Ahl's Simple Benchmark

[Dean Sadites]

10 '  Ahl's Simple Benchmark
20 FOR N=1 TO 100: A=N
30 FOR I=1 TO 10
40 A=SQR(A): R=R+RND(1)
50 NEXT I
60 FOR I=1 TO 10
70 A=A^2: R=R+RND(1)
80 NEXT I
90 S=S+A: NEXT N
100 PRINT ABS(1010-S/5)
110 PRINT ABS(1000-R)

In November 1983, Creative Computing magazine published a simple benchmark
test (shown above) that they used whenever they reviewed new personal
computers. In the January 1984 issue, the editor and author of the benchmark,
David H. Ahl, stated that "the Creative Computing benchmark is a short test of
computational speed, accuracy, and the random generator in Basic. We have
taken note of the criticisms of this simple test and are in the process of devising
a more comprehensive one." Several months later, they concluded that the
revised benchmarks weren't any more accurate and decided to stick with the
original.

However, what makes this benchmark test interesting is not its effectiveness or
suitability as a benchmark, but rather, the amount of historical data available
enabling us to compare current systems with those from 20 years ago. Back
then, virtually all of the tests were conducted against BASIC interpreters that
shipped with most personal computers. I've listed some of the more popular
systems below, along with some tests I conducted myself using six popular
BASIC compilers, including VB.Net. The compiled executables were run on an
old Compaq Armada 1750 (Pentium II 366 MHz), and the timings were averaged
over 5000 iterations. So far, PB/DLL is the clear winner. (Note that I have
dispensed with the accuracy and random number generator results.)

I'd be interested in seeing results from C++, Delphi and PB/DLL with inline
assembly. Of course, they would need to be run on the same system used for
the other compiler tests. I'm thinking of putting up a Web page where the
different compiled executables could be downloaded, so that the entire batch
of tests could be run against a common system. The page could provide a
database of previous results and allow new result sets to be submitted as
faster systems (and compilers) are introduced.

Compiled                  dd:hh:mm:ss
  
PB/DLL 6.1                       0:00.00145
VB.Net 7.0                       0:00.00377
Visual Basic 6.0                 0:00.00482
FirstBasic 1.0                   0:00.00227
PowerBASIC 3.2                   0:00.00244
QuickBASIC 4.5                   0:00.00873
  
  
Interpreted               dd:hh:mm:ss
  
Pentium 90 (GWBASIC)             0:00.13
Cray 1                           0:00.01
DEC Rainbow 100                  0:20
IBM PC (4.77 MHz)                0:24
Coleco Adam                      0:47
Macintosh                        1:36
Apple III                        1:48
Vic 20                           1:49
Commodore 64                     1:53
Apple II+/e                      1:53
Timex/Sinclair 1000 (fast mode)  2:43
TI 99/4A                         3:46
Sinclair Spectrum                4:39
TRS-80 Model 100                 4:54
Atari 400/800                    6:48
Timex/Sinclair 1000 (slow mode) 16:55
HP-11C Calculator               30:34
TI SR-50 Calculator       12:17:00:00

[This message has been edited by Dean Sadites (edited February 17, 2002).]

[Dean Sadites]

FYI, here's the PB code that was used:

#compile exe

Function PBMain()
  Local x As Integer
  Local n As Integer
  Local i As Integer
  Local a As Single
  Local r As Single
  Local s As Single
  Local t As Single

  t = Timer
  For x = 1 To 5000
    ' Main benchmark
    For n = 1 To 100
      a = n
      For i = 1 To 10
        a = Sqr(a)
        r = r + Rnd(1)
      Next

      For i = 1 To 10
        a = a ^ 2
        r = r + Rnd(1)
      Next
      s = s + a
    Next

  Next

  MsgBox "Time: " & Format$((Timer - t)/5000, "0.00000") & " seconds"
End Function

And here's the VB.Net code that was used:

Imports System
Imports System.Math
Imports Microsoft.VisualBasic

Public Module Ahl
  Public Sub Main()
    Dim x As Integer
    Dim n As Integer
    Dim i As Integer
    Dim a As Single
    Dim r As Single
    Dim s As Single
    Dim t As Double

    t = Timer
    For x = 1 To 5000

      ' Main benchmark
      For n = 1 To 100
        a = n
        For i = 1 To 10
          a = Sqrt(a)
          r = r + Rnd(1)
        Next

        For i = 1 To 10
          a = a ^ 2
          r = r + Rnd(1)
        Next
        s = s + a
      Next

    Next
    Console.WriteLine("Time: " & Format((Timer - t) / 5000, "0.00000") & " seconds")
  End Sub
End Module

[This message has been edited by Dean Sadites (edited February 17, 2002).]

[Lance Edmonds]

In those days, CPU differences were of little importance and small variance, but today, the differences in performance for certain types of code can vary dramatically according to the brand and design of the processor, FPU, RAM/Cache combination, etc.

Or to put it another way, such tests are of interest value only... they serve little real point.

Search the forum for "benchmark" and you'll find these discussions time and time again, often with "interesting" and occasionally unexpected results...


------------------
Lance
PowerBASIC Support
mailto:[email protected][email protected]</A>

[Steve Hutchesson]

Dean,

I knew that HP 11c was a leading edge toy in its time, still runs
on its original set of batteries and I bought it in 1984. Damn it
was slow though but it certainly beat crunching numbers by hand.

Regards,

[email protected]

------------------

[Dean Sadites]

Lance Edmonds wrote:

> In those days, CPU differences were of little importance,

I'm not really sure what you're saying here; so I'll let you
elaborate before I respond.


> but today, the differences in performance for certain types
> of code can vary greatly according to the brand and design
> of the processor, FPU, RAM/Cache combination, etc.

I think that's always been true. No benchmark test shows
anything other than what the code is specifically testing.
This often favors one processor (or interpreter, compiler,
etc.) over another by taking advantage of special features
or optimizations offered, such as PB's optimization of
For/Next loops as pointed out by Steve Hutchesson in another
thread.


> Or to put it another way, such tests are of interest value
> only...

Of course, they're also good for plugging a product, like the
100,000,000-iteration floating-point loop used to show that
PB/DLL is 2300% faster than VB
http://www.powerbasic.com/products/pbdll32/

Dean


------------------

[Lance Edmonds]

No elaboration needed. Did you do a forum search for "benchmark"?

------------------
Lance
PowerBASIC Support
mailto:[email protected][email protected]</A>

[Tom Hanlin]

That bit of code was written back in the days when it could take a fair
sequence of CPU instructions just to add two 16-bit numbers. At the
time, handling SQR, RND, and ^ put quite a load on a machine. With modern
CPUs and their built-in floating point units, Ahl's little test is about
worthless. It was also designed for interpreters-- many modern
compilers have special tests for simple FOR..NEXT loops of this kind,
employing special-purpose optimizations that do nothing more or less
than make the compiler look better on such simple-minded benchmarks,
without improving the performance of real code in the slightest. As
PowerBASIC compilers don't bother with such tricks, they may well
appear slower than compilers that were written to fool benchmarks.

------------------
Tom Hanlin
PowerBASIC Staff

[David J Walker]

Benchmark fiddling has been going on for a long time.
Back in the late '80s there was a compiler (JPI as I recollect) that had a facility for spotting a Sieve of Eratosthenes, which was a common test round about then, and running it in optimised machine language.

------------------

[Steve Hutchesson]

Tom has a good point here in that some of the current compilers
perform very simplistic forms of optimisation such as short
circuiting unnecessary loops, early exit code modifications and
similar techniques that only cover up very bad coding styles.

Sad to say there are no easy tricks when the code becomes even slighlty
more complex and in a real world situation where highly complex
code is normal, this type of optimisation is effectively worthless.

I have always commended Bob Zale's approach of manually optimising
the different functions for basic directly in assembler as it
continues to deliver the goods in performance terms across many
different styles of code written in basic.

It is normal for different languages to have different advantages
in what they do well and this will at times show up with different
types of code designs performing better or worse depending on the
language they are written with but cross compiler benchmarking is
a lot more complex than some may imagine.

The only useful comparisons are task specific ones. Write the same
capacity in two different languages and clock what they both do in
real time. The results tell the story, the rest is rhetoric.

Regards,

[email protected]

------------------