Announcement

Collapse
No announcement yet.

Performance Benchmark Program

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Performance Benchmark Program

    For anyone interested in an interesting "benchmark performance" program, IBM has posted a specification, test files and expected output at:

    http://www2.hursley.ibm.com/decimal/telco.html

    Summary from that page:
    This benchmark specifies a small program which is intended to capture the essence of a telephone company billing application, with a realistic balance between Input/Output activity and application calculations.

    This is expected to allow

    - the comparison of Input/Output time with calculation time
    - the investigation of different implementation approaches.
    This might be a fun thing for those interested in performance issues.

    Since this is a "batch" application, the specs do not include any information about any kind of user interface, such as a "browse for input/output files" or a "progress bar" but adding those might be fun, too.

    In my opinion the important thing about this is that IBM has designed this benchmark application, "with a realistic balance between Input/Output activity and application calculations." That is, this is not one of the "do-nothing loop" programs which are sometimes posted here as examples of "performance" benchmark programs.

    I'm not so sure those who choose to participate should all post all their code, but for that perhaps we could ask the Forum hosts?

    Any Ideas for somehow collecting and making available the various solutions from which others may learn? I suppose I could make my web site available but about all I could do is load files out there, since I do not have a clue how create a decent HTML interface for others to browse... I was thinking something like a page such as...

    Code:
       IBM Benchmark Program
       
       (Text with link to benchmark specs and explanation)
    
    
                                  Solutions
    
       Name         Hardware     Timing Info     Link to Source Code file
       -----        -----------   -----------     ------------------------
       John Smith   PII/400       10.8 sec         <link to something.bas>
       Mary Jones   PIV/1.8Ghz     5.2 sec         <link to something.bas>
       Pat Johnson  PII/800        8.7 sec         <link to something.bas>
    Comments?

    ------------------
    Michael Mattias
    Tal Systems Inc.
    Racine WI USA
    mailto:[email protected][email protected]</A>
    www.talsystems.com



    [This message has been edited by Michael Mattias (edited May 11, 2004).]
    Michael Mattias
    Tal Systems (retired)
    Port Washington WI USA
    [email protected]
    http://www.talsystems.com

  • #2
    Source code may be posted in the Source Code forum. Large programs
    may be sent to mailto:[email protected][email protected]</A> for inclusion in the Downloads
    section. Please be sure to note whether your code is copyrighted or
    in the public domain (it can't be both). If copyrighted, you must
    describe the conditions under which the code may be distributed. If
    public domain, you've given your code to the world at large, and can
    not restrict the distribution in any way.

    ------------------
    Tom Hanlin
    PowerBASIC Staff

    Comment


    • #3
      IMHO, that bench is more a binary FP vs decimal test case.
      Have you tried implementing it in PowerBASIC and getting the right results for the million records data set?

      sumT = 1004737.58
      sumB = 57628.30
      sumD = 25042.17

      Bye!


      ------------------
      Try TrID file identifier! Over 1070 filetypes and counting...
      Give a powerfull scriptable Lua interface to your application with PowerBLua
      PBTracer - Tracer / Profiler for PowerBASIC (soon!)
      -- The universe tends toward maximum irony. Don't push it.

      File Extension Seeker - Metasearch engine for file extensions / file types
      Online TrID file identifier | TrIDLib - Identify thousands of file formats

      Comment


      • #4
        ok, i did a version.

        http://www.powerbasic.com/support/pb...ad.php?t=24241

        mcm

        Michael Mattias
        Tal Systems (retired)
        Port Washington WI USA
        mma[email protected]
        http://www.talsystems.com

        Comment


        • #5
          This is interesting and will take me more time to digest, but in the mean time
          here is what the output is showing for the large file:

          SumT 1,004,737.58
          SumB 57,628.30
          SumD 25,042.17
          Processed 1,000,000 Records in 21.49 Seconds

          PBWIN 7.04 on PIV @1.5 & 650MB - W2K (SP4)

          EDIT: 20040521:
          Correct file found from Michael's reference. New Results Shown.

          ------------------
          Roger...
          (Mail to rdrines at SpamCop dot Net)



          [This message has been edited by Roger Rines (edited May 21, 2004).]
          Roger...

          Comment


          • #6
            Did you use the correct test file? You want the BINARY file, not the BCD.

            You want the one in this archive:
            http://www2.hursley.ibm.com/decimal/expon180-1e6b.zip

            You will note on the page http://www2.hursley.ibm.com/decimal/telcoSpec.html
            In either case, if sumD is unexpectedly 0 then probably the BCD data file is being used when the testcase expects the binary data file.


            [This message has been edited by Michael Mattias (edited May 20, 2004).]
            Michael Mattias
            Tal Systems (retired)
            Port Washington WI USA
            [email protected]
            http://www.talsystems.com

            Comment


            • #7
              Originally posted by Michael Mattias:
              OK, I did a version.
              On my system (Athlon XP 2200+) this take 7.50sec; the reference implementation in C# 2.0sec.
              It's to be noted that .NET FileStream & StreamWriter does buffer the I/O.

              Bye!

              ------------------
              Try TrID file identifier! Over 1070 filetypes and counting...
              Give a powerfull scriptable Lua interface to your application with PowerBLua
              PBTracer - Tracer / Profiler for PowerBASIC (soon!)



              [This message has been edited by Marco Pontello (edited May 21, 2004).]
              -- The universe tends toward maximum irony. Don't push it.

              File Extension Seeker - Metasearch engine for file extensions / file types
              Online TrID file identifier | TrIDLib - Identify thousands of file formats

              Comment


              • #8
                >On my system (Athlon XP 2200+) this take 7.50sec

                An interesting note I got from Mike Cowlishaw of IBM / UK as a response to my note that although the benchmark was written primarily for COBOL programmers...

                Point of the benchmark is *not* to measure language A against language B (or OpSys X against Y). It is: here's a problem, here's the data as it is usually presented (2 options), code it up in the most maintainable way, what's the result.
                (He mentioned the operating system because I pointed out that the test file stores the call duration in a format called "COMP-5" .. which is supposed to be "binary integers with the MSB-LSB ordering native to the operating system." As you note in the code, I had to change the order of the bytes because the test file was produced on an IBM mainframe which does not store binary integers in Intel order. I guarantee this would run a whole lot faster if the call duration were stored in a 32-bit Intel-ordered long integer, which would hold a call duration of more than 68 years. But, in real life, we encounter "weird" choices for data storage every day, and the condition of the specification was that duration is a maximum 15-decimal digit value, and the data come off an IBM mainframe).

                Again, I posted this because I've seen posted here too many "do nothing" loops which are posited as 'benchmarks.'

                MCM

                Michael Mattias
                Tal Systems (retired)
                Port Washington WI USA
                [email protected]
                http://www.talsystems.com

                Comment


                • #9
                  I read the note on the site, and also read many comments about that "telco bench" on the newsgroups around.
                  The point, IMHO, is that this is mostly a decimal (decimal FP, instead of binary FP or fixed point) "show case".
                  Note the dist rate: the fact that's 0.00894 isn't a casual choice, nor the requested rounding.
                  Mike Cowlishaw is indeed one of the big supporter of hardware implementation of decimal FP (we have FPUs, maybe one day will have also DFPUs...).

                  I also made a PB version of the bench, with buffered input & output, that run on my system on 1.7sec.
                  But the result was nearly 20 unit above (or below? don't remember) off the correct/specified result.

                  As you have seen, you have to do some serious "hacking" to correct theese errors, at the expense of performance and (more important) code clarity and maintenability.
                  I think that's really no point in doing this "job" in this way; for this specific result, if I had to choose I will use a different dev system (one that implement decimal FP).

                  I know that's the artist, not the paintbrush... but I think a different paintbrush may be better suited!


                  Speaking of code that do something...
                  How about a code that take all the lines (an arbitrary number, let's say even 2M) passed from standard input, and write them to stadard ouput in reversed order, for example?
                  Or something that spellcheck a list of words against another list?
                  These are others jobs in witch different languages may be more or less suited...

                  Bye!

                  ------------------
                  Try TrID file identifier! Over 1070 filetypes and counting...
                  Give a powerfull scriptable Lua interface to your application with PowerBLua
                  PBTracer - Tracer / Profiler for PowerBASIC (soon!)


                  [This message has been edited by Marco Pontello (edited May 21, 2004).]
                  -- The universe tends toward maximum irony. Don't push it.

                  File Extension Seeker - Metasearch engine for file extensions / file types
                  Online TrID file identifier | TrIDLib - Identify thousands of file formats

                  Comment


                  • #10
                    I also made a PB version of the bench, with buffered input & output, that run on my system on 1.7sec.
                    But the result was nearly 20 unit above (or below? don't remember) off the correct/specified result
                    Hmm, mine took longer to run, but gets the right answer.

                    I wonder: Which one of us will Mr. Donald Trump keep as his Apprentice?

                    Michael Mattias
                    Tal Systems (retired)
                    Port Washington WI USA
                    [email protected]
                    http://www.talsystems.com

                    Comment


                    • #11
                      Originally posted by Michael Mattias:
                      Hmm, mine took longer to run, but gets the right answer.

                      I wonder: Which one of us will Mr. Donald Trump keep as his Apprentice?


                      Uhm... if that was the point, I suppose I could simply compile the reference version, and use that!
                      Less development time, no testing needed, smaller exe, less time to give the right result!

                      But I "relaunch" with the reverse STDIN problem above mentioned.
                      This is a PB version:
                      Code:
                      #DIM ALL
                      
                      %InitialBufferSize = 50000
                      %BufferSizeIncrement = 100000
                      
                      FUNCTION PBMAIN
                        
                        DIM Counter AS LONG
                        DIM BufferSize AS LONG
                        DIM inpf AS LONG
                        DIM i AS LONG
                        
                        DIM t AS SINGLE, tt AS SINGLE
                        t = TIMER
                        
                        BufferSize = %InitialBufferSize
                        DIM Buffer(BufferSize) AS STRING 
                        
                        inpf = FREEFILE
                        OPEN HANDLE GETSTDIN FOR INPUT AS inpf LEN=4096
                        
                        DO WHILE NOT EOF(inpf)
                          INCR Counter
                          IF Counter > BufferSize THEN
                            BufferSize = BufferSize + %BufferSizeIncrement
                            REDIM PRESERVE Buffer(BufferSize)
                          END IF
                          LINE INPUT #inpf, Buffer(Counter)
                        LOOP
                      
                        FOR i = Counter TO 1 STEP -1
                          STDOUT Buffer(i)
                        NEXT i
                      
                        tt = TIMER
                      
                        PRINT tt-t;"sec."
                      
                      END FUNCTION
                      And this is a Lua version (that I could use inside a PB program):
                      Code:
                      t = os.clock()
                      local lines = {}
                      local numl = 0
                      
                      for l in io.lines() do
                          numl = numl + 1
                          lines[numl] = l
                      end
                      for i= numl, 1, -1 do
                          io.write(lines[i], "\n")
                      end
                      
                      tt = os.clock()
                      print(tt-t.."sec.")
                      Note that the Lua ver is faster (about half the time, for a 1M lines)...
                      But, aside from speed (one can surely equal and probably beat the Lua ver using PB, just going a little lower level, with custom memory allocation, etc.), it's simpler.

                      Bye!

                      ------------------
                      Try TrID file identifier! Over 1070 filetypes and counting...
                      Give a powerfull scriptable Lua interface to your application with PowerBLua
                      PBTracer - Tracer / Profiler for PowerBASIC (soon!)


                      [This message has been edited by Marco Pontello (edited May 21, 2004).]
                      -- The universe tends toward maximum irony. Don't push it.

                      File Extension Seeker - Metasearch engine for file extensions / file types
                      Online TrID file identifier | TrIDLib - Identify thousands of file formats

                      Comment

                      Working...
                      X