Announcement

Collapse
No announcement yet.

What is the fastest PB command to read data at a position in a large string

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • What is the fastest PB command to read data at a position in a large string

    I am doing a 3D migration (of a ground penetrating radar volume) and am trying to discover what the bottlenecks are in the processing time. The only command in the code that i can see that might be a bottleneck (?) is a statement dum=ASC(file3d$,imptr)*256+ASC(file3d$,imptr+1) embedded in a 3D for loop. Is this the fastest way to get data from large character string (written as 16 bit big endian). The code is working but it is just slower than i thought it should be. Thanks for any advice.

  • #2
    A pointer and a single PEEK for each pair of bytes should be a lot quicker.
    Something along the lines of

    Code:
    DIM ptr as DWORD
    ...
    FOR ptr = STRPTR(file3$) TO  STRPTR(file3$) + LEN(file3d$) - 1. STEP 2
    ...
    dum = PEEK(WORD,ptr)
    ...
    NEXT

    Comment


    • #3
      Stuart knows what he is doing.
      It is probably the CVI(file3d$) or CVI(reverse(file3d$))
      If it is the reverse of the 2-bytes there is probably a function for it.
      Use [ code ] [ /code ] in source code before pasting to keep indentation

      Comment


      • #4
        Dean,
        If you have to reverse the word endianness then I would go
        with a TYPE/UNION that make it easy using pointer

        Code:
        #COMPILE EXE '#Win#
        #DIM ALL
        #INCLUDE "Win32Api.inc"
        
        TYPE WordByteType BYTE
         b0 AS BYTE
         b1 AS BYTE
        END TYPE
        
        UNION WordUnion
         WordByteType
         w AS WORD
        END UNION
        '_____________________________________________________________________________
        
        FUNCTION PBMAIN() AS LONG
         LOCAL sFile AS STRING
         LOCAL pb    AS BYTE POINTER
         LOCAL wVal  AS WordUnion
        
         sFile = MKWRD$(&h3412) & MKWRD$(&h7856) & MKWRD$(&hCDAB) 'Two bytes word
        
         pb = STRPTR(sFile) 'For big-endian
         wVal.b0 = @pb[1]   'Test only on the first word of the "file", most signifianr byte
         wVal.b1 = @pb[0]   'Test only on the first word of the "file", least signifianr byte
        
         MessageBox(%HWND_DESKTOP, "Little-endian 0x1234 = 0x" & HEX$(wVal.w), "Word & endian", %MB_OK OR %MB_TOPMOST)
        
        END FUNCTION
        '_____________________________________________________________________________
        '

        Comment


        • #5
          And if endianness is alright, I'd go wit a word pointer...

          Code:
          #COMPILE EXE '#WIN#
          #DIM ALL
          #INCLUDE "Win32Api.inc"
          '_____________________________________________________________________________
          
          FUNCTION PBMAIN() AS LONG
           LOCAL sFile AS STRING
           LOCAL pw    AS WORD POINTER
          
           sFile = MKWRD$(&h1234) & MKWRD$(&h5678) & MKWRD$(&hABCD) 'Two bytes word
           pw = STRPTR(sFile)
           MessageBox(%HWND_DESKTOP, "Word 0 0x" & HEX$(@pw[0]) & $CRLF & _
                                     "Word 1 0x" & HEX$(@pw[1]) & $CRLF & _
                                     "Word 2 0x" & HEX$(@pw[2]), "Word pointer", %MB_OK OR %MB_TOPMOST)
          END FUNCTION
          '_____________________________________________________________________________
          '

          Comment


          • #6
            Originally posted by Mike Doty View Post
            Stuart knows what he is doing.
            It is probably the CVI(file3d$) or CVI(reverse(file3d$))
            If it is the reverse of the 2-bytes there is probably a function for it.
            CVI is probably slower because PB internally will need to resolve file3d$ to a memory location on every iteration. Getting the STRPTR once and going directly to the memory location each time will involve less cycles.

            Comment


            • #7
              Test of PEEK v PTR on a 100MB string.
              This test has PTR at 160 million TIX, PEEK at 120 million TIX.
              If anyone can see a way to speed up PtrTest while remaining comparable to PeekTest, please contribute.
              Code:
              #COMPILE EXE
              #DIM ALL
              
              FUNCTION PBMAIN() AS LONG
                  ? "Pointer: " & STR$(PtrTest)
                  ? "Peek: " & STR$(PeekTest)
              END FUNCTION
              
              FUNCTION PtrTest () AS QUAD
                  LOCAL x,lngResult AS LONG
                  LOCAL p AS WORD PTR
                  LOCAL s AS STRING
                  LOCAL q AS QUAD
                  s = STRING$(100000000,"A")
                  p = STRPTR(s)
                  TIX q
                  FOR x = 0 TO FIX(LEN(s)/2) -1
                      lngResult = @p[x]
                  NEXT
                  TIX END q
                  FUNCTION = q '
              END FUNCTION
              
              FUNCTION PeekTest () AS QUAD
                  LOCAL x,lngResult AS LONG
                  LOCAL p AS DWORD
                  LOCAL s AS STRING
                  LOCAL q AS QUAD
                  s = STRING$(100000000,"A")
                  p = STRPTR(s)
                  TIX q
                  FOR x = p TO p + LEN(s) -2 STEP 2
                      lngResult = PEEK(WORD,x)
                  NEXT
                  TIX END q
                  FUNCTION = q
              END FUNCTION

              Comment


              • #8
                For comparison, replacing
                PEEK(WORD,x)
                with
                ASC(s,x)*256+ASC(s,x+1)
                resulted in TIX going from 120 million to 3,483 million- i.e. the OPs original code takes about 30 times as long as the PEEK alternative

                Comment


                • #9
                  Stuart,
                  You are right, PEEK() is significantly faster than pointer here.
                  -
                  Dean,
                  If your file is made of big endian word
                  then an assembly function to swap word's bytes before processing might be usefull...

                  See...
                  Last edited by Pierre Bellisle; 18 Mar 2020, 12:48 PM.

                  Comment


                  • #10
                    Also another way to reverse the byte order one word at a time is to use ROTATE
                    Code:
                    #COMPILE EXE '#Win#
                    #INCLUDE "Win32Api.inc"
                    '_____________________________________________________________________________
                    '
                    FUNCTION PBMAIN() AS LONG
                     LOCAL sW1     AS STRING
                     LOCAL index   AS DWORD
                     LOCAL wordVal AS WORD
                    
                     sW1 = "BADCFEHGJI" '10 bytes, 5 word
                    
                     FOR index = STRPTR(sW1) TO STRPTR(sW1) + LEN(sW1) - 2 STEP 2
                       wordVal = PEEK(WORD, index)
                       ROTATE LEFT wordVal, 8
                       MessageBox(%HWND_DESKTOP, "0x" & HEX$(wordVal, 4) & $CRLF & _
                                  MKWRD$(wordVal), "Word peek", %MB_OK OR %MB_TOPMOST)
                     NEXT
                    
                    END FUNCTION
                    '_____________________________________________________________________________
                    '

                    Comment


                    • #11
                      Originally posted by Pierre Bellisle View Post
                      If your file is made of big endian word
                      then an assembly function to swap word's bytes before processing might be usefull...
                      Yes, unfortunately
                      PEEK(WORD,x) and @p(x) do not return the same values as ASC(s,x)*256+ASC(s,x+1) , you do need to reverse the bytes

                      This one takes twice as long (235 million TIX) but it's still 15 times as fast as the OP's ASC version (and combines pointers and PEEKS )
                      Code:
                      FUNCTION PeekTest2 () AS QUAD
                      LOCAL x,lngResult AS LONG
                      LOCAL p AS DWORD
                      LOCAL s AS STRING
                      LOCAL q AS QUAD
                      LOCAL lp AS BYTE PTR
                      s = STRING$(100000000,"A")
                      p = STRPTR(s)
                      lp = VARPTR(lngResult)
                      TIX q
                      FOR x = p TO p + LEN(s) -2 STEP 2
                          @lp[1] = PEEK(BYTE,x)
                          @lp[0] = PEEK(BYTE,x+1)
                      NEXT
                      TIX END q
                      FUNCTION = q
                      END FUNCTION
                      Note, I tried it with POKE instead of pointers, but that was slower.
                      Code:
                      lp1 = lp + 1
                      
                      ...
                      POKE BYTE,lp1, PEEK(BYTE,x)
                      POKE BYTE,lp,PEEK(BYTE,x+1)
                      Came in at 350 million ticks i.e. about 50% slower than @lp[1][email protected][0]

                      Comment


                      • #12
                        Originally posted by Pierre Bellisle View Post
                        Also another way to reverse the byte order one word at a time is to use ROTATE
                        Code:
                        #COMPILE EXE '#Win#
                        #INCLUDE "Win32Api.inc"
                        '_____________________________________________________________________________
                        '
                        FUNCTION PBMAIN() AS LONG
                        LOCAL sW1 AS STRING
                        LOCAL index AS DWORD
                        LOCAL wordVal AS WORD
                        
                        sW1 = "BADCFEHGJI" '10 bytes, 5 word
                        
                        FOR index = STRPTR(sW1) TO STRPTR(sW1) + LEN(sW1) - 2 STEP 2
                        wordVal = PEEK(WORD, index)
                        ROTATE LEFT wordVal, 8
                        MessageBox(%HWND_DESKTOP, "0x" & HEX$(wordVal, 4) & $CRLF & _
                        MKWRD$(wordVal), "Word peek", %MB_OK OR %MB_TOPMOST)
                        NEXT
                        
                        END FUNCTION
                        '_____________________________________________________________________________
                        '
                        (Note, you need to ensure that the result is typed as a WORD or INTEGER (depending on what values the characters represent). I initially made the mistake of leaving it as LONG, and that didn't work with ROTATE LEFT 8 )
                        For the other versions, WORD and LONG return the same results, only INTEGER returns negative values.

                        This came in at at about 680 million TIX versus Post #11's, 235 million TIX, i.e. about three times as long

                        Looks like Post #11 is the current best solution.
                        Last edited by Stuart McLachlan; 12 Mar 2020, 07:26 AM.

                        Comment


                        • #13
                          Interesting. On my system using Stuart's post #7 the pointer method is fairly stable while the PEEK method varies quite a bit. At any rate the POINTER method is typically faster than the PEEK method.

                          Initial Results:
                          POINTER test TIX count 163,494,078
                          PEEK test TIX count 179,854,840

                          Results after replacing PEEK(WORD, x) with ASC(s,x)*256+ASC(s,x+1):
                          POINTER test TIX count 176,148,357
                          ^PEEK test TIX count 3,139,356,330

                          Comment


                          • #14
                            Here is another one based on a union/type...

                            Click image for larger version

Name:	EndianWord.png
Views:	213
Size:	10.4 KB
ID:	790849

                            Code:
                            #COMPILE EXE '#Win#
                            #INCLUDE "Win32Api.inc"
                            
                            TYPE WordByteType BYTE
                             b0 AS BYTE
                             b1 AS BYTE
                            END TYPE
                            
                            UNION WordUnion
                             WordByteType
                             w AS WORD
                            END UNION
                            '_____________________________________________________________________________
                            
                            FUNCTION WordEndianUnion() AS QUAD
                             LOCAL x          AS LONG
                             LOCAL wordResult AS WordUnion
                             LOCAL p          AS DWORD
                             LOCAL s          AS STRING
                             LOCAL q          AS QUAD
                             LOCAL bp         AS BYTE POINTER
                            
                             s = STRING$(100000000,"A")
                             p = STRPTR(s)
                             bp = VARPTR(wordResult)
                             TIX q
                             FOR x = p TO p + LEN(s) -2 STEP 2 'Set wordResult.w
                               wordResult.b0 = PEEK(BYTE, x)
                               wordResult.b1 = PEEK(BYTE, x + 1)
                             NEXT
                             TIX END q
                             FUNCTION = q
                            
                            END FUNCTION
                            '_____________________________________________________________________________
                            
                            FUNCTION PeekTest2() AS QUAD
                             LOCAL x          AS LONG
                             'LOCAL lngResult AS LONG - Used a word if more handling have to be done later in code, change is in line below.
                             LOCAL wordResult AS WORD
                             LOCAL p          AS DWORD
                             LOCAL s          AS STRING
                             LOCAL q          AS QUAD
                             LOCAL bp         AS BYTE POINTER
                            
                             s = STRING$(100000000,"A")
                             p = STRPTR(s)
                             bp = VARPTR(wordResult)
                             TIX q
                             FOR x = p TO p + LEN(s) -2 STEP 2 'Set wordResult
                               @bp[1] = PEEK(BYTE, x)
                               @bp[0] = PEEK(BYTE, x + 1)
                             NEXT
                             TIX END q
                             FUNCTION = q
                            
                            END FUNCTION
                            '_____________________________________________________________________________
                            
                            FUNCTION PBMAIN() AS LONG
                            
                             MessageBox(%HWND_DESKTOP, _
                             "PeekTest2" & $TAB & $TAB & FORMAT$(PeekTest2(), "0,") & $CRLF & _
                             "WordEndianUnion"  & $TAB & FORMAT$(WordEndianUnion(), "0,"), _
                             "Endian word", %MB_OK OR %MB_TOPMOST)
                            
                            END FUNCTION
                            '_____________________________________________________________________________
                            '

                            Comment


                            • #15
                              Sorry for the late response... Pierre, Stuart, Mikes Thanks for all the posts here and depth of information... Will implement!

                              Comment


                              • #16
                                Using PB_BenchMark version 7, I found that the PEEK test is 1.1 times faster than the POINTER test when compared in 10,000 iterations.

                                The s string was made a global and assigned outside the test threads.

                                NOTE:
                                Changes to PB_BenchMark 7
                                GLOBAL Test1 AS DOUBLE 'changed 2020-MAR-12
                                GLOBAL Test2 AS DOUBLE 'changed 2020-MAR-12

                                LOCAL FinalDet AS DOUBLE 'changed 2020-MAR-12

                                Variance = 0.0000050 ' changed 2020-03-12 for POINTER and PEEK test

                                Due to the 10,000 iterations of 50,000,000 loops it took several minutes to get results.

                                Comment


                                • #17
                                  Originally posted by Pierre Bellisle View Post
                                  Here is another one based on a union/type...
                                  ...
                                  We have a winner! Pierre's post #14 using UNION is considerable faster than the previous best post #11 which used PTR/PEEK
                                  On my machine it comes in around 160 milllion compared to the previous 235 million

                                  We have a solution which is 20 times as fast as the original ASC() + ASC() method.

                                  Comment


                                  • #18
                                    Using PB_BenchMark version 7, I found that the original POINTER test is 1.4 times faster than the new Pierre PEEK test when compared in 10,000 iterations. Which means that Stuart's original PEEK test is 1.1 * 1.4 = 1.54 times faster than Pierre's new PEEK test.

                                    Original POINTER test 1.4 times faster than Pierre's PEEK test
                                    Code:
                                    FOR x = 0 TO FIX(LEN(s)/2) -1
                                        lngResult = @p[x]
                                    NEXT
                                    Stuart's Original PEEK test is 1.1 times faster than original POINTER test
                                    Code:
                                    FOR x = p TO p + LEN(s) -2 STEP 2
                                        lngResult = PEEK(WORD,x)
                                    NEXT
                                    Pierre's new PEEK test
                                    Code:
                                     
                                    FOR x = p TO p + LEN(s) -2 STEP 2 'Set wordResult @bp[1] = PEEK(BYTE, x) @bp[0] = PEEK(BYTE, x + 1) NEXT

                                    Comment


                                    • #19
                                      Originally posted by Jim Fritts View Post
                                      Using PB_BenchMark version 7, I found that the original POINTER test is 1.4 times faster than the new Pierre PEEK test when compared in 10,000 iterations. Which means that Stuart's original PEEK test is 1.1 * 1.4 = 1.54 times faster than Pierre's new PEEK test.
                                      Correct. 160 million versus 120 million on the 100MB string.
                                      The problem is that you are comparing apples to oranges
                                      Your missed the point that the first PTR and PEEK tests were just reading two bytes.
                                      The last two test also reverse those big-endian bytes .

                                      You can't compare the two sets. Pierre's final version is the fastest to do what is required i.e. get 2 x Big-endian byte values into a PB WORD or INTEGER

                                      Comment


                                      • #20
                                        Only the stated FOR routines in #18 were compared. Nothing else. I believe they all share one thing in common. 50,000,000 loops.

                                        Comment

                                        Working...
                                        X