Announcement

Collapse
No announcement yet.

Optimization file/access

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Optimization file/access

    I'm making some code to compete with a C program that was made in
    linux. I know PB can smoke this program, I just need some
    optimization in my code. Here are the rules of the game:

    - 10 meg text file terminated with a CR
    - I load the test file into memory and parse it (which I don't need
    help doing)
    - Then I give back the time of the completed operation

    The problem I'm running into is that this C program does all this
    on a p133 in about 14 seconds... Mine does it so far at about
    35 seconds and 10 seconds of it is just opening the file and
    putting the information in memory and then saving it into a new
    file. Any help in speeding up the opening and reading the
    information faster?

    This program will run on a p133, I was also wondering if it could
    be increased if I multi-threaded the file -> buffer -> file access?
    Code:
    #COMPILE EXE '"A:\bunk.exe"
    #INCLUDE "Win32api.inc"
    
    FUNCTION PBMAIN() AS LONG
    g$ = time$
    OPEN "C:\greg.log" FOR BINARY AS #1
    sbuffer$=STRING$(LOF(1),0)
    GET 1,,sbuffer$
    CLOSE #1
    
    ...
    ... REPLACE CHR$(32) with "," In sbuffer$
    ...
    ...
    
    OPEN "c:\greg.out" FOR OUTPUT AS #1
    CLOSE #1
    OPEN "C:\greg.out" FOR BINARY AS #1
    PUT 1,,sbuffer$
    CLOSE #1
    g$ = g$ & $CRLF & TIME$
    MSGBOX g$
    END FUNCTION
    ------------------
    -Greg

    [This message has been edited by Gregery D Engle (edited August 22, 2000).]
    -Greg
    [email protected]
    MCP,MCSA,MCSE,MCSD

  • #2
    Maybe you could open the input and output files at the same time.
    Then read in a single byte and replace it if CHR(32), and write
    it out to the open'ed file.

    I guess this would break the rules of the game (i.e. the file
    contents are not loaded into memory)?

    Maybe you could also get Windows to map the file into memory
    (using CreateFileMapping - somehow!) and then read from the
    memory into your output file?

    Just a few thoughts

    ------------------

    Comment


    • #3
      This function opens a 12 MB file in 0.6 sec. on my 233 MHz machine:
      Code:
      function GetFile(byval File as string, Buffer as string) as long
       
          local hFile as long
       
          err = 0 : hFile = freefile    
       
          open File for binary as hFile
          if err then exit function
          get$ hFile, lof(hFile), Buffer
          close hFile
        
          function = 1
        
      end function
      You can try to replace
      Code:
          open File for binary as hFile
      with
      Code:
          open File for binary as hFile len = 16384
      Regards
      Peter

      ------------------


      [This message has been edited by Peter P Stephensen (edited August 23, 2000).]
      [email protected]
      www.dreammodel.dk

      Comment


      • #4
        Well if were on the subject, can anyone speed this up. I had only
        5 min to put this in production about a year ago.

        I gusse I could use case (that would help)

        $COMPILE EXE
        $INCLUDE "WIN32API.INC"
        $INCLUDE "COMMCTRL.inc"
        $INCLUDE "APIC.INC"

        DECLARE FUNCTION GetAPICALLS ()AS STRING
        DECLARE FUNCTION Parse%(instring$, searchChar$, occurence%, returnString$)
        DECLARE FUNCTION Exist(BYVAL filespec AS STRING) AS LONG

        CALLBACK FUNCTION OkButton()
        CMD$ = Command$
        LOCAL n AS INTEGER
        LOCAL value AS INTEGER
        LOCAL P AS INTEGER
        LOCAL ReportName AS STRING

        If Exist(CMD$) = 0 Then
        MSGBOX "Report File " + "["+CMD$+"]" + " does not exist!", %MB_ICONSTOP, "ACI Report Cleaner VER 13.0"
        FUNCTION = 0
        Exit Function
        End If

        HoldNAME$ = EXTRACT$(CMD$, ANY ".")
        HoldNAME2$ = EXTRACT$(CMD$, ANY ".")
        Holdname$ = Holdname$ & ".FIX"
        Holdname2$ = Holdname2$ & ".REP"

        Open CMD$ For Input As #1
        Open Holdname$ For Output As #2
        Open Holdname2$ For Output As #3
        Open "return.txt" For Output As #4
        While Not EOF(1)
        Line Input #1, a$

        If Parse%(a$, "BTA400U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA402U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA409U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA410U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA110U.2", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA110U.3", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA110U.4", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA411R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA420R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA270R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA621R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA631R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA632R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA407U-A", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA425R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA621R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA631R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA635R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA636R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA640R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA656R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA666R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA250U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA250U.1", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA250U.2", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA347U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA380U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA404U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA407U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA409U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If

        '
        If Parse%(a$, "BTA440U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA440U.2", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA440U.3", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA441U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA441U.2", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA441U.3", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA442U.3", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA181R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA306R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA346R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA348R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA351R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA356R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA360R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA361R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA370R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA375R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA411R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA625R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA640R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA660", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA661R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA665", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA675R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA110U.1", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA130U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA406U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA406u-A", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA412U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA710U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA110U.1", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA111U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA110U.4", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA442U.1", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA441U.1", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA403U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA405U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA406U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA412U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA406U-A", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA414U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA400U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA665R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA641R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA130U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA710U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA720R", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If
        '
        If Parse%(a$, "BTA715U", 1, returnString$) = 0 Then
        EXIT IF
        Else
        Print #4, returnString$
        'PRINT# 4,reportname
        Print #3, Chr$(12)
        End If


        If Parse%(a$, "1DATE:", 1, returnString$) = 0 Then
        N = Len(a$)
        N = N - 1
        a$ = Right$(a$, N)
        Print #2, a$
        Print #3, a$
        Else

        Print #2, " " + returnString$
        Print #3, " " + returnString$
        End If


        Wend

        Close 4
        Close 3
        Close 2
        Close 1
        MsgBox "***** ReBuilt File: " + CMD$ + " to File: " + Holdname$ + " and " + Holdname2$ + " *****", , "Cleaner For ACI Files"
        DIALOG END CBHNDL, 1
        End Function

        CALLBACK FUNCTION CancelButton()
        DIALOG END CBHNDL, 0
        End Function

        Function Exist(ByVal filespec As String) As Long
        LOCAL hDir AS LONG
        LOCAL FindData AS WIN32_FIND_DATA
        FindData.dwFileAttributes = %FILE_ATTRIBUTE_DIRECTORY
        hDir = FindFirstFile(ByVal StrPtr(filespec), FindData)
        IF hDir = %INVALID_HANDLE_VALUE THEN
        Exit Function 'file not found
        End If
        FUNCTION = %TRUE
        FindClose hDir
        End Function

        Function Parse%(instring$, searchChar$, occurence%, returnString$)
        pntr% = 1
        For i% = 1 To occurence%
        pntr% = InStr(pntr%, instring$, searchChar$)
        If pntr% = 0 Then Exit For
        INCR pntr%
        Next i%
        If pntr% = 0 Then
        FUNCTION = 0
        Exit Function
        End If
        endPos% = InStr(pntr% + 1, instring$, searchChar$)
        If endPos% = 0 Then
        returnString$ = Mid$(instring$, pntr%)
        Else
        returnString$ = Mid$(instring$, pntr%, endPos% - pntr%)
        End If
        FUNCTION = 1
        End Function

        Function PBMAIN()
        #REGISTER NONE
        LOCAL hDlg AS LONG
        LOCAL result AS LONG
        pea459$ = GetAPICALLS()

        If Date$ > pea459$ Then
        MsgBox "ERROR: File Error", , "ERROR:"
        Exit Function
        Else

        End If

        DIALOG NEW 0, "ACI Cleaner", ,, 160, 50, 0, 0 TO hDlg

        ' ** Add controls to it
        CONTROL ADD LABEL, hDlg, 400, "Please Wait Cleaning ACI File...", 14, 12, 134, 12, 0
        CONTROL ADD BUTTON, hDlg, %IDOK, "OK", 34, 32, 40, 14, _
        %BS_DEFAULT CALL OkButton
        CONTROL ADD BUTTON, hDlg, %IDCANCEL, "Cancel", 84, 32, 40, _
        14, 0 CALL CancelButton
        DIALOG SHOW MODAL hDlg TO result
        End Function





        ------------------
        Inv. Mark Nelson

        Comment


        • #5
          Mark - Please put the code brackets around your code [ code] [ /code]

          DorianDarby - Opening and changing bit and saving it to file at
          the same time won't be againt the rules, but I
          think that might slow it down actually. I think
          if I have like 100 threads all reading a section of
          a file it will be good... not sure yet.

          Peter - Thanks, I forgot about that but it still isn't as
          fast as I'm looking for. I'll come up with some home
          grown method.

          Thanks guys,


          ------------------
          -Greg
          -Greg
          [email protected]
          MCP,MCSA,MCSE,MCSD

          Comment


          • #6
            take a look at
            http://www.powerbasic.com/support/pb...ead.php?t=3924

            regards
            peter

            ------------------
            [email protected]
            www.dreammodel.dk

            Comment


            • #7
              If I understand your game, you want to find the quickest
              way to open read and close your file..?
              You dont want to change your parsing-routine (You could gain most here)
              Code:
                x%=FreeFile
                Open "myfile" for binary as #x%
                Get$ x%,LOF(x%),Buffer$
                 ..Parse your Buffer
                Put x%,1,Buffer$
                SetEof x%
                Close x%
              But I think you should write your parser in ASM to get best
              performance.


              ------------------
              Fred
              mailto:[email protected][email protected]</A>
              http://www.oxenby.se

              Fred
              mailto:[email protected][email protected]</A>
              http://www.oxenby.se

              Comment


              • #8
                Originally posted by Fred Oxenby:
                If I understand your game, you want to find the quickest
                way to open read and close your file..?
                You dont want to change your parsing-routine (You could gain most here)
                Code:
                  x%=FreeFile
                  Open "myfile" for binary as #x%
                  Get$ x%,LOF(x%),Buffer$
                   ..Parse your Buffer
                  Put x%,1,Buffer$
                  SetEof x%
                  Close x%
                But I think you should write your parser in ASM to get best
                performance.


                Fred,

                The downfall is that its taking a good 10 seconds just to read the
                file into the buffer and then save it back to the file... I
                haven't even parsed anything yet... Just to load and save takes
                10 seconds. I think if I speed that up a little it will be
                better. I think I'm going to try and multi thread it, and each
                thread will reach 1024k from the file and processes it and save
                it to a new file...

                Thanks

                ------------------
                -Greg
                -Greg
                [email protected]
                MCP,MCSA,MCSE,MCSD

                Comment


                • #9
                  In answer to Mark Nelson's request:

                  For speed, convert all integers% to longs&.

                  Since you are doing a lot of repetition of a simple task, you could forget the PARSE% function entirely.
                  Create a new array of items (I believe you have 78 of them), and do an INSTR on each item.
                  If each line only held a single item, you could exit the FOR early.

                  Code:
                  LOCAL itemNo AS LONG
                  DIM Items(1:78) AS STRING
                  Items(1) = "BTA400U"
                  Items(2) = "BTA402U"
                  '....
                  Items(78) = "BTA715U"
                  
                  '...
                  
                    WHILE NOT EOF(1)
                      LINE INPUT #1, a$
                      FOR itemNo = 1 TO 78
                        IF INSTR(a$, Items(itemNo) ) <> 0 THEN
                          PRINT #4, Items(itemNo)
                          PRINT #3, CHR$(12)
                        END IF
                      NEXT itemNo
                  
                      IF INSTR(a$, "1DATE:" ) = 0 THEN
                        N = LEN(a$)
                        N = N - 1
                        a$ = RIGHT$(a$, N)
                        PRINT #2, a$
                        PRINT #3, a$
                      ELSE
                        PRINT #2, " 1DATE:"
                        PRINT #3, " 1DATE:"
                     END IF
                  ------------------
                  [email protected]
                  :) IRC :)

                  Comment


                  • #10
                    Unless you're low on memory, reading and writing the entire file at a time
                    is pretty sure to be a best bet. You may find it faster to use GET$ and PUT$
                    rather than preallocating a string buffer. The redundant OPENs for output
                    can also be trimmed. Use a literal instead of a function in REPLACE.
                    Code:
                    #COMPILE EXE
                    FUNCTION PBMAIN() AS LONG
                        g$ = TIME$
                        OPEN "c:\greg.log" FOR BINARY AS #1
                        GET$ 1, , sBuffer$
                        CLOSE #1
                    ...
                    ... REPLACE " " WITH "," IN sBuffer$
                    ...
                        OPEN "c:\greg.out" FOR BINARY AS #1
                        SETEOF #1
                        PUT$ 1, sBuffer$
                        CLOSE #1
                        MSGBOX g$ & $CRLF & TIME$
                    END FUNCTION
                    ------------------
                    Tom Hanlin
                    PowerBASIC Staff

                    Comment


                    • #11
                      Greg,

                      I have always found GET$ to be plenty fast enough but if you are after
                      getting the maximum speed out of disk access, there may be some advantage
                      in using the API disk functions.

                      CreateFile()
                      ReadFile()
                      WriteFile()
                      SetEndOfFile()
                      CloseHandle()

                      It has been my experience that OLE string memory allocates faster than
                      functions like GlobalAlloc() so either normal basic string or directly
                      calling,

                      SysAllocStringByteLen()
                      SysFreeString()

                      should get your speed up.

                      I have a sneaking suspicion that,

                      CoTaskMemAlloc()
                      CoTaskMemRealloc()
                      CoTaskMemFree()

                      are faster again but they appear to be a bit more complicate to use.

                      The disk IO times you mentioned sound reasonably slow which just may be a
                      limitation of the actual hardware. What I have no data on is the IO speed
                      difference between LINUX and WINDOWS. This is a factor that is probably
                      not within your control.

                      Good luck with the quest.

                      [email protected]

                      ------------------
                      hutch at movsd dot com
                      The MASM Forum

                      www.masm32.com

                      Comment


                      • #12
                        I found a BYTE pointer better at replacing a single character than Replace.
                        Give something like this a shot.
                        Code:
                        Function PBMain
                          Register l As Long
                          Dim a$
                          Dim bp As Byte Ptr
                          Dim tt As Single
                          Dim t  As Single
                          'Create a worst case 10mb file
                          Open "D:\HexStuff.YYY" For output As #1
                          Print #1, String$(1024000*10, 32)
                          Close 1
                         
                          tt = timer
                          Open "D:\HexStuff.YYY" For Binary As #1
                          Get$ 1, Lof(1), a$
                          t = timer
                              'Replace " " With "," in a$
                         
                          bp = StrPtr(a$)
                          For l = 0 To Len(a$)-1
                            If @bp[l] = 32 Then @bp[l] = 44
                          Next l
                          t = timer - t  
                          Seek #1,1
                          Put$ 1, a$
                          Close 1
                          
                          tt = timer - tt
                          Print "Total Elapsed Time:";tt;"  Character Substitution Time:";t
                        End Function
                        ------------------
                        Ron

                        [This message has been edited by Ron Pierce (edited August 24, 2000).]

                        Comment


                        • #13
                          Note that you should avoid using indexed pointers where you can use
                          just plain pointers, if you want the best speed. Instead of this:
                          Code:
                            For l = 0 To Len(a$)-1
                              If @bp[l] = 32 Then @bp[l] = 44
                            Next l
                          Try this:
                          Code:
                            For l = 0 To Len(a$)-1
                              If @bp = 32 Then @bp = 44
                              Incr bp
                            Next l

                          ------------------
                          Tom Hanlin
                          PowerBASIC Staff

                          Comment


                          • #14
                            Originally posted by Tom Hanlin:
                            Note that you should avoid using indexed pointers where you can use
                            just plain pointers, if you want the best speed. Instead of this:
                            Code:
                              For l = 0 To Len(a$)-1
                                If @bp[l] = 32 Then @bp[l] = 44
                              Next l
                            Try this:
                            Code:
                              For l = 0 To Len(a$)-1
                                If @bp = 32 Then @bp = 44
                                Incr bp
                              Next l

                            Why are index pointers slower then regular pointers?

                            ------------------
                            -Greg
                            -Greg
                            [email protected]
                            MCP,MCSA,MCSE,MCSD

                            Comment


                            • #15
                              Greg,

                              If you are doing simple BYTE replacement without changing the string
                              length, don't waste your time with pointers, indexed or otherwise, the
                              overhead is too high.

                              The SUB below does a scan of the string, if it matches the search
                              character, it replaces it, if not, it does not waste the time writing the
                              same character.

                              This one should hurry the LINUX guys up a bit.

                              [email protected]

                              Code:
                                  DECLARE SUB ReplaceByte(ByVal src as LONG, ByVal ln as LONG)
                              
                                  Open "mybigfile.ext" for binary as #1
                                    Get$ #1,lof(1),a$
                                    ReplaceByte StrPtr(a$),len(a$)
                                    Put$ #1, a$
                                  Close #1
                              
                              '##########################################################################
                              
                              SUB ReplaceByte(ByVal src as LONG, ByVal ln as LONG)
                              
                                  #REGISTER NONE
                              
                                  ! mov ecx, ln
                                  ! mov esi, src
                                  ! add ecx, esi
                              
                                lblSt:
                                  ! mov al, [esi]
                                  ! cmp al, 32        ; byte to replace
                                  ! jne nxt1
                                  ! mov al, 44        ; byte to replace it with
                                  ! mov [esi], al
                                nxt1:
                                  ! inc esi
                                  ! cmp esi, ecx
                                  ! jne lblSt
                              
                              END SUB
                              
                              '##########################################################################
                              ------------------
                              hutch at movsd dot com
                              The MASM Forum

                              www.masm32.com

                              Comment


                              • #16
                                Originally posted by Steve Hutchesson:
                                Greg,

                                If you are doing simple BYTE replacement without changing the string
                                length, don't waste your time with pointers, indexed or otherwise, the
                                overhead is too high.

                                The SUB below does a scan of the string, if it matches the search
                                character, it replaces it, if not, it does not waste the time writing the
                                same character.

                                This one should hurry the LINUX guys up a bit.

                                [email protected]

                                Code:
                                    DECLARE SUB ReplaceByte(ByVal src as LONG, ByVal ln as LONG)
                                
                                    Open "mybigfile.ext" for binary as #1
                                      Get$ #1,lof(1),a$
                                      ReplaceByte StrPtr(a$),len(a$)
                                      Put$ #1, a$
                                    Close #1
                                
                                '##########################################################################
                                
                                SUB ReplaceByte(ByVal src as LONG, ByVal ln as LONG)
                                
                                    #REGISTER NONE
                                
                                    ! mov ecx, ln
                                    ! mov esi, src
                                    ! add ecx, esi
                                
                                  lblSt:
                                    ! mov al, [esi]
                                    ! cmp al, 32        ; byte to replace
                                    ! jne nxt1
                                    ! mov al, 44        ; byte to replace it with
                                    ! mov [esi], al
                                  nxt1:
                                    ! inc esi
                                    ! cmp esi, ecx
                                    ! jne lblSt
                                
                                END SUB
                                
                                '##########################################################################
                                Steve,

                                Actually this contest does have a purpose we are converting a
                                internet proxy log into comma delimeted text and some fields
                                are space delimited within this log file so sometimes I have this:

                                replace chr$(32) & chr$(32) & chr$(32) with "," ..

                                furthermore, the first field is a unix date/time stamp and that
                                is a fixed 11 charactors. So its much harder then this

                                Thanks



                                ------------------
                                -Greg
                                -Greg
                                [email protected]
                                MCP,MCSA,MCSE,MCSD

                                Comment


                                • #17
                                  Greg,

                                  Here is try number 2. The following function reads through the string,
                                  writes the first space it finds as a comma and throws following spaces
                                  away until it finds a non space character which it writes. The penalty
                                  with this function is that you have to allocate an extra buffer which
                                  takes extra time.

                                  This will convert a string of the type,
                                  Code:
                                    a$ = "      This     is a     comma     delimited     string    "
                                    
                                        to
                                    
                                    b$ = ",This,is,a,comma,delimited,string,"
                                  If you have problems with the leading and trailing commas on each line,

                                  change$ = chr$(44,13,10,44)
                                  Replace change$ With "" In b$

                                  or variations of same will solve that problem.

                                  Regards,

                                  [email protected]

                                  Code:
                                  '##########################################################################
                                    
                                    FUNCTION RemoveSpaces(ByVal src as LONG, _
                                                          ByVal dst as LONG, _
                                                          ByVal ln as LONG) as LONG
                                    
                                        #REGISTER NONE
                                    
                                        ! mov ecx, ln
                                        ! mov esi, src
                                        ! mov edi, dst
                                        ! add ecx, esi
                                      ' ---------------------------------------------
                                      rsSt:
                                        ! mov al, [esi]     ; read byte
                                        ! inc esi
                                        ! cmp al, 32        ; search byte
                                        ! jne rsNxt
                                      ' ---------------------------------------------
                                        ! mov al, 44
                                        ! mov [edi], al
                                        ! inc edi           ; write comma
                                      ' ---------------------------------------------
                                      rsSl:
                                        ! mov al, [esi]     ; read next byte
                                        ! inc esi
                                        ! cmp esi, ecx
                                        ! je rsOut          ; exit on length match
                                        ! cmp al, 32
                                        ! je rsSl           ; loop if next byte is space
                                        ! mov [edi], al     ; write it if its not
                                        ! inc edi
                                        ! jmp rsSt          ; exit to start of main loop
                                      ' ---------------------------------------------
                                      rsNxt:
                                        ! mov [edi], al
                                        ! inc edi
                                        ! cmp esi, ecx      ; check byte count
                                        ! jne rsSt          ; if not matching, return to rsSt
                                      ' ---------------------------------------------
                                      rsOut:
                                    
                                        ! sub edi, dst      ; get byte write count
                                    
                                        ! mov FUNCTION, edi
                                    
                                    END FUNCTION
                                    
                                  '##########################################################################
                                  [This message has been edited by Steve Hutchesson (edited August 24, 2000).]
                                  hutch at movsd dot com
                                  The MASM Forum

                                  www.masm32.com

                                  Comment


                                  • #18
                                    Hutch,

                                    Whats wrong with my code:

                                    It causes GPF, and if I do a msgbox(STRPTR(Sbuffer)) it displays
                                    a number 1.34234234+E .... (overflow?) I'm lost on what I
                                    am doing wrong.

                                    Thanks,
                                    Greg

                                    Code:
                                    #COMPILE EXE
                                    GLOBAL Sbuffer$
                                    GLOBAL newBuffer$
                                    '##########################################################################
                                    
                                    FUNCTION RemoveSpaces(BYVAL src AS LONG, _
                                    BYVAL dst AS LONG, _
                                    BYVAL ln AS LONG) AS LONG
                                    #REGISTER NONE
                                    
                                    ! mov ecx, ln
                                    ! mov esi, src
                                    ! mov edi, dst
                                    ! add ecx, esi
                                    ' ---------------------------------------------
                                    rsSt:
                                    ! mov al, [esi] ; read byte
                                    ! inc esi
                                    ! cmp al, 32 ; search byte
                                    ! jne rsNxt
                                    ' ---------------------------------------------
                                    ! mov al, 44
                                    ! mov [edi], al
                                    ! inc edi ; write comma
                                    ' ---------------------------------------------
                                    rsSl:
                                    ! mov al, [esi] ; read next byte
                                    ! inc esi
                                    ! cmp esi, ecx
                                    ! je rsOut ; exit on length match
                                    ! cmp al, 32
                                    ! je rsSl ; loop if next byte is space
                                    ! mov [edi], al ; write it if its not
                                    ! inc edi
                                    ! jmp rsSt ; exit to start of main loop
                                    ' ---------------------------------------------
                                    rsNxt:
                                    ! mov [edi], al
                                    ! inc edi
                                    ! cmp esi, ecx ; check byte count
                                    ! jne rsSt ; if not matching, return to rsSt
                                    ' ---------------------------------------------
                                    rsOut:
                                    
                                    ! sub edi, dst ; get byte write count
                                    ! mov FUNCTION, edi
                                    END FUNCTION
                                    
                                    FUNCTION PBMAIN() AS LONG
                                    'DIM sbuffer$
                                    'DIM newbuffer$
                                    DIM g&
                                    DIM tt AS LONG
                                    
                                    OPEN "C:\greg.log" FOR BINARY AS #1
                                    GET$ 1,LOF(1),sbuffer$
                                    CLOSE #1
                                      tt = TIMER
                                    'MSGBOX STR$(STRPTR(sBuffer$))
                                    
                                    g& = RemoveSpaces(STRPTR(sBuffer$), STRPTR(NewBuffer$), LEN(sbuffer$))
                                    
                                      tt = TIMER - tt
                                      MSGBOX "Total Elapsed Time:" & STR$(tt) & "  Character Substitution Time:" ' & STR$(t)
                                    OPEN "c:\greg.out" FOR BINARY AS #1
                                    SETEOF #1
                                    PUT$ 1, NewBuffer$
                                    CLOSE #1
                                    END FUNCTION
                                    ------------------
                                    -Greg
                                    -Greg
                                    [email protected]
                                    MCP,MCSA,MCSE,MCSD

                                    Comment


                                    • #19
                                      Hi Greg

                                      the problem with your code is that your are passing an empty
                                      buffer (NewBuffer$) to the RemoveSpaces function. You need to
                                      allocate:

                                      NewBuffer$ = SPACE$(LEN(sBuffer$))

                                      after reading the file in sBuffer$ and before calling RemoveSpaces().

                                      Once you've done that the function works fine.

                                      Cheers

                                      Florent




                                      [This message has been edited by Florent Heyworth (edited August 25, 2000).]

                                      Comment


                                      • #20
                                        Florent,

                                        AHHH I would never had figured that out... Thanks

                                        ------------------
                                        -Greg
                                        -Greg
                                        [email protected]
                                        MCP,MCSA,MCSE,MCSD

                                        Comment

                                        Working...
                                        X