Optimization file/access

[Gregery D Engle]

I'm making some code to compete with a C program that was made in
linux. I know PB can smoke this program, I just need some
optimization in my code. Here are the rules of the game:

- 10 meg text file terminated with a CR
- I load the test file into memory and parse it (which I don't need
help doing)
- Then I give back the time of the completed operation

The problem I'm running into is that this C program does all this
on a p133 in about 14 seconds... Mine does it so far at about
35 seconds and 10 seconds of it is just opening the file and
putting the information in memory and then saving it into a new
file. Any help in speeding up the opening and reading the
information faster?

This program will run on a p133, I was also wondering if it could
be increased if I multi-threaded the file -> buffer -> file access?

#COMPILE EXE '"A:\bunk.exe"
#INCLUDE "Win32api.inc"

FUNCTION PBMAIN() AS LONG
g$ = time$
OPEN "C:\greg.log" FOR BINARY AS #1
sbuffer$=STRING$(LOF(1),0)
GET 1,,sbuffer$
CLOSE #1

...
... REPLACE CHR$(32) with "," In sbuffer$
...
...

OPEN "c:\greg.out" FOR OUTPUT AS #1
CLOSE #1
OPEN "C:\greg.out" FOR BINARY AS #1
PUT 1,,sbuffer$
CLOSE #1
g$ = g$ & $CRLF & TIME$
MSGBOX g$
END FUNCTION

------------------
-Greg

[This message has been edited by Gregery D Engle (edited August 22, 2000).]

[DorianDarby]

Maybe you could open the input and output files at the same time.
Then read in a single byte and replace it if CHR(32), and write
it out to the open'ed file.

I guess this would break the rules of the game (i.e. the file
contents are not loaded into memory)?

Maybe you could also get Windows to map the file into memory
(using CreateFileMapping - somehow!) and then read from the
memory into your output file?

Just a few thoughts

------------------

[Peter P Stephensen]

This function opens a 12 MB file in 0.6 sec. on my 233 MHz machine:

function GetFile(byval File as string, Buffer as string) as long
 
    local hFile as long
 
    err = 0 : hFile = freefile    
 
    open File for binary as hFile
    if err then exit function
    get$ hFile, lof(hFile), Buffer
    close hFile
  
    function = 1
  
end function

You can try to replace

    open File for binary as hFile

with

    open File for binary as hFile len = 16384

Regards
Peter

------------------


[This message has been edited by Peter P Stephensen (edited August 23, 2000).]

[Mark A Nelson]

Well if were on the subject, can anyone speed this up. I had only
5 min to put this in production about a year ago.

I gusse I could use case (that would help)

$COMPILE EXE
$INCLUDE "WIN32API.INC"
$INCLUDE "COMMCTRL.inc"
$INCLUDE "APIC.INC"

DECLARE FUNCTION GetAPICALLS ()AS STRING
DECLARE FUNCTION Parse%(instring$, searchChar$, occurence%, returnString$)
DECLARE FUNCTION Exist(BYVAL filespec AS STRING) AS LONG

CALLBACK FUNCTION OkButton()
CMD$ = Command$
LOCAL n AS INTEGER
LOCAL value AS INTEGER
LOCAL P AS INTEGER
LOCAL ReportName AS STRING

If Exist(CMD$) = 0 Then
MSGBOX "Report File " + "["+CMD$+"]" + " does not exist!", %MB_ICONSTOP, "ACI Report Cleaner VER 13.0"
FUNCTION = 0
Exit Function
End If

HoldNAME$ = EXTRACT$(CMD$, ANY ".")
HoldNAME2$ = EXTRACT$(CMD$, ANY ".")
Holdname$ = Holdname$ & ".FIX"
Holdname2$ = Holdname2$ & ".REP"

Open CMD$ For Input As #1
Open Holdname$ For Output As #2
Open Holdname2$ For Output As #3
Open "return.txt" For Output As #4
While Not EOF(1)
Line Input #1, a$

If Parse%(a$, "BTA400U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA402U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA409U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA410U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA110U.2", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA110U.3", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA110U.4", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA411R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA420R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA270R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA621R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA631R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA632R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA407U-A", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA425R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA621R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA631R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA635R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA636R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA640R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA656R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA666R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA250U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA250U.1", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA250U.2", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA347U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA380U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA404U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA407U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA409U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If

'
If Parse%(a$, "BTA440U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA440U.2", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA440U.3", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA441U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA441U.2", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA441U.3", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA442U.3", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA181R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA306R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA346R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA348R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA351R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA356R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA360R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA361R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA370R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA375R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA411R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA625R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA640R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA660", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA661R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA665", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA675R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA110U.1", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA130U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA406U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA406u-A", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA412U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA710U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA110U.1", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA111U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA110U.4", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA442U.1", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA441U.1", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA403U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA405U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA406U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA412U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA406U-A", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA414U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA400U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA665R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA641R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA130U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA710U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA720R", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If
'
If Parse%(a$, "BTA715U", 1, returnString$) = 0 Then
EXIT IF
Else
Print #4, returnString$
'PRINT# 4,reportname
Print #3, Chr$(12)
End If


If Parse%(a$, "1DATE:", 1, returnString$) = 0 Then
N = Len(a$)
N = N - 1
a$ = Right$(a$, N)
Print #2, a$
Print #3, a$
Else

Print #2, " " + returnString$
Print #3, " " + returnString$
End If


Wend

Close 4
Close 3
Close 2
Close 1
MsgBox "***** ReBuilt File: " + CMD$ + " to File: " + Holdname$ + " and " + Holdname2$ + " *****", , "Cleaner For ACI Files"
DIALOG END CBHNDL, 1
End Function

CALLBACK FUNCTION CancelButton()
DIALOG END CBHNDL, 0
End Function

Function Exist(ByVal filespec As String) As Long
LOCAL hDir AS LONG
LOCAL FindData AS WIN32_FIND_DATA
FindData.dwFileAttributes = %FILE_ATTRIBUTE_DIRECTORY
hDir = FindFirstFile(ByVal StrPtr(filespec), FindData)
IF hDir = %INVALID_HANDLE_VALUE THEN
Exit Function 'file not found
End If
FUNCTION = %TRUE
FindClose hDir
End Function

Function Parse%(instring$, searchChar$, occurence%, returnString$)
pntr% = 1
For i% = 1 To occurence%
pntr% = InStr(pntr%, instring$, searchChar$)
If pntr% = 0 Then Exit For
INCR pntr%
Next i%
If pntr% = 0 Then
FUNCTION = 0
Exit Function
End If
endPos% = InStr(pntr% + 1, instring$, searchChar$)
If endPos% = 0 Then
returnString$ = Mid$(instring$, pntr%)
Else
returnString$ = Mid$(instring$, pntr%, endPos% - pntr%)
End If
FUNCTION = 1
End Function

Function PBMAIN()
#REGISTER NONE
LOCAL hDlg AS LONG
LOCAL result AS LONG
pea459$ = GetAPICALLS()

If Date$ > pea459$ Then
MsgBox "ERROR: File Error", , "ERROR:"
Exit Function
Else

End If

DIALOG NEW 0, "ACI Cleaner", ,, 160, 50, 0, 0 TO hDlg

' ** Add controls to it
CONTROL ADD LABEL, hDlg, 400, "Please Wait Cleaning ACI File...", 14, 12, 134, 12, 0
CONTROL ADD BUTTON, hDlg, %IDOK, "OK", 34, 32, 40, 14, _
%BS_DEFAULT CALL OkButton
CONTROL ADD BUTTON, hDlg, %IDCANCEL, "Cancel", 84, 32, 40, _
14, 0 CALL CancelButton
DIALOG SHOW MODAL hDlg TO result
End Function





------------------

[Gregery D Engle]

Mark - Please put the code brackets around your code [ code] [ /code]

DorianDarby - Opening and changing bit and saving it to file at
the same time won't be againt the rules, but I
think that might slow it down actually. I think
if I have like 100 threads all reading a section of
a file it will be good... not sure yet.

Peter - Thanks, I forgot about that but it still isn't as
fast as I'm looking for. I'll come up with some home
grown method.

Thanks guys,


------------------
-Greg

[Peter P Stephensen]

take a look at
http://www.powerbasic.com/support/pb...ead.php?t=3924

regards
peter

------------------

[Fred Oxenby]

If I understand your game, you want to find the quickest
way to open read and close your file..?
You dont want to change your parsing-routine (You could gain most here)

  x%=FreeFile
  Open "myfile" for binary as #x%
  Get$ x%,LOF(x%),Buffer$
   ..Parse your Buffer
  Put x%,1,Buffer$
  SetEof x%
  Close x%

But I think you should write your parser in ASM to get best
performance.


------------------
Fred
mailto:[email protected][email protected]</A>
http://www.oxenby.se

[Gregery D Engle]

Originally posted by Fred Oxenby:
If I understand your game, you want to find the quickest
way to open read and close your file..?
You dont want to change your parsing-routine (You could gain most here)

  x%=FreeFile
  Open "myfile" for binary as #x%
  Get$ x%,LOF(x%),Buffer$
   ..Parse your Buffer
  Put x%,1,Buffer$
  SetEof x%
  Close x%

But I think you should write your parser in ASM to get best
performance.

Fred,

The downfall is that its taking a good 10 seconds just to read the
file into the buffer and then save it back to the file... I
haven't even parsed anything yet... Just to load and save takes
10 seconds. I think if I speed that up a little it will be
better. I think I'm going to try and multi thread it, and each
thread will reach 1024k from the file and processes it and save
it to a new file...

Thanks

------------------
-Greg

[Ian Cairns]

In answer to Mark Nelson's request:

For speed, convert all integers% to longs&.

Since you are doing a lot of repetition of a simple task, you could forget the PARSE% function entirely.
Create a new array of items (I believe you have 78 of them), and do an INSTR on each item.
If each line only held a single item, you could exit the FOR early.

LOCAL itemNo AS LONG
DIM Items(1:78) AS STRING
Items(1) = "BTA400U"
Items(2) = "BTA402U"
'....
Items(78) = "BTA715U"

'...

  WHILE NOT EOF(1)
    LINE INPUT #1, a$
    FOR itemNo = 1 TO 78
      IF INSTR(a$, Items(itemNo) ) <> 0 THEN
        PRINT #4, Items(itemNo)
        PRINT #3, CHR$(12)
      END IF
    NEXT itemNo

    IF INSTR(a$, "1DATE:" ) = 0 THEN
      N = LEN(a$)
      N = N - 1
      a$ = RIGHT$(a$, N)
      PRINT #2, a$
      PRINT #3, a$
    ELSE
      PRINT #2, " 1DATE:"
      PRINT #3, " 1DATE:"
   END IF

------------------
[email protected]

[Tom Hanlin]

Unless you're low on memory, reading and writing the entire file at a time
is pretty sure to be a best bet. You may find it faster to use GET$ and PUT$
rather than preallocating a string buffer. The redundant OPENs for output
can also be trimmed. Use a literal instead of a function in REPLACE.

#COMPILE EXE
FUNCTION PBMAIN() AS LONG
    g$ = TIME$
    OPEN "c:\greg.log" FOR BINARY AS #1
    GET$ 1, , sBuffer$
    CLOSE #1
...
... REPLACE " " WITH "," IN sBuffer$
...
    OPEN "c:\greg.out" FOR BINARY AS #1
    SETEOF #1
    PUT$ 1, sBuffer$
    CLOSE #1
    MSGBOX g$ & $CRLF & TIME$
END FUNCTION

------------------
Tom Hanlin
PowerBASIC Staff

[Steve Hutchesson]

Greg,

I have always found GET$ to be plenty fast enough but if you are after
getting the maximum speed out of disk access, there may be some advantage
in using the API disk functions.

CreateFile()
ReadFile()
WriteFile()
SetEndOfFile()
CloseHandle()

It has been my experience that OLE string memory allocates faster than
functions like GlobalAlloc() so either normal basic string or directly
calling,

SysAllocStringByteLen()
SysFreeString()

should get your speed up.

I have a sneaking suspicion that,

CoTaskMemAlloc()
CoTaskMemRealloc()
CoTaskMemFree()

are faster again but they appear to be a bit more complicate to use.

The disk IO times you mentioned sound reasonably slow which just may be a
limitation of the actual hardware. What I have no data on is the IO speed
difference between LINUX and WINDOWS. This is a factor that is probably
not within your control.

Good luck with the quest.

[email protected]

------------------

[Guest]

I found a BYTE pointer better at replacing a single character than Replace.
Give something like this a shot.

Function PBMain
  Register l As Long
  Dim a$
  Dim bp As Byte Ptr
  Dim tt As Single
  Dim t  As Single
  'Create a worst case 10mb file
  Open "D:\HexStuff.YYY" For output As #1
  Print #1, String$(1024000*10, 32)
  Close 1
 
  tt = timer
  Open "D:\HexStuff.YYY" For Binary As #1
  Get$ 1, Lof(1), a$
  t = timer
      'Replace " " With "," in a$
 
  bp = StrPtr(a$)
  For l = 0 To Len(a$)-1
    If @bp[l] = 32 Then @bp[l] = 44
  Next l
  t = timer - t  
  Seek #1,1
  Put$ 1, a$
  Close 1
  
  tt = timer - tt
  Print "Total Elapsed Time:";tt;"  Character Substitution Time:";t
End Function

------------------
Ron

[This message has been edited by Ron Pierce (edited August 24, 2000).]

[Tom Hanlin]

Note that you should avoid using indexed pointers where you can use
just plain pointers, if you want the best speed. Instead of this:

  For l = 0 To Len(a$)-1
    If @bp[l] = 32 Then @bp[l] = 44
  Next l

Try this:

  For l = 0 To Len(a$)-1
    If @bp = 32 Then @bp = 44
    Incr bp
  Next l

------------------
Tom Hanlin
PowerBASIC Staff

[Gregery D Engle]

Originally posted by Tom Hanlin:
Note that you should avoid using indexed pointers where you can use
just plain pointers, if you want the best speed. Instead of this:

  For l = 0 To Len(a$)-1
    If @bp[l] = 32 Then @bp[l] = 44
  Next l

Try this:

  For l = 0 To Len(a$)-1
    If @bp = 32 Then @bp = 44
    Incr bp
  Next l

Why are index pointers slower then regular pointers?

------------------
-Greg

[Steve Hutchesson]

Greg,

If you are doing simple BYTE replacement without changing the string
length, don't waste your time with pointers, indexed or otherwise, the
overhead is too high.

The SUB below does a scan of the string, if it matches the search
character, it replaces it, if not, it does not waste the time writing the
same character.

This one should hurry the LINUX guys up a bit.

[email protected]

    DECLARE SUB ReplaceByte(ByVal src as LONG, ByVal ln as LONG)

    Open "mybigfile.ext" for binary as #1
      Get$ #1,lof(1),a$
      ReplaceByte StrPtr(a$),len(a$)
      Put$ #1, a$
    Close #1

'##########################################################################

SUB ReplaceByte(ByVal src as LONG, ByVal ln as LONG)

    #REGISTER NONE

    ! mov ecx, ln
    ! mov esi, src
    ! add ecx, esi

  lblSt:
    ! mov al, [esi]
    ! cmp al, 32        ; byte to replace
    ! jne nxt1
    ! mov al, 44        ; byte to replace it with
    ! mov [esi], al
  nxt1:
    ! inc esi
    ! cmp esi, ecx
    ! jne lblSt

END SUB

'##########################################################################

------------------

[Gregery D Engle]

Originally posted by Steve Hutchesson:
Greg,

If you are doing simple BYTE replacement without changing the string
length, don't waste your time with pointers, indexed or otherwise, the
overhead is too high.

The SUB below does a scan of the string, if it matches the search
character, it replaces it, if not, it does not waste the time writing the
same character.

This one should hurry the LINUX guys up a bit.

[email protected]

    DECLARE SUB ReplaceByte(ByVal src as LONG, ByVal ln as LONG)

    Open "mybigfile.ext" for binary as #1
      Get$ #1,lof(1),a$
      ReplaceByte StrPtr(a$),len(a$)
      Put$ #1, a$
    Close #1

'##########################################################################

SUB ReplaceByte(ByVal src as LONG, ByVal ln as LONG)

    #REGISTER NONE

    ! mov ecx, ln
    ! mov esi, src
    ! add ecx, esi

  lblSt:
    ! mov al, [esi]
    ! cmp al, 32        ; byte to replace
    ! jne nxt1
    ! mov al, 44        ; byte to replace it with
    ! mov [esi], al
  nxt1:
    ! inc esi
    ! cmp esi, ecx
    ! jne lblSt

END SUB

'##########################################################################

Steve,

Actually this contest does have a purpose we are converting a
internet proxy log into comma delimeted text and some fields
are space delimited within this log file so sometimes I have this:

replace chr$(32) & chr$(32) & chr$(32) with "," ..

furthermore, the first field is a unix date/time stamp and that
is a fixed 11 charactors. So its much harder then this

Thanks



------------------
-Greg

[Steve Hutchesson]

Greg,

Here is try number 2. The following function reads through the string,
writes the first space it finds as a comma and throws following spaces
away until it finds a non space character which it writes. The penalty
with this function is that you have to allocate an extra buffer which
takes extra time.

This will convert a string of the type,

  a$ = "      This     is a     comma     delimited     string    "
  
      to
  
  b$ = ",This,is,a,comma,delimited,string,"

If you have problems with the leading and trailing commas on each line,

change$ = chr$(44,13,10,44)
Replace change$ With "" In b$

or variations of same will solve that problem.

Regards,

[email protected]

'##########################################################################
  
  FUNCTION RemoveSpaces(ByVal src as LONG, _
                        ByVal dst as LONG, _
                        ByVal ln as LONG) as LONG
  
      #REGISTER NONE
  
      ! mov ecx, ln
      ! mov esi, src
      ! mov edi, dst
      ! add ecx, esi
    ' ---------------------------------------------
    rsSt:
      ! mov al, [esi]     ; read byte
      ! inc esi
      ! cmp al, 32        ; search byte
      ! jne rsNxt
    ' ---------------------------------------------
      ! mov al, 44
      ! mov [edi], al
      ! inc edi           ; write comma
    ' ---------------------------------------------
    rsSl:
      ! mov al, [esi]     ; read next byte
      ! inc esi
      ! cmp esi, ecx
      ! je rsOut          ; exit on length match
      ! cmp al, 32
      ! je rsSl           ; loop if next byte is space
      ! mov [edi], al     ; write it if its not
      ! inc edi
      ! jmp rsSt          ; exit to start of main loop
    ' ---------------------------------------------
    rsNxt:
      ! mov [edi], al
      ! inc edi
      ! cmp esi, ecx      ; check byte count
      ! jne rsSt          ; if not matching, return to rsSt
    ' ---------------------------------------------
    rsOut:
  
      ! sub edi, dst      ; get byte write count
  
      ! mov FUNCTION, edi
  
  END FUNCTION
  
'##########################################################################

[This message has been edited by Steve Hutchesson (edited August 24, 2000).]

[Gregery D Engle]

Hutch,

Whats wrong with my code:

It causes GPF, and if I do a msgbox(STRPTR(Sbuffer)) it displays
a number 1.34234234+E .... (overflow?) I'm lost on what I
am doing wrong.

Thanks,
Greg

#COMPILE EXE
GLOBAL Sbuffer$
GLOBAL newBuffer$
'##########################################################################

FUNCTION RemoveSpaces(BYVAL src AS LONG, _
BYVAL dst AS LONG, _
BYVAL ln AS LONG) AS LONG
#REGISTER NONE

! mov ecx, ln
! mov esi, src
! mov edi, dst
! add ecx, esi
' ---------------------------------------------
rsSt:
! mov al, [esi] ; read byte
! inc esi
! cmp al, 32 ; search byte
! jne rsNxt
' ---------------------------------------------
! mov al, 44
! mov [edi], al
! inc edi ; write comma
' ---------------------------------------------
rsSl:
! mov al, [esi] ; read next byte
! inc esi
! cmp esi, ecx
! je rsOut ; exit on length match
! cmp al, 32
! je rsSl ; loop if next byte is space
! mov [edi], al ; write it if its not
! inc edi
! jmp rsSt ; exit to start of main loop
' ---------------------------------------------
rsNxt:
! mov [edi], al
! inc edi
! cmp esi, ecx ; check byte count
! jne rsSt ; if not matching, return to rsSt
' ---------------------------------------------
rsOut:

! sub edi, dst ; get byte write count
! mov FUNCTION, edi
END FUNCTION

FUNCTION PBMAIN() AS LONG
'DIM sbuffer$
'DIM newbuffer$
DIM g&
DIM tt AS LONG

OPEN "C:\greg.log" FOR BINARY AS #1
GET$ 1,LOF(1),sbuffer$
CLOSE #1
  tt = TIMER
'MSGBOX STR$(STRPTR(sBuffer$))

g& = RemoveSpaces(STRPTR(sBuffer$), STRPTR(NewBuffer$), LEN(sbuffer$))

  tt = TIMER - tt
  MSGBOX "Total Elapsed Time:" & STR$(tt) & "  Character Substitution Time:" ' & STR$(t)
OPEN "c:\greg.out" FOR BINARY AS #1
SETEOF #1
PUT$ 1, NewBuffer$
CLOSE #1
END FUNCTION

------------------
-Greg

[Florent Heyworth]

Hi Greg

the problem with your code is that your are passing an empty
buffer (NewBuffer$) to the RemoveSpaces function. You need to
allocate:

NewBuffer$ = SPACE$(LEN(sBuffer$))

after reading the file in sBuffer$ and before calling RemoveSpaces().

Once you've done that the function works fine.

Cheers

Florent




[This message has been edited by Florent Heyworth (edited August 25, 2000).]

[Gregery D Engle]

Florent,

AHHH I would never had figured that out... Thanks

------------------
-Greg