Reading Binary Files

[Allen Nugent]

I just ported a VB routine that reads a binary data file into arrays to PB DLL 5.0, hoping for a big performance improvement. For a file of 40,000 4-byte reals, it takes 11 sec on a 486 DX2 ... the same as it did when coded in VB 4. How can I speed it up?

(NB. I once wrote a similar DLL under Microsoft Fortran 5.1, using 16-bit Windows API calls for the I/O, and it was almost too fast to measure.)

[Florent Heyworth]

Hi Allen

post your code so we can have a look at it...

Cheers

Florent

------------------

[Allen Nugent]

Thanks for asking, Florent. Here is the code.

FUNCTION ReadVelBinary(FileSpec$, BYVAL mDims&, BYVAL mExtSigs&, BYVAL mPoints&, _
BYVAL nVelSigs&, BYVAL nExtSigs&, BYREF nPoints&, _
BYVAL pt1&, BYVAL u1&, BYVAL ex1&) EXPORT AS INTEGER

DIM u!(1 To mDims&, 1 To mPoints&) AT u1&
DIM pt#(1 To mPoints&) AT pt1&
DIM ex!(1 To mExtSigs&, 1 To mPoints&) AT ex1&

nErr% = 0
iFile% = Freefile
On Error GoTo InputError
Open FileSpec$ For Binary Access Read As #iFile%

Get #iFile%, , nPoints&

If EOF (iFile%) Then GoTo CloseFile
For j& = 1& To nPoints&
GoSub ReadRecord
Next j&

GoTo CloseFile

'______________________ Subroutines and Branches ______________________

ReadRecord:

Get #iFile%, , pt#(j&)

If EOF (iFile%) Then GoTo EndFile
For i& = 1& To nVelSigs&
Get #iFile%, , u!(i&, j&)
If EOF (iFile%) Then GoTo EndFile
Next i&

For i& = 1& To nExtSigs&
Get #iFile%, , ex!(i&, j&)
If EOF (iFile%) Then GoTo EndFile
Next i&

Return

InputError:

nErr% = ERRCLEAR
On Error GoTo 0

EndFile:

nPoints& = j& - 1

CloseFile:

ReadVelBinary = nErr%
Close (iFile%)

EXIT FUNCTION

END FUNCTION




------------------

[Semen Matusovski]

alten --
this question was discussed, for example, in http://www.powerbasic.com/support/pb...ad.php?t=14618
to have good speed, it's necessary to read by big blocks (1-2 cluster' size, 4-16 kb).
i prefer to read in strings (get$, then to use ptr).



[this message has been edited by semen matusovski (edited april 18, 2000).]

[Peter Stephensen]

Allen,

As far as i can see the binary file is packed like this:

nPoints&
-------------
pt#(1)
u!(1, 1)
...
u!(nVelSigs&, 1)
ex!(1, 1)
...
ex!(nExtSigs&, 1)
-------------
...
-------------
pt#(nPoints&)
u!(1, nPoints&)
...
u!(nVelSigs&, nPoints&)
ex!(1, nPoints&)
...
ex!(nExtSigs&, nPoints&)
-------------

The file would be much easier to read if it was packed like this:
nPoints&
-------------
pt#(1)
...
pt#(nPoints&)
-------------
u!(1, 1)
...
u!(nVelSigs&, nPoints&)
------------
ex!(1, 1)
...
ex!(nExtSigs&, nPoints&)
------------

So, if it is possible to change the binary file...

Regards
Peter


------------------

[Guest]

Allen, you could use the API ReadFile to avoid the For..next loops
If you use pointers, I think it would be easy to read all the arrays
into memory with one ReadFile Or Get # call.

#Option Version4
#Register None
#Dim All
#include "Win32API.inc"
 
$DataFile = "ReadFile.Dat"
 
Function PbMain
  Call ReadData
  StdOut "Done!"
End Function
 
Sub ReadData
  Register l    As Long
  Local s       As Single
  Local hFile   As Long
  Local Dummy() As Long   ' use any 4 byte vartype
  Local RetVal  As Long
  Local btr     As Long   ' Bytes To Read (not to be confused with B.T.O. )
  Local btw     As Long   ' Bytes To Write
  Local bw      As Long   ' Bytes Actually Written
  Local br      As Long   ' Bytes Actually Read
  Local WinErr  As Long   'holds Last Error
  local sa      as SECURITY_ATTRIBUTES
  Local sp      As Single Ptr
  Local lp      As Long Ptr
 
  Redim Dummy (1 to 40000) As Long
  btr = 40000& * 4&
  
  lp = VarPtr(Dummy(1&))      ' long int array starts at beginning of Dummy()
  sp = VarPtr(Dummy(20001&))  ' single array starts at element 20001 of Dummy()
  For l = 0 To 19999
    @lp[l] = l+1
  Next
 
  For l = 0 To 19999
    @sp[l] = l+1.12345
  Next
 
    hFile = CreateFile($DataFile, %GENERIC_WRITE + %GENERIC_READ + %TRUNCATE_EXISTING, 0, ByVal %Null, %OPEN_ALWAYS, 0&, BYVAL %NULL)
    StdOut "hFile:" & Str$(hFile)
 
  If ( hFile <> %INVALID_HANDLE_VALUE) Then
    btw = 20000& * 4
    RetVal = WriteFile(hFile, @lp, ByVal btw, bw, ByVal %Null)
    WinErr = GetLastError
    StdOut "RetVal:" & Str$(Retval) & "    Bytes Written(LP):" & Str$(bw) & "    Read Error:" & Str$(WinErr)
 
    RetVal = WriteFile(hFile, @sp, ByVal btw, bw, ByVal %Null)
    WinErr = GetLastError
    StdOut "RetVal:" & Str$(RetVal) & "    Bytes Written(SP):" & Str$(bw) & "    Read Error:" & Str$(WinErr)
 
    For l = 0 To 39999    'Zero All array values
      @lp[l] = 0
    Next
 
    RetVal = SetFilePointer (hFile, _
                            0&, _
                            0&, _
                            %FILE_BEGIN)
 
    RetVal = ReadFile(hFile, Dummy(1), ByVal btr, br, ByVal %Null)
    WinErr = GetLastError
    CloseHandle hFile
    StdOut "RetVal:" & Str$(Retval) & "    Bytes Read:" & Str$(br) & "    Read Error:" & Str$(WinErr)
  End If
 
  If ( WinErr = 0& ) Then
    StdOut "L1:" & Str$(@lp[0]) & "   L20000:" & Str$(@lp[19999]) & "   S1:" & Str$(@sp[0]) & "   S20000:" & Str$(@sp[19999])
  Else
    'Not a good thing...
  End If
 
End Sub

Ron



[This message has been edited by Ron Pierce (edited April 18, 2000).]

[Allen Nugent]

Peter has correctly inferred the file structure. Ron's suggestion is beyond my current programming knowledge (I think can recognise a pointer variable when I see one, but that's about it).

I'm inclined to re-write the program that creates the data files, so as to use Peter's suggestion. However, I'm tempted to try Ron's advice and get stuck into pointers. I'll have to think about it.

Thanks, all!


------------------

[Peter Stephensen]

Allen,

If you change the file structure, and if you change your function such that it returns data-pointers, you could like this:

First define the function GetFile (i use this function all the time):

FUNCTION GetFile(File AS STRING, Buffer AS STRING) AS LONG
  
    LOCAL hFile AS LONG
  
    hFile = FREEFILE
  
    OPEN File FOR BINARY AS hFile
    IF ERR THEN EXIT FUNCTION
    GET$ hFile, LOF(hFile), Buffer
    CLOSE hFile
  
    FUNCTION = 1
  
END FUNCTION

Your function could now look like this:

FUNCTION ReadVelBinary(FileSpec$, BYVAL mDims&, BYVAL mExtSigs&, BYVAL mPoints&, _
    BYVAL nVelSigs&, BYVAL nExtSigs&, nPoints&, pt1&, u1&, ex1&) EXPORT AS INTEGER
  
    STATIC Buffer AS STRING
    LOCAL pBuf AS LONG
    LOCAL pLong AS LONG PTR
  
    IF GetFile(FileSpec$, Buffer ) = 0 THEN
        MSGBOX "Could not open file"
        EXIT FUNCTION
    END IF
       
    pBuf = STRPTR(Buffer)
    pLong = pBuf
    nPoints& = @pLong
  
    pt1& = pBuf + 4
    u1&  = pt1& + nPoints&
    ex1& = u1&  + nVelSigs& * nPoints&
  
    FUNCTION = 1
  
END FUNCTION

I have not tested the code. Maybe i works

Regards
Peter
------------------


[This message has been edited by Peter Stephensen (edited April 18, 2000).]

[Peter Stephensen]

I guess it should be:

    pt1& = pBuf + 4
    u1&  = pt1& + nPoints& * 8
    ex1& = u1&  + nVelSigs& * nPoints& * 4 * 4

------------------

[Semen Matusovski]

One possible variant

   #Compile Exe
   #Dim All
   #Register None

   Sub WriteArray
     Dim m1 As Long, n1 As Long, i As Long, j As Long
     m1 = 150: n1 = 200: ReDim Ar1(1 To m1, 1 To n1) As Double
     For i = 1 To m1: For j = 1 To n1: Ar1(i, j) = i + j * 0.001: Next: Next
     Open "Ar.Dat" For Output As #1
     Print #1, Mkl$(m1) + Mkl$(n1) + Peek$(VarPtr(Ar1(1, 1)), m1 * n1 * 8);
     Close #1
   End Sub
   
   Sub ReadArray
     Dim m2 As Long, n2 As Long, i As Long, j As Long, s As String
     Open "Ar.Dat" For Binary As #1: Get$ #1, Lof(1), s: Close #1
     m2 = Cvl(s, 1): n2 = Cvl(s, 5)
     ReDim Ar2(1 To m2, 1 To n2) As Double At StrPtr(s) + 8
     MsgBox Format$(Ar2(105, 21), "#.###")
   End Sub
     
   Function PbMain
      WriteArray
      ReadArray
   End Function

------------------

[Allen Nugent]

Dear Ron,

I've had another go, based on the code you provided. I think I'm reading the file into a buffer OK, but I'm not able to parse the records of the buffer into the target arrays yet. I suspect Im trying to use pointers the way I'd like them to work (ie. stepping through the memory addresses to find my array elements) as opposed to the way they actually work (which seems to be eluding me). It may not help that one field of each record is a Double, whereas the rest are Singles. Any suggestions?

'_________________________________

' PBDLL
$COMPILE EXE

$Option Version4
$Register None
'$Dim All
$include "Win32API.inc"

DECLARE FUNCTION ReadVelBinary(FileSpec$, BYVAL mDims&, BYVAL mExtSigs&, BYVAL mPoints&, _
BYVAL nVelSigs&, BYVAL nExtSigs&, BYREF nPoints&, _
pt#(), u!(), ex!()) AS INTEGER

'DECLARE FUNCTION CreateFile LIB "KERNEL32.DLL" ALIAS "CreateFileA" (lpFileName AS ASCIIZ, BYVAL dwDesiredAccess AS DWORD, BYVAL dwShareMode AS DWORD, lpSecurityAttributes AS SECURITY_ATTRIBUTES, BYVAL dwCreationDisposition AS DWORD, _
' BYVAL dwFlagsAndAttributes AS DWORD, BYVAL hTemplateFile AS LONG) AS LONG

'_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

FUNCTION WinMain (BYVAL CurInst&, _
BYVAL PrvInst&, _
CmdLine AS ASCIIZ PTR, _
BYVAL CmdShow&) EXPORT AS LONG

Dim mPoints As Long, mDims As Long, mExtSigs As Long
Dim nVelSigs As Long, nExtSigs As Long, nPoints As Long
Dim u(1 To mDims, 1 To mPoints) As Single, pt(1 To mPoints) As Double, _
ex(1 To mExtSigs, 1 To mPoints) As Single

mPoints = 10000
mDims = 2
mExtSigs = 8

nPoints = 0 ' actual value is stored in 1st 4 bytes of input file
nVelSigs = 2
nExtSigs = 4

FileSpec$ = "C:\LDV\Results\Vad_01P.000"
nErr% = ReadVelBinary(FileSpec$, mDims, mExtSigs, mPoints, nVelSigs, nExtSigs, nPoints, _
pt(), u(), ex())

for i% = 1 to 10
pti# = pt(i%)
u1i! = u(1, i%)
u2i! = u(2, i%)
ex1i! = ex(1, i%)
next i%

END FUNCTION

'__________________________________________________________________________________

FUNCTION ReadVelBinary(FileSpec$, BYVAL mDims&, BYVAL mExtSigs&, BYVAL mPoints&, _
BYVAL nVelSigs&, BYVAL nExtSigs&, BYREF nPoints&, _
pt#(), u!(), ex!()) EXPORT AS INTEGER

DIM u!(1 To mDims&, 1 To mPoints&)
DIM pt#(1 To mPoints&)
DIM ex!(1 To mExtSigs&, 1 To mPoints&)

Register i&, j&
Local hFile as Long ' file handle
Local Buffer() as Long ' input buffer
Local nBuffer& ' number of 4-byte records in input buffer
Local ReturnValue as Long
Local BytesToRead as Long
Local BytesRead as Long
Local WinErr as Long

Local cp as String Ptr
Local lp as Long Ptr
Local sp as Single Ptr
Local dp as Double Ptr

nErr% = 0


GOTO Using_Pointers

' This method is too slow >>>

iFile% = Freefile
On Error GoTo InputError
Open FileSpec$ For Binary Access Read As #iFile%

Get #iFile%, , nPoints&
If EOF (iFile%) Then GoTo CloseFile
For j& = 1& To nPoints&
GoSub ReadRecord
Next j&

GoTo CloseFile

ReadRecord:

Get #iFile%, , pt#(j&)
If EOF (iFile%) Then GoTo EndFile
For i& = 1& To nVelSigs&
Get #iFile%, , u!(i&, j&)
If EOF (iFile%) Then GoTo EndFile
Next i&
For i& = 1& To nExtSigs&
Get #iFile%, , ex!(i&, j&)
If EOF (iFile%) Then GoTo EndFile
Next i&

Return

'******************************************************

Using_Pointers:

' Using API to read ...

hFile = CreateFile (ByCopy FileSpec$, %GENERIC_READ, 0, ByVal %NULL, _
%OPEN_ALWAYS, 0&, ByVal %NULL)
If (hFile = %INVALID_HANDLE_VALUE) Then
MSGBOX "ReadVelBinary (DLL): file not found'
nPoints& = 0&
EXIT FUNCTION
End If

ReDim Buffer(1)
ReturnValue = ReadFile (hFile, Buffer(1), (4&), BytesRead, Byval %Null)
WinErr = GetLastError

nPoints& = Buffer(1)
BytesToRead = 4& + nPoints& * (8& + 4& * (nVelSigs& + nExtSigs&))
nBuffer& = nPoints& * (2& + nVelSigs& + nExtSigs&)

ReDim Buffer (1& To nBuffer&) 'element 1 contains 1st half of pt#(1)
ReturnValue = ReadFile (hFile, Buffer(1), ByVal BytesToRead, BytesRead, Byval %Null)
WinErr = GetLastError

CloseHandle hFile
MSGBOX "ReturnValue: " & Str$(ReturnValue) & " Bytes read: " & Str$ (BytesRead) _
& " Read Error:" & Str$(WinErr)
If ( WinErr <> 0& ) Then
MSGBOX "ReadVelBinary (DLL): Error reading file'
nPoints& = 0&
EXIT FUNCTION
End If

'...

' Using PB to read ...

' iFile% = Freefile
' Open FileSpec$ For Binary Access Read As #iFile%
' Get #iFile%, , Buffer ' NFG: Can't read into a non-string buffer !!!!
' Close #iFile%

'...

dp = VarPtr(Buffer(1&))
For j& = 1& To nPoints&
pt#(j&) = @dp
sp = dp + 8&
For i& = 1& To nVelSigs&
u!(i&, j&) = @sp
sp = sp + 4&
Next i&
For i& = 1& To nExtSigs&
ex!(i&, j&) = @sp
sp = sp + 4&
Next i&
dp = sp
Next j&
nErr% = ERRCLEAR

for k% = 1 to 10
pti# = pt#(k%)
u1i! = u!(1, k%)
u2i! = u!(2, k%)
ex1i! = ex!(1, k%)
next k%

GoTo Finish

'______________________ Clean-up ______________________

InputError:

nErr% = ERRCLEAR
On Error GoTo 0

EndFile:

nPoints& = j& - 1

CloseFile:

Close (iFile%)

Finish:

ReadVelBinary = nErr%
EXIT FUNCTION

END FUNCTION




------------------

[Allen Nugent]

ADDENDUM

I think I know what I've been doing wrong, but not how to fix it.

The array "Buffer" contains Doubles and Singles, but is defined as all Singles (DIM'ed to include an extra 4 bytes for each Double).

If I skip over the Double, at the 1st element by doing this...

sp = VarPtr(Buffer(1&))
sJunk1! = @sp[0]
sJunk2! = @sp[1]
sJunk3! = @sp[2]

then the Single in sJunk3! ia correct.

However, if I try to read the Double, like this...

dp = VarPtr(Buffer(1&))
dJunk1# = @dp[0]

or this...

sp = VarPtr(Buffer(1&))
dp = sp
dJunk1# = @dp[0]

or this...

sp = VarPtr(Buffer(1&))
dJunk1# = @sp[0]

I get "????" in the "Evaluate" window for dJunk1#.

So, how can I extract both Singles and Doubles from the same buffer?



------------------

[Guest]

Allen, I think the easiest cure would be using a UDT if the data arrangement would
would support this. If I understand the code it looks like you may be dealing with
data whose layout is subject to change. Is it possible to define one or more UDT's which
would always work with your data keeping in mind that arrays within a UDT must be hard coded.

Are mDims and mExtSigs constants or are will they vary?

----------------------------------------------------------------------------------

#Register None
#Dim All
#Option Version4
 
Global mPoints  As Long
Global mDims    As Long
Global mExtSigs As Long
 
Type VelBinaryType
  u(1 To 2)   As Single
  pt          As Double
  ex(1 To 8)  As Single
End Type
Global VelBinary() As VelBinaryType
Global vp          As VelBinaryType Ptr
 
Function PBMain
  mPoints = 10000
     ' mDims = 2
     ' mExtSigs = 8
 
  Redim VelBinary(1 To mPoints) as Global VelBinaryType
  vp = VarPtr(VelBinary(1)) 

  Call ReadData
End Function
 
 
Sub ReadData
  'Read the entire file in one read at the address of VelBinary(1)
  Local BytesToRead As Long
  BytesToRead = ( mPoints * Len(VelBinaryType) )
  Print "BytesToRead:"; BytesToRead, "  Len(VelBinaryType):"; Len(VelBinaryType)
End Sub

----------------------------------------------------------------------------------
Then you could use a pointer to the UDT to access each record:
Dim up As VelBinaryType Ptr
Dim db As Double
Dim sg As Single

up = VarPtr(VelBinary(1))
db = @up[0].pt
sg = @up[0].ex(7)

Or modify the array vars:
@up[124].pt = 12345.54321# ' Modify Double in array element 125
----------------------------------------------------------------
I hope something like tthis is usable.

Ron

[This message has been edited by Ron Pierce (edited April 20, 2000).]

[Semen Matusovski]

I prefer to use At. ReDim At doesn't allocate new memory.

   #Compile Exe
   #Register None
   #Dim All
   %mDims = 2
   %mExtSigs = 8
   Type VelBinaryType
      u(1 To %mDims)      As Single
      pt                  As Double
      ex(1 To %mExtSigs)  As Single
   End Type
 
  Function PbMain
     Dim mPoints  As Long, Buffer As String
     Open "Test.Dat" For Binary As #1
     If Lof(1) < 4 Then MsgBox "Error": Exit Function    
     Get$ #1, Lof(1), Buffer$: Close #1
     mPoints = Cvl(Buffer$, 1)
     ReDim VelBinary(1 To mPoints) As VelBinaryType At StrPtr(Buffer$) + 4
  End Function

------------------

[Allen Nugent]

Ron,

In an earlier posting within this forum. Pter Stephensen correctly decribed the file structure. As you surmise, the file structure is variable: pt# is always at the start of a record, but nVelSigs& and nExtSigs& vary, depending on what data the user selected to write, in the program that created the original data file. Therefore, I can't define a structure to describe the records... I would have to code for all possible permutations of structures (nVelSigs& can range from 1 to 2, and nExtSigs& from 0 to 8).

Also, pardon my ignorance, but what is a "UDT" when it's at home?

Thanks again!



------------------

[Allen Nugent]

ADDENDUM

To ameliorate confuscation...

"pt#" is also an array (I seem to have mislead both Ron and Semen, in this regard). Please refer to Peter's ellucidation of the file structure.

"mDims&", "mVelSigs&", "mExtSigs&" are the absolute maximum sizes of the relevant arrays (set in the VB calling routine);

"nDims&", "nVelSigs&", "nExtSigs&" are the actual sizes of the arrays to be extracted from the particular data file being read.

(The programmer's bane is to provide too many options to the user!)




------------------

[Allen Nugent]

FURTHER ADDENDUM

Semen seems to be telling me to read the file into a String buffer. Would this allow me to extract both Double and Single real numbers, as I choose?



------------------

[Semen Matusovski]

Originally posted by Allen Nugent:
Semen seems to be telling me to read the file into a String buffer. Would this allow me to extract both Double and Single real numbers, as I choose?

Yes, I tell you about it the second day.
I assume, that you read PB Help about pointers, VarPtr, StrPtr.

So, if you read whole file by Get$ #1, Lof(1), a$ first byte of file (if to numerate from 1) will be located at address StrPtr(a$); second - at StrPtr(a$) + 1 and so on.

Unlike Vb, you can operate with variables enough simple.

For example, DOUBLE begins in file from byte no. x1.
To get it, you can do (for example) following:

Dim pDouble As Double Ptr
Dim Dbl As Double
...
pDouble = StrPtr(a$) + x1 - 1
Dbl = @pDouble

If SINGLE begins in file from byte no. x2.

Dim pSingle As Single Ptr
Dim Sgl As Single
...
pSingle = StrPtr(a$) + x2 - 1
Sgl = @pSingle

and so on for ANY type.
You can mix such statements like you want.

If array of doubles (1 To n) begins in file from byte no. x3, you can ReDim ArDouble(1 To n) As Double At StrPtr(a$) + x3 - 1 and then simply to work with array ArDouble.
If array of singles (1 To n) begins in file from byte no. x4, you can ReDim ArSingle(1 To n) As Single At StrPtr(a$) + x4 - 1.

If you have two-dimensions arrays Array(1 To m, 1 To n), you should exactly imagine, that in PB location of elements will be following
Array(1, 1)
Array(2, 1)
...
Array(m, 1)

Array(1, 2)
Array(2, 2)
...
Array(m, 2)

Array(1, n)
Array(2, n)
...
Array(m, n)

------------------