Faster way to fill structure?

[Mike Doty]

'Opening a file and reading into a string is very quick, but then placing
'the values into a structure in the example below takes 16 seconds
'to process 8000 records.
'
'If the file is opened for random access and each record is just read
'it takes under 1-second. Is LSET CLIENT slow because of the MID$ ?

TYPE ClientType
  Everything AS STRING * 203
END TYPE
  FUNCTION PBMAIN AS LONG
    LOCAL CLIENT AS ClientType
    LOCAL RecordNumber AS LONG
    LOCAL TotalRecords AS LONG
    LOCAL sData          AS STRING
    OPEN "\grid\file101.dat" FOR BINARY AS #1
    TotalRecords = LOF(1)\LEN(CLIENT)
    GET$ #1, LOF(1),sData
    CLOSE #1
    FOR RecordNumber = 1 TO TotalRecords         'This takes 16 seconds for 8000 records
      LSET CLIENT = MID$(sData,(RecordNumber-1) *LEN(CLIENT)+1)
    NEXT
 
    OPEN "\grid\file101.dat" FOR RANDOM AS #1  LEN  = LEN(CLIENT) 'under a second for 8000 records
    TotalRecords = LOF(1)\LEN(CLIENT)
    FOR RecordNumber = 1 TO TotalRecords
       GET #1,RecordNumber,CLIENT
    NEXT
   END FUNCTION

[Michael Mattias]

>Is LSET CLIENT slow because of the MID$ ?

Yes.

Except, it is not "slow," is it "slower than reading from disk"

Imagine that... reading the disk is faster than doing it in memory..in this application.

Maybe whomever it is who is always saying "optimization is always application-specific" is not full of crap after all.

MCM

[Mike Doty]

Interesting, I thought it would be faster having the file in memory and
just looping through the string. Maybe using STRPTR or something might
get them a little bit closer. After seeing this, I'm no longer going to
keep the file in memory and use a TYPE. This is a good example of
what not to do.

[Adam J. Drake]

Mike,

You can still use the technique of keeping the file in memory and looping through it. You can use the following using a pointer to access each record:

#COMPILE EXE
#DIM ALL

TYPE ClientInformation
    Everything  AS STRING * 203
END TYPE

FUNCTION PBMAIN () AS LONG

    LOCAL x     AS LONG
    LOCAL CI    AS ClientInformation
    LOCAL CIptr AS ClientInformation PTR
    LOCAL s, f  AS SINGLE
    LOCAL fdata AS STRING
    
    OPEN "C:\FILE101.DAT" FOR RANDOM AS #1 LEN = 203
    FOR x = 1 TO 8000
        PUT #1, x, CI
    NEXT
    CLOSE #1

    s = TIMER
    OPEN "C:\FILE101.DAT" FOR BINARY ACCESS READ AS #1 LEN = 16384
    GET$ #1, LOF(1), fData
    CLOSE #1
    
    CIPtr = STRPTR(fData)
    FOR x = 0 TO 7999
        TYPE SET CI = @CIPtr[x]
    NEXT
    
    f = TIMER
    
    MSGBOX FORMAT$(f-s),, "In Memory using Pointer"
    
END FUNCTION

[Michael Mattias]

Memory Mapped Files instead of RANDOM disk file access 5-8-04

[Paul Squires]

I wonder if something like the following would work. Since the string is already in memory then try overlaying it with an absolute array:

   Dim MyArray( TotalRecords ) As ClientType At StrPtr(sData)

[Adam J. Drake]

That will work great provided the array size stays the same...

[Mike Doty]

'Yes, that was it. Extremely fast, now!

DIM MyArray(1 TO TotalRecords) AS ClientType AT STRPTR(sData)
MSGBOX  MyArray(TotalRecords).lastname

[Mike Trader]

So if I understand this right:
If a very long string sData is created initially then REDIMing will be very fast as there are no memory allocation steps.

sData = STRING$(1000000) ' 1MB

TotalRecords = 20

 DIM MyArray( TotalRecords ) As ClientType At StrPtr(sData)

NewRec = 10
 REDIM MyArray( TotalRecords+NewRec ) As ClientType At StrPtr(sData)

Where ClientType is a UDT of say 1000 bytes so MyArray will use 30k max

Do I have this right?

[Mike Doty]

I would also like to know.
If the absolute array is an overlay why couldn't the array just be 1-element and the pointer incremented? For that matter, why even
use an array? Page 227 of the manual under DIM.

[Michael Mattias]

If the absolute array is an overlay why couldn't the array just be 1-element and the pointer incremented? For that matter, why even use an array?

You wouldn't.

You'd either use an incremented (or offset) pointer to get elements, or you'd use a PB array.

[John Gleason]

Where ClientType is a UDT of say 1000 bytes so MyArray will use 30k max

Do I have this right?

You do! Furthermore, it makes no difference how big the UDT is (as long as it fits the allocated string). A 22K UDT DIMS just as fast as the 1K UDT.

[Mike Doty]

'Would something like this save memory?

 
TYPE ClientType
  me AS STRING * 1
END TYPE
FUNCTION PBMAIN () AS LONG
  DIM sData AS STRING
  sData = "ABC"
  DIM x AS LONG
  FOR x = 0 TO 2
    REDIM MyArray(0) AS clientType AT STRPTR(sData) + x * LEN(clientType)
    PRINT MyArray(0).me
  NEXT
  WAITKEY$
END FUNCTION

[Mike Trader]

Mike,
The memory is svaed because you are dimensioning the array on top of memory that has allready been allocated for the string.

How the Compiler keeps track of all this is amazing to me.

You could do this with pointers

pMyArray as ClientData PTR

Then

pMyArray = STRPTR(sData) ' First UDT array element


pMyArray = STRPTR(sData) + SIZEOF(ClientData)*9 ' element 9

Then

PRINT @pMyArray.me

just forget the array altogether.

[Mike Trader]

John,

That raises another question then, if the string is being updated in one thread and a second thread is REDIMing an array on top of it is there a risk of a GPF?

I assume not because context switching will take care of the serializing of the operations as per this thread:

http://www.powerbasic.com/support/pbforums/showthread.php?t=14382&highlight=thread

[John Gleason]

REDIMing an array on top of it is there a risk of a GPF?

Like you, I would think not since the two operations seem unrelated, but some testing to confirm it probably wouldn't hurt.

[Michael Mattias]

That raises another question then, if the string is being updated in one thread and a second thread is REDIMing an array on top of it is there a risk of a GPF?

A GPF is best case scenario. Well, I for one would prefer a GPF to silent no-warning data corruption.

FWIW: 'REDIM AT' allocates NO memory at all. Ever.

MCM

[Adam J. Drake]

If you have to make modifications to the size of the array, you should not use an array that was dimensioned using 'AT'...

Try something like this to read it fast, and then do what you want to the array:

#COMPILE EXE
#DIM ALL
                        
TYPE ClientInformation
    Everything  AS STRING * 203
END TYPE

FUNCTION PBMAIN () AS LONG

    LOCAL x     AS LONG
    LOCAL CIptr AS ClientInformation PTR
    LOCAL s, f  AS SINGLE
    LOCAL fdata AS STRING
    LOCAL CITmp AS ClientInformation

    DIM CI(1 TO 8000) AS ClientInformation

    OPEN "C:\FILE101.DAT" FOR RANDOM AS #1 LEN = 203
    FOR x = 1 TO 8000
        PUT #1, x, CITmp
    NEXT
    CLOSE #1

    s = TIMER
    OPEN "C:\FILE101.DAT" FOR BINARY ACCESS READ AS #1 LEN = 16384
    GET$ #1, LOF(1), fData
    CLOSE #1

    CIPtr = STRPTR(fData)
    FOR x = 0 TO 7999
        TYPE SET CI(x+1) = @CIPtr[x]
    NEXT

    f = TIMER

    MSGBOX FORMAT$(f-s),, "In Memory using Pointer"

END FUNCTION

[Michael Mattias]

FWIW, when you alter a string in any way, the address of the data may change, although the address of the string handle does not...meaning in addition your precise corruption or protection error will vary depending on WHAT data type you are REDIM'ing "AS" .... i.e you might get lucky.

The bottom line is, don't do this. (Subjective, biased opinion).

[John Gleason]

Adam wrote:

If you have to make modifications to the size of the array, you should not use an array that was dimensioned using 'AT'...

Mike Trader,
I believe now Adam is correct and I was wrong earlier saying you can REDIM AT an array you already DIM'ed or REDIM'ed AT earlier. It doesn't work as I originally expected and you can test the failure in the code below. However, the fastest "workaround" is to simply create a second array name AT the same STRPTR address as shown. Still only one copy in memory of the data is needed.

Michael wrote:

when you alter a string in any way, the address of the data may change

I can only hope that's not true, because I have a ton of code predicated on the idea that that the string stays put until I erase or re-create it. Again see the example below.

#COMPILE EXE
#DIM ALL

FUNCTION PBMAIN () AS LONG
    LOCAL ii AS LONG
    LOCAL strX, theString AS STRING
    
    theString = "john      gleason   programmermichael   mattias   adviser   adam      drake     analyst   "
    REDIM nameNjobArr(8) AS STRING * 10 AT STRPTR(theString)
    FOR ii = 0 TO 8
       strX = strX & nameNjobArr(ii) & $CRLF
    NEXT
    ? strX
    
    strX = ""
                         'changing theString is okay as long as you do it "in place". eg. MID$=, @pointer, asm.
                         'The memory block accessed is therefore exactly the same, but just has different contents.
    MID$(theString, 1) = "john      gleason   coder     michael   mattias   designer  adam      drake     lawmaker  "
    FOR ii = 0 TO 8
       strX = strX & nameNjobArr(ii) & $CRLF
    NEXT
    ? strX

    strX = ""
'    REDIM nameNjobArr(2) AS STRING * 30 AT STRPTR(theString) '<< REDIM'ing same array name does not work properly
    REDIM nameWithJobArr(2) AS STRING * 30 AT STRPTR(theString)  'DIM or REDIM'ing new array name AT works properly
    FOR ii = 0 TO 2
       strX = strX & nameWithJobArr(ii) & $CRLF
    NEXT
    ? strX

END FUNCTION