storing a dynamic string array

[Ralph Berger]

Hi there,

is there a better way of saving a dynamic string array to disk as
looping all through it and save it line by line ?

Are there any documents about pb's internal array header ?

Best rgds
Ralph

[Lance Edmonds]

Internally, PowerBASIC uses OLE string engine for dynamic strings. No official documentation has been released on the array headers or handles.

If performance is your reason for this question, maybe you need to revise the way you are holding the data in memory? ie, use a single large "buffer" (and use pointers to manipulate the contents) instead of an array? What exactly is the problem you wish to solve?

------------------
Lance
PowerBASIC Support
mailto:[email protected][email protected]</A>

[Borje Hagsten]

Writing each element to disk can take a long time, but if you use
a string buffer when you save the array, it can speed up the process
a lot. The following example is written from memory, so it may need
some adjustment, but just to show you the general idea:

  LOCAL I AS LONG, bPos AS LONG, ub AS LONG, Buffer AS STRING
 
  ub = UBOUND(Array): bPos = 1: Buffer = SPACE$(32000)
 
  FOR I = 0 TO ub - 1
    MID$(Buffer, bPos) = Array(I) & $CRLF
    bPos = bPos + LEN(Array(I)) + 2
    IF bPos > 30000 OR I = ub - 1 THEN  'dump it to disk
      'the PRINT statement automatically adds a CRLF, so..
      PRINT# 1, LEFT$(Buffer, bPos - 3)
      bPos = 1: Buffer = SPACE$(32000)
    END IF
  NEXT

------------------

[Semen Matusovski]

I want to join to Borje - buffering is very important - first of all, for writing, but not less for reading.
I tested following code on my PC (550Mhz, UW SCSI)

#Compile Exe
#Register None
#Dim All
#Include "WIN32API.INC"

Function PbMain
  %MaxBufferSize = 8192
  Dim strAr(100000 : 900000) As String
  Local i As Long, l As Long, lb As Long, ub As Long, bPos As Long
  lb = LBound(strAr): ub = UBound(strAr)
  For i = lb To ub: strAr(i) = String$(115, "*") + " No." + Str$(i): Next

  ' Variant 1
  Local Buffer As String * %MaxBufferSize
  ' Variant 2 (approx. eq.1)
  'Local Buffer As String: Buffer = Space$(%MaxBufferSize)
  ' Variant 3 (not effective)
  'Local Buffer As Asciiz * (%MaxBufferSize + 1): Buffer = Space$(%MaxBufferSize)

  Local t1 As Double: t1 = Timer
  Open "~1" For Output As #1
  For i = lb To ub
    l = Len(strAr(i)) + 2
    If (%MaxBufferSize - bPos) < l Then _
        Print# 1, Left$(Buffer, bPos);: bPos = 0
    Mid$(Buffer, bPos + 1, l) = strAr(i) + $CRLF
    bPos = bPos + l
  Next
  Print# 1, Left$(Buffer, bPos);
  Close #1
  MsgBox "Writing time is " + Str$(Timer - t1)
  ReDim strAr(100000 : 900000) As String
  t1# = Timer
  Open "~1" For Input As #1
  For i = lb To ub
    Line Input #1, strAr(i)
  Next
  Close #1
  MsgBox "Reading time is " + Str$(Timer - t1)
End Function

To create 100 mb file is necessary about 8 sec (not bad).
Reading time (w/o buffering takes about 36 sec).
With size of buffer not all clear (at least, on my PC, because SCSI uses own cash). I feel difference when decrease size less than 1 Kb. But for ordinary PC, I think, it's necessary to take 1,5-2 * cluster size. Perhaps, 8 Kb is enough.

------------------


[This message has been edited by Semen Matusovski (edited February 26, 2000).]

[Wyman A Belts]

Ralph, I use a pretty easy/fast method that goes like this: When you save your dynamic strings collate them into one big string and save each strings length in a byte/int/long array (depending on the maximum length of your strings) at the same time. Save the index array and the string array as binary files. To retrieve them just open the index file and the string file and load them into memory in one shot. The number of strings in your file is the length of the index file divided by 1,2 or 4 (depending on what index size you used). To access an element, simply add up the indexes up to that index number-1 and this'll give you the start character of the string. The length of the string is in index number. So, to get element 5....x$ = mid$(BigString, Index(1)+Index(2)+Index(3)+Index(4), Index(5)). If these things are getting too big for memory you can do the same thing with get$ statements. Obviously it has it's limitations, especially if you need to edit the existing records. You can toy around with the general idea to make it work for you..ie after loading the two files into memory, put them into a dynamic array and destroy the string and index array to free memory. My goal was to make fast disk accesses across a network. Anyway, it's just another idea.

------------------

[Michael Mattias]

Lance, I think you comments re OLE strings apply only to 32-bit PB/DLL. Yes, PB still ships and supports a 16-bit compiler; and yes, 16-bit applications are still in use. In fact, the U.S. Government's HCFA (Health Card Financing Administration) mandates that software provided to Medicare providers by authorized TPAs work on 16-bit systems!

Anyway, assuming this is a 32-bit question, why do we need to save the string array to disk anyway? You can handle 2Gb of user data in the 32-bit environment in memory/Windows-managed virtual memory; are we trying to manage more than 2GB?



------------------
Michael Mattias
Racine WI USA
[email protected]

[Ralph Berger]

Thanks for all the replies. I tried to use a string array to
keep message texts of different size in memory. Found a much
smarter solution for my problem. I'll use a "structured file".
Pls look at my posting in programming - topic "in file - file system".

The aproach is a little bit like wyman's suggestion. But instead
of an index array a "in file file system" is more flexible.
The GNU CPP source i use for this is part of the Sinclair QL
emulator for linux. Shall i post it ?

Best rgds
Ralph

------------------

[Florent Heyworth]

Ralph

yes please: I'd love to look at the CPP source. If it's too long to post my e-mail address is mailto:[email protected][email protected]</A>

Thanks

Florent



[This message has been edited by Florent Heyworth (edited February 26, 2000).]