GET and arrays of UDTs

[chrisjudge]

I have an app which normally needs to read several arrays of
different UDTs from disk files. Since many of these UDTs
contain dynamic strings, I can't simply read the entire array
directly, but must read the array into a buffer and then "parse"
it into the individual elements.

To further complicate the matter, I am not yet a PB purist - the
target arrays are VB6 arrays - and worse still, the UDTs are
ported from a 16bit app and VB6 DWORD aligns the data in a UDT.

The parsing step I perform is very fast, but I'm always
interested in performance tuning so I'm wondering about
coding a generic UDT parsing routine in PB.

I wrote such a routine in assembly for parsing VB3's UDT arrays,
but I couldn't figure out if there was a pointer in the array
descriptor that pointed at a "structure map" of the UDT - i.e.,
given an array of type MyType, is there something in the array
descriptor that points to a data structure that defines the
structure of MyType. Since I couldn't find such a pointer, I
had to pass a "structure map" to the routine whenever I wanted
to use it. So, the parsing routine was generic, but I had no
way of generating the structure map on the fly - it had to be
hard-coded and if I wanted to change my UDT, I'd have to find
every line of code where I called the parsing routine and
modify the hard-coded structure map string.

Confused?

Given the following:

Type MyType
A As String
B As Integer
C As Long
D As String
End Type
Global MyArray() As MyType

I would hard code the following structure map string:

chr$(3) & chr$(1) & chr$(6) & chr$(1)

This structure map means:

1. chr$(3) --> there are three blocks in this UDT
1. chr$(1) --> this block is a dynamic string
2. chr$(6) --> the next six characters can be loaded directly
3. chr$(1) --> this block is a dynamic string

In pseudocode, the parsing routine is:

Sub ReadUDTArray(ByVal lpArray&, lElementCount&, lBlockSize&,
lpStrucMap&, hFile&)

' hey this is pseudocode, assume all DIMs are here

hBuffer = GlobalAlloc(GMEM_FIXED,lBlockSize)
lpBuffer = GlobalLock(hBuffer)
Fill buffer from hFile
hMemCpy byteStrucLen, lpStrucMap, 1
lBuffOffset = 0
lOffset = 0
For X = 1 to lElementCount
For Y = 1 to byteStrucLen
Select case @(lpStrucMap + Y)
case 1 ' this is a dynamic string
hMemCpy iSLen, lpBuffer + lBuffOffset, 2
lBuffOffset = lBuffOffset + 2
hString = AllocateString(iSLen)
lpString = DerefString(hString)
hMemCpy lpString, lpBuffer + lBuffOffset, iSLen
lBuffOffset = lBuffOffset + iSLen
hMemCpy lpArray + lOffset, hString, 2
lpArray = lpArray + 2
case else ' non-dynamic string (I didn't use variants)
hMemCpy lpArray + lOffset, hString, @(lpStrucMap + Y)
lpArray = lpArray + @(lpStrucMap + Y)
End Select
Next
Next
unlock buffer and free it
end sub

This routine worked beautifully under VB3, but I would like to do
two things:

1. convert it to work under Win32/VB6
2. figure out how or if I can generate the structure map within
the parsing routine itself

Any thoughts?

chris judge,
formfill, inc.

[Fred Oxenby]

A silly Question:
How do you save an UDT to disk that has elements of dynamic string ?
In VB6 the UDT element dynamic string will be a pointer to an StringDescriptor,
and this will only be valid within that SUB/Function Class or application at the time the UDT is written


------------------
Fred
mailto:[email protected][email protected]</A>
http://www.oxenby.se



[This message has been edited by Fred Oxenby (edited March 01, 2000).]

[Colin Schmidt]

I don’t know if this is what your looking for or not, but it might help.

For my own mini-database program that I’m currently working on for a small client, I needed to have the table layouts be able to change without my program knowing it. What I did is have each file, or part/section of a file start with a template for the rest of the data. Your file loader then always starts by reading the template field and then uses that template to load the rest of the data. You then have a generic file loader that is as flexible as you make it.

Colin Schmidt

------------------
Colin Schmidt & James Duffy, Praxis Enterprises, Canada

[chrisjudge]

OK. I don't think anyone got my question, but to
answer Fred Oxenby's question:

Saving a UDT with dynamic string elements is easy.
There are two ways to do it. Assuming you have:

Type MyUDT
A As String
B As Integer
End Type

and

Redim MyArray(100) As MyUDT

you can save a single element of the array with a simple PUT:

For X = 1 to 100
Put hFile, , MyArray(1)
Next

Assume further than MyArray(1).A = "dynamic string" and that
MyArray(1).B = 0, what will be written to the file in the first
is iteration of the loop is (data in hex):

00 0E 64 79 6E 61 6D 69 63 20 73 74 72 69 6E 6D 00 00

To overcome the performance penalty of writing one element of
MyArray at a time, you can copy the data to a buffer. Using
the same UDT and array from the example above:

For X = 1 to 100
NeedSize = NeedSize + 4 + len(MyArray(X).A)
Next
hBuffer = GlobalAlloc(GMEM_FIXED, NeedSize)
lpBuffer = GlobalLock(hBuffer)
For X = 1 to 100
StringToBuffer lpBuffer, lOffset, MyArray(X).A
hMemCpy byval lpBuffer + lOffset, MyArray(X).B, 2
lOffset = lOffset + 2
Next
WriteFile hFile, byval lpBuffer, lOffset
GlobalUnlock hBuffer
GlobalFree hBuffer

where StringToBuffer is:

Sub StringToBuffer(lpBuffer&, lOffset&, S$)
Dim iSLen%
iSLen = len(S)
hMemCpy byval lpBuffer + lOffset, iSLen, 2
lOffset = lOffset + 2
hMemCpy byval lpBuffer + lOffset, byval S, iSLen
lOffset = lOffset + iSLen
End Sub

Reading an array of UDTs (with dynamic string or not) is simply
the reverse. Believe it or not, this seemingly roundabout code
is many, many times faster than using PUT (or GET) in a loop.

Anyway, my original question dealt with retrieving a pointer to
some sort of a structure map for a UDT from the array descriptor
of an array which has been declared as a UDT. This so that I
could convert the above routines from VB to PBDll without
having to maintain two sets of UDT definitions.

chris judge

------------------

[Greg Turgeon]

Chris--
   
 [quote]OK. I don't think anyone got my question, but...[/quote]
   
Fred has pointed out quite nicely the first issue to consider: the 
error which exists in the UDT.
   
   Type MyType
     A As String
     B As Integer
     C As Long
     D As String
   End Type
   
Dynamic strings are not allowed in UDTs (for a very understandable 
reason).  The compiler might accept this UDT, but its members simply 
cannot be accessed as you apparently believe they can be.
   
Fix the UDT.  Only then judge what's happening.

------------------
-- Greg
[email protected]

[chrisjudge]

Greg:

You stated:

"Dynamic strings are not allowed in UDTs (for a very understandable
reason). The compiler might accept this UDT, but its members simply
cannot be accessed as you apparently believe they can be."

Maybe you missed part of this thread, the part where I stated:

"...the target arrays are VB6 arrays - and worse still, the UDTs
are ported from a 16bit app and VB6 DWORD aligns the data in a UDT."

I realize that I may not have made myself completely clear, but
the UDTs and arrays are in VB code. My original ReadUDTArray
routine was written in assembly and I would like to port it to
PB. Porting it as written is no problem, I was checking to see
if anyone here knew if one can obtain a pointer to a "UDT map"
from the data contained within an array descriptor.

As for the inability to access dynamic strings contained within
a UDT, I can very easily post a VB3 or VB6 example that shows that
members of a UDT can be accessed this way. If you maintain
that it can't be done, well let's call Fox 'cause my computer
and those of my customer's must be possessed by some demon.

Would you prefer the VB3 version which uses ReadUDTArray or
would you prefer to see it work under VB6 which simply uses
CopyMemory?



------------------

[Lance Edmonds]

if anyone here knew if one can obtain a pointer to a "UDT map" from the data contained within an array descriptor.

You are probably going to have to ask a VB guru about this... It certainly cannot be done to PB UDT structures as the UDT member addresses are sorted out at compile time. At runtime there is no "map" of a UDT structure provided by PowerBASIC - it just does not exist.

The solution appears to be fairly simple however - create your own "map" in VB and pass it to the PB code, and then it can use that information to deal with the raw VB data in the UDT array.

A few years ago I wanted a routine to reset a UDT structure based on the member data types (ie, string bytes were set to CHR$(32) and numeric variables were to be set to 0). I solved the problem by writing a "table" that describes the UDT and from that I wrote a generic routine to parse the table and set the bytes accordingly.

From memory, a table of "S32ILDZ64" translated to a UDT of:
TYPE MyUDTType
A AS STRING * 32
B AS INTEGER
C AS LONG
d AS DOUBLE
E AS ASCIIZ * 64
END TYPE

Good luck with your project!


------------------
Lance
PowerBASIC Support
mailto:[email protected][email protected]</A>

[Fred Oxenby]

Stamina ASM-Library (DLL former Muscle) have a function
close to what Lance describe, for use with 32-bit Visual Basic.
This is called DxUDTPack, but please understand this note
MicroDexterity Inc enclose:

Important
If there are dynamic elements in UDTVar (objects, dynamic arrays,
variable-length strings, or variants containing those types),
you must maintain UDTVar in scope while the byte array is in use.

A Variable-length string in a VB-UDT is a pointer which looses its meaning when you write the UDT to the disk, for someone else
to read.



------------------
Fred
mailto:[email protected][email protected]</A>
http://www.oxenby.se

[chrisjudge]

Lance,

I had already solved the problem by passing my own "map" and I
suspected that a compiler (VB or PB) would remove any such UDT
map from the run-time code, but I just wanted to check with
other programmers. Seems you're the only one who understood
my question. Thanks,

chris judge,
formfill, inc.

[Colin Schmidt]

Chris,

So, the parsing routine was generic, but I had no
way of generating the structure map on the fly - it had to be
hard-coded and if I wanted to change my UDT, I'd have to find
every line of code where I called the parsing routine and
modify the hard-coded structure map string.

What I was suggesting, as I knew of no UDT maps in run-time memory, was a way to prevent you from having to hard code the map into your application. You stated that at the moment you have the current map for the UDT hard coded into your file loader so that you could then pass it to your parsing routine. What I was suggesting was that you simply take this map out of your application and put in into your disk file. Then when loading the file from disk, you first read the map, then load the data into your buffer, and then pass the map and the buffer to your parsing routine. You can then change your UDT with out having to go through your hard code changing the map.

Colin Schmidt

------------------
Colin Schmidt & James Duffy, Praxis Enterprises, Canada

[chrisjudge]

Colin:

Thanks for the clarification. However, your method does not meet
my requirements for one main reason: it is not a drop-in
replacement for GET.

Consider the following VB snippet:

Type udtExample
A As String
B As long
End Type
...
Dim Ex as udtExample
...
Get hFile, , Ex

In this example, VB knows to read an integer, allocate the amount
of space specified by the integer, read that many characters into
Ex.A and then, finally, read four more bytes into Ex.B.

There are two ways in which this could be done (it's probably
done via method #2):

1. VB maintains a link during run-time between the variable Ex
and a map of udtExample, or
2. during compilation, VB converts the simple (Get hFile,,Ex)
into:

Get hFile, , iStrLen
Ex.A = Space$(iStrLen)
Get hFile, , Ex.A
Get hFile, , Ex.B

Whichever method is used, our pals at MS could have included
a variation of GET (say MemGet) in which the source is a
memory address rather than a file. However, if method 2 is used,
then the only way I can simulate the desired MemGet is to pass
a "parsing map" to the MemGet or to write my own pre-compiler
which would precompile the following:

MemGet lpBuffer, Ex

to

hMemCpy iStrLen, byval lpBuffer, 2
Ex.A = Space$(iStrLen)
hMemCpy Ex.A, byval lpBuffer + 2, iStrLen
hMemCpy Ex.B, byval lpBuffer + 2 + iStrLen, 4

The precompiler route would be cumbersome unless it could be performed
by some sort of VB IDE plug-in and it's not optimal because the
resulting code would still be VB code rather than optimised
assembly or PB code.

In the 16-bit version of the app I'm working on, I wrote a
parsing routine which included as a parameter a map of the udt
being read. When porting to VB6, I discovered that VB6 now
DWORD aligns udt elements thus requiring my parsing routine
to keep track of the current udt position's alignment. The data
being read from file to memory to udt is byte aligned and "should"
stay that way for backwards compatibility (I have one customer
with over 1GB of byte-aligned data. So, rather than modifying
my dll or bloating my product with two file-loading routines (one
for byte-aligned data and one for DWORD aligned data), I abandoned
the dll and manually performed the "precompiling" described above.

So maybe PB will introduce a MemGet and a udt that supports
dynamic strings. Getting that to jive with unions might be
tough, but when you control the compiler you can do anything.

chris judge,
formfill, inc.


------------------