Announcement

Collapse
No announcement yet.

GET and arrays of UDTs

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • GET and arrays of UDTs

    I have an app which normally needs to read several arrays of
    different UDTs from disk files. Since many of these UDTs
    contain dynamic strings, I can't simply read the entire array
    directly, but must read the array into a buffer and then "parse"
    it into the individual elements.

    To further complicate the matter, I am not yet a PB purist - the
    target arrays are VB6 arrays - and worse still, the UDTs are
    ported from a 16bit app and VB6 DWORD aligns the data in a UDT.

    The parsing step I perform is very fast, but I'm always
    interested in performance tuning so I'm wondering about
    coding a generic UDT parsing routine in PB.

    I wrote such a routine in assembly for parsing VB3's UDT arrays,
    but I couldn't figure out if there was a pointer in the array
    descriptor that pointed at a "structure map" of the UDT - i.e.,
    given an array of type MyType, is there something in the array
    descriptor that points to a data structure that defines the
    structure of MyType. Since I couldn't find such a pointer, I
    had to pass a "structure map" to the routine whenever I wanted
    to use it. So, the parsing routine was generic, but I had no
    way of generating the structure map on the fly - it had to be
    hard-coded and if I wanted to change my UDT, I'd have to find
    every line of code where I called the parsing routine and
    modify the hard-coded structure map string.

    Confused?

    Given the following:

    Type MyType
    A As String
    B As Integer
    C As Long
    D As String
    End Type
    Global MyArray() As MyType

    I would hard code the following structure map string:

    chr$(3) & chr$(1) & chr$(6) & chr$(1)

    This structure map means:

    1. chr$(3) --> there are three blocks in this UDT
    1. chr$(1) --> this block is a dynamic string
    2. chr$(6) --> the next six characters can be loaded directly
    3. chr$(1) --> this block is a dynamic string

    In pseudocode, the parsing routine is:

    Sub ReadUDTArray(ByVal lpArray&, lElementCount&, lBlockSize&,
    lpStrucMap&, hFile&)

    ' hey this is pseudocode, assume all DIMs are here

    hBuffer = GlobalAlloc(GMEM_FIXED,lBlockSize)
    lpBuffer = GlobalLock(hBuffer)
    Fill buffer from hFile
    hMemCpy byteStrucLen, lpStrucMap, 1
    lBuffOffset = 0
    lOffset = 0
    For X = 1 to lElementCount
    For Y = 1 to byteStrucLen
    Select case @(lpStrucMap + Y)
    case 1 ' this is a dynamic string
    hMemCpy iSLen, lpBuffer + lBuffOffset, 2
    lBuffOffset = lBuffOffset + 2
    hString = AllocateString(iSLen)
    lpString = DerefString(hString)
    hMemCpy lpString, lpBuffer + lBuffOffset, iSLen
    lBuffOffset = lBuffOffset + iSLen
    hMemCpy lpArray + lOffset, hString, 2
    lpArray = lpArray + 2
    case else ' non-dynamic string (I didn't use variants)
    hMemCpy lpArray + lOffset, hString, @(lpStrucMap + Y)
    lpArray = lpArray + @(lpStrucMap + Y)
    End Select
    Next
    Next
    unlock buffer and free it
    end sub

    This routine worked beautifully under VB3, but I would like to do
    two things:

    1. convert it to work under Win32/VB6
    2. figure out how or if I can generate the structure map within
    the parsing routine itself

    Any thoughts?

    chris judge,
    formfill, inc.

  • #2
    A silly Question:
    How do you save an UDT to disk that has elements of dynamic string ?
    In VB6 the UDT element dynamic string will be a pointer to an StringDescriptor,
    and this will only be valid within that SUB/Function Class or application at the time the UDT is written


    ------------------
    Fred
    mailto:[email protected][email protected]</A>
    http://www.oxenby.se



    [This message has been edited by Fred Oxenby (edited March 01, 2000).]
    Fred
    mailto:[email protected][email protected]</A>
    http://www.oxenby.se

    Comment


    • #3
      I don’t know if this is what your looking for or not, but it might help.

      For my own mini-database program that I’m currently working on for a small client, I needed to have the table layouts be able to change without my program knowing it. What I did is have each file, or part/section of a file start with a template for the rest of the data. Your file loader then always starts by reading the template field and then uses that template to load the rest of the data. You then have a generic file loader that is as flexible as you make it.

      Colin Schmidt

      ------------------
      Colin Schmidt & James Duffy, Praxis Enterprises, Canada

      Comment


      • #4
        OK. I don't think anyone got my question, but to
        answer Fred Oxenby's question:

        Saving a UDT with dynamic string elements is easy.
        There are two ways to do it. Assuming you have:

        Type MyUDT
        A As String
        B As Integer
        End Type

        and

        Redim MyArray(100) As MyUDT

        you can save a single element of the array with a simple PUT:

        For X = 1 to 100
        Put hFile, , MyArray(1)
        Next

        Assume further than MyArray(1).A = "dynamic string" and that
        MyArray(1).B = 0, what will be written to the file in the first
        is iteration of the loop is (data in hex):

        00 0E 64 79 6E 61 6D 69 63 20 73 74 72 69 6E 6D 00 00

        To overcome the performance penalty of writing one element of
        MyArray at a time, you can copy the data to a buffer. Using
        the same UDT and array from the example above:

        For X = 1 to 100
        NeedSize = NeedSize + 4 + len(MyArray(X).A)
        Next
        hBuffer = GlobalAlloc(GMEM_FIXED, NeedSize)
        lpBuffer = GlobalLock(hBuffer)
        For X = 1 to 100
        StringToBuffer lpBuffer, lOffset, MyArray(X).A
        hMemCpy byval lpBuffer + lOffset, MyArray(X).B, 2
        lOffset = lOffset + 2
        Next
        WriteFile hFile, byval lpBuffer, lOffset
        GlobalUnlock hBuffer
        GlobalFree hBuffer

        where StringToBuffer is:

        Sub StringToBuffer(lpBuffer&, lOffset&, S$)
        Dim iSLen%
        iSLen = len(S)
        hMemCpy byval lpBuffer + lOffset, iSLen, 2
        lOffset = lOffset + 2
        hMemCpy byval lpBuffer + lOffset, byval S, iSLen
        lOffset = lOffset + iSLen
        End Sub

        Reading an array of UDTs (with dynamic string or not) is simply
        the reverse. Believe it or not, this seemingly roundabout code
        is many, many times faster than using PUT (or GET) in a loop.

        Anyway, my original question dealt with retrieving a pointer to
        some sort of a structure map for a UDT from the array descriptor
        of an array which has been declared as a UDT. This so that I
        could convert the above routines from VB to PBDll without
        having to maintain two sets of UDT definitions.

        chris judge

        ------------------

        Comment


        • #5
          Code:
          Chris--
             
           
          OK. I don't think anyone got my question, but...
          Fred has pointed out quite nicely the first issue to consider: the error which exists in the UDT. Type MyType A As String B As Integer C As Long D As String End Type Dynamic strings are not allowed in UDTs (for a very understandable reason). The compiler might accept this UDT, but its members simply cannot be accessed as you apparently believe they can be. Fix the UDT. Only then judge what's happening.

          ------------------
          -- Greg
          [email protected]

          Comment


          • #6
            Greg:

            You stated:

            "Dynamic strings are not allowed in UDTs (for a very understandable
            reason). The compiler might accept this UDT, but its members simply
            cannot be accessed as you apparently believe they can be."

            Maybe you missed part of this thread, the part where I stated:

            "...the target arrays are VB6 arrays - and worse still, the UDTs
            are ported from a 16bit app and VB6 DWORD aligns the data in a UDT."

            I realize that I may not have made myself completely clear, but
            the UDTs and arrays are in VB code. My original ReadUDTArray
            routine was written in assembly and I would like to port it to
            PB. Porting it as written is no problem, I was checking to see
            if anyone here knew if one can obtain a pointer to a "UDT map"
            from the data contained within an array descriptor.

            As for the inability to access dynamic strings contained within
            a UDT, I can very easily post a VB3 or VB6 example that shows that
            members of a UDT can be accessed this way. If you maintain
            that it can't be done, well let's call Fox 'cause my computer
            and those of my customer's must be possessed by some demon.

            Would you prefer the VB3 version which uses ReadUDTArray or
            would you prefer to see it work under VB6 which simply uses
            CopyMemory?



            ------------------

            Comment


            • #7
              if anyone here knew if one can obtain a pointer to a "UDT map" from the data contained within an array descriptor.
              You are probably going to have to ask a VB guru about this... It certainly cannot be done to PB UDT structures as the UDT member addresses are sorted out at compile time. At runtime there is no "map" of a UDT structure provided by PowerBASIC - it just does not exist.

              The solution appears to be fairly simple however - create your own "map" in VB and pass it to the PB code, and then it can use that information to deal with the raw VB data in the UDT array.

              A few years ago I wanted a routine to reset a UDT structure based on the member data types (ie, string bytes were set to CHR$(32) and numeric variables were to be set to 0). I solved the problem by writing a "table" that describes the UDT and from that I wrote a generic routine to parse the table and set the bytes accordingly.

              From memory, a table of "S32ILDZ64" translated to a UDT of:
              TYPE MyUDTType
              A AS STRING * 32
              B AS INTEGER
              C AS LONG
              d AS DOUBLE
              E AS ASCIIZ * 64
              END TYPE

              Good luck with your project!


              ------------------
              Lance
              PowerBASIC Support
              mailto:[email protected][email protected]</A>
              Lance
              mailto:[email protected]

              Comment


              • #8
                Stamina ASM-Library (DLL former Muscle) have a function
                close to what Lance describe, for use with 32-bit Visual Basic.
                This is called DxUDTPack, but please understand this note
                MicroDexterity Inc enclose:
                Important
                If there are dynamic elements in UDTVar (objects, dynamic arrays,
                variable-length strings, or variants containing those types),
                you must maintain UDTVar in scope while the byte array is in use.
                A Variable-length string in a VB-UDT is a pointer which looses its meaning when you write the UDT to the disk, for someone else
                to read.



                ------------------
                Fred
                mailto:[email protected][email protected]</A>
                http://www.oxenby.se

                Fred
                mailto:[email protected][email protected]</A>
                http://www.oxenby.se

                Comment


                • #9
                  Lance,

                  I had already solved the problem by passing my own "map" and I
                  suspected that a compiler (VB or PB) would remove any such UDT
                  map from the run-time code, but I just wanted to check with
                  other programmers. Seems you're the only one who understood
                  my question. Thanks,

                  chris judge,
                  formfill, inc.

                  Comment


                  • #10
                    Chris,

                    So, the parsing routine was generic, but I had no
                    way of generating the structure map on the fly - it had to be
                    hard-coded and if I wanted to change my UDT, I'd have to find
                    every line of code where I called the parsing routine and
                    modify the hard-coded structure map string.
                    What I was suggesting, as I knew of no UDT maps in run-time memory, was a way to prevent you from having to hard code the map into your application. You stated that at the moment you have the current map for the UDT hard coded into your file loader so that you could then pass it to your parsing routine. What I was suggesting was that you simply take this map out of your application and put in into your disk file. Then when loading the file from disk, you first read the map, then load the data into your buffer, and then pass the map and the buffer to your parsing routine. You can then change your UDT with out having to go through your hard code changing the map.

                    Colin Schmidt

                    ------------------
                    Colin Schmidt & James Duffy, Praxis Enterprises, Canada

                    Comment


                    • #11
                      Colin:

                      Thanks for the clarification. However, your method does not meet
                      my requirements for one main reason: it is not a drop-in
                      replacement for GET.

                      Consider the following VB snippet:

                      Type udtExample
                      A As String
                      B As long
                      End Type
                      ...
                      Dim Ex as udtExample
                      ...
                      Get hFile, , Ex

                      In this example, VB knows to read an integer, allocate the amount
                      of space specified by the integer, read that many characters into
                      Ex.A and then, finally, read four more bytes into Ex.B.

                      There are two ways in which this could be done (it's probably
                      done via method #2):

                      1. VB maintains a link during run-time between the variable Ex
                      and a map of udtExample, or
                      2. during compilation, VB converts the simple (Get hFile,,Ex)
                      into:

                      Get hFile, , iStrLen
                      Ex.A = Space$(iStrLen)
                      Get hFile, , Ex.A
                      Get hFile, , Ex.B

                      Whichever method is used, our pals at MS could have included
                      a variation of GET (say MemGet) in which the source is a
                      memory address rather than a file. However, if method 2 is used,
                      then the only way I can simulate the desired MemGet is to pass
                      a "parsing map" to the MemGet or to write my own pre-compiler
                      which would precompile the following:

                      MemGet lpBuffer, Ex

                      to

                      hMemCpy iStrLen, byval lpBuffer, 2
                      Ex.A = Space$(iStrLen)
                      hMemCpy Ex.A, byval lpBuffer + 2, iStrLen
                      hMemCpy Ex.B, byval lpBuffer + 2 + iStrLen, 4

                      The precompiler route would be cumbersome unless it could be performed
                      by some sort of VB IDE plug-in and it's not optimal because the
                      resulting code would still be VB code rather than optimised
                      assembly or PB code.

                      In the 16-bit version of the app I'm working on, I wrote a
                      parsing routine which included as a parameter a map of the udt
                      being read. When porting to VB6, I discovered that VB6 now
                      DWORD aligns udt elements thus requiring my parsing routine
                      to keep track of the current udt position's alignment. The data
                      being read from file to memory to udt is byte aligned and "should"
                      stay that way for backwards compatibility (I have one customer
                      with over 1GB of byte-aligned data. So, rather than modifying
                      my dll or bloating my product with two file-loading routines (one
                      for byte-aligned data and one for DWORD aligned data), I abandoned
                      the dll and manually performed the "precompiling" described above.

                      So maybe PB will introduce a MemGet and a udt that supports
                      dynamic strings. Getting that to jive with unions might be
                      tough, but when you control the compiler you can do anything.

                      chris judge,
                      formfill, inc.


                      ------------------

                      Comment

                      Working...
                      X