Announcement

Collapse
No announcement yet.

Help with a file format

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Help with a file format

    Hello. I am trying to write a program that will read and write
    PalmOS database files. Unfortunately none of the tools
    currently available meet my needs (or don't work with PB) so I
    am going to write my own. However, file formats are not my
    strong suit. Below is the format to .PDB files that the Palm
    devices use. Can someone point me in the right direction on
    how to read these from PB?

    A .pdb file consists of a Database Header (78 bytes) followed
    by a table of Record Entry Headers (one for each database
    record). The data for all records follows the table of Record
    Entry Headers.


    --- Database Header (78 bytes) ---

    Field Bytes Value
    ----- ----- -----
    DB Name 32 <Database Name>
    Flags 2 $00, $00
    Version 2 $00, $00
    Creation Time 4 $06, $D1, $44, $AE
    Modificaton Time 4 $06, $D1, $44, $AE
    Backup Time 4 $00, $00, $00, $00
    Modification Number 4 $00, $00, $00, $00
    App Info Offset 4 $00, $00, $00, $00
    Sort Info Offset 4 $00, $00, $00, $00
    Type 4 <Database ID>
    Creator 4 <Application ID>
    unique ID seed 4 $00, $00, $00, $00
    next record list id 4 $00, $00, $00, $00
    Number of records 2 <Number of Records>

    --- Record 0 Entry Header (8 bytes) --- (One entry for each record)

    Field Bytes Value
    ----- ----- -----
    Offset 4 <offset to record 0 from the beginning of the file>
    Record attrib (1 byte)
    & unique ID (3 bytes) 4 $40, $6F, $80, $00

    .
    .
    .

    --- Record n Entry Header (8 bytes) --- (One entry for each record)

    Field Bytes Value
    ----- ----- -----
    Offset 4 <offset to record n from the beginning of the file>
    Record attrib (1 byte)
    & unique ID (3 bytes) 4 $40, $6F, $80, $00

    --- Record 0 Data --- (One entry for each record)

    .
    .
    .

    --- Record n Data --- (One entry for each record)

    --- End of File ----


  • #2
    This is just a simple, untested record retrieval. You need to have some information about
    record lengths to do a proper job. You also need to know the structure of the Time stamp
    variables (CreationTime, etc.)
    Code:
    '######################################################################################
    #COMPILE EXE
    '#INCLUDE "WIN32API.INC"
    
    TYPE HeaderStruc
      dbName             AS STRING * 32
      Flags              AS INTEGER
      Version            AS INTEGER
      CreationTime       AS LONG
      ModificationTime   AS LONG
      BackupTime         AS LONG
      ModificationNumber AS LONG
      AppInfoOffset      AS LONG
      SortInfoOffset     AS LONG
      dbType             AS LONG
      Creator            AS LONG
      IDseed             AS LONG
      nextRecordListID   AS LONG
      NumberOfRecords    AS INTEGER
    END TYPE
    
    TYPE RecordStruc
      RecordOffset  AS LONG
      RecordInfo    AS LONG ' Attrib as Byte, ID AS 3 BYTES
    END TYPE
    '######################################################################################
    FUNCTION PBMAIN() AS LONG
      LOCAL header     AS HeaderStruc, _
            recInfo    AS RecordStruc, _
            record     AS STRING, _
            ndx        AS LONG, _
            fhandle    AS LONG, _
            fileLen    AS LONG, _
            recNo      AS LONG, _
            nextOffset AS LONG
    
      DIM recInfo(1:1) AS RecordStruc
      fhandle = FREEFILE
    ' Open a file with a fixed name
      OPEN "myfile.pdb" FOR BINARY AS # fhandle
    
    ' Retrieve the header
      GET # fhandle, 1, header
      IF header.NumberOfRecords < 1 THEN
        MSGBOX "No data in your file"
        FUNCTION = 0
        EXIT FUNCTION
      END IF
      
    ' Retrieve the Record Structure Info for each record, place in array
      REDIM recInfo(1:header.NumberOfRecords)
      FOR ndx = 1 TO header.NumberOfRecords
        GET # fhandle, , recInfo(ndx)
      NEXT ndx
      
    ' What is not mentioned is the length of a record.
    ' Without that knowledge, it becomes more difficult,
    ' but not impossible to display.
      fileLen = LOF(fhandle)
    ' .ie. let us look at the contents of the first record.
      recNo = 1
    ' find the offset of the next record in the file.
    ' If this is the last record, then set the offset to read to the end of the file.
      nextOffset = fileLen + 1
      FOR ndx = 1 TO header.NumberOfRecords
        IF recInfo(ndx).RecordOffset > recInfo(recNo).RecordOffset THEN
          nextOffset = MIN&(nextOffset, recInfo(ndx).RecordOffset )
        END IF
      NEXT ndx
    ' Place file file pointer
      SEEK fhandle, recInfo(recNo).RecordOffset
    ' Read the data into a record string
      GET$ # fhandle, nextOffset - recInfo(recNo).RecordOffset, record
    ' and Display it - Binary data may display unreadable characters!
      MSGBOX "Record: " + record
      
      CLOSE # fhandle
      
    END FUNCTION
    '######################################################################################
    regards,

    ------------------
    [email protected]
    :) IRC :)

    Comment


    • #3
      Thanks, that helped a bit. However, I'm just having trouble
      reading the header. For example, in the Palm documentation
      for the PDB format it says that the number of records is stored
      as a WORD value, and the creation times are stored as DWORD
      values. When I read the data into variables I have DIMmed in
      this manner, it is not returning accurate values (i.e. it is
      reporting 77 records when it should be 68, and the times are
      not accurate).

      Any ideas?

      Comment


      • #4
        www.wotsit.org lists 2 files outlining the PDB format:

        http://www.wotsit.org/download.asp?f=pdb

        http://www.wotsit.org/download.asp?f=pdb-fmt



        ------------------
        Lance
        PowerBASIC Support
        mailto:[email protected][email protected]</A>
        Lance
        mailto:[email protected]

        Comment


        • #5
          Thanks.. those don't tell me any more than the Palm API
          docs though. But I have at least figured out what the
          problem is... the Motorola processors save DWORD values in a
          different format that Intel processors. So all I need to do
          now is figure out a way to convert it.


          ------------------

          Comment


          • #6
            Right, the old little-endian (Intel) vs big-endian (Motorola) problem... You should be able to just reverse the byte order to get an Intel Dword.

            For example, if you load the big-endian DWORDS into a string, you can use STRREVERSE$ and convert back to DWORD, ie:
            Code:
            #COMPILE EXE
            FUNCTION PBMAIN
                a??? = &H0FFEEDDCC ' big-endian value
                b??? = CVDWD(STRREVERSE$(MKDWD$(a???)))
                MSGBOX HEX$(a???) & "=>" & HEX$(b???)
            END FUNCTION
            There are other ways, such as assembling the value with pointers, etc, but this is fairly fast and quite simple.

            ------------------
            Lance
            PowerBASIC Support
            mailto:[email protected][email protected]</A>
            Lance
            mailto:[email protected]

            Comment


            • #7
              Thanks Lance.. I wasn't aware of the string reverse function and
              was doing it the "hard way" ....


              ------------------

              Comment


              • #8
                Here's another way:

                Code:
                #COMPILE EXE
                FUNCTION PBMAIN
                    a??? = &H0FFEEDDCC ' big-endian value
                    b??? = a??? 'Copy
                    ! bswap a??? 'Byte swap
                    MSGBOX HEX$(b???) & "=>" & HEX$(a???)
                END FUNCTION
                Cheers

                Florent

                ------------------

                Comment


                • #9
                  Well this code works (thnaks guys) but I still can't seem
                  to make it into a decimal number that makes sense.


                  ------------------

                  Comment


                  • #10
                    I'm only glossing over this thread, but what if you created a union between a decimal number and a dword, then make the big endian swap as neccessary in the dword and the decimal equivelent through the union will be corrected.

                    ------------------
                    George W. Bleck
                    Senior System Engineer
                    KeySpan Corporation
                    <b>George W. Bleck</b>
                    <img src='http://www.blecktech.com/myemail.gif'>

                    Comment


                    • #11
                      A union?


                      ------------------

                      Comment


                      • #12
                        Technically it would work (although you still need to go through the byte reversal procedure), but why bother with that arrangement as it would offer less advantage than the techniques outlined above?! Makes no sense to me... sorry!


                        ------------------
                        Lance
                        PowerBASIC Support
                        mailto:[email protected][email protected]</A>
                        Lance
                        mailto:[email protected]

                        Comment


                        • #13
                          Well using the technique above I'm still not pulling out the
                          correct values. I know what they should be since I created the
                          database on my Palm device. I also double checked in Windows
                          with a program I have that displays the data in the PDB files.

                          I must be missing something really obvious.

                          Comment


                          • #14
                            How about posting the binary of the values, aalong with the expected integer representations - maybe we can correlate them and suggest a solution.


                            ------------------
                            Lance
                            PowerBASIC Support
                            mailto:[email protected][email protected]</A>
                            Lance
                            mailto:[email protected]

                            Comment


                            • #15
                              OK Lance... thank you to you and everyone else who has been
                              trying to help me. I feel a little bad about it, like you guys
                              are doing my work for me.. but I am really getting frustrated
                              trying to figure this out.

                              I posted my PDB file to the following URL: http://www.seekersofwisdom.com/EMPIN.PDB

                              (note that the filename, EMPIN.PDB is upper case). It's a
                              small (4k) file.

                              Here is what the values SHOULD be:
                              DB Name: EMPIN
                              Attributes: 0x0008
                              Version: 0
                              Creation Date: -1217854540
                              Modfication Date: -1217854540
                              Last Backup Date: -1217854540
                              Modification Number: 0
                              AppInfoArea: 624 (Size = 158) (not important, not sure what its for)
                              SortInfoArea: 0 (not important)
                              Database Type: DB99
                              Creator ID: DBOS
                              UniqueIDSeed: 0
                              NextRecordListID: 0
                              NumberOfRecord: 68

                              This information is pulled from my table with a reliable Windows
                              tool I have to read PDB database information.

                              I can also post my code here if that would be helpful... the
                              error could always be coming out of something I'm doing there.


                              Comment


                              • #16
                                Not sure if this is helpfull but I found a PDB list program and it's source code written in C.

                                I have uploaded it to my website if you care to compare it to what you are working on.

                                http://www.blecktech.com/listfull.zip

                                Hope that helps...

                                ------------------
                                George W. Bleck
                                Senior System Engineer
                                KeySpan Corporation
                                <b>George W. Bleck</b>
                                <img src='http://www.blecktech.com/myemail.gif'>

                                Comment


                                • #17
                                  This should be enough to keep you going for a while.
                                  This post contains the date/time conversions.
                                  Credit for most of this goes to Egbert Zijlema's Julian Date routines.

                                  I moved away from the ASM byteSwap conversion due to compile errors.

                                  Code:
                                  '###########################################
                                  #COMPILE EXE
                                  #REGISTER NONE
                                  #INCLUDE "WIN32API.INC"
                                  DECLARE FUNCTION DateToJulian(inDate AS STRING) AS DWORD
                                  DECLARE FUNCTION JulToDate(BYVAL Jul AS DWORD) AS STRING
                                  DECLARE FUNCTION SecondsToTime(BYVAL inSeconds AS DWORD) AS STRING
                                  
                                  TYPE HeaderStruc
                                    dbName             AS ASCIIZ * 32
                                    Flags              AS WORD
                                    Version            AS WORD
                                    CreationTime       AS DWORD
                                    ModificationTime   AS DWORD
                                    BackupTime         AS DWORD
                                    ModificationNumber AS DWORD
                                    AppInfoOffset      AS DWORD
                                    SortInfoOffset     AS DWORD
                                    dbType             AS STRING * 4
                                    Creator            AS STRING * 4
                                    IDseed             AS DWORD
                                    nextRecordListID   AS DWORD
                                    NumberOfRecords    AS WORD
                                  END TYPE
                                  
                                  TYPE RecordStruc
                                    RecordOffset  AS DWORD ' 4 <offset to record 0 from the beginning of the file>
                                    RecordAttrib  AS BYTE  ' Attrib as Byte
                                    ID            AS STRING * 3
                                  END TYPE
                                  
                                  TYPE WordStruc
                                    byte0 AS BYTE
                                    byte1 AS BYTE
                                  END TYPE
                                  
                                  TYPE dWordStruc
                                    word0 AS WordStruc
                                    word1 AS WordStruc
                                  END TYPE
                                  '###########################################
                                  FUNCTION PBMAIN() AS LONG
                                    LOCAL header     AS HeaderStruc, _
                                          recInfo    AS RecordStruc, _
                                          record     AS STRING, _
                                          ndx        AS LONG, _
                                          fhandle    AS LONG, _
                                          fileLen    AS LONG, _
                                          recNo      AS LONG, _
                                          nextOffset AS LONG, _
                                          NoRecs     AS WORD, _
                                          firstRec   AS WORD, _
                                          recordLen  AS WORD, _
                                          Created    AS DWORD, _
                                          fieldOffset AS DWORD, _
                                          sortOffset AS DWORD, _
                                          buffer     AS STRING, _
                                          tString    AS STRING, _
                                          bufferLen  AS DWORD, _
                                          dataOffset AS DWORD, _
                                          testValue  AS DWORD, _
                                          noFields   AS WORD, _
                                          fldPtr     AS DWORD, _
                                          baseDate   AS LONG
                                  
                                  LOCAL inDate AS STRING, julDate AS LONG, remainder AS DWORD, testDate AS LONG
                                  
                                  ' Dates are calculated in seconds from Jan 1, 1904 = Julian Date: 2416481
                                    baseDate = 2416481
                                  
                                    DIM recInfo(1:1) AS RecordStruc
                                    fhandle = FREEFILE
                                  ' Open a file with a fixed name - Zero-based Offset
                                    OPEN "EmpIn.pdb" FOR BINARY AS # fhandle BASE = 0
                                  
                                  ' Retrieve the header
                                    GET # fhandle, 0, header
                                    IF header.NumberOfRecords < 1 THEN
                                      MSGBOX "No data in your file"
                                      FUNCTION = 0
                                      EXIT FUNCTION
                                    END IF
                                  ' Mark the position from which to read record Info.
                                    dataOffset = SEEK(fhandle)
                                  
                                    NoRecs = header.NumberOfRecords
                                    CALL SwapWord(noRecs)
                                    Created = header.CreationTime
                                    CALL SwapDword(Created)
                                  
                                  ' Convert Creation Date to Display Date
                                    testValue = Created \ (60*60*24)  ' days
                                    remainder = Created - testvalue * (60*60*24)
                                    testDate = baseDate + testValue
                                    inDate = JulToDate(testDate)
                                    inDate = inDate + " - " +  SecondsToTime(remainder)
                                  
                                    MSGBOX "Program: " + header.dbName + CHR$(13) + _
                                           "Creation Time: " + inDate + CHR$(13) + _
                                           "database Type: " + header.dbType + CHR$(13) + _
                                           "Creator: " + header.Creator + CHR$(13) + _
                                           "No Records: " + FORMAT$(NoRecs)
                                  
                                  ' Retrieve the Record Structure Info for each record, place in array
                                    firstRec = 0
                                    testValue = LOF(fhandle)  ' This will hold the address of the first record
                                    SEEK # fhandle, dataOffset
                                    REDIM recInfo(1:noRecs)
                                    FOR ndx = 1 TO noRecs
                                      GET # fhandle, , recInfo(ndx)
                                      CALL SwapDword(recInfo(ndx).RecordOffset)
                                      IF recInfo(ndx).RecordOffset < testValue THEN
                                        firstRec = ndx
                                        testValue = recInfo(ndx).RecordOffset
                                      END IF
                                    NEXT ndx
                                  
                                  ' If there is field Information, then recover it.
                                    fieldOffset = header.AppInfoOffset
                                    CALL SwapDword(fieldOffset)
                                    IF fieldOffset > 0 THEN
                                      sortOffset = header.SortInfoOffset
                                      CALL SwapDword(sortOffset)
                                      IF SortOffset > 0 THEN
                                        bufferLen = sortOffset - fieldOffset
                                      ELSE
                                        bufferLen = testValue - fieldOffset
                                      END IF
                                      SEEK fhandle, fieldOffset
                                      GET$ # fhandle, bufferLen, buffer
                                  ' Number of Fields: 2nd dWord
                                      noFields = CVDWD(MID$(buffer, 5, 2))
                                      CALL SwapWord(noFields)
                                      DIM fieldNames(1:noFields) AS STRING
                                      fldPtr = 9
                                      FOR ndx = 1 TO noFields
                                        fieldNames(ndx) = TRIM$(MID$(buffer, fldPtr, 32), ANY CHR$(0,32) )
                                        IF ndx < noFields THEN
                                          fldPtr = INSTR(fldPtr+32, buffer, CHR$(&h20,0,&h50) ) + 5
                                        END IF
                                      tString =  MID$(buffer, fldPtr+3)
                                      NEXT ndx
                                  
                                      tString = "No. Fields: " + FORMAT$(noFields)
                                      FOR ndx = 1 TO noFields
                                        tString = tString + CHR$(13) + fieldNames(ndx)
                                      NEXT tString
                                      MSGBOX tString
                                    END IF
                                  
                                  'Retrieve Record ID of record 3:
                                    recNo = 3
                                    testValue = CVDWD(CHR$(0) + recInfo(recNo).ID)
                                    CALL SwapDword(testValue)
                                  ' find the offset of the next record in the file.
                                  ' If this is the last record, then set the offset to read to the end of the file.
                                    nextOffset = LOF(fhandle) + 1
                                    FOR ndx = 1 TO noRecs
                                      IF recNo <> ndx AND recInfo(ndx).RecordOffset > recInfo(recNo).RecordOffset THEN
                                        nextOffset = MIN&(nextOffset, recInfo(ndx).RecordOffset )
                                      END IF
                                    NEXT ndx
                                  ' Place file file pointer
                                    SEEK fhandle, recInfo(recNo).RecordOffset
                                  ' Read the data into a record string
                                    GET$ # fhandle, nextOffset - recInfo(recNo).RecordOffset, record
                                  
                                  ' display Record 3
                                    tString = "Record 3: " + CHR$(13) + "Test ID: " + FORMAT$(testValue)
                                    fldPtr = 1
                                    FOR ndx = 1 TO noFields
                                      testValue = INSTR(fldPtr, record, CHR$(0))
                                      tString = tString + CHR$(13) + fieldNames(ndx) + ":  " + MID$(record, fldPtr, testValue - fldPtr)
                                      fldPtr = testValue + 1
                                    NEXT ndx
                                    MSGBOX tString
                                  
                                    CLOSE # fhandle
                                  
                                  END FUNCTION
                                  '###########################################
                                  SUB SwapWord(inWord AS WORD)
                                    DIM tempWord  AS WordStruc
                                  
                                    LSET tempWord = MKWRD$(inWord)
                                    SWAP tempWord.byte0, tempWord.byte1
                                    inWord = CVWRD(tempWord)
                                  END SUB
                                  '###########################################
                                  SUB SwapDword(inDword AS DWORD)
                                    DIM tempdWord AS dWordStruc
                                  
                                    LSET tempdWord = MKDWD$(inDword)
                                    SWAP tempdWord.word0.byte0, tempdWord.word0.byte1
                                    SWAP tempdWord.word1.byte0, tempdWord.word1.byte1
                                    SWAP tempdWord.word0, tempdWord.word1
                                    inDword = CVDWD(tempdWord)
                                  END SUB
                                  '################################################################
                                  FUNCTION SecondsToTime(BYVAL inSeconds AS DWORD) AS STRING
                                  ' Converts Seconds into 24 hour format
                                    LOCAL hours   AS DWORD, _
                                          minutes AS DWORD, _
                                          seconds AS DWORD, _
                                          tString AS STRING, _
                                          temp    AS STRING
                                  
                                    seconds = inSeconds
                                    minutes = inSeconds\60
                                    seconds = inSeconds MOD 60
                                    hours = minutes\60
                                    minutes = hours * 60 - minutes
                                    temp = SPACE$(2)
                                    RSET temp = FORMAT$(hours)
                                    tString = temp + ":"
                                    RSET temp = FORMAT$(minutes)
                                    tString = tString + temp + ":"
                                    RSET temp = FORMAT$(seconds)
                                    tString = tString + temp
                                    FUNCTION = tString
                                  
                                  END FUNCTION
                                  '###########################################
                                  FUNCTION DateToJulian(inDate AS STRING) AS DWORD
                                  ' Originally by Egbert Zijlema
                                  ' Returns Julian Date (DWORD) from "String" Date formatted as: MM/DD/YYYY.
                                    LOCAL year         AS DWORD, _
                                          month        AS DWORD, _
                                          day          AS DWORD, _
                                          Elapsed      AS DWORD, _
                                          InvalidLeaps AS DWORD, _
                                          ValidLeaps   AS DWORD, _
                                          dMonths      AS DWORD
                                  
                                    month = VAL(LEFT$(inDate,2))
                                    day   = VAL(MID$(inDate,4,2))
                                    year  = VAL(MID$(inDate,7,4))
                                  
                                    IF month < 3 THEN                      ' January or February?
                                      month = month + 12                   ' 13th or 14th month ....
                                      DECR year                            ' .... of prev. year
                                    END IF
                                  
                                    Elapsed = INT((year + 4712) * 365.25)  ' days in years elapsed
                                    InvalidLeaps = year \ 100              ' substract century leapdays
                                    ValidLeaps = year \ 400                ' re-add valid ones
                                    dMonths = INT(30.6 * (month - 1) + .2) ' days of months elapsed + adjustment
                                    FUNCTION = Elapsed - InvalidLeaps + ValidLeaps + dMonths + day
                                  
                                  END FUNCTION
                                  '###########################################
                                  FUNCTION JulToDate(BYVAL Jul AS DWORD) AS STRING
                                  ' Originally by Egbert Zijlema
                                  ' Returns "String" Date formatted as: MM/DD/YYYY from Julian Date (DWORD)
                                  
                                    LOCAL help  AS DWORD, _
                                          Year  AS DWORD,_
                                          Month AS DWORD,_
                                          Day   AS DWORD,_
                                          oText AS STRING * 10
                                  
                                    Jul   = Jul + 68569
                                    help  = 4 * Jul \ 146097
                                    Jul   = Jul - ((146097 * help + 3) \ 4)
                                    Year  = 4000 * (Jul + 1) \ 1461001
                                    Jul   = Jul - (1461 * Year \ 4) + 31
                                    Month = 80 * Jul \ 2447
                                    Day   = Jul - (2447 * Month \ 80)
                                    Month = Month + 2 - (12 * (Month \ 11))
                                    Year  = 100 * (help - 49) + Year + (Month \ 11)
                                  
                                    oText = RIGHT$(STR$(Month),2) + "/" _
                                            + RIGHT$(STR$(Day),2) + "/" _
                                            + RIGHT$(STR$(Year),4) + "/"
                                    REPLACE " " WITH "0" IN oText
                                    FUNCTION = oText
                                  END FUNCTION
                                  '###########################################
                                  regards,


                                  [This message has been edited by Ian Cairns (edited July 10, 2001).]
                                  :) IRC :)

                                  Comment


                                  • #18
                                    THANK YOU IAN!!!!

                                    That works beautifully...!

                                    Ariel


                                    ------------------

                                    Comment


                                    • #19
                                      You are welcome. This is some of what I have to do much of the time.

                                      regards,

                                      ------------------
                                      [email protected]
                                      :) IRC :)

                                      Comment

                                      Working...
                                      X