Tweet
include file for use with pb/win or pb/cc
------------------
michael mattias
tal systems inc.
racine wi usa
mailto:[email protected][email protected]</a>
www.talsystems.com
Code:
' file: mmfli.inc ' function to use a memory-mapped file in lieu of line input ' author : michael mattias tal systems inc racine wi ' placed in public domain by author. ' date : may 31 2004 ' compiler: pb/windows v 7.02 (full test); pb/cc 3.03 (for clean compile only) ' winapi header file win32api.inc: may 9 2002 ' ------------------------------------------------------- ' this program demonstrates the use of the function "mmf_li-cb" ' quite simply what this function does is parse any file to find records delimted by cr, lf or ' any combination of cr and lf, and sends the resulting records to a user-defined callback function ' along with an action code and a user parameter value. ' this function can be used for... ' a. parsing large files without the need to load the entire file into a string. this produces a dramatic ' reduction in resource requirements. (see: http://www.powerbasic.com/support/pb...ad.php?t=23526 ' b. handling unix-delimited files (x'0a') identically to pc-delimited files (x'0d0a') without ' knowing or caring which format the file is in ' c avoiding two bugs in the powerbasic line input # function: ' 1. if file is a unix-delimited file, if the file is too big line input fails to return ' 2. line input terminates a record on cr, not crlf as documented ' ' note: as written, this software ignores blank lines. the pb line input function will, by design, return a ' null string if found in the file. ' '---------------------------------------------------------------------- ' user-called function: declare function mmf_li_cb (szfile as asciiz, byval cbaddr as dword, byval dwuser as dword) as long ' -------------------------------------------------------------------------------------------- ' parameters ' szfile [in] = file to be processed ' cbaddr [in] = address of callback function ' dwuser [in] = user-defined value passed to callback function on each call ' returns : 0 = success, enumeration completed ' other = enumeration never completed, either because of error or user request ' input file: records are delimited by a cr (x0d) or lf (x0a); any mutliple occurrences of ' either delimiter is treated as a single delimiter. ' ' callback function prototype and parameters ' ---------------------------------------------------- declare function mmf_li_cbfunction (byval statuscode as long, byval dwoffset as long, sdata as string, byval dwuser as dword) as dword ' status codes and dwoffset: ' when status code is dwoffset is sdata is ' ------------------- ----------- -------------------- ' %mmfli_error an error code n/a ' %mmfli_filesize size of file n/a ' %mmfli_data offset from zero of the cr or "the data" ' lf which terminated this record ' i.e., how much of the file has been processed ' %mmfli_eof total number of chars processed n/a ' the callback is called ' - when the file is successfully mapped, called with %mmfli_filesize ' - for each line of data, called with %mmfli_data. sdata will never be null ' - at end of file, called with mmfli_eof ' - on error, called with mmfli_error. return value is ignored in this case; enumeration stops. ' ' the callback function should return true (nonzero) to continue enumeration or ' false (0) to terminate the enumeration. ' ------------------------------------------------------------------------------------------------ #if not %def(%winapi) #include "win32api.inc" #endif #if not %def (%invalid_handle_value_long) %invalid_handle_value_long = -1& #endif ' ===================================================================== ' file mapping functions used by the main user function ' these file mapping functions are available as a 'standalone' #include file at: ' http://www.powerbasic.com/support/pb...ad.php?t=24225 ' memory map a file for input only ' returns: true = file was mapped and return value is the base pointer for the mapped object. ' false=error (e.g.,file not found, memory problems) ' parameters: ' szfilein [in] file to be mapped ' hsys [out] control value needed to unmap the file ' hfilemapping [out] control value needed to unmap the file ' cbmapping [out] size (number of bytes) of mapped file function mapthisfile (szfilein as asciiz, hsys as long, hfilemapping as long, cbmapping as long) as dword ' note we need the values of hsys, hfilemapping and pifdata to unmap the file local w32 as win32_find_data, hsearch as long local pifdata as dword ' check that we can find the file, and get its size if we can hsearch = findfirstfile (szfilein, w32) if hsearch <> %invalid_handle_value_long then cbmapping = w32.nfilesizelow closehandle hsearch else function = 0 exit function end if ' open the file for reading and get system handle. we could open with pb open and then open handle but ' that would just complicate this hsys = createfile (szfilein, %generic_read, %file_share_read, byval %null, %open_existing, %file_attribute_normal, byval %null) if istrue hsys then hfilemapping = createfilemapping (hsys, byval %null, %page_readonly, byval %null, byval %null, byval %null) if istrue hfilemapping then ' map the file to our memory space and get pointer to data... pifdata = mapviewoffile (hfilemapping, %file_map_read, byval %null, byval %null, byval %null) function = pifdata else function = 0 closehandle hsys exit function end if else function = 0 exit function end if end function function unmapmappedfile (pifdata as dword, hsys as long, hfilemapping as long) as long unmapviewoffile pifdata closehandle hfilemapping closehandle hsys end function ' ============================================= ' memory-mapped files line input functions ' and equates '============================================== ' callback status codes: %mmfli_filesize = 1& %mmfli_data = 2& %mmfli_eof = 3& %mmfli_error = 10& 'parsing constants %mmfli_cr = &h0d? %mmfli_lf = &h0a? ' error codes %mmfli_cantmapfile = 101& ' --------------------------------------------' ' user-called function ' --------------------------------------------' function mmf_li_cb (szfile as asciiz, byval cbaddr as dword, byval dwuser as dword) as long local pifdata as dword, hsys as long, hfilemapping as long, cbmapping as long local psrc as byte ptr, ncharthisrecord as long, nchardone as long local cbstatus as long, s as string, dwoffset as dword, cbret as long local inrecord as long, dwrecstart as long ' map the file to memory pifdata = mapthisfile (szfile, hsys, hfilemapping, cbmapping) ' --------------------------------------------------------------------------------------- ' call the callback function, either send filesize if mapping successful or error if not ' --------------------------------------------------------------------------------------- if istrue pifdata then cbstatus = %mmfli_filesize dwoffset = cbmapping else cbstatus = %mmfli_error dwoffset = %mmfli_cantmapfile end if call dword cbaddr using mmf_li_cbfunction (cbstatus, dwoffset, s, dwuser) to cbret ' if we had an error mapping the file, we exit if isfalse pifdata then function = %true ' set non-success return value exit function end if ' did user want to terminate the enumeration after seeing the filesize? ' if so, free the resources we allocated and exit with 'abort' return value if isfalse cbret then unmapmappedfile pifdata, hsys, hfilemapping function = %true ' set abort code exit function end if ' if we get here, we have mapped the file and the user did not ask for the enumeration to stop. psrc = pifdata ' point to start of filedata dwoffset = 0 ' get to zero-based value ncharthisrecord = 0 ' well, not yet anyway nchardone = 0 inrecord = %false cbret = %true ' do while nchardone < cbmapping ' for each character of the file select case as const @psrc case %mmfli_cr, %mmfli_lf ' terminates a record if we are in one if istrue inrecord then ' either cr or lf ends a record, and any number of consecutive cr or lf counts as one (1) delimiter ' that is, a null string is never sent to the callback function cbstatus = %mmfli_data s = peek$(dwrecstart, ncharthisrecord) call dword cbaddr using mmf_li_cbfunction (cbstatus, dwoffset, s, dwuser) to cbret ' does user want to terminate? if isfalse cbret then exit do end if inrecord = %false 'else ' we are not currently building a record, so this is just another occurrence ' of the delimiter, which we treat as a single record delimiter ' this is what would have to be changed a bit to send null strings when they occur end if case else ' a non-delimiter character if inrecord then ' incr ncharthisrecord ' keep track of how many characters we will have to move later else ' start a new record inrecord = %true dwrecstart = psrc ncharthisrecord = 1 ' because this character is included end if end select ' and increment our offset and other control values incr psrc incr dwoffset incr nchardone loop ' -------------------------------------------------------------- ' when we get here, we either have no more characters to process ' or the user asked for enumeration to end ' -------------------------------------------------------------- if istrue cbret then ' we were not forced out of loop by user request ' if we were building a record but file did not end with a cr or lf, send a data callback if inrecord then cbstatus = %mmfli_data s = peek$(dwrecstart, ncharthisrecord) call dword cbaddr using mmf_li_cbfunction (cbstatus, dwoffset, s, dwuser) to cbret end if ' send the end of file message: cbstatus = %mmfli_eof s = " call dword cbaddr using mmf_li_cbfunction (cbstatus, dwoffset, s, dwuser) to cbret ' we just ignore the return value here, since we won't be sending any more messages anyway end if ' and, whenever we get this far, we've mapped the file so we must clean up unmapmappedfile pifdata, hsys, hfilemapping ' and set function return value; 0 if enumeration completed, nonzero otherwise function = isfalse (cbret) ' if cbret is true function returns zero (success) end function ' ==================================================================================================== ' simple callback function provided for reference. dwuser (user supplied value) is a long passed by ' reference and simply counts records. pass to main user function as "byval varptr(nrec)" ' or the equivalent. ' this function writes each record to a pc-delimited (crlf) text file. this output may then be ' used safely by the powerbasic "line input #" functions. ' ==================================================================================================== function mmfli_prototype_callback_function (byval statuscode as long, byval dwoffset as long, sdata as string, nrec as long) as dword static hfile as long select case as const statuscode case %mmfli_filesize hfile = freefile open "mmfli_pc_file.txt" for output as hfile function = %true ' continue enumeration case %mmfli_data print #hfile, sdata incr nrec function = %true case %mmfli_eof close hfile hfile = 0 ' reset static function = %true case %mmfli_error function = 0 ' false to terminate true to continue enumeration ' not really necessary to process this message, as the enumeration function ' itself will exit immediately on error end select end function ' *** end of file mmfli.inc
michael mattias
tal systems inc.
racine wi usa
mailto:[email protected][email protected]</a>
www.talsystems.com
Comment