Announcement

Collapse
No announcement yet.

"print #" causes a spurious error (or termination) every 4GiB. use "put$ #" instead.

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • "print #" causes a spurious error (or termination) every 4GiB. use "put$ #" instead.

    Every time "print #1, a$;" increases the file size past a multiple of 4GB, it causes a spurious Error 51, or some other error, or silent termination of the process.
    (I wonder whether this problem originates in PB's run-time library or the Windows API.)
    Surprisingly, "put$ #1, a$" does not cause the errors. (To use "put$ #" instead of "print #", the file must be "open"ed "for binary" rather than "for output" or "for append".)
    (I don't know why I wrote to the file using "print #" instead of "put$". I always used "get$" to read from the large file.)

    To narrow down what happens and when, I wrote a program that predicts each error, and reports each error that is predicted and/or happens. (Initially I thought I could predict and selectively ignore the spurious errors, as a work-around. But I can't ignore silent termination of the process.)

    I saw this bug with PBCC 6.04 2013-03-01 and then with PBCC 6.03 2011-12-28. (PowerBASIC might have quietly "taken back" PBCC 6.04, leaving 6.03 the new latest version.)
    (Older PBCC 2.21 2002-02-01 might not have had this bug. But PBCC 2.21 won't compile on 64-bit Windows.)
    I compiled and ran it only on Windows 7, 64-bit.

    Code:
    #dim all  ' (no defint, etc.)
    
    function PBMAIN() as long
    
    ?"filltest2.bas 20171030
    ?"  This program demonstrates a problem with ""print #"" writing to a file.
    ?"  Every time ""print #"" increases the file size past a multiple of 4GB, a spurious error occurs.
    ?"  The write size determines the error number.
    ?"  If the write size is a multiple of 256, the error is Error 51, ""Internal error"".
    ?"  If the write size modulo 256 is not zero, the error is that value, but sometimes the program terminates silently.
    ?"  With spurious Error 51 after write, the file is correctly written. Ignoring that Error 51 is safe.
    ?"  With other spurious Error numbers after write, the file is correctly written. Ignoring them is less safe, because they could indicate a real error.
    ?"  Ignoring only the exact predicted errors is less unsafe.
    ?"  However, some write sizes trigger silent termination of the process, which cannot be ignored.
    ?"  Using ""put$ #"" instead of ""print #"" solves this problem, so no other work-around is needed.
    ?
    
    dim blocksize as quad
    blocksize = 1048576  ' 1048576 == 1MiB  8388608 == 8MiB
    
    ' Results table:
    ' blocksize----    Outcome--------------
    '       0 +   1 -> Error   1
    '     256       -> Error  51
    '  524288       -> Error  51
    '  131072       -> Error  51
    '  131072 +   1 -> Error   1
    '  131072 +  15 -> Error  15
    '  131072 +  51 -> Error  51
    '  131072 +  63 -> Error  63
    '  524288       -> Error  51
    ' 1048576       -> Error  51
    ' 8388608 -   1 -> process silently dies
    ' 8388608       -> Error  51 = "Internal error"
    ' 8388608 +   1 -> Error   1
    ' 8388608 +   2 -> Error   2
    ' 8388608 +   3 -> Error   3
    ' 8388608 +  63 -> Error  63
    ' 8388608 + 127 -> Error 127
    ' 8388608 + 225 -> Error 225
    ' 8388608 + 235 -> Error 235
    ' 8388608 + 240 -> Error 240
    ' 8388608 + 241 -> process silently dies
    ' 8388608 + 242 -> process silently dies
    ' 8388608 + 243 -> process silently dies
    ' 8388608 + 245 -> process silently dies
    ' 8388608 + 254 -> process silently dies
    ' 8388608 + 255 -> process silently dies
    ' 8388608 + 256 -> Error  51
    ' 8388608 + 257 -> Error   1
    ' 8388608 + 512 -> Error  51
    ' Repeating writes of each given size gives the same results, whether writing from the first byte, or some extra bytes were written first.
    ' The size of the write that reaches to or past 4GiB decides the spurious error number, not the ending byte of the write.
    
    dim sPattern as string
    sPattern = space$(blocksize)
    
    if isfile("T:\temp") then
      print "Killing the old file. Wait...";
      kill "T:\temp"
      locate ,1
      print "Old file has been deleted.   "
    end if
    
    open "T:\temp" for output access write lock shared as #1 base = 0  '' Doing it this way does causes the error.
    '' open "T:\temp" for binary access write lock shared as #1 base = 0  '' Doing it this way does not cause the error.
    
    ' Optional: Write some bytes before the repeated writes.
    ' (This confirmed that crossing each exact n x (4 GiB) boundary triggers the error.)
    ''print #1, space$(13);
    
    dim FilePos as quad  ' (Int64) position within file
    FilePos = seek(#1)  ' Collect the file position.
    
    do
      ' Optional: Taper down to 1-byte writes before crossing the 4GiB boundary. Even a 1-byte write errs (with Error 1!).
      ' (This confirmed that the size of the write decides the error number, not the ending byte of the write.)
      ''if FilePos + blocksize >= 4294967296 and (blocksize > 256) then
      ''  blocksize /= 2
      ''  sPattern = space$(blocksize)
      ''end if
    
      dim FilePosNext as quad
      FilePosNext = FilePos + blocksize  ' Calculate the expected file position after the write.
    
      ' Display the file position before and after the write. (12 hex digits covers 0 to 256TiB - 1.)
      locate ,1
      print hex$(FilePos, 12) " - " hex$(FilePosNext, 12);
    
      ' PREDICT the error. (Every time file size reaches the next multiple of 4GiB due to the write, a spurious error happens.)
      dim predictederror as integer
      if (FilePos and &hFFFFFFFF00000000) = (FilePosNext and &hFFFFFFFF00000000) then  '' IF the write will NOT cross a 4GiB boundary,
        predictederror = 0  ' predict no error.
      else  '' (IF it WILL cross a 4GiB boundary,)
        predictederror = blocksize mod 256  ' predict the error: length of the write mod 256.
        if predictederror = 0 then  ' IF predicted error is 0,
          predictederror = 51  ' change the prediction to Error 51.
        end if
        if predictederror <= 240 then  ' IF predicted error is in 0:240,
          print "  predictederror =" predictederror;  ' display the predicted error.
        else
          print "  predictederror =" predictederror " (but program will silently terminate)";  ' display the predicted error AND predict crash.
        end if
      end if
    
      ' Write the block to the file:
      print #1, sPattern;  '' Doing it this way causes the spurious error (or sometimes silent termination).
      '' put$ #1, sPattern  '' Doing it this way does not cause the spurious error.
    
      FilePos = seek(#1)  ' Collect the file position.
    
      ' Verify that the file position advanced as expected. (This might stop the program if the drive fills up or other error occurs.)
      if FilePos <> FilePosNext then
        print "  position error" FilePos "<>" FilePosNext
        exit loop
      end if
    
      if predictederror <> 0 or err <> 0 then  ' IF an error was predicted OR has occurred (or both),
        print "  error =" err  ' Display the actual error (and go to new line so that it stays on display).
      end if
    
      if predictederror = 0 and err <> 0 then  ' IF no error was predicted, but an error occurred anyway,
        print "Write ended by Error" err ": """error$(err)"""."  ' (treat it as a real error.) display the error number and description.
        exit loop  ' stop writing.
      end if
    
      errclear  ' Clear any error.  ((Real errors don't stop it. The program might never terminate.))
    
      if instat then print:print "Write ended by keystroke.":exit loop
    loop
    print
    
    print "Hit a key to finish.";
    input flush:waitkey$
    
    end function

  • #2
    Well done, nice clear report.

    Error 51 (Internal Error) is a catch-all code that AFAIK always relates to a device error of some kind. Any other error codes that you receive may not be trustworthy.

    Sure looks like a bug to me. It's unlikely, but did you rule out interference (like delayed disk-writes) from your antivirus software?

    And Yes, after PowerBASIC, Inc. was sold to Drake Software they stepped back to the .03 versions. I can't seem to find the announcement at the moment.
    "Not my circus, not my monkeys."

    Comment


    • #3
      (Older PBCC 2.21 2002-02-01 might not have had this bug. But PBCC 2.21 won't compile on 64-bit Windows.)
      I compiled and ran it only on Windows 7, 64-bit.
      Did you mean "32-bit" the second time you said "64-bit"?

      The old compilers were themselves 16-bit programs that created 32-bit executables. At PBWin 8/PBCC 4 or PBWin 9/PBCC 5 (I forget which) the compilers were made 32-bit. 64-bit Windows does not support 16-bit programs. So, the old compilers won't run on 64-bit Windows, the new ones will. The compiled code code from either should run on 64-bit Windows.

      Cheers,
      Dale

      Comment


      • #4
        Eric,

        The statement about .03 versions was in the Announcements section. That section has since been cleaned out.

        Dan,

        About the PRINT # error. Did it occur running on 32-bit Windows and 64-bit Windows, or just one?

        Cheers,
        Dale

        Comment


        • #5
          Why open the file "output, lock shared?

          If you OPEN it FOR OUTPUT no other process can even see it. (As written there is no LOCKing so SHAREing will fail miserably anyway!)

          Maybe it will work better if you don't try to force it.

          Disclaimer: N/A if this is a totally fabricated test program and in Real Life an existing file is opened FOR APPEND and is in fact SHARED and proper LOCKing is included. But then the test program should have replicated at least the OPEN so I doubt this is the case.

          MCM
          Michael Mattias
          Tal Systems Inc.
          Racine WI USA
          mmattias@talsystems.com
          http://www.talsystems.com

          Comment


          • #6
            I tried other block sizes and never get an error.
            Are you defining 4-gigabyte correctly in hex? Click image for larger version  Name:	filltest2.png Views:	1 Size:	4.6 KB ID:	766511
            Code:
            #DIM ALL
            #BREAK ON
            $File = "c:\temp\junk.txt"
            FUNCTION PBMAIN() AS LONG
             LOCAL x,StartByte,EndByte,BlockSize,GigaByte4 AS QUAD
             LOCAL sPattern AS STRING
             GigaByte4 = &h100000000  '4 of them
            
             REM if (FilePos and &hFFFFFFFF00000000) = (FilePosNext and &hFFFFFFFF00000000)
             REM Not sure what this is all about?
            
             blocksize = 1
             sPattern = SPACE$(blocksize)
             KILL $File:ERRCLEAR
             OPEN $File FOR OUTPUT ACCESS WRITE LOCK SHARED AS #1  '' Doing it this way does causes the error.
             IF ERR THEN ? "open error";ERR:WAITKEY$:END
             StartByte = GigaByte4
             EndByte   = GigaByte4*3
             SEEK #1, StartByte
             IF ERR THEN ? "Seek error";ERR:GOTO EndProgram
             FOR x = StartByte TO EndByte STEP GigaByte4
              SEEK #1,x
              PRINT #1, sPattern;
              ? USING$("Bytes #,",LOF(1))
              IF ERR THEN ? "PRINT # error";ERR:GOTO EndProgram
             NEXT
            EndProgram:
             CLOSE #1
             ? "Done, please press a key";WAITKEY$
            END FUNCTION
            Last edited by Mike Doty; 5 Nov 2017, 11:58 AM.

            Comment


            • #7
              The step back to distribution of .03 for PB/WIN 10 and PB/CC 6 occurred prior to our acquiring the products from PowerBASIC, Inc.

              I have been testing with this application myself - and can consistently make it fail every time with an error 51. I won't say this for certain because I'm not quite deep enough into the compiler code yet to troubleshoot well, however, sequential file writes I believe are a buffered write, and as this is still a 32 bit compiler - this much data on a buffered write is pushing the boundaries of those buffers. Closing and re-opening the file just before I hit that mark allows me to write beyond that mark without an error:

              Code:
              '---------------------------------------------------------------------------------------------------------------------------
              #BREAK ON
              '---------------------------------------------------------------------------------------------------------------------------
              #COMPILE EXE
              #DIM ALL
              '---------------------------------------------------------------------------------------------------------------------------
              FUNCTION PBMAIN () AS LONG
              
                  LOCAL sDataBlock    AS STRING
                  LOCAL x             AS LONG
              
                  ON ERROR GOTO ErrorHandler
              
                  sDataBlock = SPACE$(1048576)
                  PRINT
                  PRINT
              
                  OPEN "C:\ADAMTEMP_4GB.TXT" FOR OUTPUT AS #1
                  FOR x = 1 TO 4095
                      LOCATE , 1
                      PRINT FORMAT$(x);
                      PRINT #1, sDataBlock;
                  NEXT
                  CLOSE #1
              
                  PRINT
                  PRINT
                  PRINT
              
                  OPEN "C:\ADAMTEMP_4GB.TXT" FOR APPEND AS #1
                  FOR x = 1 TO 10
                      LOCATE , 1
                      PRINT FORMAT$(x);
                      PRINT #1, sDataBlock;
                  NEXT
                  CLOSE #1
              
              ExitFunc:
              
                  PRINT
                  PRINT
                  PRINT "Done."
                  WAITKEY$
              
                  EXIT FUNCTION
              
              ErrorHandler:
              
                  CLOSE
                  PRINT ERR
                  PRINT ERROR$
                  GOTO ExitFunc
              
              END FUNCTION
              '---------------------------------------------------------------------------------------------------------------------------
              Adam Drake
              Drake Software

              Comment


              • #8
                Does FLUSH just before the mark (instead of CLOSE and re-OPEN) help?

                I think I did some test code on this too. I'll see if I can find it and try FLUSH as well.

                Cheers,
                Dale

                Comment


                • #9
                  I tried using FLUSH to see what it would do, and it still trips the Error 51. I tried creating an API test to push a file beyond the 4GB mark, and it does not trip an error - looks like I've got work to do:

                  Code:
                  '-----------------------------------------------------------------------------------------------------------------------------
                  #COMPILE EXE
                  #DIM ALL
                  '-----------------------------------------------------------------------------------------------------------------------------
                  #INCLUDE "WIN32API.INC"
                  '-----------------------------------------------------------------------------------------------------------------------------
                  %BLOCKSIZE = 1048576
                  '-----------------------------------------------------------------------------------------------------------------------------
                  FUNCTION PBMAIN () AS LONG
                  
                      LOCAL lpFileName    AS ASCIIZ * %MAX_PATH
                      LOCAL fileHandle    AS DWORD
                      LOCAL errFlag       AS LONG
                      LOCAL x             AS LONG
                      LOCAL sDataBuffer   AS STRING
                      LOCAL BytesWritten  AS DWORD
                      LOCAL LastError     AS LONG
                  
                      lpFileName = "C:\ADAMTEMP_4GB.TXT"
                  
                      fileHandle = CreateFile(lpFileName, %GENERIC_WRITE, 0, BYVAL %NULL, %CREATE_ALWAYS, %FILE_ATTRIBUTE_NORMAL, %NULL)
                      IF fileHandle = %INVALID_HANDLE_VALUE THEN
                          STDOUT "Failed to open file."
                          EXIT FUNCTION
                      END IF
                  
                      sDataBuffer = SPACE$(%BLOCKSIZE)
                  
                      FOR x = 1 TO 4100
                          LOCATE , 1
                          PRINT FORMAT$(x);
                          errFlag = WriteFile(fileHandle, STRPTR(sDataBuffer), %BLOCKSIZE, BytesWritten, BYVAL %NULL)
                          IF errFlag = 0 THEN
                              LastError = GetLastError()
                              PRINT
                              PRINT
                              PRINT "Error: " + FORMAT$(LastError)
                              EXIT FOR
                          END IF
                      NEXT
                  
                      CloseHandle fileHandle
                  
                      PRINT
                      PRINT
                      PRINT "Done."
                      WAITKEY$
                  
                  END FUNCTION
                  '-----------------------------------------------------------------------------------------------------------------------------
                  Adam Drake
                  Drake Software

                  Comment


                  • #10
                    Originally posted by Adam J. Drake View Post
                    The step back to distribution of .03 for PB/WIN 10 and PB/CC 6 occurred prior to our acquiring the products from PowerBASIC, Inc.
                    In case you are not aware of it; I would like to draw your attention to a flaw in .03
                    which was corrected in .04.

                    The code below demonstrate the issue. At least did it at the time I sent it to support; many years ago.

                    ---
                    Code:
                    'Perhaps I have missed something concerning #OPTION ANSIAPI, but it seems
                    'like there is some trouble with
                    'this compiler directive and some error codes, as shown below.
                    'Various error codes are triggered, but not error 76,
                    'as expected.
                    
                    #Compile Exe
                    #Dim All
                    #Debug Error On
                    $Debug Display On
                    #Option AnsiApi                       ' This line triggers wrong error codes as shown below
                    
                    '-------------------------------------------------------------------------------------------
                    Function PBMain () As Long
                    
                      Local hFile,hWin As Long
                      Local s As String
                    
                     ' s = "C:\NotAnExistingFolder\Test.txt" ' Not an existing folder.
                                                            ' Returns error code 2 with no error description provided.
                                                            ' Notice that long folder name is used.Compare with short folder name below.
                    
                      s = "C:\NoFolder\Test.txt"           ' A not existing folder. On my system it returns error code 116 on first run.
                                                           ' Then various error codes is returned on subsequent(s) run(s).
                                                           ' Notice the use of short folder name. Error code is different now compared to
                                                           ' the use of long folder name as stated above.
                                                           ' No error description is provided.
                      Try
                        Txt.Window("Option Ansi issue", 0, 0) To hWin
                        hFile = FreeFile
                        Open s For Output As #hFile
                      Catch
                        Txt.Print Error$(Err) + Str$(Err)
                      Finally
                    
                      End Try
                    
                      Close #hFile
                      Txt.Print "Press a key to end the program..."
                      Txt.WaitKey$
                    
                    End Function

                    Comment

                    Working...
                    X