Upload client

[Martin Draper]

I am currently writing an web upload client and I have run into a problem and need some help. The basic client works with text files no problem. But when I try word files (.doc) I only get about 6 bytes of the file. This due to the fact that there is (1A character) End of file marker in the header of the file.

The problem is in STDIN LINE Temp, it does not handle binary files. It stops
reading at the End of file marker.

What I need is a way to read binary files from stdin that ignores the end of file marker.


This is the code

'------------------------------------------------------------------
' This the web based document manager version 1.0
' by M C Draper Advanced Business Concepts
'
' Changes 14/3/2009 Initial Version
'------------------------------------------------------------------

#COMPILE EXE
#DIM ALL

$INCLUDE "\pbcc40\file.inc"
$INCLUDE "\pbcc40\PBCGI.INC"
$INCLUDE "\pbcc40\CGIlib.inc"

SUB LoadPage(HtmlPage AS STRING)
'
' thie routine will check if the files exits, then open and read the file
' and output throught cgi to the client
'
DIM filehandle AS LONG
DIM Buffer AS STRING
'
IF FileExists(HtmlPage) = 0 THEN EXIT SUB
'
' Open the file
Filehandle = FREEFILE
OPEN Htmlpage FOR INPUT AS Filehandle
'
' read the contents of the file
WHILE NOT EOF(FileHandle)
LINE INPUT# Filehandle, Buffer
WriteCGI buffer
WEND

CLOSE FileHandle
END SUB

SUB ListFiles
LoadPage("menupage.html")
END SUB

SUB UploadFile
ON ERROR RESUME NEXT

LOCAL Temp AS STRING
LOCAL Buffer AS STRING
LOCAL Boundary AS STRING
LOCAL Content AS STRING
LOCAL Context AS STRING
LOCAL Filename AS STRING
LOCAL FileNameStart AS INTEGER
LOCAL File AS STRING

LOCAL Filehandle AS LONG

STDIN LINE Boundary
writecgi "Boundary = "+Boundary + "<br>"

STDIN LINE Content
writecgi "Content = "+Content + "<br>"

STDIN LINE Context
writecgi "Context = "+Context + "<br>"


Buffer = ""

WHILE INSTR(Temp,Boundary) = 0
STDIN LINE Temp
'
' don't add the boundaty to the file
IF INSTR(Temp,Boundary) = 0 THEN Buffer = Buffer + Temp + $CRLF
'writecgi temp
WEND

writecgi "File = " + Buffer + "<br>"
FileNameStart = INSTR(Content,"filename=") + 10

FileName = MID$(Content,FileNameStart,LEN(Content) - FileNameStart)
writecgi Filename
File = RIGHT$(Filename,LEN(filename) - INSTR(-1,Filename,"\"))
writecgi ">"+file+"<"
'
'
' Save the file to disk
' You cam savein the web directory as it is write protected. You havew to a separte directory
' for the uploaded files.
'
Filehandle = FREEFILE
OPEN "c:\" +File FOR OUTPUT AS Filehandle
PRINT# Filehandle, Buffer
CLOSE Filehandle
writecgi "error = " + STR$(ERR)

END SUB

FUNCTION PBMAIN () AS LONG

DIM UserLine AS STRING


IF COMMAND$ = "ListFiles" THEN ListFiles:EXIT FUNCTION
IF COMMAND$ = "UploadFile" THEN UploadFile:EXIT FUNCTION

END FUNCTION

[Mike Doty]

Is this 1 program?  
Would the cgi program read the file?
If so, then within the cgi program:
 
  hFile = FREEFILE
  DO
    ERRCLEAR
    OPEN sFileName FOR BINARY LOCK WRITE AS #hFile
    IF ERR THEN WriteCGI  "<LI>Error opening "  + STR$(ERR)
    INCR attempt
  LOOP UNTIL ERR = 0 OR attempt > 10  'don't write if somebody else is at same time.
  IF attempt > 10 THEN
     WriteCGI "<LI>Site too busy, please try later"
     EXIT FUNCTION
  END IF
  GET$ #hFile, LOF(hFile), sBuffer
  CLOSE #hFile
  'Modify sBuffer and write out new file
END FUNCTION

[Martin Draper]

Thanks Mike but that is not the problem.

The upload from the web client works just fine which is evidenced by the fact that a text file does not have a problem. The problem comes when I upload a .doc file which contains a EOF characters '1A'. The problem, I think, is the 'STDIN LINE' function, it stops when it reads the EOF character.

When uploading a binary file you have to ignore the characters and continue till you get to the physical end of the file. Just same way you read a binary file but from the stdin.

I have tried reading by ignoring stdineof and kept reading until I get the second boundary at the end of the file. That did not work and resulted in loop.

[Mike Doty]

STDIN is for reading text from the standard input device..

http://www.powerbasic.com/support/pbforums/showthread.php?t=24345

Plus, you do not need to read the file and then use writecgi with text files.
Just point browser to the file with STDOUT "LOCATION: " + URL + $CRLF

[Martin Draper]

I had a thought, when I looked up the STDIN LINE function there was also a function to get the handle for stdin which I read could be used to to treat the stdin as file stream using open handle.

Sp I changed the code to use this code to read

Filehandle = FREEFILE
OPEN HANDLE GETSTDIN FOR INPUT AS Filehandle LEN = 4096

WHILE NOT EOF(Filehandle)
LINE INPUT #Filehandle, temp
writecgi temp
Buffer = Buffer + Temp
WEND

Now this works fine with text files but does not work with the .doc files due to the EOF markers in the file. So I wrote a simple program to test reading a .doc file as a binary file and this works just fine.

Filehandle = FREEFILE
OPEN "C:\healthjob.doc" FOR BINARY AS Filehandle

WHILE NOT EOF(Filehandle)
GET$ #Filehandle,256, Temp
Buffer = Buffer + Temp
WEND
90 CLOSE Filehandle

So I modified the getstdin function above use the binary file mode the same way as the test program.

[CODE]
Filehandle = FREEFILE
OPEN HANDLE GETSTDIN FOR Binary AS Filehandle

WHILE NOT EOF(Filehandle)
GET$ #Filehandle,256, Temp
writecgi temp
Buffer = Buffer + Temp
WEND
[/QUOTE]

Now this should work but NOPE it just sits there. I must be missing something
any ideas PLEASE !

[Mike Doty]

.DOC files can not be treated as text files. They have fonts, etc.
Any binary character could be at any position.

If you want to display a .DOC file to a browser, point the browser
to the .doc file using the LOCATION statement.

Plus, the browser does not know what type of file is in memory. Reading 256 characters of a binary file and adding $CRLF will not work.
To display .DOC files requires that the client has software associated to the .DOC extension and you point the browser to the file.

[Martin Draper]

Mile you are missing the point, I am trying to upload a .doc file to a web based server to be stored which happens everyday with php, perl, vb and every other web language.

With the common gateway interface there are three ways data is sent to a web server GET, POST and Command line interface. The command line interface is out because you are limited to command line buffer size which 256characters (depending on the web server).

With GET method you are limited to the maxsize of the QueryString which is no where big enough to upload files. That only leaves the STDIN which all the webservers used to to transfer files. So by defination the STDIN has to be able handle any data so it has to be able to handle Binary.

GETSTDIN allows you to get the current handle for the standard input and use
the open handle to read the data from the stream. The open handle allows the stream to be opened as binary to read binary data which is what I am currently trying and it can't get to work.

I am not trying to display the a doc file, the writecgi is for diagnositic purposes to see what is happening. I an try to get file to server so it can be cataloged and stored as part of web document mangement system.

[Mike Doty]

Filehandle = FREEFILE
OPEN "C:\healthjob.doc" FOR BINARY AS Filehandle
 
WHILE NOT EOF(Filehandle)
   GET$ #Filehandle,256, Temp
    Buffer = Buffer + Temp
WEND
90 CLOSE Filehandle

A cgi program only has access to files on the server.
This is attempting to read a file on the client.

Maybe someone has some javascript to send files. http://javascript.internet.com/forms/upload-filter.html
Does this look like what you are trying to do?
The cgi side of this javascript to write to the server should do it. Can't work on this right now. Hope this helped.

[Mike Doty]

This could be used as a template to send to a cgi program:
Reposted here so you get an email message.

http://javascript.internet.com/forms/upload-filter.html

[Michael Mattias]

You can try reading STDIN as a regular disk file with something like..

  hSys = GETSTDIN 
  hFile = FREEFILE 
  OPEN HANDLE hSys FOR BINARY AS hFIle 
  nBYte = LOF (hFile) 
  Buff$   =  STRING$(nByte, $NUL) 
  GET$      hFile, BUff$
  CLOSE     hFile

If STDIN is complete (that is, the LOF() is the complete byte count), that will get you the file data as a string and you may do what you will with it.

I don't know how STDIN and STDOUT work well enough to know exactly how the "size" of STDIN is determined, but this has got to be a head start of sorts.

Another thought...

If that LOF() thing does not work out, you can get the data a byte at a time until you hit (logical) end-of-file

  hSys = GETSTDIN 
  hFile = FREEFILE 
  OPEN HANDLE hSys FOR BINARY AS hFIle 
  DO 
      GET    hFile, , BYteVar?
      IF ISFALSE ERR THEN            ' did not hit EOF 
            Buff  =  Buff & CHR$(ByteVar)
      ELSE
            EXIT DO 
      END IF
  LOOP 
  CLOSE     hFile

MCM

[Shawn Anderson]

this is what I use:

HTML

<html>
<body>

	<form enctype="multipart/form-data" 
		    method="post" 
		    name="uFile" 
		    action="http://www.mydomain.com/cgi/uploadDoc.exe">
		    
	      DOC file : <input type="FILE" name="docFile" id="docFile" size="40"><p>
	
				<input type="submit">
	</form>  
	
</body>	
</html>

and the PB source:

' upload a word document from a web page

#Include "win32api.inc"     
Declare Function webdataStdIn() As String         
Declare Sub WriteCGI (ByVal st As String)          

Function PbMain()    
    Local fNumber As Long     
    Local fName As String
    Local sParams As String    
    Local inFile As String  
    Local outImage As String  
    Local tempBuf As String  
    LOCAL filen AS STRING    
    Local errMsg As String       
    Local fileID As String   
    Local sp1,sp2 As Long       
  
  ' error trap  
    On Error GoTo errOut                 
    
  ' write the start of the returning html  
    writeCGI "<html><body>"    
    
  ' get file from stdin                
    sParams=webdataStdIn()                     
 
  ' make sure it is a word document  
    If InStr(sParams,"Content-Type: application/msword")=0 Then   
        errMsg="error: not a word document"
        GoTo errOut
    End If
    
  ' the actual data of the file is the remainder of the file after /msword + 4 more characters
    inFile=Trim$(Remain$(sParams,"Content-Type: application/msword"))   
    inFile=Right$(infile,Len(infile)-4)  
    
   ' get the actual file name       
    sp1=InStr(sParams,"fileID")+7
    sp2=InStr(sp1,sParams,"-")
    fileID=Trim$(Mid$(sParams,sp1,sp2-sp1))    
    Replace Chr$(0) With "" In fileID
    Replace Chr$(10) With "" In fileID
    Replace Chr$(13) With "" In fileID
   
   ' write the image to local disk
     fNumber = FreeFile      
     outImage="edocUploads\myDoc.doc"
     Open outImage For Binary As #fNumber
     Put$ #fNumber, inFile
     Close #fNumber           
     
   ' give user response
     writeCGI "done"   
     writeCGI "</body></html>"   
     
Exit Function
    errOut:
     writeCGI errMsg
     writeCGI "</body></html>"     
     
End Function

' use for binary files             
' I think this is by Dave N and Don D.
Function webdataStdIn() As String
   Dim hInput As Long
   Dim iToRead As Long
   Dim iRead As Long
   Dim iResult As Long
   Dim sBuffer As String
   Dim sOutBuffer As String       

   iToRead = Val(Environ$("CONTENT_LENGTH"))
   hInput = GetStdHandle(%STD_INPUT_HANDLE)
   If hInput Then
      Do
         If iToRead > 32000 Then
            sBuffer = Space$(32000)
         ElseIf iToRead = 0 Then
            Exit Do
         Else
            sBuffer = Space$(iToRead)
         End If

         iResult = ReadFile(hInput, ByVal StrPtr(sBuffer), _
                            Len(sBuffer), iRead, ByVal %Null)

         ' If there was an error, return nothing
         If iResult = 0 Then
            Exit Do

         ' We're done if iRead is 0
         ElseIf iRead < 1 Then
            Exit Do

         ' Otherwise, accumulate the buffer
         Else
            sOutBuffer = sOutBuffer + Left$(sBuffer, iRead)
            If Len(sOutBuffer) >= iToRead Then Exit Do

         End If
      Loop
   End If

   Function = sOutBuffer

End Function


'------------------------------------------------------------------------------       
' taken from pbcgi.inc
' WriteCGI writes an HTML string to the web server. It automatically takes care
' of the necessary header.       '
Sub WriteCGI (ByVal st As String) 
    Static header As Long      
    If IsFalse header Then
        STDOUT "Content-type: text/html"
        STDOUT
        header = -1
    End If  
    Replace "''" With $Dq In st 
    STDOUT st     
End Sub

'------------------------------------------------------------------------------
' CgiParam parses raw CGI data to return the parameter you specify. The target
' parameter name is not case-sensitive.
'
Function CgiParam (ByVal sParmList As String, ByVal sTarget As String) As String

    Local ix    As Long
    Local sParm As String

    sTarget = UCase$(sTarget) + "="

    For ix = 1 To ParseCount(sParmList, "&")
        sParm = Parse$(sParmList, "&", ix)
        If UCase$(Left$(sParm, Len(sTarget))) = sTarget Then
            Function = DecodeCGI(Mid$(sParm, Len(sTarget) + 1))
            Exit For
        End If
    Next

End Function

'------------------------------------------------------------------------------
' DecodeCGI decodes the special characters in a CGI string.
'
Function DecodeCGI (ByVal sInput As String) As String

    Local pbInput   As Byte Ptr
    Local pbOutput  As Byte Ptr
    Local ncbInput  As Long
    Local ncbOutput As Long
    Local ncHex     As Long

    ncbInput = Len(sInput)
    If ncbInput = 0 Then Exit Function

    pbInput = StrPtr(sInput)
    pbOutput = pbInput

    Do
        Select Case Const @pbInput

        Case 37   ' "&"
            Incr pbInput
            Decr ncbInput
            ncHex = Min(2, ncbInput)
            If ncHex Then
                @pbOutput = Val("&H" + Peek$(pbInput, ncHex))
                pbInput = pbInput + ncHex
                ncbInput = ncbInput - ncHex
            End If

        Case 43   ' "+"
            @pbOutput = 32   ' $SPC
            Incr pbInput
            Decr ncbInput

        Case Else
            @pbOutput = @pbInput
            Incr pbInput
            Decr ncbInput
        End Select

        Incr pbOutput
        Incr ncbOutput

    Loop While ncbInput

    Function = Left$(sInput, ncbOutput)

End Function

Macro cgiDecode = DecodeCGI

[Michael Mattias]

First of all, please edit your prior posts' code tags. You have a couple up there where you open a "[ code]" section and end it with an "end quote" "[ /quote]" tag.

I could not find some of the stuff you had already tried, which included the OPEN HANDLE of STDIN.

Now this code here:

Filehandle = FREEFILE
OPEN HANDLE GETSTDIN FOR Binary AS Filehandle 
WHILE NOT EOF(Filehandle)
   GET$ #Filehandle,256, Temp 
   writecgi temp
   Buffer = Buffer + Temp
WEND

.. if it just sits there, it's because EOF() is not returning TRUE. EOF() is funky when files opened FOR BINARY are involved: it does not return TRUE until you actually read PAST the EOF.

FWIW, the GET$() function may also be sensitive to 0x1A (EOF marker), as GET$ is a "text" function.

Try the byte-at-a-time code "as byte" I suggested above and testing for EOF after the GET and see what happens.


MCM

[Mike Doty]

Nice job, Shawn.
I was thinking about embedding some javascript, your method is better.

[Michael Mattias]

Or,mayb GET$() with a byte count of 256 is waiting...

PB/WIn 9.01 help:

Remarks
GET$ reads Count& characters from file number filenum&, and assigns them to sVar$. File filenum& must have been opened in binary mode. Characters are read starting at the current file pointer position, which can be set with the SEEK statement. When the file is first opened, the pointer is at the beginning of the file (unless the LEN clause is used in the corresponding OPEN statement, position 1 is used by default). After GET$, the file pointer position will have been advanced by Count& bytes.

Nowhere on this help page does it say what happens if there are less than Count& bytes available from current SEEK() to EOF.

I shall write to support to find out. Maybe it's waiting until 256 bytes are available.

MCM

[Mike Doty]

Michael,
That code is incorrect. The server would have to be reading a file from itself.

[Martin Draper]

Hi Michael,
I tried the lof and it returns 0 for opening the file "for Binary" or "For Input".

I tried your second idea and it does not read any data. The len of the
data being return is always zero. To make things simplier I tested the upload client with a simple text file ( The Source code this the client)

Filehandle = FREEFILE
OPEN HANDLE GETSTDIN FOR BINARY AS Filehandle
DO
GET$ #Filehandle,256, Temp
Buffer = Buffer + Temp
writecgi STR$(LEN(temp))
IF ISFALSE ERR THEN EXIT DO
LOOP

I trapped the error in this case and it returns error 52.

I changed the code to use the handle as input instead of binary and it works but it does not find the end of file

Filehandle = FREEFILE
OPEN HANDLE GETSTDIN FOR INPUT AS Filehandle

DO
'GET$ #Filehandle,256, Temp
LINE INPUT #Filehandle, temp
Buffer = Buffer + Temp
writecgi STR$(LEN(temp))
writecgi "error = "+STR$(ERR)
IF ISTRUE ERR THEN EXIT DO
writecgi temp +"<br>"
'IF INSTR(temp,Boundary) > 0 THEN EXIT LOOP
LOOP

[Michael Mattias]

You sure the OPEN ...

OPEN HANDLE GETSTDIN FOR BINARY AS Filehandle

.. is succeeding? No, you are not because you are not testing ERR.

Oops, need to test that myself to see if that bug was fixed (OPEN HANDLE was not returning ERR on failure). Don't see an entry in history.txt file.

[Michael Mattias]

Well, the GET$ thing was a thought but...

When you specify a count parameter in the GET$ statement that is greater
than the remaining number of bytes in the file, only the remaining bytes
are read and copied to the string parameter. I have logged a request
that this be added to the help file.

MCM

[Martin Draper]

Michael,
I changed the code to test err on opening the handle and it fails.

Filehandle = FREEFILE
OPEN HANDLE GETSTDIN FOR BINARY AS Filehandle
IF ISTRUE ERR THEN writecgi " Open Failed": EXIT SUB

It Looks like that bug has not been fixed in 4.03 of PBCC40.

[Michael Mattias]

If you got a non-zero ERR, that means it *is* fixed.

Now, why it won't open, that's a good question.

Hmm... trying to open STDIN in an output-capable mode (the default for FOR BINARY) can't be right.

It can't hurt to try opening it 'FOR BINARY ACCESS READ' and maybe LOCK SHARED, too.

MCM