Rserver2 to get a file using TCP

[Mike Doty]

Anyone know how to use RSERVER2 to rsGet a large file?
I'm only receiving a small portion of the file back.
rsPut works great for uploading files to the server.
RServer2 is by Don Dickinson.
 
ReadFileFromServer:
   iRet = rsLof(hSocket, hFile)
   REM msgbox "Bytes in remote file " + FORMAT$(iRet,"#,")
   '- Read data from remote file
   rsSeek hSocket, hFile, 1
 
   ERRCLEAR
   sReturn = ""
   sBuffer = ""
   iRet = rsGet(hSocket, hFile, iRet, sBuffer)
   sReturn = sBuffer                              'place first buffer into sReturn
 
   IF iRet = 0 THEN                               'attempt receiving rest of data
      result = tcpSafeReceive(hSocket, BytesToSend - LEN(sReturn), sBuffer)
      sReturn = sReturn + sBuffer
   END IF
   MSGBOX "Bytes received back" + STR$(LEN(sReturn))
   rsClose hSocket, hFile
   rsDisconnect hSocket
END FUNCTION

[Kevin Powick]

[edit]

Mike, After posting the following, I saw some other posts on the board that indicate you probably know this stuff. However, I also found an example of the TcpSafeReceive function and it does not seem to address the issue of large file transfers. It appears that it too would result in a crashed system if the file to be received was very large.

[/edit]

Hi Mike,

I'm not familiar with Rserver2, but I think you're looking at a design issue with your code.

When you need to pull down a file from a remote server, it may be the case that the file is very large. You shouldn't attempt to buffer the entire file into memory because it may max out the memory on your system and force it to start using swap space. This is very inefficient and can bring your system to crawl or even crash it. Imagine trying to get a 10, 20 or 100 GB file.

The approach you need, and it looks like the RServer2 library provides for it, is to bring the file down in manageable "chunks", which you write out to disk as you receive them, building the file on the fly.

Again, I don't know RServer2, but from the little code you posted, it seems like all the functions you need are there. It's an iterative process that I imagine would go something like this

1. Use a manageable buffer size, maybe 128K
   
2. Set a total bytes received counter to 0
   
3. Get remote file size
   
4. Seek to beginning of file
   
5. If remaining bytes to receive (file size - total bytes received) are less than buffer size, then set buffer size = remaining bytes to receive
   
6. Get buffer
   
7. Write buffer to disk
   
8. Add buffer size to total bytes received counter. If total bytes received = file size, you are done.
   
9. Seek to next position on remote file. I don't know how this lib works, so this may be automatic and the file pointer may already be advanced to the next position. If not, the seek point should be total bytes received + 1.
   
10. Go back to step 5

I hope this helps, but I may be way off the mark with regard to this library, but that's generally the approach you should likely take for efficient remote file transfer.

--
Kevin Powick

[Mike Doty]

Maximum reliable buffer size of rsGet only 4000 bytes

Thank you, Kevin!
rsGet works great with buffer sizes up to about 4,000.
rsPut has no buffer size limit so it is many times faster.
 
DO
  BytesLeft = FileSize - BytesReceived                   'bytes left to be read
  IF BytesLeft < BufferSize THEN BufferSize = BytesLeft  'don't need full buffer at end
  iRet = rsGet(hSocket, hFile, BufferSize, sBuffer)      'get data
  IF iRet <> 0 THEN ErrorNumber = 99:EXIT DO             'exit if error
  PUT #hNewFile,, sBuffer                                'write buffer
  BytesReceived = BytesReceived + LEN(sBuffer)           'number of bytes received
LOOP UNTIL BytesReceived = FileSize