Announcement

Collapse
No announcement yet.

Rserver2 to get a file using TCP

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Rserver2 to get a file using TCP

    Code:
    Anyone know how to use RSERVER2 to rsGet a large file?
    I'm only receiving a small portion of the file back.
    rsPut works great for uploading files to the server.
    RServer2 is by Don Dickinson.
     
    ReadFileFromServer:
       iRet = rsLof(hSocket, hFile)
       REM msgbox "Bytes in remote file " + FORMAT$(iRet,"#,")
       '- Read data from remote file
       rsSeek hSocket, hFile, 1
     
       ERRCLEAR
       sReturn = ""
       sBuffer = ""
       iRet = rsGet(hSocket, hFile, iRet, sBuffer)
       sReturn = sBuffer                              'place first buffer into sReturn
     
       IF iRet = 0 THEN                               'attempt receiving rest of data
          result = tcpSafeReceive(hSocket, BytesToSend - LEN(sReturn), sBuffer)
          sReturn = sReturn + sBuffer
       END IF
       MSGBOX "Bytes received back" + STR$(LEN(sReturn))
       rsClose hSocket, hFile
       rsDisconnect hSocket
    END FUNCTION

  • #2
    [edit]

    Mike, After posting the following, I saw some other posts on the board that indicate you probably know this stuff. However, I also found an example of the TcpSafeReceive function and it does not seem to address the issue of large file transfers. It appears that it too would result in a crashed system if the file to be received was very large.

    [/edit]

    Hi Mike,

    I'm not familiar with Rserver2, but I think you're looking at a design issue with your code.

    When you need to pull down a file from a remote server, it may be the case that the file is very large. You shouldn't attempt to buffer the entire file into memory because it may max out the memory on your system and force it to start using swap space. This is very inefficient and can bring your system to crawl or even crash it. Imagine trying to get a 10, 20 or 100 GB file.

    The approach you need, and it looks like the RServer2 library provides for it, is to bring the file down in manageable "chunks", which you write out to disk as you receive them, building the file on the fly.

    Again, I don't know RServer2, but from the little code you posted, it seems like all the functions you need are there. It's an iterative process that I imagine would go something like this
    1. Use a manageable buffer size, maybe 128K
    2. Set a total bytes received counter to 0
    3. Get remote file size
    4. Seek to beginning of file
    5. If remaining bytes to receive (file size - total bytes received) are less than buffer size, then set buffer size = remaining bytes to receive
    6. Get buffer
    7. Write buffer to disk
    8. Add buffer size to total bytes received counter. If total bytes received = file size, you are done.
    9. Seek to next position on remote file. I don't know how this lib works, so this may be automatic and the file pointer may already be advanced to the next position. If not, the seek point should be total bytes received + 1.
    10. Go back to step 5


    I hope this helps, but I may be way off the mark with regard to this library, but that's generally the approach you should likely take for efficient remote file transfer.

    --
    Kevin Powick
    Last edited by Kevin Powick; 1 Sep 2009, 10:01 AM.
    --
    Kevin Powick

    Comment


    • #3
      Maximum reliable buffer size of rsGet only 4000 bytes

      Code:
      Thank you, Kevin!
      rsGet works great with buffer sizes up to about 4,000.
      rsPut has no buffer size limit so it is many times faster.
       
      DO
        BytesLeft = FileSize - BytesReceived                   'bytes left to be read
        IF BytesLeft < BufferSize THEN BufferSize = BytesLeft  'don't need full buffer at end
        iRet = rsGet(hSocket, hFile, BufferSize, sBuffer)      'get data
        IF iRet <> 0 THEN ErrorNumber = 99:EXIT DO             'exit if error
        PUT #hNewFile,, sBuffer                                'write buffer
        BytesReceived = BytesReceived + LEN(sBuffer)           'number of bytes received
      LOOP UNTIL BytesReceived = FileSize
      Last edited by Mike Doty; 2 Sep 2009, 08:34 AM.

      Comment

      Working...
      X