WinInet FTP problem, help wanted

[Michael Mattias]

Not directly on point, but let me ask another question...

When downloading (or uploading I guess, I had not really thought about that yet), I am currently using BINARY mode.

However, some files are "text" files... and some (a lot of?) servers are Unix-based.

If you use ASCII mode from Unix to PC, LF are converted to CRLF, and CRLF are converted to LF when going the other way; at least that's how I understand the difference between ASCII mode and BINARY mode.

So this begs the question: is their any "rule" or other method one may use to determine that any particular file is a "text" file and these conversions should be made? Or do I need to put the onus on the user to tell me?

I'd sure hate to convert LF to CRLF if the file being downloaded is NOT a text file, you know?

At this point I cannot see I can do anything EXCEPT force the user to decide. I allow the user to set up multiple masks for upload/download from any particular server... the screen looks kind of like...

LOCAL FILES      UPLOAD AS                        OVERWRITE?   DELETE AFTER?

D:\folder\*.*    outbound/[FILENAME].[FILEEXT]     YES          NO 
C:\folder\*.edi  outbound/DATA.[CCYYMMDD].[HHMMSS] NO           YES
E:\*.dat         outbound/DATA_[SERIALNO].DAT      YES          YES

Do I need to add another option column for "CRLF=>LF?" i.e uploading to a 'nix application expecting "text;" which, if true, would make the upload mode ASCII rather than BINARY?

"In context" I am not nuts about asking the user to make another decision; then again, these setups are often done by outside consultants (eg moi) so it might not really be all that confusing.


Comments?

Thanks,
MCM

[Paul Purvis]

without historical information, what can you really do, not much.
if you could, i would do what is needed on your side(downloading) and let the other end worry about whether to convert the file or not when it comes to CRLFs.

i would only download in binary mode.
you might want to consider saving either all files or files transferred within a certain date going back some time, i might suggest all files from 3 months to date that would give you a 60 to 90 day history.
paul

on a second thought go another month or forever with directories separated by month, yyyymm?????? then let the user delete files only as needed, maybe once a year, leaving the current year or last 13 months. naturally the size of files will make the factor.

i wonder if you transfer the file by text mode that it will also remove any end of file makers chr$(26).
i always did like a chr$(26) character on the end of all my text files and even on my data files if the records are delimited by a crlf and it makes it easy on editors too.

[Michael Mayerhoffer]

crlf and lf is the difference between unix text files and windows and dos text files. Most windows prgrams can not deal with unix text files. Unix has no problems working with windows text files.

I do not recall having problems sending Windows text files in binary mode, it does not drop cr. The problems comes sending binaries as text.

Binary should be what you want to use all the time.

Added note....depends on type of text and system, bottom line use binary

[Michael Mattias]

I was thinking about this some more after i posted this, and pretty well decided that I'll make "convert Unix Text to PC Text" and vice-versa a separate utility function, and just leave all the uploads and downloads binary.

FWIW, here is the product announcement.... which also explains a bit more about how this product is not so much about FTP, but about data-management and user interface. I'm really trying to make the data communications a 'bit player' in the app.

http://tech.groups.yahoo.com/group/EDI-L/message/22190

EDI-L is "the" listserve for Electronic Data Interchange professionals, and has been for over fifteen years. Yes, it does go back to before there was such a thing as 'easy' Worldwide Web browsing.. everything was on "bulletin board service" and later on Compuserve, until about five years ago when it went onto Yahoo! Groups.

Anyone who has an "EDI" requirement would be well-advised to search the message archives when a project comes up. But let me save you some time... don't bother searching the "Files" area because most of that stuff is useless.


MCM

[John McWilliams]

And Scott and me didn't even get a mention

[Michael Mattias]

>And Scott and me didn't even get a mention

You are not known in that world. I am.

Besides, did you really want your names associated with the part of the announcement where I say..

...it is not terribly sophisticated vis-a-vis all the various options available with File Transfer Protocol.... the bits and bytes and nuts and bolts of FTP are not what this product is all about...

[John McWilliams]

>And Scott and me didn't even get a mention

You are not known in that world. I am.

Besides, did you really want your names associated with the part of the announcement where I say..

Just yankin your chain a bit and you didn't mind your name being associated with it.

[Michael Mattias]

FWIW, I got that "Connect non-passive of possible and passive only if you can't do non-passive" thing coded and working. I kind of souped up the connect function to faciliate logging ...

' -----------------------------------------------------------------------------------------
' WRAPPER FUNCTIONS TO CONNECT / DISCONECT TO FTP SERVER.
' Returns: TRUE (connected), FALSE (not connected, text explanation inszLastResponseInfo)
' WHen succes, returns hInternet and hConnect for later use with Disconnect functon
' 1.2.2 add param bPasv (I-O). If true, tries only for passive mode connection...if false
' first tries active mode, then passive if that does not work.
' On success, sets bPasv to false if Active mode connection or true if passive mode connection
' ----------------------------------------------------------------------------------------

FUNCTION ConnectToFtpServer (BYVAL hWnd AS LONG, szServer AS ASCIIZ, szUserId AS ASCIIZ, szPassWord AS ASCIIZ, _
                             hInternet AS LONG, hConnect AS LONG, szLastResponseInfo AS ASCIIZ,bPasv AS LONG) AS LONG

    LOCAL cchLastResponseInfo  AS DWORD
    LOCAL iErrLastResponse     AS DWORD
    LOCAL dwFlags AS DWORD
    LOCAL szAgent AS ASCIIZ * 32
    LOCAL LE      AS LONG
    LOCAL iRet    AS LONG
    LOCAL szText  AS ASCIIZ * %MAX_PATH
    LOCAL iTry    AS LONG

    szAgent = "TSVANFTP"

    ' get internet handle first

    ' Open an Internet session as predefined in the registry
    hInternet = InternetOpen(BYVAL VARPTR(szAgent), _
                          BYVAL %INTERNET_OPEN_TYPE_PRECONFIG, _
                          BYVAL %NULL, _
                          BYVAL %NULL, _
                          BYVAL 0)
    LE   =  GetLastError
    ' If it fails exit with false and text message
    IF hInternet = %NULL THEN
        szLastResponseInfo  = SystemErrorMEssageText (LE)
        FUNCTION = %FALSE
        EXIT FUNCTION
    ELSE
        ' make two tries to connect to the named server; in active mode or passive mode
        cchLastResponseInfo =   %LAST_RESPONSE_INFO_SIZE    ' In/out
        FOR iTry = 1 TO 2

              IF iTry = 1   THEN             ' try non-passive connection unless forced passive

                  IF ISFALSE bPasv THEN     ' try the active mode connection
                       dwFlags             =   %NULL
                  ELSE
                       dwFlags  = %INTERNET_FLAG_PASSIVE
                  END IF
              ELSE                 ' try #2, is always tried for passive mode connection
                 dwFlags = %INTERNET_FLAG_PASSIVE
              END IF

              ' needs to be passive for getting  last error?
              ' NOTE: NULL is connecting just fine!!! So does PASV. Changed back.

              hConnect = InternetConnect(BYVAL hINternet, _
                                      BYVAL VARPTR(szServer), _
                                      BYVAL %INTERNET_DEFAULT_FTP_PORT, _
                                      bvaz (szUserId), _
                                      bvaz (szPassword), _
                                      BYVAL %INTERNET_SERVICE_FTP, _
                                      BYVAL dwFlags, _
                                      BYVAL 0)
               LE = GetLastError()

               IF ISTRUE hConnect THEN ' succesfull connection
                   EXIT FOR    ' force exit if success on iTry = 1
               END IF

        NEXT iTry
        ' -------------------
        ' 03/06/08. I cannot get this to work correctly. It works in other places
        ' I will stick with GetLasterror for now because I have to get this working
       ' CALL  InternetGetLastResponseInfo (iErrLastResponse, szLastResponseInfo, cchLastResponseInfo) to iRet
        ' the parameter is incorrect (87). ????
        ' BUffer too long?
        'IF iSFALSE iRET THEN
        '    LE   = GetLastError
        '    szText = SystemErrorMessageText (LE)
        '    MSGBOX USING$("&", szText),, "Could not get last responseinfo"
        'END IF
       ' returning iErrLastResponse = 0 when called immedaiately after InternetConnect with bad password

        ' "To retrieve extended error information, call GetLastError.
        '  An application can also use InternetGetLastResponseInfo to determine
        ' why access to the service was denied."
        ' InternetGetLastResponseINfo not working

        'LE  = GetLastError  ' with bad password comes back 12014.  BUt internet GetLast
        ' is returning nothing!!


      IF ISFALSE hConnect THEN
           szLastResponseInfo    = SystemErrorMessageText (LE)
           InterNetcloseHandle hInternet
           hInternet = %NULL
           FUNCTION = %FALSE
      ELSE
           FUNCTION = %TRUE   ' we are connected
           bPasv    = ISTRUE  (dwFlags=%INTERNET_FLAG_PASSIVE)  ' where dwflags was set when connect succeeded.

      END IF

   END IF  ' if we got an iNternet handle or not

END FUNCTION

By setting "bPasv" in this function, the main calling function can write to the log showing which kind of connection was made. I also left room to "force" it if I want to some day.

MCM

[Paul Franks]

I haven't used ftp in a while, since it's been essentially banned from the company due to its inherent insecurity.

Most ftp programs will try to guess whether to use ascii or binary mode by looking at the first few bytes of the file for CRLF or newlines. They frequently guess wrong, however, particularly with PDF files, which can look like ascii files, but most definitely aren't, so it's always a good idea to set the transfer type manually.

When we used to run automated ftp transfers, we always used Expect, which is a nice scripting language perfectly suited to such tasks. There's probably a version available for Windows, but the Windows built-in ftp app is pretty lame, so I would recommend one of the many freebie versions out there.

http://en.wikipedia.org/wiki/Expect

[John Spikowski]

For what it's worth, I'll share my Windows FTP story and maybe it will save others some pain.

I have a MLS (real estate) search engine I wrote and I have to poll a web service to get new and changed photo filenames. From the response I would build a FTP script to download the photos. The MLS FTP server is hammered with over 200 people like myself trying to download 20,000 photos a day. (normal and thumbnails)

I must have tried at least 10 different off-the-shelf FTP programs that supported scripting. They all failed to deal with the over worked FTP server that would just go away for 5 minutes at a time or give you a IP and port to connect to download the file and never respond. (this hung 90% of the FTP clients I tried) I ended up trying to write my own FTP download only client in PowerBASIC using the examples here on the forum. I was still having to baby sit the transfers because the last thing you want to have happen is get behind.

I decided to try lftp under Linux and it worked without a burp. It handled anything this FTP server would throw at it and lftp would recover and finish the script.

I don't know if lftp comes in a Windows version or not but my gut feeling is that the Windows API for FTP support SUCKS! IMHO

John

[Michael Mattias]

have a MLS (real estate) search engine I wrote and I have to poll a web service to get new and changed photo filenames. From the response I would build a FTP script to download the photos. The MLS FTP server is hammered with over 200 people like myself trying to download 20,000 photos a day. (normal and thumbnails)

???
There are 20,000 new or changed photos each day?

(If there are, don't read the rest of this as you may feel insulted that I did not consider you may already be doing this).

If not, and you just download them because its convenient to replace them all... you might want to change your application to use FtpFindFirstFile/FtpFindNextFile to get the file date-time on the server, and compare to the date-time stamp on your local pc, and only download new or changed.

That doesn't deal with an overworked FTP server, but at least you will be able to say it ain't you who's eating up bandwidth and cycles.. the Moral High Ground if you will.

MCM

[Paul Franks]

I don't know if lftp comes in a Windows version or not

It doesn't, but there are plenty of others that do, including several that are cross-platform:

http://en.wikipedia.org/wiki/Comparison_of_FTP_client_software

Many ftp programs will allow you to check the timestamp of files, and only download those that are new or have changed. We use wget with the
--timestamp option to do this for http downloads, but it also works with ftp:

If you wished to mirror the GNU archive every week, you would use a command like the following, weekly:

wget --timestamping -r ftp://ftp.gnu.org/pub/gnu/

[John Spikowski]

Michael,

Here is the 6 PM run. It's 7440 photos (big/tmb) total and this cron process runs every three hours. (8 times a day) If any of the photos in a listing set are changed, the whole set needs to be downloaded again. The first part is a ScriptBasic program that queries the web service to get the photo file names and the BIG and TMB FTP runs are identical so you have to double the photo count shown. I have the script e-mail this report to me after every run.

John

EPIC - Linux

Execution Time: 2008-04-17 18:00:01
Query Run Time: 2008-04-17T14:15:00 2008-04-17T17:15:00

RESI 2784
COND  398
VACL  251
MANU   69
MULT   68
BUSO    7
COMI   61
FARM   33
RENT   49

Program End Time: 2008-04-17 18:01:11  Photo Count: 3720


BIGPHOTO

---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
<--- 250 Directory changed to /photos/bigphoto
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.

--- BIGPHOTO Complete ---
Thu Apr 17 18:34:22 PDT 2008


TMBPHOTO

---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
<--- 250 Directory changed to /photos/tmbphoto
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.
**** Timeout - reconnecting
---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
<--- 220 Serv-U FTP Server v6.4 for WinSock ready....
<--- 230 User logged in, proceed.

--- TMBPHOTO Complete ---
Thu Apr 17 19:04:32 PDT 2008

[Michael Mattias]

It any of the photos in a listing set is changed, the whole set needs to be downloaded again.

Yuck. Send me the contact info for the service provider. He clearly could benefit from an improved design. (He will ingratiate himself with users like you).

[Paul Franks]

If any of the photos in a listing set are changed, the whole set needs to be downloaded again.

I'm probably missing something, but why couldn't lftp's "mirror" option be used on the two directories? It will only download the files which have changed...

[John Spikowski]

I'm probably missing something, but why couldn't lftp's "mirror" option be used on the two directories? It will only download the files which have changed...

We don't have directory read access. There are 1000 sub-directories on the FTP server with thousands of photos in each. The FTP server only accepts a full path to the file requested.

This is a very political non-profit group that runs the MLS and if you step on the wrong toes, you loose your access. There were a few users that couldn't get a run to complete so they would reconnect after every photo. This caused HUGE delays and when I complained, I was raked over the coals.

I have to live with what they provide and like it.

John

[Paul Franks]

I have to live with what they provide and like it.

It's never simple when it gets political...

FWIW, here's a reference to a Windows version (using Cygwin) of lftp:

http://www.hackmyidea.com/wordpress/2008/01/20/windows-binary-for-lftp/

[John Spikowski]

I tried to install the Windows version of lftp and the install.bat failed trying to register the two DLL's. (no entry points found)

I was able to run the program in a console in the directory I downloaded and unziped the files in.

Looks like a manual file copy is in order.

BTW: I wonder if this is a scaled down version as there is no mention of a que or any of the other options that are in the Linux version.

John

lftp man page

[Paul Franks]

I think it's a bit out of date:

C:\temp\lftp>lftp --version
Lftp | Version 2.3.8 | Copyright (c) 1996-2001 Alexander V. Lukyanov
This is free software with ABSOLUTELY NO WARRANTY. See COPYING for details.
Send bug reports and questions to <[email protected]>.

C:\temp\lftp>lftp --help
Usage: lftp [OPTS] <site>
`lftp' is the first command executed by lftp after rc files
 -f <file>           execute commands from the file and exit
 -c <cmd>            execute the commands and exit
 --help              print this help and exit
 --version           print lftp version and exit
Other options are the same as in `open' command
 -e <cmd>            execute the command just after selecting
 -u <user>[,<pass>]  use the user/password for authentication
 -p <port>           use the port for connection
 <site>              host name, URL or bookmark name

C:\temp\lftp>lftp
lftp> help
        !<shell_command>                    (commands)
        alias [<name> [<value>]]            anon
        bookmark [SUBCMD]                   cache [SUBCMD]
        cat [-b] <files>                    cd <rdir>
        chmod mode file...                  close [-a]
        debug [<level>|off] [-o <file>]     exit [<code>|bg]
        get [OPTS] <rfile> [-o <lfile>]     help [<cmd>]
        jobs [-v]                           kill all|<job_no>
        lcd <ldir>                          lftp [OPTS] <site>
        ls [<args>]                         mget [OPTS] <files>
        mirror [OPTS] [remote [local]]      mkdir [-p] <dirs>
        module name [args]                  more <files>
        mput [OPTS] <files>                 mrm <files>
        mv <file1> <file2>                  nlist [<args>]
        open [OPTS] <site>                  pget [OPTS] <rfile> [-o <lfile>]
        put [OPTS] <lfile> [-o <rfile>]     pwd [-p]
        quote <cmd>                         reget [OPTS] <rfile> [-o <lfile>]
        rels [<args>]                       renlist [<args>]
        reput <lfile> [-o <rfile>]          rm [-r] [-f] <files>
        rmdir [-f] <dirs>                   scache [<session_no>]
        set [OPT] [<var> [<val>]]           site <site_cmd>
        source <file>                       user <user|URL> [<pass>]
        version                             wait [<jobno>]
        zcat <files>                        zmore <files>
        bzcat <files>                       bzmore <files>
lftp>

However, if you're willing to install the entire Cygwin package, a much newer version of lftp is available in the "net" category. Cygwin will give you a 'nix environment under Windows.

[Paul Franks]

Just updated my Cygwin install to include lftp, here's the result

C:\cygwin\bin>lftp --version
LFTP | Version 3.7.1 | Copyright (c) 1996-2008 Alexander V. Lukyanov

LFTP is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
There is absolutely no warranty for LFTP.  See COPYING for details.

Send bug reports and questions to <[email protected]>.

Libraries used: Readline 5.2, Expat 1.95.8, OpenSSL 0.9.8g 19 Oct 2007, libiconv 1.11

C:\cygwin\bin>lftp --help
Usage: lftp [OPTS] <site>
`lftp' is the first command executed by lftp after rc files
 -f <file>           execute commands from the file and exit
 -c <cmd>            execute the commands and exit
 --help              print this help and exit
 --version           print lftp version and exit
Other options are the same as in `open' command
 -e <cmd>            execute the command just after selecting
 -u <user>[,<pass>]  use the user/password for authentication
 -p <port>           use the port for connection
 <site>              host name, URL or bookmark name

C:\cygwin\bin>lftp
lftp :~> help
        !<shell-command>                    (commands)
        alias [<name> [<value>]]            anon
        bookmark [SUBCMD]                   cache [SUBCMD]
        cat [-b] <files>                    cd <rdir>
        chmod [OPTS] mode file...           close [-a]
        [re]cls [opts] [path/][pattern]     debug [<level>|off] [-o <file>]
        du [options] <dirs>                 exit [<code>|bg]
        get [OPTS] <rfile> [-o <lfile>]     glob [OPTS] <cmd> <args>
        help [<cmd>]                        jobs [-v]
        kill all|<job_no>                   lcd <ldir>
        lftp [OPTS] <site>                  ls [<args>]
        mget [OPTS] <files>                 mirror [OPTS] [remote [local]]
        mkdir [-p] <dirs>                   module name [args]
        more <files>                        mput [OPTS] <files>
        mrm <files>                         mv <file1> <file2>
        [re]nlist [<args>]                  open [OPTS] <site>
        pget [OPTS] <rfile> [-o <lfile>]    put [OPTS] <lfile> [-o <rfile>]
        pwd [-p]                            queue [OPTS] [<cmd>]
        quote <cmd>                         repeat [OPTS] [delay] [command]
        rm [-r] [-f] <files>                rmdir [-f] <dirs>
        scache [<session_no>]               set [OPT] [<var> [<val>]]
        site <site_cmd>                     source <file>
        user <user|URL> [<pass>]            version
        wait [<jobno>]                      zcat <files>
        zmore <files>                       history -w file|-r file|-c|-l [cnt]
lftp :~>