Announcement

Collapse
No announcement yet.

WinInet FTP problem, help wanted

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Paul Purvis
    replied
    Michael,

    i have had different problems with tcp connections over the years. one of the biggest things i have done lately that has solved many connection problems is changing the Tcp receive window.

    i figure the majority of sites being accessed are not using a MTU of 1500, that is just a guess. we use a lot of dsl connections at our locations and it is best not to set the MTU over 1492 on most DSL connections.

    i started setting the MTU on all equipment to 1492, and after doing many hours of test, i have come to conclusion a TCP receive window of 63888 is best for a MTU of 1492.

    now everything is much smoother, transferring files with inhouse programs(my speed doubled and transfer time was cut in half) and my remote control programs(pcanywhere) now run very smooth with only a few disconnects.

    even web site browsing is much faster, i am using w2kpro.
    on winxp machines i also set those to the mtu of 1492 and tcp receive windows of 63888.

    one location is on a cable modem where 1500 is best used but i left the modem router set to a MTU of 1500 and set all machines with a MTU of 1492 to match my locations elsewhere.

    if you want to set the tcp receive window with a mtu of 1500 try 64240.

    even though setting a tcp receive window higher than 63888 or 64240 will increase download speeds of large items such as files being downloaded or playing of video over the internet, it may not be best for other purposes.

    hope this helps you out and others while using highspeed broadband internet connections.

    paul
    Last edited by Paul Purvis; 19 May 2008, 01:09 PM.

    Leave a comment:


  • BOB MECHLER
    replied
    Just saw the thread and what I'm adding to the discussion may have already have been touched on. I have an application where a 126 megabit file needs to be uploaded to a backup site everynight. I use active and set the number of retries on the ftp server to at least 20 or so. I use FileZilla ftp server and client. I used to get the server reset thing when I used passive and not enough possible retries. I also create a unique name each time I send the file.

    Bob Mechler

    Leave a comment:


  • Michael Mattias
    replied
    >did you try that program that was timing out on a win 98 machine.

    Nope. Actually I'm not testing anything new on Win/9x anymore. #OPTION VERSION 5 is now de rigeur for me.

    Leave a comment:


  • Paul Purvis
    replied
    to MCM
    did you try that program that was timing out on a win 98 machine.
    just food for thought.
    paul

    Leave a comment:


  • John Spikowski
    replied
    Originally posted by Paul Franks View Post
    Great! Cygwin gives you a bash shell and access to hundreds of 'nix command line apps, many of which are far better than the Windows-only equivalents.

    You might want to check out wget, as well (it should be part of the Cygwin base install, IIRC).
    If I were to install this again, I would be much more selective. The ALL options installs things like GTK and other X based libraries that are useless.

    3.08 GB (3,315,990,528 bytes)

    167,487 files and 9,323 directories

    The *nix utilities are nice to have. I want to try Bash shell scripting to see how that works under Windows. Cygwin is using the Windows API under the covers so I don't know if this is just Windows in *nix clothing.

    John
    Last edited by John Spikowski; 19 Apr 2008, 03:55 PM.

    Leave a comment:


  • Paul Franks
    replied
    Originally posted by John Spikowski View Post
    Update:

    I have installed the full version of Cygwin on my XP Pro system so I'm good.
    Great! Cygwin gives you a bash shell and access to hundreds of 'nix command line apps, many of which are far better than the Windows-only equivalents.

    You might want to check out wget, as well (it should be part of the Cygwin base install, IIRC).

    Leave a comment:


  • Michael Mattias
    replied
    Wait a minute... we have another current thead here which may be of some interest and might use the same solution....let me find that.

    Yes, here it is..http://.http://www.powerbasic.com/su...ad.php?t=37152

    Perhaps you could query the files' timestamps INDEPENDENT of the FTP function? If you have a file name, you should be able to InternetOpenUrl it. What I don't know is if you'll be able to get the timestamp if your permissions do not include directory "read" access.. then again, I don't know how you could possibly download a file via FTP without such "read" access.

    For that matter, you might be able to "dir" (windows' server) or "ls" (unix server) the file and get back the output of that command, which could include a timestamp you could parse out.

    Worth a try?
    MCM
    Last edited by Michael Mattias; 19 Apr 2008, 09:45 AM.

    Leave a comment:


  • John Spikowski
    replied
    With all their rules, forms and playing favorites, I have to wonder sometimes if selling real estate is still the goal.

    John

    Leave a comment:


  • Paul Purvis
    replied
    John,

    years a ago, before internet, my cousin and i worked on a program to run the mls out of business and hopefully make us rich.
    he even became a real estate agent to better learn the needs of the market.

    i was like microsoft, but without their money, i made a mistake of not seeing the internet coming, we were going to dump everything (pictures/data) on a cdrom, with only brand new pictures/data being accessed in the back ground by dial up modem, we had a lot of everything worked out that would work.

    i quit on it when the internet became about because i had no knowledge on how to use it and no large money to invest, just skills and time, and figured some others had a big jump on us with the internet.
    so a year of hard work on the side went straight out the window, not to mention learning the marketplace, but after seeing your post, humm.

    you are far to right about mls, at least in those days, the product was never as good as it should of been, and protected like something sacred.

    paul

    Leave a comment:


  • John Spikowski
    replied
    Any chance of sharing the lftp binaries needed to run under Windows?

    John

    Update:

    I have installed the full version of Cygwin on my XP Pro system so I'm good.
    Last edited by John Spikowski; 18 Apr 2008, 11:27 PM.

    Leave a comment:


  • Paul Franks
    replied
    Just updated my Cygwin install to include lftp, here's the result

    Code:
    C:\cygwin\bin>lftp --version
    LFTP | Version 3.7.1 | Copyright (c) 1996-2008 Alexander V. Lukyanov
    
    LFTP is free software, covered by the GNU General Public License, and you are
    welcome to change it and/or distribute copies of it under certain conditions.
    There is absolutely no warranty for LFTP.  See COPYING for details.
    
    Send bug reports and questions to <[email protected]>.
    
    Libraries used: Readline 5.2, Expat 1.95.8, OpenSSL 0.9.8g 19 Oct 2007, libiconv 1.11
    
    C:\cygwin\bin>lftp --help
    Usage: lftp [OPTS] <site>
    `lftp' is the first command executed by lftp after rc files
     -f <file>           execute commands from the file and exit
     -c <cmd>            execute the commands and exit
     --help              print this help and exit
     --version           print lftp version and exit
    Other options are the same as in `open' command
     -e <cmd>            execute the command just after selecting
     -u <user>[,<pass>]  use the user/password for authentication
     -p <port>           use the port for connection
     <site>              host name, URL or bookmark name
    
    C:\cygwin\bin>lftp
    lftp :~> help
            !<shell-command>                    (commands)
            alias [<name> [<value>]]            anon
            bookmark [SUBCMD]                   cache [SUBCMD]
            cat [-b] <files>                    cd <rdir>
            chmod [OPTS] mode file...           close [-a]
            [re]cls [opts] [path/][pattern]     debug [<level>|off] [-o <file>]
            du [options] <dirs>                 exit [<code>|bg]
            get [OPTS] <rfile> [-o <lfile>]     glob [OPTS] <cmd> <args>
            help [<cmd>]                        jobs [-v]
            kill all|<job_no>                   lcd <ldir>
            lftp [OPTS] <site>                  ls [<args>]
            mget [OPTS] <files>                 mirror [OPTS] [remote [local]]
            mkdir [-p] <dirs>                   module name [args]
            more <files>                        mput [OPTS] <files>
            mrm <files>                         mv <file1> <file2>
            [re]nlist [<args>]                  open [OPTS] <site>
            pget [OPTS] <rfile> [-o <lfile>]    put [OPTS] <lfile> [-o <rfile>]
            pwd [-p]                            queue [OPTS] [<cmd>]
            quote <cmd>                         repeat [OPTS] [delay] [command]
            rm [-r] [-f] <files>                rmdir [-f] <dirs>
            scache [<session_no>]               set [OPT] [<var> [<val>]]
            site <site_cmd>                     source <file>
            user <user|URL> [<pass>]            version
            wait [<jobno>]                      zcat <files>
            zmore <files>                       history -w file|-r file|-c|-l [cnt]
    lftp :~>

    Leave a comment:


  • Paul Franks
    replied
    I think it's a bit out of date:

    Code:
    C:\temp\lftp>lftp --version
    Lftp | Version 2.3.8 | Copyright (c) 1996-2001 Alexander V. Lukyanov
    This is free software with ABSOLUTELY NO WARRANTY. See COPYING for details.
    Send bug reports and questions to <[email protected]>.
    
    C:\temp\lftp>lftp --help
    Usage: lftp [OPTS] <site>
    `lftp' is the first command executed by lftp after rc files
     -f <file>           execute commands from the file and exit
     -c <cmd>            execute the commands and exit
     --help              print this help and exit
     --version           print lftp version and exit
    Other options are the same as in `open' command
     -e <cmd>            execute the command just after selecting
     -u <user>[,<pass>]  use the user/password for authentication
     -p <port>           use the port for connection
     <site>              host name, URL or bookmark name
    
    C:\temp\lftp>lftp
    lftp> help
            !<shell_command>                    (commands)
            alias [<name> [<value>]]            anon
            bookmark [SUBCMD]                   cache [SUBCMD]
            cat [-b] <files>                    cd <rdir>
            chmod mode file...                  close [-a]
            debug [<level>|off] [-o <file>]     exit [<code>|bg]
            get [OPTS] <rfile> [-o <lfile>]     help [<cmd>]
            jobs [-v]                           kill all|<job_no>
            lcd <ldir>                          lftp [OPTS] <site>
            ls [<args>]                         mget [OPTS] <files>
            mirror [OPTS] [remote [local]]      mkdir [-p] <dirs>
            module name [args]                  more <files>
            mput [OPTS] <files>                 mrm <files>
            mv <file1> <file2>                  nlist [<args>]
            open [OPTS] <site>                  pget [OPTS] <rfile> [-o <lfile>]
            put [OPTS] <lfile> [-o <rfile>]     pwd [-p]
            quote <cmd>                         reget [OPTS] <rfile> [-o <lfile>]
            rels [<args>]                       renlist [<args>]
            reput <lfile> [-o <rfile>]          rm [-r] [-f] <files>
            rmdir [-f] <dirs>                   scache [<session_no>]
            set [OPT] [<var> [<val>]]           site <site_cmd>
            source <file>                       user <user|URL> [<pass>]
            version                             wait [<jobno>]
            zcat <files>                        zmore <files>
            bzcat <files>                       bzmore <files>
    lftp>
    However, if you're willing to install the entire Cygwin package, a much newer version of lftp is available in the "net" category. Cygwin will give you a 'nix environment under Windows.

    Leave a comment:


  • John Spikowski
    replied
    I tried to install the Windows version of lftp and the install.bat failed trying to register the two DLL's. (no entry points found)

    I was able to run the program in a console in the directory I downloaded and unziped the files in.

    Looks like a manual file copy is in order.

    BTW: I wonder if this is a scaled down version as there is no mention of a que or any of the other options that are in the Linux version.

    John

    lftp man page
    Last edited by John Spikowski; 18 Apr 2008, 03:26 PM.

    Leave a comment:


  • Paul Franks
    replied
    Originally posted by John Spikowski View Post
    I have to live with what they provide and like it.
    It's never simple when it gets political...

    FWIW, here's a reference to a Windows version (using Cygwin) of lftp:

    http://www.hackmyidea.com/wordpress/...nary-for-lftp/

    Leave a comment:


  • John Spikowski
    replied
    I'm probably missing something, but why couldn't lftp's "mirror" option be used on the two directories? It will only download the files which have changed...
    We don't have directory read access. There are 1000 sub-directories on the FTP server with thousands of photos in each. The FTP server only accepts a full path to the file requested.

    This is a very political non-profit group that runs the MLS and if you step on the wrong toes, you loose your access. There were a few users that couldn't get a run to complete so they would reconnect after every photo. This caused HUGE delays and when I complained, I was raked over the coals.

    I have to live with what they provide and like it.

    John

    Leave a comment:


  • Paul Franks
    replied
    Originally posted by John Spikowski View Post
    If any of the photos in a listing set are changed, the whole set needs to be downloaded again.
    I'm probably missing something, but why couldn't lftp's "mirror" option be used on the two directories? It will only download the files which have changed...

    Leave a comment:


  • Michael Mattias
    replied
    It any of the photos in a listing set is changed, the whole set needs to be downloaded again.
    Yuck. Send me the contact info for the service provider. He clearly could benefit from an improved design. (He will ingratiate himself with users like you).

    Leave a comment:


  • John Spikowski
    replied
    Michael,

    Here is the 6 PM run. It's 7440 photos (big/tmb) total and this cron process runs every three hours. (8 times a day) If any of the photos in a listing set are changed, the whole set needs to be downloaded again. The first part is a ScriptBasic program that queries the web service to get the photo file names and the BIG and TMB FTP runs are identical so you have to double the photo count shown. I have the script e-mail this report to me after every run.

    John

    Code:
    EPIC - Linux
    
    Execution Time: 2008-04-17 18:00:01
    Query Run Time: 2008-04-17T14:15:00 2008-04-17T17:15:00
    
    RESI 2784
    COND  398
    VACL  251
    MANU   69
    MULT   68
    BUSO    7
    COMI   61
    FARM   33
    RENT   49
    
    Program End Time: 2008-04-17 18:01:11  Photo Count: 3720
    
    
    BIGPHOTO
    
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    <--- 250 Directory changed to /photos/bigphoto
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    
    --- BIGPHOTO Complete ---
    Thu Apr 17 18:34:22 PDT 2008
    
    
    TMBPHOTO
    
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    <--- 250 Directory changed to /photos/tmbphoto
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    **** Timeout - reconnecting
    ---- Connecting to ftp.nwmls.com (72.5.57.177) port 21
    <--- 220 Serv-U FTP Server v6.4 for WinSock ready....
    <--- 230 User logged in, proceed.
    
    --- TMBPHOTO Complete ---
    Thu Apr 17 19:04:32 PDT 2008

    Leave a comment:


  • Paul Franks
    replied
    Originally posted by John Spikowski View Post
    I don't know if lftp comes in a Windows version or not
    It doesn't, but there are plenty of others that do, including several that are cross-platform:

    http://en.wikipedia.org/wiki/Compari...lient_software

    Many ftp programs will allow you to check the timestamp of files, and only download those that are new or have changed. We use wget with the
    --timestamp option to do this for http downloads, but it also works with ftp:

    If you wished to mirror the GNU archive every week, you would use a command like the following, weekly:

    wget --timestamping -r ftp://ftp.gnu.org/pub/gnu/

    Leave a comment:


  • Michael Mattias
    replied
    have a MLS (real estate) search engine I wrote and I have to poll a web service to get new and changed photo filenames. From the response I would build a FTP script to download the photos. The MLS FTP server is hammered with over 200 people like myself trying to download 20,000 photos a day. (normal and thumbnails)
    ???
    There are 20,000 new or changed photos each day?

    (If there are, don't read the rest of this as you may feel insulted that I did not consider you may already be doing this).

    If not, and you just download them because its convenient to replace them all... you might want to change your application to use FtpFindFirstFile/FtpFindNextFile to get the file date-time on the server, and compare to the date-time stamp on your local pc, and only download new or changed.

    That doesn't deal with an overworked FTP server, but at least you will be able to say it ain't you who's eating up bandwidth and cycles.. the Moral High Ground if you will.

    MCM

    Leave a comment:

Working...
X