WinInet FTP problem, help wanted

[Michael Mattias]

I have been working on an application using the file upload I had posted in the Source COde forum at WinInet FTP Upload (Overwrite or Append) Demo 3-10-08

A 'now and then' problem has come up:

Every "now and then" the FtpPutFile function fails with ERROR_INTERNET_CONNECTION_RESET (value 12031d).

I spent about an hour going thru MSDN looking for what can cause this, but everything came back to the same old crummy explanation in the SDK doc: "The connection with the server has been reset."

Does anyone know what causes this error? For that matter, does anyone understand what it means, "The connection with the server has been reset?"

It 'seems' to occur only when overwriting an existing file. And it "may" (I'd have to re-set up the test data and hope I can make it happen) have something to do with the same LOCAL file being sent twice in the same connection session to two different folders on the server.


Basically, the demo I posted above is all the salient "internet" code. The application itself does this:

  Get filespecs to upload: may be any number each containing source name/mask, target name/token
  Connect to FTP site 
  FOR EACH Spec
       Build list of local files found matching source spec 
       FOR each file file found 
           create output name (replaceable tokens can be used) 
           FtpPutFile in default (binary) mode  (default behavior, create if not found, overwrite if found) 
       NEXT 
  NEXT Spec
  Disconnect

One of the MSDN forum thread entries led me to believe it might be related to caching, but that thread died before the question was fully addressed, so that's equally pot luck.

But if caching is an issue, maybe I should disconnect/reconnect more often? Does that sound reasonable? This is a batch operation, so frankly it would be fine to disconnect/reconnect for each file.


Any pointers or suggestions will be appreciated.

Thanks,

[Joe Byrne]

But if caching is an issue, maybe I should disconnect/reconnect more often? Does that sound reasonable? This is a batch operation, so frankly it would be fine to disconnect/reconnect for each file.

Well, what happened when you tried it?

Sorry, couldn't resist.

I'm not in my office, but someplace I have some notes on a situation that sounds vaguely familiar. I'll see if I can dig it up later tonight.

[Michael Mattias]

I am trying it, it's just that even when (if?) if works, I would be trying to prove a negative.

So I connect/disconnect, run about 20 test runs without error. Now what? Do I have so much as a clue that this was in fact the problem?

If it NEVER worked, that's one thing, but when it only fails "now and then" is when it gets spooky and I'd kind of like to know WHY I should have been disconnecting/reconnecting more often.


MCM

[Michael Mattias]

Disconnecting/Reconnecting solved the 12031 problem, but then I ran into another probkle,,, program not returning from call to InternetConnect. Task manager/kill time.

So I changed some load file settings, and now it worked!

One thing I noticed... the change I made, I added the token [HHMMSS] to the "save as" name. This is replaced with current 24 hour time for each file... so... I force a 1 second delay every time this token is used so it is never duplicated.

"So what if," I asked myself, " I would always delay one second between files to be sent?".

So I tried that with new settings, old settings, bad settings, you name it, and now the program will not fail.

Which begs the question: should I have expected some kind of "overload" situation here, that is, some kind of situation where I should a delay should have resolved whatever is going on here?

This is what I mean by spooky. I think I fixed it, but I don't even know what was broken!

This generic "Hit it with a hammer and see if that fixes it" approach is not really my style at all.

[Michael Mayerhoffer]

Michael,

ERROR_INTERNET_CONNECTION_RESET < that means Server closed connection. Why - connection was established and most likely it thinks you are not reachable for some reason. So it performs a reset.


How can this happen - Firewall or router (NAT) blocks second (data) port request and if it fails, boom, you get reset. Also port usage along with firewall or router (NAT) used could make it intermittent looking. I have been up and down this road several times.

It could be other things, this fits the error and your symptoms the best, unless there is a server issue. If you could get someone to look at the server logs that would be a big help.

BTW - Did you try other FTP software and did it fail also ? Did you change firewall settings ?

Check this out this out for full details.

http://www.slacksite.com/other/ftp.html

If you don't think that is it, umm is the host Windows and what flavor ? or not windows.

[Michael Mattias]

>BTW - Did you try other FTP software

There is no other FTP software which does this. That's why I am writing it.

> Did you change firewall settings ?

Nope.

>Check this out this out for full details.

That stuff is way beyond my pay grade. I am using "passive mode" if that helps.

Basically this application is the demo I developed and posted at the above link, with some other code to get the source filenames and modify the target filenames, that's it.

Except in this app I don't do 'append' at all... if the target file exists the only options are "overwrite" or "skip". (I have 'business logic' reasons for not allowing an append, not technical issues).

Well, I have the software out at a couple of places for pre-release evaluation (it's actually in production use at two places). I may just have to wait for feedback.

[Michael Mattias]

Why - connection was established and most likely it thinks you are not reachable for some reason. So it performs a reset.

Hmmmm.....

It thinks I'm not reachable because I am uploading files essentially non-stop, meaning I'm never giving the server the chance to verify I'm still there?

Suggesting that periodic delays serve as kind of 'breathers' for the server, so it can get around to verifying I'm still there?

?????

-----------------------------
Straw Grasping in Wisconsin

[John McWilliams]

Mr. Mattias, have you looked at latency on the line? An overloaded network could cause the reset. Ping the target and see what the latency is.

[Joe Byrne]

I am using "passive mode" if that helps.

You might try not using passive mode. If you can connect to the FTP server, and get a data connection back without passive mode, the likelihood of a more 'stable' connection is far greater.

I don't use passive mode for my servers (its available, but I tell my customers not to use it) as active mode provides a much more reliable transmission in my experience. Your FTP client shouldn't have a problem with active mode, but the server might, depending on how they've firewalled it.

[Michael Mattias]

Oooh, oooh, actual tips...i.e, "some things to try."

>Ping the target and see what the latency is.

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Michael>ping www.talsystems.com

Pinging www.talsystems.com [209.225.111.245] with 32 bytes of data:

Reply from 209.225.111.245: bytes=32 time=27ms TTL=54
Reply from 209.225.111.245: bytes=32 time=25ms TTL=54
Reply from 209.225.111.245: bytes=32 time=26ms TTL=54
Reply from 209.225.111.245: bytes=32 time=24ms TTL=54

Ping statistics for 209.225.111.245:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 24ms, Maximum = 27ms, Average = 25ms

C:\Documents and Settings\Michael>

I have no clue what this means; but I sure hope it's good news. If not, how much time have I got left, doctor?

If you can connect to the FTP server, and get a data connection back without passive mode, the likelihood of a more 'stable' connection is far greater.

I did not know that. (Like you hadn't been able to guess I didn't know that).

But as noted in the demo*, I tried both active and passive and both worked fine with absolutely no difference in performance.

So what you are suggesting is, my 'connect' function should try Active; if that fails for some appropriate reason code, then try passive? I can do that.

In its primary design use, security is NOT a consideration in this application.

But I got five bucks someone is going to whine about passive being more secure and I'm using active if available.

Well, when I set up the code I made active/passive an INI file setting, albeit I did NOT put this on the setup screen for the user to maintain. Maybe I should just pass the buck and let the user decide. I really don't like that, since I know I will get questions from Suzy User, "Should I check the "Passive" box?" and I won't know what to tell her, which will no doubt dramatically reduce my sex appeal to Suzy, which won't be a Good Thing.


MCM
* strange how a guy who doesn't know Internet connections from beans is creating demos, huh?

[Joe Byrne]

The latency time in your ping test looks good, but it really doesn't tell you much. ICMP traffic is pretty useless for testing the quality of a connection. It does tell you however, that at the very exact and specific time that you did the ping, the turn around time was pretty good. Now, in 2 minutes ......

I would bet that your problems will go away if you use Active instead of passive mode. If you want to "automate" the setting, try an active connection first. If it fails, switch to passive and try that. If it fails too, then warn the user that you can't connect. I would do this every time I make the connection, in fact, this is how I do it.

There really isn't a "security" issue with active vs. passive. The difference is who is 'footing the bill' for whatever security there is with FTP. With active, its up to the client needs to have the proper ports open, with Passive, the Server has to open different ports which may or may not be an issue. Its really just a matter of defining who specifies the route that the data will flow across.

Oh yea, BTW, I know how old you are at our age, 'sex appeal' was lost long ago

[Michael Mattias]

Ok, I will make the change on connection: try active first, passive only if unable to connect active.

Thanks.

Oh yea, BTW, I know how old you are at our age, 'sex appeal' was lost long ago

Mr. Byrne does not speak for all thread participants.

[Joe Byrne]

Mr. Byrne does not speak for all thread participants.

[Paul Purvis]

i had seen this type of behavior some years ago(timeout issues), when i was the ftp program provided by windows and using filezilla server.
i broke up my script into multiple scripts to solve my program with reconnection for every script.


is the connection wireless, that may cause problems.
i have also changed every computer's ethernet setting using drtcp program to tcp windows receive to 63888 and mtu to 1492

[John McWilliams]

Joe's point about latency is accurate but it's got another side to it. If you check the immediate latency and it's above, say, 250 ms then it's probably not a good time to initiate the contact.

here's what I get when I ping you:

Pinging www.talsystems.com [209.225.111.245] with 32 bytes of data:


Reply from 209.225.111.245: bytes=32 time=78ms TTL=45
Reply from 209.225.111.245: bytes=32 time=75ms TTL=45
Reply from 209.225.111.245: bytes=32 time=76ms TTL=45
Reply from 209.225.111.245: bytes=32 time=76ms TTL=45

Ping statistics for 209.225.111.245:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 75ms, Maximum = 78ms, Average = 76ms

Still pretty good but much slower. It has to do with the fact that I am going through a number of resources before I get to you. All of those resources effect your connection. It's not a simple thing this WAN stuff to many variables that are out of your control.

[Shawn Anderson]

Joe's right, a standard ping is only shows minimal continuity.

try pinging with a loaded packet:

ping 209.225.111.245 -t -l 1450

the ping times should go up but you shouldn't drop any packets

[Paul Purvis]

while using ftp, i have found out thought experimentation, that the QOS packet scheduler should be install on the ethernet connection to provide a load balancing strategy on ethernet connections. i just enable the QOS, i really have not studied it to provide some kind of reasonable remarks on it.

paul

[Marco Pontello]

I think that using the WinInet FTP leave too little control over how to establish & conduct the FTP connection. If I'm remembering correctly, Mike posted something about this not too long ago.
You'll find situations in witch it works perfectly; but when it will not, you haven't many (read: about none) ways to try different tricks & settings, especially if you can't access / reconfigure the NAT/router at your side, the FTP server, etc.

I think your best bet is either using a dedicated library or a command line tool like NcFTPGet / NcFTPPut that let you control almost everything.

http://www.ncftp.com/ncftp/

NcFTPPut let you also do some nice things, like for example upload a file with a temporary name, and then rename it automatically only when the upload is completed.

Bye!

[Knuth Konrad]

I think that using the WinInet FTP leave too little control over how to establish & conduct the FTP connection. If I'm remembering correctly, Mike posted something about this not too long ago.

Yepp, here's that posting: http://www.powerbasic.com/support/pb...90&postcount=3

And this is what MCM replied to Mike's posting:

It works for my client's application.

Silly me thought that was the only thing which really mattered.

[John McWilliams]

Joe's right, a standard ping is only shows minimal continuity.

try pinging with a loaded packet:

ping 209.225.111.245 -t -l 1450

the ping times should go up but you shouldn't drop any packets

Good Point!