Announcement

Collapse
No announcement yet.

WinHttpRequest Redirect to Secure Pages Fails?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • WinHttpRequest Redirect to Secure Pages Fails?

    I need a reliable way to download web pages and image files from the internet. I've been using URLDownloadToFile, but there's no easy way to set a timeout and it often hangs indefinitely. It's also quite slow.

    I have tried a number of options, including the examples on Jose's page at:
    http://www.jose.it-berater.org/smffo...p?topic=4368.0

    I adapted one of his examples to make transition to my own program easier, and it generally works well on both http and https web pages. However, if an unsecured http page redirects to a secured https page, it fails. Any ideas how to fix this?


    Code:
    ' ========================================================================================
    ' The following example shows how to open an HTTP connection, send an HTTP request, and
    ' read the response as an IStream. The data from the IStream is written to a file.
    ' ========================================================================================
    
    #COMPILE EXE
    #DIM ALL
    #INCLUDE ONCE "httprequest.inc"
    #INCLUDE ONCE "ole2utils.inc"
    
    
    FUNCTION PBMAIN
    LOCAL url AS STRING
    LOCAL s AS STRING
    
    ' Unsecured web page downloads fine
    '===================================
    'url$="http://smashedpeasandcarrots.com/sugar-cookie-icing-the-recipe/"
    
    
    ' Secured web page downloads fine
    '=================================
    'url$="https://damndelicious.net/2021/10/21/pumpkin-donut-holes/"
    
    
    ' Unsecured page fails redirect to the secure page?
    '===================================================
    url$="http://www.cdkitchen.com/recipes/recs/2516/Spare-Ribs-And-Sauerkraut104302.shtml"
    
    
    ' Secured page redirected from above loads fine
    '===============================================
    'url$="https://www.cdkitchen.com/recipes/recs/2516/Spare-Ribs-And-Sauerkraut104302.shtml"
    
    CALL url_to_string(url$,s$)
    
    
    ' Save the string to a file
    '=============================
    OPEN "testing.txt" FOR BINARY AS #1
    PUT #1, 1, s$
    CLOSE #1
    
    MSGBOX "File saved"
    
    END FUNCTION
    
    
    SUB url_to_string(url$,s$)
    LOCAL pWHttp AS IWinHttpRequest
    LOCAL vSTream AS VARIANT
    LOCAL pIStream AS IStream
    LOCAL tstatstg AS STATSTG
    LOCAL cb AS DWORD
    LOCAL cbRead AS DWORD
    LOCAL iSucceeded AS INTEGER
    LOCAL strBuffer AS STRING
    
    s$=""
    
    ' Creates an instance of the HTTP service
    '==========================================
    pWHttp = NEWCOM "WinHttp.WinHttpRequest.5.1"
    IF ISNOTHING(pWHttp) THEN EXIT SUB
    
    TRY
    
    ' Opens an HTTP connection to an HTTP resource
    '==============================================
    pWHttp.Open "GET", url$
    
    
    ' Sends an HTTP request to the HTTP server
    '==========================================
    pWHttp.Send
    
    
    ' Wait for response with a timeout of 5 seconds
    '===============================================
    iSucceeded = pWHttp.WaitForResponse(5)
    
    
    ' Get the response as a stream
    '==============================
    vStream = pWHttp.ResponseStream
    IF VARIANTVT(vStream) = %VT_UNKNOWN THEN
    pIStream = vStream
    vStream = EMPTY
    
    ' Get the size of the stream
    '============================
    pIStream.Stat(tstatstg, %STATFLAG_NONAME)
    
    
    ' Use a string as a buffer
    '==========================
    s$=SPACE$(tstatstg.cbsize)
    
    
    ' Read the stream
    '=================
    pIStream.Read STRPTR(s$), LEN(s$), cbRead
    IF cbRead < tstatstg.cbsize THEN
    s$ = LEFT$(strBuffer, cbRead)
    END IF
    pIStream = NOTHING
    
    END IF
    
    CATCH
    OleShowErrorInfo OBJRESULT
    END TRY
    
    END SUB
    Anthony Watson, Mountain Software
    www.mountainsoftware.com

  • #2

    url$="http://www.cdkitchen.com/recipes/recs/2516/Spare-Ribs-And-Sauerkraut104302.shtml"

    Remove the "s" from the file extension for the insecure connection and it works fine

    url$="http://www.cdkitchen.com/recipes/recs/2516/Spare-Ribs-And-Sauerkraut104302.html"

    Comment


    • #3
      Who needs a recipe to make spareribs and sauerkraut?
      Michael Mattias
      Tal Systems (retired)
      Port Washington WI USA
      [email protected]
      http://www.talsystems.com

      Comment


      • #4
        Originally posted by Stuart McLachlan View Post
        Remove the "s" from the file extension for the insecure connection and it works fine
        Interesting. It's kind of a hack, but this is the only page I have encountered this with, so I guess this workaround will do for now.
        Anthony Watson, Mountain Software
        www.mountainsoftware.com

        Comment


        • #5
          Originally posted by Michael Mattias View Post
          Who needs a recipe to make spareribs and sauerkraut?
          It's just a sample page out of thousands of pages that have recipes. The content of this particular page is not important.
          Anthony Watson, Mountain Software
          www.mountainsoftware.com

          Comment


          • #6
            It's just a sample page out of thousands of pages
            Really? Ya Think?

            And to think some people believe programmers are detached from real life ("smart in school, dumb on the bus"). Where DO they get that idea?
            Michael Mattias
            Tal Systems (retired)
            Port Washington WI USA
            [email protected]
            http://www.talsystems.com

            Comment


            • #7
              Originally posted by Michael Mattias View Post
              Really? Ya Think?
              Yes, really.... There is more than one page on the entire internet. Honest.
              Anthony Watson, Mountain Software
              www.mountainsoftware.com

              Comment


              • #8
                Originally posted by Stuart McLachlan View Post
                Remove the "s" from the file extension for the insecure connection and it works fine
                As expected with most hacks, this was too easy to be true... While it does download a page, it's a "file not found" page, not the actual page.

                The problem appears to be with the SHTML page type, not with the http or https security status.

                Any idea how to get WinHttpRequest to work with SHTML pages?
                Anthony Watson, Mountain Software
                www.mountainsoftware.com

                Comment


                • #9
                  Yes, really.... There is more than one page on the entire internet. Honest
                  I've been posting in public since before the "Word Wide Web" made its debut in 1991... when we posted on dial-up bulletin boards. Those formative years plus the first several years of the WWW era are what helped me form my "rule of reading and understanding the posts of others", which is summarized as ..

                  "When online, take nothing literally and even less personally."

                  Sadly, not everyone reads online posts like this.
                  Michael Mattias
                  Tal Systems (retired)
                  Port Washington WI USA
                  [email protected]
                  http://www.talsystems.com

                  Comment


                  • #10
                    Since WinHtmlRequest seems to have a problem with SHTML file types, I thought I would try using URLOpenBlockingStream instead. This seems to work fine with web pages (even SHTML), but I'm getting an odd result with image files. Any idea why I'm only getting 10403 bytes instead of the full 38107 bytes the image is supposed to have? I've tried numerous images and they all seem to stop before the entire image is downloaded.

                    Code:
                    #COMPILE EXE
                    #DEBUG DISPLAY ON
                    #DEBUG ERROR ON
                    #INCLUDE "WIN32API.INC"
                    #INCLUDE "urlmon.inc"
                    
                    
                    FUNCTION PBMAIN()
                    
                    'url$="https://damndelicious.net/2021/10/21/pumpkin-donut-holes/"
                    'url$="https://www.cdkitchen.com/recipes/recs/2516/Spare-Ribs-And-Sauerkraut104302.shtml"
                    'url$="http://www.cdkitchen.com/recipes/recs/2516/Spare-Ribs-And-Sauerkraut104302.shtml"
                    'url$="https://foodandjourneys.net/lebanese-chicken-stew-with-vegetables/"
                    'url$="https://www.cdkitchen.com/recipes/recs/1129/Absolutely-The-Best-Enchiladas75972.shtml"
                    url$="https://cdn.cdkitchen.com/recipes/images/2015/01/898-3626-mx.jpg"
                    
                    CALL url_to_string(url$,s$)
                    
                    
                    ' Save the string to a file
                    '=============================
                    OPEN "test.jpg" FOR BINARY AS #1
                    PUT #1, 1, s$
                    CLOSE #1
                    
                    
                    MSGBOX "File saved: "+STR$(LEN(s$))+" bytes"
                    
                    ' Should be 38107 bytes.
                    ' Why only 10403 bytes downloaded?
                    
                    END FUNCTION
                    
                    SUB url_to_string(url$,s$)
                    LOCAL stream AS IStream
                    LOCAL hr AS LONG
                    LOCAL buffer AS ASCIIZ * 500
                    LOCAL bytesread AS DWORD
                    LOCAL u AS ASCIIZ*2048
                    
                    u$=url$
                    s$=""
                    
                    hr = URLOpenBlockingStream(NOTHING, u, stream, 0, NOTHING)
                    
                    WHILE (stream.Read(VARPTR(buffer), 500, bytesread) = %S_OK) AND (bytesread<>0)
                    s$=s$+LEFT$(buffer,bytesRead)
                    bytesread = 0
                    WEND
                    stream = NOTHING
                    
                    END SUB
                    Anthony Watson, Mountain Software
                    www.mountainsoftware.com

                    Comment


                    • #11
                      Seems to work OK if

                      LOCAL buffer AS String * 500

                      is used in the Sub
                      Rgds, Dave

                      Comment


                      • #12
                        Don't use STRINGZ (formerly ASCIIZ) for non-text data (like images). Any zero byte terminates the string.

                        Cheers,
                        Dale

                        Comment


                        • #13
                          Originally posted by Dave Biggs View Post
                          Seems to work OK if

                          LOCAL buffer AS String * 500

                          is used in the Sub
                          Good catch. Thanks! It's working great now.
                          Anthony Watson, Mountain Software
                          www.mountainsoftware.com

                          Comment


                          • #14
                            Originally posted by Dale Yarker View Post
                            Don't use STRINGZ (formerly ASCIIZ) for non-text data (like images). Any zero byte terminates the string.

                            Cheers,
                            It's obvious now that I know what the problem is. My bad for copying and pasting the code from other samples I found on the forum, but I was in a hurry and not paying attention.
                            Anthony Watson, Mountain Software
                            www.mountainsoftware.com

                            Comment


                            • #15
                              I'm still trying to figure out why many sites aren't redirecting with this code. Any ideas?

                              Code:
                              #COMPILE EXE
                              #DIM ALL
                              #INCLUDE ONCE "httprequest.inc"
                              #INCLUDE ONCE "ole2utils.inc"
                              
                              
                              FUNCTION PBMAIN
                              LOCAL url AS STRING
                              LOCAL s AS STRING
                              
                              
                              ' These URL's all download successfully
                              'url$="https://www.skinnytaste.com/feed/"
                              'url$="https://damndelicious.net/2021/10/21/pumpkin-donut-holes/"
                              'url$="https://foodandjourneys.net/lebanese-chicken-stew-with-vegetables/"
                              
                              
                              ' No errors reported, but returned string is empty?
                              'url$="https://www.cdkitchen.com/recipes/recs/1129/Absolutely-The-Best-Enchiladas75972.shtml"
                              
                              
                              ' Downloads error file stating anticrawler needs javascript?
                              'url$="https://www.foodfidelity.com/feed/"
                              
                              
                              ' Should redirect to http://feeds.feedburner.com/smittenkitchen
                              ' Error: The server returned and invalid or unrecognized response
                              url$="https://smittenkitchen.com/feed/"
                              
                              
                              ' Should redirect to http://feeds.feedburner.com/ACoupleCooks
                              ' Error: The operation has been canceled
                              'url$="https://www.acouplecooks.com/feed"
                              
                              CALL url_to_string(url$,s$)
                              
                              
                              ' Save the string to a file
                              '=============================
                              OPEN "testing.txt" FOR BINARY AS #1
                              PUT #1, 1, s$
                              CLOSE #1
                              
                              MSGBOX "File saved"
                              
                              END FUNCTION
                              
                              
                              SUB url_to_string(url$,s$)
                              LOCAL pWHttp AS IWinHttpRequest
                              LOCAL vSTream AS VARIANT
                              LOCAL pIStream AS IStream
                              LOCAL tstatstg AS STATSTG
                              LOCAL cb AS DWORD
                              LOCAL cbRead AS DWORD
                              LOCAL iSucceeded AS INTEGER
                              LOCAL strBuffer AS STRING
                              
                              s$=""
                              
                              ' Creates an instance of the HTTP service
                              '==========================================
                              pWHttp = NEWCOM "WinHttp.WinHttpRequest.5.1"
                              IF ISNOTHING(pWHttp) THEN EXIT SUB
                              
                              TRY
                              
                              ' Opens an HTTP connection to an HTTP resource
                              '==============================================
                              pWHttp.Open "GET", url$
                              
                              
                              ' Enable Redirects
                              '==================
                              pWHttp.Option(%WinHttpRequestOption_EnableRedirects)
                              pWHttp.Option(%WinHttpRequestOption_EnableHttpsToHttpRedirects)
                              
                              
                              ' Sends an HTTP request to the HTTP server
                              '==========================================
                              pWHttp.Send
                              
                              
                              ' Wait for response with a timeout of 5 seconds
                              '===============================================
                              iSucceeded = pWHttp.WaitForResponse(5)
                              
                              
                              ' Get the response as a stream
                              '==============================
                              vStream = pWHttp.ResponseStream
                              IF VARIANTVT(vStream) = %VT_UNKNOWN THEN
                              pIStream = vStream
                              vStream = EMPTY
                              
                              ' Get the size of the stream
                              '============================
                              pIStream.Stat(tstatstg, %STATFLAG_NONAME)
                              
                              
                              ' Use a string as a buffer
                              '==========================
                              s$=SPACE$(tstatstg.cbsize)
                              
                              
                              ' Read the stream
                              '=================
                              pIStream.Read STRPTR(s$), LEN(s$), cbRead
                              IF cbRead < tstatstg.cbsize THEN
                              s$ = LEFT$(strBuffer, cbRead)
                              END IF
                              pIStream = NOTHING
                              
                              END IF
                              
                              CATCH
                              OleShowErrorInfo OBJRESULT
                              END TRY
                              
                              END SUB
                              Anthony Watson, Mountain Software
                              www.mountainsoftware.com

                              Comment


                              • #16
                                This writes the download to a file and shells to it. Jose Roca includes.
                                It kills the file (if a previous one exists before writing to it) and also clears the cache.
                                Not sure what is wrong with stream method?
                                Code:
                                #INCLUDE "win32api.inc"
                                #INCLUDE "WinInet.inc"
                                FUNCTION PBMAIN AS LONG
                                 LOCAL sUrl AS STRING
                                 LOCAL sFile AS STRING
                                 LOCAL sErrorMsg AS STRING
                                 sUrl = "https://www.cdkitchen.com/recipes/recs/1129/Absolutely-The-Best-Enchiladas75972.shtml"
                                 sFile= "test.htm"
                                 DownloadFile sUrl,sFile,sErrorMsg
                                 ShellExecute (%NULL, "OPEN", sFile + $NUL, BYVAL %NULL, CURDIR$, %SW_SHOWNORMAL)
                                END FUNCTION
                                
                                FUNCTION DownloadFile(sUrl AS STRING,sFile AS STRING,sErrorMsg AS STRING) AS LONG
                                
                                  LOCAL hr AS LONG
                                  LOCAL pIBindStatusCallbackImpl AS IBindStatusCallbackImpl
                                  pIBindStatusCallbackImpl = CLASS "CIBindStatusCallbackImpl"
                                
                                  KILL sFile
                                  ERRCLEAR 'get rid of previous file so it isn't re-used
                                  hr =DeleteURLCacheEntry(sURL + $NUL)  '1 = success  clear the cache
                                  hr = URLDownloadToFile(NOTHING, sUrl+$NUL, sFile+$NUL,0,pIBindStatusCallbackImpl)
                                  IF hr THEN
                                      sErrorMsg = "Unable to find file to download"
                                      BEEP
                                  ELSE
                                    FUNCTION = 1
                                  END IF
                                END FUNCTION
                                
                                CLASS CIBindStatusCallbackImpl COMMON
                                
                                   INTERFACE IBindStatusCallbackImpl $IID_IBindStatusCallback
                                
                                      INHERIT IUNKNOWN
                                
                                      ' =====================================================================================
                                      METHOD OnStartBinding ( _                            ' VTable offset = 12
                                        BYVAL dwReserved AS DWORD _                        ' __in DWORD dwReserved
                                      , BYVAL pib AS IBinding _                            ' __in IBinding *pib
                                      ) AS LONG                                            ' HRESULT
                                
                                      END METHOD
                                
                                      ' =====================================================================================
                                      METHOD GetPriority ( _                               ' VTable offset = 16
                                        BYREF pnPriority AS LONG _                         ' __out LONG *pnPriority
                                      ) AS LONG                                            ' HRESULT
                                
                                      END METHOD
                                      ' =====================================================================================
                                      METHOD OnLowResource ( _                             ' VTable offset = 20
                                        BYVAL reserved AS DWORD _                          ' __in DWORD reserved
                                      ) AS LONG                                            ' HRESULT
                                
                                      END METHOD
                                      ' =====================================================================================
                                      METHOD OnProgress ( _                                ' VTable offset = 24
                                        BYVAL ulProgress AS DWORD _                        ' __in ULONG ulProgress
                                      , BYVAL ulProgressMax AS DWORD _                     ' __in ULONG ulProgressMax
                                      , BYVAL ulStatusCode AS DWORD _                      ' __in ULONG ulStatusCode
                                      , BYREF szStatusText AS WSTRINGZ _                   ' __in LPCWSTR szStatusText
                                      ) AS LONG                                            ' HRESULT
                                
                                      'OutputDebugString STR$(ulProgress) & STR$(ulProgressMax)
                                      IF ulProgress THEN
                                         LOCAL Percent AS LONG
                                         ulProgressMax = 1940690  '<----   replace with Pierre's code to get filesize
                                         Percent = UlProgress/UlProgressMax*100
                                         'PROGRESSBAR SET POS ghDlg, %ProgressBar1, Percent
                                         'CONTROL SET TEXT ghDlg, %StatusBar1, USING$("#, of #, bytes  (#%)",UlProgress,ulProgressMax,Percent)
                                         'IF UlProgress = UlProgressMax OR Percent = 100 THEN
                                         ' LOCAL x AS LONG
                                         ' FOR x = 1 TO 40
                                         '  PROGRESSBAR SET POS ghDlg, %ProgressBar1, Percent
                                         '  CONTROL REDRAW ghDlg,%ProgressBar1
                                         '  SLEEP 50
                                         ' NEXT
                                         'END IF
                                      END IF
                                
                                
                                      END METHOD
                                      ' =====================================================================================
                                      METHOD OnStopBinding ( _                             ' VTable offset = 28
                                        BYVAL hresult AS LONG _                            ' __in HRESULT hresult
                                      , BYREF szError AS WSTRINGZ _                        ' __in LPCWSTR szError
                                      ) AS LONG                                            ' HRESULT
                                
                                      END METHOD
                                      ' =====================================================================================
                                      METHOD GetBindInfo ( _                               ' VTable offset = 32
                                        BYREF grfBINDF AS DWORD _                          ' __out DWORD *grfBINDF
                                      , BYREF pbindinfo AS BINDINFO _                      ' __in_out BINDINFO *pbindinfo
                                      ) AS LONG                                            ' HRESULT
                                
                                      END METHOD
                                      ' =====================================================================================
                                      METHOD OnDataAvailable ( _                           ' VTable offset = 36
                                        BYVAL grfBSCF AS DWORD _                           ' __in DWORD grfBSCF
                                      , BYVAL dwSize AS DWORD _                            ' __in DWORD dwSize
                                      , BYREF pformatetc AS FORMATETC _                    ' __in FORMATETC *pformatetc
                                      , BYREF pstgmed AS STGMEDIUM _                       ' __in STGMEDIUM *pstgmed
                                      ) AS LONG                                            ' HRESULT
                                      END METHOD
                                      ' =====================================================================================
                                      METHOD OnObjectAvailable ( _                         ' VTable offset = 40
                                        BYREF riid AS GUID _                               ' __in REFIID riid
                                      , BYVAL punk AS IUNKNOWN _                           ' __in IUnknown *punk
                                      ) AS LONG                                            ' HRESULT
                                
                                      END METHOD
                                      ' =====================================================================================
                                
                                   END INTERFACE
                                
                                END CLASS

                                Comment


                                • #17
                                  Thanks Mike! I'm already using URLDownloadToFile as a backup alternative if the WinHttpRequest fails for some reason. URLDownloadToFile works, but it's slow when downloading hundreds of web sites, and it has a tendency to hang on some sites. Often the only way to cancel is to kill the application in task manager if it doesn't time out.
                                  Anthony Watson, Mountain Software
                                  www.mountainsoftware.com

                                  Comment


                                  • #18
                                    Wow. Didn't know that.

                                    Comment


                                    • #19
                                      WinHttpRequest Redirect to Secure Pages Fails?

                                      16 Nov 2021, 05:45 PM
                                      I need a reliable way to download web pages and image files from the internet. I've been using URLDownloadToFile, but there's no easy way to set a timeout and it often hangs indefinitely. It's also quite slow.

                                      I have tried a number of options, including the examples on Jose's page at:
                                      http://www.jose.it-berater.org/smffo...p?topic=4368.0

                                      I adapted one of his examples to make transition to my own program easier, and it generally works well on both http and https web pages. However, if an unsecured http page redirects to a secured https page, it fails. Any ideas how to fix this?

                                      if URLDownloadToFile is too slow, I'm curious how many threads are used?
                                      The code below correctly handles over 63-threads (just in case that is a problem.)

                                      Downloading some sites are always the same size (as you know.)

                                      This site mentioned is always a different size, but no error is ever returned
                                      https://www.cdkitchen.com/recipes/re...das75972.shtml
                                      This code names files as junk1.htm, junk2.htm, etc ...

                                      This isn't what you want, but something like SocketTools may be the solution or help you.

                                      Help needed.
                                      Code:
                                      #DIM ALL
                                      #INCLUDE "win32api.inc"
                                      #INCLUDE "WinInet.inc"
                                      GLOBAL gsURL(),gsFileName() AS STRING
                                      FUNCTION PBMAIN AS LONG
                                       LOCAL x,threads AS LONG
                                       'sUrl = "https://www.cdkitchen.com/recipes/recs/1129/Absolutely-The-Best-Enchiladas75972.shtml"
                                       threads = 9
                                       REDIM gsURL(threads)
                                       REDIM gsFileName(threads)
                                       REDIM hThread(1 TO threads) AS LONG
                                       FOR x = 1 TO threads
                                        gsURl(x) = "https://www.powerbasic.com"   'URL
                                        gsFileName(x) = USING$("junk#.htm",x)     'save as
                                        THREAD CREATE MyThread(x) TO hThread(x)
                                       NEXT
                                       ThreadWait hThread()
                                       ? "Done",%MB_SYSTEMMODAL
                                      END FUNCTION
                                      
                                      THREAD FUNCTION MyThread(BYVAL x AS LONG) AS LONG
                                       DownloadFile x
                                      END FUNCTION
                                      
                                      FUNCTION DownloadFile(x AS LONG) AS LONG
                                        LOCAL errCode AS LONG
                                        KILL gsFilename(x)
                                        ERRCLEAR
                                        DeleteURLCacheEntry(gsURL(x) + $NUL)  '1 = success  clear the cache
                                        errcode = URLDownloadToFile(BYVAL 0,gsUrl(x)+$NUL, gsFileName(x)+$NUL,0,0)
                                        IF errcode THEN ? "Download error" + STR$(errcode):FUNCTION = errCode
                                      END FUNCTION
                                      
                                      FUNCTION ThreadWait(hThread() AS LONG) AS LONG
                                       LOCAL ret,StartIndex,ThreadRemain,CurrentGroup,maxthread AS LONG
                                       maxthread = ARRAYATTR(hThread(),4)  'waitformultiplethreads.bas 11/13/21
                                       StartIndex   = 1
                                       ThreadRemain = MaxThread
                                       DO
                                        CurrentGroup = MIN(%MAXIMUM_WAIT_OBJECTS, ThreadRemain)
                                        ret = WaitForMultipleObjects(CurrentGroup,BYVAL VARPTR(hThread(StartIndex)),%TRUE,%INFINITE)
                                        StartIndex   = StartIndex   + %MAXIMUM_WAIT_OBJECTS
                                        ThreadRemain = ThreadRemain - %MAXIMUM_WAIT_OBJECTS
                                       LOOP WHILE ThreadRemain > 0
                                      END FUNCTION

                                      Comment


                                      • #20
                                        Originally posted by Mike Doty View Post
                                        if URLDownloadToFile is too slow, I'm curious how many threads are used?
                                        I currently have about 200 threads in my main application for the download operations. URLDownloadToFile is mostly sitting around waiting on the server and whatnot, so it lets the other sites continue downloading while each thread waits. I didn't just dream up 200 threads, I ran speed tests slowly increasing the number of threads until it stopped making much difference. 200 seemed to be a good balance point in my situation.

                                        Honestly, the URLDownloadToFile speed wouldn't bother me that much, especially when using multiple threads. Unfortunately, the timeout period seems to be fixed and depending on the site can get stuck waiting.

                                        WinHttpRequest is nice because it's fast, I can define the timeout times, and it doesn't seem to get stuck on a web site. Unfortunately, it seems to fail with a variety of sites for various reasons.
                                        Anthony Watson, Mountain Software
                                        www.mountainsoftware.com

                                        Comment

                                        Working...
                                        X