Announcement

Collapse
No announcement yet.

Tcp open

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Tcp open

    The following page is publicly available free of charge:
    http://finance.yahoo.com/q/hp?s=%5ED...e=5&f=2009&g=d

    It is sometimes written as
    http://finance.yahoo.com/q/hp?s=%^DJ...e=5&f=2009&g=d

    I've been trying to access it with
    TCP OPEN and then TCP RECV but can't get the OPEN to work.

    Please, does anyone have any idea how to do this ?

  • #2
    I've been trying to access it with TCP OPEN and then TCP RECV but can't get the OPEN to work.
    Allegedly failing code used?


    Error code from same?
    Michael Mattias
    Tal Systems (retired)
    Port Washington WI USA
    [email protected]
    http://www.talsystems.com

    Comment


    • #3
      Originally posted by Michael Mattias View Post
      Allegedly failing code used?


      Error code from same?
      I've forgotten how to include code in Forum.
      Please point me in right direction.

      Comment


      • #4
        Think I've managed now to send code as follows

        Code:
        #COMPILE EXE
        #DIM ALL
        FUNCTION PBMAIN () AS LONG
            LOCAL fi AS LONG
            LOCAL Entire_page AS STRING
            LOCAL yahooaddr AS STRING
            LOCAL yahoopage AS STRING
            LOCAL buffer AS STRING
            yahooaddr = "[URL]http://finance.yahoo.com[/URL]"
            yahoopage = "[URL]http://finance.yahoo.com/table.csv?s=^DJI&a=07&b=20&c=2009&d=08&e=5&f=2009%g=d[/URL]"
            fi = FREEFILE
            ' Connecting...
            TCP OPEN "http" AT yahooaddr AS #fi TIMEOUT 30000 'returns err57 - why?
            ' Could we connect to site?
            IF ERR THEN
                FUNCTION = ERRCLEAR
                EXIT FUNCTION
            END IF
            ' Send the GET request...
            TCP PRINT #fi, "GET " & yahoopage & " HTTP/1.0"
            ' Retrieve the page...
            DO
                TCP RECV #fi, 4096, buffer  'gives err24 whether or not yahoopage starts http://
                Entire_page = Entire_page + buffer
            LOOP WHILE ISTRUE LEN(buffer) AND ISFALSE ERR
            ' Close the TCP/IP port...
            TCP CLOSE #fi
        END FUNCTION

        Comment


        • #5
          Code:
          Not sure how to get pages within yahoo, but here is an example
          #COMPILE EXE
          #DIM ALL
          #INCLUDE "win32api.inc"
          FUNCTION PBMAIN () AS LONG
              LOCAL fi AS LONG
              LOCAL Entire_page AS STRING
              LOCAL yahooaddr AS STRING
              LOCAL yahoopage AS STRING
              LOCAL buffer AS STRING
              REM yahooaddr = "[URL="http://www.yahoo.com"]www.yahoo.com[/URL]"
              REM yahoopage= "[URL]http://finance.yahoo.com/q?s=ibm[/URL]
              yahooaddr = "[URL="http://www.powerbasic.com"]www.powerbasic.com[/URL]"
              yahoopage = "[URL]http://www.powerbasic.com/support/pbforums/showthread.php?t=41414[/URL]
              
              fi = FREEFILE
              ' Connecting...
              TCP OPEN "http" AT yahooaddr AS #fi TIMEOUT 10000 'returns err57 - why?
              
              ' Could we connect to site?
              IF ERR THEN
                ? STR$(ERR)
                  FUNCTION = ERRCLEAR
                  EXIT FUNCTION
              END IF
              ' Send the GET request...
              TCP PRINT #fi, "GET " & yahoopage & " HTTP/1.0"
              'TCP PRINT #fi, "Referer: [URL]http://www.myprogram.com[/URL]"
              'TCP PRINT #fi, "User-Agent: TCP OPEN Example ([URL="http://www.powerbasic.com"]www.powerbasic.com[/URL])
              TCP PRINT #fi, ""
              
              ' Retrieve the page...
              DO
                  TCP RECV #fi, 4096, buffer  'gives err24 whether or not yahoopage starts http://
                  Entire_page = Entire_page + buffer
              LOOP WHILE ISTRUE LEN(buffer) AND ISFALSE ERR
              ' Close the TCP/IP port...
              TCP CLOSE #fi
              KILL "test.htm":ERRCLEAR
              fi = FREEFILE
              OPEN "test.htm" FOR BINARY AS #fi
              '? Entire_Page 'source
              PUT$ #fi,Entire_page
              CLOSE #fi
              SLEEP 50
              ShellExecute (%NULL, "OPEN", "test.htm", BYVAL %NULL, CURDIR$, %SW_SHOWNORMAL)
          END FUNCTION

          Comment


          • #6
            Try:
            yahooaddr = "finance.yahoo.com"

            (drop the "http://")

            Comment


            • #7
              Strip off header
              $CRLF2 = $CRLF + $CRLF
              entire_page = MID$(entire_page,(INSTR(entire_page,$CRLF2) +4))

              Comment


              • #8
                I was going to post some code that would pull the header and body from the result but with a little less complex commands than Mike did but then I started adding other quick and easy statements that can be used to pull apart a URL so I ended up throwing this little demo together instead. The last few statements are just samples so never get executed>

                Code:
                #COMPILE EXE
                #DIM ALL
                
                FUNCTION PBMAIN () AS LONG
                
                    LOCAL sURL                  AS STRING
                    LOCAL sProtocol             AS STRING
                    LOCAL sHost                 AS STRING
                    LOCAL sPath                 AS STRING
                    LOCAL lPort                 AS LONG
                    LOCAL sHTTPRequest          AS STRING
                    LOCAL hSocket               AS DWORD
                    LOCAL sDataFromHTTPServer   AS STRING
                    LOCAL sHeader               AS STRING
                    LOCAL sBody                 AS STRING
                
                    sURL = "http://www.powerbasic.com/support/pbforums/showpost.php?p=322649&postcount=1"
                
                    ' Protocol (ie. HTTP, FTP)
                    sProtocol = UCASE$(EXTRACT$(sURL, "://"))
                    ' Host name or IP address
                    sHost = EXTRACT$(REMAIN$(sURL, "://"), ANY "/?")
                    ' Path and/or query being passed
                    sPath = REMAIN$(sURL, sHost)
                    ' Check to see if a port has been specified
                    IF INSTR(sHost, ":") THEN
                        ' Port specified so pull it
                        lPort = VAL(REMAIN$(sHost, ":"))
                        sHost = EXTRACT$(sHost, ":")
                    ELSE
                        ' Port not specified so use default
                        SELECT CASE sProtocol
                            CASE "HTTP"
                                lPort = 80
                            ' Other CASE statements can be added for other protocols as needed
                        END SELECT
                    END IF
                
                    ' To form an HTTP request from the above you can use this:
                    sHTTPRequest = "& & HTTP/1.1" + $CRLF + "Host: &" + $CRLF    ' Only need one CRLF here if we use TCP PRINT to send it
                
                    MSGBOX "Protocol: " + sProtocol + $CRLF + _
                           "Host: " + sHost + $CRLF + _
                           "Port: " + FORMAT$(lPort) + $CRLF + _
                           "Path: " + sPath + $CRLF + _
                           "HTTP GET request: " + $CRLF + _
                           USING$(sHTTPRequest, "GET", sPath, sHost)
                    EXIT FUNCTION
                
                    ' To open a connection to the server use this:
                    TCP OPEN PORT lPort AT sHost AS hSocket
                
                    ' To initiate getting the page, use this:
                    TCP PRINT hSocket, USING$(sHTTPRequest, "GET", sPath, sHost)
                
                    ' Once you receive the data you can get the header and body separated using these:
                    sHeader = EXTRACT$(sDataFromHTTPServer, $CRLF + $CRLF)
                    sBody = REMAIN$(sDataFromHTTPServer, $CRLF + $CRLF)
                
                END FUNCTION[
                Jeff Blakeney

                Comment


                • #9
                  Thank you, everyone. The following code works.

                  It is mostly Mike Doty's contribution with minor amendments. If you run it as shown below, the downloaded file, yahootest.htm, is shown in Internet Explorer by the shell execute command. But my version of IE complains about scripts and activex controls, and also shows a lot of text at the top of the page which I don't understand. I solved that by changing the downloaded file to yahootest.txt and commenting-out the shell command.

                  Stuart McLachlan was quite right to drop "http://" from the variable yahooaddr. That was probably the biggest fault in my original code.

                  Stripping off the header as suggested by Mike Doty didn't seem to work.

                  Jeff Blakeney's code was too advanced for me, but no doubt very good.

                  The outcome is that I have a downloaded file from which I can extract the stock market prices I want. So far, so good.

                  As usual, the most impressive part of PB is this Forum. Where else can a guy in England ask for help and get a reply from Papua New Guinea in two hours. In my youth that was science fiction stuff.

                  For what it's worth, I still find handling the Internet the hardest part of PB. So much so that I gave up in March 2008 and went back to VB. But here I am again having another go.

                  Thank you everyone.

                  Code:
                  #COMPILE EXE
                  #DIM ALL
                  #INCLUDE "win32api.inc"
                  FUNCTION PBMAIN () AS LONG
                      LOCAL fi AS LONG
                      LOCAL Entire_page AS STRING
                      LOCAL yahooaddr AS STRING
                      LOCAL yahoopage AS STRING
                      LOCAL buffer AS STRING
                      yahooaddr = "finance.yahoo.com"
                      yahoopage= "[URL]http://finance.yahoo.com/q/hp?s=%5EDJI&a=07&b=20&c=2009&d=08&e=5&f=2009&g=d[/URL]"
                   
                      fi = FREEFILE
                      ' Connecting...
                      TCP OPEN "http" AT yahooaddr AS #fi TIMEOUT 10000
                   
                      ' Could we connect to site?
                      IF ERR THEN
                        ? STR$(ERR)
                          FUNCTION = ERRCLEAR
                          EXIT FUNCTION
                      END IF
                   
                      ' Send the GET request...
                      TCP PRINT #fi, "GET " & yahoopage & " HTTP/1.0"
                      'TCP PRINT #fi, "Referer: [URL]http://www.myprogram.com[/URL]"
                      'TCP PRINT #fi, "User-Agent: TCP OPEN Example([URL="http://www.powerbasic.com"]www.powerbasic.com[/URL])
                      TCP PRINT #fi, ""
                   
                      ' Retrieve the page...
                      DO
                          TCP RECV #fi, 4096, buffer
                          Entire_page = Entire_page + buffer
                      LOOP WHILE ISTRUE LEN(buffer) AND ISFALSE ERR
                   
                      ' Close the TCP/IP port...
                      TCP CLOSE #fi
                   
                      KILL "yahootest.htm":ERRCLEAR
                   
                      fi = FREEFILE
                      OPEN "yahootest.htm" FOR BINARY AS #fi
                   
                      '? Entire_Page 'source
                      PUT$ #fi,Entire_page
                      CLOSE #fi
                      SLEEP 50
                      ShellExecute (%NULL, "OPEN", "yahootest.htm", BYVAL %NULL, CURDIR$, %SW_SHOWNORMAL)
                  END FUNCTION

                  Comment


                  • #10
                    Interesting thread..
                    The outcome is that I have a downloaded file from which I can extract the stock market prices I want. So far, so good.
                    Looks like you can request the info in the table more directly with this URL:
                    "http://ichart.finance.yahoo.com/tabl...=d&ignore=.csv"
                    Rgds, Dave

                    Comment


                    • #11
                      Wow! Dave Biggs has just astonished me. For twenty years I have painfully extracted stock market prices by disentangling the htm. Now Dave shows me a simple way to download a clean csv file. Thank you, Dave. As I said, "This Forum is a wonderful thing."

                      Comment


                      • #12
                        I have now modified my code to use Dave Biggs' much better URL. Thank you, Dave.
                        I then began to understand about header and body.
                        So I used Jeff Blakeney's code to remove the header. Thank you, Jeff.
                        Miraculously I have now downloaded a simple csv file which I can read line by line.
                        And the sun is shining, and I love you all. Isn't the world great?

                        Comment


                        • #13
                          Page with URL's?

                          'Thanks Jeff for the parsing methods.
                          'Not sure why REMAIN gave different results?
                          Code:
                          FUNCTION PBMAIN () AS LONG
                           
                            entire_page$ = CHR$("HEADER PORTION",13,10,13,10,"DATA PORTION")
                           
                            ?  MID$(entire_page$,(INSTR(entire_page$,CHR$(13,10,13,10)) +4))
                            ?  REMAIN$(entire_page$, $CRLF + $CRLF) 'same results, here
                           
                          END FUNCTION

                          Looks like you can download what you want using tags:

                          http://www.gummy-stuff.org/Yahoo-data.htm




                          .
                          Last edited by Mike Doty; 7 Sep 2009, 06:58 AM.

                          Comment


                          • #14
                            Originally posted by Dave Biggs View Post
                            Looks like you can request the info in the table more directly with this URL:
                            "http://ichart.finance.yahoo.com/tabl...=d&ignore=.csv"
                            Anyone know if the PB forum software has an option like this? I've been working occasionally on an offline reader for the forums here because I find it really slow to read messages using web based software. Right now, parsing the HTML is a bit of a pain and any changes made to these web forums would break my program until I updated it. This program would also have the added benefit of me being able to keep the messages I find interesting or useful and being able to search locally instead of having to come online to find stuff. Kind of like a POFFS program that gets updated every time you read messages.

                            Originally posted by Mike Doty View Post
                            'Thanks Jeff for the parsing methods.
                            'Not sure why REMAIN gave different results?
                            Code:

                            FUNCTION PBMAIN () AS LONG

                            entire_page$ = CHR$("HEADER PORTION",13,10,13,10,"DATA PORTION")

                            ? MID$(entire_page$,(INSTR(entire_page$,CHR$(13,10,13,10)) +4))
                            ? REMAIN$(entire_page$, $CRLF + $CRLF) 'same results, here

                            END FUNCTION
                            Mike. are you saying that using REMAIN$ didn't give you the same results as your MID$ line or not? You first state that it "gave different results" and then in the code you say "same results, here" so I'm not sure if you had a problem or not. I get the same result from both statements using the code you posted.
                            Jeff Blakeney

                            Comment


                            • #15
                              Ralph said it in post #9
                              "Stripping off the header as suggested by Mike Doty didn't seem to work."

                              Comment


                              • #16
                                Ah, now I understand. Thanks for the clarification.
                                Jeff Blakeney

                                Comment

                                Working...
                                X