Tcp open

[Ralph Brereton]

The following page is publicly available free of charge:
http://finance.yahoo.com/q/hp?s=%5ED...e=5&f=2009&g=d

It is sometimes written as
http://finance.yahoo.com/q/hp?s=%^DJ...e=5&f=2009&g=d

I've been trying to access it with
TCP OPEN and then TCP RECV but can't get the OPEN to work.

Please, does anyone have any idea how to do this ?

[Michael Mattias]

I've been trying to access it with TCP OPEN and then TCP RECV but can't get the OPEN to work.

Allegedly failing code used?


Error code from same?

[Ralph Brereton]

Allegedly failing code used?


Error code from same?

I've forgotten how to include code in Forum.
Please point me in right direction.

[Ralph Brereton]

Think I've managed now to send code as follows

#COMPILE EXE
#DIM ALL
FUNCTION PBMAIN () AS LONG
    LOCAL fi AS LONG
    LOCAL Entire_page AS STRING
    LOCAL yahooaddr AS STRING
    LOCAL yahoopage AS STRING
    LOCAL buffer AS STRING
    yahooaddr = "[URL]http://finance.yahoo.com[/URL]"
    yahoopage = "[URL]http://finance.yahoo.com/table.csv?s=^DJI&a=07&b=20&c=2009&d=08&e=5&f=2009%g=d[/URL]"
    fi = FREEFILE
    ' Connecting...
    TCP OPEN "http" AT yahooaddr AS #fi TIMEOUT 30000 'returns err57 - why?
    ' Could we connect to site?
    IF ERR THEN
        FUNCTION = ERRCLEAR
        EXIT FUNCTION
    END IF
    ' Send the GET request...
    TCP PRINT #fi, "GET " & yahoopage & " HTTP/1.0"
    ' Retrieve the page...
    DO
        TCP RECV #fi, 4096, buffer  'gives err24 whether or not yahoopage starts http://
        Entire_page = Entire_page + buffer
    LOOP WHILE ISTRUE LEN(buffer) AND ISFALSE ERR
    ' Close the TCP/IP port...
    TCP CLOSE #fi
END FUNCTION

[Mike Doty]

Not sure how to get pages within yahoo, but here is an example
#COMPILE EXE
#DIM ALL
#INCLUDE "win32api.inc"
FUNCTION PBMAIN () AS LONG
    LOCAL fi AS LONG
    LOCAL Entire_page AS STRING
    LOCAL yahooaddr AS STRING
    LOCAL yahoopage AS STRING
    LOCAL buffer AS STRING
    REM yahooaddr = "[URL="http://www.yahoo.com"]www.yahoo.com[/URL]"
    REM yahoopage= "[URL]http://finance.yahoo.com/q?s=ibm[/URL]
    yahooaddr = "[URL="http://www.powerbasic.com"]www.powerbasic.com[/URL]"
    yahoopage = "[URL]http://www.powerbasic.com/support/pbforums/showthread.php?t=41414[/URL]
    
    fi = FREEFILE
    ' Connecting...
    TCP OPEN "http" AT yahooaddr AS #fi TIMEOUT 10000 'returns err57 - why?
    
    ' Could we connect to site?
    IF ERR THEN
      ? STR$(ERR)
        FUNCTION = ERRCLEAR
        EXIT FUNCTION
    END IF
    ' Send the GET request...
    TCP PRINT #fi, "GET " & yahoopage & " HTTP/1.0"
    'TCP PRINT #fi, "Referer: [URL]http://www.myprogram.com[/URL]"
    'TCP PRINT #fi, "User-Agent: TCP OPEN Example ([URL="http://www.powerbasic.com"]www.powerbasic.com[/URL])
    TCP PRINT #fi, ""
    
    ' Retrieve the page...
    DO
        TCP RECV #fi, 4096, buffer  'gives err24 whether or not yahoopage starts http://
        Entire_page = Entire_page + buffer
    LOOP WHILE ISTRUE LEN(buffer) AND ISFALSE ERR
    ' Close the TCP/IP port...
    TCP CLOSE #fi
    KILL "test.htm":ERRCLEAR
    fi = FREEFILE
    OPEN "test.htm" FOR BINARY AS #fi
    '? Entire_Page 'source
    PUT$ #fi,Entire_page
    CLOSE #fi
    SLEEP 50
    ShellExecute (%NULL, "OPEN", "test.htm", BYVAL %NULL, CURDIR$, %SW_SHOWNORMAL)
END FUNCTION

[Stuart McLachlan]

Try:
yahooaddr = "finance.yahoo.com"

(drop the "http://")

[Mike Doty]

Strip off header
$CRLF2 = $CRLF + $CRLF
entire_page = MID$(entire_page,(INSTR(entire_page,$CRLF2) +4))

[Jeff Blakeney]

I was going to post some code that would pull the header and body from the result but with a little less complex commands than Mike did but then I started adding other quick and easy statements that can be used to pull apart a URL so I ended up throwing this little demo together instead. The last few statements are just samples so never get executed>

#COMPILE EXE
#DIM ALL

FUNCTION PBMAIN () AS LONG

    LOCAL sURL                  AS STRING
    LOCAL sProtocol             AS STRING
    LOCAL sHost                 AS STRING
    LOCAL sPath                 AS STRING
    LOCAL lPort                 AS LONG
    LOCAL sHTTPRequest          AS STRING
    LOCAL hSocket               AS DWORD
    LOCAL sDataFromHTTPServer   AS STRING
    LOCAL sHeader               AS STRING
    LOCAL sBody                 AS STRING

    sURL = "http://www.powerbasic.com/support/pbforums/showpost.php?p=322649&postcount=1"

    ' Protocol (ie. HTTP, FTP)
    sProtocol = UCASE$(EXTRACT$(sURL, "://"))
    ' Host name or IP address
    sHost = EXTRACT$(REMAIN$(sURL, "://"), ANY "/?")
    ' Path and/or query being passed
    sPath = REMAIN$(sURL, sHost)
    ' Check to see if a port has been specified
    IF INSTR(sHost, ":") THEN
        ' Port specified so pull it
        lPort = VAL(REMAIN$(sHost, ":"))
        sHost = EXTRACT$(sHost, ":")
    ELSE
        ' Port not specified so use default
        SELECT CASE sProtocol
            CASE "HTTP"
                lPort = 80
            ' Other CASE statements can be added for other protocols as needed
        END SELECT
    END IF

    ' To form an HTTP request from the above you can use this:
    sHTTPRequest = "& & HTTP/1.1" + $CRLF + "Host: &" + $CRLF    ' Only need one CRLF here if we use TCP PRINT to send it

    MSGBOX "Protocol: " + sProtocol + $CRLF + _
           "Host: " + sHost + $CRLF + _
           "Port: " + FORMAT$(lPort) + $CRLF + _
           "Path: " + sPath + $CRLF + _
           "HTTP GET request: " + $CRLF + _
           USING$(sHTTPRequest, "GET", sPath, sHost)
    EXIT FUNCTION

    ' To open a connection to the server use this:
    TCP OPEN PORT lPort AT sHost AS hSocket

    ' To initiate getting the page, use this:
    TCP PRINT hSocket, USING$(sHTTPRequest, "GET", sPath, sHost)

    ' Once you receive the data you can get the header and body separated using these:
    sHeader = EXTRACT$(sDataFromHTTPServer, $CRLF + $CRLF)
    sBody = REMAIN$(sDataFromHTTPServer, $CRLF + $CRLF)

END FUNCTION[

[Ralph Brereton]

Thank you, everyone. The following code works.

It is mostly Mike Doty's contribution with minor amendments. If you run it as shown below, the downloaded file, yahootest.htm, is shown in Internet Explorer by the shell execute command. But my version of IE complains about scripts and activex controls, and also shows a lot of text at the top of the page which I don't understand. I solved that by changing the downloaded file to yahootest.txt and commenting-out the shell command.

Stuart McLachlan was quite right to drop "http://" from the variable yahooaddr. That was probably the biggest fault in my original code.

Stripping off the header as suggested by Mike Doty didn't seem to work.

Jeff Blakeney's code was too advanced for me, but no doubt very good.

The outcome is that I have a downloaded file from which I can extract the stock market prices I want. So far, so good.

As usual, the most impressive part of PB is this Forum. Where else can a guy in England ask for help and get a reply from Papua New Guinea in two hours. In my youth that was science fiction stuff.

For what it's worth, I still find handling the Internet the hardest part of PB. So much so that I gave up in March 2008 and went back to VB. But here I am again having another go.

Thank you everyone.

#COMPILE EXE
#DIM ALL
#INCLUDE "win32api.inc"
FUNCTION PBMAIN () AS LONG
    LOCAL fi AS LONG
    LOCAL Entire_page AS STRING
    LOCAL yahooaddr AS STRING
    LOCAL yahoopage AS STRING
    LOCAL buffer AS STRING
    yahooaddr = "finance.yahoo.com"
    yahoopage= "[URL]http://finance.yahoo.com/q/hp?s=%5EDJI&a=07&b=20&c=2009&d=08&e=5&f=2009&g=d[/URL]"
 
    fi = FREEFILE
    ' Connecting...
    TCP OPEN "http" AT yahooaddr AS #fi TIMEOUT 10000
 
    ' Could we connect to site?
    IF ERR THEN
      ? STR$(ERR)
        FUNCTION = ERRCLEAR
        EXIT FUNCTION
    END IF
 
    ' Send the GET request...
    TCP PRINT #fi, "GET " & yahoopage & " HTTP/1.0"
    'TCP PRINT #fi, "Referer: [URL]http://www.myprogram.com[/URL]"
    'TCP PRINT #fi, "User-Agent: TCP OPEN Example([URL="http://www.powerbasic.com"]www.powerbasic.com[/URL])
    TCP PRINT #fi, ""
 
    ' Retrieve the page...
    DO
        TCP RECV #fi, 4096, buffer
        Entire_page = Entire_page + buffer
    LOOP WHILE ISTRUE LEN(buffer) AND ISFALSE ERR
 
    ' Close the TCP/IP port...
    TCP CLOSE #fi
 
    KILL "yahootest.htm":ERRCLEAR
 
    fi = FREEFILE
    OPEN "yahootest.htm" FOR BINARY AS #fi
 
    '? Entire_Page 'source
    PUT$ #fi,Entire_page
    CLOSE #fi
    SLEEP 50
    ShellExecute (%NULL, "OPEN", "yahootest.htm", BYVAL %NULL, CURDIR$, %SW_SHOWNORMAL)
END FUNCTION

[Dave Biggs]

Interesting thread..

The outcome is that I have a downloaded file from which I can extract the stock market prices I want. So far, so good.

Looks like you can request the info in the table more directly with this URL:
"http://ichart.finance.yahoo.com/tabl...=d&ignore=.csv"

[Ralph Brereton]

Wow! Dave Biggs has just astonished me. For twenty years I have painfully extracted stock market prices by disentangling the htm. Now Dave shows me a simple way to download a clean csv file. Thank you, Dave. As I said, "This Forum is a wonderful thing."

[Ralph Brereton]

I have now modified my code to use Dave Biggs' much better URL. Thank you, Dave.
I then began to understand about header and body.
So I used Jeff Blakeney's code to remove the header. Thank you, Jeff.
Miraculously I have now downloaded a simple csv file which I can read line by line.
And the sun is shining, and I love you all. Isn't the world great?

[Mike Doty]

Page with URL's?

'Thanks Jeff for the parsing methods.
'Not sure why REMAIN gave different results?

FUNCTION PBMAIN () AS LONG
 
  entire_page$ = CHR$("HEADER PORTION",13,10,13,10,"DATA PORTION")
 
  ?  MID$(entire_page$,(INSTR(entire_page$,CHR$(13,10,13,10)) +4))
  ?  REMAIN$(entire_page$, $CRLF + $CRLF) 'same results, here
 
END FUNCTION

Looks like you can download what you want using tags:

http://www.gummy-stuff.org/Yahoo-data.htm




.

[Jeff Blakeney]

Looks like you can request the info in the table more directly with this URL:
"http://ichart.finance.yahoo.com/tabl...=d&ignore=.csv"

Anyone know if the PB forum software has an option like this? I've been working occasionally on an offline reader for the forums here because I find it really slow to read messages using web based software. Right now, parsing the HTML is a bit of a pain and any changes made to these web forums would break my program until I updated it. This program would also have the added benefit of me being able to keep the messages I find interesting or useful and being able to search locally instead of having to come online to find stuff. Kind of like a POFFS program that gets updated every time you read messages.

'Thanks Jeff for the parsing methods.
'Not sure why REMAIN gave different results?
Code:

FUNCTION PBMAIN () AS LONG

entire_page$ = CHR$("HEADER PORTION",13,10,13,10,"DATA PORTION")

? MID$(entire_page$,(INSTR(entire_page$,CHR$(13,10,13,10)) +4))
? REMAIN$(entire_page$, $CRLF + $CRLF) 'same results, here

END FUNCTION

Mike. are you saying that using REMAIN$ didn't give you the same results as your MID$ line or not? You first state that it "gave different results" and then in the code you say "same results, here" so I'm not sure if you had a problem or not. I get the same result from both statements using the code you posted.

[Mike Doty]

Ralph said it in post #9
"Stripping off the header as suggested by Mike Doty didn't seem to work."

[Jeff Blakeney]

Ah, now I understand. Thanks for the clarification.