Web Access

[Kerry Farmer]

I would like to write a program to read a named web site, navigate around it, accept data from it and maybe even enter responses.

I have never programmed this sort of application before.

I have looked at the PB samples but they are very full of jargon and I am not even sure what they are trying to do.

I use PBCC

I feel totally confident of manipulating the data once I get access to it.

Is this exercise practical for a programmer of my ability?

Can anyone suggest how I can get started on this course of action please? Please keep the answer simple (if at all possible).

Thank you.

[John Montenigro]

Depending on what you mean (in the details) by "navigate around it, accept data from it and maybe even enter responses", I imagine you'd be creating a non-GUI web-browser... ???

If you are able to experiment in PBWIN, search the forum for "minibrowser" code, created and posted by Jose Roca (I think?)

"Is this exercise practical for a programmer of my ability?"

IMHO, I think it's more a question of how determined you are. You build your skills and abilities by trying stuff out and then determining why it didn't work the way you first thought it should!

If it's any consolation to you, I'm nearing completion on a little utility that I started over a year ago. To get it to do everything (OK, most of) what I wanted, I've had to dig into and learn about Windows features and PB etc. I build little standalone experiments, and when I've got that feature working, I add it to the app. Gradually, it has taken shape. It's obviously not a professional's product, but I'm happy with it.

It's taken me through topics such as string data, icons, and sounds in the Resource file, to managing an INI file, to saving and restoring windows positions, to hot keys, and more... I've had a blast (when I haven't been pulling my hair out), and I've learned some pretty cool stuff.

Also, I save my source files as a series of "MyApp _BAKnn.bas", and I periodically go back and see why something I tried earlier didn't work, and how many ways I had to try something before I got it to work. Pretty interesting stuff.

I'd suggest you start with a really specific specification for what you want the program to do, under what conditions, with what user and system inputs, and what your valid data should look like.

Then after some structural/organizational decisions, get coding! Do your research as you hit problems. Come to the forums and ask questions. Keep To-Do list - constantly review it, add to it, check things off as you complete.

Be very generous in comenting your code - I start by writing a detailed bulleted list of what the module is going to do. Then I copy it, insert blank lines between the bullets of the copied block, then under each point, I insert code that accomplishes that step. I also capture all my thoughts and concerns while working out a problem, then move to another copy of the file and clean out the comments that are no longer needed.

Remember to stop once in awhile for food and drink!

[Brad D Byrne]

Kerry, search forums for CGI, especially Joe Byrne's tutorial's

http://www.powerbasic.com/support/pb...&highlight=cgi

and, http://www.basicrocks.com/

[Brad D Byrne]

also, if you need help with client side languages (html, javascript, DOM, etc),

http://www.w3schools.com/

check out the examples, they are interactive so you can change them and test them out

[John Petty]

Kerry
Getting the web page is normally a very simple matter

 
LOCAL fi AS LONG
LOCAL t AS STRING 
LOCAL Entire_page AS STRING  
    fi = FREEFILE
    t ="http://full name of desired page"  'ie http://www.website.com.au/public/latest.aspx
      ' Connecting...
    TCP OPEN PORT 80 AT "url of website" AS fi TIMEOUT 30000
      'TCP OPEN PORT 80 AT host$ AS [#] fNum& [TIMEOUT timeoutval&]
      ' Could we connect to site?
    IF ERR THEN
          FUNCTION = ERRCLEAR
          EXIT FUNCTION
    END IF
      ' Send the GET request...
    TCP PRINT #fi, "GET " & t & " HTTP/1.0"

    TCP PRINT #fi, "Referer: http://url of website again"
    TCP PRINT #fi, "User-Agent: My Grabber
    TCP PRINT #fi, ""

      ' Retrieve the page...
    DO
        TCP RECV #fi, 4096, t
        Entire_page = Entire_page + t
    LOOP WHILE ISTRUE LEN(t) AND ISFALSE ERR

      ' Close the TCP/IP port...
    TCP CLOSE #fi

This will give you the web page as a string in Entire_page. To extract information from it requires some basic HTML knowledge which is not a difficult language.
Sending information back can be a little more difficult depending on whether the website is using things like in memory cookies. My first step is always to grab the page, save it to a text file then read it and decide how I will handle it after that.
John

[Kerry Farmer]

Thanks everybody

I really appreciate all the ideas.

[Kerry Farmer]

It works!

[Kerry Farmer]

I just changed the sample code (see below) a little bit to clarify what is meant exactly by the name of the page (ok it is obvious, but it fooled me for a quite a while)

And I added code to actually write out a file (in the real world I always check for err after every file instruction).

And it works for "www" web sites such as "www.powerbasic.com"

But it does not work for websites which have no "www". I get error 57 and I have tried lots of variations and simple eexamples (actually sometimes I get error 53)

Any thoughts?

Thanks

#COMPILE EXE
#DIM ALL

FUNCTION PBMAIN () AS LONG

LOCAL fi AS LONG
LOCAL t AS STRING
LOCAL Entire_page AS STRING
LOCAL webname AS STRING
KILL "webpagedata"
webname = "www.powerbasic.com"
fi = FREEFILE
t ="http://"+webname
' Connecting...
TCP OPEN PORT 80 AT webname AS fi TIMEOUT 30000
'TCP OPEN PORT 80 AT host$ AS [#] fNum& [TIMEOUT timeoutval&]
' Could we connect to site?
IF ERR THEN
FUNCTION = ERRCLEAR
EXIT FUNCTION
END IF
' Send the GET request...
TCP PRINT #fi, "GET " & t & " HTTP/1.0"

TCP PRINT #fi, t

TCP PRINT #fi, "User-Agent: My Grabber

TCP PRINT #fi, ""


' Retrieve the page...
DO
TCP RECV #fi, 4096, t
'
Entire_page = Entire_page + t
LOOP WHILE ISTRUE LEN(t) AND ISFALSE ERR

' Close the TCP/IP port...
TCP CLOSE #fi


OPEN "webpagedata" FOR OUTPUT AS 3 'write page out to file
PRINT #3, Entire_page
CLOSE 3

END FUNCTION

[John Petty]

Kerry
First a couple of comments. The simple code I posted is basically straight out of the TCP OPEN in the help file. I note you have left out the word Referer: in your second TCP PRINT, whilst quoting a referrer is optional many websites require it if you are going to any page other than the default. I hope you understand what the name of the page exactly means ie if you wished to download this thread then it would be http://www.powerbasic.com/support/pb...ad.php?t=40029. A good list of the simple request headers can be found here http://www.w3.org/Protocols/HTTP/HTRQ_Headers.html#z14

To your specific question. Can you view these sites with a normal browser like IE? Error 57 is no connection, some companies run sites without the www which are not meant for public viewing or maybe Unix system where the site does not run under administrative privilages. If you can view it in IE then try connecting to Port 8080 instead of 80.
You should also PING the site and see if it returns an IP address, if it does then try using the dotted IP (as a string) instead of the URL in the TCP OPEN.
John