Dump URL Contents Directly To String Buffer?

[Jeff Westfall]

Greetings Everyone:

I'm new to PowerBasic for Windows. Up to now I've been a DOS dinosaur writing code in C to run in MS-DOS for real-time data collection and other good stuff out in the field.

I'm also dabbling in the stock market and am working on writing a program to perform technical analysis on various stocks. Obviously, before I can perform the analysis, I must first read in the stock data. Downloading an Excel file from Yahoo! is just the ticket and I've noticed that someone has created a URLDownLoadToFile() function which performs this requirement nicely.

But I would like to skip the step of downloading the file, storing it to disk, then opening it up and performing the parsing. If possible, I would like to dump the URL contents directly to a string buffer and parse it immediately without first storing the contents in a disk file.

Has anyone experimented with or developed a URLDownLoadToString() function? If so, I would certainly like to hear from you!

Thanks.

[Stuart McLachlan]

Greetings Everyone:

But I would like to skip the step of downloading the file, storing it to disk, then opening it up and performing the parsing. If possible, I would like to dump the URL contents directly to a string buffer and parse it immediately without first storing the contents in a disk file.

Has anyone experimented with or developed a URLDownLoadToString() function? If so, I would certainly like to hear from you!

Thanks.

Take a look at the thread

http://www.powerbasic.com/support/pbforums/showthread.php?t=41132

The code I posted there does exactly that.

[Michael Mattias]

But I would like to skip the step of downloading the file, storing it to disk, then opening it up and performing the parsing. If possible, I would like to dump the URL contents directly to a string buffer and parse it immediately without first storing the contents in a disk file.

Just as a discussion point... why? Loading a disk file's contents into a string buffer is not terribly difficult; nor is KILLing the disk file when you no longer need it.

Enquiring Minds Want to Know.....

MCM

[Jeff Westfall]

The software I'm writing runs on a Thin Client computer which has a solid state disk, and this disk doesn't offer the same number of write cycles as a conventional hard drive does.

With the software potentially analyzing hundreds (or more) stocks per day I want to minimize the number of writes to avoid "wearing out" the sectors. Reading the URL data directly into a string will avoid this problem.

Another possibility would be to enable a ramdrive in memory to emulate a hard drive. But before I resort to the open file, download data, save file, close file, reopen file, extract data, close file cycle I want to try a direct memory download first.

Stuart posted an interesting suggestion that I'm going to try out.

Thanks.

[Edwin Knoppert]

Pointless, Windows reads and writes much more things constantly.
Wearing out should not be a programmers concern.

[Jeff Westfall]

True, Windows does perform many write operations to the hard drive. The only way I know to completely stop it is to turn the machine off.

But, as a programmer I must be mindful of the limitations of the underlying hardware. Hence, I can make some adjustments to minimize the hard drive writing by disabling the paging file and in this case read data directly into string memory. My code will perform write operations from time to time but it will do this sparingly.

BTW, I've taken Stuart McLachlan's idea and tweaked it a little bit and it works perfectly. I now have the equivalent of a URLDownLoadToString() function as I originally wanted.

This problem is now solved and many thanks to Stuart for pointing me in the right direction.

Cheers.

[Michael Mattias]

BTW, I've taken Stuart McLachlan's idea and tweaked it a little bit ....

...of a URLDownLoadToString() function....
... thanks to Stuart for pointing me in the right direction

I would think a more appropriate 'thank you' would be for you to post what you developed from the work shared by Mr. McLachan; perhaps as a 'reply' to the original post, so related code can be found in one place.

MCM

[Michael Burns]

Any flash drive (or SSD) has wear leveling built into its internal controller. Wearing out the drive is just not a realistic concern. With or without wear leveling, the actual data writing of a file is not the limiting issue anyway - the OS updating the FAT is and that happens constantly by the OS just running. Wear leveling extends the "life" of the flash by roughly 5 orders of magnitude versus not having wear leveling. Basically, any flash drive you own would wear out in a day or so if its built in controller did not perform wear leveling because the area the FAT resides, if it were in a fixed physical location, would wear out that quickly.

There are a lot of good papers on flash wear leveling, but a very short (but dated) application note illustrating how wear leveling gives flash usable lifetime (usually a couple of decades under constant use) can be found here:
http://www.numonyx.com/Documents/App...tes/AN1822.pdf