Announcement

Collapse
No announcement yet.

Dump URL Contents Directly To String Buffer?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Dump URL Contents Directly To String Buffer?

    Greetings Everyone:

    I'm new to PowerBasic for Windows. Up to now I've been a DOS dinosaur writing code in C to run in MS-DOS for real-time data collection and other good stuff out in the field.

    I'm also dabbling in the stock market and am working on writing a program to perform technical analysis on various stocks. Obviously, before I can perform the analysis, I must first read in the stock data. Downloading an Excel file from Yahoo! is just the ticket and I've noticed that someone has created a URLDownLoadToFile() function which performs this requirement nicely.

    But I would like to skip the step of downloading the file, storing it to disk, then opening it up and performing the parsing. If possible, I would like to dump the URL contents directly to a string buffer and parse it immediately without first storing the contents in a disk file.

    Has anyone experimented with or developed a URLDownLoadToString() function? If so, I would certainly like to hear from you!

    Thanks.

  • #2
    Originally posted by Jeff Westfall View Post
    Greetings Everyone:

    But I would like to skip the step of downloading the file, storing it to disk, then opening it up and performing the parsing. If possible, I would like to dump the URL contents directly to a string buffer and parse it immediately without first storing the contents in a disk file.

    Has anyone experimented with or developed a URLDownLoadToString() function? If so, I would certainly like to hear from you!

    Thanks.
    Take a look at the thread
    http://www.powerbasic.com/support/pb...ad.php?t=41132

    The code I posted there does exactly that.

    Comment


    • #3
      But I would like to skip the step of downloading the file, storing it to disk, then opening it up and performing the parsing. If possible, I would like to dump the URL contents directly to a string buffer and parse it immediately without first storing the contents in a disk file.
      Just as a discussion point... why? Loading a disk file's contents into a string buffer is not terribly difficult; nor is KILLing the disk file when you no longer need it.

      Enquiring Minds Want to Know.....

      MCM
      Michael Mattias
      Tal Systems (retired)
      Port Washington WI USA
      [email protected]
      http://www.talsystems.com

      Comment


      • #4
        The software I'm writing runs on a Thin Client computer which has a solid state disk, and this disk doesn't offer the same number of write cycles as a conventional hard drive does.

        With the software potentially analyzing hundreds (or more) stocks per day I want to minimize the number of writes to avoid "wearing out" the sectors. Reading the URL data directly into a string will avoid this problem.

        Another possibility would be to enable a ramdrive in memory to emulate a hard drive. But before I resort to the open file, download data, save file, close file, reopen file, extract data, close file cycle I want to try a direct memory download first.

        Stuart posted an interesting suggestion that I'm going to try out.

        Thanks.

        Comment


        • #5
          Pointless, Windows reads and writes much more things constantly.
          Wearing out should not be a programmers concern.
          hellobasic

          Comment


          • #6
            True, Windows does perform many write operations to the hard drive. The only way I know to completely stop it is to turn the machine off.

            But, as a programmer I must be mindful of the limitations of the underlying hardware. Hence, I can make some adjustments to minimize the hard drive writing by disabling the paging file and in this case read data directly into string memory. My code will perform write operations from time to time but it will do this sparingly.

            BTW, I've taken Stuart McLachlan's idea and tweaked it a little bit and it works perfectly. I now have the equivalent of a URLDownLoadToString() function as I originally wanted.

            This problem is now solved and many thanks to Stuart for pointing me in the right direction.

            Cheers.

            Comment


            • #7
              BTW, I've taken Stuart McLachlan's idea and tweaked it a little bit ....

              ...of a URLDownLoadToString() function....
              ... thanks to Stuart for pointing me in the right direction
              I would think a more appropriate 'thank you' would be for you to post what you developed from the work shared by Mr. McLachan; perhaps as a 'reply' to the original post, so related code can be found in one place.

              MCM
              Michael Mattias
              Tal Systems (retired)
              Port Washington WI USA
              [email protected]
              http://www.talsystems.com

              Comment


              • #8
                Any flash drive (or SSD) has wear leveling built into its internal controller. Wearing out the drive is just not a realistic concern. With or without wear leveling, the actual data writing of a file is not the limiting issue anyway - the OS updating the FAT is and that happens constantly by the OS just running. Wear leveling extends the "life" of the flash by roughly 5 orders of magnitude versus not having wear leveling. Basically, any flash drive you own would wear out in a day or so if its built in controller did not perform wear leveling because the area the FAT resides, if it were in a fixed physical location, would wear out that quickly.

                There are a lot of good papers on flash wear leveling, but a very short (but dated) application note illustrating how wear leveling gives flash usable lifetime (usually a couple of decades under constant use) can be found here:
                http://www.numonyx.com/Documents/App...tes/AN1822.pdf
                Last edited by Michael Burns; 31 Aug 2009, 01:56 PM. Reason: minor tweek
                Michael Burns

                Comment

                Working...
                X