Announcement

Collapse

New Sub-Forum

In an effort to help make sure there are appropriate categories for topics of discussion that are happening, there is now a sub-forum for databases and database programming under Special Interest groups. Please direct questions, etc., about this topic to that sub-forum moving forward. Thank you.
See more
See less

Huge arrays and quad integer I need

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Huge arrays and quad integer I need

    Using DOS 6.22, PB for DOS 3.5
    I can not DIM a HUGE number 5 dimentional array correctly as I
    always get a floating point array with low & high values. I
    thought all arrays were zero out at run time & another thing I
    really need 4 huge or virual arrays & only can get one at a time
    which stinks.

    If there is a place where there is more complete info on these
    commands please point the way as I do not mind reading its just
    the books do not give many examples & not much info.

    Bob


    ------------------

  • #2
    If you have really huge arrays and PB/DOS 3.5, I suggest you look at VIRTUAL arrays.

    How big are these arrays? (Show us the DIM statement).

    You may be running up against the hard limits of MS-DOS, namely 640Kb conventional (regular or HUGE arrays) or 16Mb (or is it 32Mb?) of EMS for use in VIRTUAL arrays.



    ------------------
    Michael Mattias
    Racine WI USA
    [email protected]
    Michael Mattias
    Tal Systems Inc. (retired)
    Racine WI USA
    [email protected]
    http://www.talsystems.com

    Comment


    • #3
      DIM VIRTUAL repeat5(255,255,255,255,255)
      DIM VIRTUAL repeat4(255,255,255,255)
      DIM VIRTUAL repeat3(255,255,255)
      DIM VIRTUAL repeat2(255,255)

      This was the orginal way that I tried & needed to get going &
      this way would only work if there was one virual statement & took
      over a minute to start the actual program on a P200 MHz computer.
      For the HUGE dimention I had just put in place of VIRUAL with the
      repeat5 line, I did try the AS statement also as QUAD & even made
      repeat a QUAD, also tried repeat5&&(large array in here).
      This is why I am puzzed there is no other info on HUGE dynamic &
      VIRUAL static arrays as I found many limits & problems.
      I switched to huge cause it was faster & now that I can only do
      one array at a time I need a dynamic type integer array to get
      the job done & hope I can save the info somehow & **** it back
      into my program.
      I am looking for the actual repeat of bytes in large files so the
      name repeat & numbered to identified which one I am working on.

      Bob

      ------------------

      Comment


      • #4
        Robert--

        You may need to make a minor adjustment here. Just your first DIM statement alone, if applied as quad integers, would require in excess of eight terabytes of memory to execute successfully. This is somewhat beyond the capabilities of DOS, Windows, and not a few Cray Computers.

        Might be a good idea to go back to those books a bit more...

        Best of luck,

        Bob Zale
        PowerBASIC Inc.


        ------------------

        Comment


        • #5
          I was hoping not to get to deep into the program mode but now it
          looks like I have to. I found the lenght for strings are
          variables stored the same way ? 2 bytes for pointers & the # for
          the value so now 1 integer array = 4 bytes ?.

          I can cut down to integers & start with 4 or 3 dimentional arrays
          but 0 thru 255 must stay & I could attack this a different way
          later on as this is only the basic test to give me info (stats)
          for the real program.

          I read in the Power Basic 3.5 manuals that EMS memory limit is
          16 & 32 meg which one is true ?. I also have a 30 meg ramdrive &
          will upgrade this computer to a 128 or 256 MB & I could upgrade
          to a 2 gig of ram & the ramdrive I use can go this high to.

          I was getting some memory errors but not always & thanks for the
          info now I can move on knowing what the problem is & I can check
          for overflows as I am going along also.

          Thanks again Bob


          ------------------

          Comment


          • #6
            VIRTUAL arrays can be as large as 32Mb each, but the reality is that _most_ EMS memory managers can only provide 16 to 32 Mb of EMS.

            QEMM 8.x reportedly can provide up to 64Mb of EMS, which would give you two 32Mb arrays, or 4 16Mb arrays, etc.

            However, QEMM 8.x also has it's share of problems from the information I've read, especially it's "Stealth" feature.

            Also, it pays to note that some EMS managers can only report a maximum of 16Mb of EMS available to PB/DOS, even if it can actually provide more EMS memory than that. In such cases, FRE(-11) will only report less than 16Mb of EMS free once the free EMS pool has dropped below the 16Mb level.

            Finally, it must be pointed out that buggy EMS/XMS memory managers abound. MS released several such versions and released corrected versions with the same file time/date stamp, making identification of faulty memory managers difficult. Therefore, if you encounter run-time problems when dealing with VIRTUAL arrays (and you have thououghly debugged your code, etc!), the very first suspect is the memory manager being used. For example, on Win3.x platforms, switching from the EMS/XMS memory manager in the \WINDOWS difectory, over to the one installed in the \DOS directory is often a cure.


            ------------------
            Lance
            PowerBASIC Support
            mailto:[email protected][email protected]</A>
            Lance
            mailto:[email protected]

            Comment


            • #7
              Robert -

              Even a single 3-dimensional array of integers, eg:
              Code:
              DIM VIRTUAL Repeat3(255, 255, 255) AS INTEGER
              is going to require the full 32Mb of EMS to contain, as 256 * 256 * 256 * 2 bytes per element = 33,554,432 bytes = 32,768K = 32M. As Lance correctly points out, it's something of a crapshoot as to whether your particular memory manager will supply that much EMS - and, even if you get it to work on yours, there's no guarantee that the resulting program will run on another PC unless it has exactly the same configuration as yours. If you do get away with it, though, then that'll be the only VIRTUAL array you can have, as your EMS will be completely consumed. (Unless you're using QEMM which, as Lance said, can sometimes get you 64Mb of EMS if you configure it right.)

              Adding a fourth dimension, i.e.
              Code:
              DIM VIRTUAL Repeat4(255, 255, 255, 255) AS INTEGER
              swells the memory requirements to 256 * 256 * 256 * 256 * 2 bytes per element = 8,589,934,592 bytes = 8,388,608K = 8,192M = 8 gigabytes of RAM... and under no circumstances will you ever get an array that large in PB/DOS.

              I have to confess, though, I'm both intrigued and baffled by just what you're trying to accomplish here...

              ------------------


              [This message has been edited by Gary Akins (edited February 12, 2001).]

              Comment


              • #8
                I did the single byte math yesterday & already decided on plan C
                is in effect, I will use a large file on a 60 MB ramdrive & I can
                drop the negative value also & use 2 or 3 bytes to get a high
                count that I need on at least a 3 deep array & a higher count on
                the 2 dimentiional arry, yes slow as hell but it should work.
                I am trying a compression idea that I have had for along time,
                that will only work on large files, but I maybe able to run again
                & again on the same file till the file get small, I have a big
                loss so far but I need to look at the stats on repeated bytes so
                I can do the math & figure the best way to attack this part of
                my idea. I have a lost but I have lots of flags free 72 of them
                & if I want to take a bigger loss I can double the flags with
                every bit added. If my idea works I will have this program redone
                with C or assembly for speed, first I need to prove it works.

                The fre(-11) gives me like 32 or 33 MB after I adjusted my emm386
                I have gotten screwie results with the huge & virual commands
                even on the 2 dimentional array only @ integer.
                Like I copied the way sort.bas from the example subdir does the
                array & will not work for me but running the program sort.bas
                works fine I was doing the same thing ?

                Thanks for the math tip, I thought I hade to add on a pointer of 2.

                Bob

                ------------------

                Comment


                • #9
                  Well, you do have a few bytes of array-descriptor overhead to contend with, but that's per array, not per element... I didn't add it in, so as not to confuse the issue unnecessarily. (As the wise man once said, "An ounce of inaccuracy sometimes saves a ton of explanation." )

                  So, basically, you're trying to do a brute-force pattern analysis, looking for sequences of two, three, four, or five bytes that repeat throughout the file? Well... now you know why compression algorithms like Lempel-Ziv generate their dictionaries on the fly, and limit its size, I guess. I don't think re-coding this in 'C' or ASM is going to gain you much speed; it isn't the complexity of the task that's your bottleneck, it's the size of the data set you're trying to work with...

                  ------------------

                  Comment


                  • #10
                    A library would not save me very much in the end as I would have
                    to store the byte pattern pointers then the total count of them.
                    Where as an array type file I will be able to use math as a
                    pointer & only have to store the total count. The four & five
                    bytes was an addon just to look at the stats, I am sure there
                    will not be many of these in random large files.

                    I did study data compression alittle & I have the book DATA
                    Compression by Gilbert Held third edition it is an old book, he
                    has the fifth edition out now, but I am attacking this in a
                    different way & only for large files & I knew this was going to
                    be slow, my goal is to srink a random file down to its smallest
                    size possible by running this program 1 time or 100 times & maybe
                    more if needed.

                    Thanks again for the help
                    Bob

                    ------------------

                    Comment

                    Working...
                    X