Huge arrays and quad integer I need

[Robert Jennett]

Using DOS 6.22, PB for DOS 3.5
I can not DIM a HUGE number 5 dimentional array correctly as I
always get a floating point array with low & high values. I
thought all arrays were zero out at run time & another thing I
really need 4 huge or virual arrays & only can get one at a time
which stinks.

If there is a place where there is more complete info on these
commands please point the way as I do not mind reading its just
the books do not give many examples & not much info.

Bob


------------------

[Michael Mattias]

If you have really huge arrays and PB/DOS 3.5, I suggest you look at VIRTUAL arrays.

How big are these arrays? (Show us the DIM statement).

You may be running up against the hard limits of MS-DOS, namely 640Kb conventional (regular or HUGE arrays) or 16Mb (or is it 32Mb?) of EMS for use in VIRTUAL arrays.



------------------
Michael Mattias
Racine WI USA
[email protected]

[Robert Jennett]

DIM VIRTUAL repeat5(255,255,255,255,255)
DIM VIRTUAL repeat4(255,255,255,255)
DIM VIRTUAL repeat3(255,255,255)
DIM VIRTUAL repeat2(255,255)

This was the orginal way that I tried & needed to get going &
this way would only work if there was one virual statement & took
over a minute to start the actual program on a P200 MHz computer.
For the HUGE dimention I had just put in place of VIRUAL with the
repeat5 line, I did try the AS statement also as QUAD & even made
repeat a QUAD, also tried repeat5&&(large array in here).
This is why I am puzzed there is no other info on HUGE dynamic &
VIRUAL static arrays as I found many limits & problems.
I switched to huge cause it was faster & now that I can only do
one array at a time I need a dynamic type integer array to get
the job done & hope I can save the info somehow & **** it back
into my program.
I am looking for the actual repeat of bytes in large files so the
name repeat & numbered to identified which one I am working on.

Bob

------------------

[Bob Zale]

Robert--

You may need to make a minor adjustment here. Just your first DIM statement alone, if applied as quad integers, would require in excess of eight terabytes of memory to execute successfully. This is somewhat beyond the capabilities of DOS, Windows, and not a few Cray Computers.

Might be a good idea to go back to those books a bit more...

Best of luck,

Bob Zale
PowerBASIC Inc.


------------------

[Robert Jennett]

I was hoping not to get to deep into the program mode but now it
looks like I have to. I found the lenght for strings are
variables stored the same way ? 2 bytes for pointers & the # for
the value so now 1 integer array = 4 bytes ?.

I can cut down to integers & start with 4 or 3 dimentional arrays
but 0 thru 255 must stay & I could attack this a different way
later on as this is only the basic test to give me info (stats)
for the real program.

I read in the Power Basic 3.5 manuals that EMS memory limit is
16 & 32 meg which one is true ?. I also have a 30 meg ramdrive &
will upgrade this computer to a 128 or 256 MB & I could upgrade
to a 2 gig of ram & the ramdrive I use can go this high to.

I was getting some memory errors but not always & thanks for the
info now I can move on knowing what the problem is & I can check
for overflows as I am going along also.

Thanks again Bob


------------------

[Lance Edmonds]

VIRTUAL arrays can be as large as 32Mb each, but the reality is that _most_ EMS memory managers can only provide 16 to 32 Mb of EMS.

QEMM 8.x reportedly can provide up to 64Mb of EMS, which would give you two 32Mb arrays, or 4 16Mb arrays, etc.

However, QEMM 8.x also has it's share of problems from the information I've read, especially it's "Stealth" feature.

Also, it pays to note that some EMS managers can only report a maximum of 16Mb of EMS available to PB/DOS, even if it can actually provide more EMS memory than that. In such cases, FRE(-11) will only report less than 16Mb of EMS free once the free EMS pool has dropped below the 16Mb level.

Finally, it must be pointed out that buggy EMS/XMS memory managers abound. MS released several such versions and released corrected versions with the same file time/date stamp, making identification of faulty memory managers difficult. Therefore, if you encounter run-time problems when dealing with VIRTUAL arrays (and you have thououghly debugged your code, etc!), the very first suspect is the memory manager being used. For example, on Win3.x platforms, switching from the EMS/XMS memory manager in the \WINDOWS difectory, over to the one installed in the \DOS directory is often a cure.


------------------
Lance
PowerBASIC Support
mailto:[email protected][email protected]</A>

[Gary Akins]

Robert -

Even a single 3-dimensional array of integers, eg:

DIM VIRTUAL Repeat3(255, 255, 255) AS INTEGER

is going to require the full 32Mb of EMS to contain, as 256 * 256 * 256 * 2 bytes per element = 33,554,432 bytes = 32,768K = 32M. As Lance correctly points out, it's something of a crapshoot as to whether your particular memory manager will supply that much EMS - and, even if you get it to work on yours, there's no guarantee that the resulting program will run on another PC unless it has exactly the same configuration as yours. If you do get away with it, though, then that'll be the only VIRTUAL array you can have, as your EMS will be completely consumed. (Unless you're using QEMM which, as Lance said, can sometimes get you 64Mb of EMS if you configure it right.)

Adding a fourth dimension, i.e.

DIM VIRTUAL Repeat4(255, 255, 255, 255) AS INTEGER

swells the memory requirements to 256 * 256 * 256 * 256 * 2 bytes per element = 8,589,934,592 bytes = 8,388,608K = 8,192M = 8 gigabytes of RAM... and under no circumstances will you ever get an array that large in PB/DOS.

I have to confess, though, I'm both intrigued and baffled by just what you're trying to accomplish here...

------------------


[This message has been edited by Gary Akins (edited February 12, 2001).]

[Robert Jennett]

I did the single byte math yesterday & already decided on plan C
is in effect, I will use a large file on a 60 MB ramdrive & I can
drop the negative value also & use 2 or 3 bytes to get a high
count that I need on at least a 3 deep array & a higher count on
the 2 dimentiional arry, yes slow as hell but it should work.
I am trying a compression idea that I have had for along time,
that will only work on large files, but I maybe able to run again
& again on the same file till the file get small, I have a big
loss so far but I need to look at the stats on repeated bytes so
I can do the math & figure the best way to attack this part of
my idea. I have a lost but I have lots of flags free 72 of them
& if I want to take a bigger loss I can double the flags with
every bit added. If my idea works I will have this program redone
with C or assembly for speed, first I need to prove it works.

The fre(-11) gives me like 32 or 33 MB after I adjusted my emm386
I have gotten screwie results with the huge & virual commands
even on the 2 dimentional array only @ integer.
Like I copied the way sort.bas from the example subdir does the
array & will not work for me but running the program sort.bas
works fine I was doing the same thing ?

Thanks for the math tip, I thought I hade to add on a pointer of 2.

Bob

------------------

[Gary Akins]

Well, you do have a few bytes of array-descriptor overhead to contend with, but that's per array, not per element... I didn't add it in, so as not to confuse the issue unnecessarily. (As the wise man once said, "An ounce of inaccuracy sometimes saves a ton of explanation." )

So, basically, you're trying to do a brute-force pattern analysis, looking for sequences of two, three, four, or five bytes that repeat throughout the file? Well... now you know why compression algorithms like Lempel-Ziv generate their dictionaries on the fly, and limit its size, I guess. I don't think re-coding this in 'C' or ASM is going to gain you much speed; it isn't the complexity of the task that's your bottleneck, it's the size of the data set you're trying to work with...

------------------

[Robert Jennett]

A library would not save me very much in the end as I would have
to store the byte pattern pointers then the total count of them.
Where as an array type file I will be able to use math as a
pointer & only have to store the total count. The four & five
bytes was an addon just to look at the stats, I am sure there
will not be many of these in random large files.

I did study data compression alittle & I have the book DATA
Compression by Gilbert Held third edition it is an old book, he
has the fifth edition out now, but I am attacking this in a
different way & only for large files & I knew this was going to
be slow, my goal is to srink a random file down to its smallest
size possible by running this program 1 time or 100 times & maybe
more if needed.

Thanks again for the help
Bob

------------------