Array Size?

[William Martin]

Is there a maximum array size one needs to deal with in PB8? I tinkered with compiling a DIM statement defining a string array and kept bumping it up until it seemed unreasonably large and PB never did complain that the array would be too large to manage.

(e.g. I DIMed a 3 dimensional string array 2,000 elements on a side - something like 8,000GB of data if I didn't slip a dismal point. And a lot bigger than my computer is.)

The problem I'm really considering is to move an entire large Excel workbook into PB (via VBA call & DLL) to do some intensive calculation and then back to Excel with the result. This would be as an alternative to moving individual sheets to PB repeatedly. I was trying to understand how large an array PB could reasonably handle.

Thanks.

Bill

[Paul Dixon]

Bill,
did you check the ERROR value after you dimensioned your long array?

Paul.

[William Martin]

Obviously not. I *presumed* the compiler would tell me if there were an error. Perhaps a bad assumption.

And how does one check the ERROR value when compiling to a DLL? I haven't found any way to step through the code as it actually executes when it's compiled to DLL. Obviously I don't have much PB experience.

(Ok, I added an integer variable to the call parameters and can check the ERR flag when it executes. Apparently space is only allocated at run time and not checked at compile time.)

Bill

[Michael Mattias]

I *presumed* the compiler would tell me if there were an error. Perhaps a bad assumption.

Well it does but perhaps not the way you were expecting.

The compiler will set the system ERR variable on an allocation error (array too big), but you will have to check that value to know if the DIM/REDIM succeeded or not. If ERR does not equal zero, something bad happened. The ERROR$() function will give you the error in text, saving you a trip to the help file.

Apparently [array ?] space is only allocated at run time and not checked at compile time.

True.

[William Martin]

Following up, I set up the PB routine to take a parameter from the Call, and dimension a 3D array to that size.

And then I ran an Excel VBA routine, calling the PB DLL, which keeps bumping up the array size by 1% until an error code is returned by PB.

When I DIM the array as LONG, it fails at a max dimension of 502. Curiously though, if I DIM the array as STRING, it fails at a max dimension of 509.

Since STRING takes a lot more space than LONG, can I presume that if I dimension a large STRING array it may get through the DIM ok, but still fail later on when the array elements are filled with strings? And how will it fail? Just not do some of the writes, or crash the program, or write over top of other stuff or what?

Do I have to make PB check ERR every time I fill an array element? Or how does one normally deal with this?

Thanks.

Bill

[Cliff Nichols]

Hi William,
This may help with tracking down the array size <code not shown> so you will need to piece my error handling inc file into your program.
RunTime Debug and Error Handling Part I - Find that elusive bug from the source code forum.

Oh and use "#DEBUG ERROR OFF" so all errors are executed

[Michael Mattias]

>And how will it fail?

Error 7 (memory allocation error).

Dynamic string arrays require four (4) bytes per element at DIM/REDIM time, but the space for the data are not allocated until you assign a value to that element. (or in your case, "try" to assign a value).

For all other arrays (except maybe variants?), all space which will be required is allocated at execution of the DIM/REDIM statement.

MCM

[Gösta H. Lovgren-2]

Bill,

One tried and true stub your toe seat of the pants trial and error crude semi-couth method to test beforehand is to '

'PBWIN 9.00 - WinApi 05/2008 - XP Pro SP3
#Compile Exe                                
#Dim All 
#Include "WIN32API.INC"
#Include "COMDLG32.INC"
 
Function PBMain
    Local Ans, Array_Bounds_Check, Add_on As Long
    Add_on = 10
    Array_Bounds_Check = 100 'okay to start
Add_More:  
    ReDim Arry(Array_Bounds_Check, Array_Bounds_Check, Array_Bounds_Check) As String * 100
    Arry(Array_Bounds_Check, Array_Bounds_Check, Array_Bounds_Check) = "Testing"
    Ans = MsgBox (Using$( _ 
          "Testing Array Sizes" & $CrLf & _
          "#, is okay", Array_Bounds_Check), %MB_YESNO, "Click Yes")
 
    Array_Bounds_Check = Array_Bounds_Check + Add_on
 
    If Ans = %IdYes Then GoTo Add_More      
 
End Function
'

Obviously you can change the Starting & Add On variables to anything you want (1, 10, 25, ...)

My machine chokes out (GPF's) between 230 & 240 (for Array_Bounds_Check) when Dimming String * 100. Much higher (1100+) when using a string size of 1. Between 700 - 800 when As Long.

Someone (avec programming elegance) could easily add a memory check in there too (if he wanted). And if someone wanted she could add an Error Check inside a Loop as well.

=======================================
"There are only two tragedies in life:
one is not getting what one wants,
and the other is getting it."
Oscar Wilde (1854-1900)
=======================================

[Paul Dixon]

Bill,
I just tried this on 3 computers I have here.
My main PC (512MB ram, loads of disk) allocated 1.8GB before failing.
A laptop with 256MB RAM, loads of disk, allocated 900MB
An ASUS EeePC with 512MB RAM and 4GB hard disk allocated the full 2GB.

All run WinXP SP2 or 3.

I'm sure I could increase the swap file on the first 2 so they'd match the Asus.

So 2GB isn't unreasonable to expect.

That's quite a lot of data for one speadsheet. Are you sure it's that big?
Also, for speed you need to keep the data out of the swap file if you can so it might well be faster to process one page at a time in real memory than load it all and have it swapping in and out of virtual memory.

Paul.

[William Martin]

Thanks for the solution Gosta. It will take me awhile to digest it.

No Paul, my spreadsheet is not that big. (Maybe 15MB depending on how long the actual strings are.) Still I get nervous about PB exhibiting an amorphous failure mode that may pop up unexpectedly if I don't fully understand it. For example I haven't come to terms yet with whether my program would seem to run ok now, but not later if XP's loading changes. Yet constantly checking ERR seems a bit paranoid.

Or for that matter, can the program lock up in an infinite loop if it runs out of space (depending on how the code is written)? I just don't know. It's not a failure mode I'm familiar dealing with. Just chewing over the possibilities at present.

Thanks all... I'll retreat to my cave for awhile.

Bill

[Erich Schulman]

There are two ways to look at the biggest array size: the biggest that can be allocated at runtime, and the largest range of acceptable subscripts.

This actually does compile in PBCC5:
DIM A(0 TO 999999999999999999999999999999999999999999999999999999999999999999999999999) AS LONG

I am not going to investigate the actual object code, but it does seem unlikely the compiler actually understood this as an array with far over a decillion elements.

Looks like a new wishlist item would be documented max LBOUND/UBOUND values and (when possible) compile-time bounds checking.

[Michael Mattias]

That's quite a lot of data for one speadsheet. Are you sure it's that big?

Part II of this question: Do you really need to load ALL the data into memory at one time?

Can you maybe get whatever it is you are doing to work by,,,,,

1. Only loading 'n' rows at a time?

2. Treating the spreadsheet as a database and manipulating with SQL statements?

3. Unloading the spreadsheet contents to a temporary disk file, 'doing your thing' and putting it back?

Lots of options. Application-specific. YMMV. Void where prohibited.

MCM

[Michael Mattias]

>Yet constantly checking ERR seems a bit paranoid.

ON ERROR GOTO....

[Eric Pearson]

Erich --

> but it does seem unlikely the compiler actually understood
> this as an array with far over a decillion elements

It wouldn't "understand" that until runtime, when it attempted to dim the array and failed. To do it at compile time the compiler would have to guess how much memory would be available at runtime. How much memory will a typical computer have in 2018, when the program might be executed? So when should a compile-time error be flagged in 2008?

After all, a decillion bytes should be enough for anybody, right?

William --

> Yet constantly checking ERR seems a bit paranoid.

Not checking ERR after performing an operation that can fail due to a runtime limitation is... what's the opposite of paranoid?

-- Eric

[Michael Mattias]

>what's the opposite of paranoid

In context? Silly.

[William Martin]

Part II of this question: Do you really need to load ALL the data into memory at one time?

No, but I'm trying to understand the trade-off between space and speed. I can load and operate on one sheet at a time, scan through all the sheets, and then go back to the beginning an reload the first sheet again. Loop, loop, loop... That involves thrashing a lot of the same data repeatedly between Excel and PB.

Or I can move the whole thing to PB at once and execute on it a few hours with PB and move it back again once. (It's a successive approximation of a non-linear and non-continuous problem.)

I'm just trying to understand the programming constraints and trade-offs before writing a ton of code.

Bill

[Paul Dixon]

Bill,
the "page at a time" suggestion was with 8GB of data in mind. Since you now say it's only 15MB then there is no problem loading the whole thing at once.
On a 10 year old computer I'd think nothing of creating 100MB arrays. 15MB really isn't a problem on a modern PC.

Paul.

[Michael Mattias]

> That involves thrashing a lot of the same data repeatedly between Excel and PB.

Well, I don't know how Excel does it, but "thashing" data between PB functions only involves a couple of integers (the data addresses).

Maybe you could do the whole job within Excel... but writing some functions, putting them in a Dynamic Link Library and using that in some cell formulae.

MCM

[Gösta H. Lovgren-2]

Paul is right. A 15 mb speadsheet is nothing to Load unless it's a ton of cells.

But that aside, I must be missing something here though. How can one check for an Error before Dimming an array (as in the sample snippet above). An "over dim" causes a GPF, not a checkable Error per se.

Bill, when writing this it occurs to me, if too many cells in an array is a problem, why not break it down into several arrays? Not as convenient programmically as having all the data in one 3 dim array, but no more Dim GPFs or memory constraints (effectively anyway).

==========================
He that can have patience
can have what he will.
Ben Franklin
==========================

[Chris Holbrook]

But that aside, I must be missing something here though. How can one check for an Error before Dimming an array (as in the sample snippet above).

Explicit allocation (like GLOBALMEM ALLOC) will return an error instead of GPFing.

When you have the address of the allocated memory, you can DIM your array AT that address.