Large Array memory allocation

[Hilton Schwartz]

Hi : I'm working on an application for the stock market with very large arrays.
They are all DIM'ed to the largest size needed.
The program will analyze any number of days of data, from 0 to 1,200.
At the end, there is a screen display of the results with a WAITKEY$, after which there is an EXIT FUNCTION.

When I run the program with 200, 500, or 900 days of data, all runs fine.
However, when I run 1,000 days, after the analysis and after the screen display of the results and then a key input I get "Program has encountered an error and needs to close " from M$.
I assume this is a GPF.

I had always assumed that a DIM allocated the memory needed, and so if I needed too much memory, I would get a fault as soon as I started the program. It seems here that as I ask for more and more days of analysis, and use more and more of the arrays, when I get to a certain size, something goes wrong.
But curiously, it does not go wrong when actually running the program, but when I return via EXIT FUNCTION.

Any ideas ?

Hilton Schwartz

[Michael Mattias]

"GPF" is an acronym for "general protection fault". That means your program tried to do "something" which would violate the integrity of the system memory.

You will not necessarily get a Windows' GPF as soon as you make the programming error; that fault may not show up for hundreds or thousands of lines of executed code. The only way to trap these guys when they happen is to test yourself using the tools provided.

eg, if DIM or redim fails, you will get an error 7 which you can test (ERR variable) immediately following the statement.

But I will put money on an error somewhere else in your program, since 1000 "anythings" are not going to exceed the process memory allowance.

As for the close, if you view the detail of the "needs to close" it will tell you the problem. It could be page fault, stack fault, invalid operation or something else.

I am guessing "stack fault" here, since you are able to re-create the error with the same value (1,000) each time.

[Mike Doty]

Have you tried running with #DEBUG ERROR ON ?

[Rodney Hicks]

Contrary to Michael, I do think the error will be in the DIM or REDIM statement or related to it. Bounds and so on.

However since the code isn't shown, we can help only by guessing, so I could be way wrong here.

But "very large arrays" and "largest" I am taking to be indicators of the problem.

It is very easy to dimension arrays too large for available space when constructing a Stock Market program, depending on the Data TYPE used.

Rod

[Donald Darden]

If you are using variable string arrays, then the space allocated for the array will only be used to reference the string data indirectly. The contents of each string will be store elsewhere, and the amount memory required cannot be anticipated adequately, and only determined at run time. But that is only one
possibility. It still could be a number of other things. One thing you could try is to reduce the number of competing applications, or run on a PC with more available RAM, and see if that has an effect.

If you suspect a memory size problem, the question is whether you can do much to offset it. One way is of course not to have all the data in memory at the same time, but only to call up what you need when you need it, and to free up memory by deleting or erasing arrays or variables when not needed. Another is to reuse variables as much as possilbe, which deletes their old contents automatically. Some people use a RAMDRIVE as a pseudo disk to store information temporarily in a standard file structure, so that it can be quickly moved in and out of your application.

But you really need to approximate how much memory your application requires, and measure that against the maximum that your OS can support. either in total, or for a given application. If you are not anywhere near the limit (and let's include the size of you application in this estimate), then it is likely a different cause. To approximate, you cannot simply consider the number of elements in a given array, but consider what type of array, and how extensively it is used.

[Hilton Schwartz]

Update - Large Array memory Allocation

Thank you all for your suggestions.

I checked the details given by the M$ "Has to close" message, but I cannot find where it says the fault is. It does give a lot of contents of memory and registers. Where should I be looking for the fault report, stack or otherwise ?

There is no ERR on DIM, or anywhere. I use ERR statements extensively.
I get the same error whether the the DIM are GLOBAL or not.

#DEBUG ERR is on, and, unless I missed a spot, all bounds are checked.

I have done all I can to reduce memory usage, focussed on ensuring no bounds are violated and they are set as tight as possible.

The program is way too large to post.
I currently put all my DIM's at the top of the program, after the FUNCTION PB MAIN.

I am amazed that, as Michael M mentions, the MS message can show up hundred of lines after it occurs.
Makes you wish for GWBASIC interpreted < smile >

So, it looks like I have some more debugging to do. Sigh.

Assuming the problem is using too much memory, the following are not clear to me...
1. There is no error when I run 500 days, but there is an error with 1000 days. If I were overwriting memory, why wouldn't I get an error when I use 500 days ?
2. What is the role of the swap file ? I always thought that when you needed more RAM, this would be used.
3. Why does the program run well, error aside, if I have overwritten memory somewhere ? Are the results suspect ? They look OK.
4. If I ran the program in the Debugger, an onerous task, would I get any additional information ?
5. How do I calculate memory needed for LONG and DOUBLE variables ? From the previous post, vab I assume STRING varibales are small enough not to concerm me ?

Thanks

[Paul Dixon]

Assuming the problem is using too much memory, the following are not clear to me...

You can't use too much memory. If you're checking ERR after allocating things and ERR=0 then the memory was allocated and you can use it all.
If you try to use too much then the OS will refuse to allocate the memory and ERR will tell you this.

1. There is no error when I run 500 days, but there is an error with 1000 days. If I were overwriting memory, why wouldn't I get an error when I use 500 days ?

Maybe you're overwriting memeory that you don't use until days = 1000.
Maybe you have a calculation based on the number of days which overflows?
There could be lots of reasons.

2. What is the role of the swap file ? I always thought that when you needed more RAM, this would be used.

Memory for your program is limited to 2GB.
Real memory will be used first but if the OS decides it needs more than the physical memory available then it'll use the swap file on disk to make up the difference so your code can act as though it has 2GB avaialble even though it doesn't have that much RAM installed. The OS takes care of the swapping. At that point things slow down considerably as accessing data from disk is very slow compared to accessing data from main memory.

3. Why does the program run well, error aside, if I have overwritten memory somewhere ? Are the results suspect ? They look OK.

Could be lots of reasons. Maybe you've overwritten the stack at the place storing the return address for PBMAIN so when the code ends it crashes? There are many other possibilities.
Results must be considered suspect until you understand the cause of the problem.

4. If I ran the program in the Debugger, an onerous task, would I get any additional information ?

Yes. It's not always useful but it can help track down errors in running code.

5. How do I calculate memory needed for LONG and DOUBLE variables ? From the previous post, vab I assume STRING varibales are small enough not to concerm me ?

LONGs take 4 bytes each, DOUBLEs takes 8 bytes each. Dynamic STRINGs still take up memory, even if it belongs to the OS's allocation and you can still run out.

Paul.

[Michael Mattias]

I hope you ran the program using TRACE as shown in the FAQ at FAQ: Finding memory corruption using TRACE

Null pointer and array bounds errors stick out like a sore thumb (find "error" in trace file), and potential stack problems will be evident because the 'indent' will basically push the entries way to the right of the page.

However, absent code to look at, nobody is going to find your problem.

[Rodney Hicks]

I currently put all my DIM's at the top of the program, after the FUNCTION PB MAIN.

If the previous posts don't help:

Try just posting the DIMs for us to look at, or even just a small program with the dims and nothing else. If your small program with the DIMs still crashes you know you're onto something. Developing little code snippets to show the problem sometimes show the solution.

Rod

[Richard Angell]

If you are DIMming an array (e.g.) MyDayData(1000) each time and trying to access MyDayData(500), no sweat, but at MyDayData(1000) M$ Windoze will not like your program You would need DIM MyDayData(1 TO 1000) ... if you have not already checked this possibility.

If you can post just the DIM section as suggested, someone might see a problem, but if not it can help with other data that lets us see the size of arrays you are creating in memory.

[Mike Doty]

Richard,
DIM T(1000) AS LONG gives 1001 elements 0-1000.

[Richard Angell]

Richard,
DIM T(1000) AS LONG gives 1001 elements 0-1000.

right, had my mind elsewhere

[Hilton Schwartz]

Update Large Array Allocation

I'm somewhat sheepish about the solution.

After much debugging, turns out that the 1000 days was not overwriting memory anywhere, but that data around 950 days had 3 very rare problems that caused ERR's.

How rare ?
1000 days x 4 methods of analysis x 10,000 stocks = 3 in 40 million !

On the other hand, I did learn a lot about memory and allocation, and made my program a touch more bullet-proof.

Thanks all for your help.

[Michael Mattias]

On the other hand, I did learn a lot about memory and allocation, and made my program a touch more bullet-proof.

Looking for a position as a 'spin doctor,' are we?

[Mike Doty]

What were the ERR's, he asked questioningly?

[Hilton Schwartz]

Large Array Memory Allocation

No spin intended.
I find that debugging code, even in the wrong direction, eventually leads to better coding. Don't you ? < smile >

In the section of code that caused problems, I was checking array bounds in software, not with ERR. There was also a divide by zero that crept in.
Because of the iterative nature of the analysis, I imagine a bunch of these piled up, causing the later message "M$ must close".
When I invoked the use of ERR, I got ERR 9, and this, along with clearing the divide by zero, eventually lead to the discovery of the real problem.

[Michael Mattias]

Because of the iterative nature of the analysis, I imagine a bunch of these piled up, causing the later message "M$ must close".

"Piled up" and iterative nature really have nothing to do with the error, except in the context of how your application used the erroneous results.

When you get an error 9 the statement you attempted is not executed. (Either you get error 9 or the program reads/writes the wrong memory location depending on your #DEBUG setting at compile time. Meaning of course, if your program got the GPF when run with #DEBUG ERROR OFF, you were not executing the same code executed when you ran with the TRACE).

What you do with the variable determines what happens later.

Divide by Zero errors cannot be trapped with the PB/Compilers. I've had a New Feature Suggestion in to add support for overflow checking for some time. You might want to send one in, too, for "it is said" the more who ask the more likely we are to receive.