Memory Allocation

[Dean Schrage]

Can anyone offer explanations on the following with this sample code. My machine as 3 GB memory and the /3GB switch has been set. However, I can only allocate around 0.8 GB. On a similar machine, I can allocate about 1.7 GB. However, I can never allocate above that without failure. When you run this, open taskmgr and observe the MEM allocation to see the drop off.

#COMPILE EXE
#DIM ALL

FUNCTION PBMAIN () AS LONG

    DIM N           AS LONG

    DO
        N = VAL(INPUTBOX$("Array size"))
        REDIM x(N)      AS DOUBLE
        MSGBOX "DOUBLE array, size " + STR$(N) + $CRLF + "MEM (MB) Predicted to use:  " + STR$(08 * N / 1024 / 1000), , "MEM Test"
        ERASE x()

    LOOP UNTIL N = 00

END FUNCTION

[Michael Mattias]

Check swap file size.

Also check ERR after the DIM/REDIM. If the REDIM fails the rest of your results are totally and absolutely meaningless.

Someone had this same issue about 3 weeks ago. They made the same mistake, not querying ERR. Basically the upshot of that thread was, his posted program ran perfectly on my machine with the same compiler and operating system, so it must have been a 'settings' difference, which is where I suggested checking the swap file.

IIRC the followup discussion, I seggested the design was terrible, with one PB array never changing, just sitting there eating up memory... that data should have been stored for program use in a random access disk file and read when needed (which was not all that often).


MCM

[Gösta H. Lovgren-2]

Duuno if ERROR checking helps out here or not.

'

#Compile Exe
#Dim All
                   
Function PBMain () As Long
    Local N, Element_size           As Long
    Local s$
    Do                                     
       ErrClear
        N = Val(InputBox$("Array size")) 
        If n = 0 Then Exit Loop
        If n < 0 Then
           ? Using$(" # way too big", n),," Grando Mucho Numberaro"
           Iterate Loop
        End If
        Element_size = 32
        ReDim x(N) As Quad 'eat up memory faster
        
        If Err Then
           ? Error$(Err) ,, "Error" & Str$(Err)
           'Exit Loop
        End If
        s$ = "String array size with #, Elements " & $CrLf & $CrLf & _
             " Predicted to use: #, MB "
        MsgBox Using$(s$, N, Element_size * N / 1024 / 1024), , "MEM Test"
        Erase x()
    Loop Until N = 00
End Function
'

==============================================
"Thank you for sending me a copy of your book
I'll waste no time reading it."
Moses Hadas (1900-1966)
==============================================

[Michael Mattias]

>Element_size = 32
> ReDim x(N) As Quad

QUADs now require 32 bytes?

Darned, yet another undocumented change across compiler versions!!

[Dean Schrage]

thanks all

ok ERR can be used to infer that the allocation is in error.

But I still do not understand if the boot switch is set to /3GB and I have 4GB of memory, why I cannot get access to this memory. Any ideas ?

[Michael Mattias]

>Any ideas ?

What about the swap file size? Did you check that?

[Paul Dixon]

Dean,
which compiler are you using? I think only the latest (PBWin9/PBCC5) can access to more than 2GB.

I can get a 1,716,000,000 byte array on my PC but it's very slow because the swapfile is being used a lot as I only(!) have 512MB RAM installed.

Paul.

[Gösta H. Lovgren-2]

>Element_size = 32
> ReDim x(N) As Quad

QUADs now require 32 bytes?

They do in my head, which be may roomy than your {grin}.

Actually I was thinking 32 bit processor.

===================================
"Once is happenstance.
Twice is coincidence.
Three times is enemy action."
Auric Goldfinger, in "Goldfinger"
by Ian L. Fleming (1908-1964)
===================================

[Dean Schrage]

yes, swap file size had no affect. Is that the case, that is, PB 9.0 will compile over the 1.8 GB limit? Bob Zale, can you address this if possible or if someone can, test the compilation using 9.0?

For those interested, as an aside, this level of memory usage is very typical for computational analyzes. I find the 3GB limit addressed very frequently on other BBs. They all seem to point to a ~ 1.8 GB shut off point. What gives?

thanks all for helping ...

[Paul Dixon]

Dean,

They all seem to point to a ~ 1.8 GB shut off point. What gives?

many applications, including PB before CC5/Win9, had a 2GB limit on total memory that any single process could handle. Originally, the OS occupied 2GB of address space and the program occupied the remaining 2GB from the 4GB available.

I think lots of folk complained and Microsoft listened .. a little, and later versions of the OS could be resticted to 1GB (still more than enough!) leaving 3GB to applications that could use it.
But, many applications were written with the original 2GB limit in mind and can not take advantage of the extra GB.

The 1.8GB is about right. It allows 200MB for your program code, all the stacks and bits an pieces of other stuff and the remaining 1.8GB is useable for data.

if someone can, test the compilation using 9.0?

I've run it with PBWin but my OS doesn't start with the /3GB flag so I still only have 2GB to play with, hence the cut off is still just under 2GB.
I remember trying the 3GB flag once and it caused lots of problems so I'm reluctant to try it again.

Paul.

[Paul Dixon]

PS maybe this thread may help:

http://www.powerbasic.com/support/pbforums/showthread.php?t=17095

[Dean Schrage]

Paul

thanks for digging that up!. Excellent resource. Application runs on XP Pro but now the following needs to be worked:

"No APIs are required to support application memory tuning. However, it would be ineffective to automatically provide every application with a 3 GB address space. Executables that can use the 3 GB address space are required to have the bit IMAGE_FILE_LARGE_ADDRESS_AWARE set in their image header. If you are the developer of the executable, you can specify a linker flag:

/LARGEADDRESSAWARE

I will do some searching but does anyone know if it is possible to set this LARGEADDRESSAWARE switch in a PB application?

thanks all!

[Paul Dixon]

Dean,
looking at the .exe files produced by PBWin9, it doesn't appear that the large address aware flag is set.
You could try setting it yourself to see if it works.

If you look at the .exe file in a hex editor, the PE header appears to start at location &h80.
The header starts with the letters PE so you can find it easily if it's not at that exact location.
22 bytes beyond the P in PE is a word of value &h030E. The bits of this word are defined here:

http://msdn.microsoft.com/en-us/library/ms680313.aspx

If you were to set the 6th LSB bit (change &h030E to &h032E) then the application should have the large address aware flag set.

I haven't tried it but it maybe worth giving it a go.

Paul.

[Michael Mattias]

The fuddy-duddy speaks: instead of editing compiled executables, maybe the program should be re-engineered to NOT require that much virtual memory.

Or, it should be rewritten using a compiler which supports this kind of RAM requirement.

[Dean Schrage]

Paul

Excellent! Will give this a try. Any hex editor you can recommend? I think this is a common posting but perhaps you can suggest an easy to use favorite.

As for the apparency of mem bloat, this really isn't the case especially with computational simulations. This is just the new reality of computational physics. 64 bit will buy time but only time. CFD simulations are approaching 1 billion elements.

[John Petty]

MCM I seem to recall that the initial announcements of 9 and and 5 claimed they handled 4GB of memory (they no longer do) and have posted in several discussions regarding this feature including a few days after their release but nobody from PB has ever confirmed or denied the capability or presence of the large address aware flag.
Dean merely editing in the flag without PB confirmation is extremely risky as the compiler would need to be doing all its address calculations using DWORDS or QUADS not LONGS.
If it is capapable, then running on a 64 bit CPU and 64 bit OS (32 bit program) with 6 GB of memory will actually allow you to to use the full 4GB of memory for your program.
All my comps are 64 bit capable and I own a copy of XP 64 bit Pro that I have never used ( nor have I yet upgraded to 9) while i wait for clarification.

[Paul Dixon]

Dean,
any hex editor would do. I use UltraEdit.

Paul.

[Paul Dixon]

John,
the latest help file for PBWin9 lists as a feature:

Use of all available memory (up to 4 Gb) for arrays and dynamic strings

Paul.

[Paul Dixon]

MCM,
some tasks benefit hugely from that much memory just as tasks of 15 years ago benefitted by moving to 16MB of Windows 3.0 from the 640k of DOS.
Also, it's not virtual memory Dean's after, it's real memory. He has 3GB installed and wishes to make use of it.

Paul.

[Dean Schrage]

... actually I have 4GB installed but would be satisified with accessing 3 GB

Paul, can you look at this map and tell me what to change? I am totally clueless here in the hex edit mode?

thanks very much, regards, Dean