Best Memory for DIM .. AT

[Pat Bullman]

Hello All,

I’m developing a DLL which uses DIM...AT lpMem1 to create and manipulate several arrays.

I use:

ghHeap = HeapCreate( %HEAP_GENERATE_EXCEPTIONS _
                     OR %HEAP_NO_SERIALIZE, _
                     0, 0 )

then:

lpMem1 = HeapAlloc( ghHeap, _
                    %HEAP_ZERO_MEMORY OR _ 
                    %HEAP_NO_SERIALIZE OR _
                    %HEAP_GENERATE_EXCEPTIONS, _
                    lBytes1 )

lpMem2 = HeapAlloc( ghHeap, _
                    %HEAP_ZERO_MEMORY OR _ 
                    %HEAP_NO_SERIALIZE OR _
                    %HEAP_GENERATE_EXCEPTIONS, _
                    lBytes2 )

lpMem3 = HeapAlloc( ghHeap, _
                    %HEAP_ZERO_MEMORY OR _ 
                    %HEAP_NO_SERIALIZE OR _
                    %HEAP_GENERATE_EXCEPTIONS, _
                    lBytes3 )

and also:

lpMem2 = HeapReAlloc( ghHeap, _
                      %HEAP_ZERO_MEMORY OR _
                      %HEAP_NO_SERIALIZE OR _
                      %HEAP_GENERATE_EXCEPTIONS, _
                      hExitingHandle, _
                      lNewBytes2 )

whenever I need to change the size of the arrays. The application is not finished so I cannot yet test the code, however I’m starting to think the above logic may fail.

For DIM..AT lpMem2 to work, lpMem2 must point to non-fragmented memory. However, using the above code, increasing the size of memory at lpMem2 must result in either its memory being fragmented or memory at lpMem3 being corrupted – doesn’t it?

I can only see one option and that is to redo the code to only one use of HeapAlloc() and allocate enough memory for all 3 arrays, as in:

lpMem1 = HeapAlloc( ghHeap, _
                    %HEAP_ZERO_MEMORY OR _ 
                    %HEAP_NO_SERIALIZE OR _
                    %HEAP_GENERATE_EXCEPTIONS, _
                    lBytes1 + lBytes2 + lBytes3 )

and then calculate the offsets for the other pointers, as in:

lpMem2 = lpMem1 + lBytes1
lpMem3 = lpMem2 + lBytes2

Of course, I’ll have to use MoveMemory() after every HeapReAlloc() to move ensure the pointers are correct.

Can anyone see any problems, or a more effective solution?


Pat

[Michael Mattias]

For DIM..AT lpMem2 to work, lpMem2 must point to non-fragmented memory. However, using the above code, increasing the size of memory at lpMem2 must result in either its memory being fragmented or memory at lpMem3 being corrupted – doesn’t it?

No.

It might be easier instead of counting bytes etc .. when you HeapReAlloc, just re-REDIM AT the returned pointer.

ie.

IF UBOUND(currentArray) IS TOO SMALL
   pMem =  HeapRealloc (BigEnough) 
   REDIM CurrentArray (new size) AT pMem
END IF

The contents of the existing block are unchanged; new memory added has 'undefined' content unless you specify HEAP_ZERO_MEMORY in the ReAlloc call in which case the new memory is (surprise!) all zeros.

[Michael Mattias]

BTW...

lpMem2 = HeapReAlloc( ghHeap, _
                      %HEAP_ZERO_MEMORY OR _
                      %HEAP_NO_SERIALIZE OR _
                      %HEAP_GENERATE_EXCEPTIONS, _
                      hExitingHandle, _
                      lNewBytes2 )

That parameter is the pointer returned by HeapAlloc or prior call to HeapRealloc..not some handle.... i.e in your code it would be "lpmem"

How you get hExitingHandle [sic] code not shown.

[John Petty]

Pat
I have never found a need to create a heap in 25 years of using basic. Nowdays the OLE engine as used by PB is extremely fast and safe so if I want to create an array of say udts and think it might grow then do the following allowing a 10% growth.
Mystring = SPACE$( SIZEOF(MyUdt) *1.1)
DIM etc AT STRPTR etc
Use the usual REDIM PRESERVE
For up to 10% increase
If you find you are increasing the array more than 10% then
Mystring = Mystring + SPACE$(increase)
REDIM etc AT STRPTR
And PB and Windows will do all the hard work for you
John

[Michael Mattias]

I think Mr. Bullman's concern was how to resize an absolute array ("DIM AT") without corrupting his data. I don't think he realized it was as straughtforward as it is.

You don't really need to worry about the pointers at all, since presumably you are using the array subscripts to access your data. And if you are not using the subscripts, why bother with DIM AT in the first place?

[Pat Bullman]

Michael,

I'd like you to clarify a few points for me, if you have the time:

That parameter is the pointer returned by HeapAlloc or prior call to HeapRealloc..not some handle.... i.e in your code it would be "lpmem"

Yes, sorry, bad variable naming there on my part. In my code HeapReAlloc() sits in a local function called ReMalloc() and hExitingHandle is a passed parameter - it would be more aptly named lpmem or lpExistingMem.

I said:

For DIM..AT lpMem2 to work, lpMem2 must point to non-fragmented memory. However, using the above code, increasing the size of memory at lpMem2 must result in either its memory being fragmented or memory at lpMem3 being corrupted – doesn’t it?

You answered "No". Can you explain a little more. If I've got:

hHeap = HeapCreate(..)

lpMem1 = HeapAlloc( hHeap,.., lBytes1 )
lpMem2 = HeapAlloc( hHeap,.., lBytes2 )
lpMem3 = HeapAlloc( hHeap,.., lBytes3 )

then surely a later call to

lpMem2 = HeapReAlloc( hHeap,..lpMem2, lBytes2 + 1000 )

would see the pointer lpMem3 invalidated, so in addition to

Dim MyMyArray2,... AT lpMem2, I'd also have to Dim MyMyArray3,... AT lpMem3?

In other words does a call to increase the size of memory - lpMem2 = HeapReAlloc( hHeap,..lpMem2, lBytes2 + 1000 ) -forces Windows to move data at lpMem3 and therefore I'd also need a Dim MyArray3 AT pMem3 as well?

You also said:

You don't really need to worry about the pointers at all, since presumably you are using the array subscripts to access your data. And if you are not using the subscripts, why bother with DIM AT in the first place?

Yes, I’m using array subscripts to access my data.


John,

I have never found a need to create a heap in 25 years of using basic. Nowdays the OLE engine as used by PB is extremely fast and safe so if I want to create an array of say udts and think it might grow then do the following allowing a 10% growth.

It’s hard to explain why I think I think DIM…AT is the best method here. A simplistic view is: I have a DLL which manages data in arrays which is displayed in windows also manages by the DLL. I store the pointer to each data block using SetWindowLong() which is unique to each window. So if the DLL has several window open and say a WM_PAINT is received then the DLL can reference the data simply using GetWindowLong().


Pat

[Chris Holbrook]

oops!

[Michael Mattias]

If I've got:

hHeap = HeapCreate(..)
lpMem1 = HeapAlloc( hHeap,.., lBytes1 )
lpMem2 = HeapAlloc( hHeap,.., lBytes2 )
lpMem3 = HeapAlloc( hHeap,.., lBytes3 )

then surely a later call to ...

lpMem2 = HeapReAlloc( hHeap,..lpMem2, lBytes2 + 1000 )

..would see the pointer lpMem3 invalidated,

Not so surely at all. That is, No.

HeapAlloc/HeapRealloc gurantees a contiguous block of memory with addresses between the return value and Returnvalue+blocksize.

Reallocation of the block at lpmem2 does not affect the block at lpmem3 one bit. Ever.

It’s hard to explain why I think I think DIM…AT is the best method here.

You use DIM AT for one reason only: because it is convenient to treat your data as a PowerBASIC array. i.e., it's "application-specific."

MCM

[Michael Mattias]

Speaking of making your coding life easier...

The code at malloc, realloc, free for PB/Win32 September 20, 2002 includes a set of functions for use with private heaps.

[John Petty]

Pat
I think you missunderstood my comments, totally agree on the value of using DIM AT (I use it lots). The point was that a string of say SPACE$(100000) is also gauranteed to be a contiguous block of memory and you could store its pointer just as you are now doing for the heap.
The difference is that when you change the size PB and Windows do all the cleanup, moving etc for you in a very efficient way. You just need to replace your stored pointer.
John

[Michael Mattias]

I generally use the 'direct' memory allocation functions rather than a string for one reason: scope.

PowerBASIC conveniently and nicely deallocates LOCAL variables when a prodedure goes out of scope.. but memory you allocate yourself using the WinAPI is not deallocated until you do so explictly. Very handy when you want to store a handle or pointer in a window/control's "user data" slot, considering window procedures go out of scope on each notification message.

MCM

[Pat Bullman]

Michael,

Reallocation of the block at lpmem2 does not affect the block at lpmem3 one bit. Ever

I did as you suggested and just relied on using DIM...AT with the pointer returned from HeapAlloc() OR HeapReAlloc(). And now that I got enough code finalised to enable the program to run, I've been able to test it and it seems to work.

So thank you.

My only concern is that I don't understand what's hapening. I can't wrap my head around it all. A bit of reading and further investigation on my part is warranted.


Pat

[Pat Bullman]

John,

point was that a string of say SPACE$(100000) is also gauranteed to be a contiguous block of memory and you could store its pointer just as you are now doing for the heap.

The problem with this approach is that the DLL, where this code sits, might manage several windows and each window has it's own data to store. I can't use one variable.


Pat

[John Petty]

John,The problem with this approach is that the DLL, where this code sits, might manage several windows and each window has it's own data to store. I can't use one variable.
Pat

I understood that from the start, obviously every window creates and has its own string which the DLL references by its pointer just like a heap not by name.
Michael is of course correct that you have to be very careful about scope as strings never cause memory leaks which means that derided thing called a GLOBAL needs to be used or at least STATIC.
Am just posting a reference in the Cafe on the future of this subject
John

[Michael Mattias]

....enable the program to run, I've been able to test it and it seems to work.
... I don't understand what's hapening..
.... A bit of reading and further investigation on my part is warranted.

More than "a bit" of reading and investigation methinks.

The printed PB/DOS manuals explained absolute arrays really well, with pictures and everything. But it seems you understand this well enough.

But there is not anything in print even that old which is going to explain any better than the MS-SDK doc (you DID read up on memory management in the SDK doc, right?) how the various memory-management functions work.

If you do not have a decent reference ("win32api.inc" is not a decent reference) you can start here: http://msdn.microsoft.com/library/default.asp

(Sheesh, I never could use something I did not understand. I am amazed sometimes how so many others can do that).

MCM

[Chris Holbrook]

PowerBASIC conveniently and nicely deallocates LOCAL variables when a prodedure goes out of scope..

So use STATIC or GLOBAL declarations?

[Chris Holbrook]

I can't use one variable.

Why not use an array of strings? What kind of changes do you need to make to the adat returned by the dll, BTW?

[Pat Bullman]

Michael,

The printed PB/DOS manuals explained absolute arrays really well, with pictures and everything. But it seems you understand this well enough.

Yes, I’m fine with these – once I got over the fact of how easy and powerful they are.

But there is not anything in print even that old which is going to explain any better than the MS-SDK doc (you DID read up on memory management in the SDK doc, right?) how the various memory-management functions work.

Yes, I’ve been using the SDK and that’s why all memory function have been moved from the original code where GlobalAlloc range were used.

(Sheesh, I never could use something I did not understand. I am amazed sometimes how so many others can do that).

Yes that’s why I doing further reading and testing to try to understand it all. I think what stumped me was the need to REDIM...AT after every Heap call. My intuition couldn't grasp that a pointer returned from HeapReAlloc() could be different from the original HeapAlloc() call yet the data at the original pointer was now at the new pointer (hope that makes sense).

ALSO, what I mean by understand is try to have some basic concept of what Windows is doing behind the scenes.


John

Why not use an array of strings? What kind of changes do you need to make to the adat returned by the dll, BTW?

I can see how this could work but I seems involved and convoluted when, to me, the API provides a cleaner way of doing it and the purpose of doing this project is to help me understand both PowerBASIC and the 32 bit API's.

I don't know if I've previously mentioned this but the code is a port a 16 bit DLL code.


Pat

[Michael Mattias]

>So use STATIC or GLOBAL declarations?

Fails when multi-threaded or multiple windows sharing window procedure, i.e., STATIC and GLOBAL are inherently not re-entrant.

[Gösta H. Lovgren-2]

(Sheesh, I never could use something I did not understand. I am amazed sometimes how so many others can do that).MCM

It's called "Cut & Paste". I can recco a bit of reading if you like {smarmy smile}.

==================================================
"The budget should balance,
Treasury refilled,
Public debt reduced,
Arrogance of officialdom tempered and controlled,
Assistance to foreign lands curtailed
Lest Rome become bankrupt.
People must again learn to work,
Instead of living on public assistance."
Marcus Tullius Cicero
(Just prior to the Fall of the Roman Empire)
==================================================