HUGE Arrays Not Contiguous, Or Is My Code Bad?

[Clay Clear]

I have written an OBJ procedure that duplicates the Windows
"MoveMemory" function. However, it simply refuses to move the datas
from one HUGE array to another. Both arrays are dimensioned as
(1 TO 40000) AS LONG. It only moves up to appr. the first 16000
elements of the first array into the second array. Is this because the
segments used by each array are not contiguous in memory? Or is
my code bad? If Mr.Dixon, or Mr. Hanlin, et al, want to see the
MoveMemory code, I will post it. However, you should note that
it is written to be compiled as an OBJ file, using MASM 6.14 (that
is why I singled you two guys out ). The only reason I put the
code in an OBJ file, rather than using PB inline, is because MASM 6.14
directly supports dword vars, which PB inline does not. And it is MUCH
easier to decrement the incoming "Count" var as a whole than to use
"word ptr".

Any help GRATEFULLY received.


------------------

[Michael Mattias]

HUGE arrays are not contiguous unless element sizes are powers of two. (I think that is the right arithmetic deduction).

This I do know (or least remember it this way): HUGE arrays use up space in multiples of 64K; if insufficient space is available in current segment for the next element, the space is skipped and the next element starts at offset zero of a new segment.

Heck, come to think of it, those segments may not even be contiguous.

(MS-DOS. The bad old days. How we forget.)

MCM

[Clay Clear]

Thanks, Michael. My empirical testing showed what you said, but
I needed confimration before abandoning the pursuit.



------------------

[Michael Mattias]

Hint.. (I got this from my perviously-made-available-as-freeware package to do ARRAY SORT on HUGE ARRAYS)...

Use the PB internal function ArrayCalc to get addresses.. comments from my 1997 code...

'...
' The pointer to ... array element will be at ...
' If the array is HUGE, moving more than the key length into the key union
' could cause segment overflow; if huge, I have have to deal with the
' segment shift. (Except I used PB ArrayCalc, meaning I do NOT have to worry
' about segment shifts because PowerBASIC does).

.. and leave the driving to the compiler.

MCM

[Clay Clear]

Michael,

Thank you for the suggestion, but ArrayCalc is not an option for
me. My DOS versions of "MoveMemory", "FillMemory", and "ZeroMemory"
are designed to work with scalars of all types, structures, and arrays.
The procedures have no idea of what KIND of var for which the
VARPTR32\STRPTR32 coming in is. All they know is they are supposed to
do this or that at this memory location for this many bytes.


------------------

[Tom Hanlin]

I don't know whether HUGE arrays are ever entirely contiguous. They
might be, perhaps depending on the element size, as Michael suggests.
Regardless, your memory routines are not designed to handle the full
possible size of a HUGE array. See here:

    ! mov cx, Count[0]
    ! rep movsb

You're restricted to copying a maximum of CX bytes, where CX can hold
up to 65,535. A LONG array of 40,000 elements is 160,000 bytes.

The maximum of 65,535 is, actually, a best case. A single segment has
a range of 65,536 bytes, assuming offsets of 0..65535. However, the
addresses are loaded directly:

    ! lds si, Source
    ! les di, Destination

...so, you have no guarantee of what the offsets actually are. If, for
example, the Source offset turns out to be 65535, you will only be able
to copy a single byte before the offset wraps around to 0 within the
same segment-- probably not what you had in mind.

This is not to suggest that you rewrite the routines to handle every
possible situation. They'd get much more complicated and would be slower
for more common cases. If you're feeling ambitious, though, consider the
following.

A segment ffset address may be converted to an absolute address using
the calculation "(segment * 16) + offset". There may be many combinations
of segments and offsets that resolve to the same absolute address. For
example, the base of the BIOS data section for comm ports is &H40:0, or
0:&H400. When you're using a REP instruction and need to have the maximum
range for your offsets, it may pay you to convert the segment ffset
addresses so that the segment part holds as much of the address as possible
(say, &H40:0, rather than 0:&H400). That will allow you to copy a possible
range of up to &HFFF0 bytes with a single REP. When you get to the end of
the range of a single REP, if there's still more copying to be done, you'll
need to adjust the segment ffset addresses to point to the next block.

It may be faster to handle DWORDs or WORDs at a time, rather than just BYTEs.
In that case, it would be wise to adjust the address(es) to WORD or DWORD
alignment before starting off on the REP: most CPUs will give you a speed
boost that way.


------------------
Tom Hanlin
PowerBASIC Staff

[Clay Clear]

Thanks for the tips, Tom.

I am going to leave my POSTED code as-is. MY production code is
actually an OBJ file, which I chose for size of the compiled module.
Also, my include-in-everything INC file with all of my DECLARE's
has the Destination and Source arguments declared AS ANY, so I do
not have to use the BYVAL VARPTR32() - can use the variable in standalone
fashion (except for dynamic strings & code pointers - those still
require BYVAL STRPTR32() and BYVAL CODEPTR32()).


------------------

[Clay Clear]

Tom,

Just re-read your post.

In my test coding, which was in the OBJ file, I used a cheesy method
for going beyond the range of an unsigned integer:

    P1A:
    les di, dword ptr [bp + 0ah]
    mov al, es:[di]
    les di, dword ptr [bp + 0eh]
    mov es:[di], al
    inc dword ptr [bp + 0ah]
    inc dword ptr [bp + 0eh]
    dec dword ptr [bp + 06h]
    jnz P1A

You should note that I have 486/387 modes enabled in my MASM
compiler.



------------------

[Michael Mattias]

So what's wrong with PEEK$ and POKE$?

Those 'move memory' just fine. Always have, and I suspect always will.

Ok, so a 'block' to be moved in a single instruction (BASIC statement) is limited to the current $STRING setting. Big deal, as you have to deal with a 64K max 'block' size under MS-DOS anyway.

Oops! How silly of me to forget! Using assembly language is <U>cool</U>.
Never mind.

MCM

[Tom Hanlin]

PEEK$/POKE$ would be simpler and more efficient, but an understanding
of assembly language is priceless.

------------------
Tom Hanlin
PowerBASIC Staff

[Tom Hanlin]

going beyond the range of an unsigned integer

LES wasn't an opcode I used often, and maybe I'm foggy on the details,
but I'm not sure you've improved the range, here. I'm guessing that
you're only incrementing the memory offset, not the segment, and haven't
increased the range at all...

Reloading all data from base memory, every time, would be inefficient
in any event. What you want to do, ideally, is put all the calculations
up front, and leave the "inner loop" part of the code to do as little as
possible. Think REP, then registers, then [registers], then (last) cases
like [register + number] addressing. Segment overrides (e.g., ES should
also be avoided whenever practical.


------------------
Tom Hanlin
PowerBASIC Staff