Announcement

Collapse

Maintenance

The forum could be offline for 30-60 minutes in the very near future for maintenance (said 3pm Pacific). I was behind on getting this notice. I do apologize.
See more
See less

HUGE Arrays Not Contiguous, Or Is My Code Bad?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • HUGE Arrays Not Contiguous, Or Is My Code Bad?

    I have written an OBJ procedure that duplicates the Windows
    "MoveMemory" function. However, it simply refuses to move the datas
    from one HUGE array to another. Both arrays are dimensioned as
    (1 TO 40000) AS LONG. It only moves up to appr. the first 16000
    elements of the first array into the second array. Is this because the
    segments used by each array are not contiguous in memory? Or is
    my code bad? If Mr.Dixon, or Mr. Hanlin, et al, want to see the
    MoveMemory code, I will post it. However, you should note that
    it is written to be compiled as an OBJ file, using MASM 6.14 (that
    is why I singled you two guys out ). The only reason I put the
    code in an OBJ file, rather than using PB inline, is because MASM 6.14
    directly supports dword vars, which PB inline does not. And it is MUCH
    easier to decrement the incoming "Count" var as a whole than to use
    "word ptr".

    Any help GRATEFULLY received.


    ------------------

  • #2
    HUGE arrays are not contiguous unless element sizes are powers of two. (I think that is the right arithmetic deduction).

    This I do know (or least remember it this way): HUGE arrays use up space in multiples of 64K; if insufficient space is available in current segment for the next element, the space is skipped and the next element starts at offset zero of a new segment.

    Heck, come to think of it, those segments may not even be contiguous.

    (MS-DOS. The bad old days. How we forget.)

    MCM


    Michael Mattias
    Tal Systems Inc. (retired)
    Racine WI USA
    [email protected]
    http://www.talsystems.com

    Comment


    • #3
      Thanks, Michael. My empirical testing showed what you said, but
      I needed confimration before abandoning the pursuit.



      ------------------

      Comment


      • #4
        Hint.. (I got this from my perviously-made-available-as-freeware package to do ARRAY SORT on HUGE ARRAYS)...

        Use the PB internal function ArrayCalc to get addresses.. comments from my 1997 code...
        '...
        ' The pointer to ... array element will be at ...
        ' If the array is HUGE, moving more than the key length into the key union
        ' could cause segment overflow; if huge, I have have to deal with the
        ' segment shift. (Except I used PB ArrayCalc, meaning I do NOT have to worry
        ' about segment shifts because PowerBASIC does).
        .. and leave the driving to the compiler.

        MCM
        Michael Mattias
        Tal Systems Inc. (retired)
        Racine WI USA
        [email protected]
        http://www.talsystems.com

        Comment


        • #5
          Michael,

          Thank you for the suggestion, but ArrayCalc is not an option for
          me. My DOS versions of "MoveMemory", "FillMemory", and "ZeroMemory"
          are designed to work with scalars of all types, structures, and arrays.
          The procedures have no idea of what KIND of var for which the
          VARPTR32\STRPTR32 coming in is. All they know is they are supposed to
          do this or that at this memory location for this many bytes.


          ------------------

          Comment


          • #6
            I don't know whether HUGE arrays are ever entirely contiguous. They
            might be, perhaps depending on the element size, as Michael suggests.
            Regardless, your memory routines are not designed to handle the full
            possible size of a HUGE array. See here:
            Code:
                ! mov cx, Count[0]
                ! rep movsb
            You're restricted to copying a maximum of CX bytes, where CX can hold
            up to 65,535. A LONG array of 40,000 elements is 160,000 bytes.

            The maximum of 65,535 is, actually, a best case. A single segment has
            a range of 65,536 bytes, assuming offsets of 0..65535. However, the
            addresses are loaded directly:
            Code:
                ! lds si, Source
                ! les di, Destination
            ...so, you have no guarantee of what the offsets actually are. If, for
            example, the Source offset turns out to be 65535, you will only be able
            to copy a single byte before the offset wraps around to 0 within the
            same segment-- probably not what you had in mind.

            This is not to suggest that you rewrite the routines to handle every
            possible situation. They'd get much more complicated and would be slower
            for more common cases. If you're feeling ambitious, though, consider the
            following.

            A segment ffset address may be converted to an absolute address using
            the calculation "(segment * 16) + offset". There may be many combinations
            of segments and offsets that resolve to the same absolute address. For
            example, the base of the BIOS data section for comm ports is &H40:0, or
            0:&H400. When you're using a REP instruction and need to have the maximum
            range for your offsets, it may pay you to convert the segment ffset
            addresses so that the segment part holds as much of the address as possible
            (say, &H40:0, rather than 0:&H400). That will allow you to copy a possible
            range of up to &HFFF0 bytes with a single REP. When you get to the end of
            the range of a single REP, if there's still more copying to be done, you'll
            need to adjust the segment ffset addresses to point to the next block.

            It may be faster to handle DWORDs or WORDs at a time, rather than just BYTEs.
            In that case, it would be wise to adjust the address(es) to WORD or DWORD
            alignment before starting off on the REP: most CPUs will give you a speed
            boost that way.


            ------------------
            Tom Hanlin
            PowerBASIC Staff

            Comment


            • #7
              Thanks for the tips, Tom.

              I am going to leave my POSTED code as-is. MY production code is
              actually an OBJ file, which I chose for size of the compiled module.
              Also, my include-in-everything INC file with all of my DECLARE's
              has the Destination and Source arguments declared AS ANY, so I do
              not have to use the BYVAL VARPTR32() - can use the variable in standalone
              fashion (except for dynamic strings & code pointers - those still
              require BYVAL STRPTR32() and BYVAL CODEPTR32()).


              ------------------

              Comment


              • #8
                Tom,

                Just re-read your post.

                In my test coding, which was in the OBJ file, I used a cheesy method
                for going beyond the range of an unsigned integer:

                Code:
                    P1A:
                    les di, dword ptr [bp + 0ah]
                    mov al, es:[di]
                    les di, dword ptr [bp + 0eh]
                    mov es:[di], al
                    inc dword ptr [bp + 0ah]
                    inc dword ptr [bp + 0eh]
                    dec dword ptr [bp + 06h]
                    jnz P1A


                You should note that I have 486/387 modes enabled in my MASM
                compiler.



                ------------------

                Comment


                • #9
                  So what's wrong with PEEK$ and POKE$?

                  Those 'move memory' just fine. Always have, and I suspect always will.

                  Ok, so a 'block' to be moved in a single instruction (BASIC statement) is limited to the current $STRING setting. Big deal, as you have to deal with a 64K max 'block' size under MS-DOS anyway.

                  Oops! How silly of me to forget! Using assembly language is <U>cool</U>.
                  Never mind.

                  MCM
                  Michael Mattias
                  Tal Systems Inc. (retired)
                  Racine WI USA
                  [email protected]
                  http://www.talsystems.com

                  Comment


                  • #10
                    PEEK$/POKE$ would be simpler and more efficient, but an understanding
                    of assembly language is priceless.

                    ------------------
                    Tom Hanlin
                    PowerBASIC Staff

                    Comment


                    • #11
                      going beyond the range of an unsigned integer
                      LES wasn't an opcode I used often, and maybe I'm foggy on the details,
                      but I'm not sure you've improved the range, here. I'm guessing that
                      you're only incrementing the memory offset, not the segment, and haven't
                      increased the range at all...

                      Reloading all data from base memory, every time, would be inefficient
                      in any event. What you want to do, ideally, is put all the calculations
                      up front, and leave the "inner loop" part of the code to do as little as
                      possible. Think REP, then registers, then [registers], then (last) cases
                      like [register + number] addressing. Segment overrides (e.g., ES should
                      also be avoided whenever practical.


                      ------------------
                      Tom Hanlin
                      PowerBASIC Staff

                      Comment

                      Working...
                      X