Announcement

Collapse
No announcement yet.

memory allocation with PBDLL

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • memory allocation with PBDLL

    Some questions:

    1) Is the memory space available to a program dependent on the
    physical RAM present, or will Window's memory manager allocate
    virtual memory if my app needs more than what's physically available?

    2) Is there a fixed limit to the memory space available to GLOBAL vars/arrays?
    I'm getting error 9s & 7s currently as a result of dimensioning my global
    arrays while also allocating over 5MB for (GLOBAL) fixed length strings.

    3) Can GLOBAL arrays be dimensioned at an address (overlaying fixed length
    strings)? If so, can they be redimmed (PRESERVE) at the same address?



    ------------------
    Bernard Ertl
    Bernard Ertl
    InterPlan Systems

  • #2
    1) Short answer: "yes". Short ramble: Depends on what kind(s) of memory you're
    talking about, what version(s) of Windows, what custom settings, what API
    functions might be involved... It is not possible to give a good short answer.
    Maybe someone will feel ambitious enough to write an essay on the subject or
    can provide a good web link!

    2) No fixed limits that I can think of under the circumstances.

    3) Dunno, have you tried it? Seems like it should work. There would be no need
    to REDIM PRESERVE, because DIM AT doesn't initialize a memory area. It's more or
    less like setting a pointer.


    ------------------
    Tom Hanlin
    PowerBASIC Staff

    Comment


    • #3
      Hi Tom,

      Sorry, sometimes my mind gets way ahead of my fingers. I'll try
      to clarify Q1:

      1) Is the memory space which PB uses when allocating vars & arrays
      using DIM limited to the amount of free physical RAM available, or
      does Windows & PB allow us to DIM vars & arrays beyond the limits
      of physical RAM (and thus invoke the virtual memory manager)?

      On to a puzzle --> Q2...

      I've got an app that has been running stable for several months.
      This morning I added :

      Code:
      GLOBAL MyArraySpace1  AS STRING * 800000
      GLOBAL MyArraySpace2  AS STRING * 800000
      GLOBAL MyArraySpace3  AS STRING * 800000
      GLOBAL MyArraySpace4  AS STRING * 800000
      GLOBAL MyArraySpace5  AS STRING * 800000
      GLOBAL MyArraySpace6  AS STRING * 800000
      GLOBAL MyArraySpace7  AS STRING * 800000
      In order to overlay some LONG arrays within one of my functions.
      Adding those lines caused the initialization of my GLOBAL arrays
      (28 of mixed types: LONGS, UDTs & Variable Length Strings to 30,000
      elements each) to produce error 9. I did track the source of the
      error to the dimensioning of the arrays.

      I reduced the 800,000 to 100,000 for each of the above MyArraySpace#
      declarations and my GLOBAL arrays generated error 7. I further
      reduced the 100,000 to 50,000 and suddenly everything worked again.
      Any ideas why I might experience this?

      I'm testing on a Win95 with 32MB RAM (bare bones).

      Q3--> No, I didn't try it. I have a small memory problem to fix first.
      I hope to try it if I am able to dimension enough fixed length string space.


      ------------------
      Bernard Ertl

      [This message has been edited by Bern Ertl (edited April 24, 2001).]
      Bernard Ertl
      InterPlan Systems

      Comment


      • #4
        Bern
        I suspect you are running out of disk space not memory.
        Using 98 SE I loaded 20 10,000,000 byte fixed len global string with no problem on my 192 Meg machine
        I don't have any 95 machines left to test but do seem to remember that NT4 has a fixed size swap file.
        Have you tried running System Monitor while loading the program to see whats happening.


        ------------------

        Comment


        • #5
          1. The memory available to your application (well, each process) is the sum of RAM and free disk space; Windows(r) will handle what it uses transparently to the application. Other than a couple of cheap-n-dirties, I've never written an application which uses as much memory as you are, but all my test programs worked OK.

          2. If you are running Win98, there is a known problem with memory fragmentation which appears as "insufficient memory" to PB. I did not see which OS you are using, but that might be a consideration as well.

          3. You pointed out you had tracked your error #9s to the array dimensioning, so that's good; but since error 9 is a subscript error and not a memory error, you should probably be tracking subscripts at run-time as well.
          If it does not cause a GPF, a bad subscript can cause one of those errors which don;t show up until elsewhere in the program.

          MCM



          [This message has been edited by Michael Mattias (edited April 25, 2001).]
          Michael Mattias
          Tal Systems Inc. (retired)
          Racine WI USA
          [email protected]
          http://www.talsystems.com

          Comment


          • #6
            Bern,

            I have a sneaking suspicion that the problem is simply running
            out of available ram on the machine. From memory, PB uses
            GlobalAlloc() for these types or operations and this function does
            not swap extra memory requirements to disk.

            I would be tempted to try normal dynamic string as it is supplied
            by the OLE string engine and it may have a better strategy for handling
            high memory usage.

            Regards,

            [email protected]

            ------------------
            hutch at movsd dot com
            The MASM Forum

            www.masm32.com

            Comment


            • #7
              I checked my HD and there is 830MB free. That should be more than
              enough if Windows is using it all.

              I bumped the STRINGs allocations back to 100,000 to generate
              the error 7. I tried running while the system monitor was active.
              I've never used the system monitor before and I'm not quite sure
              what to look for.

              I monitored the Free Memory, Other Memory, Allocated Memory &
              Disk Cache Size.

              The before value indicates strings allocated, arrays about to be
              dimensioned.

              The after value indicates right after error is generated (before
              app termination).

              Code:
                           Before REDIM Error    After REDIM Error
              Free Memory :         6,639,616              245,760          
              
              Other Memory :       31,846,400           30,695,424    
              
              Allocated Mem :      56,754,176           66,347,???    
              
              Cache Size :          6,197,248            5,???,???
              ??? --> #'s changed before I could record them.

              John,
              The memory allocation for the strings do not cause the error.
              Redimensioning the GLOBAL arrays in my function causes the errors.

              Michael,
              Both the error 9 & error 7 occur at exactly the same place, when
              REDIMming my GLOBAL arrays. The only changes I made were to the
              sizes of the strings. I'm not sure why REDIM would cause an error 9,
              but that's what I'm getting when the strings are "super huge" as opposed
              to just "huge".

              I placed error checking just before & just after the REDIM statements
              to confirm that the errors are generated there.



              ------------------
              Bernard Ertl
              Bernard Ertl
              InterPlan Systems

              Comment


              • #8
                Steve, we posted at the same time...

                It was my suspicion that PB was only using physical RAM too.
                Thus my initial question. I need to know for sure because this will
                be a determining factor as to the design of my app's memory structure.

                Unfortunately, dynamic strings won't work for me (see the relearning
                ASM basics thread in the Programming Forum)....



                ------------------
                Bernard Ertl
                Bernard Ertl
                InterPlan Systems

                Comment


                • #9
                  Bern,

                  I just had a quick read through that thread and it seemed that
                  your problem was related to running out of registers. In the crude
                  sense, memory is memory, OLE memory can be written to and read
                  from like any other method of managing memory.

                  The problem in part seems to be the amount of memory you require
                  which from the example its 800k x 7 which is about 5.6 meg.

                  I use OLE string memory in MASM regularly and it has no problems at
                  all. Once you have the start address, you can do what you like
                  with it. There are a couple of things from that thread, you can
                  use any of the 6 available registers for anything you like, you
                  are not required to use any particular register for base addressing
                  or similar as it used to be done in DOS 16 bit code.

                  You can also forget segments as they must not be used in 32 bit flat
                  memory model. I know that working in limited size machines poses
                  some very interesting problems and it may be that the magnitude of
                  memory usage you require will need an entirely different strategy
                  on a limited size machine.

                  Regards,

                  [email protected]

                  ------------------
                  hutch at movsd dot com
                  The MASM Forum

                  www.masm32.com

                  Comment


                  • #10
                    Bern
                    On my 192 Meg 98 machine I allocated as follows
                    Code:
                    GLOBAL S1 AS STRING * 10000000
                    GLOBAL S2 AS STRING * 10000000
                    GLOBAL S3 AS STRING * 10000000
                    GLOBAL S4 AS STRING * 10000000
                    GLOBAL S5 AS STRING * 10000000
                    GLOBAL S6 AS STRING * 10000000
                    GLOBAL S7 AS STRING * 10000000
                    GLOBAL S8 AS STRING * 10000000
                    GLOBAL S9 AS STRING * 10000000
                    GLOBAL S10 AS STRING * 10000000
                    GLOBAL S11 AS STRING * 10000000
                    GLOBAL S12 AS STRING * 10000000
                    GLOBAL S13 AS STRING * 10000000
                    GLOBAL S14 AS STRING * 10000000
                    GLOBAL S15 AS STRING * 10000000
                    GLOBAL S16 AS STRING * 10000000
                    GLOBAL S17 AS STRING * 10000000
                    GLOBAL S18 AS STRING * 10000000
                    GLOBAL S19 AS STRING * 10000000
                    GLOBAL S20 AS STRING * 10000000
                    GLOBAL L1() AS LONG
                    GLOBAL L2() AS LONG
                    TYPE TT
                        l AS LONG
                        b AS BYTE
                        s AS STRING *10
                    END TYPE
                    GLOBAL T1() AS tt
                    FUNCTION PBMAIN
                        REDIM L1(20000000)
                        REDIM L2(50000000)
                        REDIM T!(1000000)
                    I then single stepped it in the debugger
                    SYstem Monitor showed the strings taking available physical memory down from 122meg to 0 and the swap file up from 100meg to 310meg the redim's in code then merely increased the swap file to 590meg no errors. It must be a 95 problem or something else. Have solved the problem I had with self modifying code, unfortunately its quarterly tax time here so am a bit busy but will try to post sample code over the weekend.
                    Hutch
                    I think you are missing the point that he needs string addresses at compile time so they must be fixed length, otherwise he could just use dynamic arrays


                    ------------------

                    Comment


                    • #11
                      Steve,

                      John is correct. The whole point of using the strings was to get
                      an offset address for the memory space, not a pointer to it. The
                      goal is/was to overlay LONG arrays and be able to reference the elements
                      with a single instruction (not having to preload the array address into
                      a register).

                      John,

                      You may be right (about Win95). I'm treading new ground here and
                      I'm not sure I understand what's happening to the memory on my test
                      machine.

                      I'll be eagerly awaiting your self modifying code sample when you
                      get a chance to post it. I think it will be a better solution overall
                      as I won't need to tie up so much memory for a max case scenario or
                      impose artificial limits. Thanks.



                      ------------------
                      Bernard Ertl
                      Bernard Ertl
                      InterPlan Systems

                      Comment


                      • #12
                        An error 7 means that a contiguous block of memory of the requested size could not be allocated. If your code does not "handle" the Error 7 condition, and you continue to reference the array (with #DEBUG ERROR ON) you'll get an Error 9 if the subscript is outside of the array bounds. Simply put, the problem is not PowerBASIC, but something related to your O/S. If this error only happens after the app has been running for some time, then it could be a memory fragmentation problem (and you can point rhe finger at Microsoft for that one!).

                        I'm completely unclear as to why you need an "offset" rather than an pointer. If you are overlaying a numeric array over a (static or global) fixed-length string using DIM..AT, computing the 'offset' from the base string address (VARPTR) is simple maths. Offset = subscript * (SIZEOF(subscript) - 1) ' assuming zero based array, and the base address is the address of the string, say, VARPTR(fixedstring)

                        ------------------
                        Lance
                        PowerBASIC Support
                        mailto:[email protected][email protected]</A>
                        Lance
                        mailto:[email protected]

                        Comment


                        • #13
                          Hi Lance,

                          I'm running under the assumption that it is an O/S problem right
                          now. Either way, I can't be sure of the O/S or condition of my end
                          users' machines, so if it (the memory allocation structure that I
                          was trying) is "flaky" on my machine, I'm not planning on using it.

                          Using an offset vs. a pointer would allow me to make optimizations
                          to a critical calculations function. I would only need one ASM
                          instruction per array reference rather than two.

                          Code:
                          MyArrayPtr = VARPTR( MyArray(0))
                          ...
                          !mov  eax,  MyArrayPtr      ; Load array address
                          !mov  eax,  [eax + ecx*4]   ; Load value of element 'ecx'
                          vs.

                          Code:
                          GLOBAL MyArrayOffset AS STRING * %MaxArraySize ' = MaxArrayElem * ElemSize
                          ...
                          DIM MyArray( %MaxArrayElem) AT MyArrayOffset
                          ...
                          !mov  eax,  MyArrayOffset[ecx]  ; Load value of element 'ecx\4'


                          ------------------
                          Bernard Ertl
                          Bernard Ertl
                          InterPlan Systems

                          Comment


                          • #14
                            Bern,
                            Code:
                            !mov  eax,  MyArrayPtr      ; Load array address
                            !mov  eax,  [eax + ecx*4]   ; Load value of element 'ecx'
                            If you have enough registers available in your algorithm, I would
                            be inclined to use,
                            Code:
                                ! mov edx, MyArrayPtr
                                ! mov eax, [edx+ecx*4]
                            as it reduces one dependency in the EAX register.
                            Code:
                                !mov  eax,  MyArrayOffset[ecx]  ; Load value of element 'ecx\4'
                            This is simpler code but with pairing, it may not work any faster so
                            that advantage may not be worth chasing.

                            Regards,

                            [email protected]


                            ------------------
                            hutch at movsd dot com
                            The MASM Forum

                            www.masm32.com

                            Comment


                            • #15
                              Steve,

                              I don't have enough registers. I need to reference elements in six
                              different arrays while reserving the esi & edi registers (ASM is
                              mixed with PB & they are used for quick data transfer).

                              The offset method is quicker than two !mov's (at least on my machine).
                              I've already tested for that.



                              ------------------
                              Bernard Ertl
                              Bernard Ertl
                              InterPlan Systems

                              Comment

                              Working...
                              X