Announcement

Collapse
No announcement yet.

Array Size?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Array Size?

    Is there a maximum array size one needs to deal with in PB8? I tinkered with compiling a DIM statement defining a string array and kept bumping it up until it seemed unreasonably large and PB never did complain that the array would be too large to manage.

    (e.g. I DIMed a 3 dimensional string array 2,000 elements on a side - something like 8,000GB of data if I didn't slip a dismal point. And a lot bigger than my computer is.)

    The problem I'm really considering is to move an entire large Excel workbook into PB (via VBA call & DLL) to do some intensive calculation and then back to Excel with the result. This would be as an alternative to moving individual sheets to PB repeatedly. I was trying to understand how large an array PB could reasonably handle.

    Thanks.

    Bill

  • #2
    Bill,
    did you check the ERROR value after you dimensioned your long array?

    Paul.

    Comment


    • #3
      Obviously not. I *presumed* the compiler would tell me if there were an error. Perhaps a bad assumption.

      And how does one check the ERROR value when compiling to a DLL? I haven't found any way to step through the code as it actually executes when it's compiled to DLL. Obviously I don't have much PB experience.

      (Ok, I added an integer variable to the call parameters and can check the ERR flag when it executes. Apparently space is only allocated at run time and not checked at compile time.)

      Bill
      Last edited by William Martin; 13 Oct 2008, 03:12 PM.

      Comment


      • #4
        I *presumed* the compiler would tell me if there were an error. Perhaps a bad assumption.
        Well it does but perhaps not the way you were expecting.

        The compiler will set the system ERR variable on an allocation error (array too big), but you will have to check that value to know if the DIM/REDIM succeeded or not. If ERR does not equal zero, something bad happened. The ERROR$() function will give you the error in text, saving you a trip to the help file.

        Apparently [array ?] space is only allocated at run time and not checked at compile time.
        True.
        Michael Mattias
        Tal Systems (retired)
        Port Washington WI USA
        [email protected]
        http://www.talsystems.com

        Comment


        • #5
          Following up, I set up the PB routine to take a parameter from the Call, and dimension a 3D array to that size.

          And then I ran an Excel VBA routine, calling the PB DLL, which keeps bumping up the array size by 1% until an error code is returned by PB.

          When I DIM the array as LONG, it fails at a max dimension of 502. Curiously though, if I DIM the array as STRING, it fails at a max dimension of 509.

          Since STRING takes a lot more space than LONG, can I presume that if I dimension a large STRING array it may get through the DIM ok, but still fail later on when the array elements are filled with strings? And how will it fail? Just not do some of the writes, or crash the program, or write over top of other stuff or what?

          Do I have to make PB check ERR every time I fill an array element? Or how does one normally deal with this?

          Thanks.

          Bill

          Comment


          • #6
            Hi William,
            This may help with tracking down the array size <code not shown> so you will need to piece my error handling inc file into your program.
            RunTime Debug and Error Handling Part I - Find that elusive bug from the source code forum.

            Oh and use "#DEBUG ERROR OFF" so all errors are executed
            Engineer's Motto: If it aint broke take it apart and fix it

            "If at 1st you don't succeed... call it version 1.0"

            "Half of Programming is coding"....."The other 90% is DEBUGGING"

            "Document my code????" .... "WHYYY??? do you think they call it CODE? "

            Comment


            • #7
              >And how will it fail?

              Error 7 (memory allocation error).

              Dynamic string arrays require four (4) bytes per element at DIM/REDIM time, but the space for the data are not allocated until you assign a value to that element. (or in your case, "try" to assign a value).

              For all other arrays (except maybe variants?), all space which will be required is allocated at execution of the DIM/REDIM statement.

              MCM
              Michael Mattias
              Tal Systems (retired)
              Port Washington WI USA
              [email protected]
              http://www.talsystems.com

              Comment


              • #8
                Bill,

                One tried and true stub your toe seat of the pants trial and error crude semi-couth method to test beforehand is to '
                Code:
                'PBWIN 9.00 - WinApi 05/2008 - XP Pro SP3
                #Compile Exe                                
                #Dim All 
                #Include "WIN32API.INC"
                #Include "COMDLG32.INC"
                 
                Function PBMain
                    Local Ans, Array_Bounds_Check, Add_on As Long
                    Add_on = 10
                    Array_Bounds_Check = 100 'okay to start
                Add_More:  
                    ReDim Arry(Array_Bounds_Check, Array_Bounds_Check, Array_Bounds_Check) As String * 100
                    Arry(Array_Bounds_Check, Array_Bounds_Check, Array_Bounds_Check) = "Testing"
                    Ans = MsgBox (Using$( _ 
                          "Testing Array Sizes" & $CrLf & _
                          "#, is okay", Array_Bounds_Check), %MB_YESNO, "Click Yes")
                 
                    Array_Bounds_Check = Array_Bounds_Check + Add_on
                 
                    If Ans = %IdYes Then GoTo Add_More      
                 
                End Function
                '
                Obviously you can change the Starting & Add On variables to anything you want (1, 10, 25, ...)

                My machine chokes out (GPF's) between 230 & 240 (for Array_Bounds_Check) when Dimming String * 100. Much higher (1100+) when using a string size of 1. Between 700 - 800 when As Long.

                Someone (avec programming elegance) could easily add a memory check in there too (if he wanted). And if someone wanted she could add an Error Check inside a Loop as well.

                =======================================
                "There are only two tragedies in life:
                one is not getting what one wants,
                and the other is getting it."
                Oscar Wilde (1854-1900)
                =======================================
                Last edited by Gösta H. Lovgren-2; 13 Oct 2008, 07:53 PM. Reason: Crystal Clear Clarity
                It's a pretty day. I hope you enjoy it.

                Gösta

                JWAM: (Quit Smoking): http://www.SwedesDock.com/smoking
                LDN - A Miracle Drug: http://www.SwedesDock.com/LDN/

                Comment


                • #9
                  Bill,
                  I just tried this on 3 computers I have here.
                  My main PC (512MB ram, loads of disk) allocated 1.8GB before failing.
                  A laptop with 256MB RAM, loads of disk, allocated 900MB
                  An ASUS EeePC with 512MB RAM and 4GB hard disk allocated the full 2GB.

                  All run WinXP SP2 or 3.

                  I'm sure I could increase the swap file on the first 2 so they'd match the Asus.

                  So 2GB isn't unreasonable to expect.

                  That's quite a lot of data for one speadsheet. Are you sure it's that big?
                  Also, for speed you need to keep the data out of the swap file if you can so it might well be faster to process one page at a time in real memory than load it all and have it swapping in and out of virtual memory.

                  Paul.

                  Comment


                  • #10
                    Thanks for the solution Gosta. It will take me awhile to digest it.

                    No Paul, my spreadsheet is not that big. (Maybe 15MB depending on how long the actual strings are.) Still I get nervous about PB exhibiting an amorphous failure mode that may pop up unexpectedly if I don't fully understand it. For example I haven't come to terms yet with whether my program would seem to run ok now, but not later if XP's loading changes. Yet constantly checking ERR seems a bit paranoid.

                    Or for that matter, can the program lock up in an infinite loop if it runs out of space (depending on how the code is written)? I just don't know. It's not a failure mode I'm familiar dealing with. Just chewing over the possibilities at present.

                    Thanks all... I'll retreat to my cave for awhile.

                    Bill

                    Comment


                    • #11
                      There are two ways to look at the biggest array size: the biggest that can be allocated at runtime, and the largest range of acceptable subscripts.

                      This actually does compile in PBCC5:
                      DIM A(0 TO 999999999999999999999999999999999999999999999999999999999999999999999999999) AS LONG

                      I am not going to investigate the actual object code, but it does seem unlikely the compiler actually understood this as an array with far over a decillion elements.

                      Looks like a new wishlist item would be documented max LBOUND/UBOUND values and (when possible) compile-time bounds checking.
                      Erich Schulman (KT4VOL/KTN4CA)
                      Go Big Orange

                      Comment


                      • #12
                        That's quite a lot of data for one speadsheet. Are you sure it's that big?
                        Part II of this question: Do you really need to load ALL the data into memory at one time?

                        Can you maybe get whatever it is you are doing to work by,,,,,

                        1. Only loading 'n' rows at a time?

                        2. Treating the spreadsheet as a database and manipulating with SQL statements?

                        3. Unloading the spreadsheet contents to a temporary disk file, 'doing your thing' and putting it back?

                        Lots of options. Application-specific. YMMV. Void where prohibited.

                        MCM
                        Michael Mattias
                        Tal Systems (retired)
                        Port Washington WI USA
                        [email protected]
                        http://www.talsystems.com

                        Comment


                        • #13
                          >Yet constantly checking ERR seems a bit paranoid.

                          ON ERROR GOTO....
                          Michael Mattias
                          Tal Systems (retired)
                          Port Washington WI USA
                          [email protected]
                          http://www.talsystems.com

                          Comment


                          • #14
                            Erich --

                            > but it does seem unlikely the compiler actually understood
                            > this as an array with far over a decillion elements

                            It wouldn't "understand" that until runtime, when it attempted to dim the array and failed. To do it at compile time the compiler would have to guess how much memory would be available at runtime. How much memory will a typical computer have in 2018, when the program might be executed? So when should a compile-time error be flagged in 2008?

                            After all, a decillion bytes should be enough for anybody, right?

                            William --

                            > Yet constantly checking ERR seems a bit paranoid.

                            Not checking ERR after performing an operation that can fail due to a runtime limitation is... what's the opposite of paranoid?

                            -- Eric
                            "Not my circus, not my monkeys."

                            Comment


                            • #15
                              >what's the opposite of paranoid

                              In context? Silly.
                              Michael Mattias
                              Tal Systems (retired)
                              Port Washington WI USA
                              [email protected]
                              http://www.talsystems.com

                              Comment


                              • #16
                                Originally posted by Michael Mattias View Post
                                Part II of this question: Do you really need to load ALL the data into memory at one time?
                                No, but I'm trying to understand the trade-off between space and speed. I can load and operate on one sheet at a time, scan through all the sheets, and then go back to the beginning an reload the first sheet again. Loop, loop, loop... That involves thrashing a lot of the same data repeatedly between Excel and PB.

                                Or I can move the whole thing to PB at once and execute on it a few hours with PB and move it back again once. (It's a successive approximation of a non-linear and non-continuous problem.)

                                I'm just trying to understand the programming constraints and trade-offs before writing a ton of code.

                                Bill

                                Comment


                                • #17
                                  Bill,
                                  the "page at a time" suggestion was with 8GB of data in mind. Since you now say it's only 15MB then there is no problem loading the whole thing at once.
                                  On a 10 year old computer I'd think nothing of creating 100MB arrays. 15MB really isn't a problem on a modern PC.

                                  Paul.

                                  Comment


                                  • #18
                                    > That involves thrashing a lot of the same data repeatedly between Excel and PB.

                                    Well, I don't know how Excel does it, but "thashing" data between PB functions only involves a couple of integers (the data addresses).

                                    Maybe you could do the whole job within Excel... but writing some functions, putting them in a Dynamic Link Library and using that in some cell formulae.

                                    MCM
                                    Michael Mattias
                                    Tal Systems (retired)
                                    Port Washington WI USA
                                    [email protected]
                                    http://www.talsystems.com

                                    Comment


                                    • #19
                                      Paul is right. A 15 mb speadsheet is nothing to Load unless it's a ton of cells.

                                      But that aside, I must be missing something here though. How can one check for an Error before Dimming an array (as in the sample snippet above). An "over dim" causes a GPF, not a checkable Error per se.

                                      Bill, when writing this it occurs to me, if too many cells in an array is a problem, why not break it down into several arrays? Not as convenient programmically as having all the data in one 3 dim array, but no more Dim GPFs or memory constraints (effectively anyway).

                                      ==========================
                                      He that can have patience
                                      can have what he will.
                                      Ben Franklin
                                      ==========================
                                      Last edited by Gösta H. Lovgren-2; 14 Oct 2008, 03:53 PM.
                                      It's a pretty day. I hope you enjoy it.

                                      Gösta

                                      JWAM: (Quit Smoking): http://www.SwedesDock.com/smoking
                                      LDN - A Miracle Drug: http://www.SwedesDock.com/LDN/

                                      Comment


                                      • #20
                                        Originally posted by Gösta H. Lovgren-2 View Post
                                        But that aside, I must be missing something here though. How can one check for an Error before Dimming an array (as in the sample snippet above).
                                        Explicit allocation (like GLOBALMEM ALLOC) will return an error instead of GPFing.

                                        When you have the address of the allocated memory, you can DIM your array AT that address.

                                        Comment

                                        Working...
                                        X