Announcement

Collapse
No announcement yet.

Saving An Entire Array At Once ?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Saving An Entire Array At Once ?

    Hello Folks, I am trying to save array DATA_1?(0:32770) as fast as possible to file "a.dat" and would like to hear feedback in how to do this. So far I have found the BSAVE command the very best but think there might be another way such as using assembly language for arrays bigger than 65536 bytes.
    BSAVE happened in less time then a TIMER tic but all other ways involving opening files took fraction of seconds.

    IS there any way to save an entire VIRTUAL array with a single command and not using contiguous loops ?
    :wavey:
    Contact: gmvoeth(AT)hotmail(dot)com nospam

  • #2
    Nothing is going to be faster than BSAVE.

    It is not worth the time to do it in assembly language... why not? Because BSAVE is compiled into what the PB folks think is prett-good assembly language for saving an array to disk

    You can't save any array > 64K under MS-DOS in one shot because MS-DOS can't handle more than 64k in one shot. (16 bit registers, you know). Whatever you do or the compiler does for you, be assured "somewhere" it's broken up into chunks of 64 K or less.

    FWIW..
    >BSAVE happened in less time then a TIMER tic

    If you need to save a whole array more than 18 times per second, you need some serious design assistance.

    MCM
    Michael Mattias
    Tal Systems Inc. (retired)
    Racine WI USA
    [email protected]
    http://www.talsystems.com

    Comment


    • #3
      Originally posted by Michael Mattias View Post
      Nothing is going to be faster than BSAVE.

      It is not worth the time to do it in assembly language... why not? Because BSAVE is compiled into what the PB folks think is prett-good assembly language for saving an array to disk

      You can't save any array > 64K under MS-DOS in one shot because MS-DOS can't handle more than 64k in one shot. (16 bit registers, you know). Whatever you do or the compiler does for you, be assured "somewhere" it's broken up into chunks of 64 K or less.

      FWIW..
      >BSAVE happened in less time then a TIMER tic

      If you need to save a whole array more than 18 times per second, you need some serious design assistance.MCM
      <====

      There just no need for such comments as that last line you injected the human wants and desires are unlimited even if unreasonable.
      Other than that your comments seem reasonable.
      Memory should be straight line do you know of an operating system like MSDOS that will handle all RAM as straight line memory like in the old DIGITAL EQUIPMENT CORPORATION PDP1140 series industrial control computers ???

      There exists the possibility that the BSAVE routine first disables all interrupts prior to executing its main routine and that is why no tic counter tics are counted ?? It is my understanding that the CPU is resposibl through interrupts to maintain the seconds after midnight but that it is possible to issue a command to stop all such interrupts and dedicate the machine fully to any task the programmer desires.
      Last edited by Geoffrey Voeth; 9 May 2008, 12:16 PM.
      Contact: gmvoeth(AT)hotmail(dot)com nospam

      Comment


      • #4
        Originally posted by Geoffrey Voeth View Post
        <====
        do you know of an operating system like MSDOS that will handle all RAM as straight line memory like in the old DIGITAL EQUIPMENT CORPORATION PDP1140 series industrial control computers ???
        It sounds like FreeDOS-32 is what you're looking for, other than it being far from ready for production use. And I am assuming Linux or *BSD would not meet your needs.
        Erich Schulman (KT4VOL/KTN4CA)
        Go Big Orange

        Comment


        • #5
          Originally posted by Erich Schulman View Post
          It sounds like FreeDOS-32 is what you're looking for, other than it being far from ready for production use. And I am assuming Linux or *BSD would not meet your needs.
          For Goodness Sake that idea is three years old and no more progress is indicated it seems there is no serious effort for anyone to make an operating system with unlimited straight line memory.
          Contact: gmvoeth(AT)hotmail(dot)com nospam

          Comment


          • #6
            Memory should be straight line do you know of an operating system like MSDOS that will handle all RAM as straight line memory ..
            (Assuming by "straight line" you mean "linear addressing")

            If it did, then it wouldn't be like MS-DOS. MS-DOS uses a 20-bit "paragraphffset" address, which is only linear within each 64K block which starts from offset zero.
            Michael Mattias
            Tal Systems Inc. (retired)
            Racine WI USA
            [email protected]
            http://www.talsystems.com

            Comment


            • #7
              But Windows/32 is kind of like MS-DOS, and that DOES use linear addressing within the entire 32-bit range from x000000 to xFFFFFFFF

              MCM
              Michael Mattias
              Tal Systems Inc. (retired)
              Racine WI USA
              [email protected]
              http://www.talsystems.com

              Comment


              • #8
                it is possible to issue a command to stop all such interrupts and dedicate the machine fully to any task the programmer desires
                Yes, it is possible.

                See the Intel x86 (or whatever chip you are working with) programming reference for details.
                Michael Mattias
                Tal Systems Inc. (retired)
                Racine WI USA
                [email protected]
                http://www.talsystems.com

                Comment


                • #9
                  do you know of an operating system like MSDOS that will handle all RAM as straight line memory
                  It can be done in MSDOS itself, if you are careful. Do a Google search for "real mode flat memory". However, I never tried it, because I am not a very careful programmer... And it would not serve your purpose of saving an array of more than 64 KB, because for that you would still have to use the DOS services which are programmed for writing a maximum of 64KB(-1) at once.

                  Comment


                  • #10
                    Intel imposed the 64k addressing boundary on their early CPU architectures, because somebody decided that nobody would ever need more than 1 MB or RAM, ever. So instead of having the segment addresses do a full offset from the addressing range allowed with 16 bits, they caused an overlap where only four addressing bits were actually appended. Some old code used some carefully crafted segment+address math to obscure jump locations or table lookup values, trying to protect their code from prying eyes.

                    MSDOS merely followed the hardware design. Later, when they wanted to address more memory, they used tricks with address line 20 to map in a different memory chunk at the very top end of the memory range, and we saw new drivers under DOS to allow us to address expanded and extended memory. People never really got a handle on which was which.

                    With Windows, Microsoft finally decided to take over managing memory and try to make it transparent to the user. The expansion of the original architecture design added instructions that could actually address a full 32-bits of memory range, and Windows was able to take advantage of this. DOS, however, was designed for the older architecture, and though it could run on the newer platforms, it still had the same limitations, because it still relied on the old 16-bit design and legacy addressing modes.

                    So the original shortsightedness was probably due to some director over an engineering department with Intel who decided that 1 MB of RAM was all that was ever going to be needed, and saw this as a way to keep the chip die small and not require ever more complex pin arrangements to support all the addressing lines that would be needed otherwise. The advances in chip design and multilayer fabrication processes would eventually prove him wrong, but that does happen. But Bill Gates once thought
                    that 64k was all the RAM that would ever be needed, so at least he was in good company.

                    Every time an interrupt is generated, the first thing that usually happens is that further interrupts are inhibited immediately. Some very short processes or time critical processes may complete their tasks before they permit further interrupts. Others, willing to give way to other possible interrupts, and may enable interrupts almost immediately, thus permitting some other process to preceed them if need be. You cannot tell which method is used by any one process without examining the code involved.

                    Comment


                    • #11
                      Not sure how you're using the BSAVE command, nor how fast the following works with that large of an array, but I used this technique a lot back in my DOS days (even a time or two in Windows) and found it to be darn good for my needs. Sure glad I never delete any of these "old" tools
                      Code:
                      '                           ARRAYSAV.BAS
                      '        ---------------------------------------           Eric J. Pearson
                      '        ARRAY SAVE/LOAD ROUTINES FOR POWERBASIC 
                      '        ---------------------------------------             August 1991
                      '
                      ' These sample programs demonstrate a method for BSAVEing and BLOADing
                      ' numeric and flex string arrays (i.e. swapping data directly between memory
                      ' and disk) with PowerBASIC.  This technique is much faster than saving and
                      ' loading arrays as text files (i.e. creating data files with PRINT#), and
                      ' takes less program overhead.  And because the files are undelimited (i.e.
                      ' no CR/LF, commas, quotes, etc.) and because numeric values are stored in
                      ' "compressed" form (compared to storing numbers as text) the data files
                      ' take up much less disk space.  Finally, the technique allows numeric values
                      ' to be saved/loaded directly, without the relatively slow task of translation
                      ' to/from text.
                      '
                      ' Numeric variables of any type, and flex strings, in arrays with any number
                      ' of dimensions, can be saved and loaded with routines based on these samples.
                      ' (Sorry, but "normal" string arrays can't be BSAVEd or BLOADed.)
                      '
                      ' Generally speaking, a small "custom" sub is built for each array that is
                      ' to be saved/loaded, and a "standard" sub handles the actual bload/bsave...
                      '
                      ' INITIALIZATION:
                      
                      	'define variable types (*REQUIRED* values)...
                      	%IntegerVar =  2   'INTEGER (%)
                      	%LongIntVar =  4   'LONG INTEGER (&)
                      	%QuadIntVar =  8   'QUAD INTEGER (&&)
                      	%SinglePrec =  4   'SINGLE PRECISION (!)
                      	%DoublePrec =  8   'DOUBLE PRECISION (#)
                      	%ExtPrec    = 10   'EXTENDED PRECISION (##)
                      	%BCDFixed   =  8   'BCD FIXED POINT (@)
                      	%BCDFloat   = 10   'BCD FLOATING POINT (@@)
                      	%FlexString =  0   'FLEX STRING ($$)
                      
                      	'for convenience (arbitrary values)...
                      	%Load       =  1
                      	%Save       = -1
                      
                      ' DEMONSTRATION #1 (1-dimensional integer array):
                      
                      	'dimension the array
                      	DIM STATIC Array1%(256)
                      
                      	'load demonstration data into the array
                      	Array1%(1) = 10001
                      
                      	'save the array on disk
                      	CALL ArrayOne(%Save)
                      
                      	'clear the demo value from the array
                      	ERASE Array1%
                      
                      	'load the array from disk
                      	CALL ArrayOne(%Load)
                      
                      	'display the demonstration data
                      	PRINT Array1%(1)    'prints 10001 if everything worked right.
                      
                      ' DEMONSTRATION #2 (3-dimensional, double precision array):
                      
                      	DIM STATIC Array2#(1:64,0:10,0:2)
                      	Array2#(2,2,2) = 20000000.2
                      	CALL ArrayTwo(%Save)
                      	ERASE Array2#
                      	CALL ArrayTwo(%Load)
                      	PRINT Array2#(2,2,2) 'prints 20000000.2 if everything worked right.
                      
                      ' DEMONSTRATION #3 (one-dimensional, flex string array):
                      
                      	DIM STATIC Array3$$(0:1024)  '(each element to be 8 characters)
                      	'8 chars per element * 1025 elements = 8200 bytes, so...
                      	MAP Big3$$ * 8200
                      
                      	'map the array into the big string
                      	FOR Element% = 0 TO 1024
                      		ElBase% = (Element%*8)+1
                      		MAP Big3$$, FROM ElBase% TO ElBase%+7 AS Array3$$(Element%)
                      	NEXT
                      
                      	'load demonstration data into the array
                      	LSET Array3$$(16) = "TESTING!"
                      
                      	'save the array
                      	CALL ArrayThree(%Save)
                      
                      	'clear the demonstration data
                      	'(don't ERASE the array or the MAP will be lost)
                      	LSET Array3$$(16) = "        "
                      
                      	'load the array
                      	CALL ArrayThree(%Load)
                      
                      	PRINT Array3$$(16)  'prints "TESTING!" if everything worked right.
                      
                      	END
                      
                      SUB ArrayOne(Direction%)
                      
                      	'THIS IS A "CUSTOM" SUB, SET UP FOR A PARTICULAR ARRAY
                      
                      	SHARED Array1%()
                      	LOCAL  Segment&,Start&,Finish&,VarType%
                      
                      	Segment& = VARSEG(Array1%(0))   'first element
                      	Start&   = VARPTR(Array1%(0))   'first element
                      	Finish&  = VARPTR(Array1%(256)) 'last element
                      	VarType% = %IntegerVar
                      	CALL LoadOrSave("Array1",Direction%,Segment&,Start&,Finish&,VarType%)
                      
                      END SUB
                      
                      
                      SUB ArrayTwo(Direction%)
                      
                      	'THIS IS A "CUSTOM" SUB, SET UP FOR A PARTICULAR ARRAY
                      
                      	SHARED Array2#()
                      	LOCAL  Segment&,Start&,Finish&,VarType%
                      
                      	Segment& = VARSEG(Array2#(1,0,0))     'first element
                      	Start&   = VARPTR(Array2#(1,0,0))     'first element
                      	Finish&  = VARPTR(Array2#(64,10,2))   'last element
                      	VarType% = %DoublePrec
                      	CALL LoadOrSave("Array2",Direction%,Segment&,Start&,Finish&,VarType%)
                      
                      END SUB
                      
                      
                      SUB ArrayThree(Direction%)
                      
                      	'THIS IS A "CUSTOM" SUB, SET UP FOR A PARTICULAR ARRAY
                      
                      	SHARED Big3$$  'note that Array3$$() itself is not shared
                      	LOCAL  Segment&,Start&,Finish&,VarType%
                      
                      	'(note use of STRSEG and STRPTR, not VARSEG and VARPTR as above)
                      	Segment& = STRSEG(Big3$$)          'first byte of big string
                      	Start&   = STRPTR(Big3$$)          'first byte of big string
                      	Finish&  = Start& + LEN(Big3$$)    'last byte of big string
                      	VarType% = %FlexString             '
                      
                      	CALL LoadOrSave("Array3",Direction%,Segment&,Start&,Finish&,VarType%)
                      
                      END SUB
                      
                      
                      SUB LoadOrSave(FileName$,WhichWay%,SegmentAdr&,StartAdr&,FinishAdr&,WordLen%)
                      
                      	'THIS "STANDARD" ROUTINE IS USED BY ALL "CUSTOM" SUBS ABOVE
                      
                      	LOCAL Length&
                      
                      	FileName$ = FileName$ + ".ARR"  'MY DEFAULT EXTENSION FOR ARRAY FILES
                      
                      	Length&  = FinishAdr& - StartAdr& + WordLen%  'ARRAY LENGTH IN BYTES
                      
                      	DEF SEG = SegmentAdr&
                      
                      	IF WhichWay% = %Save THEN
                      		BSAVE FileName$,StartAdr&,Length&
                      	ELSE
                      		BLOAD FileName$,StartAdr&
                      	END IF
                      
                      	DEF SEG
                      
                      END SUB
                      
                      '------------------------------------------------------------- end of ARRAY.BAS
                      Software makes Hardware Happen

                      Comment


                      • #12
                        Wow, what memories!

                        VARSEG, STRSEG, DEF SEG, BLOAD, BSAVE,'$$' data types....

                        Ah, they just don't make keywords like they used to......
                        Michael Mattias
                        Tal Systems Inc. (retired)
                        Racine WI USA
                        [email protected]
                        http://www.talsystems.com

                        Comment


                        • #13
                          Coming From THEOS multi-user basic in the 80's, it was nice to see the addition of mat in the newer versions of pbwin and pbcc.

                          THEOS not only allows mat for numeric but also string arrays and
                          a MAT WRITE #file%[,RECORD KEY]:string_arrayname$ or numeric_arrayname

                          This would write the array to a file starting with item 1.

                          and then on the other side,
                          MAT READ #file%[,RECORD KEY]:arrayname
                          Client Writeup for the CPA

                          buffs.proboards2.com

                          Links Page

                          Comment


                          • #14
                            Talk about memories....and relevant, too!
                            Code:
                            [72030,3563] Michael Mattias
                            RPD.ZIP
                              Bin,  Bytes:    21248, Count:   141, 22-Aug-95
                            
                              Title   : REDIM PRESERVE plus Virtual Array Memory for PB
                              Keywords: ARRAY REDIM PRESERVE SAVE RESTORE SOURCE FREE 3.1
                            
                              Routines to allow PowerBASIC programmers to REDIM regular and
                              HUGE fixed-element-size arrays (including TYPE arrays) with PRESERVE and
                              use disk files as "virtual array memory." Includes Unit source, demo source
                              and documentation. Requires PB 3.1+. Copyright but free. From Michael
                              Mattias.
                            There are some 'save and restore' array routine in here which ain't all bad...
                            Attached Files
                            Michael Mattias
                            Tal Systems Inc. (retired)
                            Racine WI USA
                            [email protected]
                            http://www.talsystems.com

                            Comment

                            Working...
                            X