Saving An Entire Array At Once ?

[Geoffrey Voeth]

Hello Folks, I am trying to save array DATA_1?(0:32770) as fast as possible to file "a.dat" and would like to hear feedback in how to do this. So far I have found the BSAVE command the very best but think there might be another way such as using assembly language for arrays bigger than 65536 bytes.
BSAVE happened in less time then a TIMER tic but all other ways involving opening files took fraction of seconds.

IS there any way to save an entire VIRTUAL array with a single command and not using contiguous loops ?
:wavey:

[Michael Mattias]

Nothing is going to be faster than BSAVE.

It is not worth the time to do it in assembly language... why not? Because BSAVE is compiled into what the PB folks think is prett-good assembly language for saving an array to disk

You can't save any array > 64K under MS-DOS in one shot because MS-DOS can't handle more than 64k in one shot. (16 bit registers, you know). Whatever you do or the compiler does for you, be assured "somewhere" it's broken up into chunks of 64 K or less.

FWIW..
>BSAVE happened in less time then a TIMER tic

If you need to save a whole array more than 18 times per second, you need some serious design assistance.

MCM

[Geoffrey Voeth]

Nothing is going to be faster than BSAVE.

It is not worth the time to do it in assembly language... why not? Because BSAVE is compiled into what the PB folks think is prett-good assembly language for saving an array to disk

You can't save any array > 64K under MS-DOS in one shot because MS-DOS can't handle more than 64k in one shot. (16 bit registers, you know). Whatever you do or the compiler does for you, be assured "somewhere" it's broken up into chunks of 64 K or less.

FWIW..
>BSAVE happened in less time then a TIMER tic

If you need to save a whole array more than 18 times per second, you need some serious design assistance.MCM

<====

There just no need for such comments as that last line you injected the human wants and desires are unlimited even if unreasonable.
Other than that your comments seem reasonable.
Memory should be straight line do you know of an operating system like MSDOS that will handle all RAM as straight line memory like in the old DIGITAL EQUIPMENT CORPORATION PDP1140 series industrial control computers ???

There exists the possibility that the BSAVE routine first disables all interrupts prior to executing its main routine and that is why no tic counter tics are counted ?? It is my understanding that the CPU is resposibl through interrupts to maintain the seconds after midnight but that it is possible to issue a command to stop all such interrupts and dedicate the machine fully to any task the programmer desires.

[Erich Schulman]

<====
do you know of an operating system like MSDOS that will handle all RAM as straight line memory like in the old DIGITAL EQUIPMENT CORPORATION PDP1140 series industrial control computers ???

It sounds like FreeDOS-32 is what you're looking for, other than it being far from ready for production use. And I am assuming Linux or *BSD would not meet your needs.

[Geoffrey Voeth]

It sounds like FreeDOS-32 is what you're looking for, other than it being far from ready for production use. And I am assuming Linux or *BSD would not meet your needs.

For Goodness Sake that idea is three years old and no more progress is indicated it seems there is no serious effort for anyone to make an operating system with unlimited straight line memory.

[Michael Mattias]

Memory should be straight line do you know of an operating system like MSDOS that will handle all RAM as straight line memory ..

(Assuming by "straight line" you mean "linear addressing")

If it did, then it wouldn't be like MS-DOS. MS-DOS uses a 20-bit "paragraphffset" address, which is only linear within each 64K block which starts from offset zero.

[Michael Mattias]

But Windows/32 is kind of like MS-DOS, and that DOES use linear addressing within the entire 32-bit range from x000000 to xFFFFFFFF

MCM

[Michael Mattias]

it is possible to issue a command to stop all such interrupts and dedicate the machine fully to any task the programmer desires

Yes, it is possible.

See the Intel x86 (or whatever chip you are working with) programming reference for details.

[Hans Ruegg]

do you know of an operating system like MSDOS that will handle all RAM as straight line memory

It can be done in MSDOS itself, if you are careful. Do a Google search for "real mode flat memory". However, I never tried it, because I am not a very careful programmer... And it would not serve your purpose of saving an array of more than 64 KB, because for that you would still have to use the DOS services which are programmed for writing a maximum of 64KB(-1) at once.

[Donald Darden]

Intel imposed the 64k addressing boundary on their early CPU architectures, because somebody decided that nobody would ever need more than 1 MB or RAM, ever. So instead of having the segment addresses do a full offset from the addressing range allowed with 16 bits, they caused an overlap where only four addressing bits were actually appended. Some old code used some carefully crafted segment+address math to obscure jump locations or table lookup values, trying to protect their code from prying eyes.

MSDOS merely followed the hardware design. Later, when they wanted to address more memory, they used tricks with address line 20 to map in a different memory chunk at the very top end of the memory range, and we saw new drivers under DOS to allow us to address expanded and extended memory. People never really got a handle on which was which.

With Windows, Microsoft finally decided to take over managing memory and try to make it transparent to the user. The expansion of the original architecture design added instructions that could actually address a full 32-bits of memory range, and Windows was able to take advantage of this. DOS, however, was designed for the older architecture, and though it could run on the newer platforms, it still had the same limitations, because it still relied on the old 16-bit design and legacy addressing modes.

So the original shortsightedness was probably due to some director over an engineering department with Intel who decided that 1 MB of RAM was all that was ever going to be needed, and saw this as a way to keep the chip die small and not require ever more complex pin arrangements to support all the addressing lines that would be needed otherwise. The advances in chip design and multilayer fabrication processes would eventually prove him wrong, but that does happen. But Bill Gates once thought
that 64k was all the RAM that would ever be needed, so at least he was in good company.

Every time an interrupt is generated, the first thing that usually happens is that further interrupts are inhibited immediately. Some very short processes or time critical processes may complete their tasks before they permit further interrupts. Others, willing to give way to other possible interrupts, and may enable interrupts almost immediately, thus permitting some other process to preceed them if need be. You cannot tell which method is used by any one process without examining the code involved.

[Joe Byrne]

Not sure how you're using the BSAVE command, nor how fast the following works with that large of an array, but I used this technique a lot back in my DOS days (even a time or two in Windows) and found it to be darn good for my needs. Sure glad I never delete any of these "old" tools

'                           ARRAYSAV.BAS
'        ---------------------------------------           Eric J. Pearson
'        ARRAY SAVE/LOAD ROUTINES FOR POWERBASIC 
'        ---------------------------------------             August 1991
'
' These sample programs demonstrate a method for BSAVEing and BLOADing
' numeric and flex string arrays (i.e. swapping data directly between memory
' and disk) with PowerBASIC.  This technique is much faster than saving and
' loading arrays as text files (i.e. creating data files with PRINT#), and
' takes less program overhead.  And because the files are undelimited (i.e.
' no CR/LF, commas, quotes, etc.) and because numeric values are stored in
' "compressed" form (compared to storing numbers as text) the data files
' take up much less disk space.  Finally, the technique allows numeric values
' to be saved/loaded directly, without the relatively slow task of translation
' to/from text.
'
' Numeric variables of any type, and flex strings, in arrays with any number
' of dimensions, can be saved and loaded with routines based on these samples.
' (Sorry, but "normal" string arrays can't be BSAVEd or BLOADed.)
'
' Generally speaking, a small "custom" sub is built for each array that is
' to be saved/loaded, and a "standard" sub handles the actual bload/bsave...
'
' INITIALIZATION:

	'define variable types (*REQUIRED* values)...
	%IntegerVar =  2   'INTEGER (%)
	%LongIntVar =  4   'LONG INTEGER (&)
	%QuadIntVar =  8   'QUAD INTEGER (&&)
	%SinglePrec =  4   'SINGLE PRECISION (!)
	%DoublePrec =  8   'DOUBLE PRECISION (#)
	%ExtPrec    = 10   'EXTENDED PRECISION (##)
	%BCDFixed   =  8   'BCD FIXED POINT (@)
	%BCDFloat   = 10   'BCD FLOATING POINT (@@)
	%FlexString =  0   'FLEX STRING ($$)

	'for convenience (arbitrary values)...
	%Load       =  1
	%Save       = -1

' DEMONSTRATION #1 (1-dimensional integer array):

	'dimension the array
	DIM STATIC Array1%(256)

	'load demonstration data into the array
	Array1%(1) = 10001

	'save the array on disk
	CALL ArrayOne(%Save)

	'clear the demo value from the array
	ERASE Array1%

	'load the array from disk
	CALL ArrayOne(%Load)

	'display the demonstration data
	PRINT Array1%(1)    'prints 10001 if everything worked right.

' DEMONSTRATION #2 (3-dimensional, double precision array):

	DIM STATIC Array2#(1:64,0:10,0:2)
	Array2#(2,2,2) = 20000000.2
	CALL ArrayTwo(%Save)
	ERASE Array2#
	CALL ArrayTwo(%Load)
	PRINT Array2#(2,2,2) 'prints 20000000.2 if everything worked right.

' DEMONSTRATION #3 (one-dimensional, flex string array):

	DIM STATIC Array3$$(0:1024)  '(each element to be 8 characters)
	'8 chars per element * 1025 elements = 8200 bytes, so...
	MAP Big3$$ * 8200

	'map the array into the big string
	FOR Element% = 0 TO 1024
		ElBase% = (Element%*8)+1
		MAP Big3$$, FROM ElBase% TO ElBase%+7 AS Array3$$(Element%)
	NEXT

	'load demonstration data into the array
	LSET Array3$$(16) = "TESTING!"

	'save the array
	CALL ArrayThree(%Save)

	'clear the demonstration data
	'(don't ERASE the array or the MAP will be lost)
	LSET Array3$$(16) = "        "

	'load the array
	CALL ArrayThree(%Load)

	PRINT Array3$$(16)  'prints "TESTING!" if everything worked right.

	END

SUB ArrayOne(Direction%)

	'THIS IS A "CUSTOM" SUB, SET UP FOR A PARTICULAR ARRAY

	SHARED Array1%()
	LOCAL  Segment&,Start&,Finish&,VarType%

	Segment& = VARSEG(Array1%(0))   'first element
	Start&   = VARPTR(Array1%(0))   'first element
	Finish&  = VARPTR(Array1%(256)) 'last element
	VarType% = %IntegerVar
	CALL LoadOrSave("Array1",Direction%,Segment&,Start&,Finish&,VarType%)

END SUB


SUB ArrayTwo(Direction%)

	'THIS IS A "CUSTOM" SUB, SET UP FOR A PARTICULAR ARRAY

	SHARED Array2#()
	LOCAL  Segment&,Start&,Finish&,VarType%

	Segment& = VARSEG(Array2#(1,0,0))     'first element
	Start&   = VARPTR(Array2#(1,0,0))     'first element
	Finish&  = VARPTR(Array2#(64,10,2))   'last element
	VarType% = %DoublePrec
	CALL LoadOrSave("Array2",Direction%,Segment&,Start&,Finish&,VarType%)

END SUB


SUB ArrayThree(Direction%)

	'THIS IS A "CUSTOM" SUB, SET UP FOR A PARTICULAR ARRAY

	SHARED Big3$$  'note that Array3$$() itself is not shared
	LOCAL  Segment&,Start&,Finish&,VarType%

	'(note use of STRSEG and STRPTR, not VARSEG and VARPTR as above)
	Segment& = STRSEG(Big3$$)          'first byte of big string
	Start&   = STRPTR(Big3$$)          'first byte of big string
	Finish&  = Start& + LEN(Big3$$)    'last byte of big string
	VarType% = %FlexString             '

	CALL LoadOrSave("Array3",Direction%,Segment&,Start&,Finish&,VarType%)

END SUB


SUB LoadOrSave(FileName$,WhichWay%,SegmentAdr&,StartAdr&,FinishAdr&,WordLen%)

	'THIS "STANDARD" ROUTINE IS USED BY ALL "CUSTOM" SUBS ABOVE

	LOCAL Length&

	FileName$ = FileName$ + ".ARR"  'MY DEFAULT EXTENSION FOR ARRAY FILES

	Length&  = FinishAdr& - StartAdr& + WordLen%  'ARRAY LENGTH IN BYTES

	DEF SEG = SegmentAdr&

	IF WhichWay% = %Save THEN
		BSAVE FileName$,StartAdr&,Length&
	ELSE
		BLOAD FileName$,StartAdr&
	END IF

	DEF SEG

END SUB

'------------------------------------------------------------- end of ARRAY.BAS

[Michael Mattias]

Wow, what memories!

VARSEG, STRSEG, DEF SEG, BLOAD, BSAVE,'$$' data types....

Ah, they just don't make keywords like they used to......

[Fred Buffington]

Coming From THEOS multi-user basic in the 80's, it was nice to see the addition of mat in the newer versions of pbwin and pbcc.

THEOS not only allows mat for numeric but also string arrays and
a MAT WRITE #file%[,RECORD KEY]:string_arrayname$ or numeric_arrayname

This would write the array to a file starting with item 1.

and then on the other side,
MAT READ #file%[,RECORD KEY]:arrayname

[Michael Mattias]

Talk about memories....and relevant, too!

[72030,3563] Michael Mattias
RPD.ZIP
  Bin,  Bytes:    21248, Count:   141, 22-Aug-95

  Title   : REDIM PRESERVE plus Virtual Array Memory for PB
  Keywords: ARRAY REDIM PRESERVE SAVE RESTORE SOURCE FREE 3.1

  Routines to allow PowerBASIC programmers to REDIM regular and
  HUGE fixed-element-size arrays (including TYPE arrays) with PRESERVE and
  use disk files as "virtual array memory." Includes Unit source, demo source
  and documentation. Requires PB 3.1+. Copyright but free. From Michael
  Mattias.

There are some 'save and restore' array routine in here which ain't all bad...