Noob: Supercharging UDTs with Classes

[Craig Scarborough]

In the "Powerbasic Listens" thread I mentioned that I'd love to see Powerbasic support dynamic arrays within UDTs. There are two schools of thought here, and it seems that that overhead and therefore loss of speed that would occur were this to be implemented may prevent this from being added as a feature.

With that, Paul Squires was kind enough to say a) he would like to see this, too and b) the use of classes gives him much of the feature set he was looking for anyway, so it's a very plausible workaround.

I'm a complete noob at objects - never programmed with them, never had to. However, I'm currently working on my own project where I think either they (or dynamic arrays within UDTs) would be very handy. Basically, I'm creating an array of UDTs that store information on a per person basis. Within each person's UDT I need to store a lot of historical information (in yet another UDT). For the new guy, there's obviously not going to be much info. Frankly, they may never attain much historical info EVER - but I need to be able to track them to eternity. There will be select people, however, that will record a lot of history and therefore the array of their statistics UDT would be fairly large. If I were to just dimension the UDT for a worst case scenario, I'd be looking at an array like this: stats(20, 3, 5, 10) as tStat.

For our purposes I'm not going to show the tStat type declaration, but it's about 1K in data. I'm looking to keep track of 30000 people to start, adding about 1000 per year (not real-time, program-time, so this could happen fairly quickly). Unless my calculations are off, I'm quickly bogging down huge amounts of memory if each person has an array of 20 * 3 * 5 * 10 = 3000 * 1k = 3 MB (?!).

So - the hope was to implement a dynamic array that expands as the history of the person requires it, rather than creating the giant array referenced above.

I'm wondering - are CLASSES really going to be the answer for my problem here? I have a number of other requirements here as well, some that naturally lend itself to an array - I need the ability to sort the people based on data stored within that statistics UDT. I also need to change the data of the most current historical record - a LOT. Wouldn't the use of an array of obects like this be cumbersome? I guess what I'm asking is: are CLASSES the be-all end-all here, and should I immediately start re-writing my program to support them for this unique case?

Thanks in advance for any advice!

-Craig

[Michael Mattias]

>are CLASSES the be-all end-all here

Neither classes, nor objects, nor "variable-size UDTs", nor indexed disk files, nor PB arrays, nor linked lists, nor anything else is ever a 'be-all end-all' for anything.

Each application is different. You have many tools from which to choose; one or more may suit any particular application. No one choice will be wrong or right; some will be inferior and some will be superior; some will be simple to program, some will not.

Just don't lock yourself in to a particular mindset, eg..

>I need the ability to sort the people based on data stored within that statistics UDT.

I don't think you do need to do that. I will bet what you really need is the ability to report retrieved statistics and sort that info by people. How you store that info, and whether or not you maintain that information sorted or only sort it 'as required' is one of those 'design decisions' you need to make before writing that first line of code

However, you start with plus six (+6) style points because...

> Unless my calculations are off, I'm quickly bogging down huge amounts of memory....

.. you are actually thinking about resource usage. I don't see that here all that often. Good for you.


MCM

[Jeff Blakeney]

Well, using a class can give you some interesting abilities. You can have all the properties you need made available to your program. One benefit is being able to have read only or write only properties. You can also encapsulate other methods in the object for dealing with your data. You can write whatever sort routine or routines that you want and include them in the object. You can even include file operation methods to make it easier to load and save records to a file and this could include keeping an index file of where each record starts in that data file to improve performance. You can even have private methods that help out your public methods and because they are encapsulated in the object, you don't need to worry about name conflicts if you include the object in another project. Don't forget you can also write smaller objects and then create new objects that inherit those objects. You can sort of do this with UDTs but then your syntax for dealing with them gets messy (ie. tCustomer.Address.Line1 as opposed to using an object an using oCustomer.Address1).

I was never a fan of object oriented programming but this use of them is quite attractive to me.

[Michael Mattias]

The use of CLASSES is quite attractive for the purpose of creating REUSEABLE code.

Or, I suppose, for those who may have experience using OO-syntax.

Or, I suppose, for those who want to create a programming library for fun or profit.

Or, I suppose, for those who are seeking a better way to 'protect' data-variable values across multiple occurences of things.

Lots of plusses. But a lot of plusses does not a silver bullet make.

MCM

[Scott Slater]

For our purposes I'm not going to show the tStat type declaration, but it's about 1K in data. I'm looking to keep track of 30000 people to start, adding about 1000 per year (not real-time, program-time, so this could happen fairly quickly). Unless my calculations are off, I'm quickly bogging down huge amounts of memory if each person has an array of 20 * 3 * 5 * 10 = 3000 * 1k = 3 MB (?!).

Have you considered storing some of this data in a temporary file, or SQL table? It would take some of the load off of the memory requirement.

You can have an array of objects, and/or arrays inside of an object to handle different user records.

[Scott Slater]

I'm wondering - are CLASSES really going to be the answer for my problem here? I have a number of other requirements here as well, some that naturally lend itself to an array - I need the ability to sort the people based on data stored within that statistics UDT. I also need to change the data of the most current historical record - a LOT. Wouldn't the use of an array of obects like this be cumbersome? I guess what I'm asking is: are CLASSES the be-all end-all here, and should I immediately start re-writing my program to support them for this unique case?

Again, it is probably more if an implementation issue. Sounds like a job for SQL from what you're describing. You can mix that with Objects or UDTs (or both).

Have a look at SQL Tools or SQLitening. You can then encapsulate the database read/writes into an object which is a pretty neat way of handling data. The code would be something like:

User.Name = "John Smith"
User.City = "Phoenix"
if User.Save then
  User.Sort(%ByLastHistory)
  .....

[Michael Mattias]

>Have a look at SQL Tools or SQLitening.

Um, how about OLE via ADO? That even uses OO-syntax, which apparently is of interest to Mr. Scarborough.

DEMO TO GET YOU STARTED:
Generic 'ADO' Connection and Query Tester (CC 5+/Win 9+) 11-02-08

Assuming, of course, the application calls for permanent storage in some kind of RDBMS.



MCM

[Craig Scarborough]

Still working through this, but I'm trying to minimize the number of disk read/writes and maximize the usage of memory to keep that application quick. I'm thinking of storing the recent years statistics in memory and past years on disk (since they will be utilized much less often). By only storing this year's in memory, I can cut down the size of the array immensely. If I can do this, I maybe able to bypass switching to objects entirely.

Thanks all for your input!

[Michael Mattias]

Well, if your statistics info can be stored on disk in fixed-size records, you could use something like...

Memory Mapped Files instead of RANDOM disk file access 5-8-04

If you want to map ALL the data, it will be limited to 600-800 Mb depending on your system.

Advantage is, it does not use "user" memory as would loading all the data to a PB array.

Actually, if you can describe your statistics record, and give us some idea of volume, and how often you have to read or write one or more records, some folks will give you some ideas. But right now we are just sucking our thumbs because "any one of the above" could be a good choice for all we know.

[Christopher Carroll]

Still working through this, but I'm trying to minimize the number of disk read/writes and maximize the usage of memory to keep that application quick. I'm thinking of storing the recent years statistics in memory ...

From previous hint/comment:

...Sounds like a job for SQL from what you're describing. You can mix that with Objects or UDTs (or both)....Have a look at SQL Tools or SQLitening....

A SQLite database can be stored in memory by naming the database ":memory:" when opening it. See www.sqlite.org/inmemorydb.html. This gives one method to retrieve data from this year, or from archive - just different locations.

[Michael Mattias]

[EDITORIAL]
Storing "everything" in RAM is overrated. Good application design and programming technique produces excellent performance.
[/EDITORIAL]

[John Strasser]

Based upon what I can read here, the question of having all that info "live" may turn out to be a better question to ask.

Back when I designed software for a living (as compared to my current hobby status) the most applicable question I ever asked was:

How live do you need your live data?

Carrying around that much info seems like a lot of overhead to me...

As for oop vs udt, the needs of the program (data encapsulation, organization and maintenance, learning curve, etc) should determine what you choose. Not data storage. You could even have your class (or udt) have a pointer to another class which reads in the volume of data so that when you call the method or property of the "data" class it automatically loads the data from disk, traps errors, and whathaveyou.