DIALOG/Listview question

[BOB MECHLER Member Find all posts View Profile Close BOB MECHLER]

I have a Dialog that needs to fill a ListView control with a subset of records from a 1 million record file where I have up to 10 different criteria items each record has to pass before it is placed in the listview.

My first method filled a table with the results while displaying a smallish dialog with a progress bar. Then after the scan and I have a table, I started another progress bar indicating that it was now loading the listview. Then when it finished loading the listview I show the results screen and allow them to sort by columns, save the result to a another file (dbf) (for loading into Excel or Access) or a text file or discard it.

That method has worked fine until we started getting those million record files to scan where there was no way to really optimize the scan. (Not using SQL)

Someone at work here asked, "why spend the time creating the initial table, just load the listview directly as a record qualifies". I had to agree that was a great idea and would eliminate 40% to 80% of the processing time.

Anyway, I've come up with the following method that seems to work but want to check to see if I'm just lucky.

Steps.
1. Do a Dialog New and add one control to it, the Listview control.

2. Start looping through the records adding to the listview (not yet shown with Dialog Show Modal) and update a progress bar in another window that is visible. Allow the run to be terminated during the Scan phase.

While the scan progress bar shows the progress it can be minimized and cause the results screen to also be minimized by passing a flag.

The scans can take anywhere from 2 seconds to 10 minutes depending on the size of the file being scanned and whether we can optimize the scan using a sorted field in the query.

3. When the scan is complete then add all the other controls which include the number of hits and total number of records etc

4. Then do a Dialog Show Modal and start processing the dialog callback.

5. If they chose Excel or a Text file, then I build the table by reading the listview row by row. (Since the results listview can be sorted, I had to do this anyway)

The question is: Is there any problem creating a new dialog and then not showing it for as long as it takes for the listview to fill? (So far in testing it holds at least 750,000 records in the listview)

The time savings by not doing the initial results table is 40% to 80% depending on the hardware being used.

Bob Mechler

[Michael Mattias Member Find all posts View Profile Close Michael Mattias]

Show/no show depends on the user and the application.

When I had similar problem with loading up a control with rows from "really large" files, I solved that by using a virtual listview. That is, you never have to add anything to the listview control, because it's all virtual.

That reduces your loading operations to...

- Scan table or file or whatever, loading info into something handy. (A PB array sounds good) (This will take as long as it does now)
- Make one call to Listview_SetItemCountEx to tell control how many rows there are. This will take maybe a couple of millseconds on a bad day.

Then you just have to respond to WM_NOTIFY/LVN_GETDISPINFO by looking up in your array the text which goes in the target row and column.

Basically you end up doing what your colleauge suggested: you only handle display info when that info actually has to be displayed. Very, very efficient and user-friendly.

Can't help with the scan, but 10 minutes sounds a little spooky. No 'online' program should take 10 minutes to do anything.

What you could consider in this case is doing the load in a separate thread of exectuion, and every 'X' records tell the virtual listview there are now <X> records in the control. That would make what is available actually available to the user at that time.

Another technique you can use is (assuming database).. first do a query to get a COUNT() of the number of rows actual detail query will return.... then depending on that number change your load/display strategy to fit the number of rows with which you will hav to deal.

Of course, you should also look at creating indexes to match your query requirements. A ten-minute query simply must be doing some kind of full table scan, so some kind of indexing should help that a lot.

MCM

[BOB MECHLER Member Find all posts View Profile Close BOB MECHLER]

Good information and you gave me some key words to look up.

The user actually expects to send the result grid directly to Excel or a text file almost immediately after they make sure they are getting the information they want.

Sometimes they just get a few records with very specific criteria and other times they want everything for a specific customer for the last year.

Since I asked the question I've been testing more and the increase in speed is about 80%. So even the longest query should take between 2 to 5 minutes. I forgot to mention they can select up to 6 related tables and cause the primary table to lookup additional information in other tables to satisfy field information in the listview.

I've got it set so they can minimize the progress dialog and forget about it while they do other work.

One customer I have has about 2000 predefined queries already. It's about a 200 user setup.

I do however have other parts of the system where your suggestions would very useful.

It's just that I was unsure how much time you could have between issuing a Dialog New and the Dialog Show Modal. Apparently you can manipulate and fill any control added after the Dialog New and do as much interim processing as you want before you issue the Dialog Show Modal to start callback processing.

Bob Mechler

[Cliff Nichols Member Find all posts View Profile Close Cliff Nichols]

unless I mis-read

UMMMM "Threads" may be the keyword

(anything beyond 3 seconds is annoying)