funny!! MCM------------------
Wash DC Area
Borje's "Poff's" is likely the BEST tool for learning PB.. http://www.tolkenxp.com/pb
and a few PBTool's:http://sweetheartgames.com/PBTools/PBTools.html
-
hehe funny!! MCM
------------------
Wash DC Area
Borje's "Poff's" is likely the BEST tool for learning PB.. http://www.tolkenxp.com/pb
and a few PBTool's:http://sweetheartgames.com/PBTools/PBTools.html -
..and just to demo the best source for performance improvement is design...
- Yes, I got the 95% improvement by going to LPSTR_TEXTCALLBACK and have stuck with that.
But - I got another 50% from what was left by..
- Changing the algorithm I used to expand the tree when initially presented. I was using a 'generic' treeview enumerator I had developed, but given the specific data possible, I was able to rewrite that to "never mind checking this, because that can never happen"
- I found a place where I had forgotten to disable the redraw when the display was being changed; now I <U>always</U> disable redraw, set up <U>everything</U> for user presentation, <U>then</U> enable redraw and force a repaint.
- When applicable, I moved a couple of operations which include a progress bar to a separate thread, and simply posted bar-update messages. So what if the bar is not quite in synch? Show me a user who can tell the difference between 50% complete and 55% percent complete when there is nothing but a bar (no numbers)? Better still, show me one who cares.
- I did larger memory allocations earlier, meaning fewer <U>re-</U>allocations (REDIM or HeapRealloc)).
No, I didn't even bother looking at how I did loop counters. I went where they keep the money.
[This message has been edited by Michael Mattias (edited March 28, 2004).]Leave a comment:
-
Prelim results:
Changing to external storage of text and specifying LPSTR_TEXTCALLBACK in psxText member of TV_ITEM : Load time reduced to less than five (5) percent of time required when control must store the text.
Ninety-Five (95) percent improvement is a good start.
Leave a comment:
-
Who reads data from a file this way? I memory-map the file and parse it character by character. (You have to parse ANSI EDI data character-by-character anyway, so why not let Windows carry the load?)Now read the data in quickly using high speed buffers (large blocks of data)
of at least 16 to 32 KB in size (after about 8 KB the buffer
size only increases speed of reading slightly).
Besides, my "worst case scenario" is a 30 MB file.. which results in one (1) interchange containing one (1) functional group containing one transaction set (1).
Unfortunately, the "segments" is where I have a little problem...Here is my scan report:
MCMCode:EDI File Summary Analysis of File S:\pbwin70\work\testdata\2551121.clp Created : Tuesday, January 20, 2004 4:11 PM Accessed: Thursday, February 26, 2004 6:00 AM Written : Tuesday, January 20, 2004 4:11 PM Interchange 000000069 Group Count 1 Group ID HC # 1069 Transaction Count 1 Transaction Set 837 # 1 Segment Count 1156344 *** END OF REPORT ***
Leave a comment:
-
Michael,
Some suggestions:
The data file looks like it would be quite large. The best way
to load it is to avoid reading it line by line, but to read it
in using buffers. Assuming you have to load say 500,000 items
which are 50 characters each you would be loading about 23 meg
of data. By using 32 KB buffers to read the data file in blocks
it would take about 5 to 20 seconds to load (depending upon CPU
and speed of harddrive).
Now to fill the treeview with this much data would definitely
be problematic. Using what you posted, the "Functional group" level
would be 2 levels in the tree. The next levels beyond this
should not be filled in the tree.
One way to tackle this so the end user doesn't have to wait too
long is to use a tab control with say 10 tabs on it (1 to 10).
Put a Treeview control on each tab. Disable all the tabs on the
tab control and disable the treeview on the first tab. Now read
the data in quickly using high speed buffers (large blocks of data)
of at least 16 to 32 KB in size (after about 8 KB the buffer
size only increases speed of reading slightly).
Fill the first treeview until you reach some predefined limit.
Only fill the tree to the first two levels. The higher levels
should be in memory by this time and arranged in some kind of
data array (just not loaded into the treeview). Once done, enable
the first tab on the tab control plus the treeview associated
with that tab. The user can now access this data. When the user
selects a level not yet loaded (3rd level and above) load the
data into the treeview dynamically. By processing the TVN_ITEMEXPANDING
notification message, you can fill the next levels in the treeview
when needed. If the level is expanding, fill in the extra data.
If the level is collapsing then delete the items in that level.
By dynamically loading and unloading data as needed for levels
3 and above, the treeview should never become overloaded with too
much data.
Now continue reading in the rest of the data in the file and fill
the second treeview on tab 2 of the tab control. When it is filled
(up to level 2 and the rest in memory), then enable the second
tab on the tab control and its treeview.
Continue to do this until as many tabs/treeviews are filled to
handle all the data.
You could use a thread to fill each treeview , one at a time.
Since a treeview can't be accessed until it is filled and enabled
there shouldn't be a problem with the thread getting in the way
of user interaction. I would use a Global array for storing the
data. Using critical sections, a worker thread could be filling
the global array with data. When the array is filled, simply send
a message to the main thread (use sendmessage to force a contect
switch) and fill the treeview from the main GUI thread. Once done
sendmessage would return and the worker thread would continue on.
You can get fancy with the tab control, by using imagelists and
displaying one image for a loaded treeview/tab and another for
a treeview/tab in the process of being loaded and another for an
unused treeview/tab.
Assume the character R (ready) stands for the loaded image, L (loading) for the
process of being loaded image and U for unused, the tabs on
the tab control would lool like this to the end user:
The end user is fully aware of what data is loaded, whatCode:_________________________________________________ | R | L | U | U | U | U | U | U | | |_____|_____|_____|_____|_____|_____|_____|_____ | | | | | | | | | | | | | | | | | | | | | | | | | | |____________________________________________________|
data is in the process of being loaded and what data does
not exist yet. As long as one of the tabs displays the
in processing of loading (L) image, the users knows the
data is still being read in.
------------------
Chris Boss
Computer Workshop
Developer of "EZGUI"
http://ezgui.com
[This message has been edited by Chris Boss (edited March 17, 2004).]Leave a comment:
-
OK, the data does seem to fit the "tree" model.
Have you looked into the threaded background fill?
------------------
Every day I try to learn one thing new,
but new things to learn are increasing exponentially.
At this rate I’m becoming an idiot faster and faster !!!
------------------
George W. Bleck
Lead Computer Systems Engineer
KeySpan Corporation
My EmailLeave a comment:
-
Basic ANSI ASC X12 EDI:Since we don't know the data structure/relationship it's hard to figure out a more appropriate GUI (if one would apply
"Tree" is perfect for this.Code:FILE - Interchange - Functional group - Transaction set - Segment - Segment - Segment - Transaction set - Segment - Segment - Segment - Transaction set - Segment - Segment - Segment - Functional group - Transaction set - Segment - Segment - Segment - Transaction set - Segment - Segment - Segment - Transaction set - Segment - Segment - Segment - Interchange - Functional group - Transaction set - Segment - Segment - Segment - Transaction set - Segment - Segment - Segment - Transaction set - Segment - Segment - Segment - Functional group - Transaction set - Segment - Segment - Segment - Transaction set - Segment - Segment - Segment - Transaction set - Segment - Segment - Segment
I started on this project yesterday. OK, so I worked on other parts first whilst digesting all these ideas.
First thing I'm going to try is storing the text externally to see what that does.
MCM
[This message has been edited by Michael Mattias (edited March 17, 2004).]Leave a comment:
-
mike,
sounds like re-thinking the gui may be in order. that many items in
a tree view sounds like a nightmare to search for a piece of data.
since we don't know the data structure/relationship it's hard to figure out
a more appropriate gui (if one would apply).
alternative using treeview...
you could do an on-demand fill using threads so when the client clicks on a node
it starts filling the items in the background, my bff example in the source code
forum does this. http://www.powerbasic.com/support/pb...ad.php?t=23778
------------------
every day i try to learn one thing new,
but new things to learn are increasing exponentially.
at this rate i’m becoming an idiot faster and faster !!!
------------------
george w. bleck
lead computer systems engineer
keyspan corporation
my email
[this message has been edited by george bleck (edited march 16, 2004).]Leave a comment:
-
Could you impose an arbitrary structure above level 3 to subdivide the data?Originally posted by Michael Mattias:
... The technique involves only loading those nodes at a given level, when required
That's a good idea, and I have thought about it..
My problem is the one you suggest: in most cases, the vast majority of the the nodes will be at a single level (level three).
So I could load up levels one and two, but as soon as the user clicks on the little box to expand level two, WHAMMO, I'm loading a boatload of stuff.. and in this case, the user gets to wait at exactly the wrong time. Users can be reasonably patient on "file/load" but they just won't go for "please wait" when they click to expand a tree node.
Maybe I will just do some kind of "load only if visible" thing and trap scrollbar, arrow and "page key" messages or something like that. That way I'd only have to load 20-30 nodes at a crack. Figgering out what would be visible might not be a whole lotta fun, though... (and then there's the 'search' feature, where I (currently) enumerate the tree... can't enumerate nodes which haven't been added yet, so I'd have to figure out another way to handle the search).
MCM
e.g. Put in Alpha indexes A-Z, thereby breaking the data into another 26 pseudo branches. You get the idea. Who wants to scroll a bazillion items anyway?
Cheers,
Michael.
[This message has been edited by Michael Puckett (edited March 16, 2004).]Leave a comment:
-
> ... The technique involves only loading those nodes at a given level, when required
That's a good idea, and I have thought about it..
My problem is the one you suggest: in most cases, the vast majority of the the nodes will be at a single level (level three).
So I could load up levels one and two, but as soon as the user clicks on the little box to expand level two, WHAMMO, I'm loading a boatload of stuff.. and in this case, the user gets to wait at exactly the wrong time. Users can be reasonably patient on "file/load" but they just won't go for "please wait" when they click to expand a tree node.
Maybe I will just do some kind of "load only if visible" thing and trap scrollbar, arrow and "page key" messages or something like that. That way I'd only have to load 20-30 nodes at a crack. Figgering out what would be visible might not be a whole lotta fun, though... (and then there's the 'search' feature, where I (currently) enumerate the tree... can't enumerate nodes which haven't been added yet, so I'd have to figure out another way to handle the search).
MCM
Leave a comment:
-
Hi Michael,
If you don't mind spending $500 buck
, try the SFT/Tree DLL
http://www.softelvdm.com/treedll.html
You probably could recupe the cost by the price of the software
yuo're selling. I don't have this control but I am tempted though,
although for me $500 is a bit much for a control.
Regards
Steven
------------------
Leave a comment:
-
Forgive me if this has already been suggested and I missed
it ...
I remember reading (can't remember if it was Petzold or Balena)
about a "demand loaded" tree technique that *may* help out your
performance woes if the data is indeed a tree. If the data is
essentially flat (all nodes at the primary level) there won't be
any improvement.
Anyway, sorry to ramble ... The technique involves only loading
those nodes at a given level, when required.
For example,
Hope that makes sense/is not a repeat of previous suggestion(s) orCode:Initial load +-- Root | +-- Node 1 | +-- Node 2 | +-- Node 3 | +-- Node 4 | +-- Node 5 Initally 5 nodes loaded, now click Node 1: +-- Root | +-- Node 1 | | +-- Node 1.1 | | +-- Node 1.2 | | +-- Node 1.3 | | +-- Node 1.4 | | +-- Node 1.5 | +-- Node 2 | +-- Node 3 | +-- Node 4 | +-- Node 5 Nodes 1.1 thru 1.5 loaded, now click Node 1.1 +-- Root | +-- Node 1 | | +-- Node 1.1 | | | +-- Node 1.1.1 | | | +-- Node 1.1.2 | | | +-- Node 1.1.3 | | | +-- Node 1.1.4 | | | +-- Node 1.1.5 | | +-- Node 1.2 | | +-- Node 1.3 | | +-- Node 1.4 | | +-- Node 1.5 | +-- Node 2 | +-- Node 3 | +-- Node 4 | +-- Node 5 | Nodes 1.1.1 thru 1.1.5 loaded.
your initial post; like idea (1).
Found the article:
Page 514. "Loading nodes on demand.", from the book "Programming
Visual Basic 6.0", Francesco Balena.
As well as a coding example by Brad Martinex: "TVFastLoad: How to
load the TreeView quickly".
http://www.mvps.org/btmtz/treeview (about 2/3 down page)
http://www.mvps.org/btmtz/treeview/tvfastload.zip
Cheers,
Michael
[This message has been edited by Michael Puckett (edited March 16, 2004).]Leave a comment:
-
Michael,
If you end up using the listview, be sure to use it as a virtual listview. It's basically an ownerdraw listview, the os sends you a message and tells you what cells to fill and you draw them with data stored in an array. If you don't do that you'll end up with the same speed penalty.
Russ Srole
------------------
Leave a comment:
-
if you're thinking of a listview ... you might want to checkout siGrid from www.softwareinnovators.com - their pb grid is very, very fast.
same with elias's grid - i don't recall the url, but it looks pretty good too.
--don
------------------
Don Dickinson
Author of ddoc Print and Preview A dll-based print-preview engine for windows
www.greatwebdivide.com
Author of Tsunami Tcp Data Server
http://ttds.greatwebdivide.comLeave a comment:
-
>but i seem to recall that you can specify how many nodes windows should allocate at a time.
I found that for the LISTview control (LVM_SETITEMCOUNT), but I can't find one for the TREEview control...
.. but that gives me another idea...
Maybe I can use a listview control with the "indent" feature instead of a treeview...should look close enough for government work..
I already disable the drawing (WM_SETREDRAW) whilst the tree is loading, so I don't know what benefits LockWindowUpdate might bring.
MC
Leave a comment:
-
i haven't worked with a treeview via the api lately, but i seem to recall that you can specify how many nodes windows should allocate at a time. by allocating a bunch of nodes at once (say 1000) and using LockWindowUpdate while loading, you can speed it up significantly.
i'd beware of the multi-threaded interface ... that can be tricky to deal with (see the Add Software/Windows components applet in the control panel of win2k for an example how not to do this).
--don
------------------
Don Dickinson
Author of ddoc Print and Preview A dll-based print-preview engine for windows
www.greatwebdivide.com
Author of Tsunami Tcp Data Server
http://ttds.greatwebdivide.comLeave a comment:
-
Treeview Control Replacement?
I have been using a treeview control to display ANSI EDI data , but of late I have run into a great number of "really large" files... that is, I am trying to put several hundred thousand items into the tree control.
Needless to say, this can take approximately "forever" to add all those nodes to the tree.
I have been thinking about how to make this faster and had a couple of ideas...
1. Instead of asking the tree control to hold the text of each node, hold the text separately and use style LPSTR_TEXTCALLBACK and just go get the text when Windows asks for it.
2. Load the tree from a separate thread.. allowing the user to work with "what's already been loaded" whilst the balance of the data continue to load.
3. Eschewing the treeview control entirely and painting the visible portion of the tree myself when Windows asks (e.g, on a static control or a proprietary window class window). (The "tree" is just so perfect a presentation).
I'll be playing around with all these methods over the next couple of weeks, but in the meantime if anyone has any suggestions or comments on the above thoughts I'd really appreciate those ideas.
Thanks,
------------------
Michael Mattias
Tal Systems Inc.
Racine WI USA
mailto:[email protected][email protected]</A>
www.talsystems.com
Leave a comment: