Announcement

Collapse
No announcement yet.

Real-time search match delay

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Real-time search match delay

    I am working on a program that provides the ability to search "real-time" against a fields in a database. To better clarify, as you type search text it auto-narrows to only matched text on-the-fly.

    Originally I was just reacting to EN_CHANGE, but the data set is just large enough to cause typing to be appear sluggish as search text is entered.

    My next thought was to work with a timer, such that, as you type it does a KILLTIMER/SETTIMER of .75 seconds. Once the timer expires, it then grabs the search text, performs the search, and then finally kills the timer for good. This allows the person to type normally until they pause about 3/4 of a second before initiating the search.

    It all works as expected, but are there any ramifications to constantly doing a KILLTIMER/SETTIMER over and over that quickly as someone types?

    Code examples to help explain...

    Timer Section
    Code:
            CASE %WM_TIMER
                SELECT CASE AS LONG CB.WPARAM
                    CASE %Timers.cRealTimeSearch
                        sub_FindMatch CB.HNDL
                        KILLTIMER CB.HNDL, %Timers.cRealTimeSearch
                    CASE %Timers.cStatusbarTimeout
                        CONTROL SET TEXT CB.HNDL, %dlgPRSG.sbarStatus, ""
                        KILLTIMER CB.HNDL, %Timers.cStatusbarTimeout
                END SELECT 
    EN_CHANGE Section
    Code:
                    CASE %dlgPRSG.edtTextTtoMatch
                        IF CB.CTLMSG = %EN_CHANGE THEN
                            KILLTIMER CB.HNDL, %Timers.cRealTimeSearch
                            SETTIMER CB.HNDL, %Timers.cRealTimeSearch, 750, BYVAL %NULL
                        END IF 
    Last edited by George Bleck; 17 Feb 2017, 11:22 AM.
    <b>George W. Bleck</b>
    <img src='http://www.blecktech.com/myemail.gif'>

  • #2
    Why use killtimer? I normally let my timers run during the life of the program, no need to kill them...
    I just use a static or global variable inside the timer that I load with the desired time (if you have a 10mS timer it would be 75)
    Then decrement it every loop, and reload it when activity is detected. If it reaches '0' , you can do what needs to be done...
    Regards,
    Peter

    Comment


    • #3
      I guess that could be done but from a resource perspective it sounds wasteful to have a timer running that is unused which includes extra messages that go into the queue and be processed (whether countdown is active or inactive). You also now need to process the timer countdown checks numerous times rather then a one time trigger when a specific time has elapsed.

      Be aware I am also running two different timers, one on a 4000ms count down, the other on a 750ms countdown. I would then need to find a common multiple between the two (e.x. 50ms) and then decrements two different counters every 50ms.
      <b>George W. Bleck</b>
      <img src='http://www.blecktech.com/myemail.gif'>

      Comment


      • #4
        Although thinking back to my original code, it might be better to do the KILLTIMER as soon as the timer message processes (i.e. before the sub_FindMatch) just in case the search takes longer than the timer period.

        In fact why not do it to both, can't hurt...

        Code:
                CASE %WM_TIMER
                    SELECT CASE AS LONG CB.WPARAM
                        CASE %Timers.cRealTimeSearch
                            KILLTIMER CB.HNDL, %Timers.cRealTimeSearch
                            sub_FindMatch CB.HNDL
                        CASE %Timers.cStatusbarTimeout
                            KILLTIMER CB.HNDL, %Timers.cStatusbarTimeout
                            CONTROL SET TEXT CB.HNDL, %dlgPRSG.sbarStatus, ""
                    END SELECT 
        <b>George W. Bleck</b>
        <img src='http://www.blecktech.com/myemail.gif'>

        Comment


        • #5
          I don't think you'll see much performance difference with a counter or timer more or less. Modern computers are fast and the OS already uses a huge number of resources.
          On a Windows7 computer with only a few programs running there are already hundreths of threads active (and 33000 open handles!), giving a lot more overhead as a simple timer.

          Even on old computers some 15 years ago I often used a dozen timers without noticable influance on performance.
          Regards,
          Peter

          Comment


          • #6
            Another way that I like and is very light on the system
            is a simple time compare using GetTickCount().

            Here is an example of what I mean, code is formatted to be easy to follow
            by using an edit control to hold the search string from a listbox.
            If nothing is type after a defined delay, a new search string will be build.

            Pierre

            Code:
            #COMPILE EXE '#Win 9.07#
            #DIM ALL
            #INCLUDE "Win32Api.inc"
            
            GLOBAL hDlg AS DWORD
            
            %Edit    = 101
            %Listbox = 201
            '______________________________________________________________________________
            
            CALLBACK FUNCTION DlgProc
             LOCAL  sEdit     AS STRING
             STATIC sEditPrev AS STRING
             STATIC TickCount AS DWORD
             LOCAL  Index     AS LONG
             STATIC Bypass    AS LONG
             LOCAL  ItemIndex AS LONG
            
             SELECT CASE CBMSG
               CASE %WM_COMMAND
                 SELECT CASE LOWRD(CBWPARAM)
            
                   CASE %Edit
                     SELECT CASE CBCTLMSG
                       CASE %EN_CHANGE 'Sent after the screen update
                         IF Bypass = %FALSE THEN
                           CONTROL GET TEXT CBHNDL, %Edit TO sEdit
                           IF TickCount = 0 THEN TickCount = GetTickCount() 'First time init
                           IF GetTickCount() > TickCount + 1000 THEN 'If miliseconds timeout is elapsed
                             FOR Index = 1 TO LEN(sEdit) 'Extract the last character typed
                               IF ASC(sEdit, Index) <> ASC(sEditPrev, Index) THEN
                                 sEdit = CHR$(ASC(sEdit, Index))
                                 Bypass = %TRUE 'Avoid working on self action
                                 CONTROL SET TEXT CBHNDL, %Edit, sEdit
                                 SendDlgItemMessage(hdlg, %Edit, %EM_SETSEL, -2, -2) 'Set the caret at the end
                                 EXIT FOR
                               END IF
                             NEXT
                           END IF
            
                           IF LEN(sEdit) THEN
                             ItemIndex = SendDlgItemMessage(hdlg, %Listbox, %LB_FINDSTRING, -1, BYVAL STRPTR(sEdit)) 'Search listbox
                             IF ItemIndex <> %LB_ERR THEN
                               SendDlgItemMessage(hdlg, %Listbox, %LB_SETCURSEL, ItemIndex, 0) 'Select what was found
                             END IF
                           END IF
                         ELSE
                           Bypass = %FALSE 'Reset bypass
                         END IF
                         sEditPrev = sEdit 'Be ready to detect next char
                         TickCount = GetTickCount() 'Get time for timeout
                      END SELECT
            
                 END SELECT
             END SELECT
            
            END FUNCTION
            '______________________________________________________________________________
            
            FUNCTION PBMAIN()
            
             DIALOG NEW %HWND_DESKTOP ,"Search", , , 125, 115, %WS_CAPTION OR %WS_MINIMIZEBOX OR %WS_SYSMENU, 0 TO hDlg
            
             DIM sListArray(0 TO 9) AS STRING
             sListArray(00) = "Zero"
             sListArray(01) = "One"
             sListArray(02) = "Two"
             sListArray(03) = "Three"
             sListArray(04) = "Four"
             sListArray(05) = "Five"
             sListArray(06) = "Six"
             sListArray(07) = "Seven"
             sListArray(08) = "Eight"
             sListArray(09) = "Nine"
             CONTROL ADD LISTBOX, hDlg, %Listbox, sListArray(), 5, 5, 115, 100, _
                         %WS_CHILD OR %WS_VISIBLE OR %WS_VSCROLL OR _
                         %WS_TABSTOP OR %LBS_NOTIFY, %WS_EX_CLIENTEDGE
            
             CONTROL ADD TEXTBOX, hDlg, %Edit, "", 5, 100, 115, 12
             CONTROL SET FOCUS hDlg, %Edit
            
             DIALOG SHOW MODAL hDlg CALL DlgProc
            
            END FUNCTION
            '______________________________________________________________________________
            '

            Comment


            • #7
              Pierre Bellisle You example does not quite fit what I am looking for. It needs to accept any number of keypresses until there is a pause by the client (i.e. a pause of "x" milliseconds assumes they are done typing), THEN it searches. Yours immediately searches as the person types. As there are 3500 items in the database, if I search immediately on each keypress it slows the person's typing. All I can see yours doing is clearing the search box after the delay and the person types something new.

              I have modified your code to perform the effect I was looking to validate. In my example, type an "e"... 3/4 of second after you type the "e" it finds all words with "e". Erase the "e" and let the listbox reset (3/4 of a second after you are done hitting backspace). Now type "ee", but pause 1 second between each "e" as you type it, it slowly narrows the field. Clear the search box again to let the listbox reset, now type "ee" fast (no delay between "e"s), see what I mean now?
              Code:
              #COMPILE EXE '#Win 9.07#
              #DIM ALL
              #INCLUDE "Win32Api.inc"
              
              GLOBAL hDlg AS DWORD
              GLOBAL sListArray() AS STRING
              
              
              %Edit    = 101
              %Listbox = 201
              '______________________________________________________________________________
              
              CALLBACK FUNCTION DlgProc
               LOCAL  sEdit     AS STRING
               STATIC sEditPrev AS STRING
               STATIC TickCount AS DWORD
               LOCAL  Index     AS LONG
               STATIC Bypass    AS LONG
               LOCAL  ItemIndex AS LONG
              
               SELECT CASE CBMSG
                 CASE %WM_TIMER
                   KILLTIMER CB.HNDL, 1
                   CONTROL GET TEXT CB.HNDL, %Edit TO sEdit
                   LISTBOX RESET CB.HNDL, %Listbox
                   FOR Index = LBOUND(sListArray) TO UBOUND(sListArray)
                     IF sEdit = "" THEN
                       LISTBOX ADD CB.HNDL, %Listbox, sListArray(Index)
                     ELSEIF INSTR(sListArray(Index),sEdit) THEN
                       LISTBOX ADD CB.HNDL, %Listbox, sListArray(Index)
                      END IF
                   NEXT Index
                 CASE %WM_COMMAND
                   SELECT CASE LOWRD(CBWPARAM)
              
                     CASE %Edit
                       SELECT CASE CBCTLMSG
                         CASE %EN_CHANGE 'Sent after the screen update
                           KILLTIMER CB.HNDL, 1
                           SETTIMER CB.HNDL, 1, 750, BYVAL %Null
                        END SELECT
              
                   END SELECT
               END SELECT
              
              END FUNCTION
              '______________________________________________________________________________
              
              FUNCTION PBMAIN()
              
               DIALOG NEW %HWND_DESKTOP ,"Search", , , 125, 115, %WS_CAPTION OR %WS_MINIMIZEBOX OR %WS_SYSMENU, 0 TO hDlg
              
               DIM sListArray(0 TO 9) AS STRING
               sListArray(00) = "zero"
               sListArray(01) = "one"
               sListArray(02) = "two"
               sListArray(03) = "three"
               sListArray(04) = "four"
               sListArray(05) = "five"
               sListArray(06) = "six"
               sListArray(07) = "seven"
               sListArray(08) = "eight"
               sListArray(09) = "nine"
               CONTROL ADD LISTBOX, hDlg, %Listbox, sListArray(), 5, 5, 115, 100, _
                           %WS_CHILD OR %WS_VISIBLE OR %WS_VSCROLL OR _
                           %WS_TABSTOP OR %LBS_NOTIFY, %WS_EX_CLIENTEDGE
              
               CONTROL ADD TEXTBOX, hDlg, %Edit, "", 5, 100, 115, 12
               CONTROL SET FOCUS hDlg, %Edit
              
               DIALOG SHOW MODAL hDlg CALL DlgProc
              
              END FUNCTION
              '______________________________________________________________________________ 
              Last edited by George Bleck; 17 Feb 2017, 05:04 PM.
              <b>George W. Bleck</b>
              <img src='http://www.blecktech.com/myemail.gif'>

              Comment


              • #8
                Ah! Now I see!
                In that case, I guess that your timer approach is quite good.
                And since SETTIMER/KILLTIMER occur only on typed key, this is lighter than air. :-)

                A little variant...

                Pierre

                Code:
                #COMPILE EXE '#Win 9.07#
                #DIM ALL
                #INCLUDE "Win32Api.inc"
                
                GLOBAL hDlg AS DWORD
                
                %Edit    = 101
                %Listbox = 201
                %Timer   = 301
                '______________________________________________________________________________
                
                CALLBACK FUNCTION DlgProc
                 LOCAL  sEdit     AS STRING
                 LOCAL  ItemIndex AS LONG
                
                 SELECT CASE CBMSG
                
                   CASE %WM_TIMER
                     CONTROL GET TEXT CBHNDL, %Edit TO sEdit
                     IF LEN(sEdit) THEN
                       ItemIndex = SendDlgItemMessage(hdlg, %Listbox, %LB_FINDSTRING, -1, BYVAL STRPTR(sEdit)) 'Search listbox
                       IF ItemIndex <> %LB_ERR THEN
                         SendDlgItemMessage(hdlg, %Listbox, %LB_SETCURSEL, ItemIndex, 0) 'Select item
                       END IF
                     END IF
                
                   CASE %WM_COMMAND
                     SELECT CASE LOWRD(CBWPARAM)
                       CASE %Edit
                         SELECT CASE CBCTLMSG
                           CASE %EN_CHANGE 'Sent after the screen update
                             KillTimer(CBHNDL, %Timer)
                             SetTimer(CBHNDL, %Timer, 750, BYVAL %NULL)
                          END SELECT
                     END SELECT
                
                 END SELECT
                
                END FUNCTION
                '______________________________________________________________________________
                
                FUNCTION PBMAIN()
                
                 DIALOG NEW %HWND_DESKTOP ,"Search", , , 125, 115, %WS_CAPTION OR %WS_MINIMIZEBOX OR %WS_SYSMENU, 0 TO hDlg
                
                 DIM sListArray(0 TO 9) AS STRING
                 sListArray(00) = "Zero"
                 sListArray(01) = "One"
                 sListArray(02) = "Two"
                 sListArray(03) = "Three"
                 sListArray(04) = "Four"
                 sListArray(05) = "Five"
                 sListArray(06) = "Six"
                 sListArray(07) = "Seven"
                 sListArray(08) = "Eight"
                 sListArray(09) = "Nine"
                 CONTROL ADD LISTBOX, hDlg, %Listbox, sListArray(), 5, 5, 115, 100, _
                 %WS_CHILD OR %WS_VISIBLE OR %WS_VSCROLL OR %WS_TABSTOP OR %LBS_NOTIFY, %WS_EX_CLIENTEDGE
                
                 CONTROL ADD TEXTBOX, hDlg, %Edit, "", 5, 100, 115, 12
                 CONTROL SET FOCUS hDlg, %Edit
                
                 DIALOG SHOW MODAL hDlg CALL DlgProc
                
                END FUNCTION
                '______________________________________________________________________________
                '

                Comment


                • #9
                  Originally I was just reacting to EN_CHANGE, but the data set is just large enough to cause typing to be appear sluggish as search text is entered.
                  Just do the search in a separate thread. That's what I always do with bulky operations, so the main thread has no extra load...
                  Regards,
                  Peter

                  Comment


                  • #10
                    Not sure how how you mean that would be implemented... a new thread on each keypress? That would cause overlapping updates to the control (in my case a listview).
                    <b>George W. Bleck</b>
                    <img src='http://www.blecktech.com/myemail.gif'>

                    Comment


                    • #11
                      No, a separate thread to handle searching and listview update. I use SetEvent() to signal the thread there's work to be done...
                      Regards,
                      Peter

                      Comment


                      • #12
                        George,
                        maybe you need to look at how efficient your search is?

                        I do something very similar to what you describe but I can do the search on 10,000+ items using multiple partial matches and return the first 25 matches in less time than it takes to type the next key. There is no need for any delay.

                        Web browsers often do the same as a URL is typed in, the drop down box shows a number of potential matches of previous websites with what you've typed so far.

                        Google does the same across the internet when you type something into the search box and they index millions of items.

                        There should be no need for timers and delays if you improve the search.

                        Comment


                        • #13
                          Paul Dixon , I am not sure how to build an effective/quick index based search for the type of search i need to perform. The search is a case insensitive partial anywhere match (by necessity).

                          I am dealing with a fixed database of information describing some legacy evidence. The information is both the email addresses of the mailbox and the filename of the evidence itself There is no standard to the email address naming or file name. Many times I am not give very much or accurate criteria to go on. So for example If I need to find all the "Joseph" emails I may need to look for.....

                          Code:
                          john.joseph@company.com           josephj20110127.pst      (lastname, middle of email)
                          joseph.rand@company.com           jrand20120714.pst        (firstname, begin of email)
                          rjoseph@company.com               robjoseph20111216.pst    (lastname, middle of email)
                          yassir@copany.com                 yjosph@20110518.pst      (not part of email address, misspelled in file name)
                          josephabanks@company.com          clothesrecycle20160112   (generic email address, beginning)
                          petermaryandjoseph@company.com    groupa20120515.pst       (generic email address, middle of email)
                          janeuser@company.com              josephja20100612.pst     (match in the file name, maiden name of user)
                          joemama@company.com               joesephmama20111201.pst  (match in file name, proper name of user)
                          josefsmith@company.com            jossmi20090704.pst       (different spelling in email)
                          Note there is no standard in where I find it, and there are even misspellings or alternate spellings, hence the need to look for partial matches so I can eyeball potential missed items. I might need to look for "jo", "ph" etc, to find appropriate matches.

                          Granted these are samples, not actual data, but it accurately describes the problems I run into.
                          Last edited by George Bleck; 18 Feb 2017, 08:32 AM.
                          <b>George W. Bleck</b>
                          <img src='http://www.blecktech.com/myemail.gif'>

                          Comment


                          • #14
                            George,
                            Nothing tested but a few ideas might be..

                            Don't add hundreds of items to a listbox. I suspect adding an item is quite slow.
                            Maybe kill the listbox each time and re-create it ready populated with the new contents supplied in one go using an array.

                            Try doing the population the other way. Instead of searching for each matching item and adding it to the lisbox, try populating the listbox initially with all items and then delete the ones that don't match. (won't work well if the user can backspace while typing).

                            If the displayed part of the listbox is quite small then why populate the whole thing?
                            Initially, you could try just populating the few items that can actually be seen on screen.

                            Maybe use something other than a listbox.
                            A listox is trying to sort and insert and create strings and move strings, which all slow it down.
                            I use rows of labels which can be updated very easily without the overheads of a listbox.


                            I've just tried your original code usin %EN_CHANGE and it doesn't run too slowly on my computer.


                            Paul.

                            Comment


                            • #15
                              The example code I show is actually fast and works perfect for what I am doing. I was asking to see if the method was valid or could potentially cause an issue.

                              I do a killtimer/settimer on each keypress and was curious if that would cause any known issue (have not seen any or read of any).

                              If no one sees any issues and I do not run into any I am going to keep the method as it works perfectly. This was just to "air the idea" and see if it was problematic in any way.
                              <b>George W. Bleck</b>
                              <img src='http://www.blecktech.com/myemail.gif'>

                              Comment


                              • #16
                                Hi George.
                                I thought I once pulled a stunt of making an interger array of characters in a data set using binary of 0 and 1's then just matched on that data having those characters in the search box and I used a base of 0 to mark the array items that matched the individual items.
                                But i do remember building a smaller index of the larger index to make my searches faster while the person types.
                                That should make the search only slow on the first character typed. Yes you have manage another index but the time to build it or reload the 2nd index wil not take long. Possible your second index can be the same size as your primary index and just load up array items that matched from the primary character array.
                                Naturally the fewer characters you have in your array the better like not doing searches on period characters.
                                This did well for me on street addresses.
                                You did say legacy data.
                                From memory I likely did ON Chage.
                                I think I also tried keeping the number count of each character in the array but there was too many errors typed for the search item.
                                I also created a third array to narrow down on any second character typed but it was not worth the coding confusion.
                                So a search on ppurvis would give the same results as purvis, sivrupp or any mix. I also allow for a space in the searched for text to do multiple searches but build that secondary index off the first character in the text string.
                                p purvis

                                Comment


                                • #17
                                  George

                                  I do not think I follow all this debate

                                  But one possibility may be that you take each email address and convert it to all possible search entities. That is - entities that are easy to search on. That process may be quite long. But then your search becomes much quicker.

                                  It depends on the parameters of the system whether this is a good idea or not. It makes more sense for stable data than it does for changing data - and then you have to consider the rate of change etc etc

                                  Kerry
                                  [I]I made a coding error once - but fortunately I fixed it before anyone noticed[/I]
                                  Kerry Farmer

                                  Comment


                                  • #18
                                    Thanks all for the suggestions, but I am not looking for "other ways" to solve this, I am looking to vette if the current, perfectly working solution, could potentially cause problems :-) (as stated in Post #1).

                                    Although Paul Purvis, you made me think of adding a Levenshtein distance capability... that might be my next feature as it is quite easy.. Thanks!
                                    Last edited by George Bleck; 19 Feb 2017, 09:51 AM.
                                    <b>George W. Bleck</b>
                                    <img src='http://www.blecktech.com/myemail.gif'>

                                    Comment


                                    • #19
                                      I suppose George I could of said that I was using a culling method to speed my searches up on real time lookups for bad formatted data or badly formatted search criteria.
                                      p purvis

                                      Comment

                                      Working...
                                      X