No announcement yet.

Time to Check New Emails - Slow!

  • Filter
  • Time
  • Show
Clear All
new posts

  • Time to Check New Emails - Slow!

    When checking for new email, the Outlook progress bar shows that the process takes just a couple of seconds. I have typically about 300 or so emails on the server at at any time.

    However, in my own email client, the process of checking for new emails takes perhaps 30s or so - very noticeably longer than Outlook.takes - painfully slow!

    My approach is simply to query the server about how many emails it has, then call for the UID of each email on the server. I compare the received UIDs to a local database of UIDs that I've downloaded. I download emails with a new UID and add that UID to the local database. Just checking which emails need downloading is the lengthy part of the process.

    I'm at a loss to explain how Outlook downloads new emails so quickly, whereas my approach takes so long.

    Is there a different approach than the one I described that anyone uses that compares speed-wise with what Outlook seems to be able to do?

  • #2
    Barring a single thread solution, one approach might be to get the number of emails on the server, then create multiple threads, each of which connects to the server and queries the UID for a subset of the available emails.

    So, for example, if the server has 200 emails, then perhaps create 10 threads where each thread requests the UID for 10 emails (1-20, 21-30, 31-40, etc.).

    I'm not sure what the limitation is on the number of server connections, nor what would be an optimum number of threads.

    Is there a downside to a multi-thread solution? I guess some experimentation is in order to see what issues the thread approach brings to the party.


    • #3
      But, a multi-thread solution is not very satisfying. It is surprising to me that querying a server for the UID of available emails would take as long as I've described. Nor having worked with email servers all that much, perhaps I simply have an unreal expectation !?


      • #4
        300 messages or 300 unread messages?
        How many accounts are being checked?
        Is this a pop3 email checker?

        If so, I only check counts and don't read the emails here.
        The email reader is launched if new email.
        Full program available upon request by email.
        No UID checking.

        THREAD FUNCTION CheckEmailThread(BYVAL x AS DWORD) AS LONG  'check all accounts
            IF gEndEmailCheckerFlag THEN BEEP:EXIT DO
            TotalNumberOfMessages = 0
            'put each account to check here on a single line
            sHost = "":sUser = "user1":sPassword="xxxx"  :GOSUB doit
            sHost = "":suser = "abc"  :sPasword ="yyyy"  :GOSUB doit
            IF gRunOnce THEN
              gDuration = gRunOnce 'restore previous duration in seconds
              gRunOnce = 0
            END IF
            IF TotalNumberOfMessages THEN  'must be 1 or more messages
             IF TotalNumberOfMessages > TotalNumberOfMessagesPrevious THEN  'and more than previous loop
                 result = SHELL($LaunchEmailProgram)
                SLEEP 1000
                END IF
             END IF
            TotalNumberOfMessagesPrevious = TotalNumberOfMessages
            Counter = 0
              IF gRunOnce THEN EXIT DO  'get out as soon as detected
              INCR Counter
              IF gEndEmailCheckerFlag THEN EXIT LOOP
              IF Counter => gDuration THEN EXIT DO
              SLEEP 1000
          LOOP UNTIL gEndEmailCheckerFlag
          ghDlg = 0  'end dialog modeless message pump
        DoIt: 'check account
          Result = Pop(sHost,sUser,sPassword,NumberOfMessages&,MsgSize&)
          IF NumberOfMessages THEN
             TotalNumberOfMessages = TotalNumberOfMessages + NumberOfMessages
          END IF
          SLEEP 50
        END FUNCTION instead of google


        • #5
          Hi Mike, thanks for responding!

          Single account, 300 emails on a POP3 server.

          The only way I know to determine if an email has been "read" - meaning downloaded to my client - is if the UID of the email on the server matches the one I keep on my PC.

          In that case, your question of 300 on the server vs 300 unread doesn't apply, does it?

          After I check the number of emails on the server, let's say there are 300, then I send 300 requests to the server for the UID of each email on the server, which I compare to a list I keep locally.

          Do you know of a way to determine if an email has been downloaded previously other than by checking the UID?

          All I want to do is download new emails, but since there's no "already downloaded" flag on the server, then tracking the UIDs is the only approach I'm aware of for determining which emails are new.


          • #6
            Yes, delete from server after read.
            Pop3 email checker running in a loop.
             count = checkcount
             if count then
              'shell to eMclient.exe  (read_email_from_server,delete_email_from_server)
              sleep xxxxx
            loop until gdone
   instead of google


            • #7
              I don't want to delete from the server so that my phone will have access to the email as well.

              I do delete emails from the server, but only after they are a certain age (15 days or so).


              • #8
                read email, forward copy to another account, delete email.
       instead of google


                • #9
                  My approach is simply to query the server about how many emails it has, then call for the UID of each email on the server. I compare the received UIDs to a local database of UIDs that I've downloaded.
                  So are you issuing a STAT or LIST pop3 command, then get UIDL x (x being the email number)?
                  or are you getting complete UIDL list in one request by using UIDL without a value?


                  • #10
                    Sorry but just the one account. Not wanting to make it difficult - just what it is.

                    I did not know there was an "all" option. When you get the list, is it in the same order as the email number? That sounds like a very good idea. I'll go look into that immediately. I can get the list, then do a replace of the local UID in the list, leaving behind only the UIDs I haven't downloaded - or some other approach.

                    I'll let you know how it works ... ok, couldn't wait ... getting the list took about 0.3s. Very cool! I need to work with the returned list to make sure it's what I think I asked for, but if it's the list I need, the 30s drops down to almost zero.

                    Thanks for the tip!


                    • #11
                      I changed emClient to not delete emails on server until I delete them in my emClient.
                      Downloading email does not affect the count so if count > previous_count then "you have mail"
                      I don't have 300 emails yet to see how long it takes.

             instead of google


                      • #12
                        Hey Mike,
                        Your email world and mine have some differences. I keep all emails locally forever - deleting only spam for the most part. But, I could have a "Delete from Server" command in my email client to help keep the online content from growing without bounds.


                        • #13

                          I offer this in case it may be useful. Take with a grain of NaCl.

                          Managing email is such a personal thing. As a k-12 school admin with 3000 users, I have seen it all. Not that you are in that group Gary.

                          What I expect it to have all my messages on all my clients/devices; PC I’m using eM and outlook; and to be able to search them all.


                          My work email goes back to 1999 and personal back to 1983. What I have been doing is to move a copy of all new messages as they arrive to a folder called “Z!.AllMail” so it sorts correctly.

                          And when I’m done with a message I just delete it from the inbox.

                          After I migrated one of my schools to Exchange 2013 then to 2016 other users and I started having issues syncing large amounts of messages (over 30000/folder). It looked like a timeout issue. My solution was to keep only 15k-20k messages to each folder. My looks like:

                          Z!.AllMail (current year)

                          And I know that this says nothing about your original query, Sorry

                          BASIC shampoo - DO:LATHER:RINSE:LOOP UNTIL CLEAN <>0


                          • #14
                            Hi Frank!

                            Thanks for your comments. It's always interesting to see how other folks handle similar tasks.

                            You're describing a boatload of emails. Up to 20K per year over a period of a couple of decades? You're way more into email than I am!

                            called “Z!.AllMail” so it sorts correctly.
                            I use that technique myself, with an underscore for items I want at the top and Z for items I want at the bottom. I do that in file/folder naming as well so that File Explorer puts things where I want them to be.


                            • #15
                              I've been working with the option to download the list of UIDs in a single command and it continues to work quite well - dropping the wait time for getting new emails to just a few seconds (other than the first time where I actually download hundreds of emails). The documentation for the MarshallSoft See4PB product discusses the list option but I just never noticed it until you mentioned it. Thanks again!


                              • #16
                                Gary -- since you have the SEE4PB library, have you given thought to using IMAP rather than POP?
                                Real programmers use a magnetized needle and a steady hand


                                • #17
                                  Hi Bud!
                                  Thought about it, yes. But what I've read doesn't indicate there's any significant advantage for cases like mine where all I"m doing is downloading emails from the server to the PC.

                                  If you have some arguments in favor of IMAP, I'm happy to listen.


                                  • #18
                                    I guess somewhere in the conversation I got the idea that there was a requirement or desire to leave e-mail on the server. If you are following the usual POP practice of download, then delete from server, then you are right, probably no real difference.
                                    Real programmers use a magnetized needle and a steady hand


                                    • #19
                                      He wrote the email client.
                                      I just check the number of emails on the server with a simple pop3 checker that shells to emClient.
                                      I haven't noticed any difference in checking the count leaving them on the server.
                                      I think he could could forward them to multiple email addresses on the server and delete.
                                      Several email servers I have used (in the forward section) add,, with option to save a copy if wanted.

                                      He is actually downloading them and leaving on the server so multiple devices can access them.
                                      I think it could be done using forwarding, but he is the actual email client so it may not apply.

                                      All I know about IMAP is it synchronizes each device in real-time (which could be an advantage.)
                             instead of google


                                      • #20
                                        Yes, I leave the email on the server so that other devices can access the email.

                                        The suggestion from Rod, to download the entire list of available UIDs on the server at one time rather than asking for one UID at a time solved the problem for me.
                                        Downloading the list seems to take barely a second or so, then my local app makes short work of comparing the server list and my local list of previously download UIDs.