Announcement

Collapse
No announcement yet.

Client/Server Problems

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Client/Server Problems

    Hi Everyone,

    I am trying to convert my Cheetah database product to a client/server
    format. I have run into a problem. I try to add a bunch of records in
    a loop and it takes forever to do it (e.g. 25 records takes over 8 seconds).
    I dusted off my old copy of CodeBase and 25 records takes a second (in
    Client/Server mode).

    I am obviously missing the point somewhere along the way.

    Here is what I have done: I have one main DLL which works with both the
    client EXE and the server EXE. This DLL contains all of my database and
    index routines. When the client app starts, a call is made to a server
    connection routine found in the DLL. The DLL connects to the server and
    saves the server's handle in a global variable. From here on everytime the
    client app calls the DLL, the DLL checks to see if there is a server handle
    available. If it is, then the request is simply packaged in a string
    and sent to the server. The server then dispatches the request to its
    own DLL to perform the database function. The result (usually an errorcode)
    is then sent back to the client. That's it.

    On the client side:
    I connect using "TCP OPEN PORT %DEFAULT_PORT AT m_Server$ AS hSocket&"
    hSocket& is then saved to a global variable. I have never had a problem
    connecting.

    Say for example the "create database" is now called by the client. The
    DLL checks the global hSocket& > 0 and if it is then creates a send buffer
    and sends it using "TCP SEND hSocket&, Buffer$". Right after that there is
    a "TCP RECV hSocket&, LEN(Buffer$), Buffer$". Then the function exits.

    On the Server side:
    I am (now) using Erik Olsen's HTTP server as a guide although I experienced
    the same problem with Don Dickinson's rserver (Sorry Don you can't get me
    via email).

    When %FD_ACCEPT is fired the following occurs: "TCP ACCEPT fnum AS hSocket"
    and then a new thread is created "THREAD CREATE SocketThread(hSocket) to idThread(hSocket)"
    In the thread I use "TCP RECV hSocket&, DataLen, Buffer$" to receive the
    data being sent from the client. That buffer is processed to obtain the request
    and a SELECT/CASE determines which database routine in the DLL to call. The DLL
    routine does the processing and sends a "TCP SEND" back to the client.

    Does this methodology sound correct??? Any ideas about why this could be
    so slow? I know its not much to go on but the code is so huge there is no
    way I would be able to post it here. Maybe I am opening and closing
    too many connections. I apologize for being such a rookie at this.

    For those of you who have developed similar systems - have you experienced
    similar slow results when sending many TCP request to the server in succession.

    I am doing all the tests on a LOCAL machine - not over a network. This
    doesn't seem to be a factor because CodeBase runs fast on the same machine....

    Any help is greatly appreciated.

    Thanks,







    ------------------
    Paul Squires
    mailto:[email protected][email protected]</A>
    Paul Squires
    FireFly Visual Designer (for PowerBASIC Windows 10+)
    Version 3 now available.
    http://www.planetsquires.com

  • #2
    This sounds about right, but just so we're on the same page - here's how I implemented a similar thing (codebase works like this, I'm pretty sure) ...

    1...
    Connect to server (and stay connected until an explicit disconnect is ordered)

    2...
    Append calls are buffered in a transaction (in memory). When the transaction is done (the user is done appending records), the whole append buffer is sent in one block.

    3...
    On the server, this block is processed, the records appended, and notice sent back to the client that it was completed.

    4...
    Continue processing other calls - when done, disconnect.

    Is that your approach? I suspect you might be sending the appends one at a time - if so, you'll save alot of time by buffering them and sending them in a block.

    Best Regards,
    Don

    p.s. still don't know what's up with the email - glad we have the web board Feel free to email me any code you want me to look at.


    ------------------
    dickinson.basicguru.com
    Don Dickinson
    www.greatwebdivide.com

    Comment


    • #3
      Don, you are spot on! I was appending one at a time. I wasn't sure
      if I was approaching the TCP syntax correctly and maybe that was
      what was slowing down the appends. It makes much more sense to
      buffer these tasks on the client and then process them in blocks
      on the server.

      I suspect you send blocks of 8K to 16K maybe??? I remember reading
      posts in POFFS from people who tested network connections and
      if I am not mistaken the suggestion was 16K - but I'm not sure - I'll
      have to search this info out again.

      Thanks for your help Don, I plan to switch my code back to the
      rserver skeleton because I find it easier to follow and less
      likely that I will screw up threads.

      Thanks,



      ------------------
      Paul Squires
      mailto:[email protected][email protected]</A>
      Paul Squires
      FireFly Visual Designer (for PowerBASIC Windows 10+)
      Version 3 now available.
      http://www.planetsquires.com

      Comment


      • #4
        I don't recall exactly the block size I'm using - I think it's a constant e.g. %XFER_BLOCK_SIZE = 16384 or whatever so I can change it at will.
        I think you can go up to 64k without too many troubles, but I don't know what the optimal size is.
        I usually put a header structure in front of the data structure that tells the server how much data to expect. The server knows it needs to grab X bytes for the header. After it's grabbed those, then it checks the header's datalength member to see if it should wait for more. The header also contains a checksum, an instructional code (e.g. append data), and other misc data (the table handle) information.
        I might have time to work rserver into a multi-threaded model in the near future. If I can, I'll post the code at an ftp site somewhere and post notice here.

        Best Regards,
        Don


        ------------------
        dickinson.basicguru.com
        Don Dickinson
        www.greatwebdivide.com

        Comment


        • #5
          Don,

          Of the 7 or 8 functions that I have converted so far, I have used
          a header block like you describe - similar to the way you outline
          in your rserver code. Using a checksum is a very good idea that I
          will also incorporate just to be safe.

          I would be very interested in any changes you decide to make to
          your rserver code

          I also looked at Erik Olsen's HTTP server code. That code has
          a nice feature of running the server from a tray icon. I think
          I'll "steal" that code and integrate it as well.

          Thanks,



          ------------------
          Paul Squires
          mailto:[email protected][email protected]</A>
          Paul Squires
          FireFly Visual Designer (for PowerBASIC Windows 10+)
          Version 3 now available.
          http://www.planetsquires.com

          Comment


          • #6
            One more thing ... i also put flags in the header for encryption and compression. I use zlib for compression (it's free, fast, and simple). Just compress the buffer and and make an entry in the header for the un-compressed and compressed buffer sizes. The compression helps speed the transmission.
            --Don

            ------------------
            dickinson.basicguru.com
            Don Dickinson
            www.greatwebdivide.com

            Comment


            • #7
              Don, if you find the time would you be so kind enough as to post an example of using zlib simply to compress/decompress strings? I've used it to unpack a zip before and it works an absolute treat for that, but I'd also love to know how to apply it to strings rather than files, that would add another dimension to things

              Thanks,
              Wayne


              ------------------
              -

              Comment


              • #8
                Don. that's another good idea! You say you encrypt *and* compress. Are
                you using two separate routines for this or are you considering the
                zlib compression to be your "encryption" ?

                Also, does zlib come with source code so I can compile it directly
                into my DLL (so I don't have to distribute a separate DLL). Then again
                I guess any string compression routine (as long as it is fast) would
                be a good idea.

                I'll have to search POFFS to see what some of the Forum Gurus have
                contributed in the past. Hopefully there may be some inline assembler
                routines that could do the trick.

                Thanks,



                ------------------
                Paul Squires
                mailto:[email protected][email protected]</A>
                Paul Squires
                FireFly Visual Designer (for PowerBASIC Windows 10+)
                Version 3 now available.
                http://www.planetsquires.com

                Comment


                • #9
                  For encryption - I usually use my own algorithm. There are various algorithms floating around here as well as a wrapper for the M$ crypto api that i wrote.
                  Per compression - I don't know of a way to easily embed zlib. I did write some lz77 compression routines (posted to the source forum, i think) that I use in that situation. They work good as long as the data is small (< 1mb)

                  --Don

                  ------------------
                  dickinson.basicguru.com
                  Don Dickinson
                  www.greatwebdivide.com

                  Comment


                  • #10
                    Wayne, I'll post my partial translation of the zlib header and the wrapper I wrote in the code forum shortly. The compression of strings does add 4 bytes of overhead to specify the length of the string. The wrapper is a good start or example for someone, but you can definately poke holes in the efficiency of what I've done. I should also give credit to James Fuller - I can't remember the extent I borrowed from him as I wrote this a while back, but I know he helped or I copied some from him.

                    --Don

                    ------------------
                    dickinson.basicguru.com
                    Don Dickinson
                    www.greatwebdivide.com

                    Comment


                    • #11
                      paul, i just emailed you a file attachment with some code that might help - it's a beta, multi-threaded version of rserver. hopefully the email will go through. if not, let me know and i'll make alternate delivery arrangements.

                      Best regards,
                      Don

                      ------------------
                      dickinson.basicguru.com
                      Don Dickinson
                      www.greatwebdivide.com

                      Comment


                      • #12

                        I believe there are restrictions on using zlib in a commercial application???

                        James

                        ------------------

                        Comment


                        • #13
                          Originally posted by jcfuller:

                          I believe there are restrictions on using zlib in a commercial application???

                          James

                          What problem?

                          http://www.info-zip.org/pub/infozip/...b_license.html

                          ------------------
                          Best Regards
                          Peter Scheutz

                          [This message has been edited by Peter Scheutz (edited June 01, 2001).]
                          Best Regards
                          Peter Scheutz

                          Comment


                          • #14
                            My emails to you are still getting rejected - no big deal, below is a link to the code ...

                            63.225.155.227/don/rserver2.zip

                            Any feedback would be great - if it works ok for both of us, then I'll post it on my code page of basicguru.com

                            Best Regards,
                            Don

                            ------------------
                            dickinson.basicguru.com
                            Don Dickinson
                            www.greatwebdivide.com

                            Comment


                            • #15
                              Thanks Don, I just downloaded it. I will be on the road next
                              week so I'll test it out while on the plane or in the hotel. I'll
                              let you know if I experience any problems with it once I get
                              back next Friday.

                              Thanks,



                              ------------------
                              Paul Squires
                              mailto:[email protected][email protected]</A>
                              Paul Squires
                              FireFly Visual Designer (for PowerBASIC Windows 10+)
                              Version 3 now available.
                              http://www.planetsquires.com

                              Comment

                              Working...
                              X