Announcement

Collapse
No announcement yet.

TCP - Getting extra FD_READ notifications

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • TCP - Getting extra FD_READ notifications

    I am getting more FD_READ notifications than I should.

    When my server sends me a message (approx 60k in one SEND), I receive an FD_READ and go into my RECV loop until I've received all the data using an 8k buffer. The message is received correctly.

    However, I also sometimes get additional (1 to 3) FD_READ notifications with EOF true (no data) immediately after I've read my message. Where are these coming from? Am I doing something to cause them?

    Thanks

    Garry

  • #2
    Pure SDK syntax? PB-proprietary 'TCP' statements/functions? One from column A, one from column B?
    Michael Mattias
    Tal Systems (retired)
    Port Washington WI USA
    [email protected]
    http://www.talsystems.com

    Comment


    • #3
      I'm using all PB-proprietary functions
      TCP OPEN
      TCP NOTIFY
      TCP RECV

      Comment


      • #4
        Here's my sequence from the point of view of the client (who's receiving the extra FD_READs)
        1. Connect to server
        2. Send signon message (no response from server)
        3. Send subscriptions request (no response from server)
        4. some time later...
        5. Receive message from server
        Reading the message from the server is often followed by additional FD_READs with an empty buffer - If Eof(readFnum) returns true

        Is it possible that the previous messages I sent to the server somehow get ACK'd with an empy buffer from the lower TCP layers (I am not doing any response in my server) ?

        Comment


        • #5
          Originally posted by garry star View Post
          I am getting more FD_READ notifications than I should.
          ...
          Garry
          What is FD_READ? I can't find it either in PB Help or in WinApi?

          Also, Garry, you'll get a lot more responsive/useful help here if you post code. Executable example preferred, or at least a longish snippet of the actual offending code.

          ================================
          And when he is out of sight,
          quickly also is he out of mind.
          Thomas a Kempis
          ================================
          It's a pretty day. I hope you enjoy it.

          Gösta

          JWAM: (Quit Smoking): http://www.SwedesDock.com/smoking
          LDN - A Miracle Drug: http://www.SwedesDock.com/LDN/

          Comment


          • #6
            Some code

            After opening the port, I do a notify:

            Tcp Notify gDsFnum, Recv Close To hMainWnd As %TCP_CONNECT

            The dialogproc handling the NOTIFY messages looks like this

            Function TcpDialogProc(ByVal hDlg As Long, _
            ByVal Msg As Long, _
            ByVal wParam As Long, _
            ByVal lParam As Long) As Long

            Local sBuffer As String
            Function = 0

            Select Case As Long Msg
            Case %TCP_CONNECT
            Select Case LoWrd(LPARAM)

            Case %FD_READ
            If wParam = %INVALID_SOCKET Then Exit Function
            HandleRead(wParam, sBuffer)
            Function = 1
            End Select

            Case Else
            End Select


            HandleRead has a loop to read the actual data
            Do
            If Eof(readFnum) Then
            Debug "Ignore empty buffer"
            Else
            Try
            Tcp Recv readFnum, 8192, sPacket
            Catch
            If Err = 24 Then Debug "Ignore timeout error"
            End Try
            End If
            bytes = bytes + Len(sPacket)
            sBuffer = sBuffer + sPacket
            Loop Until Len(sPacket) = 0 Or IsTrue Eof(readFnum) Or IsTrue Err

            Currently I'm checking for Eof first and just ignoring the additional FD_READ notifications when there is nothing to read

            Comment


            • #7
              http://www.powerbasic.com/support/pb...ad.php?t=13533

              Comment


              • #8
                Gary,

                try putting your code snippet into [ CODE ][ /CODE ] (without the spaces) brackets. This will preserve your source code formatting.

                Example:
                Code:
                Tcp Notify gDsFnum, Recv Close To hMainWnd As %TCP_CONNECT
                
                The dialogproc handling the NOTIFY messages looks like this
                
                Function TcpDialogProc(ByVal hDlg As Long, _
                   ByVal Msg As Long, _
                   ByVal wParam As Long, _
                   ByVal lParam As Long) As Long

                Comment


                • #9
                  Mike
                  I have seen this posted before, but it makes no sense, TCP RECV is a blocking statement so is not returned until 1 of the 3 possible conditions is met ie full buffer, EOF or ERR so sleep is meaningless unless you are saying M$ has a problem. I have programs running PB syntax also used by a number of others that download hundreds of 30K to 60K web pages in a 6 hour period. Computer range in speed from 5YO and laptops to high speed multicore. Internet connections ranging from hopeless 3rd world to 28 MB ADSL2. Never seen the problem.

                  Gary
                  I have to assume the "server" is not a standard web server as the steps you list in 1 to 5 would be strange, ie the client cannot just "receive a file sometime later" unless it was also set up as a server. If it is a standard web server then it should be acknowledging ever transmission from the client and usually the client would not send the next information until that has been received. If you are not using a "keep alive" request then it would normally be sending a memory cookie so that at the next stage it knows who it is dealing with and that the credentials were accepted.
                  TCP is of course an acknowledge protocol unlike UDP in which there is no gaurentee of delivery or error. This is at a low layer level and I don't know what would be passed back to TCP RECV if there were no actual message returned.

                  Comment


                  • #10
                    I spent weeks trying to figure out why packets over 1460 bytes were not reliable without using TCPSafeReceive.
                    I've noticed PB examples usually use 1024 bytes.
                    Note: counting the number of reads needed can not be done as winsock may change the size of each packet.
                    Last edited by Mike Doty; 3 Apr 2009, 12:54 PM.

                    Comment


                    • #11
                      Mike
                      I am not sure what you mean 1460 byte packets, I assume the amount of data you send in a single TCP SEND or TCP PRINT statement, the original ethernet standard allows 1500 bytes of data to be included in a maximum packet size of 1518 bytes, later standards allow maximum packet size of 1522 adding 4 extra control bytes and of course OS's and hardware can increase this on LAN's. Once you go over the internet then the IP protocol consumes 20 bytes of those 1500 so giving a normal MTU of 1480 ( though an IP packet can have a maximum size of 64K bytes) and I have seen a number of carriers halve this for various reasons. If the carrier uses ATM then the packets are broken down to 48 bytes of data (into which is all the IP headers) + 5 bytes of overhead for its own protocol.
                      The point of that long description? This is all handled at lower levels than the normal application can see and of course protocol and MTU changes across the internet are never seen, so should be totally ignored as the data should be fragmented and reassembled as needed.
                      For most downloads I use a 4K buffer and rarely use a callback notify scheme rather i put that whole portion of the internet access in a thread and allow my main program to carry on doing anything else it can while the thread is suspended with all the blocking statements. When the data is received (and in my case parsed) I pass it back to the main program (or an error) obviously using an appropriate synchronisation, normally one or more mutexs. I have never had to use this "safe" method or had problems.
                      John

                      Comment


                      • #12
                        Gosta,

                        In Help look in TCP NOTIFY and UDP NOTIFY. The value for %FD_READ in WSOCK32.INC

                        Cheers,
                        Dale

                        Comment


                        • #13
                          Originally posted by Dale Yarker View Post
                          Gosta,

                          In Help look in TCP NOTIFY and UDP NOTIFY. The value for %FD_READ in WSOCK32.INC

                          Cheers,
                          Thanks Dale. I figured it was something like that (a specific Include) but as code was not shown ....

                          =====================================
                          "Ask her to wait a moment
                          I am almost done."
                          Carl Friedrich Gauss (1777-1855),
                          while working,
                          when informed that his wife is dying
                          =====================================
                          It's a pretty day. I hope you enjoy it.

                          Gösta

                          JWAM: (Quit Smoking): http://www.SwedesDock.com/smoking
                          LDN - A Miracle Drug: http://www.SwedesDock.com/LDN/

                          Comment


                          • #14
                            In help, look for the code in:
                            An ECHO client and server using TCP

                            Comment


                            • #15
                              Mike
                              I hope you don't mind me continuing in this thread but it has caused me a lot of thinking. In both the problems you have quoted and Gary they have been on the client side of a TCP application. I can't understand where TCP NOTIFY fits in as it is meant for the server side of an application.
                              John

                              Comment


                              • #16
                                John,
                                This isn't my thread.
                                I'm not sure why NOTIFY was used for help on %FD_READ.
                                I pointed to the help file on using %FD_READ.
                                The complete Echo Server and Echo Client sample can be found in your PB\SAMPLES\INTERNET\TCP folder.
                                Finally, it should be noted that there is no direct correlation between the number of TCP SEND statements executed, compared to the number of %FD_READ messages received. This is because Winsock may concatenate multiple data packets and issue a lesser number of %FD_READ messages in response. Therefore, it is usually necessary to construct your code so that it continues to read data from the incoming data stream until either the returned string is empty, or an error is detected. For example:
                                DIM InBuffer AS STRING
                                ...
                                CASE %FD_READ
                                InBuffer = ""
                                IF hEcho = %INVALID_SOCKET THEN EXIT SELECT

                                DO
                                TCP RECV hEcho, 1024, buffer
                                IF LEN(buffer) = 0 OR ISTRUE ERR THEN EXIT LOOP
                                InBuffer = InBuffer + buffer
                                TCP SEND hEcho, buffer
                                LogEvent $DQ + Buffer + $DQ
                                LOOP
                                ...
                                Code:
                                Here is a threaded client that I used to find the problem I've mentioned:
                                There is no NOTIFY on the client side as mentioned.
                                 
                                #PBFORMS CREATED V1.51
                                #COMPILE EXE
                                #DIM ALL
                                '------------------------------------------------------------------------------
                                '   ** Includes **
                                '------------------------------------------------------------------------------
                                #PBFORMS BEGIN INCLUDES
                                #IF NOT %DEF(%WINAPI)
                                    #INCLUDE "WIN32API.INC"
                                #ENDIF
                                #PBFORMS END INCLUDES
                                '   ** Constants **
                                '------------------------------------------------------------------------------
                                #PBFORMS BEGIN CONSTANTS
                                %IDD_DIALOG1  = 1001
                                %IDC_LISTBOX1 =  201
                                #PBFORMS END CONSTANTS
                                '------------------------------------------------------------------------------
                                '   ** Declarations **
                                '------------------------------------------------------------------------------
                                DECLARE CALLBACK FUNCTION ShowDIALOG1Proc()
                                DECLARE FUNCTION ShowDIALOG1(BYVAL hParent AS DWORD) AS LONG
                                DECLARE FUNCTION AttachMENU1(BYVAL hDlg AS DWORD) AS DWORD
                                #PBFORMS DECLARATIONS
                                '------------------------------------------------------------------------------
                                %PacketLength       = 10000 'whatever you want
                                %TimesToSendReceive = 500
                                %Threads            = 10
                                $Server             = "127.0.0.1"
                                GLOBAL gCS AS CRITICAL_SECTION, ghDlg AS DWORD
                                '------------------------------------------------------------------------------
                                '   ** Main Application Entry Point **
                                '------------------------------------------------------------------------------
                                 FUNCTION PBMAIN AS LONG
                                  LOCAL result AS LONG
                                 
                                  REDIM gCounter(%Threads) AS LONG
                                 
                                  InitializeCriticalSection gCS
                                  ShowDIALOG1 %HWND_DESKTOP
                                 
                                  DO WHILE THREADCOUNT > 1
                                    DIALOG SET TEXT ghDlg, "Users"+ STR$(THREADCOUNT-1)
                                    SLEEP 1000
                                  LOOP
                                 
                                DeleteCriticalSection     gCS
                                END FUNCTION
                                SUB StartThread(ThreadNumber AS LONG)
                                  LOCAL result AS DWORD
                                  THREAD CREATE TEST(ThreadNumber) TO RESULT
                                  THREAD CLOSE result TO result
                                END SUB
                                '   ** CallBacks **
                                '------------------------------------------------------------------------------
                                CALLBACK FUNCTION ShowDIALOG1Proc()
                                    SELECT CASE AS LONG CBMSG
                                        CASE %WM_INITDIALOG
                                            ' Initialization handler
                                 
                                        CASE %WM_NCACTIVATE
                                            STATIC hWndSaveFocus AS DWORD
                                            IF ISFALSE CBWPARAM THEN
                                                ' Save control focus
                                                hWndSaveFocus = GetFocus()
                                            ELSEIF hWndSaveFocus THEN
                                                ' Restore control focus
                                                SetFocus(hWndSaveFocus)
                                                hWndSaveFocus = 0
                                            END IF
                                 
                                        CASE %WM_DESTROY
                                        CASE %WM_COMMAND
                                            ' Process control notifications
                                            SELECT CASE AS LONG CBCTL
                                               CASE 101 TO 109  'Button Numbers 1-9 start a thread
                                                 IF CBCTLMSG = %BN_CLICKED OR CBCTLMSG = 1 THEN
                                                   StartThread CBCTL-100
                                                 END IF
                                            END SELECT
                                        CASE %WM_SYSCOMMAND
                                            IF (CBWPARAM AND &HFFF0) = %SC_CLOSE THEN
                                              'If threads still running, don't end dialog
                                              Lb  "X clicked. Threads running" + STR$(THREADCOUNT)
                                              IF THREADCOUNT > 1 THEN
                                                 RemoveMenu(GetSystemMenu(CBHNDL, %True), %SC_CLOSE, %MF_BYCOMMAND)
                                                 drawMenuBar CBHNDL
                                                 FUNCTION = 1
                                              END IF
                                            END IF
                                    END SELECT
                                END FUNCTION
                                '------------------------------------------------------------------------------
                                '------------------------------------------------------------------------------
                                '   ** Dialogs **
                                '------------------------------------------------------------------------------
                                FUNCTION ShowDIALOG1(BYVAL hParent AS DWORD) AS LONG
                                    LOCAL lRslt AS LONG
                                #PBFORMS BEGIN DIALOG %IDD_DIALOG1->->
                                    LOCAL hDlg  AS DWORD
                                    DIALOG NEW hParent, "Dialog1", 355, 282, 270, 117, %WS_POPUP OR _
                                        %WS_DLGFRAME OR %WS_THICKFRAME OR %WS_CAPTION OR %WS_SYSMENU OR _
                                        %WS_CLIPCHILDREN OR %WS_VISIBLE OR %DS_MODALFRAME OR %DS_CENTER OR _
                                        %DS_3DLOOK OR %DS_NOFAILCREATE OR %DS_SETFONT, %WS_EX_CONTROLPARENT _
                                        OR %WS_EX_LEFT OR %WS_EX_LTRREADING OR %WS_EX_RIGHTSCROLLBAR, TO _
                                        hDlg
                                    CONTROL ADD LISTBOX, hDlg, %IDC_LISTBOX1, , 0, 15, 270, 87, %WS_CHILD OR _
                                        %WS_VISIBLE OR %WS_VSCROLL, %WS_EX_CLIENTEDGE OR %WS_EX_LEFT OR _
                                        %WS_EX_LTRREADING OR %WS_EX_RIGHTSCROLLBAR
                                #PBFORMS END DIALOG
                                    ghDlg = hdlg
                                    DIALOG SET TEXT hDlg, "(" + $Server + ")  Packet length" + STR$(%PacketLength)
                                    LOCAL ButtonNumber AS LONG
                                    FOR ButtonNumber = 1 TO 9
                                         CONTROL ADD BUTTON,  hDlg, ButtonNumber + 100, "User" + STR$(ButtonNumber), (ButtonNumber-1)*30, 0, 30, 15, _
                                        %WS_CHILD OR %WS_VISIBLE OR %BS_TEXT OR %BS_PUSHBUTTON OR %BS_CENTER _
                                        OR %BS_VCENTER, %WS_EX_LEFT OR %WS_EX_LTRREADING
                                    NEXT
                                    DIALOG SHOW MODAL hDlg, CALL ShowDIALOG1Proc TO lRslt
                                #PBFORMS BEGIN CLEANUP %IDD_DIALOG1
                                #PBFORMS END CLEANUP
                                    FUNCTION = lRslt
                                END FUNCTION
                                THREAD FUNCTION TEST(BYVAL ThreadNumber AS LONG) AS LONG
                                   LOCAL counter&,nSocket&, sBuffer$, sPacket$, Errors&, Result AS LONG
                                   LOCAL sDataSent AS STRING
                                   nSocket = FREEFILE   'Always send complete packets until the end!!!
                                   TCP OPEN PORT 999 AT $Server AS nSocket TIMEOUT 5000
                                   IF ERR THEN LB "Error opening":EXIT FUNCTION 'put into listbox
                                   DIALOG SET TEXT ghDlg, "Users"+ STR$(THREADCOUNT-1)
                                   RemoveMenu(GetSystemMenu(ghDlg, %false), %SC_CLOSE, %MF_BYCOMMAND)
                                   DrawMenuBar ghDlg 'don't allow ending
                                FOR Counter = 1 TO %TimesToSendReceive
                                    'send data here
                                    sDataSent = STRING$(%PacketLength,"X")
                                    TCP SEND nSocket, sDataSent
                                    IF ERR THEN
                                      LB "Send error" + STR$(ERRCLEAR) + " in Thread" + STR$(ThreadNumber) + " attempt" + STR$(Counter)
                                      EXIT FOR   'put into listbox
                                    END IF
                                    sPacket = "":    sBuffer = ""
                                    'receive data back here
                                    result = tcpSafeReceive(nSocket, %PacketLength,sPacket)   'Don Dickinson
                                    IF result THEN   'success
                                       'double check that data sent is the same as data that came back
                                       IF sPacket <> sDataSent THEN
                                         INCR Errors
                                          LB "Receive error, data sent does not match data received"
                                          EXIT FOR  'exit on first error
                                        'ELSE
                                        '  ? sPacket
                                        END IF
                                    ELSE
                                       LB "Receive error, user" + STR$(ThreadNumber)  'put into listbox
                                       EXIT FOR
                                    END IF
                                NEXT
                                  TCP CLOSE nSocket
                                  result = THREADCOUNT -1
                                  IF result =  1 THEN    'if only main thread then
                                    RemoveMenu(GetSystemMenu(ghDlg, %True), %SC_CLOSE, %MF_BYCOMMAND)  'enable
                                    DrawMenuBar ghDlg
                                    DIALOG SET TEXT ghDlg, "No users"
                                  ELSE
                                    DIALOG SET TEXT ghDlg, "Users" + STR$(result)
                                  END IF
                                END FUNCTION
                                '------------------------------------------------------------------------------
                                'tcpSaveReceive.Inc
                                 
                                FUNCTION tcpSafeReceive(BYVAL hSocket AS LONG, BYVAL iBufferLen AS LONG, _
                                                        recBuff AS STRING) AS LONG
                                   DIM iLeft AS LONG
                                   DIM sBuffer AS STRING
                                   recBuff = ""
                                   iLeft = iBufferLen
                                   DO
                                      sBuffer = SPACE$(iLeft)
                                      ON ERROR RESUME NEXT
                                      sBuffer = SPACE$(iBufferLen)
                                      TCP RECV hSocket, iLeft, sBuffer
                                      IF ERR THEN
                                         FUNCTION = %False
                                         EXIT FUNCTION
                                      END IF
                                      recBuff = recBuff + sBuffer
                                      IF LEN(recBuff) >= iBufferLen THEN
                                         EXIT DO
                                      END IF
                                      iLeft = iBufferLen - LEN(recBuff)
                                      SLEEP 5
                                   LOOP
                                   FUNCTION = %True
                                END FUNCTION
                                SUB EnterCritical
                                   EnterCriticalSection gCS
                                END SUB
                                SUB LeaveCritical
                                   LeaveCriticalSection gCS
                                END SUB
                                SUB LB(s AS STRING)
                                  LISTBOX ADD ghDlg, %IDC_LISTBOX1, s
                                END SUB

                                Comment


                                • #17
                                  What I learned

                                  FYI, based on what I read here and previous work with TCP in C++, I modified my message to have a constant sized header with the length within the first few bytes (20). I then do

                                  recv of this header (size 20)
                                  parse the length value
                                  recv of the remainder of the message (may still need a recv loop)

                                  The sender is another pwbasic program that sends the entire message (60k or more) in one send, after constructing the header in the first 20 bytes

                                  This works reliably (and quickly) for me.

                                  Garry

                                  Comment


                                  • #18
                                    tcp send size limit?

                                    Is there a limit to the size of message that I can call Tcp Send with? I'm sending up to 500k and it seems to work but I was just wondering.

                                    Thanks...

                                    Comment


                                    • #19
                                      Originally posted by garry star View Post
                                      FYI, based on what I read here and previous work with TCP in C++, I modified my message to have a constant sized header with the length within the first few bytes (20). I then do
                                      ...

                                      This works reliably (and quickly) for me.

                                      Garry
                                      Well, it isn't quite as reliable as I thought.

                                      I am doing at least 2 Tcp Recvs for each message (first to get the length, second to read the remainder of the message.) I then ignore any FD_READ if there is no data to be read. However, once in a while, I must be ignoring an FD_READ that I shouldn't since my server gets into a state where it doesn't receive an FD_READ when my client sends (and there are no errors on the send).

                                      I think this means that I need to go back to doing a single Tcp Recv for each FD_READ and then continue reading/accumulating my message only after receiving another FD_READ.

                                      Can anyone confirm that this is the correct approach?

                                      Thanks!

                                      Garry

                                      Comment


                                      • #20
                                        See help file under echo server.
                                        See TcpSafeReceive in my posting above.
                                        the complete echo server and echo client sample can be found in your pb\samples\internet\tcp folder.
                                        Finally, it should be noted that there is no direct correlation between the number of tcp send statements executed, compared to the number of %fdçi6d messages received. This is because winsock may concatenate multiple data packets and issue a lesser number of %fd_read messages in response. Therefore, it is usually necessary to construct your code so that it continues to read data from the incoming data stream until either the returned string is empty, or an error is detected.

                                        Comment

                                        Working...
                                        X