TCP data transmission

[Vince Delbrugge]

I have an application where we transmit fields of data and we need to use network communications. I transmit about a dozen fields in a record, transmitting one record per second.

I have written some code that works- at least for a while. I transmit the code using a program compiled with PB/CC 5.0, and I have been receiving my sample code on another PC using the commercial software Hyperterminal.

The problem is that after some time, the Hyperterminal software locks-up. The Task Manager indicates that it is not responding.

If I shorten the transmitted record, i.e., fewer fields, then the time that it takes the Hyperterminal to lock-up increases. This sounds like some sort of buffering capacity problem. I attached a concise example of my code below that gives this problem below. I would appreciate any feedback and suggestions.

#COMPILE EXE
#DIM ALL
FUNCTION PBMAIN () AS LONG

TCP OPEN PORT 23 AT "192.168.1.102" AS #2
DIM asciistring AS STRING
WHILE NOT INSTAT
asciistring="15,0,8.6544,.5,8.35,49.28,0,0,9.86,3,0,0,6.57,2"
TCP PRINT #2, asciistring$
SLEEP 1000
WEND

TCP CLOSE #2
END FUNCTION

[Mike Doty]

Can the data contain 13,10 which might be interpreted incorrectly?
IF ERR THEN after the TCP PRINT might help trap the error.

[Vince Delbrugge]

Thank you for the reply.

I modified the code as its shown below and ran it with the same results on the Hyperterminal. My program continued to execute, and I terminated normally. I don't have any experience using the debugging options, so I am not sure that I coded it correcly.

How does the 13,10 data apply? Should it be a part of the data string?

Here is another note. If I use Hyperterminal instead of my program to transmit data from a file, then I do not have any problems on the receiving PC. I need my program to transmit the data in real-time.

I attached an image of the Hyperterminal window after the error for reference.

#COMPILE EXE
#DIM ALL
FUNCTION PBMAIN () AS LONG
PRINT "start"
TCP OPEN PORT 23 AT "192.168.1.102" AS #2
DIM asciistring AS STRING
WHILE NOT INSTAT
asciistring="15,0,8.6544,.5,8.35,49.28,0,0,9.86,3,0,0,6.57,2"
TCP PRINT #2, asciistring$
IF ERR THEN
PRINT ERR
END IF
PRINT "ok"
SLEEP 1000
WEND


TCP CLOSE #2
END FUNCTION

[Mike Doty]

13,10 is the $CRLF at the end of each packet using TCP PRINT.
Is there any chance that hyperterminal is sending something
back to your transmit program that needs to be processed?

[Vince Delbrugge]

It probably is. I can check with the tech support for the software.

What type of information would it send? Is it related to the buffer capacity?

[Mike Doty]

Check if hyperterminal has an option to specify what terminates each record since you are sending $CRLF using TCP PRINT.
You might try TCP PRINT ..... followed by a semicolon or use
TCP SEND. Hyperterminal might even be set to just look for $CR at the end of each record.

[Vince Delbrugge]

I tried the options for TCP SEND and TCP PRINT with no change.

I worked with the tech support for Hyperterminal. I transmitted the data over the internet to their Hyperterminal software. Although their system did not freeze, it did indicate a problem- I attached part of their description below:

"It’s been over 20 minutes now and HyperTerminal isn’t freezing.
I did notice a series of errors coming in starting at around 12 minutes onward “TCP ZeroWindow”, which I believe is being generated by the script since it is not receiving the data from HyperTerminal. I don’t think this would cause HyperTerminal to freeze though (it isn’t on my computer). I saw the same error in my sped up version after 2 minutes
You might try modifying the script so it at least pretends to read the data from HyperTerminal (even if it just dumps it)."

I want to use PB/CC to write the receiving software to avoid the Hyperterminal; I have to do this for my field application anyway. Hyperterminal usually works for initial testing; maybe that is not the case here.

I get error 421 with the following code below for my receiving PC. The error refers to the ip address not being a string. I guess it can't interpret it as a number because of the excess decimal points, but I also get error 420 if I declare it as a string and use quotation marks. Does anyone have any ideas as to how to address this problem and write the code for the receiving PC?

I appreciate the help.



#COMPILE EXE
#DIM ALL
FUNCTION PBMAIN () AS LONG
LOCAL buffer$
PRINT "start"
TCP OPEN SERVER ADDR 192.168.1.102 PORT 23 AS #2

WHILE NOT INSTAT
TCP RECV #2, 4096, buffer$
IF ERR THEN
PRINT ERR
END IF
PRINT buffer$
SLEEP 1000
WEND

TCP CLOSE #2
END FUNCTION

[Mike Doty]

\pbcc50\samples\internet\tcp\echoserv.bas should give you a good example.

[Neil Croft]

Try using a different terminal client and see if the problem stays. Also try and work out the exact amount of data that's transferred.

Hyperterminal's pretty poor.

Have you got capture text enabled to somewhere it can't write?

[Vince Delbrugge]

I started using the Echoserve and Echoclient routines with some modifcations yesterday. They are more useful to me now after stumbling around with my own code written from scratch. I transmitted with my own PB/CC code and received it on the Echoserve routine for a couple of hours yesterday without any freezes. The next step is to make sure the quality of my data records is good throughout the transmission.


My manual has some comments comparing UDP and TCP. In general, it seems like TCP may partition a data record while UDP will not do that. For my application, I would rather miss a few complete records using UDP than receive a portion of a record with TCP.

Am I interpreting the comparison between TCP and UDP correctly?

Thank you again for the comments and help.

[Neil Croft]

TCP should retransmit lost packets which will then be assembled in the IP stack. UDP just sends 'em and be damned. In theory therefore, TCP ensures you get all your packets on a slow/noisy/bad network whereas UDP doesn't.

This is fine as long as you allow for the occasions when even TCP fails to get through.

I once had a client network's firewall lock up because someone had set it's logging to TCP and the log server wasn't listening on TCP. Once the firewall ran out of room to hold the logs itself (as it wasn't offloading them anywhere), it locked up. Fail closed may be more secure but it took a lot of finding to fix. With UDP, it sends the log packets out and doesn't care/know if they arrive at all.

[Mike Doty]

Here is a PBCC server that allows large buffer sizes and will
also terminate faster by passing a known terminator.

 
'==============================================================================
'
'  TCP Echo Server example for the PowerBASIC Console Compiler
'  Copyright (c) 1999-20086 PowerBASIC, Inc.
'  All Rights Reserved.
'
'==============================================================================
#COMPILER PBCC 5
#COMPILE CON
#DIM ALL
%PortNumber = 9999
%BufferSize = 100000
$Terminator = "<END OF FILE HERE>"   'place on end of data sent for faster terminating
GLOBAL gEmptyBuffer AS LONG  'just for testing
'------------------------------------------------------------------------------
' Include files and settings
'
%CCWIN = 1                ' Include GUI API calls
%USEMACROS = 1
#INCLUDE "WIN32API.INC"
#INCLUDE "WS2_32.INC"
%TCP_ACCEPT = %WM_USER + 4093  ' Any value larger than %WM_USER + 500
%TCP_ECHO   = %WM_USER + 4094  ' Any value larger than %WM_USER + 500
'------------------------------------------------------------------------------
' Global variables
'
GLOBAL hEcho    AS LONG
GLOBAL hServer  AS LONG
GLOBAL hwndTCP  AS LONG
GLOBAL hThread  AS LONG
'------------------------------------------------------------------------------
' Event logging routine
'
SUB LogEvent (BYVAL Buffer AS STRING)
    Buffer = TIME$ & " - " & Buffer
    PRINT Buffer
END SUB
 
'------------------------------------------------------------------------------
' Callback function to handle events for the GUI window
'
FUNCTION TcpProc (BYVAL hWnd AS LONG, BYVAL wMsg AS LONG, _
                  BYVAL wParam AS LONG, BYVAL lParam AS LONG) EXPORT AS LONG
    STATIC hServer AS LONG
    STATIC hEcho   AS LONG
    LOCAL  sBuffer AS STRING
    LOCAL  sPacket AS STRING
    LOCAL  BufferCount AS LONG
    SELECT CASE wMsg
    CASE %WM_CREATE
        hServer = FREEFILE
        TCP OPEN SERVER PORT %PortNumber AS hServer TIMEOUT 5000
        IF ERR THEN
            sBuffer = "Couldn't create socket!"
        ELSE
            TCP NOTIFY hServer, ACCEPT TO hWnd AS %TCP_ACCEPT
            sBuffer = "Connected to Port" + STR$(%PortNumber)
        END IF
        LogEvent sBuffer
        hEcho = %INVALID_SOCKET
        FUNCTION = 1
    CASE %TCP_ACCEPT
        SELECT CASE LO(WORD, lParam)
        CASE %FD_ACCEPT
            hEcho = FREEFILE
            TCP ACCEPT hServer AS hEcho
            TCP NOTIFY hEcho, RECV CLOSE TO hWnd AS %TCP_ECHO
        END SELECT
        FUNCTION = 1
    CASE %TCP_ECHO
        SELECT CASE LO(WORD, lParam)
        CASE %FD_READ
            IF hEcho <> %INVALID_SOCKET THEN
                ' Perform a receive-loop until there is no data left (ie, the end of stream)
           '---------------------------------------------------------------------------
                #IF 0
                sBuffer = ""
                sPacket = ""
                DO
                  TCP RECV hEcho, 1024, sBuffer
                  sPacket = sPacket & sBuffer
                LOOP UNTIL sBuffer = "" OR ISTRUE EOF(hEcho) OR ISTRUE ERR
                #ENDIF
           '---------------------------------------------------------------------------
                'BufferCount   = 0 'not needed, just for testing
                gEmptyBuffer = 0  'not needed, just for testing
                DO
                  INCR BufferCount       'for tsting only
                  tcpSafeReceive hEcho, %BufferSize, sBuffer
                  REM tcpSafeReceive hEcho, %BufferSize, sPacket
                  sPacket = sPacket + sBuffer
                  IF LEN(sPacket) THEN
                      ? FORMAT$(LEN(sPacket),"#,")
                  END IF
                LOOP UNTIL sBuffer = "" OR ISTRUE EOF(hEcho) OR ISTRUE ERR
                ' Optional send back response
                IF LEN(sPacket) THEN
                   TCP SEND hEcho, "Server received " + FORMAT$(LEN(sPacket),"#,") + " bytes received, ok."
                   LogEvent FORMAT$(LEN(sPacket),"#,") + " bytes.   Buffers read" + STR$(BufferCount)  + " gEmptyBuffer " + FORMAT$(gEmptyBuffer,"#,")
 
                END IF
            ELSE
                LogEvent "* FD_READ Error!"
            END IF
        CASE %FD_CLOSE
            TCP CLOSE hEcho
            hEcho = %INVALID_SOCKET
            ? "Closed hEcho socket"
            ? STRING$(60,"-")
        END SELECT
        FUNCTION = 1
    CASE %WM_DESTROY
        TCP CLOSE hServer
    END SELECT
    FUNCTION = DefWindowProc(hWnd, wMsg, wParam, lParam)
END FUNCTION
 
'------------------------------------------------------------------------------
' Create the GUI window to receive TCP event notification messages
'
FUNCTION MakeWindow () AS LONG
    LOCAL wce         AS WndClassEx
    LOCAL szClassName AS ASCIIZ * 64
    LOCAL hWnd        AS LONG
    LOCAL hInst       AS LONG
    STATIC registered AS LONG
    hInst = GetModuleHandle(BYVAL %NULL)
    IF ISFALSE registered THEN
        szClassName          = "PBTCPCOMM"
        wce.cbSize           = SIZEOF(wce)
        wce.style            = %CS_HREDRAW OR %CS_VREDRAW
        wce.lpfnWndProc      = CODEPTR(TcpProc)
        wce.cbClsExtra       = 0
        wce.cbWndExtra       = 0
        wce.hInstance        = hInst
        wce.hIcon            = %NULL
        wce.hCursor          = %NULL
        wce.hbrBackground    = %NULL
        wce.lpszMenuName     = %NULL
        wce.lpszClassName    = VARPTR(szClassName)
        wce.hIconSm          = %NULL
        RegisterClassEx wce
        registered = %TRUE
    END IF
    hWnd = CreateWindow(szClassName, _
                        "TCP Handler", _
                        %WS_OVERLAPPEDWINDOW, _
                        5, 5, 10, 10, _
                        %NULL, _
                        %NULL, _
                        hInst, _
                        BYVAL %NULL)
    IF ISFALSE hWnd THEN
        hWnd = GetLastError
    ELSE
        ShowWindow hWnd, %SW_HIDE
        UpdateWindow hWnd
    END IF
    FUNCTION = hWnd
END FUNCTION
 
'------------------------------------------------------------------------------
' Spawn a thread to create/own the GUI window and run a thread message pump.
' Each thread must operate it's own pump the GUI windows it owns.
'
FUNCTION WindowThread (BYVAL nIgnored AS LONG) AS LONG
    LOCAL Msg AS tagMsg
    hwndTCP = MakeWindow
    DO WHILE IsWindow(hwndTCP) AND GetMessage(Msg, %NULL, 0, 0)
        TranslateMessage Msg
        DispatchMessage Msg
    LOOP
END FUNCTION
 
'------------------------------------------------------------------------------
' Main program entry point...
'
FUNCTION PBMAIN () AS LONG
    ' Process socket messages in a separate thread
    THREAD CREATE WindowThread(%NULL) TO hThread
    ' If the user presses the Esc key, exit the server
    PRINT "Press ESC to end the ECHO Server with buffer size " + FORMAT$(%BufferSize,"#,")
    DO UNTIL WAITKEY$ = $ESC
    LOOP
    SendMessage hwndTCP, %WM_CLOSE, 0, 0
END FUNCTION
  '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
'  tcpSafeReceive
'  Specify a length of data to wait for and this function
'  retrieves data until all data has filled the buffer or
'  an error occurs.
'  Returns: %True on success, %False on error
'  by Don Dickinson
'  Modified 9/9/09 to also exit if buffer is empty
'~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
FUNCTION tcpSafeReceive(BYVAL hSocket AS LONG, BYVAL iBufferLen AS LONG, _
                        recBuff AS STRING) AS LONG
   DIM iLeft AS LONG
   DIM sBuffer AS STRING
   recBuff = ""
   iLeft = iBufferLen
   DO
      sBuffer = SPACE$(iLeft)
      TCP RECV hSocket, iLeft, sBuffer
      IF ERR THEN
         FUNCTION = %False
         EXIT FUNCTION
      ELSEIF LEN(sBuffer) = 0 THEN
        INCR gEmptyBuffer  'for testing only
        FUNCTION = %False
        EXIT FUNCTION
      END IF
      recBuff = recBuff + LEFT$(sBuffer,iLeft)
 
      IF LEN(recBuff) >= iBufferLen THEN
        REM ? "Normal exit"
         EXIT DO
      ELSEIF RIGHT$(recBuff,LEN($Terminator)) = $Terminator THEN
        RecBuff = LEFT$(recBuff,LEN(recBuff)- LEN($Terminator))   'fast terminator
        ? "Terminator found"
        EXIT DO
      END IF
 
      iLeft = iBufferLen - LEN(recBuff)
      SLEEP 5
   LOOP
   FUNCTION = %True
END FUNCTION

Here is a client to test it.

#COMPILE EXE
#DIM ALL
%Bytes = 25000
%LoopsWanted = 2
$Terminator = "<END OF FILE HERE>
%PortNumber = 9999
$Site = "192.168.10.103"
%TimeOut = 20000
FUNCTION PBMAIN () AS LONG
  COLOR 7,1,1
  CLS
  LOCAL Counter AS LONG
  LOCAL sBuffer AS STRING
  LOCAL sPacket AS STRING
  LOCAL Socket  AS LONG
  FOR counter = 1 TO %LoopsWanted
    Socket = FREEFILE
    TCP OPEN PORT %PortNumber AT $Site AS #Socket TIMEOUT %Timeout
    IF ERR THEN ? "Error opening port" + STR$(%PortNumber) + " error" + STR$(ERRCLEAR):GOTO EndHere
    TCP SEND #Socket, STRING$(%Bytes,"D") + $Terminator
    IF ERR THEN ? "TCP SEND error" + STR$(ERRCLEAR):GOTO EndHere
    ? "Sent, wait for response"
    sBuffer = ""
    sPacket = ""
    DO
      TCP RECV #Socket, 1024, sbuffer
      sPacket = sPacket + sbuffer
    LOOP UNTIL sBuffer = "" OR ISTRUE EOF(#Socket) OR ISTRUE ERR
    IF ERR THEN
      ? "TCP RECV error" + STR$(ERRCLEAR)
    ELSE
      ? sPacket
    END IF
    TCP CLOSE #Socket
    SLEEP 1000
   NEXT
EndHere:
   SLEEP 5000
END FUNCTION

[Vince Delbrugge]

Gentlemen,

I will try the server and client routines I was sent.

I attached the client routine that I modified using the Echo client from PB/CC. I also attached two graphics images of the output.

Note that after about 10 minutes or so, the data format that the server printed in the concole changed. The change reflected that the data records were being merged. The time for this change is on the order of the time that the Hyperterminal locked-up.

Both of the server routines, the Echoserve and the one that Mike sent, have the advanced programming that the PB/CC manual describes. I think that I am swimming in the deep water now.

[Mike Doty]

On server comment out the line to send back to client.
Slower: You can also open/close the port with each send to let server know it is the end of a record.

#COMPILE EXE
#DIM ALL
$Terminator = "<END OF FILE HERE>
$Site = "192.168.1.102"
%PortNumber = 23
%Timeout = 10000
FUNCTION PBMAIN () AS LONG
 
  LOCAL nSocket&,asciistring$
  CLS
  MOUSE ON
  nSocket = FREEFILE
  TCP OPEN PORT %PortNumber AT $Site AS #nSocket TIMEOUT %Timeout
  IF ERR THEN ? "Error opening port" + STR$(%PortNumber) + " error" + STR$(ERRCLEAR):SLEEP 2000:EXIT FUNCTION
 
  WHILE NOT INSTAT
    asciistring = "15,0,8.6544,.5,8.35,49.28,0,0,9.86,3,0,0,6.57,2"  '47 bytes
    SendIt nSocket,asciistring
    IF ERR THEN ? "TCP SEND error"  + STR$(ERRCLEAR) ELSE  ? "ok"
  WEND
  TCP CLOSE #nSocket
  ? "Done ..."
  SLEEP 2000
 
END FUNCTION
SUB SendIt(MySocket AS LONG, s AS STRING)
   TCP SEND #MySocket, s + $Terminator
   SLEEP 210  'minimum 210 milliseconds between sends so packets don't get combined
END SUB

[Vince Delbrugge]

Here is the source code for both the server and client that worked for over an hour with no flaws.

Mike, in this server routine, will taking out the SEND command have an efect? I think that it is out of the loop in this routine.

Thank you for the help. Now I have to integrate the code to input the data using RS-232 communication.

Can a client link to multiple servers simultaneously?

[Mike Doty]

The TCP SEND from the server to the client isn't required.
Is that what you are referring to?
There should be some sort ot checksum on the server that the data is correct.

[Mike Doty]

If interested in testing over the internet, please let me know.
Added 8-byte header to do a checksum on the server.
TCP RECV on server modified to allow large buffer sizes.

I would test each send with the server.

#COMPILE EXE
#DIM ALL
$Site = "192.168.10.103"  'local
'$Site = "255.255.255.255"  'internet
%PortNumber = 31187
%Timeout = 30000
TYPE TcpSendType       'TCP SEND starts with this data
  BytesSent AS DWORD   'length of buffer to send
  Checksum  AS DWORD   'call asmchecksum(s$) to get this
END TYPE
FUNCTION PBMAIN () AS LONG
  LOCAL nSocket&, sBuffer$,sPacket$,counter&,AsciiString$
  LOCAL Header AS TcpSendType
  CLS
  MOUSE ON
  nSocket = FREEFILE
  TCP OPEN PORT %PortNumber AT $Site AS #nSocket TIMEOUT %Timeout
  IF ERR THEN ? "Error opening port" + STR$(%PortNumber) + " error" + STR$(ERRCLEAR):SLEEP 2000:EXIT FUNCTION
  WHILE NOT INSTAT
    INCR counter
    'asciistring = "15,0,8.6544,.5,8.35,49.28,0,0,9.86,3,0,0,6.57,2"  '8 byte header + 47 bytes = 55
    asciistring  =  STRING$(RND(1,9999),"x")
    
    Header.BytesSent = LEN(AsciiString) + SIZEOF(TcpSendType)
    Header.CheckSum  = AsmCheckSum (asciistring)
    sPacket = Header + AsciiString
    ? "Sending "  + FORMAT$(Header.BytesSent,"#,"); " bytes (checksum" + STR$(Header.CheckSum);") ";
    TCP SEND nSocket, sPacket
    
    IF ERR = 0 THEN     'successful send, wait for response
      sPacket = ""
      sBuffer = ""
      DO
       TCP RECV nSocket, 1024, sBuffer
          sPacket = sPacket + sBuffer
          SLEEP 5
       LOOP UNTIL sBuffer = "" OR ISTRUE EOF(nSocket) OR ISTRUE ERR
       IF sPacket = "Ok" THEN   'server sends back the number of bytes received
         ? sPacket
       ELSE
         ? LEFT$(sPacket,1000) 'in case some error sending  back
         BEEP
         SLEEP 5000
         EXIT DO
       END IF
    ELSE
       ? "TCP send error" + STR$(ERRCLEAR)
       BEEP
       SLEEP 5000
       EXIT DO
    END IF
   WEND
  ? "Done ..."
  TCP CLOSE #nSocket
  SLEEP 5000
END FUNCTION
FUNCTION AsmCheckSum(strBuffer AS STRING) AS DWORD
  ' [URL]http://www.powerbasic.com/support/pbforums/showthread.php?t=23094[/URL]
  'Scott Slater, modified by Mark Smit for zero length strings
    ASM xor     ebx,ebx         ; zero ebx
    ASM mov     eax,strBuffer   ; eax = pointer to string handle
    ASM mov     eax,[eax]       ; eax = pointer to string data
    ASM and     eax,eax         ; check eax for null pointer
    ASM jz      done
    ASM mov     ecx,[eax-4]     ; move length of string into ecx
mainloop:
    ASM xor     edx,edx
    ASM mov     dl,[eax]       ; get current character into dl
    ASM add     ebx,edx         ; Add current value to total
    ASM inc     eax             ; get next character
    ASM dec     ecx             ; decrease ecx (length) counter
    ASM jnz     mainloop        ; iterate if not zero
done:
    ASM mov     function,ebx    ; return checksum value to caller
END FUNCTION

[Mike Stefanik]

Just to point out, you don't need to checksum data written over stream sockets. TCP already does that; it uses a sliding window protocol that ensures that the data arrives intact, with the bytes in the same order that they were written. The only thing that it does not guarantee is that you'll be able to read the data in the same amount that it was written. In other words, if the sender writes 1024 bytes to the socket, there's no guarantee that you'll get all of the data in a single read operation; you may need to perform multiple read operations to get all of the data.

[Mike Doty]

Thank you.
I can see dropping the checksum.

The receive routine being used is many times faster than the
tcp recv loop with only 1024 bytes as in the echo server example.
It adjusts the buffer size to the bytes left after the first tcp recv.
The echo server example receives early EOF's in my tests with large buffers.

I would consider all data sensitive over the internet.
Would you send the record length?
Would you compress/uncompress?
Would you encrypt?

[Mike Stefanik]

The two most common methods for sending structured data over stream sockets is either prefix it with the length, or use a special byte sequence that indicates the end of a record; you could actually do both, if you wanted some redundancy.

As far as compression and encryption, that really depends on the payload. If it contains sensitive information, naturally you'd want to encrypt it. Compression really depends on the amount of data being exchanged; with smaller amounts, you'll spend more time compressing and decompressing the data with little payoff in terms of bandwitdth reduction. The larger those data records are, the more value in implementing some kind of compression scheme. If the data size is variable, then you could always implement some kind of threshold where you only compress data that's (for example) more than 20K or something along those lines.

So, really, the answer to all of your questions is "it depends".