Announcement

Collapse
No announcement yet.

Threads, performance, response times

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Roger Rines
    replied
    This exercise seemed like a good way to learn more about thread performance. From the code provided by Paul & Chris, I created three different code modules, but only created 1 test file with 10,000,000 records so the testing would push the processing time in each section further and each module would be using the same data:
    Code:
     Record_Count = 10,000,000
     
    ------------------ Paul's Code #1 - PBCC 4.04
        No Thread = 17.266
      With Thread = 10.406
             Diff =  6.86
         Increase = 39.73%
    ------------------ Chris' Code #1 - PBWin 8.04
        No Thread = 18.594
      With Thread = 10.688
             Diff =  7.906
         Increase = 42.52%
    ------------------ Paul's Code #2 - PBCC 4.04
        No Thread = 18.696
      With Thread =  9.064
             Diff =  9.632
         Increase = 51.52%
    ------------------ END
     
    ------------------ Computer Hardware & OS - CPU = XEON x 2 :
    OS Name	Microsoft Windows XP Professional
    Version	5.1.2600 Service Pack 2 Build 2600
    OS Manufacturer	Microsoft Corporation
    System Name	R9
    System Manufacturer	ASUSTeK COMPUTER INC.
    System Model	NCCH-DL
    System Type	X86-based PC
    Processor	x86 Family 15 Model 3 Stepping 4 GenuineIntel ~3200 Mhz
    Processor	x86 Family 15 Model 3 Stepping 4 GenuineIntel ~3200 Mhz
    Processor	x86 Family 15 Model 3 Stepping 4 GenuineIntel ~3200 Mhz
    Processor	x86 Family 15 Model 3 Stepping 4 GenuineIntel ~3200 Mhz
    BIOS Version/Date	Phoenix Technologies, LTD ASUS NCCH-DL ACPI BIOS Revision 1003, 10/1/2004
    SMBIOS Version	2.3
    Windows Directory	C:\WINDOWS
    System Directory	C:\WINDOWS\system32
    Boot Device	\Device\HarddiskVolume2
    Locale	United States
    Hardware Abstraction Layer	Version = "5.1.2600.2180 (xpsp_sp2_rtm.040803-2158)"
    User Name	R9\Roger
    Time Zone	Pacific Daylight Time
    Total Physical Memory	2,048.00 MB
    Available Physical Memory	1.33 GB
    Total Virtual Memory	2.00 GB
    Available Virtual Memory	1.96 GB
    Page File Space	3.85 GB
    Page File	C:\pagefile.sys
    It seems that some threading can provide a lot of performance if the machine executing the threads has the capability to handle the work. In the test cases given in this posting and generated on my machine, it sure seems likes threads can save a lot of time because of their ability to better utilize the hardware. This performance is improved even more when the adjustments Paul made in his second code posting are employed.

    Leave a comment:


  • David Roberts
    replied
    Got it.

    With the 2 error very early on in the FOR/NEXT LOOP there is nothing left of the file to read so strNull becomes empty. ClearFileCache gets an empty buffer and so has an early bath. In fact there is no need for ClearFileCache to use the same buffer - my logic is adrift there.

    Funny thing is that on introducing error checking on GET I don't get one - should have got an end of file error. Must look into that now. Blast it!

    With ClearFileCache2 it uses its own buffer and has no regard for how PBMain collects data so doesn't 'fall over'.

    Actually, I think I'll remove ClearFileCache from the Source Code forum - the idea is OK but it was poorly implemented.

    Leave a comment:


  • David Roberts
    replied
    Breathes sigh of relief.

    Still cannot figure out why ClearFilecache came beck so quickly on the second outing with the 2 error. I introduced the same error to ClearFileCache2 and the cache clearing took just as long, that is it behaved as expected. I'll leave it for now and look again - might spot something then. The reason may not be worth knowing but I won't know that until I find it.

    Leave a comment:


  • Gösta H. Lovgren-2
    replied
    Readings after correcting the code (removing the 2):

    Code:
     Writing took =  4.38
     
     Clearing First Cache =  2.06
     First Read =  2.52
     Clearing Second Cache =  1.95
     Second Read =  2.53
     Clearing Third Cache =  1.95
    All is copesetic again PB Land.

    ================================================================
    "Every time I see an adult on a bicycle,
    I no longer despair
    for the future of the human race."
    H.G. Wells
    ================================================================

    Leave a comment:


  • David Roberts
    replied
    Sorry, Gösta.

    > Your relative observations may not change, just the absolute timings.

    Isn't true. I couldn't figure out why you were getting such small values for second and subsequent filecache clearings. They should be all the same, or thereabouts, since ClearFileCache is reading the file.

    When I removed the offending 2 your timings made more sense.

    I thought only the GET timings would be affected.

    I'll check out that error and see what is going on 'under the bonnet'.

    Leave a comment:


  • David Roberts
    replied
    I've added a new version of ClearFileCache in the Source Code forum.

    Leave a comment:


  • David Roberts
    replied
    Gösta, I found a typo in my code. A 2 crept in on reading the files. The first was a typo and then the error was duplicated on a 'Copy and Paste'. The 10248576 should read 1048576.

    The moral here, of course, and I should know better after goodness knows how long is not to use use numbers in code but use a declaration of some sort. Of course when testing something it is very easy to slip into bad practice. We shouldn't, but I guess that I may not be the only one. Your relative observations may not change, just the absolute timings. I'll have a look at what you are doing.

    Leave a comment:


  • Gösta H. Lovgren-2
    replied
    Does the Cache have a Cache?

    Been playing around this morning with the code. I put a Kill in before the first write and added a couple more timers. Here are the results:

    Writing took = 3.95

    Clearing First Cache = 1.98
    First Read = 9.50
    Clearing Second Cache = 0.02
    Second Read = 10.08
    Clearing Third Cache = 0.02

    Clearing the cache the first time takes pretty consistently 100 times longer than subsequent Clear Caches.

    Here's a run without the Kill:
    Writing took = 0.00

    Clearing First Cache = 1.97
    First Read = 9.50
    Clearing Second Cache = 0.00
    Second Read = 10.25
    Clearing Third Cache = 0.00
    Is there a Catch to the Cache?

    Okay, as I'm writing this it occurs to me the compiler is setting up internal values (or clears memory to work) when it first runs ClearCache and that's what takes the time and it isn't necessary subsequently. Does that make sense?

    Okay, it doesn't. Here's another run with the first ClearCache remmed:
    Writing took = 0.00

    Clearing First Cache = 0.00
    First Read = 10.17
    Clearing Second Cache = 0.02
    Second Read = 9.81
    Clearing Third Cache = 0.02
    Really curious. Anyway, if any want to try it themselves here's my David's code with my junk added:


    '
    Code:
    'http://www.powerbasic.com/support/pbforums/showthread.php?t=35260
    #Dim      All
    #Compile  Exe 
    #Register None
    #Tools    Off
    %NOMMIDS = 1
    %NOGDI = 1
    #Include "WIN32API.inc" ' 1 November 2006
    '#Include "C:\Power Basic\Includes\Clear_File_Cache.inc"
    '#Include "C:\Power Basic\Includes\clipboard.inc"
    $sFile = "c:\test"
    '         
    'Clipboard stuff gotten from Poffs but I forget from whom
    Function Clipboard_Set_Text Alias "Clipboard_Set_Text" _
                  (ByVal sText As String) Export As Long
      Local hData As Long, hGlob As Long
    ' ** Create a global memory object and copy the data into it
      hData = GlobalAlloc(%GMEM_MOVEABLE Or %GMEM_DDESHARE, Len(sText) + 1)
      hGlob = GlobalLock(hData)
      Poke$ hGlob, sText + Chr$(0)
      GlobalUnlock hData
    ' ** Open the clipboard
      If IsFalse(OpenClipboard(%Null)) Then
        GlobalFree hData
        Exit Function
      End If
    ' ** Paste the data into the clipboard
      EmptyClipboard  'WinAPI
      Function = SetClipboardData(%CF_TEXT, hData) 'WinAPI
      CloseClipboard  'WinAPI
    End Function
    Function Clipboard_Get_Text Alias "Clipboard_Get_Text"() Export As String
        Local zPtr As Asciiz Ptr
        OpenClipboard 0 'WinAPI
        zPtr = GetClipboardData(%CF_TEXT)
        If zPtr <> 0 Then Function = @zPtr
        CloseClipboard   'WinAPI
    End Function
    '                   
    '         
    ' '             
    Sub ClearFileCache(sFile As String , Buffer As String)
       Local hFile, Chunk, nbLoaded As Long, ptrBuffer As Dword Ptr
       Dim zFile As Asciiz * 256
     
       zFile = sFile
       hFile = CreateFile(zFile, %GENERIC_READ, 0, ByVal 0, %OPEN_EXISTING, %FILE_FLAG_SEQUENTIAL_SCAN Or %FILE_FLAG_NO_BUFFERING, ByVal 0)
       Chunk = Len(Buffer)
       ptrBuffer = StrPtr(Buffer)
       Do
         ReadFile( hFile, ByVal ptrBuffer, ByVal Chunk, nbLoaded, ByVal 0 )
       Loop Until nbLoaded = 0
       CloseHandle(hFile)
    End Sub
     '                   
    Function PBMain( ) As Long
      Dim strNull As String
      Local i As Long, t1, t2 As Ext
      Local Clr_Cache01, Clr_Cache02, Clr_Cache03, Write_01 As Ext
      strNull = Nul$(1048576) ' 1 MB
     
    '  Kill $sfile 'just to test
      Write_01 = Timer
      If Dir$($SFile) = "" Then
        ' Write a 100MB file
        Open $sFile For Binary As #1
        For i = 1 To 100
          Put$ #1, strNull
        Next
        Close #1                   
        ?"Wrote " & $sfile
      End If          
      Write_01 = Timer - Write_01
     
      ' Clear filecache after creation, if done, 
      ' And before before first Read
     
    ?"Clearing original Cache"
     Clr_Cache01 = Timer
      ClearFileCache($sFile, strNull)  
    '? Using$("Clearing any existing cache took #.## seconds", Timer - t1)  
     Clr_Cache01 = Timer - Clr_Cache01
     
      ' Read file
      t1 = Timer
      t1 = Timer
      Open $sFile For Binary As #1
      For i = 1 To 100
        Get$ #1, 10248576, strNull
      Next
      Close #1
      t1 = Timer - t1
     ?"Clearing Read cache"
     
      ' Clear file cache after first reading and before second read
     Clr_Cache02 = Timer
      ClearFileCache($sFile, strNull)
      Clr_Cache02 = Timer - Clr_Cache02
    '? Using$("Clearing Read Cache took #.## seconds", Timer - t3)  
     
      ' Read file again
      t2 = Timer
      Open $sFile For Binary As #1
      For i = 1 To 100
        Get$ #1, 10248576, strNull
      Next
      Close #1
      t2 = Timer - t2
     Clr_Cache03 = Timer
      ClearFileCache($sFile, strNull)
      Clr_Cache03 = Timer - Clr_Cache03
     
      Local m$
     
        M$ = Using$(" Writing took = ##.##", Write_01) + $CrLf + $CrLf  + _
             Using$(" Clearing First Cache = ##.##", Clr_Cache01) & $CrLf & _
             Using$(" First Read = ##.##",t1) +  $CrLf  + _
             Using$(" Clearing Second Cache = ##.##", Clr_Cache02) & $CrLf & _
             Using$(" Second Read = ##.##",t2) & $CrLf & _ 
             Using$(" Clearing Third Cache = ##.##", Clr_Cache03)
        ? m$
        Clipboard_Set_Text(m$)     
     End Function
     '

    Leave a comment:


  • David Roberts
    replied
    Blimey. Thanks Gösta.

    Leave a comment:


  • Gösta H. Lovgren-2
    replied
    Dave, your comment sparked a memory so I looked Paul Squires JF forum and found this:

    http://planetsquires.com/support/ind...b&topic=1836.0

    It's a quirk in JFP. Can't have a comment on the same line as Compile:

    Compile Exe ' 8.04 '<== won't run after compile


    Compile Exe '<== will run after compile


    ================================================================

    I do not feel obliged to believe
    the same God who has endowed us
    with sense, reason, and intellect
    intended us to forgo their use.
    ~ Galileo Galilei
    ================================================================

    Leave a comment:


  • David Roberts
    replied
    I don't believe this - I'm sure I've done it before - I must check.

    A comment following #COMPILE.EXE is getting treated as a compile only in JellyFish when Build and Execute is chosen. I added the comment just before posting.

    Leave a comment:


  • Gösta H. Lovgren-2
    replied
    Originally posted by David Roberts View Post
    >
    That is exactly what my GetFileSize does but if the system handle is available why use a sledgehammer when three lines of PB does the job?
    Because, .... errrr.... ahhh .... hrrrrummmmph..... REAL men use the API.

    ================================================================
    "Some editors are failed writers,
    but so are most writers."
    T. S. Eliot (1888-1965)
    ================================================================

    Leave a comment:


  • David Roberts
    replied
    > Real Men Don't Need No Stinkin' Handle...Write once, use many....The Power of MACROs.....<insert platitude of choice here>...

    That is exactly what my GetFileSize does but if the system handle is available why use a sledgehammer when three lines of PB does the job?

    Leave a comment:


  • David Roberts
    replied
    > Must be something in your code that causes JellyFish to not run the code after compiling.

    I have seen this with JellyFish on a few occasions leaving me to run the exe directly. It used to happen on my old Win98 machine as well as my XP Pro machine. If you load the bas into the PB IDE then it should work with 'Run>Compile and Execute'. I can only think that on editing a non-printable character gets embedded that throws JellyFish but not the PB IDE.

    Sometimes, I have rewritten sections of code and deleted the section copied to find that all is well again, removing the offending character without knowing what it was.

    It happens so rarely that I've just accepted it especially since, to my knowledge, nobody had reported this behaviour.

    > At any rate neat code. (Contest material?)

    No, its just a trick and I cannot even prove why it works.

    Leave a comment:


  • Gösta H. Lovgren-2
    replied
    Originally posted by David Roberts View Post
    In the meantime you guys may like to have a look at this to avoid restarting your machines or writing massive files, Gösta.
    Am having a peculiar problem with your code, David. I C&P'ed, and while it compiles fine, it doesn't display anything on the screen, it doesn't appear to run, just compiles. I tried sprinkling the code with MsgBoxes but none display.

    {just occurred to me} Just chkd the folder and the .exe is there, and runs as advertised. Must be something in your code that causes JellyFish to not run the code after compiling. Have another prog loaded into JF at the same time which compiles and runs as normal. Strange.

    I've been playing around with the code a little and found that clearing the cache the first time takes about 2 seconds but clearing the cache the second time (after reading the file) is effectively instantaneous (.02 if at all).

    At any rate neat code. (Contest material?)

    Leave a comment:


  • Michael Mattias
    replied
    Real Men Don't Need No Stinkin' Handle...Write once, use many....The Power of MACROs.....<insert platitude of choice here>...

    Code:
    MACRO EQ32(a,b) = (BITS???(a)=BITS???(b))
    
    MACRO FUNCTION ANY_FILESIZE (szFile) 
    
       MACROTEMP 32, hSearch , fsize 
    
       DIM w32 AS WIN32_Find_Data, hSearch AS LONG, fsize AS LONG
       hSearch       = FindFirstFile (szFile, W32)
       IF eq32(hsearch, %INVALID_HANDLE_VALUE) THEN
          fsize = -1&   ' "not found" 
       ELSE
          FindClose hSearch
          fSize =  w32.nFilesizeLow
       END IF
    END MACRO = fSize
    
    ...
    
       fsize = ANY_FILESIZE (szFile)
    (Fails if file open and size is being changed since directory is not updated until file is CLOSEd)
    (Also, not written for filesize > 2 Gb)

    Leave a comment:


  • David Roberts
    replied
    Paul Purvis asked me a little while back why I used a function called GetFileSize instaed of LOF and I told him that the handle returned by CREATEFILE didn't work.

    Now I realise that I could have simply written

    Open Handle hSys As #1
    Filelength = Lof(#1)
    Close #1

    Live and learn.

    Leave a comment:


  • Michael Mattias
    replied
    While OPEN HANDLE can be used to effect an OPEN with 'non-default' options otherwise not (yet?) controllable thru PB syntax, it's more common use would be passing an open file handle to a function which resides in another code module.

    Since PB internal handles (e g "#12" or whatever is returned by FREEFILE) are only valid in the code module in which that handle was obtained, you can't pass it to such a function and expect it to be valid. Instead you do something like...

    Code:
    ' MAIN.BAS
    #COMPILE EXE 
    
    DECLARE FUNCTION SupportFunction LIB "MyDLL.DLL" (BYVAL h AS LONG) AS LONG
    
    ...
      hFile = FREEFILE
      OPEN   "byfile"  FOR OUTPUT as hFile
      hSys  = FILEATTR(hFIle, 2)    ' get system handle
      CALL    SupportFunction ( hSys)
      PRINT #hFile, "End of Report"
      CLOSE hFile
    
    ' =================================================
    ' 
    #COMPILE DLL
    
    FUNCTION SupportFunction (BYVAL hSys AS LONG) EXPORT AS LONG
    
        hFile = FREEFILE
        OPEN   HANDLE hSys FOR OUTPUT AS hFile 
    
        PRINT #hFile, "Hello World"    ' <<< PB syntax not "Writefile"
    
        CLOSE hFile     ' does not close underlying system handle
    
        FUNCTION = %TRUE
    END FUNCTION
    MCM

    Leave a comment:


  • David Roberts
    replied
    That is an interesting aspect of RTFM. It is a long time since I read OPEN in detail. It was read on first getting PBWin but I probably glossed over any reference to the API as it meant little too me at the time. It has occurred to me of late that some aspects of the manual may be worth reading again as a very much higher percentage may be absorbed now compared to the initial read. I tend to use the manual when using an aspect of PB that I have not ventured into much, or at all.

    On the issue of "buffersize and alignment requirements" I adhere exactly to the requirements in a working app but the idea used in the link above totally ignores any such requirements. Perhaps I should look at it again to see if the results differ any by adherence. Whilst on this subject I have not been able to find any comments about possible mishaps if such requirements are not adhered to.

    Leave a comment:


  • Michael Mattias
    replied
    Just for the record....

    You don't need to use ReadFile and WriteFile to use the FILE_FLAG_NO_BUFFERING (or any other "non standard" 'open' flags)...
    Code:
     hSys = CreateFile (......  special flags....)
     hFile  = FREEFILE
     OPEN HANDLE hSys for <PB syntax mode> AS Hfile
     ' -------------------------------------
     ' GET/PUT here 
     ' --------------------------------------
     CLOSE hFile
     CloseHandle hSys
    This does not eliminate the buffersize and alignment requirements for the "var" in the PUT or GETwhen using the FILE_FLAG_NO_BUFFERING option , but some might find PUT/GET a bit easier - or at least a bit more familiar - to work with.

    MCM

    Leave a comment:

Working...
X