Announcement

Collapse
No announcement yet.

Empty out the R/W cache?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Empty out the R/W cache?

    Any way of flushing (emptying, nulling out, etc.) the read/write cache?

    I'm writing a program to test my actual network speed under various conditions and I'm getting unexpected results.

    I suspect it's because the computer(s) are reading the cache instead of the file being processed.

    Help, please?
    There are no atheists in a fox hole or the morning of a math test.
    If my flag offends you, I'll help you pack.

  • #2
    The cache SHOULD be current. (yes I have my own doubts). That is, you SHOULD be reading the cache, not the disk file.

    You can try opening the file without buffering or specify write-though on output. See demo here: Create and use a file which will be deleted on close and play with the "Flags" options in the CreateFile calls to create necessary conditions.

    Or, if you are testing by sending the same data repeatedly, that's why the cache would be used... because it IS current. (Code not shown).

    MCM
    Michael Mattias
    Tal Systems (retired)
    Port Washington WI USA
    [email protected]
    http://www.talsystems.com

    Comment


    • #3
      if you want to turn off caching at the disk level you can do it by right clicking on the drive in my computer. then going to the hardware tab, select the drive, and choose the properties button. if you're using vista, then you'll have to click the "change settings" button in order to change anything. anyway, on one of the tabs in the settings screen there is a checkbox for write caching. in vista its not a checkbox, its two radio buttons - optimizing for quick removal disables write caching.

      -don
      Don Dickinson
      www.greatwebdivide.com

      Comment


      • #4
        I've used FlushFileBuffers to clear the cache for a file.

        little example:
        Code:
           OPEN "c:\r1.dat" FOR BINARY AS #1
           hFil = FILEATTR(#1, 2)
           FlushFileBuffers hFil
        Last edited by John Gleason; 9 Dec 2008, 07:25 PM. Reason: add example

        Comment


        • #5
          My understanding is that the SDK FlushFileBuffers is the same as the PB Flush statement. The internal buffers are manually flushed to disk whilst the file is still open instead of waiting for them to be flushed automatically on closing the file. This has nothing to do with the cache.

          To stop caching we need to employ, as Mr Mattias mentioned, FILE_FLAG_NO_BUFFERING in dwFlagsAndAttributes when using CreateFile. It is not straightforward. For example, the buffers addresses should be sector aligned and for that I use VirtualAlloc as recommended in the SDK.

          Have a look at post #2 here.

          We have an analogy with the registry. The RAM portions are flushed to disk periodically. Effectively, the registry is buffered. It is not, however, cached.
          Last edited by David Roberts; 10 Dec 2008, 08:34 AM.

          Comment


          • #6
            What exactly are you trying to test?

            Maybe this is not the best method.

            I'm no network guy but I know there are some here; maybe there is some 'better' way to get whatever info you are after.

            Provide the 'what' and I'm sure you will get several 'hows.'
            Michael Mattias
            Tal Systems (retired)
            Port Washington WI USA
            [email protected]
            http://www.talsystems.com

            Comment


            • #7
              FILE_FLAG_NO_BUFFERING in dwFlagsAndAttributes when using CreateFile. It is not straightforward.
              Actually, it IS straightforward.

              What it isn't is quick and easy.
              Michael Mattias
              Tal Systems (retired)
              Port Washington WI USA
              [email protected]
              http://www.talsystems.com

              Comment


              • #8
                Well, MM (and others who are curious).

                I'm trying to get a realistic idea of how fast my home network is actually operating at under various conditions. Example: It will be slower if I am listening to streaming media.

                Currently, I am generating a really large file and writing it from one computer to another. Using TIMER results, I then calculate how long it takes to write the file, do some math and display the results.

                Then, I read the file, see how long it takes to get the entire file into memory, do some more math and display the results.

                On one of my terminals, (800mhz machine with 98SE), I get a write speed of 6mBPS and a read speed of 20mBPS (that's bits, not bytes).

                On the other machine (1.x ghz dual core running xp), I am getting 46mBPS and a read speed of 6.x gigBPS

                Out of a 10/100 switch? I don't think so.

                Well, thinking about it...I wasn't planning on posting the source code until I get all the kinks worked out but, here it is. Written in CC4.04.

                Maybe somebody can determine what I'm doing wrong.
                Code:
                REM ***********************************************************     '
                REM * Get a reasonably accurate speed measure of your network *     '
                REM ***********************************************************     '
                                                                                    '
                    #INCLUDE "win32api.inc"                                         '
                    DECLARE FUNCTION BrowseForFolder(LONG, STRING) AS STRING        '
                                                                                    '
                FUNCTION PBMAIN                                                     '
                    LOCAL x, y, z, Packets           AS LONG                        '
                    LOCAL bytes, bps                 AS QUAD                        '
                    LOCAL Start, Finish, Difference  AS DOUBLE                      '
                    LOCAL FileName                   AS STRING                      '
                '*******************************************************************'
                    FOR x = 0 TO 255                                                '<:  Build up the test
                    TestPacket$ = TestPacket$ + CHR$(x)                             ' |  packet.
                    NEXT x                                                          '<:
                                                                                    '
                    f1$ = "###,###,###"                                             '
                    f$  = "##,###.#######"                                          '
                                                                                    '
                    Packets = 50000                                                 '   Write 50K packets
                                                                                    '
                '*******************************************************************'
                    COLOR 14,1                                                      '
                    CLS                                                             '
                    FileName = browseforfolder(0, "Select destination folder...")   '   Output path/filename
                    IF FileName = "" THEN EXIT FUNCTION                             '
                    IF RIGHT$(FileName,1) <> "\" THEN FileName = FileName+"\"       '
                    FileName = FileName + "SpeedTest.txt"                           '
                                                                                    '
                '*******************************************************************'
                                                                                    '
                    OPEN FileName FOR BINARY AS #1                                  '
                    LOCATE  1, 5 : COLOR 0,7 : PRINT;" Write test " : COLOR 14,1    '
                                                                                    '
                    Start = TIMER                                                   '
                    FOR x = 1 TO Packets                                            '<: Test the write speed
                    PUT$ #1,TestPacket$                                             ' |
                    NEXT x                                                          '<:
                    CLOSE #1                                                        '
                    finish = TIMER                                                  '
                    Difference = Finish - Start                                     '   In seconds
                                                                                    '
                '*******************************************************************'   Start calculations.
                                                                                    '
                    LOCATE  3, 1 : PRINT;"   Packet Length: ";_                     '
                                   USING$(f1$,LEN(TestPacket$) * Packets)           '
                    LOCATE  4, 1 : PRINT;"      Time Start: ";USING$(f$,start);     '
                    LOCATE  5, 1 : PRINT;"     Time Finish: ";USING$(f$,Finish);    '
                    LOCATE  6, 1 : PRINT;" Time Difference: ";USING$(f$,Difference);'
                                                                                    '
                    bytes = (LEN(TestPacket$) * Packets) / difference               '
                    bps = bytes * 8                                                 '
                                                                                    '
                    LOCATE  7, 1 : PRINT;"Bytes per second: ";USING$(f1$,bytes);    '
                    LOCATE  8, 1 : PRINT;" Bits per second: ";USING$(f1$,bps)       '
                    LOCATE 10, 1 : PRINT;"  Path/File Name: ";FileName              '
                                                                                    '
                '*******************************************************************'
                    LOCATE  1,45 : COLOR 0,7 : PRINT;" Read test: "; : COLOR 14,1   '
                                                                                    '
                    OPEN FileName FOR BINARY AS #1                                  '   Now perform a read test
                    Start = TIMER                                                   '
                    GET$ #1,LOF(1),TestPacket$                                      '   Read entire file.
                    CLOSE #1                                                        '
                    Finish = TIMER                                                  '
                                                                                    '
                    Difference = Finish - Start                                     '
                                                                                    '
                    LOCATE  3,40 : PRINT;"   Packet Length: ";_                     '
                                   USING$(f1$,LEN(TestPacket$))                     '
                    LOCATE  4,40 : PRINT;"           Start: ";USING$(f$,start);     '
                    LOCATE  5,40 : PRINT;"          Finish: ";USING$(f$,Finish);    '
                    LOCATE  6,40 : PRINT;"      Difference: ";USING$(f$,Difference);'
                                                                                    '
                    bytes = LEN(TestPacket$) / difference                           '   Bytes per second
                    LOCATE  7,40 : PRINT;"Bytes per second: ";USING$(f1$,bytes)     '
                                                                                    '
                    bps = bytes * 8                                                 '   Bits per second
                    LOCATE  8,40 : PRINT;" Bits per second: ";USING$(f1$,bps)       '
                    KILL FileName                                                   '   Kill off the test file.
                    WAITKEY$                                                        '
                    END FUNCTION                                                    '
                
                REM ******************************
                REM * Select a folder            *
                REM * Ripped from another thread *
                REM ******************************
                
                FUNCTION BrowseForFolder(hWndOwner AS LONG, sPrompt AS STRING) AS STRING
                 LOCAL lpIDList AS LONG, szPath AS ASCIIZ * %MAX_PATH, udtBI AS BrowseInfo
                 udtBI.hWndOwner = hWndOwner
                 udtBI.lpszTitle = STRPTR(sPrompt)
                 udtBI.ulFlags   = %BIF_RETURNONLYFSDIRS
                 lpIDList = SHBrowseForFolder(udtBI)
                 IF lpIDList THEN
                   SHGetPathFromIDList BYVAL lpIDList, szPath
                   CoTaskMemFree lpIDList
                   FUNCTION = szPath
                 END IF
                END FUNCTION
                There are no atheists in a fox hole or the morning of a math test.
                If my flag offends you, I'll help you pack.

                Comment


                • #9
                  You perhaps can use overlapped (asynchronous) I-O. That will give your program a callback when the I-O actually completes.

                  I played around with this once and left this 'starter' code here...Asynchronous (Overlapped) I-O Demo

                  Can't find it right now but "I think" there is an option to not get signalled until the data are actually on the disk.

                  Failing that, you can set up a FindFirstChangeNotifucation() object and wait on that. I'm pretty sure that works off of directoiry changes but I don't know if directory changes are also cached the same as writes.

                  Unbuffered I-O would also serve these particular purposes, since WriteFile() does not return until the write completes, and if it ain't caching that means the disk file itself is updated. Not sure this is really "network" capacity but it sounds good.

                  I'd also believe that there simply has to be some third-party "test your network speed" software available somewhere... maybe some group of network geeks has a web site. That is, you may not have to re-invent this particular wheel.

                  Running these down ought to keep you busy - for a little while, anyway.



                  MCM
                  Michael Mattias
                  Tal Systems (retired)
                  Port Washington WI USA
                  [email protected]
                  http://www.talsystems.com

                  Comment


                  • #10
                    Deleted - think again, Roberts.

                    Right.

                    What exactly are you trying to test?
                    100MB file created.

                    Read twice.

                    1st read: 2.33s
                    2nd read: 0.20s

                    So, 2nd read is from cache.

                    Read twice again.

                    1st read: 0.23s
                    2nd read: 0.22s

                    So, both reads from cache.

                    Execute ClearFileCache ( from post #5's link ) then read twice.

                    1st read: 2.59s
                    2nd read: 0.23s

                    So, cache had been cleared and 1st read had to go to the disk.

                    Execute ClearFileCache before both reads.

                    1st read: 2.34s
                    2nd read: 2.52s

                    So, both reads had to go to the disk.

                    ClearFileCache does what it says on the box - it clears the file cache.

                    *********************************

                    Run again but execute ClearFileCache before app closes. Cache is now cleared.

                    Read.

                    On 2nd Opening use FlushFileBuffers before reading.

                    1st read: 2.34s
                    2nd read: 0.27s

                    So, FlushFileBuffers has not cleared the cache.

                    Repeat using PB's Flush.

                    Same result - cache not cleared.

                    Using FlushFileBuffers and Flush was a waste of time - the buffers had been flushed on closing the file after the first read - the cache, on the other hand, was left intact.

                    Real world use.

                    HashA on file: 1.6s
                    HashB on file: 0.34s

                    HashB is faster.

                    but HashB used the cache.

                    HashA on file: 1.6s
                    ClearFilecache
                    HashB on file: 1.9s

                    HashA is faster and really is faster.

                    I cannot think of ClearFileCache having any use other than creating a level playing field as in the hashing example above.
                    Last edited by David Roberts; 10 Dec 2008, 01:19 PM.

                    Comment


                    • #11
                      Good job Dave, I'm convinced. I'll be using clearFileCache in the future almost for sure.

                      Comment


                      • #12
                        Thanks Dave and John. After mod'ing Daves example for CC, mod'ed my program to include the function.

                        Here are the results after running the program several times. The figures are representative.
                        Code:
                                800Mhz machine 98SE        1.x Ghz with xp
                        -----------------------------------------------
                        Write:  7,472,816 BPS                46,167,720 BPS
                        Read: 28,248,272 BPS                69,002,696 BPS
                        So Dave's ClearFileCache does, indeed seem to work.

                        I'll be posting the ?finalized? source code in the next couple of days. I want to play with it some more, just to make sure.
                        There are no atheists in a fox hole or the morning of a math test.
                        If my flag offends you, I'll help you pack.

                        Comment

                        Working...
                        X