Empty out the R/W cache?

[Mel Bishop]

Any way of flushing (emptying, nulling out, etc.) the read/write cache?

I'm writing a program to test my actual network speed under various conditions and I'm getting unexpected results.

I suspect it's because the computer(s) are reading the cache instead of the file being processed.

Help, please?

[Michael Mattias]

The cache SHOULD be current. (yes I have my own doubts). That is, you SHOULD be reading the cache, not the disk file.

You can try opening the file without buffering or specify write-though on output. See demo here: Create and use a file which will be deleted on close and play with the "Flags" options in the CreateFile calls to create necessary conditions.

Or, if you are testing by sending the same data repeatedly, that's why the cache would be used... because it IS current. (Code not shown).

MCM

[Don Dickinson]

if you want to turn off caching at the disk level you can do it by right clicking on the drive in my computer. then going to the hardware tab, select the drive, and choose the properties button. if you're using vista, then you'll have to click the "change settings" button in order to change anything. anyway, on one of the tabs in the settings screen there is a checkbox for write caching. in vista its not a checkbox, its two radio buttons - optimizing for quick removal disables write caching.

-don

[John Gleason]

I've used FlushFileBuffers to clear the cache for a file.

little example:

   OPEN "c:\r1.dat" FOR BINARY AS #1
   hFil = FILEATTR(#1, 2)
   FlushFileBuffers hFil

[David Roberts]

My understanding is that the SDK FlushFileBuffers is the same as the PB Flush statement. The internal buffers are manually flushed to disk whilst the file is still open instead of waiting for them to be flushed automatically on closing the file. This has nothing to do with the cache.

To stop caching we need to employ, as Mr Mattias mentioned, FILE_FLAG_NO_BUFFERING in dwFlagsAndAttributes when using CreateFile. It is not straightforward. For example, the buffers addresses should be sector aligned and for that I use VirtualAlloc as recommended in the SDK.

Have a look at post #2 here.

We have an analogy with the registry. The RAM portions are flushed to disk periodically. Effectively, the registry is buffered. It is not, however, cached.

[Michael Mattias]

What exactly are you trying to test?

Maybe this is not the best method.

I'm no network guy but I know there are some here; maybe there is some 'better' way to get whatever info you are after.

Provide the 'what' and I'm sure you will get several 'hows.'

[Michael Mattias]

FILE_FLAG_NO_BUFFERING in dwFlagsAndAttributes when using CreateFile. It is not straightforward.

Actually, it IS straightforward.

What it isn't is quick and easy.

[Mel Bishop]

Well, MM (and others who are curious).

I'm trying to get a realistic idea of how fast my home network is actually operating at under various conditions. Example: It will be slower if I am listening to streaming media.

Currently, I am generating a really large file and writing it from one computer to another. Using TIMER results, I then calculate how long it takes to write the file, do some math and display the results.

Then, I read the file, see how long it takes to get the entire file into memory, do some more math and display the results.

On one of my terminals, (800mhz machine with 98SE), I get a write speed of 6mBPS and a read speed of 20mBPS (that's bits, not bytes).

On the other machine (1.x ghz dual core running xp), I am getting 46mBPS and a read speed of 6.x gigBPS

Out of a 10/100 switch? I don't think so.

Well, thinking about it...I wasn't planning on posting the source code until I get all the kinks worked out but, here it is. Written in CC4.04.

Maybe somebody can determine what I'm doing wrong.

REM ***********************************************************     '
REM * Get a reasonably accurate speed measure of your network *     '
REM ***********************************************************     '
                                                                    '
    #INCLUDE "win32api.inc"                                         '
    DECLARE FUNCTION BrowseForFolder(LONG, STRING) AS STRING        '
                                                                    '
FUNCTION PBMAIN                                                     '
    LOCAL x, y, z, Packets           AS LONG                        '
    LOCAL bytes, bps                 AS QUAD                        '
    LOCAL Start, Finish, Difference  AS DOUBLE                      '
    LOCAL FileName                   AS STRING                      '
'*******************************************************************'
    FOR x = 0 TO 255                                                '<:  Build up the test
    TestPacket$ = TestPacket$ + CHR$(x)                             ' |  packet.
    NEXT x                                                          '<:
                                                                    '
    f1$ = "###,###,###"                                             '
    f$  = "##,###.#######"                                          '
                                                                    '
    Packets = 50000                                                 '   Write 50K packets
                                                                    '
'*******************************************************************'
    COLOR 14,1                                                      '
    CLS                                                             '
    FileName = browseforfolder(0, "Select destination folder...")   '   Output path/filename
    IF FileName = "" THEN EXIT FUNCTION                             '
    IF RIGHT$(FileName,1) <> "\" THEN FileName = FileName+"\"       '
    FileName = FileName + "SpeedTest.txt"                           '
                                                                    '
'*******************************************************************'
                                                                    '
    OPEN FileName FOR BINARY AS #1                                  '
    LOCATE  1, 5 : COLOR 0,7 : PRINT;" Write test " : COLOR 14,1    '
                                                                    '
    Start = TIMER                                                   '
    FOR x = 1 TO Packets                                            '<: Test the write speed
    PUT$ #1,TestPacket$                                             ' |
    NEXT x                                                          '<:
    CLOSE #1                                                        '
    finish = TIMER                                                  '
    Difference = Finish - Start                                     '   In seconds
                                                                    '
'*******************************************************************'   Start calculations.
                                                                    '
    LOCATE  3, 1 : PRINT;"   Packet Length: ";_                     '
                   USING$(f1$,LEN(TestPacket$) * Packets)           '
    LOCATE  4, 1 : PRINT;"      Time Start: ";USING$(f$,start);     '
    LOCATE  5, 1 : PRINT;"     Time Finish: ";USING$(f$,Finish);    '
    LOCATE  6, 1 : PRINT;" Time Difference: ";USING$(f$,Difference);'
                                                                    '
    bytes = (LEN(TestPacket$) * Packets) / difference               '
    bps = bytes * 8                                                 '
                                                                    '
    LOCATE  7, 1 : PRINT;"Bytes per second: ";USING$(f1$,bytes);    '
    LOCATE  8, 1 : PRINT;" Bits per second: ";USING$(f1$,bps)       '
    LOCATE 10, 1 : PRINT;"  Path/File Name: ";FileName              '
                                                                    '
'*******************************************************************'
    LOCATE  1,45 : COLOR 0,7 : PRINT;" Read test: "; : COLOR 14,1   '
                                                                    '
    OPEN FileName FOR BINARY AS #1                                  '   Now perform a read test
    Start = TIMER                                                   '
    GET$ #1,LOF(1),TestPacket$                                      '   Read entire file.
    CLOSE #1                                                        '
    Finish = TIMER                                                  '
                                                                    '
    Difference = Finish - Start                                     '
                                                                    '
    LOCATE  3,40 : PRINT;"   Packet Length: ";_                     '
                   USING$(f1$,LEN(TestPacket$))                     '
    LOCATE  4,40 : PRINT;"           Start: ";USING$(f$,start);     '
    LOCATE  5,40 : PRINT;"          Finish: ";USING$(f$,Finish);    '
    LOCATE  6,40 : PRINT;"      Difference: ";USING$(f$,Difference);'
                                                                    '
    bytes = LEN(TestPacket$) / difference                           '   Bytes per second
    LOCATE  7,40 : PRINT;"Bytes per second: ";USING$(f1$,bytes)     '
                                                                    '
    bps = bytes * 8                                                 '   Bits per second
    LOCATE  8,40 : PRINT;" Bits per second: ";USING$(f1$,bps)       '
    KILL FileName                                                   '   Kill off the test file.
    WAITKEY$                                                        '
    END FUNCTION                                                    '

REM ******************************
REM * Select a folder            *
REM * Ripped from another thread *
REM ******************************

FUNCTION BrowseForFolder(hWndOwner AS LONG, sPrompt AS STRING) AS STRING
 LOCAL lpIDList AS LONG, szPath AS ASCIIZ * %MAX_PATH, udtBI AS BrowseInfo
 udtBI.hWndOwner = hWndOwner
 udtBI.lpszTitle = STRPTR(sPrompt)
 udtBI.ulFlags   = %BIF_RETURNONLYFSDIRS
 lpIDList = SHBrowseForFolder(udtBI)
 IF lpIDList THEN
   SHGetPathFromIDList BYVAL lpIDList, szPath
   CoTaskMemFree lpIDList
   FUNCTION = szPath
 END IF
END FUNCTION

[Michael Mattias]

You perhaps can use overlapped (asynchronous) I-O. That will give your program a callback when the I-O actually completes.

I played around with this once and left this 'starter' code here...Asynchronous (Overlapped) I-O Demo

Can't find it right now but "I think" there is an option to not get signalled until the data are actually on the disk.

Failing that, you can set up a FindFirstChangeNotifucation() object and wait on that. I'm pretty sure that works off of directoiry changes but I don't know if directory changes are also cached the same as writes.

Unbuffered I-O would also serve these particular purposes, since WriteFile() does not return until the write completes, and if it ain't caching that means the disk file itself is updated. Not sure this is really "network" capacity but it sounds good.

I'd also believe that there simply has to be some third-party "test your network speed" software available somewhere... maybe some group of network geeks has a web site. That is, you may not have to re-invent this particular wheel.

Running these down ought to keep you busy - for a little while, anyway.



MCM

[David Roberts]

Deleted - think again, Roberts.

Right.

What exactly are you trying to test?

100MB file created.

Read twice.

1st read: 2.33s
2nd read: 0.20s

So, 2nd read is from cache.

Read twice again.

1st read: 0.23s
2nd read: 0.22s

So, both reads from cache.

Execute ClearFileCache ( from post #5's link ) then read twice.

1st read: 2.59s
2nd read: 0.23s

So, cache had been cleared and 1st read had to go to the disk.

Execute ClearFileCache before both reads.

1st read: 2.34s
2nd read: 2.52s

So, both reads had to go to the disk.

ClearFileCache does what it says on the box - it clears the file cache.

*********************************

Run again but execute ClearFileCache before app closes. Cache is now cleared.

Read.

On 2nd Opening use FlushFileBuffers before reading.

1st read: 2.34s
2nd read: 0.27s

So, FlushFileBuffers has not cleared the cache.

Repeat using PB's Flush.

Same result - cache not cleared.

Using FlushFileBuffers and Flush was a waste of time - the buffers had been flushed on closing the file after the first read - the cache, on the other hand, was left intact.

Real world use.

HashA on file: 1.6s
HashB on file: 0.34s

HashB is faster.

but HashB used the cache.

HashA on file: 1.6s
ClearFilecache
HashB on file: 1.9s

HashA is faster and really is faster.

I cannot think of ClearFileCache having any use other than creating a level playing field as in the hashing example above.

[John Gleason]

Good job Dave, I'm convinced. I'll be using clearFileCache in the future almost for sure.

[Mel Bishop]

Thanks Dave and John. After mod'ing Daves example for CC, mod'ed my program to include the function.

Here are the results after running the program several times. The figures are representative.

        800Mhz machine 98SE        1.x Ghz with xp
-----------------------------------------------
Write:  7,472,816 BPS                46,167,720 BPS
Read: 28,248,272 BPS                69,002,696 BPS

So Dave's ClearFileCache does, indeed seem to work.

I'll be posting the ?finalized? source code in the next couple of days. I want to play with it some more, just to make sure.