Announcement

Collapse
No announcement yet.

Unexpected Variation in File Loading Time

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Unexpected Variation in File Loading Time

    To remove caching from calculating the time to load multiple files (such as in the discussion on fast loading of images), I wrote this test code. It generates a fresh set of 50 images each time, using a randomly generated name. The same image is used in all files but file names are different each time you run the code. I assume that will eliminate the effects of caching.

    But when I run the code, I get times that vary from 1s to 5s.

    How could the OS possibly get such varying results from the same, simple task? I might have expected some small variation, but 5X? Just doesn't sound right.

    Perhaps my sample code simple isn't doing what I thought.

    Code:
    #Include "win32api.inc"
    Global qFreq, qStart, qStop As Quad
    
    Function PBMain()
       Local fName$, temp$, i,iRnd As Long
       'clear/create the files to read in so cache won't play into the answer
       'assumes there is a file "bigbaby.bmp" in the EXE folder
       Randomize Timer
       iRnd = Rnd(1,1000) * 1000
       If IsFalse IsFolder("bmp") Then MkDir "bmp"
       temp$ = Dir$("bmp\*.*")
       If Len(temp$) Then Kill "bmp\*.bmp"
       For i = 1 To 50 : FileCopy "bigbaby.bmp", "bmp\bigbaby" + Format$(i+iRnd,"0000000") + ".bmp" : Next i
    
    ? "Ready to Read Images"
    
       'read in every file to temp$
       QueryPerformanceFrequency qFreq
       QueryPerformanceCounter   qStart
       fName$ = Dir$("bmp\*.bmp")
       While Len(fName$)
          Open "bmp\" + fName$ For Binary As #1
          Get$ #1, Lof(1), temp$
          Close #1
          fName$ = Dir$(Next)
       Wend
       QueryPerformanceCounter   qStop
       ? Format$((qStop-qStart)/qFreq,"###.000") & " seconds"
    End Function

  • #2
    You're not removing the caching from the calculation.
    The cache works on both writes and reads.
    When you copy the files, they're all copied to cache and then to disk.
    Then you read them back and the ones still in the cache come from the cache.

    Comment


    • #3
      Good morning, Paul!

      That's good to know. I thought it was only on reads.

      But even so, any idea on why there is such a variation on the time to complete the task?

      And, for that matter, how can I clear the cache?

      Comment


      • #4
        I don't know how to clear cache.
        Might turn off caching policy while testing in device manager.
        Click image for larger version

Name:	beene.png
Views:	76
Size:	6.5 KB
ID:	784578
        https://duckduckgo.com instead of google

        Comment


        • #5
          Just ran across this ... SetSystemFileCacheSize

          Among other things, it says this ...

          To flush the cache, specify (SIZE_T) -1.
          Looks promising. I'll go check it out.

          Comment


          • #6
            I haven't looked at what that call does but "Flush the cache" doesn't necessarily mean empty it.
            It means to write the content of the cache to the permanent storage but if you immediately try to read it back then you still read back the cached version as it's still in the cache as well as on the disc.

            Comment


            • #7
              A couple of ideas
              https://stackoverflow.com/questions/478340/clear-file-cache-to-repeat-performance-testing

              and
              https://stackoverflow.com/questions/...e-system-cache


              This is also a simple (but slow?) method:

              "Create a large file, however many GB you need, and each time you want to reset your file cache, make a copy of the file. Then make sure you delete the old file.
              So, create BIGFILE1.DAT, copy it to BIGFILE2.DAT, and then delete BIGFILE1.DAT (which removes it from the disk and the cache). Next time, just reverse the process."



              And then there's a CLI utility here:

              https://www.delphitools.info/2013/11...ws-file-cache/

              Comment


              • #8
                Also note: hard drives have their own internal cache which may be up to 64MB is size, so it may not be the windows cache which is causing the variability.

                Comment


                • #9
                  My surprise is the 1s to 5s range of results. unless cache is emptied, I would expect very similar results from run to run. Yes?

                  Comment


                  • #10
                    > ...how can I clear the cache?

                    Gary,
                    To clear Windows cache, you may also use Sysinternals / Windows Internals, RamMap.exe, and select "Empty", "Empty stanby list"...

                    Comment

                    Working...
                    X