No announcement yet.

File CreationTime mystery

  • Filter
  • Time
  • Show
Clear All
new posts

  • #21
    Originally posted by John Montenigro View Post
    Seems some other readers are getting a bit testy... I'll post the "answer" tomorrow...
    Not to sound testy but today is tomorrow (at least according to my TimeStamp).

    "Everyone is a genius at least once a year;
    a real genius has his original ideas closer together."
    Georg Lichtenberg (1742-1799)
    It's a pretty day. I hope you enjoy it.


    JWAM: (Quit Smoking):
    LDN - A Miracle Drug:


    • #22
      Back in DOS/FAT there were only the last-modified-date. When copying files, the new file would get the current time. If you needed to keep the timestamp, your tool had to read the old stamp and transfer it to the target. Nothing new there. But everything did that, so that was never an issue.

      DST/NTFS as Garstang describes it is inaccurate. NTFS always store file dates using UTF/GMT and the creator's timezone is dropped. And it is correct that Windows Explorer seems to be inconsistent when displaying file dates (but I have never seen anything else do it differently either). Everybody seems to simply convert UTF using current DST info with no regard to whether the file was CREATED, MODIFIED or LAST ACCESSED inside or outside e.g. DST! But that is not an NTFS error! That is an error on the display-tool's hand. Information lost in translation, so to say.

      However, it is debatable whether it is correct to keep tabs of such considerations too - it can be said that Windows Explorer's behaviour is correct. The impression is merely cosmetic. There is no one-answer-only here, but I would defend Explorer's behaviour:

      E.g. imagine a file created in France at 15:00 hours (military time for you US citizens). This being local time, Friday 13th some month and year. Would it be correct to assume that when the file was compared to a file created in New York at 11:00 AM (= 11:00 for you Europeans ) still at the same Friday 13th - that the French file is older? Or is it the newer one? It is after all created later in the day ... UTF says otherwise: 15:00 in France is 14:00 UTF and 11:00 in New York is 17:00 UTF. If one says that the two files are transactions recorded for the same database? Should the US transactions get precedence? I don't think so. So, clearly, timezones are out of the equation. Now, should DST be brought into the equation? It can't. Same thing applies, but in a minor scale.


      However, I am sidestepping:
      I for one do not think the problem at hand is something to be disregarded. Unless John has a good solution.

      I've had enough problems over the years with failing caching algorithms not to care, and this immediately looked like a variation over the same theme.

      I ran the test myself, and I can confirm it using XP SP3. Anyone with other OS-es? (You don't need to patch the timestamps yourself. Simply wait one minute between each repeated attempt).

      And it is uglier: First I tested it using the same source and target file to confirm. Then I used another, (much) larger source file. The correct, larger amount of data arrived at the target, but the old created timestamp was kept! The other two dates (modified and last access) were correctly set.

      To me, this seems to be worhty of a bug report to Microsoft. Does anyone know a workaround yet? Or something else that occurs due to the same(?) error? Or?

      Last edited by Vidar Hanto; 16 Dec 2008, 10:12 PM. Reason: Print error


      • #23
        Sorry for the delay - two court days and other business meetings, and it's only due to a snowstorm that I'm sitting here now. Otherwise, I wouldn't have gotten back till Monday...

        Vidar, thanks for confirming the odd behavior. If I may point out that you mention waiting between changes... that's part of it... But of course, if you were restamping a list of filenames under program control, you wouldn't want to wait for 30 or more seconds between each operation...

        Others have run into this odd scenario:

        I've contacted authors of other apps that manage to accomplish what I set out to do, but I haven't heard back yet on how they get around it...

        SO! While I'm waiting for that info, let me shed some light on the nature of the problem itself.

        It's called "file tunneling".

        An excerpt from the above link:
        The Microsoft Windowsproducts listed at the beginning of this artice contain file system tunneling capabilities to enable compatibility with programs that rely on file systems being able to hold onto file meta-info for a short period of time. This occurs after deletion or renaming and re-introducing a new directory entry with that meta-info (if a create or rename occurs to cause a file of that name to appear again in a short period of time).

        When a name is removed from a directory (rename or delete), its short/long name pair and creation time are saved in a cache, keyed by the name that was removed. When a name is added to a directory (rename or create), the cache is searched to see if there is information to restore. The cache is effective per instance of a directory. If a directory is deleted, the cache for it is removed.

        These paired operations can cause tunneling on "name."


        delete(name)/rename(source, name)

        rename(name, newname)/create(name)

        rename(name, newname)/rename(source, name)

        The idea is to mimic the behavior MS-DOS programs expect when they use the safe save method. They copy the modified data to a temporary file, delete the original and rename the temporary to the original. This should seem to be the original file when complete. Windows performs tunneling on both FAT and NTFS file systems to ensure long/short file names are retained when 16-bit applications perform this safe save operation.

        That's just the beginning... lots more to read...

        What's the significance? Well, for anyone who still thinks about file date/time stamps as if they were simple items in the file's header (DTA?), then forget it! This meta-data scheme makes the dates one step removed from "actual" file data. And if you are trying to manage file date/time stamps, you'll need to find some way around it.

        Yes, as soon as I find something reliable that does not require shortening the wait time in the Registry settings, I'll post it...

        I hope you had some fun with this - always another thing to learn!


        • #24
          Originally posted by Michael Mattias View Post
          >and I DO do it myself, and the change DOES NOT HAPPEN as intended.

          Failing code not shown.
          Code was posted with instructions; please refer to post #1 in this thread.


          • #25
            Code in Post #1 does not show using SetFileTime() function.

            It shows using PowerbASIC filecopy function. Or maybe it's the Windows FileCopy function, the PB function being CopyFile.

            In any event the code to try is ..

               TS1 =  GetFileTimeOfInterest (SourceFile) 
               Copy   SourceFile to Target file 
               SetFileTime (of interest)  of Target file = TS1
            Michael Mattias
            Tal Systems (retired)
            Port Washington WI USA
            [email protected]


            • #26

              While your pseudocode looks reasonable, it doesn't work. I've used at least 5 variations of SetFileTime() routines found on the forums. (That's why the instructions directed the use of a non-code tool...)

              Here's just one:

              '0 = LastWriteTime (visible in dos)
              '1 = LastAccessTime
              '2 = CreationTime
              Function SetFileDateTime( FileName As ASCIIZ, fSysTime As SYSTEMTIME, ByVal tMode As Long ) EXPORT As Long
                  Dim hFile       As Long
                  Dim lpFileTime1 As FILETIME
                  Dim lpFileTime2 As FILETIME
                  If Clng( VarPtr( FileName ) ) = 0 Then Exit Function
                  hFile = GetAttr( FileName )
                  If ErrClear Then Exit Function
                  hFile = 0
                  hFile = CreateFile( FileName, _
                      %GENERIC_READ Or %GENERIC_WRITE _
                      , %FILE_SHARE_READ Or %FILE_SHARE_WRITE _
                      , ByVal 0& _
                      , %OPEN_EXISTING _
                      , %FILE_FLAG_BACKUP_SEMANTICS _
                      , 0 _
                  If hFile <> %INVALID_HANDLE_VALUE Then
                      Call SystemTimeToFileTime   ( fSysTime, lpFileTime1 )
                      Call LocalFileTimeToFileTime( lpFileTime1, lpFileTime2 )
                      Select Case tMode
                      Case 0: FUNCTION = SetFileTime( hFile, ByVal 0&, ByVal 0&, lpFileTime2 )
                      Case 1: FUNCTION = SetFileTime( hFile, ByVal 0&, lpFileTime2, ByVal 0& )
                      Case 2: FUNCTION = SetFileTime( hFile, lpFileTime2, ByVal 0&, ByVal 0& )
                      End Select
                      Call CloseHandle( hFile )
                  End If
              End Function
              Works fine for the LastWrite and LastAccess stamps but not for the CreationTime on most everything, EXCEPT for a moved/copied/deleted/renamed and re-created file in the same directory.

              At the bottom of the MS article is a list of OS file systems for which this behavior is inherent (by design).
              Last edited by John Montenigro; 19 Dec 2008, 02:07 PM.


              • #27
                Well, OK, we have some code now....

                I'm not sure I like the use of FILE_FLAG_BACKUP_SEMANTICS.
                FILE_FLAG_BACKUP_SEMANTICS Windows NT/2000/XP: Indicates that the file is being opened or created for a backup or restore operation. The system ensures that the calling process overrides file security checks, provided it has the SE_BACKUP_NAME and SE_RESTORE_NAME privileges. For more information, see Changing Privileges in a Token.
                You can also set this flag to obtain a handle to a directory. Where indicated, a directory handle can be passed to some functions in place of a file handle.
                I wonder if it's preserving the CreateDate because of that? You can try NULL for flags... that's where I'd start anyway.

                I'd try it but I have to go get a haircut right now.

                Michael Mattias
                Tal Systems (retired)
                Port Washington WI USA
                [email protected]


                • #28
                  While you were out getting a haircut, I shoveled 6" snow off the driveway and front walk...

                  Once you try the procedure outlined above and see the results yourself, your speculations are more likely to return actual results...

                  Here are a few more routines I tried from various authors on the forums. I started with them "as found", you may have to edit off some "export" statements and such, but at first I did not edit the substance, except to use CreationTime instead of LastWriteTime in the proper parameter position.

                  In later tests, I did try changing the PB OPEN to API CreateFile and FindFirstFile, to ensure that I was getting a valid handle, and other such variations. I used simple calls using just FILETIME structures, and more complex variations using WIN32_FIND_DATA structures. Oh yeah, I had also tried PB's DIR..TO with a DIRDATA structure...

                  Regardless of the code used, the results seem to be the same, and seem to be determined not by application code but by the OS (which I think is the "file tunneling" feature). THAT was the mystery!

                  The greater mystery is how to get around it without waiting the timeout period for each filename in a list, and without messing with the Registry and then restoring it...(would that even be an acceptable design strategy?)

                  Function SetFileDateandTime(ccsFileSpec As String,ft As SYSTEMTIME)Export As Long   'common32.bas
                  Local hFile     As Long
                  Local Result    As Long
                  Local ccsFileTime  As FILETIME
                  Local fLocTime  As FILETIME
                  hFile = CreateFile(ccsFileSpec + Chr$(0), ByVal %GENERIC_READ Or %GENERIC_WRITE, _
                                      ByVal 0, ByVal %NULL, ByVal %OPEN_ALWAYS, _
                                      ByVal %FILE_ATTRIBUTE_NORMAL, ByVal %NULL)
                  ' convert system date/time to file structure.
                  SystemTimeToFileTime ft, fLocTime
                  ' convert local file time to UTC file time
                  ' setfiletime expects a time that is relative to utc time.
                  LocalFileTimeToFileTime fLocTime, ccsFileTime
                  ' set the file time
                  Result = SetFileTime(ByVal hFile, ByVal %NULL, ByVal %NULL, ccsFileTime)
                  CloseHandle hFile
                  Function = Result
                  End Function
                  ' SetFileDate - Set the time and date of a file
                  'Scott T?
                  SUB SetFileDate(sFileName AS STRING, _
                                  iYear     AS LONG, iMonth  AS LONG, iDay    AS LONG, _
                                  iHour     AS LONG, iMinute AS LONG, iSecond AS LONG) EXPORT
                      LOCAL nFile          AS LONG
                      LOCAL hFile          AS DWORD
                      LOCAL CreationTime   AS FILETIME
                      LOCAL LastAccessTime AS FILETIME
                      LOCAL LastWriteTime  AS FILETIME
                      LOCAL ST             AS SYSTEMTIME
                      LOCAL LT             AS FILETIME
                      LOCAL lpReOpenBuff   AS OFSTRUCT
                          nFile = FREEFILE
                          OPEN sFileName FOR INPUT ACCESS READ LOCK SHARED AS #nFile
                          EXIT SUB
                          hFile = FILEATTR(nFile,2)
                          GetFileTime hFile, CreationTime, LastAccessTime, LastWriteTime
                          CLOSE #nFile
                          FileTimeToLocalFileTime LastWriteTime, LT
                          FileTimeToSystemTime LT, ST
                          ST.wYear   = iYear
                          ST.wMonth  = iMonth
                          ST.wDay    = iDay
                          ST.wHour   = iHour
                          ST.wMinute = iMinute
                          ST.wSecond = iSecond
                          SystemTimeToFileTime ST, LT
                          LocalFileTimeToFileTime LT, LastWriteTime
                          SetFileTime hFile, CreationTime, LastAccessTime, LastWriteTime
                      END TRY
                  END SUB
                  SUB SetDateAndTime(BYVAL sFilename AS STRING, BYVAL sfiletimeinfo AS STRING )  'Pierre B?
                   LOCAL CreationTime   AS FILETIME
                   LOCAL LastAccessTime AS FILETIME
                   LOCAL LastWriteTime  AS FILETIME
                   LOCAL hFile          AS DWORD
                   LOCAL FileNum        AS DWORD
                   IF LEN(sfiletimeinfo) <> 24 THEN EXIT SUB 'rem bad date and time returned
                     'OPEN sFilename FOR INPUT ACCESS READ LOCK SHARED AS #FileNum 'See next line
                     OPEN sFilename FOR BINARY ACCESS WRITE AS #FileNum 'Need write access
                     hFile = FILEATTR(#FileNum, 2) 'This is the handle needed for GetFileTime/SetFileTime
                                                   '2 = The operating system file handle for the file
                     EXIT SUB
                     'Use TYPE SET to copy back from a string
                     TYPE SET CreationTime   = MID$(sfiletimeinfo, 1, 8)
                     TYPE SET LastAccessTime = MID$(sfiletimeinfo, 9, 8)
                     TYPE SET LastWriteTime  = MID$(sfiletimeinfo, 17, 8)
                     SetFileTime hFile, CreationTime, LastAccessTime, LastWriteTime
                     CLOSE #FileNum
                   END TRY
                  END SUB
                  Last edited by John Montenigro; 19 Dec 2008, 04:07 PM.


                  • #29
                    **** barber closed up and took off. Drove about 20 miles round trip thru the snow for nothing. Because of all the snow (we got about 12"), I was so intent on the road that I missed "the moment" when my odometer finally turned over to 100,000 miles.... got back and lo and behold it read 100005.

                    I really don't think the vehicle will be around to hit 200,000... I bought it new in '96. At this rate it would not hit 200,000 until sometime in 2021... come to think of it, I''ll be doing well to be around myself in 2021.

                    Michael Mattias
                    Tal Systems (retired)
                    Port Washington WI USA
                    [email protected]


                    • #30
                      Not to worry - you're about 22 messages away from your forum odometer clicking over a "big one"!!


                      • #31
                        So, it seems like this is a variation over the old cache bug problem, then. And a bug report to MS won't do, they already know about it and according to John's description above, it is apparently by design. So it's not a bug, even.

                        The question remains then, how to cope with it?
                        Off the top of my head, I see three alternatives:
                        1) Don't bother. Will do for many (most?) purposes, but not all.
                        2) Make sure your own SW is immune to the problem. This is probably the only universal solution (you can't modify the client's PC can you? Well - you shouldn't), but doesn't cover 3rd party SW.
                        3) Get a clever workaround that also covers 3rd party SW, but this can only be applied to PCs you are in control of. According to the above, this workaround seems to be to outwait the cache.

                        Which brings me to the only solution I have applied to such cases with universal success: Disable the cache completely. Anything else have just changed the face of the problem.

                        I can't test anything now, but disabling this cache may or may not hurt performance. There are so much caching all over the place nowadays, so it may go unnoticed. Maybe the harddrive's HW cache is sufficient all by itself? I would suspect (a guess, not tested!) that this only applies to local drives as file access over networks would rely on the server's local file handling.

                        Two examples from (ancient?) history:
                        Remember the "blue screen of death" in Windows 3.0/3.1? That was due to SMARTDRV.SYS doing its default caching when DOS 5.0 hit the streets. Disable it, and there were no problem. Doing so did hurt performance a bit, but not nearly as much as the blue screen did ...

                        Remember Novell DOS drivers? Version 2.something didn't write to server disk data sets that was smaller than the driver's cache. A disaster for database transactions. There were corrupt databases all over the place. Again, disabling the cache was the solution. This time, the performance hit wasn't seen - it had to be measured.



                        • #32
                          You just got me thinking Michael... I dont like the feeling that someday i will know my life is getting close to an end.

                          Good thing that doesnt have to happen. If i overwritte myself with a BAT extension my age might change back to 22.


                          • #33

                            I have not found any information about a "file tunneling" cache that could be disabled. I doubt that I have uncovered everything there is to know about it, but so far, I've seen nothing - except to disable the timeout.

                            I've never done anything with Registry settings, so I'm a few days away from being able to test it, AND I don't know if this would be a good workaround:

                            1. Capture the current registry timeout setting,
                            2. change it to 0,
                            3. process the list of files,
                            4. change the registry timeout back to its original setting.

                            I don't like this because the application has no knowledge if the timeout change would adversely affect other operations occurring on the drive. For example, if there were another process that depended on the timeout for recovery purposes, this strategy would adversely interfere.

                            Another possible strategy:

                            1. capture the filenames and their CreationTime to a file,
                            2. allow the timeout to expire,
                            3. read the file and apply the saved timestamps.

                            This is less invasive of the user's environment, but it would require the app to go "on hold" for awhile, and the user may wonder why it didn't "complete" such a "simple" operation more quickly... (User's wonder could be averted if the app provided sufficient messages to explain what's going to happen, what's happening, and what happened...)

                            So, I'm still thinking about alternative workarounds.

                            Any other ideas?
                            Last edited by John Montenigro; 20 Dec 2008, 08:53 AM.


                            • #34
                              Save the creation dates of interest in your own disk file or database. Change when wanted. Always get creation dates for 'files of interest' from your own storage, ignoring what Windows says it is.
                              Michael Mattias
                              Tal Systems (retired)
                              Port Washington WI USA
                              [email protected]


                              • #35

                                In doing further research, I found this page:

                                Excerpt (I added bolding and italic for emphasis)

                                The ReplaceFile function combines several steps within a single function. An application can call ReplaceFile instead of calling separate functions to save the data to a new file, rename the original file using a temporary name, rename the new file to have the same name as the original file, and delete the original file. Another advantage is that ReplaceFile not only copies the new file data, but also preserves the following attributes of the original file:

                                Creation time
                                Short file name
                                Object identifier
                                Named streams not already in the replacement file

                                For example, if the replacement file is encrypted, but the replaced file is not encrypted, the resulting file is not encrypted.

                                The resulting file has the same file ID as the replacement file.

                                The backup file, replaced file, and replacement file must all reside on the same volume.
                                What caught my attention is that the ReplaceFile function preserves the CreationTime of the original file!!!
                                <SMALLPARADE> Yea! </SMALLPARADE>

                                The article indicates that WinBase.h is the appropriate header file. I got with PBWIN9, but to avoid having to edit out all the definitions duplicated in, I've just copied the relevant DECLARE and flags into my code...

                                The article also indicates that Windows 2000 Professional is the minimum client, and although I'm running XP Pro SP3, it does compile, run, and is returning errors about permissions - but they're appropriate to the situation, which verifies that the ReplaceFile function is available and operating...

                                SO... I've been playing with it a bit. I was trying to trick it by supplying the same file as both ReplaceFile and ReplacementFile - it didn't like that.

                                So, a workaround... I will supply my original SourceFile as the ReplacementFile and a copy of it as the ReplaceFile, expecting that I'll get the file I want as the BackupFile...

                                Yes this will be slower than a "normal" process because I'll have to make a copy of each file first and delete the copy afterwards, but if this forces the OS to give me the file I need (with intact CreationTime stamp), AND if it eliminates my having to mess with the timeout period or its Registry settings, then I'll consider it a success...

                                We shall see...

                                I'll update this post in a day or so... (maybe I can persuade Santa to give me some time this week) Merry Happy everything!


                                • #36
                                  For example, if the replacement file is encrypted, but the replaced file is not encrypted, the resulting file is not encrypted
                                  Hmm...nah, it could not be that simple to decrypt a file, could it?
                                  Michael Mattias
                                  Tal Systems (retired)
                                  Port Washington WI USA
                                  [email protected]


                                  • #37
                                    I don't know - haven't tried that yet... might just come out as junk...

                                    But it would be interesting to see what it actually does.


                                    • #38
                                      I doubt it will decrypt, but if I ever saw something that could have been missed in testing.....
                                      Michael Mattias
                                      Tal Systems (retired)
                                      Port Washington WI USA
                                      [email protected]


                                      • #39
                                        Originally posted by Adam J. Drake View Post

                                        Are you moving or copying a file?

                                        If moving - not copying - you can use the NAME command to move a file.

                                        If copying - then you would need to use what Michael suggested and ensure FileTime is preserved yourself.

                                        EDIT: Scratch that suggestion - apparently moving by using the NAME command also resets the creation date/time stamp. You will need to ensure file date/time stamps are preserved on your own if you are going to depend on them...

                                        I found that NAME actually can be used to preserve the CreationTime, IF you reverse the process!!!!

                                        Name SourceFile As DestinationFile    ' << make backup FIRST, by MOVing original
                                        FileCopy DestinationFile, SourceFile  ' << restore original from backup copy
                                        This way, when the OS preserves the CreationTime from the DELETED original and imposes it on the new copy, the result is precisely what was wanted -- both files are identical, right down to their CreationTime stamps!!!

                                        Not one API call to SetFileTime!!! No intermediate variables!!! No messing with the Registry!!! No messing with "file tunneling" delay times!!!

                                        Clean and simple!!! (though quite counter-intuitive...)


                                        • #40
                                          ... and then there is the folder date

                                          can the same method be applied to a folder date?
                                          sometimes I do Q&D (quick and dirty) backups of an entire folder.
                                          I delete an entire folder from my backup disk,
                                          then copy the same folder from workdisk to backup
                                          a month or so later I want to manually check when I made the last backup.
                                          instead of seeing a date of eg: 19 Sept 2008, I find 14 May 2002.