File CreationTime mystery

[Gösta H. Lovgren-2]

Seems some other readers are getting a bit testy... I'll post the "answer" tomorrow...

Not to sound testy but today is tomorrow (at least according to my TimeStamp).

=======================================================
"Everyone is a genius at least once a year;
a real genius has his original ideas closer together."
Georg Lichtenberg (1742-1799)
=======================================================

[Vidar Hanto]

Back in DOS/FAT there were only the last-modified-date. When copying files, the new file would get the current time. If you needed to keep the timestamp, your tool had to read the old stamp and transfer it to the target. Nothing new there. But everything did that, so that was never an issue.

DST/NTFS as Garstang describes it is inaccurate. NTFS always store file dates using UTF/GMT and the creator's timezone is dropped. And it is correct that Windows Explorer seems to be inconsistent when displaying file dates (but I have never seen anything else do it differently either). Everybody seems to simply convert UTF using current DST info with no regard to whether the file was CREATED, MODIFIED or LAST ACCESSED inside or outside e.g. DST! But that is not an NTFS error! That is an error on the display-tool's hand. Information lost in translation, so to say.

However, it is debatable whether it is correct to keep tabs of such considerations too - it can be said that Windows Explorer's behaviour is correct. The impression is merely cosmetic. There is no one-answer-only here, but I would defend Explorer's behaviour:

E.g. imagine a file created in France at 15:00 hours (military time for you US citizens). This being local time, Friday 13th some month and year. Would it be correct to assume that when the file was compared to a file created in New York at 11:00 AM (= 11:00 for you Europeans ) still at the same Friday 13th - that the French file is older? Or is it the newer one? It is after all created later in the day ... UTF says otherwise: 15:00 in France is 14:00 UTF and 11:00 in New York is 17:00 UTF. If one says that the two files are transactions recorded for the same database? Should the US transactions get precedence? I don't think so. So, clearly, timezones are out of the equation. Now, should DST be brought into the equation? It can't. Same thing applies, but in a minor scale.

----

However, I am sidestepping:
I for one do not think the problem at hand is something to be disregarded. Unless John has a good solution.

I've had enough problems over the years with failing caching algorithms not to care, and this immediately looked like a variation over the same theme.

I ran the test myself, and I can confirm it using XP SP3. Anyone with other OS-es? (You don't need to patch the timestamps yourself. Simply wait one minute between each repeated attempt).

And it is uglier: First I tested it using the same source and target file to confirm. Then I used another, (much) larger source file. The correct, larger amount of data arrived at the target, but the old created timestamp was kept! The other two dates (modified and last access) were correctly set.

To me, this seems to be worhty of a bug report to Microsoft. Does anyone know a workaround yet? Or something else that occurs due to the same(?) error? Or?

ViH

[John Montenigro]

Sorry for the delay - two court days and other business meetings, and it's only due to a snowstorm that I'm sitting here now. Otherwise, I wouldn't have gotten back till Monday...

Vidar, thanks for confirming the odd behavior. If I may point out that you mention waiting between changes... that's part of it... But of course, if you were restamping a list of filenames under program control, you wouldn't want to wait for 30 or more seconds between each operation...

Others have run into this odd scenario:

http://www.eggheadcafe.com/software/...changed-d.aspx

http://www.tech-archive.net/Archive/...4-06/5191.html


I've contacted authors of other apps that manage to accomplish what I set out to do, but I haven't heard back yet on how they get around it...


SO! While I'm waiting for that info, let me shed some light on the nature of the problem itself.

It's called "file tunneling".

http://support.microsoft.com/?scid=k...2190&x=12&y=13
http://support.microsoft.com/default...en-us%3B172190

An excerpt from the above link:

The Microsoft Windowsproducts listed at the beginning of this artice contain file system tunneling capabilities to enable compatibility with programs that rely on file systems being able to hold onto file meta-info for a short period of time. This occurs after deletion or renaming and re-introducing a new directory entry with that meta-info (if a create or rename occurs to cause a file of that name to appear again in a short period of time).

When a name is removed from a directory (rename or delete), its short/long name pair and creation time are saved in a cache, keyed by the name that was removed. When a name is added to a directory (rename or create), the cache is searched to see if there is information to restore. The cache is effective per instance of a directory. If a directory is deleted, the cache for it is removed.

These paired operations can cause tunneling on "name."

delete(name)/create(name)

delete(name)/rename(source, name)

rename(name, newname)/create(name)

rename(name, newname)/rename(source, name)

The idea is to mimic the behavior MS-DOS programs expect when they use the safe save method. They copy the modified data to a temporary file, delete the original and rename the temporary to the original. This should seem to be the original file when complete. Windows performs tunneling on both FAT and NTFS file systems to ensure long/short file names are retained when 16-bit applications perform this safe save operation.

That's just the beginning... lots more to read...

What's the significance? Well, for anyone who still thinks about file date/time stamps as if they were simple items in the file's header (DTA?), then forget it! This meta-data scheme makes the dates one step removed from "actual" file data. And if you are trying to manage file date/time stamps, you'll need to find some way around it.

Yes, as soon as I find something reliable that does not require shortening the wait time in the Registry settings, I'll post it...

I hope you had some fun with this - always another thing to learn!

[John Montenigro]

>and I DO do it myself, and the change DOES NOT HAPPEN as intended.

Failing code not shown.

Code was posted with instructions; please refer to post #1 in this thread.

[Michael Mattias]

Code in Post #1 does not show using SetFileTime() function.

It shows using PowerbASIC filecopy function. Or maybe it's the Windows FileCopy function, the PB function being CopyFile.

In any event the code to try is ..

   TS1 =  GetFileTimeOfInterest (SourceFile) 
   Copy   SourceFile to Target file 
   SetFileTime (of interest)  of Target file = TS1

[John Montenigro]

MCM,

While your pseudocode looks reasonable, it doesn't work. I've used at least 5 variations of SetFileTime() routines found on the forums. (That's why the instructions directed the use of a non-code tool...)

Here's just one:

'0 = LastWriteTime (visible in dos)
'1 = LastAccessTime
'2 = CreationTime
Function SetFileDateTime( FileName As ASCIIZ, fSysTime As SYSTEMTIME, ByVal tMode As Long ) EXPORT As Long

    Dim hFile       As Long
    Dim lpFileTime1 As FILETIME
    Dim lpFileTime2 As FILETIME

    If Clng( VarPtr( FileName ) ) = 0 Then Exit Function

    ErrClear
    hFile = GetAttr( FileName )
    If ErrClear Then Exit Function

    hFile = 0
    hFile = CreateFile( FileName, _
        %GENERIC_READ Or %GENERIC_WRITE _
        , %FILE_SHARE_READ Or %FILE_SHARE_WRITE _
        , ByVal 0& _
        , %OPEN_EXISTING _
        , %FILE_FLAG_BACKUP_SEMANTICS _

        , 0 _
        )

    If hFile <> %INVALID_HANDLE_VALUE Then

        Call SystemTimeToFileTime   ( fSysTime, lpFileTime1 )
        Call LocalFileTimeToFileTime( lpFileTime1, lpFileTime2 )

        Select Case tMode
        Case 0: FUNCTION = SetFileTime( hFile, ByVal 0&, ByVal 0&, lpFileTime2 )
        Case 1: FUNCTION = SetFileTime( hFile, ByVal 0&, lpFileTime2, ByVal 0& )
        Case 2: FUNCTION = SetFileTime( hFile, lpFileTime2, ByVal 0&, ByVal 0& )
        End Select

        Call CloseHandle( hFile )

    End If

End Function

Works fine for the LastWrite and LastAccess stamps but not for the CreationTime on most everything, EXCEPT for a moved/copied/deleted/renamed and re-created file in the same directory.

At the bottom of the MS article is a list of OS file systems for which this behavior is inherent (by design).

[Michael Mattias]

Well, OK, we have some code now....

I'm not sure I like the use of FILE_FLAG_BACKUP_SEMANTICS.

FILE_FLAG_BACKUP_SEMANTICS Windows NT/2000/XP: Indicates that the file is being opened or created for a backup or restore operation. The system ensures that the calling process overrides file security checks, provided it has the SE_BACKUP_NAME and SE_RESTORE_NAME privileges. For more information, see Changing Privileges in a Token.
You can also set this flag to obtain a handle to a directory. Where indicated, a directory handle can be passed to some functions in place of a file handle.

I wonder if it's preserving the CreateDate because of that? You can try NULL for flags... that's where I'd start anyway.

I'd try it but I have to go get a haircut right now.

MCM

[John Montenigro]

While you were out getting a haircut, I shoveled 6" snow off the driveway and front walk...

Once you try the procedure outlined above and see the results yourself, your speculations are more likely to return actual results...

Here are a few more routines I tried from various authors on the forums. I started with them "as found", you may have to edit off some "export" statements and such, but at first I did not edit the substance, except to use CreationTime instead of LastWriteTime in the proper parameter position.

In later tests, I did try changing the PB OPEN to API CreateFile and FindFirstFile, to ensure that I was getting a valid handle, and other such variations. I used simple calls using just FILETIME structures, and more complex variations using WIN32_FIND_DATA structures. Oh yeah, I had also tried PB's DIR..TO with a DIRDATA structure...

Regardless of the code used, the results seem to be the same, and seem to be determined not by application code but by the OS (which I think is the "file tunneling" feature). THAT was the mystery!

The greater mystery is how to get around it without waiting the timeout period for each filename in a list, and without messing with the Registry and then restoring it...(would that even be an acceptable design strategy?)
-jhm

Function SetFileDateandTime(ccsFileSpec As String,ft As SYSTEMTIME)Export As Long   'common32.bas
Local hFile     As Long
Local Result    As Long
Local ccsFileTime  As FILETIME
Local fLocTime  As FILETIME

hFile = CreateFile(ccsFileSpec + Chr$(0), ByVal %GENERIC_READ Or %GENERIC_WRITE, _
                    ByVal 0, ByVal %NULL, ByVal %OPEN_ALWAYS, _
                    ByVal %FILE_ATTRIBUTE_NORMAL, ByVal %NULL)

' convert system date/time to file structure.
SystemTimeToFileTime ft, fLocTime
' convert local file time to UTC file time
' setfiletime expects a time that is relative to utc time.
LocalFileTimeToFileTime fLocTime, ccsFileTime
' set the file time
Result = SetFileTime(ByVal hFile, ByVal %NULL, ByVal %NULL, ccsFileTime)
CloseHandle hFile
Function = Result
End Function

' SetFileDate - Set the time and date of a file
'Scott T?
SUB SetFileDate(sFileName AS STRING, _
                iYear     AS LONG, iMonth  AS LONG, iDay    AS LONG, _
                iHour     AS LONG, iMinute AS LONG, iSecond AS LONG) EXPORT

    LOCAL nFile          AS LONG
    LOCAL hFile          AS DWORD
    LOCAL CreationTime   AS FILETIME
    LOCAL LastAccessTime AS FILETIME
    LOCAL LastWriteTime  AS FILETIME
    LOCAL ST             AS SYSTEMTIME
    LOCAL LT             AS FILETIME
    LOCAL lpReOpenBuff   AS OFSTRUCT

    TRY
        nFile = FREEFILE
        OPEN sFileName FOR INPUT ACCESS READ LOCK SHARED AS #nFile
    CATCH
        EXIT SUB
    FINALLY
        hFile = FILEATTR(nFile,2)
        GetFileTime hFile, CreationTime, LastAccessTime, LastWriteTime
        CLOSE #nFile
        FileTimeToLocalFileTime LastWriteTime, LT
        FileTimeToSystemTime LT, ST
        ST.wYear   = iYear
        ST.wMonth  = iMonth
        ST.wDay    = iDay
        ST.wHour   = iHour
        ST.wMinute = iMinute
        ST.wSecond = iSecond
        SystemTimeToFileTime ST, LT
        LocalFileTimeToFileTime LT, LastWriteTime
        SetFileTime hFile, CreationTime, LastAccessTime, LastWriteTime
    END TRY

END SUB

SUB SetDateAndTime(BYVAL sFilename AS STRING, BYVAL sfiletimeinfo AS STRING )  'Pierre B?
 LOCAL CreationTime   AS FILETIME
 LOCAL LastAccessTime AS FILETIME
 LOCAL LastWriteTime  AS FILETIME
 LOCAL hFile          AS DWORD
 LOCAL FileNum        AS DWORD
 
 IF LEN(sfiletimeinfo) <> 24 THEN EXIT SUB 'rem bad date and time returned
 
 TRY
   'OPEN sFilename FOR INPUT ACCESS READ LOCK SHARED AS #FileNum 'See next line
   OPEN sFilename FOR BINARY ACCESS WRITE AS #FileNum 'Need write access
   hFile = FILEATTR(#FileNum, 2) 'This is the handle needed for GetFileTime/SetFileTime
                                 '2 = The operating system file handle for the file
 CATCH
   EXIT SUB
 
 FINALLY
   'Use TYPE SET to copy back from a string
   TYPE SET CreationTime   = MID$(sfiletimeinfo, 1, 8)
   TYPE SET LastAccessTime = MID$(sfiletimeinfo, 9, 8)
   TYPE SET LastWriteTime  = MID$(sfiletimeinfo, 17, 8)
 
   SetFileTime hFile, CreationTime, LastAccessTime, LastWriteTime
 
   CLOSE #FileNum
 
 END TRY
 
END SUB

[Michael Mattias]

**** barber closed up and took off. Drove about 20 miles round trip thru the snow for nothing. Because of all the snow (we got about 12"), I was so intent on the road that I missed "the moment" when my odometer finally turned over to 100,000 miles.... got back and lo and behold it read 100005.

I really don't think the vehicle will be around to hit 200,000... I bought it new in '96. At this rate it would not hit 200,000 until sometime in 2021... come to think of it, I''ll be doing well to be around myself in 2021.

MCM

[John Montenigro]

Not to worry - you're about 22 messages away from your forum odometer clicking over a "big one"!!

[Vidar Hanto]

So, it seems like this is a variation over the old cache bug problem, then. And a bug report to MS won't do, they already know about it and according to John's description above, it is apparently by design. So it's not a bug, even.

The question remains then, how to cope with it?
Off the top of my head, I see three alternatives:
1) Don't bother. Will do for many (most?) purposes, but not all.
2) Make sure your own SW is immune to the problem. This is probably the only universal solution (you can't modify the client's PC can you? Well - you shouldn't), but doesn't cover 3rd party SW.
3) Get a clever workaround that also covers 3rd party SW, but this can only be applied to PCs you are in control of. According to the above, this workaround seems to be to outwait the cache.

Which brings me to the only solution I have applied to such cases with universal success: Disable the cache completely. Anything else have just changed the face of the problem.

I can't test anything now, but disabling this cache may or may not hurt performance. There are so much caching all over the place nowadays, so it may go unnoticed. Maybe the harddrive's HW cache is sufficient all by itself? I would suspect (a guess, not tested!) that this only applies to local drives as file access over networks would rely on the server's local file handling.


Two examples from (ancient?) history:
Remember the "blue screen of death" in Windows 3.0/3.1? That was due to SMARTDRV.SYS doing its default caching when DOS 5.0 hit the streets. Disable it, and there were no problem. Doing so did hurt performance a bit, but not nearly as much as the blue screen did ...

Remember Novell DOS drivers? Version 2.something didn't write to server disk data sets that was smaller than the driver's cache. A disaster for database transactions. There were corrupt databases all over the place. Again, disabling the cache was the solution. This time, the performance hit wasn't seen - it had to be measured.

ViH

[Elias Montoya]

You just got me thinking Michael... I dont like the feeling that someday i will know my life is getting close to an end.

Good thing that doesnt have to happen. If i overwritte myself with a BAT extension my age might change back to 22.

[John Montenigro]

Vidar,

I have not found any information about a "file tunneling" cache that could be disabled. I doubt that I have uncovered everything there is to know about it, but so far, I've seen nothing - except to disable the timeout.

I've never done anything with Registry settings, so I'm a few days away from being able to test it, AND I don't know if this would be a good workaround:

1. Capture the current registry timeout setting,
2. change it to 0,
3. process the list of files,
4. change the registry timeout back to its original setting.

I don't like this because the application has no knowledge if the timeout change would adversely affect other operations occurring on the drive. For example, if there were another process that depended on the timeout for recovery purposes, this strategy would adversely interfere.


Another possible strategy:

1. capture the filenames and their CreationTime to a file,
2. allow the timeout to expire,
3. read the file and apply the saved timestamps.

This is less invasive of the user's environment, but it would require the app to go "on hold" for awhile, and the user may wonder why it didn't "complete" such a "simple" operation more quickly... (User's wonder could be averted if the app provided sufficient messages to explain what's going to happen, what's happening, and what happened...)


So, I'm still thinking about alternative workarounds.

Any other ideas?

[Michael Mattias]

Save the creation dates of interest in your own disk file or database. Change when wanted. Always get creation dates for 'files of interest' from your own storage, ignoring what Windows says it is.

[John Montenigro]

Hmmmm...

In doing further research, I found this page:
http://msdn.microsoft.com/en-us/libr...12(VS.85).aspx

Excerpt (I added bolding and italic for emphasis)

The ReplaceFile function combines several steps within a single function. An application can call ReplaceFile instead of calling separate functions to save the data to a new file, rename the original file using a temporary name, rename the new file to have the same name as the original file, and delete the original file. Another advantage is that ReplaceFile not only copies the new file data, but also preserves the following attributes of the original file:

Creation time
Short file name
Object identifier
DACLs
Encryption
Compression
Named streams not already in the replacement file

For example, if the replacement file is encrypted, but the replaced file is not encrypted, the resulting file is not encrypted.

The resulting file has the same file ID as the replacement file.

The backup file, replaced file, and replacement file must all reside on the same volume.

What caught my attention is that the ReplaceFile function preserves the CreationTime of the original file!!!
<SMALLPARADE> Yea! </SMALLPARADE>

The article indicates that WinBase.h is the appropriate header file. I got WinBase.inc with PBWIN9, but to avoid having to edit out all the definitions duplicated in Win3API.inc, I've just copied the relevant DECLARE and flags into my code...

The article also indicates that Windows 2000 Professional is the minimum client, and although I'm running XP Pro SP3, it does compile, run, and is returning errors about permissions - but they're appropriate to the situation, which verifies that the ReplaceFile function is available and operating...

SO... I've been playing with it a bit. I was trying to trick it by supplying the same file as both ReplaceFile and ReplacementFile - it didn't like that.

So, a workaround... I will supply my original SourceFile as the ReplacementFile and a copy of it as the ReplaceFile, expecting that I'll get the file I want as the BackupFile...

Yes this will be slower than a "normal" process because I'll have to make a copy of each file first and delete the copy afterwards, but if this forces the OS to give me the file I need (with intact CreationTime stamp), AND if it eliminates my having to mess with the timeout period or its Registry settings, then I'll consider it a success...

We shall see...

I'll update this post in a day or so... (maybe I can persuade Santa to give me some time this week) Merry Happy everything!

[Michael Mattias]

For example, if the replacement file is encrypted, but the replaced file is not encrypted, the resulting file is not encrypted

Hmm...nah, it could not be that simple to decrypt a file, could it?

[John Montenigro]

I don't know - haven't tried that yet... might just come out as junk...

But it would be interesting to see what it actually does.

[Michael Mattias]

I doubt it will decrypt, but if I ever saw something that could have been missed in testing.....

[John Montenigro]

John,

Are you moving or copying a file?

If moving - not copying - you can use the NAME command to move a file.

If copying - then you would need to use what Michael suggested and ensure FileTime is preserved yourself.

EDIT: Scratch that suggestion - apparently moving by using the NAME command also resets the creation date/time stamp. You will need to ensure file date/time stamps are preserved on your own if you are going to depend on them...

Adam,

I found that NAME actually can be used to preserve the CreationTime, IF you reverse the process!!!!

Name SourceFile As DestinationFile    ' << make backup FIRST, by MOVing original
FileCopy DestinationFile, SourceFile  ' << restore original from backup copy

This way, when the OS preserves the CreationTime from the DELETED original and imposes it on the new copy, the result is precisely what was wanted -- both files are identical, right down to their CreationTime stamps!!!

Not one API call to SetFileTime!!! No intermediate variables!!! No messing with the Registry!!! No messing with "file tunneling" delay times!!!

Clean and simple!!! (though quite counter-intuitive...)

[Eric von Bostel]

... and then there is the folder date

can the same method be applied to a folder date?
sometimes I do Q&D (quick and dirty) backups of an entire folder.
I delete an entire folder from my backup disk,
then copy the same folder from workdisk to backup
a month or so later I want to manually check when I made the last backup.
instead of seeing a date of eg: 19 Sept 2008, I find 14 May 2002.