Announcement

Collapse
No announcement yet.

File CreationTime mystery

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Michael Mattias
    replied
    Q&D (quick and dirty) backups.. I delete an entire folder from my backup disk,
    then copy the same folder from workdisk to backup..so later I want to manually check when I made the last backup.
    Code:
      Source$ = "D:\data"
      BackupDrive$ =  "L:"
      Call DeleteSubfolders (BackupDrive$) 
      Dest$ = BackupDrive$ & "\" & DATE$
      MKDIR  Dest$
      CALL Backup (Source$, Dest$)
    MCM

    Leave a comment:


  • Eric von Bostel
    replied
    ... and then there is the folder date

    can the same method be applied to a folder date?
    sometimes I do Q&D (quick and dirty) backups of an entire folder.
    I delete an entire folder from my backup disk,
    then copy the same folder from workdisk to backup
    a month or so later I want to manually check when I made the last backup.
    instead of seeing a date of eg: 19 Sept 2008, I find 14 May 2002.

    Leave a comment:


  • John Montenigro
    replied
    Originally posted by Adam J. Drake View Post
    John,

    Are you moving or copying a file?

    If moving - not copying - you can use the NAME command to move a file.

    If copying - then you would need to use what Michael suggested and ensure FileTime is preserved yourself.

    EDIT: Scratch that suggestion - apparently moving by using the NAME command also resets the creation date/time stamp. You will need to ensure file date/time stamps are preserved on your own if you are going to depend on them...
    Adam,

    I found that NAME actually can be used to preserve the CreationTime, IF you reverse the process!!!!

    Code:
    Name SourceFile As DestinationFile    ' << make backup FIRST, by MOVing original
    FileCopy DestinationFile, SourceFile  ' << restore original from backup copy
    This way, when the OS preserves the CreationTime from the DELETED original and imposes it on the new copy, the result is precisely what was wanted -- both files are identical, right down to their CreationTime stamps!!!

    Not one API call to SetFileTime!!! No intermediate variables!!! No messing with the Registry!!! No messing with "file tunneling" delay times!!!

    Clean and simple!!! (though quite counter-intuitive...)

    Leave a comment:


  • Michael Mattias
    replied
    I doubt it will decrypt, but if I ever saw something that could have been missed in testing.....

    Leave a comment:


  • John Montenigro
    replied
    I don't know - haven't tried that yet... might just come out as junk...

    But it would be interesting to see what it actually does.

    Leave a comment:


  • Michael Mattias
    replied
    For example, if the replacement file is encrypted, but the replaced file is not encrypted, the resulting file is not encrypted
    Hmm...nah, it could not be that simple to decrypt a file, could it?

    Leave a comment:


  • John Montenigro
    replied
    Hmmmm...

    In doing further research, I found this page:
    http://msdn.microsoft.com/en-us/libr...12(VS.85).aspx

    Excerpt (I added bolding and italic for emphasis)

    The ReplaceFile function combines several steps within a single function. An application can call ReplaceFile instead of calling separate functions to save the data to a new file, rename the original file using a temporary name, rename the new file to have the same name as the original file, and delete the original file. Another advantage is that ReplaceFile not only copies the new file data, but also preserves the following attributes of the original file:

    Creation time
    Short file name
    Object identifier
    DACLs
    Encryption
    Compression
    Named streams not already in the replacement file

    For example, if the replacement file is encrypted, but the replaced file is not encrypted, the resulting file is not encrypted.

    The resulting file has the same file ID as the replacement file.

    The backup file, replaced file, and replacement file must all reside on the same volume.
    What caught my attention is that the ReplaceFile function preserves the CreationTime of the original file!!!
    <SMALLPARADE> Yea! </SMALLPARADE>

    The article indicates that WinBase.h is the appropriate header file. I got WinBase.inc with PBWIN9, but to avoid having to edit out all the definitions duplicated in Win3API.inc, I've just copied the relevant DECLARE and flags into my code...

    The article also indicates that Windows 2000 Professional is the minimum client, and although I'm running XP Pro SP3, it does compile, run, and is returning errors about permissions - but they're appropriate to the situation, which verifies that the ReplaceFile function is available and operating...

    SO... I've been playing with it a bit. I was trying to trick it by supplying the same file as both ReplaceFile and ReplacementFile - it didn't like that.

    So, a workaround... I will supply my original SourceFile as the ReplacementFile and a copy of it as the ReplaceFile, expecting that I'll get the file I want as the BackupFile...

    Yes this will be slower than a "normal" process because I'll have to make a copy of each file first and delete the copy afterwards, but if this forces the OS to give me the file I need (with intact CreationTime stamp), AND if it eliminates my having to mess with the timeout period or its Registry settings, then I'll consider it a success...

    We shall see...

    I'll update this post in a day or so... (maybe I can persuade Santa to give me some time this week) Merry Happy everything!

    Leave a comment:


  • Michael Mattias
    replied
    Save the creation dates of interest in your own disk file or database. Change when wanted. Always get creation dates for 'files of interest' from your own storage, ignoring what Windows says it is.

    Leave a comment:


  • John Montenigro
    replied
    Vidar,

    I have not found any information about a "file tunneling" cache that could be disabled. I doubt that I have uncovered everything there is to know about it, but so far, I've seen nothing - except to disable the timeout.

    I've never done anything with Registry settings, so I'm a few days away from being able to test it, AND I don't know if this would be a good workaround:

    1. Capture the current registry timeout setting,
    2. change it to 0,
    3. process the list of files,
    4. change the registry timeout back to its original setting.

    I don't like this because the application has no knowledge if the timeout change would adversely affect other operations occurring on the drive. For example, if there were another process that depended on the timeout for recovery purposes, this strategy would adversely interfere.


    Another possible strategy:

    1. capture the filenames and their CreationTime to a file,
    2. allow the timeout to expire,
    3. read the file and apply the saved timestamps.

    This is less invasive of the user's environment, but it would require the app to go "on hold" for awhile, and the user may wonder why it didn't "complete" such a "simple" operation more quickly... (User's wonder could be averted if the app provided sufficient messages to explain what's going to happen, what's happening, and what happened...)


    So, I'm still thinking about alternative workarounds.

    Any other ideas?
    Last edited by John Montenigro; 20 Dec 2008, 08:53 AM.

    Leave a comment:


  • Elias Montoya
    replied
    You just got me thinking Michael... I dont like the feeling that someday i will know my life is getting close to an end.

    Good thing that doesnt have to happen. If i overwritte myself with a BAT extension my age might change back to 22.

    Leave a comment:


  • Vidar Hanto
    replied
    So, it seems like this is a variation over the old cache bug problem, then. And a bug report to MS won't do, they already know about it and according to John's description above, it is apparently by design. So it's not a bug, even.

    The question remains then, how to cope with it?
    Off the top of my head, I see three alternatives:
    1) Don't bother. Will do for many (most?) purposes, but not all.
    2) Make sure your own SW is immune to the problem. This is probably the only universal solution (you can't modify the client's PC can you? Well - you shouldn't), but doesn't cover 3rd party SW.
    3) Get a clever workaround that also covers 3rd party SW, but this can only be applied to PCs you are in control of. According to the above, this workaround seems to be to outwait the cache.

    Which brings me to the only solution I have applied to such cases with universal success: Disable the cache completely. Anything else have just changed the face of the problem.

    I can't test anything now, but disabling this cache may or may not hurt performance. There are so much caching all over the place nowadays, so it may go unnoticed. Maybe the harddrive's HW cache is sufficient all by itself? I would suspect (a guess, not tested!) that this only applies to local drives as file access over networks would rely on the server's local file handling.


    Two examples from (ancient?) history:
    Remember the "blue screen of death" in Windows 3.0/3.1? That was due to SMARTDRV.SYS doing its default caching when DOS 5.0 hit the streets. Disable it, and there were no problem. Doing so did hurt performance a bit, but not nearly as much as the blue screen did ...

    Remember Novell DOS drivers? Version 2.something didn't write to server disk data sets that was smaller than the driver's cache. A disaster for database transactions. There were corrupt databases all over the place. Again, disabling the cache was the solution. This time, the performance hit wasn't seen - it had to be measured.

    ViH

    Leave a comment:


  • John Montenigro
    replied
    Not to worry - you're about 22 messages away from your forum odometer clicking over a "big one"!!

    Leave a comment:


  • Michael Mattias
    replied
    **** barber closed up and took off. Drove about 20 miles round trip thru the snow for nothing. Because of all the snow (we got about 12"), I was so intent on the road that I missed "the moment" when my odometer finally turned over to 100,000 miles.... got back and lo and behold it read 100005.

    I really don't think the vehicle will be around to hit 200,000... I bought it new in '96. At this rate it would not hit 200,000 until sometime in 2021... come to think of it, I''ll be doing well to be around myself in 2021.

    MCM

    Leave a comment:


  • John Montenigro
    replied
    While you were out getting a haircut, I shoveled 6" snow off the driveway and front walk...

    Once you try the procedure outlined above and see the results yourself, your speculations are more likely to return actual results...

    Here are a few more routines I tried from various authors on the forums. I started with them "as found", you may have to edit off some "export" statements and such, but at first I did not edit the substance, except to use CreationTime instead of LastWriteTime in the proper parameter position.

    In later tests, I did try changing the PB OPEN to API CreateFile and FindFirstFile, to ensure that I was getting a valid handle, and other such variations. I used simple calls using just FILETIME structures, and more complex variations using WIN32_FIND_DATA structures. Oh yeah, I had also tried PB's DIR..TO with a DIRDATA structure...

    Regardless of the code used, the results seem to be the same, and seem to be determined not by application code but by the OS (which I think is the "file tunneling" feature). THAT was the mystery!

    The greater mystery is how to get around it without waiting the timeout period for each filename in a list, and without messing with the Registry and then restoring it...(would that even be an acceptable design strategy?)
    -jhm



    Code:
    Function SetFileDateandTime(ccsFileSpec As String,ft As SYSTEMTIME)Export As Long   'common32.bas
    Local hFile     As Long
    Local Result    As Long
    Local ccsFileTime  As FILETIME
    Local fLocTime  As FILETIME
    
    hFile = CreateFile(ccsFileSpec + Chr$(0), ByVal %GENERIC_READ Or %GENERIC_WRITE, _
                        ByVal 0, ByVal %NULL, ByVal %OPEN_ALWAYS, _
                        ByVal %FILE_ATTRIBUTE_NORMAL, ByVal %NULL)
    
    ' convert system date/time to file structure.
    SystemTimeToFileTime ft, fLocTime
    ' convert local file time to UTC file time
    ' setfiletime expects a time that is relative to utc time.
    LocalFileTimeToFileTime fLocTime, ccsFileTime
    ' set the file time
    Result = SetFileTime(ByVal hFile, ByVal %NULL, ByVal %NULL, ccsFileTime)
    CloseHandle hFile
    Function = Result
    End Function
    Code:
    ' SetFileDate - Set the time and date of a file
    'Scott T?
    SUB SetFileDate(sFileName AS STRING, _
                    iYear     AS LONG, iMonth  AS LONG, iDay    AS LONG, _
                    iHour     AS LONG, iMinute AS LONG, iSecond AS LONG) EXPORT
    
        LOCAL nFile          AS LONG
        LOCAL hFile          AS DWORD
        LOCAL CreationTime   AS FILETIME
        LOCAL LastAccessTime AS FILETIME
        LOCAL LastWriteTime  AS FILETIME
        LOCAL ST             AS SYSTEMTIME
        LOCAL LT             AS FILETIME
        LOCAL lpReOpenBuff   AS OFSTRUCT
    
        TRY
            nFile = FREEFILE
            OPEN sFileName FOR INPUT ACCESS READ LOCK SHARED AS #nFile
        CATCH
            EXIT SUB
        FINALLY
            hFile = FILEATTR(nFile,2)
            GetFileTime hFile, CreationTime, LastAccessTime, LastWriteTime
            CLOSE #nFile
            FileTimeToLocalFileTime LastWriteTime, LT
            FileTimeToSystemTime LT, ST
            ST.wYear   = iYear
            ST.wMonth  = iMonth
            ST.wDay    = iDay
            ST.wHour   = iHour
            ST.wMinute = iMinute
            ST.wSecond = iSecond
            SystemTimeToFileTime ST, LT
            LocalFileTimeToFileTime LT, LastWriteTime
            SetFileTime hFile, CreationTime, LastAccessTime, LastWriteTime
        END TRY
    
    END SUB
    Code:
    SUB SetDateAndTime(BYVAL sFilename AS STRING, BYVAL sfiletimeinfo AS STRING )  'Pierre B?
     LOCAL CreationTime   AS FILETIME
     LOCAL LastAccessTime AS FILETIME
     LOCAL LastWriteTime  AS FILETIME
     LOCAL hFile          AS DWORD
     LOCAL FileNum        AS DWORD
     
     IF LEN(sfiletimeinfo) <> 24 THEN EXIT SUB 'rem bad date and time returned
     
     TRY
       'OPEN sFilename FOR INPUT ACCESS READ LOCK SHARED AS #FileNum 'See next line
       OPEN sFilename FOR BINARY ACCESS WRITE AS #FileNum 'Need write access
       hFile = FILEATTR(#FileNum, 2) 'This is the handle needed for GetFileTime/SetFileTime
                                     '2 = The operating system file handle for the file
     CATCH
       EXIT SUB
     
     FINALLY
       'Use TYPE SET to copy back from a string
       TYPE SET CreationTime   = MID$(sfiletimeinfo, 1, 8)
       TYPE SET LastAccessTime = MID$(sfiletimeinfo, 9, 8)
       TYPE SET LastWriteTime  = MID$(sfiletimeinfo, 17, 8)
     
       SetFileTime hFile, CreationTime, LastAccessTime, LastWriteTime
     
       CLOSE #FileNum
     
     END TRY
     
    END SUB
    Last edited by John Montenigro; 19 Dec 2008, 04:07 PM.

    Leave a comment:


  • Michael Mattias
    replied
    Well, OK, we have some code now....

    I'm not sure I like the use of FILE_FLAG_BACKUP_SEMANTICS.
    FILE_FLAG_BACKUP_SEMANTICS Windows NT/2000/XP: Indicates that the file is being opened or created for a backup or restore operation. The system ensures that the calling process overrides file security checks, provided it has the SE_BACKUP_NAME and SE_RESTORE_NAME privileges. For more information, see Changing Privileges in a Token.
    You can also set this flag to obtain a handle to a directory. Where indicated, a directory handle can be passed to some functions in place of a file handle.
    I wonder if it's preserving the CreateDate because of that? You can try NULL for flags... that's where I'd start anyway.

    I'd try it but I have to go get a haircut right now.

    MCM

    Leave a comment:


  • John Montenigro
    replied
    MCM,

    While your pseudocode looks reasonable, it doesn't work. I've used at least 5 variations of SetFileTime() routines found on the forums. (That's why the instructions directed the use of a non-code tool...)

    Here's just one:


    Code:
    '0 = LastWriteTime (visible in dos)
    '1 = LastAccessTime
    '2 = CreationTime
    Function SetFileDateTime( FileName As ASCIIZ, fSysTime As SYSTEMTIME, ByVal tMode As Long ) EXPORT As Long
    
        Dim hFile       As Long
        Dim lpFileTime1 As FILETIME
        Dim lpFileTime2 As FILETIME
    
        If Clng( VarPtr( FileName ) ) = 0 Then Exit Function
    
        ErrClear
        hFile = GetAttr( FileName )
        If ErrClear Then Exit Function
    
        hFile = 0
        hFile = CreateFile( FileName, _
            %GENERIC_READ Or %GENERIC_WRITE _
            , %FILE_SHARE_READ Or %FILE_SHARE_WRITE _
            , ByVal 0& _
            , %OPEN_EXISTING _
            , %FILE_FLAG_BACKUP_SEMANTICS _
    
            , 0 _
            )
    
        If hFile <> %INVALID_HANDLE_VALUE Then
    
            Call SystemTimeToFileTime   ( fSysTime, lpFileTime1 )
            Call LocalFileTimeToFileTime( lpFileTime1, lpFileTime2 )
    
            Select Case tMode
            Case 0: FUNCTION = SetFileTime( hFile, ByVal 0&, ByVal 0&, lpFileTime2 )
            Case 1: FUNCTION = SetFileTime( hFile, ByVal 0&, lpFileTime2, ByVal 0& )
            Case 2: FUNCTION = SetFileTime( hFile, lpFileTime2, ByVal 0&, ByVal 0& )
            End Select
    
            Call CloseHandle( hFile )
    
        End If
    
    End Function
    Works fine for the LastWrite and LastAccess stamps but not for the CreationTime on most everything, EXCEPT for a moved/copied/deleted/renamed and re-created file in the same directory.

    At the bottom of the MS article is a list of OS file systems for which this behavior is inherent (by design).
    Last edited by John Montenigro; 19 Dec 2008, 02:07 PM.

    Leave a comment:


  • Michael Mattias
    replied
    Code in Post #1 does not show using SetFileTime() function.

    It shows using PowerbASIC filecopy function. Or maybe it's the Windows FileCopy function, the PB function being CopyFile.

    In any event the code to try is ..

    Code:
       TS1 =  GetFileTimeOfInterest (SourceFile) 
       Copy   SourceFile to Target file 
       SetFileTime (of interest)  of Target file = TS1

    Leave a comment:


  • John Montenigro
    replied
    Originally posted by Michael Mattias View Post
    >and I DO do it myself, and the change DOES NOT HAPPEN as intended.

    Failing code not shown.
    Code was posted with instructions; please refer to post #1 in this thread.

    Leave a comment:


  • John Montenigro
    replied
    Sorry for the delay - two court days and other business meetings, and it's only due to a snowstorm that I'm sitting here now. Otherwise, I wouldn't have gotten back till Monday...

    Vidar, thanks for confirming the odd behavior. If I may point out that you mention waiting between changes... that's part of it... But of course, if you were restamping a list of filenames under program control, you wouldn't want to wait for 30 or more seconds between each operation...

    Others have run into this odd scenario:

    http://www.eggheadcafe.com/software/...changed-d.aspx

    http://www.tech-archive.net/Archive/...4-06/5191.html


    I've contacted authors of other apps that manage to accomplish what I set out to do, but I haven't heard back yet on how they get around it...


    SO! While I'm waiting for that info, let me shed some light on the nature of the problem itself.

    It's called "file tunneling".

    http://support.microsoft.com/?scid=k...2190&x=12&y=13
    http://support.microsoft.com/default...en-us%3B172190

    An excerpt from the above link:
    The Microsoft Windowsproducts listed at the beginning of this artice contain file system tunneling capabilities to enable compatibility with programs that rely on file systems being able to hold onto file meta-info for a short period of time. This occurs after deletion or renaming and re-introducing a new directory entry with that meta-info (if a create or rename occurs to cause a file of that name to appear again in a short period of time).

    When a name is removed from a directory (rename or delete), its short/long name pair and creation time are saved in a cache, keyed by the name that was removed. When a name is added to a directory (rename or create), the cache is searched to see if there is information to restore. The cache is effective per instance of a directory. If a directory is deleted, the cache for it is removed.

    These paired operations can cause tunneling on "name."

    delete(name)/create(name)

    delete(name)/rename(source, name)

    rename(name, newname)/create(name)

    rename(name, newname)/rename(source, name)

    The idea is to mimic the behavior MS-DOS programs expect when they use the safe save method. They copy the modified data to a temporary file, delete the original and rename the temporary to the original. This should seem to be the original file when complete. Windows performs tunneling on both FAT and NTFS file systems to ensure long/short file names are retained when 16-bit applications perform this safe save operation.

    That's just the beginning... lots more to read...

    What's the significance? Well, for anyone who still thinks about file date/time stamps as if they were simple items in the file's header (DTA?), then forget it! This meta-data scheme makes the dates one step removed from "actual" file data. And if you are trying to manage file date/time stamps, you'll need to find some way around it.

    Yes, as soon as I find something reliable that does not require shortening the wait time in the Registry settings, I'll post it...

    I hope you had some fun with this - always another thing to learn!

    Leave a comment:


  • Vidar Hanto
    replied
    Back in DOS/FAT there were only the last-modified-date. When copying files, the new file would get the current time. If you needed to keep the timestamp, your tool had to read the old stamp and transfer it to the target. Nothing new there. But everything did that, so that was never an issue.

    DST/NTFS as Garstang describes it is inaccurate. NTFS always store file dates using UTF/GMT and the creator's timezone is dropped. And it is correct that Windows Explorer seems to be inconsistent when displaying file dates (but I have never seen anything else do it differently either). Everybody seems to simply convert UTF using current DST info with no regard to whether the file was CREATED, MODIFIED or LAST ACCESSED inside or outside e.g. DST! But that is not an NTFS error! That is an error on the display-tool's hand. Information lost in translation, so to say.

    However, it is debatable whether it is correct to keep tabs of such considerations too - it can be said that Windows Explorer's behaviour is correct. The impression is merely cosmetic. There is no one-answer-only here, but I would defend Explorer's behaviour:

    E.g. imagine a file created in France at 15:00 hours (military time for you US citizens). This being local time, Friday 13th some month and year. Would it be correct to assume that when the file was compared to a file created in New York at 11:00 AM (= 11:00 for you Europeans ) still at the same Friday 13th - that the French file is older? Or is it the newer one? It is after all created later in the day ... UTF says otherwise: 15:00 in France is 14:00 UTF and 11:00 in New York is 17:00 UTF. If one says that the two files are transactions recorded for the same database? Should the US transactions get precedence? I don't think so. So, clearly, timezones are out of the equation. Now, should DST be brought into the equation? It can't. Same thing applies, but in a minor scale.

    ----

    However, I am sidestepping:
    I for one do not think the problem at hand is something to be disregarded. Unless John has a good solution.

    I've had enough problems over the years with failing caching algorithms not to care, and this immediately looked like a variation over the same theme.

    I ran the test myself, and I can confirm it using XP SP3. Anyone with other OS-es? (You don't need to patch the timestamps yourself. Simply wait one minute between each repeated attempt).

    And it is uglier: First I tested it using the same source and target file to confirm. Then I used another, (much) larger source file. The correct, larger amount of data arrived at the target, but the old created timestamp was kept! The other two dates (modified and last access) were correctly set.

    To me, this seems to be worhty of a bug report to Microsoft. Does anyone know a workaround yet? Or something else that occurs due to the same(?) error? Or?

    ViH
    Last edited by Vidar Hanto; 16 Dec 2008, 10:12 PM. Reason: Print error

    Leave a comment:

Working...
X