FileWatcher Discussion

[Larry Charlton]

Discussion for the FileWatcher class found here.

I found myself running a lot of unit tests recently and the editor I was using to view the results would only load changes when it was focused. Probabbly should have downloaded a tail utility, but thought this would likely come in useful later. It uses the ReadDirectoryChangesW API call to monitor for changes. This is my first attempt at using these API by cobbling together stuff from the documentation and found around the net. Please let me know if you see anything that needs correcting.

[Gary Beene]

Hey Larry,
Thanks for the code. I've been interested in using the API for monitoring file/folder changes but haven't gotten around to doing anything with it.

The simple code below is an example of what I've used to monitor changes to a single file. It's fairly minimal, and uses a simple timer rather than the API you're using, but does the job I want. Can you talk to why, for monitoring a single file, the API would be more beneficial?

I'm more interested in using the API to monitor all file and folder changes, so that I could write an equivalent to the Everything application. Have you done any work with the API on monitoring any and all file/folder changes? Eventually I'll find time to look into it, so what you are doing is appreciated!

'Compilable Example:
#Compiler PBWin 10
#Compile Exe
#Dim All
%Unicode = 1
#Include "Win32API.inc"
%IDC_TextBox = 500
%ID_Timer    = 501
Global hDlg As Dword

Function PBMain() As Long
   Dialog New Pixels, 0, "PowerBASIC",300,300,200,80, %WS_OverlappedWindow To hDlg
   Control Add TextBox, hDlg, %IDC_TextBox,"c:\_test\x.bas", 10,10,180,20
   Dialog Show Modal hDlg Call DlgProc
End Function

CallBack Function DlgProc() As Long
   Static oldFile, tempFile As DirData, temp$
   Select Case Cb.Msg
      Case %WM_InitDialog
         SetTimer hDlg, %ID_Timer, 250, ByVal %Null
         Control Get Text hDlg, %IDC_TextBox To temp$
         temp$ = Dir$(temp$, 0, To oldFile)
      Case %WM_Timer
         Dialog Set Text hDlg, Time$
         Control Get Text hDlg, %IDC_TextBox To temp$
         temp$ = Dir$(temp$, 0, To tempFile)
         If tempFile.LastWriteTime <> OldFile.LastWriteTime Then
            KillTimer hDlg, %ID_Timer
            ? "File changed!"
         End If
      Case %WM_Destroy
         KillTimer hDlg, %ID_Timer
   End Select
End Function

[Michael Mattias]

>Can you talk to why, for monitoring a single file, the API would be more beneficial?

Less resources required. Why expend the resources needed to check a file on each timer hit when you can wait for something to actually change?

[Gary Beene]

Hi MCM,
Re:

Why expend the resources ...

Simplicity of the approach is the first thought that comes to mind, but your question raises a question I've never answered for myself nor seen quantified.

Just how much of a resource hog is a timer? I seem to remember doing a test one time - pulled up Task Manager and looked at CPU usage after starting a timer. As I recall, I could see no visual change in the graph.

... pause for new test ...

I made a short app that creates 1,000 timers, but I can't see any impact in Task Manager. I've read elsewhere that timer are very lightweight resources, but haven't found an upper limit mentioned online or on MSDN.

Can you suggest a more appropriate test to quantify resource use of timers?

Here's the test code I just used:

'Compilable Example:
#Compiler PBWin 10
#Compile Exe
#Dim All
#Debug Error On
#Debug Display On
%Unicode = 1
#Include "Win32API.inc"
Global hDlg As Dword

Function PBMain() As Long
   Dialog New Pixels, 0, "PowerBASIC",300,300,200,80, %WS_OverlappedWindow To hDlg
   Dialog Show Modal hDlg Call DlgProc
End Function

CallBack Function DlgProc() As Long
   Local i As Long
   Select Case Cb.Msg
      Case %WM_InitDialog
         For i = 1 To 1000
            SetTimer hDlg, 1000 + i, 250, ByVal %Null
         Next i
      Case %WM_Destroy
         For i = 1 To 1000
            KillTimer hDlg, 1000 + i
         Next i
   End Select
End Function

[Gary Beene]

Hi Larry,
I went out to MSDN and looked at ReadDirectoryChangesW. The bWatchSubTree argument looks of interest. %True appears to allow the API to monitor subfolders. Cool beans.

I wonder why MSDN has tossed in the term "subtree". I don't recall having seen that anywhere before. Was there a reason they don't just say Directory?

[Michael Mattias]

I wonder why MSDN has tossed in the term "subtree". I don't recall having seen that anywhere before. Was there a reason they don't just say Directory?

Frankly, I am surprised they do not say, "Folder."

Just how much of a resource hog is a timer?

Each thread in the system is routinely given CPU time, unless the thread is in a wait state.

"How much" a good question.. I would assume the calling thread gets its normal time slice, which in this case would be used up "waiting for a message" because it's a GUI thread. (It's not only waiting for WM_TIMER, there's lots of other messages it might be getting).

When the thread is in a wait state it gets nada until released, at which time it takes its regular turn again.

I think you can guess how many style points you get for using a timer or otherwise 'polling for a condition' vs using synchronization objects and functions.

MCM
Surpreme Judge (Style Points)

[Paul Purvis]

I would like to view the testing of which process spends more resources.
For two reasons, one for the reason that i am going to need(want) effecient code that loops and has time pauses in the loop and the second reason to possible learn to view and compare efficient processing times of different sets of source code on a running bases.

I am not sure about to how to judge the cpu resources on code that already runs smoothly(yeah the processes that eat up the cpu are easy to see in task manager), but i was going to use the Kerneltime and Usertime values to see what they produced. I have read about other methods of timing, but i would like to start with these.

Larry, i am not trying to distract from the subject too much bc i feel that in testing code. We try to get it to run better after it is giving results. One never knows when the code will be run on a server or a machine that resouces need to be squeezed.

The SLEEP statement can be easily replace the timer part.
I am asking MCM to provide the WAIT method source code.
I will work on the kerneltime and usertime display which should take longer bc it is me doing it and learning while doing.

[Michael Mattias]

>I am asking MCM to provide the WAIT method source code.

Easy.

Search here for "WaitForSingleObject" and/or "WaitForMultipleObjects"

I know these posts all use one or the other ...

GUI + Worker Thread + Abort Demo 11-24-07 (DDT syntax)
Waitable Timer Object Demo June 2005
Terminate Worker Threads Using Windows Events (and Using Thread Local Storage) Demo Dec 23 2005

Probably some more, too.

Just generically speaking? Using the "Wait" functions is more efficient than polling:

DO 
   IF GlobalVar =  Quit_Value THEN 
       EXIT DO 
   ELSE
      SLEEP Sleep_Interval 
   END IF 
LOOP

First, there is a thread switch required to execute this thread - because it's NOT in a wait state.

Second, every Sleep_interval the CPU must do an integer compare. Not that an integer compare is a "long" job at all.. but it's still greater than NOTHING.

MCM

[Michael Mattias]

FWIW... an 'early' 'File watcher' .. which actually watches a directory (folder):

Win32(SDK): Internet Cookie Monitor April 25, 2001

[Larry Charlton]

Gary,

What MCM said. No resources expended unless something changes. Using the API with the provided class is pretty easy

  
  Local fw As iFileWatcher
  Call StartFileWatcher( fw, newFile, CodePtr( OnFileChanged ) )

It calls a routine you write that has the following signature whenever there's a change:

Sub OnFileChanged( filename As WString )
  ' Your code to do something when the file changes goes here.
End Sub

If you want to monitor a folder or tree, just remove some code and change one parameter. I might look into providing a variation if someone doesn't beat me to it.

In terms of efficiency, the approach presented will scale well particularly if you decide to monitor a directory or a directory tree. If you use a timer, everytime it goes off, you'll need to track something about every file in the tree and then scan the tree and compare for changes. That could get bad pretty quickly. For one file, I'm guessing it doesn't matter.

Subtree is just a shorted form of Sub directory tree refering to all of the folders from a particular location in the directory down.

There is a more performant method of using ReadDirectoryChangesW that involves a few more threads and a call back function. Didn't seem to fit easily into my view of PowerBASIC coding, maybe next time.

Edit: Oh and I updated the code to use more synchronization primitives and offload processing of the change from watching for the change. I also dealt with an issue on change if you try to use a file too quickly, you often get access denied errors...

[Larry Charlton]

I've put zip files you can download at the following locations:

FileWatcher - includes exe. Watches for writes to a single file and then reloads it.

FileSystemWatcher - includes DLL for pbCC users. Watches for selected changes to a directory or directory tree and then sends a notice. Small demos of including, using an SLL, and using a DLL.

[Nathan Maddox]

compiled using PB 10.03
Ran on Win 7 Professional Version 6.1 Build 7601:Service Pack 1)
1st browed to a file on my C: managed by Tortoise SVN
It opened the file in a "notepad like" read only window with
"Load round 1" added to the top
In a seperate window with notepad I opened the file
Edited it
Saved it
Closed Notepad

I didn't get any notification.

Closed File Watcher.
Reopened it
Browed to c:\Temp.Txt
It did the same (opened it in a read only window)
I used notepad to edit c:\Temp.Txt then saved it
I didn't get any notification...

Using Trend Micro Client/Service Security Agent ..

[Michael Mattias]

From ZIP file linked in post 11

Sub OnFileChanged( filename As WString )
	Local value, c As String
	Local i, f As Long
	Static cnt As Long

	' Reload file
	ErrClear
	f = FreeFile
	For i=1 To 10
		Open filename For Binary Access Read Lock Shared As #f
		If Err=0 Then Exit For
		ErrClear
		Sleep 100
	Next
	Get$ #f, Lof(#f), value
	Close #f
	Incr cnt
	c = Format$( cnt, "#,##0" )
	value = Build$("Load round ", c, $CrLf, value )
	Control Set Text hDlg, %CTL_TEXT, value
End Sub

This procedure has no error handler when the OPEN fails ten times (FOR.. NEXT loop completes, exits fat and happy and continues without a care in the world) .

In the whole program, there is only one thread of execution. This can/will cause the program to hang on "no changes."

Suggestion:
Run the watcher in a separate thread of execution. When change detected, post a private message to notify the GUI thread. (That will be a lot easier than using MessageWaitForMultipleObjects in your own message loop).

Demo: Waitable Timer Object Demo June 2005. (Different object for the wait handle but same principle.)

(also handle open failure after nine retries).

MCM

[Larry Charlton]

Nathan
I've added an error message if you can't load (refresh) the file. This would happen if a program kept the file open over a 1 second and it didn't at least allow at least shared reading. If the writes are slow, extending the timeout count would help.

MCM
I'm not sure I follow the single thread of execution (I'm using two PowerThreads, not thread functions). The gui is running in a thread, the file watcher is running in a thread, change notification is running in a thread.

I wait for multiple objects because either the gui (signaling program exit) or the OS (signaling file change) can wake up the watcher. Watcher kicks the notifier and immediately goes back to watching.

I've not seen the "hang" condition you mentioned. I would be interested in an event sequence that caused the hang or steps to reproduce. Thanks again.

[Michael Mattias]

> (I'm using two PowerThreads, not thread functions)

Not in the filewatcher.bas or filewatcher.inc I downloaded from thread # 11 there isn't. http://www.cur-ion.net/pb/FileWatcher.zip

Bad link to old version? Or multi-threaded code is in the other ZIP file?

If code is running multiple threads of execution as described, hang will not occur.

MCM

[Larry Charlton]

Nathan
Also it might be UAC. I have to run notepad or another editor as an administrator to acutally change a file in the root of C. But when I do, I'm seeing the changes in file watcher. A change to the file isn't even needed to see the change, pressing Control+S repeatedly in notepad caused the load count to increment.

I'll poke around and see if I can duplicate it with TortiseSVN.

[Larry Charlton]

> (I'm using two PowerThreads, not thread functions)

Not in the filewatcher.bas or filewatcher.inc I downloaded from thread # 11 there isn't. http://www.cur-ion.net/pb/FileWatcher.zip

Bad link to old version? Or multi-threaded code is in the other ZIP file?

If code is running multiple threads of execution as described, hang will not occur.

MCM

filewatcher.bas is an example of using filewatcher.inc. Filewatcher.inc has two classes outlined below, both using PowerThreads. I used the link in post #11 to get the file and make the change to display the error if one occured, so I think we're using the same program.

Class cFileWatcher
Thread Method Main() As Long
Interface iFileWatcher
Inherit IPowerThread

Class cFileWatchNotifier
Thread Method Main() As Long
Interface iFileWatchNotifier
Inherit IPowerThread

[Michael Mattias]

"inherit" not found in either file
"thread" not found in either file
"pow" not found in either file

Sure you don't have the wrong files zipped up out there?

[Larry Charlton]

Below is the source I have from clicking the link you provided. I see all three value in Filewatcher.inc. I'm baffled.

Filewatcher.bas

#Compile Exe
#Dim All
%UNICODE = 1
#Include Once "win32api.inc"
#Include Once "FileWatcher.inc"
Global fw As iFileWatcher:  ' File watcher
Global filename As WString: ' File we're watching
Global hDlg As Dword:       ' Open dialog
%FILE_EXIT    = 1005
%FILE_OPEN    = 1006
%CTL_TEXT     = 1001
Function PBMain()
  Local hFont, hTopMenu, hFileMenu, hAccel, hRes As Dword
  Dim menuAccel(0) As ACCELAPI
  Dialog New Pixels, %HWND_Desktop, "File Watcher", 105, 114, 607, 483, %WS_OverlappedWindow, To hDlg
  Control Add TextBox, hDlg, %CTL_TEXT, "", 0, 0, 0, 0, %WS_Child Or _
                                                            %WS_Visible Or _
                                                            %WS_Border Or _
                                                            %WS_TabStop Or _
                                                            %WS_VScroll Or _
                                                            %WS_HScroll Or _
                                                            %ES_Left Or _
                                                            %ES_MultiLine Or _
                                                            %ES_ReadOnly Or _
                                                            %ES_AutoHScroll Or _
                                                            %ES_AutoVScroll,_
                                                            %WS_Ex_RightScrollbar
  Font New "Courier New", 10, 0, %ANSI_CHARSET To hFont
  Control Set Font hDlg, %CTL_TEXT, hFont
  Menu New Bar To hTopMenu
  Menu New PopUp To hFileMenu
  Menu Add PopUp, hTopMenu, "File", hFileMenu, %MF_Enabled
  Menu Add String, hFileMenu, "&Open" & $Tab & "Ctrl+O", %FILE_OPEN, %MF_Enabled
  Menu Add String, hFileMenu, "-", 0, 0
  Menu Add String, hFileMenu, "E&xit", %FILE_EXIT, %MF_Enabled
  Menu Attach hTopMenu, hDlg
  menuAccel(0).fVirt = %FVIRTKEY Or %FCONTROL Or %FNOINVERT
  menuAccel(0).Key = Asc("O")
  menuAccel(0).Cmd = %FILE_OPEN
  Accel Attach hDlg, menuAccel() To hAccel
  Dialog Show Modeless hDlg, Call ShowFileWatcherProc To hRes
  GetNewFile()
  If IsNothing( fw ) Then Exit Function
  Call OnFileChanged( filename ): ' Initial load
  Do
    Dialog DoEvents
  Loop While IsWin(hDlg)
  If IsObject( fw ) Then fw.Stop()
  Font End hFont
End Function
CallBack Function ShowFileWatcherProc()
  Local w, h As Long
  Select Case Const Cb.Msg
    Case %WM_Size
      Dialog Get Client hDlg To w, h
      Control Set Size hDlg, %CTL_TEXT, w, h
    Case %WM_Command
      Select Case Const Cb.Ctl
        Case %FILE_OPEN: Call GetNewFile():   Call OnFileChanged( filename )
        Case %FILE_EXIT: Dialog End hDlg
      End Select
  End Select
End Function
Sub GetNewFile()
  Local newFile As WString
  Display Openfile 0, , , "Select File To Watch", "", "All|*.*", "", "", %OFN_FileMustExist To newFile
  If newFile = "" Then Exit Sub
  If IsObject( fw ) Then fw.Stop()
  fw = Nothing
  filename = newFile
  Call StartFileWatcher( fw, newFile, CodePtr( OnFileChanged ) )
End Sub
Sub OnFileChanged( filename As WString )
  Local value, c As String
  Local i, f As Long
  Static cnt As Long
  ' Reload file
  ErrClear
  f = FreeFile
  For i=1 To 10
    Open filename For Binary Access Read Lock Shared As #f
    If Err=0 Then Exit For
    ErrClear
    Sleep 100
  Next
  If Err<>0 Then
    Close #f
    Control Set Text hDlg, %CTL_TEXT, "(unable to open file " + Error$ + ")"
  Else
    Get$ #f, Lof(#f), value
    Close #f
    Incr cnt
    c = Format$( cnt, "#,##0" )
    value = Build$("Load round ", c, $CrLf, value )
    Control Set Text hDlg, %CTL_TEXT, value
  End If
End Sub

Filewatcher.inc

%BUFSIZE = 1024
Sub NewFileWatcher( obj As iFileWatcher, filename As WString, ByVal notifyRoutine As Dword )
  obj = Class "cFileWatcher"
  obj.Initialize( filename, notifyRoutine )
End Sub
Sub StartFilewatcher( obj As iFileWatcher, filename As WString, ByVal notifyRoutine As Dword )
  Call NewFileWatcher( obj, filename, notifyRoutine )
  obj.Launch( ByVal 0 )
End Sub
Sub NotifyPattern( filename As WString )
End Sub
Class cFileWatcher
  Instance ThreadParam As Long
  Instance running_ As Long
  Instance notifier_ As iFileWatchNotifier
  Instance notifyRoutine_ As Dword
  Instance filename_ As WString
  Instance hStop As Dword
  Thread Method Main() As Long
  #Align 4
    Local buffer1 As StringZ * %BUFSIZE
    Local buffer2 As StringZ * %BUFSIZE
    Local hover As OVERLAPPED
    Local hWait() As Dword
    Local dwWaitStatus, flags, hDir As Dword
    Local fullPath, modFile As WString
    Local finished, numBytes, cnt, i As Long
    Local fn As WString Ptr
    Local fi As FILE_NOTIFY_INFORMATION
    ReDim hWait(1) As Local Dword
    If IsFalse IsFile( filename_ ) Or notifyRoutine_=0 Then Exit Method
    Call StartFileWatchNotifier( notifier_, filename_, notifyRoutine_ )
    fullPath = PathScan$( Path, filename_ )
    ' Other things we can watch for
    '%FILE_NOTIFY_CHANGE_FILE_NAME
    '%FILE_NOTIFY_CHANGE_DIR_NAME
    '%FILE_NOTIFY_CHANGE_ATTRIBUTES
    '%FILE_NOTIFY_CHANGE_SIZE
    '%FILE_NOTIFY_CHANGE_SECURITY
    flags = %FILE_NOTIFY_CHANGE_LAST_WRITE: ' Watching for file changes/writes
    hDir = CreateFile( ByVal StrPtr(fullPath), %FILE_LIST_DIRECTORY, %FILE_SHARE_READ Or %FILE_SHARE_WRITE Or %FILE_SHARE_DELETE, ByVal %NULL, %OPEN_EXISTING, %FILE_FLAG_BACKUP_SEMANTICS Or %FILE_FLAG_OVERLAPPED , %NULL)
    hWait(0) = CreateEvent( ByVal %NULL, 0, 0, ByVal %NULL )
    hWait(1) = CreateEvent( ByVal %NULL, 0, 0, ByVal %NULL )
    hStop = hWait(1)
    hover.hEvent = hWait(0)
    Do While -1
      Memory Fill VarPtr( buffer1 ), %BUFSIZE, Byte 0
      ResetEvent( hover.hEvent )
      If ReadDirectoryChangesW( hDir, VarPtr(buffer1), %BUFSIZE, %FALSE, flags, %NULL, hover, %NULL) Then
        dwWaitStatus = WaitForMultipleObjects( 2, hWait(0), %FALSE, %INFINITE )
        Select Case Const dwWaitStatus - %WAIT_OBJECT_0
          Case 0: ' Directory change signaled
            'If returnedBytes>0 Then
              Memory Copy VarPtr( buffer1 ), VarPtr( buffer2 ), %BUFSIZE
              notifier_.Changed( VarPtr( buffer2 ) )
            'End If
          Case 1: ' hStop signaled
            Exit Loop
        End Select
      End If
    Loop
    hStop = 0
    CloseHandle( hWait(0) )
    CloseHandle( hWait(1) )
    CloseHandle( hDir )
    notifier_.Stop()
    notifier_ = Nothing
  End Method
  Interface iFileWatcher
    Inherit IPowerThread
    Method Initialize( filename As WString, notifyRoutine As Dword )
      If filename="" Or notifyRoutine=0 Then Exit Method
      If filename_ = "" Or notifyRoutine_ = 0 Then
        filename_ = filename
        notifyRoutine_ = notifyRoutine
      End If
    End Method
    Property Get Running() As Long
      Property = running_
    End Property
    Property Get Filename() As WString
      Property = filename_
    End Property
    Property Get NotifyRoutine() As Dword
      Property = notifyRoutine_
    End Property
    Method Stop()
      Local fw As iFileWatcher
      fw = Me
      SetEvent( hStop )
      fw.Join( fw, 0 )
      fw = Nothing
    End Method
  End Interface
End Class
Sub StartFileWatchNotifier( obj As iFileWatchNotifier, filename As WString, ByVal notifyRoutine As Dword )
  obj = Class "cFileWatchNotifier"
  obj.Initialize( filename, notifyRoutine )
  obj.Launch( ByVal 0 )
End Sub
Class cFileWatchNotifier
  Instance ThreadParam As Long
  Instance changed_ As Dword
  Instance stopped_ As Dword
  Instance notifyRoutine_ As Dword
  Instance filename_ As WString
  Instance fni As FILE_NOTIFY_INFORMATION Ptr
  Thread Method Main() As Long
    Local cnt As Long
    Local fi As FILE_NOTIFY_INFORMATION Ptr
    Local filename, fullPath, modFile As WString
    Local i, finished As Long
    Local hWait() As Dword
    Local ofs As Long
    Dim hWait(1) As Local Dword
    Local dwWaitStatus As Dword
    hWait(0) = CreateEvent( ByVal %NULL, 0, 0, ByVal %NULL )
    hWait(1) = CreateEvent( ByVal %NULL, 0, 0, ByVal %NULL )
    changed_ = hWait(0)
    stopped_ = hWait(1)
    fullPath = PathScan$( Path, filename_ )
    filename = LCase$( PathScan$( Namex, filename_ ) )
    Do While -1
      ResetEvent( changed_ )
      dwWaitStatus = WaitForMultipleObjects( 2, hWait(0), %FALSE, %INFINITE )
      Select Case Const dwWaitStatus - %WAIT_OBJECT_0
        Case 0: ' Changed
          fi = fni
          finished = 0
          While Not finished
            finished = ( @fi.NextEntryOffset = 0 )
            If @fi.Action = %FILE_ACTION_MODIFIED And @fi.FileNameLength Then
              modFile = Peek$$( VarPtr( @fi.FileName ), @fi.FileNameLength )
              i = Len(modFile)
              While i>1 And Asc(modFile,i)<=32
                Decr i
              Wend
              modFile=Left$(modFile,i)
              modFile = LCase$( Dir$( fullPath + modFile ) )
              If modFile = LCase$( filename ) Then
                Call Dword notifyRoutine_ Using NotifyPattern( filename_ )
              End If
            End If
            fi += @fi.NextEntryOffset
          Wend
        Case 1: ' Stopped
          Exit Loop
      End Select
    Loop
  End Method
  Interface iFileWatchNotifier
    Inherit IPowerThread
    Method Initialize( filename As WString, notifyRoutine As Dword )
      If filename_<>"" Or notifyRoutine_ Then Exit Method
      filename_ = filename
      notifyRoutine_ = notifyRoutine
    End Method
    Method Stop()
      Local fwn As iFileWatchNotifier
      SetEvent( stopped_ )
      fwn = Me
      fwn.Join( fwn, 0 )
      fwn = Nothing
    End Method
    Method Changed( ByVal fi As Dword )
      fni = fi
      SetEvent( changed_ )
    End Method
  End Interface
End Class

[Michael Mattias]

:doh:

Did you know...

In Ultra-Edit if you check the "Match Case" box on the "find", neither "pow" nor "thread" will match "IPowerThread?"