PERL Beginners - About File::Monitor

Author

About File::Monitor

reader@newsguy.com

2008-01-24, 7:07 pm

Can anyone provide a real example of using File::Monitor?

I've been pounding away at perldoc File::Monitor, which seems to be
pretty thoroughly documented.... but as often seems to happen to me
its just not soaking in... how the scanning and reporting is supposed
to happen. Somehow in the course of reading through the documentation
I just start to see red and don't get what is supposed to happen.

What I really want to do is monitor a directory recursively but here
just trying to use it any basic way to start to `get' how to use it.

Taking one of the examples offered at:
http://search.cpan.org/~andya/File-...nitor/Object.pm

With a few minor (but probably wrong) changes I get nothing printed
when the file is altered during the sleep. The program just runs 10
seconds and closes.

The example seems to be saying in the comments to just run
$monitor->scan; `later' and changes will be available.

#!/usr/bin/perl

use strict;
use warnings;

use File::Monitor;
use File::Monitor::Object;

my $monitor = File::Monitor->new();

my @files = ( "./oldwork/test");
for my $file ( @files ) {
$monitor->watch( $file );
}

# First scan just finds out about the monitored files. No changes
# will be reported.
$monitor->scan;
sleep 10;

# Later perform a scan and gather any changes
for my $change ( $monitor->scan ) {
# $change is a File::Monitor::Delta

## Also tried `print "$File::Monitor::Delta\n" -hp;

print "change<$_>\n";
}

Tom Phoenix

2008-01-24, 7:07 pm

On Jan 24, 2008 12:34 PM, <reader@newsguy.com> wrote:

> Can anyone provide a real example of using File::Monitor?

Aren't the examples in the docs and the t/ directory "real" enough for
you? There's even a file in the examples/ directory; it looks pretty
real to me. Does it work for you?

http://search.cpan.org/src/ANDYA/Fi...ples/monitor.pl

> With a few minor (but probably wrong) changes I get nothing printed
> when the file is altered during the sleep. The program just runs 10
> seconds and closes.

How do you know that the file was altered "during the sleep"? For
example, it could be that you tried to change a file, but your change
was still in an output buffer, and not yet flushed to disk, when the
second scan occurred.

Did the module pass all tests before you installed it?

Hope this helps!

--Tom Phoenix
Stonehenge Perl Training

reader@newsguy.com

2008-01-24, 7:07 pm

"Tom Phoenix" <tom@stonehenge.com> writes:

> On Jan 24, 2008 12:34 PM, <reader@newsguy.com> wrote:
>
>
> Aren't the examples in the docs and the t/ directory "real" enough for
> you? There's even a file in the examples/ directory; it looks pretty
> real to me. Does it work for you?
>
> http://search.cpan.org/src/ANDYA/Fi...ples/monitor.pl

Well there is a hard and very public lesson learned. I guess I'm
ashamed to say I never thought to look in there. I installed with cpan
and unless I have trouble installing something I never do go into the
build..

Yeah, that looks like exactly the example I was after. And yes it
works.

> How do you know that the file was altered "during the sleep"? For
> example, it could be that you tried to change a file, but your change
> was still in an output buffer, and not yet flushed to disk, when the
> second scan occurred.

There is something else I guess I hadn't thought of.
I just did `echo something >> test' repeatedly during the sleep.

But I think Jays' observation on that is the problem:

Jay wrote:
> Without the recurse option, this just watches the directory, not the
> files in it. So unless you're adding or deleting files...

> Did the module pass all tests before you installed it?

I don't know that either. I didn't see any errors

> Hope this helps!

Somewhat embarrassing but yes it has.

reader@newsguy.com

2008-01-24, 7:07 pm

"Jay Savage" <daggerquill@gmail.com> writes:

[...] Thanks for the pointers

> Finally, Tom's points are important. How do you *know* that the files
> (in this case a single directory) changed *during the sleep*? Do you
> know that the output wasn't buffered? That the system didn't delay the
> writes for some reason? That you were even watching the files you
> thought you were?

The last one there seems pretty likely.

reader@newsguy.com

2008-01-25, 7:06 pm

"Jay Savage" <daggerquill@gmail.com> writes:

> Finally, Tom's points are important. How do you *know* that the files
> (in this case a single directory) changed *during the sleep*? Do you
> know that the output wasn't buffered? That the system didn't delay the
> writes for some reason? That you were even watching the files you
> thought you were?

> Your best bet to make sure the module is working would, instead of
> sleeping between scans and trying to modify the watched files from an
> external process during a narrow window, would probably be to perform
> your initial scan; open one of the watched files for writing; write to
> it; flush the buffer; close it; and then rescan.

Sounds like a plan for testing but the end result sought in this case
is a monitor that *will* see changes done by external processes in some
reasonable time frame.

(Assuming for a moment I do get it figured out with the help in this
thread how to use File::Monitor)

How long of a sleep would be required to *KNOW* a change has happened
with something external writing to files? Is there no way to guess
that closely... too many variables may come into play? Or could one
create a monitor that will *know* inside of ten seconds that a file
has been written to?

Would *knowing* about file creation have the same problems? Or is that
something the monitor would *know* for sure quickly?

Tom Phoenix

2008-01-25, 7:06 pm

On Jan 25, 2008 9:20 AM, <reader@newsguy.com> wrote:

> How long of a sleep would be required to *KNOW* a change has happened
> with something external writing to files?

You want to be sure the other task is finished? Several days of
waiting should suffice for better than 99.999% of all cases. If you
can tolerate a lower reliability rate (i.e., occasionally processing a
file before it's ready), you can cut that delay by many orders of
magnitude.

There is, in general, no way to be sure that somebody writing a file
doesn't still have some writing or post-processing yet to do. But
there are partial solutions that work well in most practical cases.
Sometimes the file format is one (like XML) that shows end-of-data on
its own. Sometimes the program writing the file can be made to lock
the file while writing it or to otherwise signal when it is finished.
Sometimes the solution is to move finished files (atomically) into the
monitored directory from a staging directory on the same mounted
volume. Sometimes the solution is to put only symbolic links to
finished files into the monitored directory. Sometimes the OS or other
tools can tell you when the file has been written (but they can be
fooled by, e.g., a shell script or makefile that writes a file in
chunks). Sometimes it works to wait for a predetermined amount of time
after some event, then to assume (i.e. hope) that the file is
finished. That last is a common solution; if a tolerable time delay
and maybe a little program logic can make errors reasonably
infrequent, and if the circumstances make errors reasonably easy to
recover from, it's not a bad solution.

Good luck with it!

--Tom Phoenix
Stonehenge Perl Training

reader@newsguy.com

2008-01-30, 10:11 pm

"Jay Savage" <daggerquill@gmail.com> writes:

> Where is all this headed? File monitoring relies on Step 4. Step 4 is
> where the directory pointers and block counts get updated, and the
> ctime and mtime get rewritten. Once your system performs Step 4,
> File::Monitor will see the change on its next run. How long that takes
> will depend on how the writing process is written, and at least
> partially on what OS its running on.

This whole thread has been helpful.

You and Tom, having taken time to explain beyond the call of duty is especially
appreciated ... thank you.

Sponsored Links