PERL Modules - dupe times in Time::HiRes

Author

dupe times in Time::HiRes

atcproductions@gmail.com

2005-03-23, 3:56 am

Monks, mongers and perlites,

I have been attempting to use it to provide 20 digit time date stamp
which includes 6 digits for microseconds. In conjunction with
Time::HiRes I have also used Time::Format which calls HiRes.

The issue that I am running into is I am getting duplicate time stamps.
The following is an excerpt of the code:

Situation: MySQL v4.0, perl v5.8, perl DBI/DBD

There is a MySQL prepare statement, then those bind parameters are
inserted in a loop. These inserts can be quite frequent, but as you can
see in the output below, even slow inserts cause dupes.

#!/usr/bin/perl
#
#####
#
# use me, require me..
#
use Time::HiRes;
use Time::Format qw(%time %strftime %manip);
...
...
...
my $ptime;
while(<STDIN> ) {
chomp;
my $event = $_;
undef $ptime;
$ptime = $time{'yyyymmddhhmmss.uuuuuu'};
my ($A,$B,$C,$D,$E,$F,$G) = split /<-->/, $event;
EXECUTE: unless ( eval{ $sth->execute($ptime,$A,$B,$C,$D,$E,$F,$G)}) {
# eval ensures no exit!
...
...
...

Here is a sample from the database:

+--------------------------+-------------------------------------+
| timeDateInserted | timeDateInsertedP |
+--------------------------+-------------------------------------+
| 20050318130506 | 20050318130506.304688 |
| 20050318130507 | 20050318130507.304688 |
| 20050318130507 | 20050318130507.355469 |
| 20050318130508 | 20050318130508.304688 |
| 20050318130509 | 20050318130509.304688 |
| 20050318130509 | 20050318130509.304688 |
| 20050318130509 | 20050318130509.414062 |
| 20050318130510 | 20050318130510.414062 |
+--------------------------+-------------------------------------+
8 rows in set (0.00 sec)

It appears that microseconds somehow are not being updated during the
loop. I have also tried using HiRes with out Format and I am getting
the same type of behavior.

Has anyone come across this type of behavior before? I'm just looking
for each row to have a unique, sequential time. The actual accuracy of
the microseconds is irrelevant as long as no 2 rows have the same exact
time. Any suggestions would be greatly appreciated.

TIA!

pat

2005-03-23, 3:56 pm

Chris,

Always good to have your opinion.

On the first count, microsecond accuracy is not required. Even
hundredth of a second would be passable.

The point remains, however, that the module is broken, and I haven't a
clue how to rewrite/fix it. Dupe times, especially across second
boundaries, are simply unacceptable. Barring someone telling me how I'm
using the module incorrectly, it remains broken.

Now the second count is interesting. As we;re talking events on a
single machine, a unique , arbitrary counter would certainly be an
interesting idea. I wish I could use perl instead of the database,
though, for performance reasons. Perhaps there is a way...

Lets suppose I had a running process/daemon running on the box that
would entertain requests for a unique (well, sequential) identifier.
Each running process would ask for an integer through IPC and get one.
This SHOULD be the fastest way to communicate between running programs
on the same box, right?

The reason I like this approach is that the numbers (high water) being
returned by this daemon need not survive a reboot. They can start over
with no problems, as they are only used for resolution within a
timeframe of a second.

For performance, I'd call this daemon instead of using the current
overhead of asking for usec resolution from Time::HiRes, so I might nod
add any overhead. In fact, I might be doing things faster that way, as
the daemon would be doing any POSIX calls or actual calculations.

Sorry for thinking out loud, but it is a very interesting approach. I
guess now, being on a modules group, I need to find the fastest module
that allows my programs to communicate with a perl daemon!

Can you see any flaws in this approach?

Are there any suggested modules that would allow speed and performance
as desired?

Thanks again for your posts!!!

pat
:)

chris-usenet@roaima.co.uk

2005-03-23, 8:55 pm

pat <pat.trainor@gmail.com> wrote:
> The point remains, however, that the module is broken, and I haven't a
> clue how to rewrite/fix it.

I've got version 1.59 with perl 5.8.4 running on Intel hardware (GNU/Linux
debian "sarge"), which appears to work just fine.

> Lets suppose I had a running process/daemon running on the box that
> would entertain requests for a unique (well, sequential) identifier.
> Each running process would ask for an integer through IPC and get one.
> This SHOULD be the fastest way to communicate between running programs
> on the same box, right?

You'll probably get a better response playing with a local file and
exclusive locking. Come to think of it, there's a FAQ on this (see
"perldoc -q increment"). It'd also be easier to maintain than a separate
IPC/socket based daemon.

> For performance, I'd call this daemon instead of using the current
> overhead of asking for usec resolution from Time::HiRes, so I might nod
> add any overhead.

With local file locking, your daemon suddenly becomes about half a dozen
lines of perl (or a trivial home-grown module to hide the details).

Regards,
Chris

pat

2005-03-24, 8:56 pm

Chris,

Thanks for the input. Sequential counters are fine, and as the entries
are being stored in a database with an auto_increment column that part
is fine. Unfortunately the solution suggested won't work when entries
in one table/db are compared to another.

Arbitrary is valid if there is only one source of entries, and they are
being compared against other entries in the same table.

When comparing/merging and 'UNION'ing tables together, the
auto_increment fails to allow a chronological sorting of events as
different tables count in the same range at different times.

A true chronological timestamp allows entries from various sources
(tables) to be merged to show precisely what happenned when, and most
importantly in which order. This is all I'm trying to achieve. If a
valid, repeatable usec time could be achieved from this module
(Time::HiRes), life would be good. As is, unfortunately, the module
does not perform consistently or accurately (at lesat in the way that I
am using it).

If the module can't be made to function accurately (see above posts for
cross-second dupes, etc.) then is there a way (perhaps through POSIX?)
to get the system time (sans whole seconds)-regardless of resolution?
Even 1/10 sec resolution is 10 times more accurate than whole seconds.
I hope I'm explaining the symptoms properly; forgive me if not.

There has to be a good solution. Ideas?

TIA!

pat
:)

chris-usenet@roaima.co.uk

2005-03-26, 8:55 pm

pat

2005-04-06, 12:08 pm

Chris/All,

Problem resolved (or at least identified). It is NOT what you might
think it is.

First let me say that we're still troubleshooting the precise cause,
but herre is how it was resolved in the application. The INSERT was/is
being done with a perl dbi/dbd prepare and subsequent high-rate
executes. THe column in question, as you may have figured out by
looking at the earlier posts, was a MySAL type DOUBLE. A TIMESTAMP(14)
would naturally not take into account the usec resolution.

Resolution (well, a reduction of symptoms at least): On a hunch, the
other developer on the project changed the column type in MySQL to
text. Instantly the problem went away, and all is well.

This means, of course, that the Time::HiRes module is not th eculprit,
and that either DBI::DBD and/or MySQL are at fault. I'll post the test
code that allows anyone to reproduce the fault next email.

Thanks for your determination, it is what kept us moving toward a
'reduction in symptoms' instead of giving up and writing our own
solution (which would not have worked!)...

pat
:)

pat

2005-04-07, 3:56 am

Sponsored Links