PERL Beginners - count in continuous time piece

Author

count in continuous time piece

Jeff Pang

2006-01-24, 6:56 pm

hi,lists,
I want to do this work: count something in continuous given time piece,such as 10 minutes.Fox example,there are the lines coming to the socket server as following:

1_feifish1_01_1000ybyg:11530514:get
1_swm_mail_01_100k8qte:13901448:get
1_dajia###_01_1006x8jh:11652267:put
....
each line is split to 3 fieles: uid,size,type

The socket server accept these lines continuely.I want the result: some a uid such as '1_swm_mail_01_100k8qte',when his total size are plus to some a number (fox example,20000000) in the past 10 minutes,then I would do something.But I have no idea about h
ow to do the count in continuous time piece.Is there any suggestion here?thanks.

--
http://home.earthlink.net/~pangj/

Chas Owens

2006-01-24, 6:56 pm

On 1/24/06, Jeff Pang <pangj@earthlink.net> wrote:
> hi,lists,
> I want to do this work: count something in continuous given time piece,su=
ch as 10
> minutes.Fox example,there are the lines coming to the socket server as fo=
llowing:
>
> 1_feifish1_01_1000ybyg:11530514:get
> 1_swm_mail_01_100k8qte:13901448:get
> 1_dajia###_01_1006x8jh:11652267:put
> ...
> each line is split to 3 fieles: uid,size,type
>
> The socket server accept these lines continuely.I want the result: some a=
uid such as
> '1_swm_mail_01_100k8qte',when his total size are plus to some a number (f=
ox
> example,20000000) in the past 10 minutes,then I would do something.But I =
have no
> idea about how to do the count in continuous time piece.Is there any sugg=
estion
> here?thanks.
snip

If you have access to whatever is writing to the socket then the
safest thing to do is to add a timestamp that the message was sent.=20
Otherwise you will need to maintain the timestamp of the arrival of
the message. See the select function on (perldoc -f select) how to
only execute when data is available.

Jeff Pang

2006-01-27, 3:55 am

Now I'm still

on this work.Maybe I have not described the problem clearly.
Fox example,there are some items coming in continuous time piece:

00:00:01 itemA 200
00:00:02 itemB 100
00:00:03 itemC 150
00:00:04 itemD 300
00:00:05 itemE 250
....
(the item appear as 'name => vaule' style. And most of the items's name are different from others)

In every 5 minutes, I'm doing it as following:

{
sleep 5*60;
my %hash = calculate_from_the_items(); #for each item,I'll plus all the historical records to a hash value,and the hash key is this item's name.
for (keys %hash){
do_something() if $hash{$_} > LIMIT;
}
clear_the_items_to_null(); #clear all the items's records to null
}

But it's a very simple resolving method.I'm not satisfied with this way.
I want to get this result when each one item is coming:

{
my $sock=shift;
while(<$sock> )
{
my $isTrue = do_judgement();#judge if it reach some a limit in last 5 minutes relative the current time
if ($isTrue){
do_something();
}
clear(); #clear this item's historical record out of 5 minutes relative the current time
}
}

How can I do it?Any suggestion is welcome.Thanks.

-----Original Message-----
>From: Chas Owens <chas.owens@gmail.com>
>Sent: Jan 24, 2006 9:47 AM
>To: Jeff Pang <pangj@earthlink.net>
>Cc: beginners@perl.org
>Subject: Re: count in continuous time piece
>
>On 1/24/06, Jeff Pang <pangj@earthlink.net> wrote:
>snip
>
>If you have access to whatever is writing to the socket then the
>safest thing to do is to add a timestamp that the message was sent.
>Otherwise you will need to maintain the timestamp of the arrival of
>the message. See the select function on (perldoc -f select) how to
>only execute when data is available.

--
http://home.earthlink.net/~pangj/

Chas Owens

2006-01-27, 6:57 pm

On 1/27/06, Jeff Pang <pangj@earthlink.net> wrote:
> Now I'm still

on this work.Maybe I have not described the proble=
m clearly.
> Fox example,there are some items coming in continuous time piece:
>
> 00:00:01 itemA 200
> 00:00:02 itemB 100
> 00:00:03 itemC 150
> 00:00:04 itemD 300
> 00:00:05 itemE 250
> ...
> (the item appear as 'name =3D> vaule' style. And most of the items's name=
are different from others)
>
> In every 5 minutes, I'm doing it as following:
>
> {
> sleep 5*60;
> my %hash =3D calculate_from_the_items(); #for each item,I'll plus =
all the historical records to a hash value,and the hash key is this item's =
name.
> for (keys %hash){
> do_something() if $hash{$_} > LIMIT;
> }
> clear_the_items_to_null(); #clear all the items's records to null
> }
>
> But it's a very simple resolving method.I'm not satisfied with this way.
> I want to get this result when each one item is coming:
>
> {
> my $sock=3Dshift;
> while(<$sock> )
> {
> my $isTrue =3D do_judgement();#judge if it reach some a limit in=
last 5 minutes relative the current time
> if ($isTrue){
> do_something();
> }
> clear(); #clear this item's historical record out of 5 minutes =
relative the current time
> }
> }
>
> How can I do it?Any suggestion is welcome.Thanks.

In your first example you are creating an aggregate based on a key.=20
In your second example you want a similar aggregate, but you don't
want to count items that are older than 5 minutes ago. This means you
will need to keep the items separate until the check and you will need
to store a timestamp (to know how old an item is). Assuming you
cannot change what is coming in from the socket, you will need to add
the timestamp that item was read (as opposed to when it was created).=20
The ideal pattern for this is an array of either arrays or hashes
depending on your preference. You will need to read in all available
records from the socket attaching the time the item was read. At the
end you should have a data structure that looks like this:

my @records =3D (
{
time =3D> 1138374300,
key =3D> 'joe',
amount =3D> 20
},
{
time =3D> 1138374362,
key =3D> 'mary',
amount =3D> 10
},
{
time =3D> 1138374462,
key =3D> 'joe',
amount =3D> 5
},
{
time =3D> 1138374500,
key =3D> 'joe',
amount =3D> 1
},
{
time =3D> 1138374530,
key =3D> 'mary',
amount =3D> 20
},
);

Assuming it is now 1138374600 then the first record needs to be
expunged with delete(). This could be achieved by doing something
like this (warning untested code):

for (@records) {
delete $_ if $_->{time} < time() - 5*60;
}

You can then loop over the remaing records creating the aggregate like this

my %sum;
for my $rec (@records) {
$sum{$rec->{key}} +=3D $rec->{amount};
}

Chas Owens

2006-01-27, 6:57 pm

On 1/27/06, Chas Owens <chas.owens@gmail.com> wrote:

> for (@records) {
> delete $_ if $_->{time} < time() - 5*60;
> }
Oops, I misused the delete() function. I needed to use the shift()=20
function instead:

shift @records while $records[0]{time} < time() - 5*60;

Jeff Pang

2006-01-27, 6:57 pm

Hello,Chas,

Thanks advanced for your good suggestions.
I tidy up all the words said by you,and write the code as following.Is it right?

while(<$sock> )
{
my ($key,$value) = split;
my $timestamp = time();
push @records, { time => $timestamp,
key => $key,
amount=> $value,
};

shift @records while $records[0]{time} < time() - 5*60;

my %sum;
for my $rec (@records) {
$sum{$rec->{key}} += $rec->{amount};
}

for (keys %sum)
{
do_something() if $sum{$_} > LIMIT;
}
}

-----Original Message-----
>From: Chas Owens <chas.owens@gmail.com>
>Sent: Jan 27, 2006 11:14 PM
>To: Jeff Pang <pangj@earthlink.net>
>Cc: beginners@perl.org
>Subject: Re: count in continuous time piece
>
>On 1/27/06, Jeff Pang <pangj@earthlink.net> wrote:
>
>In your first example you are creating an aggregate based on a key.
>In your second example you want a similar aggregate, but you don't
>want to count items that are older than 5 minutes ago. This means you
>will need to keep the items separate until the check and you will need
>to store a timestamp (to know how old an item is). Assuming you
>cannot change what is coming in from the socket, you will need to add
>the timestamp that item was read (as opposed to when it was created).
>The ideal pattern for this is an array of either arrays or hashes
>depending on your preference. You will need to read in all available
>records from the socket attaching the time the item was read. At the
>end you should have a data structure that looks like this:
>
>my @records = (
> {
> time => 1138374300,
> key => 'joe',
> amount => 20
> },
> {
> time => 1138374362,
> key => 'mary',
> amount => 10
> },
> {
> time => 1138374462,
> key => 'joe',
> amount => 5
> },
> {
> time => 1138374500,
> key => 'joe',
> amount => 1
> },
> {
> time => 1138374530,
> key => 'mary',
> amount => 20
> },
> );
>
>Assuming it is now 1138374600 then the first record needs to be
>expunged with delete(). This could be achieved by doing something
>like this (warning untested code):
>
>for (@records) {
> delete $_ if $_->{time} < time() - 5*60;
>}
>
>You can then loop over the remaing records creating the aggregate like this
>
>my %sum;
>for my $rec (@records) {
> $sum{$rec->{key}} += $rec->{amount};
>}

--
http://home.earthlink.net/~pangj/

Chas Owens

2006-01-27, 6:57 pm

On 1/27/06, Jeff Pang <pangj@earthlink.net> wrote:
> Hello,Chas,
>
> Thanks advanced for your good suggestions.
> I tidy up all the words said by you,and write the code as following.Is i=
t right?
>
> while(<$sock> )
> {
> my ($key,$value) =3D split;
> my $timestamp =3D time();
> push @records, { time =3D> $timestamp,
> key =3D> $key,
> amount=3D> $value,
> };
>
> shift @records while $records[0]{time} < time() - 5*60;
>
> my %sum;
> for my $rec (@records) {
> $sum{$rec->{key}} +=3D $rec->{amount};
> }
>
> for (keys %sum)
> {
> do_something() if $sum{$_} > LIMIT;
> }
> }
snip

That looks about right. I would have read all available records (see
perldoc -f select) from the socket adding them to @records and then
executed the rest of the code, but this way should be fine if
do_something() is not overly long. You might want to change

shift @records while $records[0]{time} < time() - 5*60;

to

shift @records while $records[0]{time} < $timestamp - 5*60;

Sponsored Links