PERL Beginners - File content question

Author

File content question

Sanjeev Sagar

2004-03-29, 6:42 pm

Hello Everyone,

We have a process where several scripts write to centralized log file. I
have to retrieve a specific set of information and display in a format.
Following is a snippet of log file

=========================
<Bunch of text lines>

Variable_name Value
1.
2.
3.
4.
..
..
..
Upto 200 lines of variables and their values
************************1 row***************

<Bunch of text files>
===============================

I have to collect the variable_name and their value block of 200 lines.
Every 15 min. this output get added in file.

I am trying to write code for this but getting lost in b/w like below

#!/usr/bin/perl -w

$infile = "/tmp/test1.log";
$outfile = "/tmp/mysqltats.out";

open (INFILE, "<$infile") || die "cannot open $infile: $!\n";
open (OUTFILE, ">$outfile") || die "cannot open $outfile: $!\n";

while ( $mystring = <INFILE> ) {

chomp $mystring;
if ( $mystring =~ /\*+/) { next; }
elsif ( $mystring =~ /^Variable_name/ ) { next; }

#Here I need to store those 200 lines in an array or something but don't
know how.
}

close (OUTFILE);
close (INFILE);

system ("cat $outfile");

Any help will be highly appreciable.

Best Regards,

<<Sagar, Sanjeev.vcf>>

Morbus Iff

2004-03-29, 6:42 pm

Where are you getting lost? I don't recall your previous
messages, so I'm not sure what "like before" means...
Anyway, here's an untested rewrite. Untested. Mmhm.

The biggest problem is making sure to reset %datastore
when you come across another <bunch of text lines> that
don't match your variable names. You didn't provide enough
data for that instance, so I've considered it irrelevant.

#!/usr/bin/perl
use warnings;
use strict;

my $infile = "/tmp/test1.log";
my $outfile = "/tmp/mysqltats.out";
my %datastore; # where we keep variables.

open (INFILE, "<$infile") || die "cannot open $infile: $!\n";
open (OUTFILE, ">$outfile") || die "cannot open $outfile: $!\n";

while (<INFILE> ) {
chomp;

next if /^\*+/; # changed from your example. you were
# checking if there was *+ anywhere in
# a string, which could potentially match
# the value of a variable. ^\&+ is stronger.

# etc., etc. this sorta syntax is more
# readable than if/else statements. the same
# can be done with "next unless [condition]".
next if /^anotherexample/;

# key: beginning of line up to first whitespace.
# value: everything else after the first whitespace.
my ($key, $value) = /^(.*)\s(.*)$/;

# you may want to throw in more tests here to make
# sure that $key is what you expect: ie., letters and
# underscores only, less than 15 characters, lowercase,
# etc., etc. pro-actively checking for sanity helps.
#
# next if $key !~ /[\w_-]*/;
# next if length($key) > 15;
# $key = lc($key);

# store the key/value into a hash, unless this
# key has already been seen (ie., the first instance
# takes precedent. remove the "unless ..." if you'd
# like the final value of a duplicated key instead.
$datastore{$key} = $value unless $datastore($key);
}

# print out a specific key value.
print "The value of 'Bob' is $key{"Bob"}\n";

# or loop through 'em all.
foreach (keys %datastore) {
print "Variable: $_ // Value: $datastore{$_}\n";
}

close (OUTFILE); close (INFILE);

--
Morbus Iff ( i put the demon back in codemonkey )
Culture: http://www.disobey.com/ and http://www.gamegrene.com/
Spidering Hacks: http://amazon.com/exec/obidos/ASIN/...5776/disobeycom
icq: 2927491 / aim: akaMorbus / yahoo: morbus_iff / jabber.org: morbus

John W. Krahn

2004-03-29, 7:33 pm

Sanjeev Sagar wrote:
>
> Hello Everyone,

Hello,

> We have a process where several scripts write to centralized log file. I
> have to retrieve a specific set of information and display in a format.
> Following is a snippet of log file
>
> =========================
> <Bunch of text lines>
>
> Variable_name Value
> 1.
> 2.
> 3.
> 4.
> .
> .
> .
> Upto 200 lines of variables and their values
> ************************1 row***************
>
> <Bunch of text files>
> ===============================
>
> I have to collect the variable_name and their value block of 200 lines.
> Every 15 min. this output get added in file.
>
> I am trying to write code for this but getting lost in b/w like below
^^^^^^^^^^^
Lost in what?

> #!/usr/bin/perl -w
>
> $infile = "/tmp/test1.log";
> $outfile = "/tmp/mysqltats.out";
>
> open (INFILE, "<$infile") || die "cannot open $infile: $!\n";
> open (OUTFILE, ">$outfile") || die "cannot open $outfile: $!\n";
>
> while ( $mystring = <INFILE> ) {
>
> chomp $mystring;
> if ( $mystring =~ /\*+/) { next; }

Your regular expression says "find one or more '*' characters anywhere
in $mystring" however finding one is the same as finding one or more.

if ( $mystring =~ /\*/) { next; }

> elsif ( $mystring =~ /^Variable_name/ ) { next; }

You could do it like this:

next if $mystring =~ /\*/ or $mystring =~ /^Variable_name/;

> #Here I need to store those 200 lines in an array or something but don't
> know how.

push @an_array, $mystring;

> }
>
> close (OUTFILE);
> close (INFILE);
>
> system ("cat $outfile");

Why not just 'print "$mystring\n";' inside the while loop.

John
--
use Perl;
program
fulfillment

Sanjeev Sagar

2004-03-29, 7:33 pm

Big Thanks !

My log file looks like below

2004-03-26 @ 00:00:01 sql_er_status@pinhpp31 -- 10881 10864 hyb01
:INFORMATIONAL
2004-03-26 @ 00:00:01 sql_er_status@pinhpp31 -- 10881 10864 hyb01 Function
(up) returned (0) return code. -- seconds -- 1:INFORMATIONAL
2004-03-26 @ 00:00:01 sql_er_lagonly_init@pinhpp31 -- 10878 10862 hyb01
Script/function sql_er_lagonly_init started at Fri Mar 26 00:00:01 CST 2004
-- seconds -- 1:INFORMATIONAL
2004-03-26 @ 00:00:01 sql_er_status_final@pinhpp31 -- 10881 10864 hyb01
Returning return code 0:INFORMATIONAL
2004-03-26 @ 00:00:01 sql_er_status_final@pinhpp31 -- 10881 10864 hyb01
Script/function sql_er_status_final ended at Fri Mar 26 00:00:01 CST 2004 --
seconds -- 1:INFORMATIONAL
2004-03-26 @ 00:00:01 sql_er_lagonly@pinhpp31 -- 10878 10862 hyb01
Script/function sql_er_lagonly started at Fri Mar 26 00:00:01 CST 2004 --
seconds -- 1:INFORMATIONAL
Variable_name Value
Aborted_clients 5592
Aborted_connects 4
Bytes_received 58302821
Bytes_sent 658785263
Com_admin_commands 0
Com_alter_table 27
Com_analyze 0
Com_backup_table 0
Com_begin 35652978
Com_change_db 172
Com_change_master 0
Com_check 0
Com_commit 35653362
Com_create_db 0
Com_create_function 0
..
..
..
200 of these parameters
2004-03-26 @ 00:00:04 sql_instance_ping@pinhpp31 -- 10880 10863 hyb01
DATABASE IS UP:INFORMATIONAL
2004-03-26 @ 00:00:04 sql_instance_ping_final@pinhpp31 -- 10880 10863 hyb01
Script/function sql_instance_ping_final started at Fri Mar 26 00:00:04 CST
2004 -- seconds -- 4:INFORMATIONAL

Any idea on reseting %datastore will be highly appreciable.

Best Regards,

-----Original Message-----
From: Morbus Iff [mailto:morbus@disobey.com]
Sent: Monday, March 29, 2004 4:45 PM
To: Sagar, Sanjeev; beginners@perl.org
Cc: Sagar, Sanjeev
Subject: Re: File content question

Where are you getting lost? I don't recall your previous messages, so I'm
not sure what "like before" means... Anyway, here's an untested rewrite.
Untested. Mmhm.

The biggest problem is making sure to reset %datastore
when you come across another <bunch of text lines> that
don't match your variable names. You didn't provide enough
data for that instance, so I've considered it irrelevant.

#!/usr/bin/perl
use warnings;
use strict;

my $infile = "/tmp/test1.log";
my $outfile = "/tmp/mysqltats.out";
my %datastore; # where we keep variables.

open (INFILE, "<$infile") || die "cannot open $infile: $!\n";
open (OUTFILE, ">$outfile") || die "cannot open $outfile: $!\n";

while (<INFILE> ) {
chomp;

next if /^\*+/; # changed from your example. you were
# checking if there was *+ anywhere in
# a string, which could potentially match
# the value of a variable. ^\&+ is stronger.

# etc., etc. this sorta syntax is more
# readable than if/else statements. the same
# can be done with "next unless [condition]".
next if /^anotherexample/;

# key: beginning of line up to first whitespace.
# value: everything else after the first whitespace.
my ($key, $value) = /^(.*)\s(.*)$/;

# you may want to throw in more tests here to make
# sure that $key is what you expect: ie., letters and
# underscores only, less than 15 characters, lowercase,
# etc., etc. pro-actively checking for sanity helps.
#
# next if $key !~ /[\w_-]*/;
# next if length($key) > 15;
# $key = lc($key);

# store the key/value into a hash, unless this
# key has already been seen (ie., the first instance
# takes precedent. remove the "unless ..." if you'd
# like the final value of a duplicated key instead.
$datastore{$key} = $value unless $datastore($key);
}

# print out a specific key value.
print "The value of 'Bob' is $key{"Bob"}\n";

# or loop through 'em all.
foreach (keys %datastore) {
print "Variable: $_ // Value: $datastore{$_}\n";
}

close (OUTFILE); close (INFILE);

--
Morbus Iff ( i put the demon back in codemonkey )
Culture: http://www.disobey.com/ and http://www.gamegrene.com/ Spidering
Hacks: http://amazon.com/exec/obidos/ASIN/...5776/disobeycom
icq: 2927491 / aim: akaMorbus / yahoo: morbus_iff / jabber.org: morbus

R. Joseph Newton

2004-03-30, 1:30 am

"Sagar, Sanjeev" wrote:

> Big Thanks !
>
> My log file looks like below

Would be better with multiple sections, and only a few representative line
per...

> 2004-03-26 @ 00:00:01 sql_er_status@pinhpp31 -- 10881 10864 hyb01
> :INFORMATIONAL
>

....

> Script/function sql_instance_ping_final started at Fri Mar 26 00:00:04 CST
> 2004 -- seconds -- 4:INFORMATIONAL
>
> Any idea on reseting %datastore will be highly appreciable.

Yes it will take an appreicable effort to come up with any ideas. What do you
want this script to do for you? Do you want to have all the log information in
memory so that you can compare between checks? In that case you will probably
want a ahs of hash references. The keys of the outer hash would probably be the
times of the checks The value for each key would be a reference to an anonymous
hash containing all of the status keys and their values for a given checktime.

You can make use of the file structure in deciding when to add one set of status
vlues to your ahs and start the next.

You know that:

Each check-time record starts with a series of lines, each of which starts with
a specifically formatted date and time string.
The end of the report is marked by a constant string;
Variable_name Value
This constant string is followed by the name-value pairs you s

to organize.

So clearly, the first date formatted string you encounter on each timecheck
should signal you to store the previous anonymous hash, and to get the date
information that will be the key of the next. Depending on whether you are
making use of the information contained in the header lines, you will process
them or not while watching for the constant string /^Variable_name Value$/
which will signal your program to start collecting the key-value pairs for the
current time-check.

Joseph

Somthing like this:

Greetings! E:\d_drive\perlStuff>perl -w
my %time_checks;
my $line = <DATA>;
while ($line) {
my $current_time;
if ($line =~ /^(\d{4}-\d{2}-\d{2} \@ \d{2}:\d{2}:\d{2})/) {
$current_time = $1;
}
$line = <DATA> until $line =~ /^Variable_name Value$/;
my $current_values = {};
$line = <DATA>;
until ($line =~ /^(\d{4}-\d{2}-\d{2} \@ \d{2}:\d{2}:\d{2})/) {
chomp $line;
last unless $line;
my ($key, $value) = split /\s+/, $line;
$current_values->{$key} = $value;
$line = <DATA>;
}
$time_checks{$current_time} = $current_values;
chomp $line;
}

foreach $time_check (sort keys %time_checks) {
print "\n\n$time_check:\n";
my $check_values = $time_checks{$time_check};
print " $_: $check_values->{$_}\n" foreach (keys %$check_values);
}

__DATA__
2004-03-26 @ 00:00:01 sql_er_status@pinhpp31 -- 10881 10864 hyb01
:INFORMATIONAL
2004-03-26 @ 00:00:01 sql_er_status@pinhpp31 -- 10881 10864 hyb01 Function
(up) returned (0) return code. -- seconds -- 1:INFORMATIONAL
Variable_name Value
Aborted_clients 5592
Aborted_connects 4
Bytes_received 500
Bytes_sent 5000
Com_admin_commands 0
Com_alter_table 27
Com_analyze 0
Com_backup_table 0
2004-03-26 @ 00:15:01 sql_er_status@pinhpp31 -- 10881 10864 hyb01
:INFORMATIONAL
2004-03-26 @ 00:15:01 sql_er_status@pinhpp31 -- 10881 10864 hyb01 Function
(up) returned (0) return code. -- seconds -- 1:INFORMATIONAL
Variable_name Value
Aborted_clients 5592
Aborted_connects 4
Bytes_received 2678
Bytes_sent 6935
Com_admin_commands 0
Com_alter_table 29
Com_analyze 0
Com_backup_table 0
2004-03-26 @ 00:30:01 sql_er_status@pinhpp31 -- 10881 10864 hyb01
:INFORMATIONAL
2004-03-26 @ 00:30:01 sql_er_status@pinhpp31 -- 10881 10864 hyb01 Function
(up) returned (0) return code. -- seconds -- 1:INFORMATIONAL
Variable_name Value
Aborted_clients 5592
Aborted_connects 4
Bytes_received 6023
Bytes_sent 10001
Com_admin_commands 0
Com_alter_table 15
Com_analyze 0
Com_backup_table 0

2004-03-26 @ 00:00:01:
Bytes_received: 500
Com_analyze: 0
Bytes_sent: 5000
Com_alter_table: 27
Com_backup_table: 0
Com_admin_commands: 0
Aborted_clients: 5592
Aborted_connects: 4

2004-03-26 @ 00:15:01:
Bytes_received: 2678
Com_analyze: 0
Bytes_sent: 6935
Com_alter_table: 29
Com_backup_table: 0
Com_admin_commands: 0
Aborted_clients: 5592
Aborted_connects: 4

2004-03-26 @ 00:30:01:
Bytes_received: 6023
Com_analyze: 0
Bytes_sent: 10001
Com_alter_table: 15
Com_backup_table: 0
Com_admin_commands: 0
Aborted_clients: 5592
Aborted_connects: 4

Sponsored Links