AWK - Shell/Awk processing question

Author

Shell/Awk processing question

sf94061@gmail.com

2008-03-25, 9:59 pm

I have a text file that has tail output from multiple application log
files. The application log files are all from the same application,
but individual nodes running on various servers. The log information
is ssh'ed (tail -200...) from the remote servers to the local text
file.

Since the retrieval process isn't very refined, I'm stuck with log
output that spans several minutes, but I'm only interested in the
output from the past minute (date +%H:%M --date "1 minute ago"). I've
tried to write a small awk process that would go through the log file
every 5 minutes and capture the output from the past minute and write
it out to individual log files locally.

Here's the format of the log file...

==> file name <==
date_stamp | time_stam (which can be now minus several past minutes) |
trigger text
date_stamp | time_stam (which can be now minus several past minutes) |
text
more text
more text
date_stamp | time_stam (which can be now minus several past minutes) |
trigger text
date_stamp | time_stam (which can be now minus several past minutes) |
text
more text
more text
date_stamp | time_stam (which can be now minus several past minutes) |
trigger text
date_stamp | time_stam (which can be now minus several past minutes) |
text
more text
more text
==> file name <==
date_stamp | time_stam (which can be now minus several past minutes) |
trigger text
date_stamp | time_stam (which can be now minus several past minutes) |
text
more text
more text
date_stamp | time_stam (which can be now minus several past minutes) |
trigger text
date_stamp | time_stam (which can be now minus several past minutes) |
text
more text
more text
date_stamp | time_stam (which can be now minus several past minutes) |
trigger text
date_stamp | time_stam (which can be now minus several past minutes) |
text
more text
more text

________________________________________

some arbitrary text that
spans multiple lines since I'm stuck with the
tail -200 output. Followed by the meat of the
content
==> /dir-path/file1.out <==
2007-03-23 hh:mm:ss [num] channels
2007-03-23 17:33:01 Some text ....
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
Arbitrary text spanning
several lines...
2007-03-23 17:34:01 [num] channels
2007-03-23 17:34:01 Some text ....
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
Arbitrary text spanning
several lines...

==> /dir-path/file2.out <==
2007-03-23 hh:mm:ss [num] channels
2007-03-23 17:33:01 Some text ....
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
Arbitrary text spanning
several lines...
2007-03-23 17:34:01 [num] channels
2007-03-23 17:34:01 Some text ....
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
[Num] text I need
Arbitrary text spanning
several lines...
________________________________________

For every file output (==> file name <==) I would like to retrieve all
the information from the last minute and dump it to a separate file.

I have a sample awk but it doesn't process correctly.

awk '{
# Here I want to capture what tail output file I'm in.
if($2~/"file_name1"/){
filelabel="file1";
}
else if($2~/"file_name2"/){
filelabel="file2";
}
etc...

# Here I'm trying to say that I set a marker that indicates that I'm
at the correct position in the file and the time stamp is correc.
if(NF==4 && $2~timeval && $4~/channels/)
start="yes";
# Here I'm at a potentially correct position, but it's not the
correct time stamp ($2!~timeval)
else if(NF==4 && $2!~timeval && $4~/channels/)
start="no";

while(start=="yes"){
if($2~/chancount/){
start="no";
next;
}
else printf("%s %s\n",filelabel, $0);
}
}' timeval=`date +%H:%M --date "1 minute ago"` ${infile}

What am I doing wrong?

Thanks.

Emma

Bill Marcum

2008-03-26, 3:58 am

["Followup-To:" header set to comp.unix.shell.]
On 2008-03-26, sf94061@gmail.com <sf94061@gmail.com> wrote:
>
>
> I have a text file that has tail output from multiple application log
> files. The application log files are all from the same application,
> but individual nodes running on various servers. The log information
> is ssh'ed (tail -200...) from the remote servers to the local text
> file.
>
> Since the retrieval process isn't very refined, I'm stuck with log
> output that spans several minutes, but I'm only interested in the
> output from the past minute (date +%H:%M --date "1 minute ago"). I've
> tried to write a small awk process that would go through the log file
> every 5 minutes and capture the output from the past minute and write
> it out to individual log files locally.
>
> Here's the format of the log file...
>
>==> file name <==
> date_stamp | time_stam (which can be now minus several past minutes) |
> trigger text
> date_stamp | time_stam (which can be now minus several past minutes) |
> text
> more text
> more text
> date_stamp | time_stam (which can be now minus several past minutes) |
> trigger text
> date_stamp | time_stam (which can be now minus several past minutes) |
> text
> more text
> more text
> date_stamp | time_stam (which can be now minus several past minutes) |
> trigger text
> date_stamp | time_stam (which can be now minus several past minutes) |
> text
> more text
> more text
>==> file name <==
> date_stamp | time_stam (which can be now minus several past minutes) |
> trigger text
> date_stamp | time_stam (which can be now minus several past minutes) |
> text
> more text
> more text
> date_stamp | time_stam (which can be now minus several past minutes) |
> trigger text
> date_stamp | time_stam (which can be now minus several past minutes) |
> text
> more text
> more text
> date_stamp | time_stam (which can be now minus several past minutes) |
> trigger text
> date_stamp | time_stam (which can be now minus several past minutes) |
> text
> more text
> more text
>
> ________________________________________

>
> some arbitrary text that
> spans multiple lines since I'm stuck with the
> tail -200 output. Followed by the meat of the
> content
>==> /dir-path/file1.out <==
> 2007-03-23 hh:mm:ss [num] channels
> 2007-03-23 17:33:01 Some text ....
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> Arbitrary text spanning
> several lines...
> 2007-03-23 17:34:01 [num] channels
> 2007-03-23 17:34:01 Some text ....
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> Arbitrary text spanning
> several lines...
>
>==> /dir-path/file2.out <==
> 2007-03-23 hh:mm:ss [num] channels
> 2007-03-23 17:33:01 Some text ....
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> Arbitrary text spanning
> several lines...
> 2007-03-23 17:34:01 [num] channels
> 2007-03-23 17:34:01 Some text ....
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> [Num] text I need
> Arbitrary text spanning
> several lines...
> ________________________________________

>
> For every file output (==> file name <==) I would like to retrieve all
> the information from the last minute and dump it to a separate file.
>
> I have a sample awk but it doesn't process correctly.
>
> awk '{
> # Here I want to capture what tail output file I'm in.
> if($2~/"file_name1"/){
> filelabel="file1";
> }
> else if($2~/"file_name2"/){
> filelabel="file2";
> }
> etc...
>
> # Here I'm trying to say that I set a marker that indicates that I'm
> at the correct position in the file and the time stamp is correc.
> if(NF==4 && $2~timeval && $4~/channels/)
> start="yes";
> # Here I'm at a potentially correct position, but it's not the
> correct time stamp ($2!~timeval)
> else if(NF==4 && $2!~timeval && $4~/channels/)
> start="no";
>
> while(start=="yes"){
> if($2~/chancount/){
> start="no";
> next;
> }
> else printf("%s %s\n",filelabel, $0);
> }
Awk programs have an implied outer loop "read a record; test for
patterns; perform actions". This while loop occurs within an action,
so $2 never changes and start doesn't change unless ($2~/chancount/).

> }' timeval=`date +%H:%M --date "1 minute ago"` ${infile}
>
> What am I doing wrong?
>
> Thanks.
>
> Emma

Hermann Peifer

2008-03-26, 7:59 am

On Mar 26, 3:16=A0am, sf94...@gmail.com wrote:
>
> Since the retrieval process isn't very refined, I'm stuck with log
> output that spans several minutes, but I'm only interested in the
> output from the past minute (date +%H:%M --date "1 minute ago"). =A0I've
> tried to write a small awk process that would go through the log file
> every 5 minutes and capture the output from the past minute and write
> it out to individual log files locally.
>

This might be a start for your awk script:

BEGIN { past_minute =3D strftime("%F %H:%M",systime()-60) }

/^=3D=3D>/ { file_name =3D $2 ; gsub("/","_",file_name) }

substr($0,1,16) =3D=3D past_minute {
# Some logic if you want to print
# or ignore the following lines
# ...

print >> file_name
}

Hermann Peifer

2008-03-26, 7:59 am

On Mar 26, 1:12=A0pm, Hermann Peifer <pei...@gmx.net> wrote:
>
> This might be a start for your awk script:
>

I should have written: ...for your *g*awk script

Hermann

Maxwell Lol

2008-03-26, 7:03 pm

[color=darkred]

You can write a program that reads via tail -f filename, It can do
something special every time the timestamp increases to the next
rounded-off value.

Sponsored Links