PERL Miscellaneous - Multiple File Grep

Author

Multiple File Grep

Blake

2004-07-28, 9:01 pm

I'm trying to figure out how to grep a bunch of log files into one
file.

Basically I have virtual hosts set up like this

/home/user/site/log

There's about 85 like that. Within /log/ there is an access_log file

So what I want to do is to be able to grep out all the hits in a
certain hour to see who's killing the server.

So what I'd like to do is do something like

find /home/ -name access_log
while { there's a log }
grep July 27 10pm
send that output to a single file

Then I can grep out the hits from that one file to see who's kiling
me.

What's the best what to do that?

Josef Moellers

2004-07-28, 9:01 pm

Blake wrote:
> I'm trying to figure out how to grep a bunch of log files into one
> file.
>=20
> Basically I have virtual hosts set up like this
>=20
> /home/user/site/log
>=20
> There's about 85 like that. Within /log/ there is an access_log file
>=20
> So what I want to do is to be able to grep out all the hits in a
> certain hour to see who's killing the server.
>=20
> So what I'd like to do is do something like
>=20
> find /home/ -name access_log
> while { there's a log }
> grep July 27 10pm=20
> send that output to a single file
>=20
> Then I can grep out the hits from that one file to see who's kiling
> me.
>=20
> What's the best what to do that?

I'd use File::Find.

--=20
Josef M=F6llers (Pinguinpfleger bei FSC)
If failure had no penalty success would not be a prize
-- T. Pratchett

ctcgag@hotmail.com

2004-07-28, 9:01 pm

phpuser@chek.com (Blake) wrote:
> I'm trying to figure out how to grep a bunch of log files into one
> file.
>
> Basically I have virtual hosts set up like this
>
> /home/user/site/log
>
> There's about 85 like that. Within /log/ there is an access_log file
>
> So what I want to do is to be able to grep out all the hits in a
> certain hour to see who's killing the server.
>
> So what I'd like to do is do something like
>
> find /home/ -name access_log
> while { there's a log }
> grep July 27 10pm
> send that output to a single file
>
> Then I can grep out the hits from that one file to see who's kiling
> me.
>
> What's the best what to do that?

system q{
grep 'July 27 10pm' /home/*/site/log/access_log > a_single_file
}q

Xho

--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB

Greg Bacon

2004-07-28, 9:01 pm

In article <6ca67b3b.0407270521.1e99ec8e@posting.google.com>,
Blake <phpuser@chek.com> wrote:

: [...]
: So what I want to do is to be able to grep out all the hits in a
: certain hour to see who's killing the server.
:
: So what I'd like to do is do something like
:
: find /home/ -name access_log
: while { there's a log }
: grep July 27 10pm
: send that output to a single file
:
: Then I can grep out the hits from that one file to see who's kiling
: me.

Consider the code below. Example usage:

% ghrp /home 27/Jul/2004:22

#! /usr/local/bin/perl

# ghrp: search for an hour and print

use warnings;
use strict;

sub usage { "Usage: $0 root-dir time-pattern\n" }

my %mon = (
Jan => 1, Feb => 2, Mar => 3, Apr => 4, May => 5, Jun => 6,
Jul => 7, Aug => 8, Sep => 9, Oct => 10, Nov => 11, Dec => 12,
);

sub date {
my $date = shift;

my($d,$m,$y,$hr,$min,$sec);

# e.g., 03/Feb/1998:17:42:15 -0500
if (m!(\d+)/(\w+)/(\d+):(\d+):(\d+):(\d+)!) {
($d,$m,$y,$hr,$min,$sec) = ($1,$2,$3,$4,$5,$6);

$m = $mon{$m} || 0;
}
else {
$d = $m = $y = $hr = $min = $sec = 0;
}

($d,$m,$y,$hr,$min,$sec);
}

sub date_asc {
$a->[2] <=> $b->[2] # year
||
$a->[1] <=> $b->[1] # month
||
$a->[0] <=> $b->[0] # day
||
$a->[3] <=> $b->[3] # hour
||
$a->[4] <=> $b->[4] # min
||
$a->[5] <=> $b->[5] # sec
}

## main
die usage unless @ARGV == 2;

my $root = shift;
die "$0: '$root' is not a directory!\n" . usage unless -d $root;

(my $time = shift) =~ s,/,\\/,g;
my $pat = eval "qr/" . $time . "/";

unless (defined $pat) {
die "$0: bad time pattern\n";
}

# from http://stein.cshl.org/WWW/docs/handout.html#Log_Parsing
my $line = qr/^\S+ \S+ \S+ \[($pat[^]]+)\] "\w+ \S+.*" \d+ \S+/;

my @hits;
for (`find $root -name access_log 2>&1`) {
chomp;

# assume this line is a warning if it's not a filename
unless (-f $_) {
warn $_ . "\n";
next;
}

my $fh;
unless (open $fh, "<", $_) {
warn "$0: open $_: $!\n";
next;
}

while (<$fh> ) {
push @hits, [ date($1), $_ ] if /$line/;
}
}

print $_ for map $_->[6], sort date_asc @hits;

__END__

Hope this helps,
Greg
--
The logic is flawless: when a private business *accidentally* kills 146
people, we need to increase the power of the government, an entity that
*deliberately* kills millions.
-- Gene Callahan

Jim Gibson

2004-07-28, 9:01 pm

In article <6ca67b3b.0407270521.1e99ec8e@posting.google.com>, Blake
<phpuser@chek.com> wrote:

> I'm trying to figure out how to grep a bunch of log files into one
> file.
>
> Basically I have virtual hosts set up like this
>
> /home/user/site/log
>
> There's about 85 like that. Within /log/ there is an access_log file
>
> So what I want to do is to be able to grep out all the hits in a
> certain hour to see who's killing the server.
>
> So what I'd like to do is do something like
>
> find /home/ -name access_log
> while { there's a log }
> grep July 27 10pm
> send that output to a single file
>
> Then I can grep out the hits from that one file to see who's kiling
> me.
>
> What's the best what to do that?

You don't always have to use Perl. If you are using Unix:

#!/bin/csh
set date = `date '+%b %d %l%p'`
find /home -name access_log -exec -H grep "$date" {} \;

(and direct output of this shell script to a file).

Sponsored Links