Home > Archive > PERL Beginners > October 2006 > Advice: Perl Multidimensional Array/Hash vs Relational Database (MySQL)?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Advice: Perl Multidimensional Array/Hash vs Relational Database (MySQL)?
|
|
| jplee3@gmail.com 2006-10-05, 6:58 pm |
| Hi all,
I've been mulling over the best solution to implement for running
reports on Windows event logs. Basically, the current setup we have is
that a bunch of Windows event logs are being logged (using evntslog) to
a central logging Linux server.These logs grow to 1.5gb+ daily with
tens of thousands of lines of text, and need to be reviewed on a daily
basis. Currently, I've been using small Perl and shell scripts to parse
data and manually review the logs. But soon I'll need to have daily and
monthly reports that essentially display alerts and statistics.
My initial instinct is to either dump all the log file text, or
parsed portions of it that I want/need, into a database and manipulate
it from there. It seems like it would be a whole lot easier and more
efficient.
The other route I've been researching is doing *everything* in Perl
- but this seems like it would be pretty expensive on memory and
performance; especially with processing thousands upon thousands of
lines of text into arrays and or hashes.
The problem is that my managers generally don't want to introduce
more 'unnecessary' stuff into the current environment. Ultimately, all
I need to do is get the reports into a spreadsheet/Excel type format
and have them uploaded to our Sharepoint site.
What are your thoughts? Any advice? Preferences?
TIA
| |
| usenet@DavidFilmer.com 2006-10-05, 6:58 pm |
| jplee3@gmail.com wrote:
> My initial instinct is to either dump all the log file text, or
> parsed portions of it that I want/need, into a database and manipulate
> it from there. It seems like it would be a whole lot easier and more
> efficient.
Maybe if you need to revisit the data. Since you must parse the logs
to create the database, why not just parse the logs to create the
report directly? Because this...
> I need to do is get the reports into a spreadsheet/Excel type format
> and have them uploaded to our Sharepoint site.
looks like a static report.
> The other route I've been researching is doing *everything* in Perl
> - but this seems like it would be pretty expensive on memory and
> performance; especially with processing thousands upon thousands of
> lines of text into arrays and or hashes.
Yeah, that's neither robust nor scalable.
> The problem is that my managers generally don't want to introduce
> more 'unnecessary' stuff into the current environment.
If you already havw a database server, it's not much to introduce. If
you don't, consider SQLite, which lets gives you (almost) full SQL
capability without needing a database server. It's not nearly as fast
as a real db, but it's robust and highly functional.
--
David Filmer (http://DavidFilmer.com)
| |
| jplee3@gmail.com 2006-10-05, 6:58 pm |
| usenet@DavidFilmer.com wrote:
> jplee3@gmail.com wrote:
>
> Maybe if you need to revisit the data. Since you must parse the logs
> to create the database, why not just parse the logs to create the
> report directly? Because this...
>
Mostly because they want to see numbers and figures - I can definitely
parse the data into XML or XLS (or whatever else) and display each log
entry in a more readable format, but they want to see the number of
occurrences along with high/mid/low priority events (and it would also
be nice to be able to link back to entire log entry, or the parsed
version of it at least). Doing something like this all in a script
would require the multidimensional array/hash implementations, right?
Of course, I suppose I could manipulate the data in XML or even
XLS/Excel but it seems like there would be a lot more tedious
workarounds required (i.e. using macros in Excel...)
>
> looks like a static report.
>
Yes, there should be a report for each day and then sort of a
'compilation' of all the reports each month.
>
> Yeah, that's neither robust nor scalable.
>
Right, I was playing around with a few scripts and thought to myself -
"this is unreasonably slow and tedious..."
>
> If you already havw a database server, it's not much to introduce. If
> you don't, consider SQLite, which lets gives you (almost) full SQL
> capability without needing a database server. It's not nearly as fast
> as a real db, but it's robust and highly functional.
>
I was initially just thinking about using Perl to parse the data and
insert to a mysql database, then doing the number crunching/report
generating either with Perl [again] or PHP and outputting it to a
static file [or mickeymouse-automating a way to get it uploaded to our
Sharepoint site. Not sure of all the ins and outs of Sharepoint...]
> --
> David Filmer (http://DavidFilmer.com)
|
|
|
|
|