Home > Archive > PERL Miscellaneous > April 2007 > Readable/writable database in Perl
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Readable/writable database in Perl
|
|
|
| Hello,
Currently, our code is utilizing a database that is neither humanly
readable or writable. My task is to modify the existing code so that
the database is both. A sample of the code is below - in this
particular example, to request a mount point. (It does some other
stuff, such as incrementing/decrementing counts, etc., but that can be
ignored in relation to this request for assistance.)
I'm not looking for the solution in the posting (that wouldn't be any
fun :); what I am looking for is some initial direction on how I might
go about accomplishing this. There are no shortage of database related
core and CPAN modules. In particular, I was told to use the 'mySQL'
approach (if that means anything to anybody).
Anyone have an idea on how I might go about this task?
Thanks to advance to all that respond.
Eric
vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
vvvvvv
sub RequestMountPoint {
my $self = shift;
my $protocol = shift;
my $bldNum = shift;
my $mntPnt = undef;
my $mntDBfile = $self->Config->HomeDir()."/mntDBfile";
my $mntsDB = {};
unless (open SEMAPHORE, "> /tmp/mnt.lock") {
$self->Config->Env->ReleaseMach();
die "problem allocating semaphore";
}
flock SEMAPHORE, Fcntl::LOCK_EX;
tie( %$mntsDB, "MLDBM", $mountsDBFile, O_CREAT|O_RDWR, 0666,
$DB_File::DB_BTREE );
my $mountPoint = $self->Config->MountPointCount();
for (my $label = 0; $label < $mountPoint; $label++){
$mntPnt = "mntpnt".$label;
unless (defined($mntsDB->{$mntPnt})){
$mntsDB->{$mntPnt} = {
BldNum => $bldNum,
Protocol => $protocol,
RefCnt => 1,
};
last;
}
if (($mntsDB->{$mntPnt}->{BldNum} == $bldNum) and
($mntsDB->{$mntPnt}->{Protocol} eq $protocol)) {
$mntsDB->{$mntPnt}->{RefCnt}++;
last;
}
$mntPnt = undef;
}
my $ref = $mntsDB->{$mntPnt}->{RefCnt};
untie(%$mntsDB);
close(SEMAPHORE);
return $mntPnt, $ref;
}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^
| |
| xhoster@gmail.com 2007-04-26, 7:03 pm |
| Eric <ecarlson@vmware.com> wrote:
> Hello,
>
> Currently, our code is utilizing a database that is neither humanly
> readable or writable. My task is to modify the existing code so that
> the database is both.
There is are good reasons that no "real" databases use human readable
and even more so, human writable) data files.
What features of real databases are you willing to surrender to accomplish
this aim? Concurrency? Isolation? Atomicity? Durability? Performance
scalability with increasing database size? All of the above?
> A sample of the code is below - in this
> particular example, to request a mount point. (It does some other
> stuff, such as incrementing/decrementing counts, etc., but that can be
> ignored in relation to this request for assistance.)
If you want us to ignore code, then you should remove before you post it,
rather than just posting it and asking us to ignore it.
> I'm not looking for the solution in the posting (that wouldn't be any
> fun :); what I am looking for is some initial direction on how I might
> go about accomplishing this. There are no shortage of database related
> core and CPAN modules. In particular, I was told to use the 'mySQL'
> approach (if that means anything to anybody).
MySQL does not use human readable or writable data files. Well, unless you
are an extraordinarily gifted human.
> Anyone have an idea on how I might go about this task?
Sure. Step one, specify exactly what the task is, including the answers to
the questions I asked above about you are willing to sacrifice.
Step two, wait for step one.
Xho
--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
| |
| Brian McCauley 2007-04-26, 7:03 pm |
| On Apr 26, 6:08 pm, Eric <ecarl...@vmware.com> wrote:
> Currently, our code is utilizing a database that is neither humanly
> readable or writable. My task is to modify the existing code so that
> the database is both.
> I'm not looking for the solution in the posting (that wouldn't be any
> fun :); what I am looking for is some initial direction on how I might
> go about accomplishing this. There are no shortage of database related
> core and CPAN modules. In particular, I was told to use the 'mySQL'
> approach (if that means anything to anybody).
Well, if you want to use a human readable database you'd do better to
use a database module that uses flat files, like say DBD::CSV or
DBD::AnyData.
| |
|
| Xho,
Thanks for your reply to my posting. My response inline:
> There is are good reasons that no "real" databases use human readable
> and even more so, human writable) data files.
I'm no expert in databases, period. But I did point out to my group
that allowing someone to edit a database might corrupt it. I am only
following instructions on this one.
> What features of real databases are you willing to surrender to accomplish
> this aim? Concurrency? Isolation? Atomicity? Durability? Performance
> scalability with increasing database size? All of the above?
As stated, I am not an expert in databases. So I don't know what we
are willing to give up. Having said that however, the database is will
never grow to great bounds. So I don't think we'd be sacrificing a
whole lot.
> If you want us to ignore code, then you should remove before you post it,
> rather than just posting it and asking us to ignore it.
In the past I have been criticized for posting to this group without
including all of the code. Seems like when I leave something out or
try to simplify things, people seem to focus on that rather than the
question I am asking. So I can't seem to win with that one.
> MySQL does not use human readable or writable data files. Well, unless you
> are an extraordinarily gifted human.
That's good information. Perhaps the person giving me this direction
is unaware of this and we need to go a different route.
> Sure. Step one, specify exactly what the task is, including the answers to
> the questions I asked above about you are willing to sacrifice.
The task is to have a humanly readable/writable database. I don't know
what we'd be willing to sacrifice, but as mentioned already, the
database will be small by database standards.
If we can't have what we want in a database, then I was told that we
could create simple text files instead.
Eric
xhoster@gmail.com wrote:
> Eric <ecarlson@vmware.com> wrote:
>
> There is are good reasons that no "real" databases use human readable
> and even more so, human writable) data files.
>
> What features of real databases are you willing to surrender to accomplish
> this aim? Concurrency? Isolation? Atomicity? Durability? Performance
> scalability with increasing database size? All of the above?
>
>
>
> If you want us to ignore code, then you should remove before you post it,
> rather than just posting it and asking us to ignore it.
>
>
> MySQL does not use human readable or writable data files. Well, unless you
> are an extraordinarily gifted human.
>
>
> Sure. Step one, specify exactly what the task is, including the answers to
> the questions I asked above about you are willing to sacrifice.
>
> Step two, wait for step one.
>
> Xho
>
> --
> -------------------- http://NewsReader.Com/ --------------------
> Usenet Newsgroup Service $9.95/Month 30GB
| |
|
| Thanks for this direction, Brian. I will certainly look into this.
Eric
Brian McCauley wrote:
> On Apr 26, 6:08 pm, Eric <ecarl...@vmware.com> wrote:
>
>
> Well, if you want to use a human readable database you'd do better to
> use a database module that uses flat files, like say DBD::CSV or
> DBD::AnyData.
| |
| Sherm Pendley 2007-04-26, 7:03 pm |
| Eric <ecarlson@vmware.com> writes:
> Xho,
>
> Thanks for your reply to my posting. My response inline:
That's the normal way of writing responses. No need to point it out. :-)
>
> In the past I have been criticized for posting to this group without
> including all of the code. Seems like when I leave something out or
> try to simplify things, people seem to focus on that rather than the
> question I am asking. So I can't seem to win with that one.
Sure you can. If you were criticized in the past, it was for not including
all of the code that was *relevant to your question*. That often means you
need to write a separate, stand-alone test script that illustrates whatever
problem you're having, but isn't loaded down with a bunch of other code,
and that's a good thing!
In the process of weeding out what's relevant and writing the test script,
you'll often find that you've solved the problem yourself. So it's not just
a matter of appeasing a cranky group, it's a good habit that will help you
become a better, more self-sufficient programmer.
>
> That's good information. Perhaps the person giving me this direction
> is unaware of this and we need to go a different route.
That depends on what they mean by "human readable."
MySQL's data files are not, by themselves, very readable. But, it does provide
a generic interface. So, other programs can be written that use the same data,
or the "raw" SQL interface can be used to run ad-hoc queries. So it's human
readable in the sense that this one particular program you're writing won't
be the only way to work with the data.
sherm--
--
Web Hosting by West Virginians, for West Virginians: http://wv-www.net
Cocoa programming in Perl: http://camelbones.sourceforge.net
| |
| Sherm Pendley 2007-04-26, 7:03 pm |
| Eric <ecarlson@vmware.com> writes:
Upside-down - please don't do that! English reads from top to bottom.
> Brian McCauley wrote:
>
> Thanks for this direction, Brian. I will certainly look into this.
Replies belong below the quoted text, like this.
sherm--
--
Web Hosting by West Virginians, for West Virginians: http://wv-www.net
Cocoa programming in Perl: http://camelbones.sourceforge.net
| |
|
|
Sherm Pendley wrote:
> Eric <ecarlson@vmware.com> writes:
>
> Upside-down - please don't do that! English reads from top to bottom.
>
>
> Replies belong below the quoted text, like this.
>
> sherm--
>
Unless you're replying to an email, in which case the reply text goes
at the top before the replied to text. :)
Eric
| |
| xhoster@gmail.com 2007-04-26, 7:03 pm |
| Eric <ecarlson@vmware.com> wrote:
> Xho,
>
> Thanks for your reply to my posting. My response inline:
>
>
> I'm no expert in databases, period. But I did point out to my group
> that allowing someone to edit a database might corrupt it. I am only
> following instructions on this one.
>
>
> As stated, I am not an expert in databases. So I don't know what we
> are willing to give up. Having said that however, the database is will
> never grow to great bounds. So I don't think we'd be sacrificing a
> whole lot.
Then something that reads the entire file in and writes the entire file
back out when done, like Config::Simple, might be a good option. Or just
using Data::Dumper as a serializer, if the humans who will be reading it
are Perl people and like that Perlish format for data.
>
>
> In the past I have been criticized for posting to this group without
> including all of the code. Seems like when I leave something out or
> try to simplify things, people seem to focus on that rather than the
> question I am asking. So I can't seem to win with that one.
The keys is that while the code you post should be "real" in the sense
that it can be run and in doing so it demonstrates the issues at hand, it
doesn't need to be identical to the end-goal code you are working on. Make
a reduced copy of the code that gets rid of all the stuff that you are
confident is irrelevant to the topic at hand. I do that not just for
posting, but also for my own testing purposes.
>
>
> That's good information. Perhaps the person giving me this direction
> is unaware of this and we need to go a different route.
>
>
> The task is to have a humanly readable/writable database. I don't know
> what we'd be willing to sacrifice, but as mentioned already, the
> database will be small by database standards.
You need to figure this out. The fact that it will be small mitigates
indexing/performance problems--you can just read and parse the whole thing
and then write out the whole thing each time. But it does little to
address the concurrency, atomicity, durability issues. You (or your boss,
since it seems you are mostly just following orders) absolutely must get a
handle on these things. Assuming the entire file will be re-written upon
any changes, then you need to figure out how to lock the file against one
Perl program accessing it while other Perl programs also accessing it,
between Perl and people accessing, and between people and other people
accessing. Will people change the file only while the program is quescient
or not running at all, or will they change it at any time? How will the
program know a change has been made, and how will it respond? If a human
is holding a lock for a long time, what should the program do? (presumably
the programs won't need to hold locks for extended period, so that
shouldn't be as much as an issue the other way around.)
>
> If we can't have what we want in a database, then I was told that we
> could create simple text files instead.
That is almost by definition what you will need to do. While it might be
possible to build some fancy structures into a file and still have it
human readable (through creative use of white-space for example), I don't
think it would be possible to do so and still have it human writable. If
the human doesn't understand the significance of the white space, then it
would not be possible for them to edit the file without corrupting it.
So it is just a matter of what modules (if any) to use to read and write
the file. I'd turn that around. Instead of looking at every possible
module and asking if the format it uses is acceptable to the humans, ask
format do the human readers/writers want to use.
Xho
--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB
| |
| Uri Guttman 2007-04-27, 4:02 am |
| >>>>> "E" == Eric <ecarlson@vmware.com> writes:
E> Sherm Pendley wrote:[color=darkred]
E> Unless you're replying to an email, in which case the reply text goes
E> at the top before the replied to text. :)
no, that is still top posting. too bad redmond ruined the email world by
their breaking of a smart and long standing netiquette rule.
uri
--
Uri Guttman ------ uri@stemsystems.com -------- http://www.stemsystems.com
--Perl Consulting, Stem Development, Systems Architecture, Design and Coding-
Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org
| |
| Joe Smith 2007-04-27, 4:02 am |
| Eric wrote:
> Currently, our code is utilizing a database that is neither humanly
> readable or writable. My task is to modify the existing code so that
> the database is both.
If it is to be readable and writable by humans but read-only to machines,
then you could do what sendmail does.
The file /etc/mail/aliases has a very simple format - it is designed to
be read and written by the system administrator. But performing a
sequential search on the plain-text file is very inefficient. So the
sendmail package comes with a 'newaliases' command. That command reads
/etc/mail/aliases and creates /etc/mail/aliases.db, which is a database
optimized for doing fast lookups. The master file is the plain-text
one and the database is derived from it.
An alternative is to have the database be the definitive version.
Whenever the data needs to be edited by a human, the entire database
could be converted to a temporary plain-text file. Then when the
human is finished editing, the database can be re-created from the
modified plain-text file. (You'd have to have some sort of locking
mechanism to ensure that no machine-generated updates are allowed
while the human is editing.)
Back in the days of Perl 4, I had a program that kept track of files
and their checksums so that I could avoid duplicates and avoid doing
unnecessary I/O to magneto-optical disks. During it development, I
made sure it had an 'export' command to dump the data into a flat file,
and an 'import' command to rebuild the dbm file from it. The flat file
would exist just long enough to do a global search-and-replace, and would
be deleted as soon as the database of binary data was rebuilt.
-Joe
| |
| Joe Smith 2007-04-27, 4:02 am |
| Eric wrote:
> Sherm Pendley wrote:
>
> Unless you're replying to an email, in which case the reply text goes
> at the top before the replied to text. :)
Only pointy-haired bosses forwarding messages with "Here, you take are of this"
are allowed to do that.
| |
| Dr.Ruud 2007-04-27, 8:02 am |
| Eric schreef:
> In the past I have been criticized for posting to this group without
> including all of the code.
Impossible. You should only share a complete (meaning compilable AND
runnable, so with test data, and using warnings AND strictures) script,
never "all of the code".
--
Affijn, Ruud
"Gewoon is een tijger."
|
|
|
|
|