DBI + XML::Twig conflict (?)

I have a strange problem where a database handle is being destroyed for no
apparent reason. My initial idea was to parse some xml, and translate it
into a database. Deleting line after line of code I came up with this short
meaningles program which exhibits the same behavior as its real-life
sibling (tested against two different mysql instalations on different
machines):

use warnings;
use strict;
use DBI;
use XML::Twig;

++$|;	#so I can see the progress in _dummy ()

my $dbh = DBI->connect ( "dbi:mysql:$random_db:localhost",
$user,
$pass,
{    AutoCommit => 0,
RaiseError => 1,
PrintError => 0,
}
);

unless ($dbh) {
print "Unable to connect:\n$DBI::errstr\n";
exit 1;
};

eval { $dbh->do ("DROP TABLE IF EXISTS some_nonexistant_table") };
die ($@) if $@;

my $objinfo_parser = XML::Twig->new (
twig_handlers => {
'Products/Product' => \&_dummy
}
);
$objinfo_parser->parseurl ('http://www.3btech.net/3btech/objinfo.xml');

$dbh->commit;
$dbh->disconnect;

exit 0;

sub _dummy {
my ($twig, $child) = @_;
print '.';
$twig->purge();
return 'dummy';
}

Note that I am not even doing any real operations on the MySQL side, thus I
don't care what database I connect to. I put the DROP TABLE there just to
make sure that commands are reaching the server, and yes I can see that in
the logs.
If I run the real program I end up stuffing about 480 out of roughly 510
products into a designated table and then the handle goes out to lunch with
the same error message.
I use the very same XML::Twig setup in another script that packs results
into a hash and returns it, and everything works flawlessly. Any help is
priceless, as after nearly two days playing with this I am totally lost...

Thanks

Peter

Please don't cross post.

Peter Rabbitson wrote:
> I have a strange problem where a database handle is being destroyed for no
> apparent reason. My initial idea was to parse some xml, and translate it
> into a database. Deleting line after line of code I came up with this shor
t
> meaningles program which exhibits the same behavior as its real-life
> sibling (tested against two different mysql instalations on different
> machines):
>
> use warnings;
> use strict;
> use DBI;
> use XML::Twig;
>
> ++$|;	#so I can see the progress in _dummy ()
>
> my $dbh = DBI->connect ( "dbi:mysql:$random_db:localhost",
>                          $user,
>                          $pass,
>                          {    AutoCommit => 0,
> 			      RaiseError => 1,
> 			      PrintError => 0,
>                          }
>                );
>
> unless ($dbh) {
>     print "Unable to connect:\n$DBI::errstr\n";
>     exit 1;
> };
>
> eval { $dbh->do ("DROP TABLE IF EXISTS some_nonexistant_table") };
> die ($@) if $@;
>
> my $objinfo_parser = XML::Twig->new (
>                           twig_handlers => {
>                                  'Products/Product' => \&_dummy
>                           }
>                    );
> $objinfo_parser->parseurl ('http://www.3btech.net/3btech/objinfo.xml');
>
> $dbh->commit;
> $dbh->disconnect;
>
> exit 0;
>
> sub _dummy {
>     my ($twig, $child) = @_;
>     print '.';
>     $twig->purge();
>     return 'dummy';
> }
>
> Note that I am not even doing any real operations on the MySQL side, thus 
I
> don't care what database I connect to. I put the DROP TABLE there just to
> make sure that commands are reaching the server, and yes I can see that in
> the logs.
> If I run the real program I end up stuffing about 480 out of roughly 510
> products into a designated table and then the handle goes out to lunch wit
h
> the same error message.

What's the error message?

http://danconia.org

> I use the very same XML::Twig setup in another script that packs results
> into a hash and returns it, and everything works flawlessly. Any help is
> priceless, as after nearly two days playing with this I am totally lost...
>
> Thanks
>
> Peter
>

> > If I run the real program I end up stuffing about 480 out of roughly 510
 
>
> What's the error message?
>

Ups... I guess I missed that:

~$ ./test > /dev/null
Issuing rollback() for database handle being DESTROY'd without explicit
disconnect().
DBD::mysql::db commit failed: MySQL server has gone away at ./test line 27.

Peter Rabbitson wrote: 
>
>
> Ups... I guess I missed that:
>
> ~$ ./test > /dev/null
> Issuing rollback() for database handle being DESTROY'd without explicit
> disconnect().
> DBD::mysql::db commit failed: MySQL server has gone away at ./test line 27
.
>
>

Thank you. No guarantees, but try setting 'InactiveDestroy' when you
create the DB handle. XML::Twig uses a fork/exec call in 'parseurl' to
retrieve the URL in one process and to parse the XML in the other. When
the retrieval is complete one of the processes closes with an 'exit'. I
think because the $dbh is shared (because it just is) you are getting
the above result. The switch I mentioned appears to be designed for this
specific case.

For more info see "InactiveDestroy" here:

[url]http://search.cpan.org/~timb/DBI-1.48/DBI.pm#ATTRIBUTES_COMMON_TO_ALL_HANDLES[/url
]

"This attribute is specifically designed for use in Unix applications
that "fork" child processes. Either the parent or the child process, but
not both, should set InactiveDestroy on all their shared handles."

HTH,

http://danconia.org

> Thank you. No guarantees, but try setting 'InactiveDestroy' when you
> create the DB handle. XML::Twig uses a fork/exec call in 'parseurl' to
> retrieve the URL in one process and to parse the XML in the other. When
> the retrieval is complete one of the processes closes with an 'exit'. I
> think because the $dbh is shared (because it just is) you are getting
> the above result. The switch I mentioned appears to be designed for this
> specific case.
>
> For more info see "InactiveDestroy" here:
>
> [url]http://search.cpan.org/~timb/DBI-1.48/DBI.pm#ATTRIBUTES_COMMON_TO_ALL_HANDLES[/u
rl]
>
> "This attribute is specifically designed for use in Unix applications
> that "fork" child processes. Either the parent or the child process, but
> not both, should set InactiveDestroy on all their shared handles."
>
> HTH,


Hm... I get it and I don't get it... Who keeps the sub _dummy - the parent
or the child? I need to use DBI in it so I guess InactiveDestroy must be set
to true there. How do I find out who is who not at the time but after the
fork? Is this portable to win32? (the final version must run on win32 as
well).
Actually from the little I know about forks I thought that both parent and
child get copies of the very same stuff. So both get a "hot" DBI and
DBD::Mysql tied to the same socket... if this is at all possible. On the
other hand I am not using DBI anymore while in the reader (child?), just the
parser, so it shouldn't matter... Wow I am even more  now :)

All above considered - I guess I would be safer downloading the file
somewhere and doing safe_parsefile to avoid any forking altogether.

Thanks for the hint

Peter Rabbitson wrote: 
>
>
>
> Hm... I get it and I don't get it... Who keeps the sub _dummy - the parent
> or the child? I need to use DBI in it so I guess InactiveDestroy must be s
et
> to true there. How do I find out who is who not at the time but after the
> fork? Is this portable to win32? (the final version must run on win32 as
> well).
> Actually from the little I know about forks I thought that both parent and
> child get copies of the very same stuff. So both get a "hot" DBI and
> DBD::Mysql tied to the same socket... if this is at all possible. On the
> other hand I am not using DBI anymore while in the reader (child?), just t
he
> parser, so it shouldn't matter... Wow I am even more  now :)
>

Ah, fun with IPC :-). The sub _dummy is the parent, because that is what
you are controlling, technically you aren't even supposed to be aware
that there is a fork (and weren't, really, until I told you :-)).  Yes
you are correct, everything is shared across the fork, the $dbh being
one of those things. Normally if you were controlling the fork/exec
model then you could decide to set InactiveDestroy in whichever place it
was needed, it would actually be the child as that is what is exiting
early (it does the retrieval). But in this case since the fork/exec is
encapsulated in a module you didn't write, you will likely want to set
it at connection time so that it is set on both of the handles. The key
in this case is that you have to do your explicit commit/disconnect
(which you were already). Theoretically another option would be to set
it during construction, then unset it in the parent after the fork,
assuming this works as I am thinking (caveat), then that might be better
as it should make sure to do the normal DBI cleanup. Of course back to
my caveat, since you don't really have control of when the fork is going
to exit this might be trickier. Though *I think* (no guarantee) that the
parser is reading from the pipe created for the fork, so as long as the
parser isn't finished the pipe and child should be active. Sorry if I am
confusing you again, the route I would take would be to set it during
construction because that is the simplest, then test, test, and test
some more. Try it different ways and see which works the best in the
tests. See if you can get it to break again.

Whether or not this works correctly on Windows I have no idea, and
wouldn't even venture to guess. You may want to read up on IPC on
windows and fork/exec for that platform.

> All above considered - I guess I would be safer downloading the file
> somewhere and doing safe_parsefile to avoid any forking altogether.
>

Possibly, but then that is more code you have to write, which can be
buggy, doesn't likely have as many people looking at and testing it,
etc.  Watch out for the slippery slope with this argument, I mean, you
could just write it all without modules, you could write it all in C,
you could write it in assembly! Boy what fun that would be.....

> Thanks for the hint
>
HTH, Good luck (and test some more),

http://danconia.org