Home > Archive > PERL Miscellaneous > March 2004 > Re: problem/help with using LWP
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Re: problem/help with using LWP
|
|
| Uri Guttman 2004-03-28, 10:22 pm |
| >>>>> "BW" == Bob Walton <invalid-email@rochester.rr.com> writes:
BW> Richard Bell wrote:[color=darkred]
BW> Then you should consider using a web scaping proxy:
BW> http://www.research.att.com/~hpk/wsp/
and check out WWW::Mechanize and its related modules. they make
following all those links and filling forms much easier. it is based on
LWP so you get all of that with many extra features designed to make
scraping and such very easy.
uri
--
Uri Guttman ------ uri@stemsystems.com -------- http://www.stemsystems.com
--Perl Consulting, Stem Development, Systems Architecture, Design and Coding-
Search or Offer Perl Jobs ---------------------------- http://jobs.perl.org
| |
| Richard Bell 2004-03-28, 10:22 pm |
| Thanks for responding.
As I mentioned in my original post, I've already used wsp (that's how
I got this far). The issue now is how to find out of a list of
roughly 50 or so cookies, which one is the session cookie that times
out. I've spent some time reading wsp's -v output and can see the
cookie traffic. What I'm not clear on is how to tell which one is
getting me in trouble.
BTW, when I installed wsp, I got some test errors and only got it to
run by forcing the make install. I'm running Fedora. Is this
expected behavior? Is forcing the make install an appropriate
solution?
I could also use some clues as to how to get a current session cookie
(I assume LWP to the page that originally dispenses it) and then place
it in the appropriate header line of the POST to actually get the data
I really want. Has anyone had any experience in this regard? What
are the problems/issues/techniques that I should be aware of?
I'll take another look at mechanize. I'd hoped to avoid working my
way through the entire form process as the form to POST to actually
get the data I want always has exactly the same content. The issue
seems to be purely one of the session cookie, not the form (or for
that matter the navigation).
BTW, I should add that I've recently returned to *nix and perl after
many years away, so I'm not experienced what most of what is new.
Please, forgive what may seem nieve questions.
Thanks again.
R
On Sun, 28 Mar 2004 04:39:59 GMT, Uri Guttman <uri@stemsystems.com>
wrote:
>
> BW> Richard Bell wrote:
>
>
> BW> Then you should consider using a web scaping proxy:
> BW> http://www.research.att.com/~hpk/wsp/
>
>and check out WWW::Mechanize and its related modules. they make
>following all those links and filling forms much easier. it is based on
>LWP so you get all of that with many extra features designed to make
>scraping and such very easy.
>
>uri
|
|
|
|
|