PERL Beginners - WWW::Mechanize java navigation

Author

WWW::Mechanize java navigation

Peter Rabbitson

2005-02-24, 3:56 pm

Hello list. I had very good luck using the WWW::Mechanize object for various
automated data collectors until I stumbled over
http://www.mymerchantview.net. As you can imagine the problems begin when I
do my first ->follow_link after which the script blows with something about
unsupported java. What can I use that would be as functional as
WWW::Mechanize while at the same time compatible with the site mentioned
above. It would be more than appreciated if somebody can submit along a few
lines of code showing a "click" on the 'Log In' tab, a "click" on 'Continue'
over at the next page and submittal of three types of authentication info on
the successive screen.

Thanks everyone.

Peter

Peter Rabbitson

2005-02-24, 8:55 pm

use WWW::Mechanize;
use Crypt::SSLeay;

my $m_crawl = WWW::Mechanize->new;
$m_crawl->get ('http://mymerchantview.net');
$m_crawl->follow_link (n => '1');

print $m_crawl->content;

the above code yields:

501 Protocol scheme 'javascript' is not supported

On Thu, Feb 24, 2005 at 02:20:29PM -0600, perlmails@thinbrowser.com wrote:
> Have you installed Crypt::SSLeay for https:// ?
> Could you print the header at least with your code?
>
>
> ----- Original Message -----
> From: "Peter Rabbitson" <rabbit@rabbit.us>
> To: <beginners@perl.org>
> Sent: Thursday, February 24, 2005 11:45 AM
> Subject: WWW::Mechanize java navigation
>
>
>

Peter Rabbitson

2005-02-24, 8:55 pm

On Thu, Feb 24, 2005 at 03:11:08PM -0600, perlmails@thinbrowser.com wrote:

<---snip whole bunch of misunderstaning--->

> You have your answer then.. I guess.
>
> How can I get WWW::Mechanize to execute this JavaScript?
> You can't. JavaScript is entirely client-based, and WWW::Mechanize is a
> client that doesn't understand JavaScript.
>

<---/snip--->

Whereas my original question was:

What can I use that would be as functional as WWW::Mechanize while at the
same time compatible with the site mentioned above.

perlmails@thinbrowser.com

2005-02-24, 8:55 pm

Have you installed Crypt::SSLeay for https:// ?
Could you print the header at least with your code?

----- Original Message -----
From: "Peter Rabbitson" <rabbit@rabbit.us>
To: <beginners@perl.org>
Sent: Thursday, February 24, 2005 11:45 AM
Subject: WWW::Mechanize java navigation

> Hello list. I had very good luck using the WWW::Mechanize object for
> various
> automated data collectors until I stumbled over
> http://www.mymerchantview.net. As you can imagine the problems begin when
> I
> do my first ->follow_link after which the script blows with something
> about
> unsupported java. What can I use that would be as functional as
> WWW::Mechanize while at the same time compatible with the site mentioned
> above. It would be more than appreciated if somebody can submit along a
> few
> lines of code showing a "click" on the 'Log In' tab, a "click" on
> 'Continue'
> over at the next page and submittal of three types of authentication info
> on
> the successive screen.
>
>
> Thanks everyone.
>
> Peter
>
> --
> To unsubscribe, e-mail: beginners-unsubscribe@perl.org
> For additional commands, e-mail: beginners-help@perl.org
> <http://learn.perl.org/> <http://learn.perl.org/first-response>
>
>

perlmails@thinbrowser.com

2005-02-24, 8:55 pm

----- Original Message -----
From: "Peter Rabbitson" <rabbit@rabbit.us>
To: <beginners@perl.org>
Sent: Thursday, February 24, 2005 3:02 PM
Subject: Re: WWW::Mechanize java navigation

> use WWW::Mechanize;
> use Crypt::SSLeay;
>
> my $m_crawl = WWW::Mechanize->new;
> $m_crawl->get ('http://mymerchantview.net');
> $m_crawl->follow_link (n => '1');
>
> print $m_crawl->content;
>
>
> the above code yields:
>
> 501 Protocol scheme 'javascript' is not supported
>

You have your answer then.. I guess.
From FAQ=WWW::Mechanize;

How can I get WWW::Mechanize to execute this JavaScript?
You can't. JavaScript is entirely client-based, and WWW::Mechanize is a
client that doesn't understand JavaScript.

>
>
> On Thu, Feb 24, 2005 at 02:20:29PM -0600, perlmails@thinbrowser.com wrote:
>
> --
> To unsubscribe, e-mail: beginners-unsubscribe@perl.org
> For additional commands, e-mail: beginners-help@perl.org
> <http://learn.perl.org/> <http://learn.perl.org/first-response>
>
>

perlmails@thinbrowser.com

2005-02-24, 8:55 pm

----- Original Message -----
From: "Peter Rabbitson" <rabbit@rabbit.us>
To: <beginners@perl.org>
Sent: Thursday, February 24, 2005 3:45 PM
Subject: Re: WWW::Mechanize java navigation

>
>
>
> On Thu, Feb 24, 2005 at 03:11:08PM -0600, perlmails@thinbrowser.com wrote:
>
>
> <---snip whole bunch of misunderstaning--->
>
>
> <---/snip--->
>
> Whereas my original question was:
>
> What can I use that would be as functional as WWW::Mechanize while at the
> same time compatible with the site mentioned above.
>

Looks like one has to regexp the javascript link for the button?
Here is also somthing that might help,
http://www.easyya.com/lwp/LWP_form_javascript.html

> --
> To unsubscribe, e-mail: beginners-unsubscribe@perl.org
> For additional commands, e-mail: beginners-help@perl.org
> <http://learn.perl.org/> <http://learn.perl.org/first-response>
>
>

Todd W

2005-02-25, 3:56 am

"Peter Rabbitson" <rabbit@rabbit.us> wrote in message
news:20050224214511.GA5378@rabbit.us...
>
>
>
> On Thu, Feb 24, 2005 at 03:11:08PM -0600, perlmails@thinbrowser.com wrote:
>
>
> <---snip whole bunch of misunderstaning--->
>
>
> <---/snip--->
>
> Whereas my original question was:
>
> What can I use that would be as functional as WWW::Mechanize while at the
> same time compatible with the site mentioned above.

You have to read through the javascript by hand and see what it does, then
reset the href value of the link you are trying to click, then you can
do ->follow_link.

For example, my bank's web site does some javascript with some radio buttons
before setting the form's "action" property. So I looked through the
javascript to see what url I need the action set to for me, and I used the
WWW::Mechanize API to set the action with perl code before ->submit()ting
the form.

Either that or write some glue to get WWW::Mechanize to support javascript
;0).

Todd W.

Chris Devers

2005-02-25, 8:56 am

On Thu, 24 Feb 2005, Peter Rabbitson wrote:

> What can I use that would be as functional as WWW::Mechanize while at
> the same time [Javascript aware].

To do this properly, you need to have a proper Javascript engine. I'm
not aware of any Javascript hooks for Perl, but then, all modern web
browsers have support for it built in, if you can leverage that.

It occurs to me that the "easiest" way to do this -- for definitions of
"easy" meaning "heavy, a bit complex, but with most of the hard work
offloaded from you" -- would be to automate a regular web browser.

What platform is this going to be running on?

If you're on a Mac, you can do this with AppleScript &/or Mac::Glue,
which makes this pretty easy with most browsers -- you can just have the
system record your actions as you go through it once, and save the
results as an AppleScript script that can be run as-is, rewritten &
optimised, or redone & extended in Perl syntax with Mac::Glue.

I believe that there are similar options available on Windows (Windows
Script Host? OLE?) but I've never used them and can't comment on how
well they play with Perl. It's probably not too bad though.

On Linux or another Unix.... you may be stuck with cobbling everything
together yourself there, but then as a Linux user that's how things are
expected to work :-)

On any platform, you might be able to get some kind of Gecko / Mozilla /
Firefox / etc wrapper that uses the Gecko engine to do the work, and
that (presumably) would run as some kind of in-browser plugin. This
might actually be worth investigating, but it would probably involve
little if any Perl code.

But anyway, short of puzzling out the Javascript by hand -- which may
be sufficient for limited problems -- getting help from software that
can really "speak" Javascript is your best bet. Parsing this out with
regular expressions is just nuts -- there's way too many ways to express
the same concept in Javascript code, and anything you write that has to
be reasonably robust is going to be nearly as complex as a full-blown
Javascript engine would have been -- at which point you've just been
wasting your time.

Poking around on CPAN, I see possibly useful modules: JavaScript.pm
(copyright 2001; abandoned?) and java script::SpiderMonkey.pm (the readme
dates to 2002, but the package has releases from Aug 2004, which isn't
so bad).

<http://search.cpan.org/~mschilli/Ja...SpiderMonkey.pm>
<http://www.cpan.org/modules/by-module/JavaScript/>
<http://search.cpan.org/search?query...monkey&mode=all>

If you can get one of these to work, you can ignore all the stuff above
about automating a web browser :-)

--
Chris Devers

rnmscott@netspace.net.au

2005-02-25, 8:56 am

Quoting Peter Rabbitson <rabbit@rabbit.us>:
>
> Whereas my original question was:
>
> What can I use that would be as functional as WWW::Mechanize while at the
> same time compatible with the site mentioned above.
>
> --

I'd be interested in an answer to this too, as I have a similar sort of problem
(with javascript on a site).

Richard

------------------------------------------------------------
This email was sent from Netspace Webmail: http://www.netspace.net.au

Sponsored Links