Browsing a site and checking/posting webmail using perl...

I am using RH9, postgreSQL and perl5. I would like to get my system to autom
atically visit a site, go to the login url, login, browse to the mail/messag
es link and get the messages one by one and save them in a postgreSQL databa
se.

Also, I would like my system to post messages from the database through the 
webmail interfaces of these sites.

I would like to do this with multiple sites. Is this possible with perl? Wha
t modules do I need? Are any readymade scripts that can be adapted to my nee
ds available?

All/any help/pointers greatly appreciated.

With best regards.
Sanjay.

On Wed, 4 Aug 2004, Sanjay Arora wrote:

> I am using RH9, postgreSQL and perl5. I would like to get my system to
> automatically visit a site, go to the login url, login, browse to the
> mail/messages link and get the messages one by one and save them in a
> postgreSQL database.

Okay. Do you have any code do do any of this?

If not, take a look at the CPAN modules DBI (you should already have
this), DBD::Pg (the PostgreSQL database driver), and LWP (available most
easily in cpan as Bundle::LWP). Also, WWW::Mechanize might be useful.
For sending mail there's many options, but Mail::Simple might be easy.

Get these installed using the CPAN shell, then look over the docs & try
some starter code. If you hit any stumbling blocks, let the list know.

> I would like to do this with multiple sites. Is this possible with
> perl? What modules do I need? Are any readymade scripts that can be
> adapted to my needs available?

Yes this is all possible, and done regularly by many people. The catch
is that everyone who does this has slightly different needs, so code for
doing all of this may not be available in one place. So, like I say,
look over the documentation for these modules and try some code to do
aspects of your project, and then start plugging it all together.



--
Chris Devers      cdevers@pobox.com
http://devers.homeip.net:8080/blog/

np: 'Mr. Lucky'
by Henry Mancini
from 'The Best Of Mancini'

Thanks for the tips, Chris. Ill start looking at the documentation of
the code you pointed.

On Wed, 2004-08-04 at 17:14, Chris Devers wrote:
> On Wed, 4 Aug 2004, Sanjay Arora wrote:
> 
>
> Okay. Do you have any code do do any of this?
>
> If not, take a look at the CPAN modules DBI (you should already have
> this), DBD::Pg (the PostgreSQL database driver), and LWP (available most
> easily in cpan as Bundle::LWP). Also, WWW::Mechanize might be useful.
> For sending mail there's many options, but Mail::Simple might be easy.
>
BTW, its not SMTP mail I want, its webmail of the particular web-sites.
If you have any seperate pointers for this, please let me know.

> Get these installed using the CPAN shell, then look over the docs & try
> some starter code. If you hit any stumbling blocks, let the list know.
> 
>
> Yes this is all possible, and done regularly by many people. The catch
> is that everyone who does this has slightly different needs, so code for
> doing all of this may not be available in one place. So, like I say,
> look over the documentation for these modules and try some code to do
> aspects of your project, and then start plugging it all together.
>
If you know of any pieces of code, you mentioned as being scattered on
the net, please let me know, if its not too much of any effort. You see
I am newbie at perl. In fact, I code with books open in front of me and
couple of helping web-site tutorials open on my screen....referring to
them every two minutes. Understanding how somebody has done something
for another website will go a long way for my coding effort. IAC, I am
talking before I have even started....guess its my nervousness as a
newbie speaking.

Ill start looking at the documentation you mentioned and revert to the
list, in case I have any problems.

Thanks again.

With best regards.
Sanjay.

Sanjay -- please direct all replies to the list, not me directly.

On Wed, 4 Aug 2004, Sanjay Arora wrote:

> On Wed, 2004-08-04 at 17:14, Chris Devers wrote: 

> BTW, its not SMTP mail I want, its webmail of the particular
> web-sites. If you have any seperate pointers for this, please let me
> know.

Nothing comes to mind, sorry. If it's (say) hotmail or yahoo that you
want, there may be code out there, but I don't know where. Also,
anything that gets information this way is a "screen scraper", which is
to say that the next time the webmail provider changes their interface,
any software that tries to work with their site is probably going to
break. If it's possible to get at the same mail some other way -- via
POP or IMAP -- that's almost definitely going to be more reliable.

> If you know of any pieces of code, you mentioned as being scattered on
> the net, please let me know, if its not too much of any effort.

Sorry, I don't have any at hand. Like I say, this sort of thing is done
often, but everyone writing such a system has different requirements so
the particulars become significant.

I suspect that you'll be much better off by breaking your project down
into simpler components -- store & retrieve information from the
database, get & post information to & from web sites, etc -- and then
once you have these working, try to get them going in tandem.

This will, almost as a side effect, get you writing nice, reusable,
modular, easy to maintain code.


Let the list know if you have more specific questions. :)



--
Chris Devers      cdevers@pobox.com
http://devers.homeip.net:8080/blog/

np: 'Timothy'
by Henry Mancini
from 'The Best Of Mancini'