PERL POE - Client:DNS under heavy load

Author

Client:DNS under heavy load

Rohan R. Almeida

2005-10-26, 8:03 am

Not sure if my earlier mail was sent successfully, so re-sending.
Apologies for this.

Hi,

I need to resolve several thousands of IPs in a relatively short time.
I'm going to split the IPs into batches and spawn a component for
these particular IPs.

So, assuming I have 1000 IPs, I plan to create 10 components
(My::Resolver) , each of which will have the job of resolving 100 IP
addresses. These 10 components will be running parallelly (inside the
POE kernel, no threads)

Is it recommended to have a Client::DNS component for each
My::Resolver or would it matter if I just spawned a single Client::DNS
component and used it for all the 10 My::Resolver components?

Additionally, can POE::Wheels be used here for any kind of performance
advantage?

Thanks for reading. I would appreciate any pointers.

--
Rohan
http://rohan.almeida.in

Rocco Caputo

2005-11-03, 7:07 pm

On Oct 26, 2005, at 08:26, Rohan R. Almeida wrote:
>
> I need to resolve several thousands of IPs in a relatively short time.
> I'm going to split the IPs into batches and spawn a component for
> these particular IPs.
>
> So, assuming I have 1000 IPs, I plan to create 10 components
> (My::Resolver) , each of which will have the job of resolving 100 IP
> addresses. These 10 components will be running parallelly (inside the
> POE kernel, no threads)

DNS lookups are usually performed over UDP. Your bottleneck will be
the number of UDP packets you can have in play before a network node
between you and the name server begins dropping them. Lost DNS
packets result in timeouts. You should be prepared to retry requests
that time out.

The pattern that seems to work best is to send out a limited number
of requests, say 10-20 initially. The number depends on how many
requests you can have in play at any given moment. As each response
(or timeout) arrives, you send out a new request and then process the
response you got. The assumption is that DNS lookup time is your
bottleneck, so you want the new request in play while you take time
to deal with the current response.

> Is it recommended to have a Client::DNS component for each
> My::Resolver or would it matter if I just spawned a single Client::DNS
> component and used it for all the 10 My::Resolver components?

Each Client::DNS object encapsulates a remote resolver. You should
only need one per name server.

> Additionally, can POE::Wheels be used here for any kind of performance
> advantage?

POE::Wheel::Run could be used to run host or nslookup 1000 times, but
I don't recommend it. There may be opportunity for optimizing things
you haven't talked about yet, but obviously I can't comment on that. :)

--
Rocco Caputo - rcaputo@pobox.com

Sponsored Links