PERL Miscellaneous - coarse-grained parallel processing of a for-loop

Author

coarse-grained parallel processing of a for-loop

guba@vi-anec.de

2007-06-28, 8:03 am

Hello,

I am searching for an easy way for a coarse-grained
parallel processing of a for-loop (image processing
with PerlMagick/Win on a QuadCore). Because no
communication is nedded the for-loop

for ($i = $count_min; $i =< $count_max; $i++) {
read selected source images
image composing
write result image on HD
};

could theoretically be splitted in 4 loops

$count_min..int($count_max/4)
int($count_max/4)+1..int($count_max/2)
int($count_max/2)+1..int(3*$count_max/4)
int(3*$count_max/4)+1..$count_max

that are processed in parallel (each on a Core).

cpan offers 300 results for "parallel" but I don't
know where to start. Thank you very much for hints!

Guenter

Peter Makholm

2007-06-28, 8:03 am

"guba@vi-anec.de" <guba@vi-anec.de> writes:

> I am searching for an easy way for a coarse-grained
> parallel processing of a for-loop (image processing
> with PerlMagick/Win on a QuadCore). Because no
> communication is nedded the for-loop

I don't know if Win32 is a problem, but I have used
Parallel::ForkManager for something alike

use Parallel::ForkManager;

my $pm = new Parallel::ForkManager($MAX_PROCESSES);
for $i ($count_min .. $count_max) {
my $pid = $pma->start and next;

read selected source images
image composing
write result image on HD

$pm->finish;
}
$pm->wait_all_children;

But taht gives you no way of communication from within the loop to the
main process.

> could theoretically be splitted in 4 loops
>
> $count_min..int($count_max/4)
> int($count_max/4)+1..int($count_max/2)
> int($count_max/2)+1..int(3*$count_max/4)
> int(3*$count_max/4)+1..$count_max

If all jobs are of equal length this would be ok but otherwise you
would be better of with some sort of work queue.

//Makholm

Dr.Ruud

2007-06-28, 7:05 pm

guba@vi-anec.de schreef:

> I am searching for an easy way for a coarse-grained
> parallel processing of a for-loop (image processing
> with PerlMagick/Win on a QuadCore). Because no
> communication is nedded the for-loop
>
> for ($i = $count_min; $i =< $count_max; $i++) {
> read selected source images
> image composing
> write result image on HD
> };
>
> could theoretically be splitted in 4 loops
>
> $count_min..int($count_max/4)
> int($count_max/4)+1..int($count_max/2)
> int($count_max/2)+1..int(3*$count_max/4)
> int(3*$count_max/4)+1..$count_max

See also the % operator (read `perldoc perlop`).

--
Affijn, Ruud

"Gewoon is een tijger."

xhoster@gmail.com

2007-06-28, 7:05 pm

"guba@vi-anec.de" <guba@vi-anec.de> wrote:
> Hello,
>
> I am searching for an easy way for a coarse-grained
> parallel processing of a for-loop (image processing
> with PerlMagick/Win on a QuadCore). Because no
> communication is nedded the for-loop
>
> for ($i = $count_min; $i =< $count_max; $i++) {
> read selected source images
> image composing
> write result image on HD
> };

Parallel::ForkManager might work well for this. It will do one fork
for each item in count_max (but not all at once, of course), so if the work
load for each individual image is very light, then the overhead of the
forking might come to dominate the workload. But I've never tried it on
Windows, so I don't know how it works there.

>
> could theoretically be splitted in 4 loops
>
> $count_min..int($count_max/4)
> int($count_max/4)+1..int($count_max/2)
> int($count_max/2)+1..int(3*$count_max/4)
> int(3*$count_max/4)+1..$count_max
>
> that are processed in parallel (each on a Core).

I frequently do something this like this, too, when each individual item
is light weight. But I don't know of any CPAN modules to facilitate it.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB

Sponsored Links