con con - 1 month ago 12
Perl Question

Parallel::ForkManager makes subroutine 1000x slower

I have a subroutine, which I've serially optimized as much as I can, approximately like

sub overlap {

my $hash_reference = shift; # pass the hash to the subroutine
my %h = %{ $hash_reference }; # refer to the hash as %h
my $standard = shift; # this is the key that will be compared against
my $compared = shift; # this is the key being compared
my $start_index = 0; # this will continually be increased
# to save computation time

# I want to parallelize here

foreach my $s ( 0 .. scalar @{ $h{$standard}{end} }-1 ) {
foreach my $c ( $start_index .. scalar @{ $h{$compared}{end} }-1 ) {
... # abbreviated for minimal working example
}
}

return ($standard_indices_met_in_compared, \@overlay);
}


This is a slow subroutine. I run it thousands of times in about 12-14 minutes, but running this again and again wastes time.

I regularly use
Parallel::ForkManager
for system processes, but this doesn't work well here.

Implementation of
Parallel::ForkManager
looks like

use Parallel::ForkManager qw();
my $manager = new Parallel::ForkManager(2);
foreach my $s ( 0 .. scalar @{ $h{$standard}{end} }-1 ) {

foreach my $c ( $start_index .. scalar @{ $h{$compared}{end} }-1 ) {
$manager->start and next;
... # abbreviated for minimal working example
}

$manager->finish;
}

$manager->wait_all_children; # necessary after all lists


I've looked at threads and such, but do not see how to apply here.

I have looked at Perl multithreading and foreach and the Perl documentation for threads, and numerous other sources, but I don't see how I can apply what has been done before in this case. Everything I see looks like it is for system commands only.

I want to write to a shared array and scalar, with no system commands. In case I'm missing something, please tell me.

How can I parallelize this
foreach
loop inside a subroutine?

Answer

Are you really trying to parallelize with a maximum of two processes only? If so, this may be the source of the perceived slowness.

There will always be an overhead associated with parallelization. You cannot guarantee a 10x speed-up if you parallelize over 10 processes.

I suggest you open up the maximum number of processes to something more reasonable and try again. If this does not help, it may be due to:

  • hardware limitations
  • something about the loop you are trying to parallelize that is forcing sequential execution (e.g. writing to the same file, DB table, updating a semaphored, shared variable...)
Comments