Perl - multithreading on foreach loop:

Question

I have a big code that somewhere in the middle of it, there's a foreach loop filling up some dictionaries. Each iteration is independant and fills those dictionaries with disjoint keys.

I'm trying to turn the "foreach" loop in the middle to multithreaded in order to decrease time.

In the following example, $a1, $b1 are the pointers to dictionaries.

I tried "thread::shared" this way:

my $a1 = {};    
my $b1 = {};
my $c1 = {};
my $d1 = {};

# a lot of code using $a1 and $b1

share($a1);
share($b1);
share($c1);
share($d1);
my @threads;
foreach my $Inst ( sort keys %{ $a1->{ports} }) {
            push( @threads, threads->create('some_func', $Inst, $a1, $b1, $c1, $d1, $e ...)); 
}
for my $thr (@threads) {
          thr->join();
}

# all the other code

But I get an error of:

Invalid value for shared scalar at ...

Any ideas how to get the data-structures filled, but not that it would interfere with the code before and after the for-each loop?

Your question is obscure -- your declare a1,b1,c1,d1 as shared and then you say that they filled on each iteration without specifying any code how it happening. If they filled on each iteration then it sounds like they should not be shared. And your sentence -- but not that it would interfere with the "#some code" section and not with the "#some other code" section" -- creates even more confusion. — Polar Bear
– Polar Bear, Commented Aug 22, 2021 at 8:24
@PolarBear Thanks for your comment, I'll try to edit the question — urie
– urie, Commented Aug 22, 2021 at 8:26
According to the documentation, shared scalar references can only store reference to shared variables or shared data. For example if $a1 references a dictionary %h1, then you need to share %h1 too.. — Håkon Hægland
– Håkon Hægland, Commented Aug 22, 2021 at 8:33
@urie "how could I avoid that" --- Use shared_clone from threads::shared. I suggested this in comments to your two previous questions, one now deleted (exact same as this one, I believe). There was a link to a post with an example as well. Did you not look at any of that at all? — zdim
– zdim, Commented Aug 22, 2021 at 8:51
@urie It seems it is not possible to make a hash shared after it has been created/declared without losing the data in the hash. If you do shared %$a1 and $a1 references a hash %h1, the data in %h1 is lost. As zdim mentioned, I think you need to use shared_clone — Håkon Hægland
– Håkon Hægland, Commented Aug 22, 2021 at 9:09

Håkon Hægland · Accepted Answer · 2021-08-22 09:15:05Z

1

It is not possible to make a hash shared after it has been created/declared without losing the data in the hash. Instead you could try use shared_clone() like this:

use feature qw(say);
use strict;
use warnings;
use threads ;
use threads::shared ;
use Data::Dumper qw(Dumper);

my %h1 = (a => 1, b => 2);
my %h2 = (c => 3, d => 4);

my $a1 = \%h1;
my $b1 = \%h2;

my $a1c = shared_clone($a1);
my $b1c = shared_clone($b1);
my $lockvar:shared;

my $nthreads = 3;
for ( 1..$nthreads ) {
    threads->create('job_to_parallelize', $a1c, $b1c, \$lockvar ) ;
}
$_->join() for threads->list();

sub job_to_parallelize {
    my ($a1, $b1, $lockvar) = @_;
    {
        lock $lockvar;
        $a1->{a}++;
        $b1->{d}++;
    }
}

print Dumper({a1c => $a1c});
print Dumper({b1c => $b1c});

Output:

$VAR1 = {
          'a1c' => {
                     'a' => 4,
                     'b' => 2
                   }
        };
$VAR1 = {
          'b1c' => {
                     'd' => 7,
                     'c' => 3
                   }
        };

answered Aug 22, 2021 at 9:15

Håkon Hægland

40.9k22 gold badges96 silver badges209 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

urie Over a year ago

How would you suggest combining everything together at the end? (In case of a complex dictionary, like dictionary of dictionaries of dictionaries)

Håkon Hægland Over a year ago

shared_clone() should be doing a deep copy. According to the documentation : "returns a shared version of its argument, performing a deep copy on any non-shared elements". Did you try it? I do not see exactly what I can add to my answer that is not clear. Please clarify.

urie Over a year ago

I did that, and it made a deep copy :) My question is this: After one loop, I have a partly-filled data-structure. How do I update the "main" data-structure with the new values? Is there an easy way besides iterating through the whole keys/values and checking if there were any updates somewhere? If it's still unclear, I'll try to give a concrete example

Håkon Hægland Over a year ago

"After one loop, I have a partly-filled data-structure." : What do you mean by "one loop" ? In your example there is only the thr->join() loop. You mean that after that loop finishes, the data structure is only partly filled? Why is it so? Please give more details.

urie Over a year ago

I mean that EACH loop builds its own data structure(hash, in my case). How do I combine all of those hashes together? I just want to explain that it's not just hash => {3 =>4, "mom"=> "dad" }, but it's a very complex hash that holds hashes and arrays inside it. So just looping over the hash won't be enough. I'll try to open another question if I won't see any solution online.

|

Collectives™ on Stack Overflow

Perl - multithreading on foreach loop:

1 Answer 1

9 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related