Problem with implementing multithreading (Parallel.ForEach) in GHPython!

ghpython
python
unhandled

(P1r4t3b0y) #1

Hello everybody,

I’m trying to implement multithreading (parallel computing), for a demanding for loop-portion of a Python script, I’m currently working on.

I’ve read @stevebaer’s post on the subject from a while back and tried to accomplish a similar thing with System.Threading.Tasks.Parallel.ForEach().

I get the following error:

Runtime error (ArgumentTypeException): The type arguments for method ‘ForEach’ cannot be inferred from the usage. Try specifying the type arguments explicitly.

Traceback:
line 28, in update

I understand that Parallel.ForEach takes 2 parameters: a list of objects of type x, and a method that takes in one object of the same type x and returns nothing.
Here may reside one problem, since in Python (at least I think so), you don’t declare the type of the items in your list, and the type of the list itself would be, well, ‘list’ and not type x.

Here’s how I call Parallel.ForEach:

import System.Threading.Tasks as tasks

tasks.Parallel.ForEach(self.agents, self._compute_agent_velocity)

I pass it a list of class instances self.agents, of the same class Agent, as an iterable, and the method self._compute_agent_velocity(). The method takes in one agent instance and returns nothing. The list of agent instances, as well the method, are both parts of the class ParticleSystem.

import Rhino.Geometry as rg
import System.Threading.Tasks as tasks
import random


class ParticleSystem:
    def __init__(self, agent_count=100, use_parallel=True):
        self.agent_count = agent_count
        self.use_parallel = use_parallel
        self.agents = [] # gets passed to Parallel.ForEach()
        
        for i in range(self.agent_count):
            agent = Agent(
                self._get_random_point(0.0, 30.0, 0.0, 30.0, 0.0, 30.0),
                self._get_random_unit_vector() * 4.0)
            agent.flock_system = self
            self.agents.append(agent)


    def update(self):
        """Method that calls Parallel.ForEach()."""
        # Regular for loop
        if not self.use_parallel:
            for agent in self.agents:
                neighbours = self._find_neighbours(agent)
                agent._compute_desired_velocity(neighbours)
        # Multithreaded for loop
        else:
            tasks.Parallel.ForEach(self.agents, self._compute_agent_desired_velocity)
            
        for agent in self.agents:
            agent.update_veloctiy_and_position()
    
    
    def _compute_agent_desired_velocity(self, agent):
        """Method that gets passed to Parallel.ForEach()."""
        neighbours = self._find_neighbours(agent)
        agent._compute_desired_velocity(neighbours)
    
    
    def _find_neighbours(self, agent):
        neighbours = []
        for neighbour in self.agents:
            dist = neighbour.position.DistanceTo(agent.position)
            if neighbour != agent and dist < 1.500:
                neighbours.append(neighbour)
        return neighbours
    
    
    def _get_random_point(self, min_x, max_x, min_y, max_y, min_z, max_z):
        x = min_x + (max_x - min_x) * random.random()
        y = min_y + (max_y - min_y) * random.random()
        z = min_z + (max_z - min_z) * random.random()
        return rg.Point3d(x, y, z)
    
    
    def _get_random_unit_vector(self, three=True):
        phi = 2.0 * math.pi * random.random()
        theta = math.acos(2.0 * random.random() - 1.0)
        x = math.sin(theta) * math.cos(phi)
        y = math.sin(theta) * math.sin(phi)
        z = math.cos(theta)
        return rg.Vector3d(x, y, z)
    


class Agent:
    def __init__(self, position, velocity):
        self.position = position
        self.velocity = velocity
        self.desired_velocity = rg.Vector3d.Zero
        self.particle_system = None
        
    
    def update_veloctiy_and_position(self):
        self.velocity = 0.97 * self.velocity + 0.03 * self.desired_velocity
        if self.velocity.Length > 8.0:
            self.velocity *= 8.0 / self.velocity.Length
        elif self.velocity.Length < 4.0:
            self.velocity *= 4.0 / self.velocity.Length
        self.position += self.velocity * self.particle_system.time_step
        
        
    def _compute_desired_velocity(self, neighbours):
        self.desired_velocity = rg.Vector3d.Zero
        bounding_box_size = 30.0
        # Control boundary
        if self.position.X < 0.0:
            self.desired_velocity += rg.Vector3d(-self.position.X, 0.0, 0.0)
        elif self.position.X > bounding_box_size:
            self.desired_velocity += rg.Vector3d(bounding_box_size - self.position.X, 0.0, 0.0)
        
        if self.position.Y < 0.0:
            self.desired_velocity += rg.Vector3d(0.0, -self.position.Y, 0.0)
        elif self.position.Y > bounding_box_size:
            self.desired_velocity += rg.Vector3d(0.0, bounding_box_size - self.position.Y, 0.0)
            
        if self.position.Z < 0.0:
            self.desired_velocity += rg.Vector3d(0.0, 0.0, -self.position.Z)
        elif self.position.Z > bounding_box_size:
            self.desired_velocity += rg.Vector3d(0.0, 0.0, bounding_box_size - self.position.Z)

Please note that I’ve abstracted the code a bit, to make it shorter and protect some parts. :wink:

Any help is greatly appreciated!


(Graham) #2

Hi @p1r4t3b0y,
Not a solution but if statements and function lookups are expensive in Python. If you can refactor your code to reduce these e.g. by using dictionary lookups or putting more logic into a single function then it should be quicker.
Also consider storing object parameters in a local variable to avoid looking them up multiple times in the same portion of the code.
-ggg


(Steve Baer) #3

You can also use the parallel helper function that I added to ghcomp without using any grasshopper code


(P1r4t3b0y) #4

Thanks @stevebaer, but could kindly take another look at my code above and maybe tell me what the mentioned error is all about or what might be wrong please?

I’m going to try the ghcomp parallel method tomorrow though.


(P1r4t3b0y) #5

Thank you, @Dancergraham! I already know about dictionary lookups, although the implementation in this script might be kinda difficult. It’s mostly about evaluating agent relationships by their ever-changing positions in space. I’ve incorporated the R-tree algorithm from RhinoCommon this afternoon though and the performance boost is immense.
However, I’m still interested in testing it with parallel computing.

What do you mean by “object parameters”? Could you cite an example from my above script for multiple look-ups of the same stuff?


(Graham) #6

Yes I was thinking about this bit where you refer repeatedly to self.velocity


(Graham) #7

Good to know… yes i was thinking that might be suitable but I’ve never tried… nested 3D hash tables?


(P1r4t3b0y) #8

That was quick, thanks! :slight_smile:
So would it be better to store self.velocity into a local variable and use that variable multiple times inside the method?

Let me know, if you want to know, how to implement the R-tree algorithm in GHPython!


(Graham) #9

Maybe… rereading your code it may just be a dict lookup anyway so no advantage


(Steve Baer) #10

The syntax for getting Parallel.For to work in IronPython is really hard to get right as this is a generic method in .NET. This is why I wrote the ghcomp.parallel helper function. If you look at my initial blog sample, I’m only dealing with an iterable list of integers and a function that takes an integer as input. Other forms are very hard to get right.


(P1r4t3b0y) #11

OK, thanks for clearing that up. I didn’t have time to implement the ghcomp.parallel today, but I’m going to look into it tomorrow.
The integration of R-tree already provided a huge performance boost. Am I correct that it doesn’t work with multithreading yet? I remember someone having mentioned that a while back still on the previous webpage.


(Graham) #12

In case you missed it: