GH Feature Request: Parallel Groups

Belive me, with the suggested FBP concept you can take one very small (very small) adult step, and done. After that no one (or very very few) will ask for individual components with concurrency, everafter) :slight_smile:

It’s been proven since the early seventies (actually sincethe sixties). One word: Simplicity.


Edit:
I forgot to say that the components shown in the picture could be the regular script components (all of them, including GhPython), or regular list components, even for the dispatching of data (Map) to the “workers”. Only thing to take “special care of” in this concept would be to cache the outport data (from any std component).

The ParallelGroup could handle caching of outports behind the scenes, holding data until the “Reduce” components (probably standard components, including std script components) in the main thread is free to receive the data (std script component already handles dynamic number of in-ports, a “std” list-component with cache-receiving indata ports could also be introduced, but simplest would be if the ParallelGroup takes care of any caching - an all-in-one-place-solution subscribing on the outports of any worker tagged as “PARALLEL”).

In this way the main thread both dispatches and receives data - swarming one ParallelGroup at the time with available threads - even guaranteeing that data flows sequentially (“singelthreaded”) through the entire GH definition / network. Standard components can be used if the ParallelGroup component is made smart enough.

Anyway, this is also why a FBP solutions are “inherently threadsafe” (most important from the user’s persective, but extremely simple to achieve also from a technical point of view) while allowing massive high performance computing of data “locally” in the network (in the dedicated parallel groups) due to the global definition taking care of orchestrating single threaded components (single threaded, just like it works right now). Only in very rare cases one would want to implement multithreaded individual components, but even then such components work well under the same orchestration concept.

Another important point: Worries about “users at risk of doing mistakes” is exactly what you don’t have to worry about in this kind of concept.

At last: FBP is a concept, a way fo doing things, rather than a “language”.

// Rolf

Massive HPC computing cruching terabytes of data in medical research centers is done in this way in these days, where the researchers wire up their own solutions. And the data centers don’t go up in smoke.