We are working on a project that involves a massive amount of computation for a big collection of point clouds and we need a solution for splitting that computation between as many cores as possible. We use 5 x dual 8 cores xeon procs for a total amount of 160 logical procs at 3Ghz each.
Now, I am aware of posts like this here Ultimate way of sharing data among instances as well as similar ones, and we tried different scenarios with moderate success. The most promissing seems to be the zeromq approach, but also this one has it’s drawbacks, one of them being the pain in the B of manually implementing that and the fact that there is no sync option.
I was reading this introduction in The Gray Book, I quote:
" You want to offload parts of your patch to separate threads
Large patches can become computationally expensive and vvvv does not allow you to use the full power of your PC by being inherently single-threaded. Using vl you can define regions of your program that you want to run asynchronously to the main patch, thus using multiple CPUs in parallel."
Well yeah,that’s preciselly what we need! The question is, how to do that?