Forum

Best way to send large data sets between instances?

#1

I’d like to to split up a patch between my cores. I have up to 3000 values which need to be transmitted at the same time. SharedMemory, UDP can’t handle those large packages. Also tried OSC via UDP.

In every instance this data spread gets additional transformations which depend on the ones before.
So I can’t just create a new spread.

Is there any good way to broadcast 3000 values?

(Solution at the moment would be to have eight similar instances, which each calculate another section of the final image. So I only need to share textures between instances and combine them to an image in a final instance. But having eight similar patches running will be extremly annoying to maintain).

#2

boygrouping?

#3

sharedmem doesn’t work? why?

#4

@sunep
same problem with boygrouping, it also sends data via TCP / UDP.
Can you create a boygroup on one machine?

@sebl:
If I exceed 4096 Bytes the node turns red and stops sending.

I also tried Netsend and Netreceive: TCP can send far more data, but it also needs a lot of CPU-power.

Is it possible to compress the data?

#5

@wrotzlaw have you seen the NOTE in the helppatch of Reader (Raw SharedMemory) regarding changing the Size?

#6

I’ve had issues with RawShared memory, from crashes on startup to missing values, I’ve started using UDP or TCP with a tokenizer and and end of message token, this works ok.

#7

I’ve been using Writer (Raw Sharedmemory) for a multiple projects on multiple machines without problems, sometimes sharing more than 20kb of data.
Try the double spread and string spread memory sharing modules in the attachment.

SharedMemoryRW.zip (9.6 kB)

#8

On the other hand, I found UDP Send node causing massive frame rate drops, so I always try to avoid it if possible.

#9

I found really useful to use velcrome’s pack Message, converting the generated message(s) to Json with the included node (AsJson) and sending and receiving with the Raw shared memory nodes. You can deliver a ton of data with this method, but you need to control when to send (2 instances available) to avoid thing going unstable.

#10

I could make sample patches if you need.

#11

@id144 please note that there was a fix for udp/tcp nodes with large bytestreams for beta33.7. so you might want to revisit those, just in case…

#12

thanks guys! reader /writer (raw shared memory) works great, excactly what I was looking for.

@lbolster JSON conversion sounds interesting. I’ll definetly check out velcromes pack. Sample patch is also welcome :)

#13

@wrotzlaw: raw shared memory drived me crazy (not stable). I also recommend catweasel method with the strings generated from velcrome’s asJson. Of course using his pack on both instances. About the example patch, I’ll see If I have the time… do you still need it?

thanks catweasel, for that method.

© vvvv group