Regex performance

I have a big array defining translations and colors of a point cloud, it s over a million lines. I am using a regular expression parser to retrieve all the data but it takes ages to load and I am only loading 50000 lines at the moment. It does work but it just tale to long to load, is there anything I can do to improve the performance?

can’t you parse it once and store the data in a mesh or so?

Yes I can that is what I am going to do, but I was wondering if I can get better results, I see that vvvv goes blank (even the OS stutters) for about a minute or so for 50000 lines, so I really fear that whenever I ll be ready to feed it 1 o 2 millions lines …

hm… if the structure of the xml is simple enough, i would do it with basic string nodes like Separate and AsValue. or you can try the regex in the contributions, maybe its faster… but 1-2 mio lines will always be slow as hell if processed in one frame.

the structure is very simple, it is a .ply file so it is like so:

x y z nx ny nz r g b

and my expression is:

^(.?)/(.?/etc etc ?

I ll try with separate that makes sense…

Yes, with Separate it s much better performing, thanks.

How long does it take?

I’ve done a dynamic plug-in that loads point clouds from text file (only x y z though) and outputs a texture for GPU particle usage. Then you can either run it re-run the plugin each time or just save/load the texture.

Would prefer tidying it up a bit before sharing it (along with some other stuff) but if it’s important I guess I could change my mind.

if you want to share you are very welcome since I am just doing the very same thing. If you want you can post it here before contributions. Anyway it is not imporant to me as something I have to do or any deadline, just osmething I am really looking into righ tnow.

@beyon - did you ever share this?

Hey. Dynamic Plugin sounds great for the job. I always wanted to do this too but never had the time.
In the meantime you could use the DataFormation module I used in my thesis and modify it so that it fits to your column count.

RegEx is so slow… Separate does the job. and a little trick is to use framedelays and a logic which does reading, separation of lines and separation of columns in subsequent frames!

e: oh thread is a bit old. anyway interested in such a plugin. and I leave the comment in case someone is searching something similar again.

mrboni, tekcor: no I didn’t get to spend much time with vvvv so I didn’t share… but here you have a demo example.

Never got around to clean up code so it’s probably a it hackish in places but you should get the idea anyway.

Asc (1.5 MB)