Hello guys
I would like to leave the pointclouds behind and move one to Gaussian Splatting
Not sure if anyone did something similar… I would like share the idea first
Compressing Gaussian Splats | PlayCanvas Blog.
I imagine to reuse the ply reader, and work on the 3d engine to ( as first step ) render splat resuing the ply logic, and applying the transformation(rot/size) / color(alpha)
Btw try out https://www.jawset.com/
@Takuma I’m sure you will like this idea
Was also looking for this topic!!
Sadly couldn’t understand how to make Splats happen in vvvv, but would definetly love to see this happening!!
I understood that the PLY have the data of each “splat” like a particle, x y z r g b
in adition to that, there is a size of the particle, not sure if as a volume or as a plane
and a transformation vector for scale and rotation
also there are more parameters, but not sure how to implement those, but with the ones I described I guess it could be relatively easy to make a spread of quads , instance them, or maybe play around the fuse way
I will post any update on this project, I will start with the parsing of the PLY file taking as a “base” the ply reader
I am also interested into splatting… This Unity implementation could be a starting point.
“bicycle” scene from the paper, with 6.1M splats and first camera in there, rendering at 1200x797 resolution, at “Medium” asset quality level (282MB asset file):
Unity, DX12 or Vulkan: 6.8ms (147FPS) - 4.5ms rendering, 1.1ms sorting, 0.8ms splat view calc. 1.3GB VRAM usage.
@motzi, @texone and @gegenlicht were talking about it in the Fuse chat.
HowTo Parse a Splat file in a custom way.vl (69.9 KB)
I thought I might make a start
Here is a splat I made in luma last night
https://www.dropbox.com/scl/fi/ke6mn9jjrz6fi81ehihzr/gs_Titterstone_sunset1.ply?rlkey=3bzfz8cxpvdgbnndfam4qb2wb&dl=1
I’m trying to parse the file using the how to parse a custom ply in the vl.io.ply nuget. The header doesn’t seem to label the RGB which is odd, nxx is I guess the normal (luma does some normal mapping in their renderer) dc could be rgb?
Anyway, I’m not really sure how/where to us the custom parser after making it, so can’t check it out.
From Introduction to 3D Gaussian Splatting
The concept looks pretty straight forward, point sprites with scale and rotation added, the only complex thing is depth sorting them before drawing. But step one is to parse the file and render as a point cloud, then work out the RGB, the transform, then sort… I think
it is automatically being picked up by the PLYReader node internally. in your howto you can see it already “working” in that the “Vertices” pad connected to the PLYReader does have values. so the parser does something. being not familiar with the format nor its application though, i cannot comment this any further.
I can’t see where or how Binary Vertices (Custom) is called, it doesn’t seem to be inside the PLYreader node. It just works by magic? I’ll take it as supernatural for now and try and carry…
yes, by the magic of adaptive nodes.
https://www.dropbox.com/scl/fi/ttvi4vxm8nxmevv5xf178/bottle-splat.zip?rlkey=o5i3fi680uaxux876iqcvl5z4&dl=1
This is a smaller splat file, seems its quite hard to make small ones in Luma, this is as basic as I could get it I’m afraid, don’t expect beauty, just a valid splat file that is <200MB
HowTo Parse a Splat file in a custom way_2.vl (158.7 KB)
And here is an updated parser, but I’m not sure how to make a structured buffer out of the data. I’ve made a SetSplat node as a fuse particle emision kind of thing, not sure this is right way to go about it so any pointers would be great.
For further info, this is a UE implimentation, and there is a touchdesigner patch available for splatting that has a bitonic sort compute that will hopefully be somewhat transferable. Gaussian Splatting | Derivative
Check the Fuse IO.PLY examples on how to read a ply file into a structuredbuffer I guess in this case it would make sense to make a custom PLYToComputeNode I will look into it if I find the time
HowTo Parse a Splat file in a custom way_2.vl (261.5 KB)
I added a ply reader to get the data in a structured buffer it is missing the spherical harmonics data this would need to be read into a separate buffer and than used in shading. I am happy to push this further let me know can also have a chat.
That looks interesting! I’ll try and work my way through it… Thanks
HowTo Parse a Splat file in a custom way_3.vl (532.1 KB)
Started confusing myself, so I’ll leave it here. I think rotation is ok, scale is 3d in the file, but we are drawing a sprite so I guess the z isn’t needed?
Tried to think about colors and sorting, but gave up, maybe tomorrow!
This looks useful though a unity splatting repo (but is dx12, which I guess might be what @tebjan is waiting for?)