On the project I am currently working on I need to record the contour data coming out of the contour node.
I have tried recording to a text file and playing back from it again.
It is not surprising stuttering heavily and considering that it is just relatively few numbers compared to recording and playing back eg. video I am thinking there must be a more efficient method. of course I can optimize by cropping values to the needed precision so I don’t get a lot of decimals.
So My question to the forum is if you have a more efficient way of recording a lot of floats? I am thinking of some sort of binary format. I have been thinking about using a database but is thinking that it can’t be optimal to wrap the whole thing into sql.
So any Ideas?
@vux I am still a little reluctant to use the database approach, my main concern is the overhead both in cpu and size from converting the floats to strings. But should give it a try. regarding the length of the recording I tried making one around 10 minutes and at 30fps the resulting textfile got pretty large, many megabytes.
@joreg I have been playing a bit around with the dynamic texture approach and it seems like I could get it to work, I do however have some doubts about if it will be a good approach. If using this method I would need to record the changing texture as video and a codec would then be used to compress it, I am not sure how that would affect the precision of the data when replayed. further more the additional trouble of having integers larger than 255 is a bit complex. also the fact that the number of points varies which makes it necessary to use a relatively large texture.
I can see how both approaches can solve the problem, but I still feel that a more elegant and optimal solution could be possible. in any case I would like to wish for a method of saving floats and perhaps even other datatypes in what I imagine would be a binary format so I would not have to convert to strings and thus increase the amount of data saved.
my main concern is the overhead both in cpu and size from >converting the floats to strings
If you run a query per frame (which will be executed in no time), you’ll have let’s say maximum 500 conversions, which is negligible to be fair. I would still not use a network database like mysql, that’s way overkill, small file based system will definitely do the job.
The other option is to write in a binary format and do your own indexing (i did that for a FFT recorder), but that means some plugin coding in that case.
Your case will be a tiny bit more complicated than mine as on each frame the number of contour can change, so the data size per frame can be different (whereas fft stays the same all the time). It’s not too hard still, but need some thoughts.
By the way do you record all the contour data, or just the center of the blobs?
Now I am testing recording… it really halts the computer, I achieve between 7 and 10 FPS when using sqlite for recording. it seems like playing back the data is no problem though so having the data in sqlite.
I have tried reducing the length of the strings written to the database by converting the floats into hex using radix and then into string, then the strings are aprox. half the length compared to just converting the regular floats to string.
at the 30 fps the camera is running I would like to record a string around 3500 chars long. so in all ideally around 105000bytes a second.