i recently made some kind of data recorder that would serialize and write to disk a spread of self-created types in VL. the problem was that the recorded files could only be opened/deserialized on the same machine. on other machines it would fail with an exception.
luckily i had master @woei at hand and together we figured, that a simple iobox (that i just used to monitor a value) appeared to be the culprit. when inspecting the decompiled code of the record we saw that this iobox gets a (random) name autoassigned which might be different on another machine. therefore deserialization fails. when removing this iobox everything is fine.
machine 1:
machine 2:
interestingly this behaviour (different names) did not always appear now when recreating the demopatch below. still, i’d consider this a bug as you would expect the behaviour of serialization/deserialization the same on different machines.
even though it is more work, I strongly recommend doing serialization manually, just by using XElement nodes.
Sure the magic automatic serialization at first glance feels like a wonderful gift, so it’s like a no-brainer that you’d want to use it, but on the other hand there are cons to this approach:
human readability of your files isn’t guaranteed
you are not the boss of what should be stored and in what format
And then also the underlying library would need to promise to always use the exact same format, which limits its ability to develop over time. And now as you discovered we’d also need to promise that our target code always looks the same, which is a burden we shouldn’t take on. All in all, I’d only try to use it for non-persistent data.
I tried the approach in the VL Project Workshop, but I’d not do it the same way again. It’s just too vague.
There are pros to manual serialization:
You can reason about a stable human readable compressed file format while still being able to adjust the data types. There is just no tight coupling.
It feels easier to reason about versioning of the files in the manual approach. To make versioning work, you’d need to drag the legacy versions of your data types with you to be able to open old files. In the manual approach, you might only want to change some bits in the way you translate a XElement holding a legacy description of an instance of your data type into an instance of your current version of that data type
Made an issue on how to advertise/categorize the auto serialize/deserialize feature.
thanks for your elaborate and clear answer.
indeed, just using the the deserialize/serializers really looks like a wonderful gift but your arguments make sense.
one more question remains: do we have any binary formats at hand to do custom serialization? the stuff i was recording were basically a bunch of floats/vectors and i guess text formats use a lot more memory (human readability is not so important here…)
Why are patch-internals like io-boxes even serialized in the first place? Can’t they opt-out?
Can’t believe @gregsn actually said this, as an argument against generic serialisation:
because as a patcher you should not even try to be boss of the serialisation format for your stuff and handcraft it. Not in this day and age anyway, and for good reason.
Writing a custom XElement serialiser should only be recommended for projects with a long-lasting duration of development and resulting compatibility issues between versions due to internal restructuring and frequent public updates.
But in most of project work you would only want to write down the contents of your record, in the rarer complex case you would want to have some moderation in what get white- or blacklisted from the transmission data, and if only for size optimisations.
Anyway, my point is, the actual serialisation work should be done by any tried-and-tested serialisation nuget you want (msgpack, json.net, fspickler, even protobuf-net).
Of course serialisation is only one reason why we need Attributes for at least Patches, Properties and Members.
In this case, the vl editor would allow you to simply “tag” the Pads you want to be in the transmission instead of handpatching cumbersome serialisation mechanisms that usually directly mimick your original Record, but in a horror mirror kind of way.
Yes and no. In general i totally agree, especially on the attributes, they can solve many problems. But input IOboxes can’t just not be serialized, because they hold data that are most probably relevant to your design. Output IOBoxes and in between link IOBoxes for debugging should of course get the “please don’t serialize me” attribute.
For serialization itself, there are indeed very good serializers and nobody really wants to patch all that boiler plate code. On the other hand VL is a rapid prototyping environment and while patching you constantly change data types, so you are running into incompatibility problems quite quickly. If you have your own hand written serializers you can easily add/remove data fields and manage different versions. Automatic serializers can be considered if you are happy with the design of the types and know that they won’t change anymore…
I understand. Glad we are on the same page. From what you write it is simply a matter of scope, from what you and me are working on everyday tasks. In many years of work I never had to maintain savefiles from many different versions. The beauty of vvvv project work, amiright!
The only reason I know something about it is due to trying to generalize the problem while coding Message. Only because of that I feel strongly about interchangeability of serializer choice, because that is what will connect vl to all the other languages eventually.
But before this goes too meta on serializers, I did some digging and found this:
So what I read from this: the Pad here does in fact have a fixed ID across all instances of said patch (even across cores or machines).
So a proposed bugfix for the initial problem would be to use the node id as the internal name for the unnamed IOBox, instead of making one up on the spot with an equally non-sensical counter algo.
disclaimer: based on the assumption that ids are only ever randomized at the node’s creation in patchland
-1 for my own idea, mostly because the proposed solution is bound to fail when copy-pasting vl snippets from patch to patch. pasted ioboxes will have to
either create a completely new internal name (which much likely will cause incompatible naming to the original, and hence incompatible serialisation artifacts)
or somehow bring information about the original’s internal name within the xml to the patch.
so that sounds bad, but also niche, you say?
maybe it is a big thing as soon as we have html5 gadgetery for drawing vl and copying to clipboard on this very forum, right?.
so after some consideration, the cleanest way I see, is locking down some PreferredSerialisationName in the xml, right next to the cryptic id. in case a copypaste action generates a naming conflict with already existing unnamed pads, a yellow warning should light up on all affected pads to engage the patcher to ctrl+e the nodes that should reset their internal name.
of course the same thing would work with the id (as in the former solution proposal) but could unneccessarily bloat serialisation artifacts. so the trade-off with the extra xattribute seems ok
It would be good to have an example of serialisation and de-serialisation for rolling your own, when I’ve tried xml so far in VL, I have failed, dont know if it was the xpath was incorrect or not, but in vvvv I’ve seen even Woei open the xpath help file when starting to patch xml.
Then again I found making my own xml in vvvv easier than the attrubute etc nodes, never seemed to quite get the result I wanted, nor when I did, decode it again afterwards!
Just a quick update fields without a user defined name will now use the serialized id (the one you see in the *.vl files) as their name. This should lead to stable field naming and should fix the original issue.
