Automatic Persistent Serialization

vvvv gamma question
1

Anyone got solutions or hints for automatic persistent serialisation?

Usecase

I want to save and load data made over many release versions of an application, potentially with updates years apart.
The data itself is records, spreads of records, dictionaries of records.
Whilst I try and keep changes minimal there could occasionally be major structural changes to the data schema.

Basically I want to ensure as much automatic backwards compatibility as possible when loading data created with an old version of the app.
And then minimise my manual programming effort to only handle the difficult cases.

I already encode an app version number in the data so I will know if it’s attempting to load data from an old version.

Problem

The automatic serialisation of vvvv gamma is great and usually a good workflow. But as you read on the Gray book serialisation page the underlying fsPickler library is not suitable for persistent use, a future version may interpret data differently.

I tried manual serialisation but I found on a large structure it’s very laborious and prone to programmer error once you have a few versions going.

So I want to use an automatic workflow as far as possible.

Maybe I even need a schema checker that tells me what changes I made between versions?
While I can easily conceive the changes I made when I update the app from version A to version B, once I am working on app version H and I want to load data created with app version A I’m much less sure.

Questions

Breaking it down I think I can always use built in serialisation for saving. FSpickler will ensure the properties are saved in a valid common format (JSON, XML), particularly if I use Include Defaults.
(In principle this means I could switch to use something other than fsPickler for deserialising in future.)

And then if I load data created with the current version of the app it can also use the built in deserialisation.

I only need to have special handling for the case of loading data from previous versions.

Does that sound right?

Then I would love to hear thoughts or solutions from others towards this problem.

Anyone use any schema checking?
Is there any solution that would tell you meaningful differences between data and schema?

1 Like
2

Hey, just facing that exactly now after making a breaking change in the Launcher’s settings model.

As of right now, “old” settings files would simply not work, and users would have to manually delete their file and restart the launcher (and lose their settings). This is not cool.

The only thing I can think of right now is indeed to come up with my own serializer (as shown in the Register Custom Serializer help patch) where I explicitly patch all “converters” based on the version of the settings file. That looks a bit dirty and painful to maintain, though.

I wonder if doing some kind of diff between the old settings file and the new settings model would help here. Every time you introduce a breaking change, you would register some sort of converter, and if while diffing you come accros a conflict that is handled by one of your registered converters, run it, otherwise throw an exception.

Some links I’ve found while writing this :

What I don’t like though is that they’re ten thousand years old, and that they strictly relate to XML. Ideally I would like it to be decoupled from the format under which settings are stored, but I don’t know if this is even possible.

So, not bringing much to the table, but very curious to hear about it as well :)

Edit : forget the last comment about XML, one eventually has to settle on something to do the comparison, no matter how the data is stored on disk… XML can be an option here, as long as we have ways to go from/to it.

1 Like
3

Maybe @robotanton can share how it is done for the vvvv settings file …

2 Likes
4

Settings are managed internally by vvvv, so I’m not accessing an XML in the Settings window.

But this is what I’m doing when I have several versions of the model / file: I introduce a version to it. As an individual element or as an attribute for the root element. Then you just first check the version, if it doesn’t exist - use a “very-old-deserializer”, if the version is there, then pipe the XML through the deserializer for that version.

For the apps, you don’t actually have to register anything, just manually check the version in the XML and use a particular operation patched for that version, right?

Something like that:
image

Best,
Anton

1 Like
5

Thanks for the insights @robotanton :)

Was using automagic (de)serialization until now, which is why I first thought about registering custom (de)serializers so they could be picked up by the Serialization nodes.

Guess it’s more efficient to explicitly patch those in static operations as you just showed.