Rounding Errors and Float32 Precision

[mburk]

This is a very weired behaviour, which I would consider a bug.
Somehow 9 * 0.1 != 0.9 because of rounding errors or incorrect precision handling.

All the input IO Boxes are set to Precision = 2

[mburk]

Also this somehow only happens for 9 * .1
8 * .1, 7 * .1 etc all work as expected

[readme]

I’m definitely not an expert on this and get how this feels counter intuitive, but it’s a problem that I ran into outside of vvvv and it’s just how computers work (I found this pretty enlightening). Also it explains why this only happens for some, not for all floating point numbers:

source: python - Limiting floats to two decimal points - Stack Overflow

some more:
15. Floating Point Arithmetic: Issues and Limitations — Python 3.10.7 documentation

Demystifying Floating Point Precision « The blog at the bottom of the sea (demofox.org)

[mburk]

Alright, makes total sense.
Maybe some additional thoughts from a user perspective:
I would somehow expect vvvv to use the set precision also for calculations and not only display it but use full precision in background…Not sure if that actually makes sense, but that’s the obious expectation.
And secondly a =(Epsilon) would be nice to have for such things.