Some thoughts about visual-programming

vvvv gamma general
1

Hi.

For the last year I’ve been exploring visual-programming for real practical applications.

My initial idea is to give DeepFaceLive (GitHub - iperov/DeepFaceLive: Real-time face swap for PC streaming or video calls) users the ability to modify the logic of program nodes interactively, without resorting to source code.

The conclusions I came to are:

  1. visual programming of simple operations is a failure. 2-3 lines of text in python, in nodes it will be like an entire screen of spaghetti nodes. It’s pretty, but not practical.

  2. visual-programming limits the speed of writing programs. By reading text, we load a larger volume of algorithms into our imagination. When reading nodes, we first take a long time to mouse and scroll, memorize the diagram, then translate it into our internal representation of the algorithm in our imagination. As a result, we produce a lot of redundant actions in the brain compared to reading text code.

  3. Nodes should contain large programs. The user only guides the logic of the data.

Therefore, the nodes should be interactively programmable in a lightweight language like python.
In this case the system can provide 3 types of programming:

  • No-Code. The user downloads ready-made apps and uses them.

  • Low-Code. The user makes programs using ready-made nodes, or modifies downloaded apps, adding functionality to suit his needs, such as outputting an image to a stream or saving a sequence of images in a folder.

  • Full-Code. Same goals as above, except the user programs nodes interactively using hot-reloaded text inside the same studio.

  1. The more abstract the node system is, the smaller the range of tasks it can accomplish. The greater the range of tasks a node system can accomplish, the more it is like directly writing lines of code, but with Basic(lang) flowcharts, which again are difficult to read compared to text code.

  2. A vertical node system can accommodate more nodes and requires mouseover on each node to read the I/O.
    Horizontal is easier to read but requires frequent horizontal scrolling.

  3. The advantage of node system over text code is that we can see the result of any function execution interactively. It is like Debug mode of code execution, only the result of any line of code is available at any moment by simply pointing the mouse.

A node system that has a fixed pipeline and has fixed views for users, such as buttons, text and numeric fields, is not difficult to implement.
The pain begins when the code produces dynamic interactive views waiting for input from the user.
For example, selecting the type of Neural Network model in a ComboBox, and depending on that, different settings appear below. Or some settings that spawn other settings.
In this case ViewController should be built in a separate window by the program/nodes itself, increasing the size of the program, and mixing Model code with ViewController code.

Conclusion.

Is it possible to design such a general-purpose node system that would be both intuitive for the user
(vvvv is not intuitive even for programmer),
able to implement MVC/MVVM code of any complexity, and async/greenlets/multi-thread friendly?

Or is it just fundamentally impossible?

2

There are several ways you can tell a computer what you want it to do.

Text coding involves writing down a list of what you want the computer to do.
Visual programming involves drawing a picture of what you want the computer to do.
Machine learning involves showing the computer what you want it to do.

If you want to talk about the general future of visual programming you don’t have to limit visual programming to just visualising the list of instructions of some other language (Eg being ‘visual python’). That would be as limited as the people who once saw no future in machine learning.

Specifically regarding your points
Regarding points 1 and 2 I simply don’t agree. You can show me your python code and I can call it hard to read or a ‘failure’ because I don’t personally connect with the forms of expression in python.
Everyone can have their own coding aesthetic. The machine doesn’t care what abstractions we use.

Your point 4 is the rebuttal to point 3. High and low level are both needed. What can always be improved is the documentation and UI so that beginners don’t end up confused by low level nodes by accident.

On point 5, sure. I think most important is that any visual language can make good use of top to down and left to right flow when it’s useful. Any node abstraction that locks you into only one wastes the potential of the other.

On point 6 It actually amazes me that ‘hover mouse for a live tooltip’ debugging hasn’t really come to text coding. I think that feature doesn’t have to belong exclusively to visual programming.

In response to your conclusion, I think if you want to design a node set to support a specific programming pattern you can do it. But I wouldn’t call it ‘general-purpose’. Such a statement will set weird expectations from programmers coming from other languages.

3 Likes