P4V - Package Manager For VVVV

vvvv beta feature
1

P4V is initiative to create a community driven package manager for VVVV, that will allow us to download only packages needed for current project, solve dependencies between packages and allow us to easily update it and search for the new and popular stuff.

Some History

Firstly, we discussed the possibilities of package manager in IRC with devvvvs about a year ago. The main topic against separated packages was, that we have addonpack and it’s easy to download just one zip.

But what about current situation? Currently we have addonpack with a lot of unusable and almost deprecated nodes. Also, people like vux developing they own packages, that completely replacing big part of VVVV functionality, such as DX9 pipeline.

So, for today we have:

  • Pack of DX11 nodes
  • Pack of OpenCV nodes
  • Community created pack of huge amount of cool and not so cool nodes
  • Contributions section on website, with more than 9 pages of interesting stuff

As we discussed with Elliot, devvvvs currently planning to introduce new download type, called — pack. For example now on main page we have addonpack as separate download, but in future we will have also dx11 pack and openCV pack.

For me this all structure already looks like a mess. So i think we should somehow simplify the process of adding new independent contributions to vvvv and make this process consistent as possible.

Solution

At node festival i meet Fibo, that already come with the same idea. So we started to work on P4V - the package manager for vvvv. It’s already working in prototype version and have it’s own webserver running on Heroku free hosting.

What It Can Now

P4V is just reuse of the Bower - universal package manager, created by Twitter. It can pull any files from git, solve dependencies and update it. So it’s doing exactly what we need, but have some limitations.

Limitations

Bower is based on NodeJS, so we need to have it installed. Also, we need to have Git added to Path variable, so we can use it from any system’s command line.

Also, the core philosophy of Bower is local packages, so every project just have a file, describing, which packages it needs and we downloading this packages to the project folder. This is good, because different projects can use different versions of plugins, but bad, because we have packs and pack is the globally installed package, if i’m understanding it right.

Another Possible Solutions

Chocolatey is another possible solution for package management. We already have vvvv and addonpack packages there. It works only on windows, but don’t require you to install any additional scripting languages. The bad thing, is that it’s should be hard to setup package management solution on linux server and also we can’t use it for managing vvvv.js packages in future, because it’s locked to Windows.

Benefits From The Custom Server

Custom server allow us to create any API that we need. For example, we can provide an API method to describe, which nodes included in this package.

Also, we can write automatic packager, that allow us to automatically create a correct contribution from the user files, that uploaded via special form.

In other case, we can integrate package search system into vvvv website.

Topics for future discussion

  • We should understand, do we really need global packages? Or we only need a global packages and nothing else?
  • Do we really care about additional installation like Node.js or we can use it’s minimal version, integrated into vvvv download?
  • Maybe vvvv should be distributed just as a package manager, that downloads new versions automatically and this versions is included into projects, because many people have a copy of vvvv in they project directories.
  • Do we need a cross platform solution? Or we can just stick with windows?
  • How current packs like openCV, dx11 and addonpack should be distributed?
  • How we can install packages? For example - we can integrate package manager to VVVV search box, so we always see which nodes we can use, but need to download it. Downloading is fully automatic, we need just click on node and wait some time. (Suggested by @elliotwoods)) and ((user:elias)
2

hey alg, nice research and summary! how about http://nuget.org/? that’s what i had in mind the last couple of weeks as it integrates itself nicely into our build process (building vvvv/plugins). i think it’s quite important to express a node’s dependencies once and during build- and runtime the same tool is used to fetch those dependencies. think of apt-get where i can easily download git for example and if i need to build another program, which depends on my git installation i need to install the git-dev package.
i hadn’t yet time to properly research nuget, maybe you did? i only used it in visual studio quite often the last couple of months, but never created a package myself.

regarding global and local packages: i’d allow both. both make sense.
as long as you’re on your development machine you don’t want to download and install the same packages everytime you work on a different project.
but once finished you want all runtime dependencies in one place so you can easily deploy your project on a different machine - having the needed packages installed in the project directory.

ad 2) we would probably ship vvvv with all the dependencies needed to run the packager - in case of nuget that’s only nuget.exe

ad 3) sounds nice to me. but isn’t that what Chocolatey seems to accomplish on windows?

ad 4) i’d prefer cross platform.

ad 5) as meta packages

ad 6) yes, in the end vvvv should be capable of installing packages via the node browser. but till we get there command line is perfectly fine by me.

3

I add some information:

we started a repo to use it as a template for packages

there is also a repo for p4v

Considering other package managers, I think it is a good idea to have p4v installed globally, as a chocolatey package (@elias: chocolatey uses nuget) depending on the vvvv addonpack.

On the other side, packages should install locally, i.e. inside a folder p4v (like node_modules folder for node.js): a package can contain “effects, modules, plugins” folders, but also assets and I really think that this stuff should be inside the same folder as the root patch.

A first step is to install a package, second step should be packages with dependencies and third step is p4v able to publish a package in some registry server that should be hosted under vvvv.org domain.

Other nice idea is to have a vvvv node that creates a synaptic like gui to search for packages and install them.

Also the philosophy of most of the package managers nowaday is to not chain the user and let him freedom, for example to use it’s own server, or to install from a private repository.

I think that the solution that found alg, using bower, it is ideal.

Finally I would like to say that having git and node.js as a dependency is good. Having git will be needed also to develop and publish tha package … and well, node.js will be needed by vvvvjs server, but that is another project :) btw both are two great piece of software, let’s take advantage and use them :)

GRANDE alg !!

4

@elias, Chocolatey is just a NuGet wrapper. Every it’s package is NuGet package. What Chocolatey team did - better integration for software packages via custom PowerShell scripts. For example it can install Git and reset Path variable, so you can start using it directly from command line, without any restarts. Also it can unzip packages, run silent installers and do user defined post installation scripts. It can be installed from command line, when NuGet is installed by default as VS extension (maybe i’m wrong here).

What is bad with NuGet + PowerShell way, is that it currently Windows only. I did a fast search, and looks like this can be solved with Mono and Pash2. It’s was predictable ), but no one knows, what problems we can get in this way.

Can you describe what is meta packages? Can’t understand this term.

So if we are going with NuGet + PowerShell we getting:

Benefits

  • Can be integrated into development pipeline.
  • Can install software dependencies, like Git, CMake, Gow, etc.
  • Easy to distribute.

Limitations

  • Hard to do cross-platform, but theoretically possible.
  • Need to research about offline packages, custom servers etc.
  • Windows XP and Vista users should install PowerShell 2 manually, maybe Chocolatey installer solving this.

Maybe vvvv.js just needs another package manger, like a plain Bower, that already 200% suits what it needs. So we can separate this two projects.

Waiting for more replies and details ).

5

Cool I see definitely the use of it.

Script files on start up and auto fix possibilities for broken patches.

and of course efficiency. not having to unzip stuff all the time.

h99 and me were working on Dependency (Patch). It checks the XML for all dependencies and collects them automatically. Would be happy to make it work with this approach.

Just don’t no so much about the existing systems but thanks for the introduction…

6

hmm i’ll try to integrate it in md.toolkit until it’ll make it to the nodebrowser

7

hei alg,

i’d like to hear more about:

really? if stuff is broken it should be fixed, if it is unusable it should be improved if it is deprecated it should be set to “legacy”. only it needs someone to point out such problems. specifically. not generally.

and:

don’t get me wrong, i am first in line for a package manager. the only reason we haven’t tried one yet is that i consider it a fulltime job to get this right. and we allready have multiple fulltimejobs. so if you guys will take care of it, horray.

but: still i can’t understand how you can think of 3 clear packs: addons, dx11, opencv (and of course possibly more to come) that will be guaranteed to work with specific versions as a mess. true: contributions are a mess by design, we never argued against that. the packs instead are exactly already a step in the direction of a package manager.

8

For sure it is better avoid dispersion and keep “one vvvv to rule them all” but with all the new possibilities opened since last node, I think alg refers to vux’s dx11 build, elliot’s opencv build.

I remember a very nice idea of vux, just before node he was talking about a context switch to have a dx11 context, a dx9 context, so you can modify the nodelist and find only the nodes you need.

Anyway, I agree with joreg that a package manager is a lot of work, but most of the work was done by Twitter and Linus :) so we can use bower and git.

I would like to remember the first rule of p4v is

“a package manager for language foo should be written with language foo”

so our effort is to interface users with software like bower, but, form a vvvv patch to make it as community driven as possible. In fact, if we have a package manager we also need people doing packages :)

Weel, it is still a lot of work, so I will be happy to see the first step done, i.e. “a user choose a package and installs it with one click (or whatever) in its project folder”. It will save a lot of time to every user (included me :)

Then in Italy we say “if they are roses they will bloom :)”.

9

@joreg about addonpack, for example Expr (Value Advanced) not working with custom pins, two Slider nodes, both with multitouch tags, very strange advanced S and R nodes, that i never use, etc. During workshops, all this nodes appearing in my students search lists and they are making errors. This is very hard to desribe, why we have such a big amount of similar nodes, sometimes with really small differences. With time, addonpack will contain more and more nodes and it’s entropy will be increased. Supporting this nodes for one or more persons is hard task, equal to building proper contribution system. Removing legacy nodes is very strange thing, because people not have the way how to manage dependencies, so at some point they will stick on the old versions of v4. Also this way not support branching easily, the problem that we facing now.

About packs, i talked with vux and he said that dx11 have some conflicts with current addonpack nodes, so usually he installing all needed nodes by hands. Sound’s like a 90’s, right? If you need exactly the names of nodes, that will conflict, first that come to my mind is VLC. In openCV pack, we have it’s own set of texture loaders, video players, kinect sensor nodes. Imaging when you installing it all together, how many duplicates you will get? Why you need such amount of duplicating nodes? About guaranties, how you can specify the versions of plugins, that should work with this pack?

I disagree, that packs and contributions are different. Currently, packs is just bigger, that’s all. Why packs is more “package ready”? Can’t get this.

Also, we need to rethink addonpack contribution process. Adding projects to global build project and editing this projects in text editor, to add dependencies - huge mess. If you need example, just remember that i need to add my last contribution twice, because you can’t correctly merge core build project. We should’t build all our nodes at once, we just need to control dependencies. Nodes should’t reference other nodes, they just need to reference VVVV core library, that’s the way toxiclibs dependencies working for example.

10

I use advanced S R nodes all the time. more stable than the native ones with enumerations

11

md, remember advanced S/R have a framedelay inside, that’s difference from native.
@joreg we have to make a cleanup thread at some point. (basically i’m off ODE nodes) and freeframe would love to replace by CV

12

@microdee S and R here as example of almost identical nodes, that just enlarge entropy. Let’s not go off-topic.

I researched NuGet today, and this is what we need. Can be distributed as exe file, theoretically i know how to make custom server, local and global packages, etc.

My question now, what do you think, do we need a tools like Git, Node.js etc available through our package manager. Or we can just use a scripts, to silently install another package manager, like a Chocolatey to install all this stuff?

Also, Elias, can you please describe how our package manager should be integrated into your workflow? Can you use two different command to get NuGet and P4V packages?

13

@antokhio: hmmm i didn’t know that! thanks!

14

@alg: i’d just like to argue that the issues you present won’t be solved with a package manager. a package manager will be great, but for other reasons. the reasons you present are off topic to a package manager, so i won’t be mad if you don’t follow up here, still some pointers:

  • you mention specific nodes you’re not satisfied with (exprAdv, srAdv, sliderMT), fair enough. but they won’t be fixed by a package manger.
  • so it seems to me what you’re talking about is a cleanup to the nodelist, and remember we talked about this already on irc sometime ago: you can already clean nodelist.xml that is shipping with vvvv and the addonpack (in lib\nodes\nodelist.xml). nodes you remove from there won’t show up in the nodebrowser. so if you’d create a little gui that lets you modify that file it could come handy for your workshops
  • also latest alpha supports filtering of node-categories from nodebrowser

not sure if everyone is aware of how this works in vvvv: we almost never actually remove nodes! we only set a nodes version to “Legacy” which lets it no longer show up in the nodebrowser. like this old patches don’t break, but you only won’t be able to create old nodes in new patches.
the package manager won’t solve this problem either. at some point packages just won’t be available for newer versions.

important point: because packs are built against specific versions and therefore guaranteed to work with the version which it names in its .zip-filename. for contributions this is not the case. the addonpack has no dependency problems by design (except external drivers). with multiple packs we are facing issues of shared libs which we’ll try to just ship with vvvv so that this issue is not visible to users.

absolutely true. definitely accepting solutions to this…

conclusion:
i think some problems can be solved other than creating a complex system of a packagemanager that will actually not solve those particular problems. as a historical reference i would like to point to pd where for years they only had a system of contributions (all hosted on different websites) and only later they introduced pd-extended which can be compared to vvvvs addonpack, because that way is convenient for the user. yes, installation of the packs is awkward and nodebrowser needs better filtering.

back on topic: of course a package-manager is the ultimate pro-tool and i am looking forward to see whats cooking.

15

It is cooking pasta with vodka :) is not that bad :)

16

@joreg i agree, that legacy nodes can be solved in this way. But the problem is that different people using different set’s of nodes.

About nodelist — thank you for this tip, editing node list can be a good way for teaching. We can provide a custom simplified nodelists for students.

PureData is not a good example here, it never had a proper package manager. Idea of extended versions, that contains all addons, is widespread, but i think it’s not a silver bullet. As we discussed before, i think extended version should contain core functionality addons, like Kinect and Arduino, but nothing more.

About packs, i have idea - packs can be described as a packages, that contains different small packages as dependencies. Or just a big global packages as well, but this is not good for addonpack.

I can’t understand, why you can guarantee working state for packs, and can’t for contributions? I think it’s really depends on maintainer.

17

that’s what i meant with a meta package…

after reading through the nuget, apt and chocolatey documentation i’d like to step back a little and start the discussion from a broader angle:

what all those package managers have in common is the abstract concept of a package (some metadata with optional compressed content) which can depend on other packages. it’s the responsibility of the package manager to figure out the dependency chain when the user decides to install a package.

installing a package means placing the package metadata to some defined directory and if there’s content in it, extracting it to the target machine based on the metadata description.
additionally some pre- and post install scripts of the package itself get called.
in the metadata and the pre- and post install scripts a package maintainer can use pre-defined variables, which describe where to put executables, documentation files, include files etc.
these pre-defined variables are set up by the package manager. in other words, the package manager defines the environment in which the content of the package will get extracted to.

and based on the used package manager, by default (without any other configuration) the defined environment is either the system environment (apt, chocolatey) or a local environment (nuget, i guess bower too?).

since our discussion is about giving the user the ability to extend her vvvv installation via packages, giving the developer the ability to write plugins in her own repository, i think we can agree that we want to focus on a local environment. so i’ll skip apt and chocolatey from now on and close in to nuget.

a local environment can be one of the following:

  • a vvvv installation - user
  • a vvvv project (v4p files, effect files, dynamic plugins) - user
  • a plugin development environment - well, developers
  • a vvvv development environment - devvvvs (not to forget)

let’s first look at it from a user perspective:
a user would start by downloading a very stripped down vvvv installation. could be the vvvv.exe containing a nice gui to manage the installed packages or just a preconfigured command line nuget.exe - doesn’t really matter. important is the preconfiguration. it says what version of vvvv it is (beta/alpha29.2 for example), what platform (x86/x64) and therefor knows what vvvv package repository to access. think of it like a distribution in debian/ubuntu. a beta installation would access packages.vvvv.org/29.2/beta (like stable in debian) and an alpha installation would access packages.vvvv.org/29.2/alpha (like unstable in debian). the packages found on those repositories could follow our own invented package format. for example let’s consider a package FOO:

effects*.fx
modules**.v4p
plugins**.dll
lib\Shared.dll (used by plugins)

and a package BAR:

plugins\Bar.dll
lib\Shared.dll (used by Bar.dll)

say we wouldn’t change anything in nuget, and simply use nuget.exe, when installing FOO and BAR we would get something like this:

vvvv45\packages\FOO\effects*.fx
vvvv45\packages\FOO\modules**.v4p
vvvv45\packages\FOO\plugins**.dll
vvvv45\packages\FOO\lib\Shared.dll
vvvv45\packages\BAR\plugins\Bar.dll
vvvv45\packages\BAR\lib\Shared.dll
vvvv45\vvvv.exe
vvvv45\nuget.exe

doesn’t look good, Shared.dll is there twice under different packages. so let’s introduce a third package called SHARED which FOO and BAR depend upon:

lib\Shared.dll

now we’d get:

vvvv45\packages\SHARED\lib\Shared.dll
vvvv45\packages\FOO\effects*.fx
vvvv45\packages\FOO\modules**.v4p
vvvv45\packages\FOO\plugins**.dll
vvvv45\packages\BAR\plugins\Bar.dll
vvvv45\vvvv.exe
vvvv45\nuget.exe

a bit better, but the dependency that Bar.dll and the plugins in FOO depend upon SHARED is only visible for nuget. vvvv doesn’t know anything about it, so when we’d start vvvv, how is vvvv supposed to know to load Shared.dll from SHARED\lib when Bar.dll asks for it? as you see there’s definitely some work to be done here.

but let’s move on to the developer perspective:
a developer would probably start from a different route. say the developer wants to write BAR. he/she would start visual studio, create a new solution called MyPlugins containg a class library project called Bar:

MyPlugins\MyPlugins.sln
MyPlugins\Bar\Bar.csproj

next step would be to add the required dependencies through the visual studio nuget interface. so let’s add SHARED and see what we get:

MyPlugins\packages\SHARED\lib\Shared.dll
MyPlugins\MyPlugins.sln
MyPlugins\Bar\Bar.csproj → modified to reference Shared.dll

so far so good, now let’s create the package BAR:

nuget pack Bar\Bar.csproj -Build

we’d get a BAR looking like this:

lib\net40\Bar.dll

and BAR depending on SHARED.
if you look back up to my initial BAR package you’ll see that the new BAR now follows the official nuget specs and not ours. we don’t know anymore that Bar.dll should be considered as a plugin by vvvv.

conclusion: we’ll certainly need to differentiate between development and (don’t have a better name now) runtime packages.
development packages following the official nuget specs, runtime packages following our (yet to invented) specs.

the good thing is that nuget used from inside visual studio has the notion of solution-level packages and normal packages. solution-level packages must not contain a lib or a content folder and their tools folder will be added to the PATH variable of the package manager. therefor it should be possible to provide a solution-level package, which contains a script to create runtime packages for vvvv. let’s call this package VVVV-SDK for now.

so what a plugin developer would do, is to add that VVVV-SDK package to the solution and instead of calling nuget pack, one would need to call
p4v pack Bar\Bar.csproj -Build
and p4v (wrapper around nuget) would know to create a runtime package for vvvv
plugins\Bar.dll
depending on SHARED

i imagine what’d be also very important for a developer is the ability to debug the stuff one writes. so that VVVV-SDK package should also provide tools to install a development version of vvvv inside the solution folder.

to summarize, i think we’d need

  • runtime packages - following our extended nuget specs, interpreted by vvvv in a special way - well thinking about it we could call them just p4v packages :)
  • development packages - following the offical nuget specs, interpreted by visual studio in a special way
  • to help setup a proper vvvv development environment the solution-level VVVV-SDK package - again following official nuget specs

the development packages could be postfixed with a ‘-dev’ like it’s done in all linux distributions. so there’d be the p4v package BAR and if someone needs it the dev package BAR-dev, where BAR-dev would only depend on other *-dev packages.

what do you think? would that approach work or is there an easier one? did i miss something fundamental? there’re probably still many details missing, but this post feels long enough now ;)

18

We are converging to the same idea, but instead of nuget I would use bower.

I think that the package should be local, I mean that you have a global vvvv installation that is exactly the current vvvv distribution.

Then you create a folder MyProject that will have a package.json or whatever meta file inside. MyProject folder will contain a root patch and classical effects, plugins, modules folders as usual.

Then you use some p4v vvvv node that creates a gui or is a vvvv module that let you install packages inside your effects, modules and plugins folders.

The package manager that I take as a reference is npm that istalls everything inside node_modules folder, so p4v could install inside a “vvvv_packages” folder and add it to nodelist in the root patch, for example.

Me and alg already did a succesfull experiment using bower and this test server

http://p4v.herokuapp.com/packages

One plus is that bower should be ok also for vvvvjs packages.

At the end Joreg, Elias, Alg and me are talking about the same identical thing, we just need some time to implement a first usable minimal version.

A good idea could be that you Elias wait for a first implementation and then see if you like the first step, give us some advice with design api and implementation and help us make it official.

19

simply because the addonpack is built by our buildserver for every release. contributions are not built automatically for every vvvv-release. so the chances of code in the addonpack to work for a new version of vvvv is much bigger. once we get the test-framework integrated in the build-process chances of things working are increasing even further.

20

That is another feature that should be added to a package managment system. Users should write and run tests before publishing a package.

In fact I started another repo (empty by now) called vvvv-ok where I will intent to implement a minimal test framework for vvvv.

Yes I know, I have to many crazy ideas … my boss often complains so in my spare time I break vvvv balls :)