Pure image treatment question: wich evolution for vvvv?

vvvv beta question
21

@joreg: i m not affaid at all about community skills ! ;-)
i hope you were not affected by this point of view. I like to push the idea more and more until the end. for me effects are part of the natural core of a video software.

so purely it appears, as mentionned previously , the devvvv’s choice is collaboration and community ;-)
ok. got it and respect. i will forget they are modules , i promise ! ;-)

ah yeah! ouch! this will rock(et) !!!

dont know how to begin, and surely there will be tons of askings!
But really thanks !

I know that a lot is present in shaders contribs, and perhaps a knowing eye should be set to them before coding.

From the following list you may perhaps diagnostic the more little shaders/operators needed, that can be combined by user to obtain the described result after .
Maybe splitting the texture in RGB channels in 3 textures and recombine it after could be also possible ?

What do you think about it ? Little shaders giving by end following results, or complete shaders ?

COLOR ENHANCEMENT:
-RGB color balance on Dark Tint / Half Tint / Lighted tint
-Curves for R G B and Global (input: spread)
-Inversion ( general, or per channels RGB )
-Min / Max value ( R G B or ALL)
-Luminosity / contrast on Dark Tint / Half Tint / Lighted tint
-Threshold ( R G B or ALL)
-Monochrome on a selected color, with contrast movable on Dark Tint / Half Tint / Lighted tint

COLOR MANIPULATION:
-Dark enhancement (rounding %inpu nearest pixels to became black or dark)
-Color reduction
-Color remplacement

AMELIORATION:
-Sharpness
-openCV dilate
-openCV erode
-Noise reduction

BLUR: ( Spreadables ? multiple sources ? on a B&W texture map ?)
-Simple Blur
-Linear Blur
-Spiral Blur ( Kalle did a nice one but very hungry about ressources)
-Gaussian Blur
-Cynetic Blur ( Linear / Radial / Zoom )
-Star Blur

NOISE:
-Noise Random mixer ( Seed / Random factor / Iterations)
-Noise expander ( X Y Pixels expansion)
-Noise on RGB ( X Y Pixels offset range)
-Noise on HUE/ VAL ( X Y Pixels offset range)
perhaps thinking of receiving a texture to generate a moving " natural " noise .

EDGES:
-Sobel
-Contour
-Laplace
-Neon
-Gaussian difference based
and avability to determine the edge size, and color
About edges, Gimp’s set of edge is really complete and nice. About Sobel.fx original it is great with its 4 techniques: B or Wh, keeping source image background or not. So rendering type is something perhaps to keep ?

COMPOSITING TOOLS:
-Mask creator
-Alpha Chanel creator
-Chroma Key ( Low threshold, high threshold, blur, color)
-Color Replacement
-Color Reduction

EFFECTS:
-Posterization
-Solarisation
-Pixelate
-Glow 2D

BLENDING:
little blending nodes of ONE operation only, with 2 TEX input, would be really great.
-Tint based
-Light
-Shadow
-Blend
-Darken
-Product
-Density of color + / -
-Lighten
-Screen
-Softlight
-Highlight
-Hardlight
-Difference
-Exclusion

Anyway, many thanks to all ( including contributors)

22

it’d be amazing to see all this come together and would be great for vvvv! nice list.

However, if this is going to happen you need some decent management of this as a project or it will just fall apart. I’ve no idea about running a project like that where lots of people can contribute and develop.

Anyone got any good ideas where to start and where to host a project like this?

23

Is this what we’re talking about?

Blend Module.zip (4.9 kB)

24

hi Gilbi, yes, but without enumeration.
and little little “nodes”, like daisy chain example from Joreg. ( Need to get ride of the Grid / scale / etc)

could be fine to have a structure of FX with inside shaders code SAME naming of variables ( to let open the possibility to any one to concatenate 2 shaders easely without fighting with variables). A sort of naming convention also.

this structure should be the more easiest simple FX possible to enable chaining and creative work:

Do a blur:

[a type of manipulation A of pixel shader](a type of manipulation A of pixel shader)

           V

[a type of manipulation B of pixel shader](a type of manipulation B of pixel shader)

           V

         Result

Do a radial blur:

[a type of manipulation B of pixel shader](a type of manipulation B of pixel shader)

           V

[a type of manipulation D of pixel shader](a type of manipulation D of pixel shader)

           V

[a type of manipulation A of pixel shader](a type of manipulation A of pixel shader)

           V

         Result
25

@Karistouf Wrong links there…

@Gilbi I think the resolution to renderer and dxTexture is redundant, also the Enable should be a Enable/Bypass, done with a switch node for example, enable should also go to the renderer to optimize.

26

Sorry io not link just brackets to figure out a node in the text.

@xdnitro: maybe creating a new wiki vvvv page to first try figure out naming conventions?

More generally to everybody: i m not writting shaders so i will not have real interresting suggestion about shaders: is this idea of very very little shaders is stupid or not ? You know shaders code so its you that could mostly define the components

27

Io: yeah bypass is a real good idea ! But im not seeing a not consuming ressources way than joreg proposal

28

You want blend modes as separate modules, That’s a bit weird isn’t it? Surely when ever you use blend modes you want to be able to easily compare different blends without having to create and patch in a new fx? How is that easier to do?
One thing you should bare in mind with things likes blurs, is they are expensive if you want them to look good… Gimp isn’t realtime so can use more processing to make a nice blur…
Trapcode star glow, again not realtime by long shot…
I can see the point of making modules out of them but chaining several together at hd could be slow, and I think that would be the same for any software, including touch?
Having said that, the best realtime blur I’ve seen is a quartzcomposer plugin by vade, and he reduces the texture dimensions the more blur you use, but obviously does a few steps as it remains smooth, but mire than one of them on hd, and it will affect your frame rate…
There is such a thing as rendering layers out to avis … I often use a mix of realtime and rendered to give the best performance… I have a feeling the the wishlist vux suggested with a absolute disable might be needed here?

29

Made these few filters this morning
please test them (for bugs etc)

I went my own way with the patching for now, later it can be fixed/ported according to the “spec”

TG5.ZIP (109.4 kB)

30

Hi Cat, i know this seems quite strange. ;-)

I m ok with you about testing convenience.

But obviously you may encounter on hudge patches troubles of compilation because of too much techniques. So by end, the shader will not compile at all, and you will remain the Tfix function.

I never gone to HD, but from what i tested, in 1280*728 on VVVV and TouchDesigner, is that TD way is less consuming ressources. Despite its opengl, wich is known to be less quick than directx.

Actually trying to figure out, and thats the main point to be discuss with everybody, is:

-complete complex shaders
or
-little shaders chained to create complex effects

On this base little separated shaders are for my point of view (and i may wrong), light weighted, more simple to use,in a compositing chain where they could appear 3 or 4 times at different steps of the imageprocessing, than a big blending mode with 16 enums.

Ah! forgetting last thing, the Ord2Enum is still buggy, am i right ? Do we have still a bug on enumeration list ? I stayed to beta23.

31

@unc: a big one ! thats really very brillant !!! damned! you worked like hell on that topic, and its running great !

what is your opinion on all those discussion ?

32

i think performance may not be a problem… You guys just need to get yourselves some good nvidia cards :D
and, i still cant find a way to retrieve texture format from texture input… Personally i’d like to always work with A16B16G16R16F, but that is not always needed, and quite hungry for videomemory
maybe we also need some “conversion” modules, to change resolution/textureformat

33

hi unc, from what you have written this morning, the Edge question is complete, playing with the 5 shaders you wrote gives all ( and more ) look aspect than a Gimp would do. Ah yeah… that is great !

TGPosterize is great !

about texture mode, well, its you that are knowing what should be done… maybe as you suggested coversion modules would be better ?

34

the point is, each module should output exactly the same type of surface it gets on input. And if user wants to change the format or resolution at some point (for optimization purposes or anything else), all it will take is to add one conversion module where it is needed
i totally dont like the idea of adding one additional input pin for each module, just to specify its output texture format

it would be nice if each module would have just texture in/out and effect-specific parameters, and nothing else

35

@karistouf
can you run some kind of a stress-test for those effects? How much of them you can stack until it kills performance? Which ones are most slow?
I need to have some clue to know what to optimize there…

36

ok, understood. ;-)
TGP Posterize with your quick blur is giving vectorized results ! very very very nice.

I have also a real question, i dont know if this can be achieved in VVVV with shaders:

Is it possible to set a transparency channel to the video ( like a png have ), and to separate in 3 component RGB the information ?
Treatement, then recomposite it.
I dont know if EX9 renderer and DX9 texture will keep it or not.

For versatility question, specific treatment on the plan of the image channel should be also interresting and very powerfull, enabling displacement, color effects and compositing very weird.

37

@unc: brilliant, this is exactly what i was preaching.
Info (EX9.Texture) has a Format output. ok it is a string including the format descriptor + additional stuff which could easily be truncated before the ‘:’, ja?

hosting

to answer xd_nitros question: lets host this on sourceforge, so it will be automatically included with the addonpack. i’ve added a directory there which can be checked out via:
http://vvvv.svn.sourceforge.net/svnroot/vvvv/modules/trunk/common/TextureFX
in order to commit you’ll need a sourceforge account and tell me your username so i can add you there.

directory structure

instead of one directory per module how about:
TextureFX\effects (which will include all the .fx files for use by any of the modules)
TextureFX\ (which will include all the modules that can reference any of the \effects)
because else in the case of two modules using the same effect we’d need to duplicate those!?

naming

lets stick to vvvvs naming conventions. most importantly all modules should be named like this
Name (EX9.Texture FX)
where name should not include any prefix like TG. note that there are explicit author and tag fields in the module-info (via ctrl+M in a module). same for the effects themselves where that info is specified via comments.

concerning the pins names please start them with capital letters and don’t use camel casing but separate words by space.

pins

as it looks every module will have the following inputs:

  • Texture In (in case of multiple inputs name them ‘Texture A’ , ‘Texture B’, …)
  • Enabled (rather than Bypass i suggest)
    and outputs
  • Texture Out
  • Compiled (as unc suggested, which i think is a good idea. just make sure then that even modules not using any effect also have this output, always set to 1)
    internally the Enabled and the effects Compiled flag can be ANDed to switch the module to bypass in case either is false.

additionally modules have parameter inputs. those must not be of types: mesh, renderstate or transform but only of types: string, value, color, enum

any additions objections to those specs? lets keep the discussion here and i’ll move this to a wikipage.

@unc: would be great if you could then adapt your samples to this spec and commit your stuff to the repository mentioned above.

38

A suggestion: Would it maybe make sense (performance and texture-format-wise) to code all the effects into one shader, and give this shader a string describing the effect chain and a spread with the params? e.g. “blend+levels+edgedetect+blend” and “0.5 0.1 1.0”
then the effect chain could easily be parsed/constructed/patched, but it’s effectively all in one shader with only the relevant parts switched on. i’m just not sure how to handle the parameter list in an effective way.

edit: OK too much activity in this thread, I think you can partially disregard what I wrote.

One question remaining: is there a performance overhead by chaining many effects together, as opposed to all of them in one shader?

Also: Please, when implementing and naming effects and blend modes, please stick to some kind of standard (the photoshop way would be best). In Resolume Avenue, for example, some blend modes have a different effect. Very confusing and frustrating sometimes.

Also II: icanhas lumakey FX (I only saw chromakey)?

39

@hunc: i will do “stress-tests” tonight. some job to do in a theatre today

but what i already feel by putting them of 2 different computer is that already they are simple, precise, and not consuming a lot !

@joreg:
About naming variable inside, are you of this point of view, of a unified naming ?

cheers

40

@joreg
yep, that spec looks totally fine
i’ll convert those effects to it, and will stick to in future ones

and btw Info (EX9.Texture) does not output all texture types… A16B16G16R16F for example - it says 64-bit float format blablabla
i’m starting to think of some stupid way to solve this:D