Decent Hap video playback solution for vvvv [GPU video decoding]

[lev]

Intro

I guess I don’t need to explain what Hap is, but just in case if you’re not aware of it, Hap is a video codec which allows to decode high-resolution videos using mainly GPU power and gives ability to play videos with alpha channel transparency. You may read more about it here.

The scope of the proposed plugin

I was pointed by multiple people that there is a lack of professional Hap player solution in vvvv, and I can develop a new plugin for vvvv (based on my existing codebase), with the following scope:
— Hap formats: Hap, Hap Q, Hap Alpha, Hap Q Alpha
— Containers: mov (preferred), avi
— Audio: yes (via VAudio)
— Loop, seek, scrub: yes
— Multiple videos: fully spreadable
— Render API: DirectX 11 (but not limited to)
— Architectures: 32-bit, 64-bit
— Supported GPUs: nvidia/ati/intel
— Best playback performance achievement: 8k@60fps Hap, 4k@60fps Hap Q

During making my past project, this plugin proved to be the only workable stable playback solution in Unity. We needed to simultaneously play 6 Full HD videos (now it’s not a limit) on a single PC.

I’ve spent some months working on similar plugin for Unity (fully workable, with the playback achievements stated above). To make the development for vvvv, a crowdfunding campaign was organized.

Crowdfunding (goal reached!)

We’re starting to accept donations today!
Donate here: http://demolition-studios.net/donations/hap4vvvv/

Campaign ends on April 1st. In case we won’t reach the goal, the made donations will be automatically sent back to all the supporters! So your money won’t get lost.

• Good to know: the initial fundraising campaign has succeeded, so the plugin will be free for non-commercial usage!

• For commercial usage, a license will be needed (thanks to vvvv store, now we can do that!), which will be possible to acquire in the second crowdfunding stage.

Initial croudfunding summary (27.01—06.02 2017)

Golden supporters:

• m-box
• wirmachenbunt
• NSYNK Gesellschaft für Kunst und Technik mbH

(perpetual license for company)

Our backers:

• David Morasz
• Sebescudie
• Shaul Tzemach
• Metrowave
• Björn
• Irwin QUEMENER
• Michael Gaidamaka

(perpetual individual license. note: i’ve decided no to require any additional upgrade for all the individual backers, but if you will use plugin a lot and will have some profit from it, you are welcome to invest in further plugin improvements)

Non-commercial supporter:
Alexander Ourth
(thanks, man!)

Additional supporters:

• schnellebuntebilder
• ViceCity Shanghai

(you will need to upgrade to a commercial license for company, we should get in touch)

Moral support:
Gareth Griffiths

My friends who gave me the idea itself and helped with organizing things:
@kbln
@idwyr
@vux

(individual license and hugs)

What’s next

Development has started, the estimated plugin arrival time to the contributions/store is March 2017.

I will probably appreciate some help from vvvv developers from time to time to be more aligned with their expectations.

Likely to introduce early bird licenses, in case if anybody did not have time to take part in the initial funding and don’t want to buy the plugin at the final vvvv store cost.

Teaser

Currently I can provide a teaser-version based on my Unity plugin. Try playing your Hap-encoded videos with it (be sure to take a look into the README file: encoding tips and links to some hi res test videos are given). The vvvv plugin is expected to work at least as good as Unity one!

The Hap player node prototype (fully discussable):

For those who interested in professional Hap playback solution for vvvv

• If you work with video in vvvv (this includes, but not limited to: VJing, VR, multi-display/projector walls and playscapes, playing alpha channel-enabled image sequences) and you think that my plugin can help you with that (and I hope it will), we encourage you to support the development!

• If you’re still unsure whether you’re going to be a commercial user or not, you should know that any donation above 50 euros will be upgradable to a full individual/company license on the second fundraising stage

• Special note for Russian citizens: please use the “Offline Donation” gateway, PayPal works for the rest of the world!

Let’s bring a better Hap playback solution into vvvv world!

[microdee]

I’ve just realized €144 is collected on vveekend vvorkshops patreon page by Sebescudie, Shaul Tzemach and Metrowave. I will pledge that into this project so it doesn’t go to waste (next week as patreon have a 5 day safety lock if you change payout settings). also I can help to make a vpack for this for vpm

[tonfilm]

what do others think?

[Rayment]

Frame access should be more accurate solution and any way - it’s quite easy to recalculate both in vvvv…

[lev]

@microdee oh… wow! I’m looking forward to give vpm a try, it looks exactly what we need here! Let’s communicate on that when the plugin will be ready. Thank you!!

[guest]

@tonfilm. Both, because of audio playback

[eno]

In my opinion this should be interchangeable with the image stack players. So a notion of a loop region is not really necessary? For the sake of synchronicity across machines, this also means that it should be possible to ‘drive’ the player by an external clock / LFO / frameindex, instead of dealing with the concept of time with play and pause in the node itself.
Maybe this is possible by setting doSeek to true permanently and just updating the seekPos pin …
As an output pin a framecount is necessary.

[eno]

maybe this also means that a fileInfo (Hap) Node would make sense, so you can avoid patching loopback logic with a framedelay on the output pins for Framecount etc.

[lev]

@Rayment @eno @tonfilm both time and frame indices input is possible, it’s just a matter of taste. something like that is possible as well, so it’d be more flexible for different needs:

I believe that some people would be more comfortable with the classic play/loop/seek interface. Should we split it into two nodes with different playback interface notations?

Right now the playback is driven by the audio clock (or the video clock, if there is no audio stream), which is rock solid.

I’ve made a quick test of having external timer as a master clock source. Results are the following:
— for video-only streams it seems to work nicely and as expected (except for efforts for supporting the reverse clock direction need)
— additional work is needed to make it work for audio-enabled videos (e.g. “smart” audio resampling is required to support various external clock speeds). The essential point is that A-V synchronization shouldn’t suffer from the frequent clock jumps / changes of the clock speed. A bit tricky, but feasible!

Have anyone tested the teaser version? Please note that I’ve changed the essential crowdfunding goal (now it’s 5 times lower), but even so we’re not nearly close to it! Guys come on!

[tonfilm]

i like enos idea of having the file info separated from the playback… because it is easy to make a simple module out of the two and pro users can patch their own playback logic.
it would even be thinkable to have a playback node that includes sound and drives the playback from the audio or video clock and another one that just takes a frame number but has no audio… or any of the listed combinations.

[lev]

@tonfilm that’s what I was thinking about! that way it should be very clear how to use these nodes

[lev]

Important update: now we have an established commercial supporter, and reaching the campaign goal will only determine whether the plugin will be released in public for non-commercial use.

Campaign ends on April 1st. In case we won’t reach the goal, the money will be automatically sent back to all the supporters!

I’ve found out today (thanks to Gareth Griffiths from Uberact) that the fact that if wasn’t explicitly stated anywhere prevented some people from supporting the development, hope it won’t be a problem now!

[tonfilm]

that’s great to hear, thanks m-box!

for developing the plugin keep in mind that the pro playback has to compete with the image stack player of vvvv, which does frame loading in a background thread and only swaps texture references in the mainloop. the idea behind the stack player is that the user sets some frame numbers that the player will load ahead of time in the background. if a specific frame number is requested for display in the mainloop it should already be in memory. automatic playback constantly loads a few frames ahead of the current time.

have a look at the helpatches of the node Player (EX9.Texture) and Player (EX9.Texture Timebased), load a DDS sequence and press CTRL+F9 to compare mainloop time.

[lev]

We’ve done it in just two days over the weekend! Thanks to everyone!
Please see the Initial croudfunding summary and What’s next sections of the top post!!

[lev]

Development WIP screenshot #1

Btw: for Hap Q, does anyone prefer getting raw YCoCg video texture over the one converted to RGB with an internal color conversion render pass? :-)

[elliotwoods]

Great work @lev!

Glad that you found a model which works for funding development and releasing it to the public (understood re: license limitations there).

If I was designing this myself i’d probably have something like the following node set:

• Reader (HAP) : Loads HAP file, outputs file reference
• Info (HAP) : Gives all file info
• AudioClock (HAP)
• VideoClock (HAP)
• FrameClock (HAP)
• HAPTexture (HAP, DX11.Texture2D)
• HAPTexture (HAP, DX11.Texture2D YUV)
• HAPTexture (HAP, EX9.Texture)
• FileStream (HAP DX11.Texture2D) <- module which wraps everything above with your current proposed interface

The Reader node accepts a Clock input. The clocks have time start/end or frame start/end for FrameClock. Also play/skip/etc controls are applied to the Clock. I’m sure that’s not quite right, but anyway I’d split the playback control elements from the file reading / texture handling / info parts

Something like that.

I generally find ‘one big node’ is good for beginners but gets frustrating quickly. So a big module / broken down into plugins is the best way to expose power and simplicity.

Feel free to ignore this of course! otherwise hope it’s useful

[sebescudie]

@elliotwoods

So a big module / broken down into plugins is the best way to expose power and simplicity.

+1 :)

[sunep]

@elliotwoods

So a big module / broken down into plugins is the best way to expose power and simplicity.

+1

[joreg]

meanwhile there is a hap survey: https://www.surveymonkey.com/r/9TWN88Y

[lev]

It’s a shame vvvv isn’t in the supported software list yet! I should work faster!
Same goes to https://github.com/Vidvox/hap/blob/master/README.md