Forum

HTML Texture endeavours


#1

Now that the new HTML Texture is out let’s talk about further development to make it industry proof:
Previously on the new HTMLTexture announcement (the one outputing IImage)

Current bugs:

  • Colors messed up Fixed in latest alpha
  • Context menu appears on right click. This should be an easy fix as CEF exposes context menu event handlers and you can modify or totally clear the context menu.
    In ChromiumFX it’s done this way:
    Fixed in latest alpha

C# code:

var client = new CfxClient();
 ...
var ContextMenuHandler = new CfxContextMenuHandler();
ContextMenuHandler.OnBeforeContextMenu += (sender, args) => args.Model.Clear();
...
client.GetContextMenuHandler += (sender, args) => args.SetReturnValue(ContextMenuHandler);

I’ll be using ChromiumFX onwards because at first impressions I’ve found it easier to use than CefGlue. And it has a “build with your own CEF build guide”.

Features and node structure
For a project we currently have at MESO we needed 2 features where you need to recompile CEF at the time of writing: h.264 codec support and real multitouch support on any third-party html document.
At first I was trying to work with the vanilla HTMLView from addonpack, but I found that for our purposes it would be cleaner to work with a concise plugin project. So I moved on to modify the HTMLTexture contribution by Gumilastik and Tonfilm.

My aim there is making it spreadable; supporting multitouch with special CEF build; creating a system which has more nodes, express input operations like object bindings and javascript executions as behavior delegates pattern (similar to layers in DX11), and have other nodes getting information from a per-slice monolithic wrapper object. Also I’m placing that vvvv wrapper object into its own assembly so other plugins can use it for their own purposes.

I’m writing down here my experiences to provide devvvvs and people here my findings about this topic. My end result won’t be as user friendly as the single monolithic nodes we have right now, but it will allow slightly more professional usage patterns. That’s also a reason why I switched from addonpack.

Multitouch
For multitouch I found this pull request https://bitbucket.org/chromiumembedded/cef/pull-requests/104/touch-processing-for-osr-different-api/diff and in the discussion I found this patch to apply: https://drive.google.com/file/d/1Sh4qrfkjql3OkdKLY08N7BuUcHUyqQIU/view (for branch 3396 (chromium 67)). Now that patch is not correct because of inconsistent folder structure so I needed to fix it (see attached file below) and even when I fixed it I needed to apply one diff myself to make it compile. Compilation of CEF takes roughly 80-90Gb storage space roughly 3-4 hours of git checkout and preparation if you have a gigabit connection and 3-4 hours compile time, while you can’t do much else because it uses all the cpu. So have an old videogame you always wanted to play lying around while you wait and forget iterative development. I won’t write full instructions here because the CEF building guide should be straightforward to follow here: https://bitbucket.org/chromiumembedded/cef/wiki/MasterBuildQuickStart. Now I managed to compile it with the touch features added and initial tests are fine, so I have 70% confidence that it will work with ChromiumFX (which I’ll start re-building today).

I attach my touch-propcodec patch to make it compile so far and some instructions:
touch-propcodec-patch-3396.zip (293.6 KB)

  1. Make sure you have Python 2.7 in the PATH environment var. Python 3.x won’t work. If you already have Python 3 there, put Python 2 above it, or delete Python 3 folders from PATH
  2. Follow instructions here until and excluding step 5: https://bitbucket.org/chromiumembedded/cef/wiki/MasterBuildQuickStart
    note: it is important to work in C:\code. If you don’t have 100 Gb free space on C you will have hard time fixing all the batch files with hard coded paths.
  3. from the attachment extract the automate folder to C:\code\automate
  4. from the attachment extract chromium_git\update.bat to C:\code\chromium_git\update.bat
  5. run that update.bat preferably from a cmd or powershell to see if it run into errors
    (this will take 3-4 hours depending on connection, it’ll download a 11 Gb compressed repo and uncompress it + couple of other operations)
  6. Extract everything from chromium_git\chromium to C:\code\chromium_git\chromium recursively and overwrite if asked
  7. Run C:\code\chromium_git\chromium\src\cef\create.bat
  8. Follow CEF build instructions from and including step 8

The ChromiumFX repo has a readme section for regenerating function bindings for new CEF releases/shenanigans which seems to be straightforward.
It’s not as straightforward as I anticipated, posted issue: https://bitbucket.org/chromiumfx/chromiumfx/issues/175/building-chromiumfx-with-own-cef-build
well managed to get over the problem.

Other interesting findings
Currently the CEF result image is copied from GPU to memory by CEF and then copied back into GPU from memory by our wrappers. This is extremely inefficient. Thankfully people are working on this issue and we have this pull request https://bitbucket.org/chromiumembedded/cef/pull-requests/158/support-external-textures-in-osr-mode/diff with this demo: https://github.com/daktronics/cef-mixer which can expose a shared texture directly from CEF so it’s not needed to copy anywhere. I’m planning to experiment with this during autumn again for our current project merging my multitouch/proprietary-codec build with this, but if anyone is quicker with this then suit yourselves ;)

Questions:
What is the price of redistributing something with the h.264 playback capability built into it?

wish me good luck and god speed!


#2

just a quick note, this is fixed in latest alphas. the addonpack was referencing an older VL.Core nuget package that had a different pixel format enum.


#3

nice! I’ll strike it out.


#4

I added instructions and necessary additional files to compile CEF with hypothetical osr touch support


#5

Indeed hiding the context menu was rather easy and should not be there anymore in upcoming build.

Features and node structure
Yes splitting things up into more nodes sounds reasonable. Personally I always postponed this until the renderer is wrapped again for VL as it comes with interface/delegate support out of the box so less boilerplate vvvv nodes to write.

Regarding the other two topics (multitouch & texture sharing) we’ll probably wait until they will be merged in the official project and go from there.

But keep us posted and thanks for your input!


#6

Update:
Now because we needed to render high resolution stuff, I needed to implement the texture sharing bit using this PR: https://bitbucket.org/chromiumembedded/cef/pull-requests/158/support-external-textures-in-osr-mode which makes it sooo much more faster than previously. Runs 4K+ resolutions silky smooth. This also means I had to update to Chromium 69. look at repo for more info about what to come, or to try compiling yourself before release: https://github.com/meso-unimpressed/MESO.HtmlTexture ;)


#7

Dávid showed his WIP at LINK, and it was seriously impressive.

He could call javascript methods easily, and set up callbacks on the vvvv side to handle incoming data, whenever the javascript calls the callback.Very bold on the topic of data binding, really.

The package was extensively spreadable, much mor modular than other attempts at HTMLTexture, and totally wholesome, leaving little else to be desired.

So anyone that traverses between html/js and vvvv can be really excited about this. It is hands down the most versatile take on HTMLTexture yet.


#8

as a quick note: (besides many other goodies) multitouch support and texture sharing is now part of official master branch and they’re also committed to release branch 3683 which is Chromium 73.0.3683.27. so it’s stable enough for my taste.
Other goodies important for us at least are:

  • API to mute audio
  • CefRenderHandler::OnVirtualKeyboardRequested

Getting raw audio buffer is very close to be merged too. They’re now only discussing documentation punctuation changes and code aesthetics.
I’m currently testing a build and will implement changes in Vanadium too