Forum

Skia Offscreen Renderer

Our goal is to render skia layers but we noticed some weird behaviour using the skia offscreen renderers.
Its really hard to pinpoint, because it acts kind of different on every machine we got.

Attached you find a Patch that should showcase the issue somehow. Just load a big image and see what happens, also restart gamma and open the patch up again.

we had problems ranging from memory leaks to crashes all kind of strange issues happening.

memorytest.vl (47.5 KB)

Thanks for the patch. The culprit seems to be a crash inside the Resize [Skia] node when connected to an image backed by GPU memory. That node is used by the viewers (tooltips and IO boxes) to scale down the image. So the IO boxes connected to those hardware accelerated offscreen renderers crash. This can be fixed on our end. Will report back once done.

There’s a new preview build coming up addressing all the issues/crashes I could find running your patch. Here’s the relevant changelog from git:

  • Fixes Resize on GPU bound images
  • The offscreen renderers share one code path now
  • Mouse and Keyboard notifications should be propagated in both versions
  • The hardware accelerated version will share one GL context per thread - fixes super expensive context switches as well as potential crashes when images move between contexts
  • The renderers will always dispose the previous rendered image/frame. Should relieve garbage collector and lead to better re-usage of resources (textures) on the Skia end
  • The LayerWidget will clone the internally rendered snapshot to ensure the image it’s handing over to the other thread will stay valid. We leave it up to the garbage collector to clean up that image - should be ok, it’s design time only.
2 Likes

Hey Elias,

thanks for looking into it. I just tried with the latest Preview 2021.4.0-0228 and i see pictures.
But, i have 3 FPS now. Allthough there would be headroom left.

There is a new patch attached that highlights 2 more findings. i only get a performance of about 25-28 fps, displaying and rendering one 4k image, no matter how big the image loaded is.

There is also a “tryout” async task renderer. Probably not the best solution but i guess it should work.
if you load a big image, and hit the write async two times, you will be rewarded with a crash. (i can send you my picture for testing purposes - 5,5 mb filesize, jpg, 4032x3024px :)

memorytest2.vl (41.4 KB)

best, leo

New build is in the making which should fix all your remaining issues. Framerate should be at stable 60 and for async image writing have a look at the attached patch - modified version of yours which adds a ImageWriter (Async) node. Should probably go to VL.Skia itself.

In any case you’ll need the latest preview build (>= 2021.4.0-0231) to run the patch:
ImageWriter_Async.vl (76.6 KB)

For reference I’ll post the relevant git changelog here as well:

  • All rendering on a thread is now done with exactly one graphics context and the lifetime of images can be controlled via ref counting
  • Replaces all usages of GLControl/SKGLControl with our own SkiaGLControl - patch editor, tooltips and renderer nodes all use the same control now
  • The SkiaGLControl as well as the offscreen renderers use one graphics context per thread. Gets rid of expensive context creation/switching.
  • The default resource limit of a Skia graphics context has been increased from 96MB to 512MB
  • Removes the Renderer (Offscreen Accelerated) node - all rendering is hardware accelerated
  • Adds new nodes Using (RefCount) and Producing (RefCount) to reference count specific resources.
  • The lifetime of the output image of the Renderer (Offscreen) is controlled by a Producing node, while nodes such as FrameDelay, S+H and HoldLatest use a Using node - that combination allows for easy feedback loops of Skia images without having to create unnecessary copies each frame. Note that this pattern could also be extended to other resource types (like OpenCV or Stride).
  • Fixes widget views used by tooltips and IO boxes not being disposed
  • Fixes Synchronizer regions (used by widget viewers) not implementing IDisposable
  • Resize node now using offscreen renderer - this improves the rendering performance of tooltips and IO boxes drastically (FPS going up from ~3 to stable 60)
  • Default value of SKImage is again one single image
  • Image and layer viewers first resize then use new ToRasterImage node - sadly the GPU image can’t be used on another thread (see https://github.com/flutter/flutter/issues/44148)
  • GPU image and GL context now get linked - prevents access violation on shutdown
8 Likes

I tried to abuse it as much as i could, but when it broke it was my fault and i clearly knew why it did that :)

Only thing i didnt quite understand is the new GRContext Node: As soon as i add it, it is active for all Renderers that operate on the Thread its in? So it also works for the offscreen renderers (if thats needed)?

If so i would wish for two things: Could we have an Integer64 for the Resource Cache Limit ( >2 GB) and could you also Expose the Number of Maximum Resources, since we did run into problems there before (if thats still a thing).

Next i will try to branch off our project to try working with the new renderers but this looks amazing so far!

The GRContext is used by all renderers (on and offscreen) in your patch. As long as they’re all running on the same thread. Should you place that node into a background thread (through reactive nodes or AsyncLoop nodes) it will give you the context of that particular thread which all renderers in that thread will use. Watch out if you pass Skia images between threads - see my post in the other thread about the async image writer.

You have access to all the methods of GRContext - so nothing should be stopping you from setting the maximum resources. But yes, we could add that as an input as well for easy access.

Also note that we switched the rendering backend as well. From Desktop OpenGL to ANGLE, which translates all calls to DirectX11. Essentially what Chrome and Firefox are doing. This should enable us to share Skia and Stride resources much better in the future - that area is still being explored.