Batch read, process and write textures

Hello, what I am looking for is Load → process → and write new texture files in a batch process (reading the contents of the dir and iterate them one by one), what I tried so far was to:

  1. put it in ASYNC (it was writing only the first two-three images)
  2. in a CACHE region (it was reading and writing back all the files but most of the times it was producing artifacts, like solid black or solid white images
  3. put it in an IF statement (more or less same as Cache region behavior)

Additionally:
a. I tried a FrameCounter and GetSlice to replace the Foreach loop (like I would do in beta)
b. Introduced a main (loop) and set it to Increment (as I would do in beta)

Any suggestions ?

Yes, like lecloneur said: use reactive. It is async but you can control it in more detail than an async region. But if you do simple stuff like cropping and rescaling, I would suggest going ffmpeg.

hello @schlonzo , thanks for the response. I am using Stride there is some Skia involved as well but only for the UI of my app.
Can you elaborate a bit more.
I am getting the contents of a folder (images) and I want to read them one by one, do some stuff with them (like cropping etc) and then save them in a different folder.

You should try with Reactive as you could potentially do the Load Process Write sequence in a background thread where all the steps would be contain in their own ForEach (Reactive).

Yes, like lecloneur said: use reactive. It is async but you can control it in more detail than an async region. But if you do simple stuff like cropping and rescaling, I would suggest going ffmpeg.

Thank you both for your responses, I did use Reactive (foreach) in order to be sure that each stage of the process will be executed properly. Although I still have few issues.
In the first place, I didn’t try yest to batch process more than 200 images at once (easy to see if this would work out as expected). But the most interesting one is that if I have, lets say, two sets of images (one hundred items per each set) first set goes fine but when I jump to the next one (second folder) then it seems that data are kept in memory. So first iteration of the second set will result images of the first set. If I trigger it again, second iteration, will produce a bunch of white (blank) images and only if I trigger it for a third time I ll get what I was looking for.

Not sure why without a patch but you should probably make sure that the first async batch is finished before starting another one.

@nissidis this should be much easier to discuss if you share you patch.

Thanks @joreg you are right, here is the patch

batch_processing.vl (51.6 KB)

needs stride

The first thing to note is, that ToObservable is not doing anything in the background. I think this needs some double exclamation mark on the node. To actually leave the main thread you need ToBackground.

And then you could do everything in one ForEach, there is no need to split it up into several regions unless you want to mix and match later.

The second thing is that FileTexture is async (non-blocking), it will not output the loaded texture in the same frame (sometimes, see “Loaded” output), and you only have one frame here. Use TextureReader for that, it will block until the texture is fully loaded.

1 Like

The next thing is, that the TextureFX nodes will also not render in a blocking way (same moment as you call them), they kind of hide their GPU draw call inside. What they do is schedule their GPU work for the next Draw call of the main loop. So this is a similar problem as the FileTexture has.

so with the current design of the TextureFX nodes, you cannot do what you are trying to do, they need the main loop to work. There are possibilities to force a draw call right away, but that is also only allowed from the main thread.

You have to use OpenCV imaging nodes, that work on the CPU and do their work in a blocking way.