Forum

C# await and Resource Provider

Dear devvvvs,

here is again a question on how properly patch c# async.

The code:

using (var csvFile = new FileStream("mfccs.csv", FileMode.Create))
{
    var serializer = new CsvFeatureSerializer(mfccs, timeMarkers, header);
    await serializer.SerializeAsync(csvFile);
}

Is it right to translate it to the code like that?
Will be FileStream disposed when an observable will be pushed?
Or I get it wrong? If I’m wrong, how can I tell to the Provider when I’m done and it can be disposed?

image

Thank you!

Best,
Anton

1 Like

I follow your async quests closely, I find them quite educational.

This time the screenshot actually left me with more questions than answers:

  • What is the New region here?
  • Would it be correct to call the Provider mechanism an original vl concept, or is there a c# lib where this is piggybacked from?
  • how does Provider fit into the grander scheme of things with Tasks and Observables etc

Looking forward to many more parallel insights

1 Like

shouldn’t there be a bind or bindNew somehere?
is this patch running asynchronously? i guess it blocks mainloop…

@velcrome:
it’s System.Resources.New
a great learning resource for those is the implementation of kinect2 or Astra devices. though the usecase is different here, i guess. for the devices, the things get pooled, so that only one resource exists and every subscriber connects to the same thing. here, the holdlatest is quite immediately after the using region (why?) so there will always only be one subscriber.

1 Like

instead of converting the task to an observable and trying to mimic the await functionality, i would just wait for it with a blocking method. for example Task.Result. then the Using region will know when the operation is finished and when it is time to release the resource.

if this blocks the mainloop, wrap an AsyncTask around it.

1 Like

Hi @tonfilm,

that was my first intent.
But there is no Task.Result for the Task (NonGeneric), right?

I must say it feels a bit awkward to replace 2 lines of code with so many regions and concepts.
Aren’t there a more elegant way as to make a blocking method out of the hot running task and then wrap it into AsyncTask?

Thank you for your help!

Anton

1 Like

@sebl and @velcrome,

yeah, these Camera repos are amazing.
We are missing tutorials on all of this Observable / Provider magic happening there under the hood.




Best,
Anton

1 Like

Yes, or/and a webinar: Dealing with resources and async patching patterns

5 Likes

it has two sides, async/await is c# syntax sugar that compiles to rather complex code. it creates some kind of state machine and passes data around (similar to observables). this is nice, because in C# it is quite hard to work with tasks manually.

in VL, on the other hand, AsyncTask and observables are nicely readable and there is no that much need for all the async/await stuff as in C#.

so it is a bit of a conflict between textual libraries using async/await and that VL doesn’t really need it that much. so the easiest way to work with it at the moment, is to make the async calls blocking and place AsyncTasks wherever you need it.

of course, we want to support that better in the language in future, because more and more libraries are only using this paradigm. it would involve things like creating anonymous/implicit operations in the patch whenever there is an await. and when the awaited task is finished, this anonymous operation gets called. then the question is, how do you assign nodes to this anonymous operation and so on…

this is a good quest for the new VL-Language repository. we can discuss problems and options there: https://github.com/vvvv/VL-Language/issues

I think there is static methods like Task.Wait or Task.WaitAll

2 Likes

Hi @tonfilm,

cool. Thank you for your explanations!
Yes, I understand everything you say and I was aware of the C# compiler unwrapping the await into rather huge statemachine. So it was unfair from my side to say, that C# has only 2 lines.
But still, of course, it would be cool if that particular elegance of C#'s await could be matched by the beautiful patches of VL.

And yes, thank you for the Task.Wait and Task.WaitAll hints. Found it.

I’ll post a quest in VL-Language repo.

Best,
Anton

2 Likes

@robotanton
Back to the initial question, yes, it is correct to patch it like that and you can be sure that the FileStream will be disposed of. Managing the lifetime of resources (like a file stream) was the main reason we came up with the idea of a resource provider. Your example shows nicely howto tie the lifetime of such a resource to an observable with the Using (Provider) [Observable] region. I should however mention that you could’ve skipped the resource provider entirely in this example with the Using [Observable] region.

Regarding async/await in VL. Yes, please open a quest in the VL-Language repository. For example introducing Await as a node shouldn’t be too hard, we can indeed rely on the C# compiler doing the heavy lifting, but the implications of such a feature are quite vast.

@sebl no, it does not block. Even though the patches inside the New and Using regions are executed on the main thread (the moment the HoldLatest node subscribes), the resulting serialization task wrapped in an observable will complete at a later time. The serialization will probably (not necessarily, depends on the library) run on a different thread. Once it completes, the used file stream will be disposed of.

@tonfilm
I’d not recommend using the AsyncTask region and waiting on the result inside as a replacement:

  1. It creates two tasks, one doing the actual work, the other just waiting on it and blocking the thread executing it.
  2. It could be that the async method being called is thread affine (for example expects to be called on the UI thread having a message pump) - in that case the whole thing would not work at all and probably crash.

@velcrome
The Provider was indeed our invention, tackling the problem of managing the lifetime of a resource, especially when resources depend on each other and/or many consumers need it.
In the grander scheme of things like tasks and observables - well from a theoretical perspective they could all be seen as monads

  • Task<T> with (return = FromResult, bind = ContinueWith)
  • Observable<T> with (return = Return, bind = SelectMany)
  • Provider<T> with (return = Return, bind = Bind)

allowing you to chain computations. Equipped with some operators (like Using (Provider) or ToObservable [Task]) you can even combine them as seen in this example. Note however that not all directions necessarily make sense. For example I’d have no clue what an operator going from Task<T> to Provider<U> would accomplish or even how to implement it without blocking the calling thread.

5 Likes

Hello @Elias,

thank you very much for the explanations and clarifying the matter!
I’m happy to hear, that blocking the thread just for waiting for the task is not the way to go.

We’ll continue here:

Best,
Anton

2 Likes

I understand that numerous topics got discussed here:

  • Translating the C# code to VL with its current ways of expression in order to achieve the same behavior.
  • For that sake, these topics were discussed:
    • Lifetime management
    • Threads
  • For that sake, these abstractions/technologies were discussed:
    • Tasks (aka futures or promises)
    • Providers
    • Observables
    • The interplay of these and some particular regions
    • On a more theoretical level Monads came up, as these 3 types actually can be classified in that way
  • But also on another level new ways of expression got asked for
    • in particular, a way to express a certain kind of Coroutine:
      • the Coroutine that handles Tasks via the async/await pattern in C# and how that could look like in VL

Regarding that Coroutine issue and the particular async/await:
I will try to summarize the ideas we had in the past to this way of expression. @tonfilm and @Elias already addressed this already briefly. When done right we’d get yield return in the same go.
Give me a moment to write this down.

In another explanatory issue, I’d like to address different evaluation strategies that vvvv users had the pleasure to use secretly:

  • Call by need: vvvv beta. Only compute a node if its output data is needed by other nodes (or if AutoEvaluate is on). Aka lazy evaluation
  • Call by value: VL. aka strict evaluation. Always evaluate everything inside a patch. If you want to optimize that: use constructs like Cache to avoid unnecessary computations.
  • Call by future: Kick off a computation that at some point returns and hands over a result. Define how to react then.

Regarding futures, it is worth noting that

  • both Tasks and Observables have the ability to express asynchronous computations:
    • Task will return one value when done
    • Observables are able to push us several values in the future
  • we currently have better support for Observables which can express more
  • we currently work with Regions (like Foreach [Reactive]) that define the continuation patches (regions make my patch look deep)
  • we’d like to support Coroutines in order to flatten that out
5 Likes