Forum

Making twitter plug in asynchronous, c# asynchronous looping through results

devvvv
#1

Hi the crazy gang,
As some of you may know, from vvvv facebook group, I am building the ultimate twitter API plug in for vvvv, a complete Twitter REST API 1.1 bridge. I have had great success porting over TWEETINVI C# API for Twitter to a vvvv plugin. Everything works well but now i am trying to make the plug in “asynchronous” so that instead of vvvv “stalling” for a second or two during search it nicely trickles the results through one by one (slowing down frame rate but still better than a complete freeze. As I said, I sue TWEETINVI C# API which is brilliant ANd comes with async versions of all fucntions. This is a simpel c# problem, I am not the best c# dev and have little knowledge of asynchronism in c#… This is what I use and works perfect as non async, liek that:

(after building serachparameters, I call:)
var tweets = Search.SearchTweets(searchParameter);
foreach(var tweet in tweets){
//DO LOTS (essentially copy tweet.text to listTweetText and so on…)
}

THAT WORKS FINE, now when i try and use the async verison of search then i can’t use “foreach” and i understand why but then what is the correct syntax? so here it is:
var tweets = SearchAsync.SearchTweets(searchParameter);
OR Task<List> = SearchAsync.SearchTweets(searchParameter);

And then… how do i recreate a foreach looop that basically goes through all the task result and do something EVERYTIME a new result comes through, something like “foreach(var tweet in tweets)” but that works asynchronously. Maybe: var results = tweets.REsults; if(results!=null){//do lots}

ALSO shouldn’t I make my outputs that are ISPREAD something like IAsyncReSult? Do I have to change anything in my output pin declaration to make the output pin “asynchronous”

Sorry this is actually mostly a C# question but it’s for a vvvv plug in that i am making for everyone to use(not just myself, this is a contribution)

THANKX!

#2

So… I had a C#/.NET developer friend have a look at my code and solution seems pretty simple:

Although I am calling an asynchronous function “SearchAsync” which is, well asynchronous, this function is being called from inside a function called Do_search(args1,args2…), so my code:

var tweets = Search.SearchTweets(searchParameter);
foreach(var tweet in tweets){
//DO LOTS (essentially copy tweet.text to listTweetText and so on…)
}

Is actually inside the Do_Search function AND That Do_Search function needs to be made async AS WELL to be executed asynchronously… so it’s not just about the “searchAsync” function but the function that calls it must be asynchronous as well.

Solution is to add “async” in the Do_search definition, like this:
public async void Do_Search(args1,args2…)

THEN change: var tweets = Search.SearchTweets(searchParameter) TO:
List tweets = await SearchAsync.SearchTweets(searchParameter);

And voila! (I haven’t tried it yet, will check it tonight and report)

HOWEVER: Is there anything i should do to make my OUTPUT PIN asynchronous??? Or will ISPREAD or ISPREAD work asynchronously? (I think so but you knwo, just asking…)

#3

i’m no pro in async thingies, but as far as i understood you fire up the async method and once you call await the main thread will wait until your async method is finished.
so if you do both in evaluate vvvv will still be blocked until the method is finished.

i guess you want to start the method async and not block v4 until it’s finished. so you would either have to grab the items of the list being populated everyframe from the main thread or have no output until the populating is finished.

#4

hi woei,

thankx for the input man, yeah so the async function declaration & await works but it doesn’t do what I am trying to do, which actually seems impossible: I am trying to trigger an event every time a new tweet comes in during a search, not just get an event/break(await) when the whole 100 tweets come in, which is what await does essentially. So, it is impossible BUT…

Brings me to my next question: Is there anyway to limit CPU usage of a vvvv plugin so that vvvv still has some ressources to run while in my case twitter search is done? At the moment you bang to search and vvvv stalls for about 1-2 seconds, which is ok obviously but would be nice to manage it a bit better…

Here is a screenshot of the plug in by the way… cheers man

EVVVVILTWEET.jpg

#5

limiting cpu usage of the plugin won’t do.

vvvv stalls because it’s busy with evaluating your plugin. (devs, correct me if i’m wrong) vvvv traverses the whole patch and executes each node sequentially. so each vvvv frame is as least as long as the sum of the execution of all nodes.

if you want the result in the same frame, you will have to wait for the result. if it’s just about the new tweet coming in, you could keep an internal list with the ones already retrieved and just add the new one, that should prevent the huge break.
if it’s about retrieving 100 tweets without stalling, you will have to do some threading magic.

#6

maybe have a look at this code:

this is not .net45 async but works anyway. i synchronize by pulling information in the update loop. functions are invoked by main thread.

#7

@WOEI: “if it’s about retrieving 100 tweets without stalling, you will have to do some threading magic.” Yeah bro that is EXACTLY what i am trying to do and you are right I need to do some threading magic!

Asynchronous c# won’t work because with twitter REST API you can’t detect just a new tweet coming in, all 100 tweet data(or less if limited in search parameter) comes in at same time, cannot be split into just one tweet event. HOWEVER it can be done with STREAM. I will make a stream version of this plug in eventually buyt it is a totally different API (twitter Stream API that is)

@Ethermammoth: Just the man I need! YES I am basing my plug in on your code bro, I am very aware of this twitterAPI project and I downloaded your whole source code from github. ORIGINALLY I started by trying to extend your plug in rather than making my own one, but as I explained on Skype, TweetSharp API, which you use in your plugin, is deprecated AND very buggy now. Problem stems from the fact that twitter recently changed their USERID to long instead of int and although TweetSharp dev is supposed to have fixed it, it’s not. Then some other guy did an unofficial fix - Doesn’t work either. I tried everything even had a c# dev mate look at fixing tweetsharp source code but “it’s too deep”. I tried a lot with tweetsharp but if you search for “#firstweet” (which I guess will return results with highest IDs) It crashes EVERY TIME. So I gave up extending your plug in and i restarted from scratch using different c# twitter api, namely: TWEETINVI. I am basing my code on yours however but as explained a little above asynchronous c#’ cannot be used to detect ONE new tweet, all 100 tweets come at the same time.

Seems that the solution is THREADING, any idea how to implement threading ethermammoth? Or even do you know if it is possible to use threading to stop vvvv from stalling?

Don’t get me wrong guys, this plug in works great as it is, I am just trying to make it a little less disruptive to vvvv so we don’t have ugly complete stalls, even if it’s just one or two seconds, it’s bit frustrating especially with things like rift etc.

This plugin will be released as contribution for everyone to play with obviously.

#8

@ethermammoth: sorry are you saying that your code uses threading? I saw some type of async stuff in your code though like IAsyncResult and some Oauth asynbc authorization but didn’t see any threading? Saying that I am not a great c# dev by any stretch so might have missed it, let me know where it is if it’s there… I am pretty new to c# to be honest and I know you are very good :)

Do you think for this if I start two threads and do the search in a 50% limited thread then that would improve vvvv stalling. Fuck man I don’t know how you have done it but your plug in seems to trickle results through, although I might be wrong but that is exactly what i am trying to achieve…

#9

hi evvvvil,

there is a very good example from @elias on how to do non-blocking async/await calls with progress indicator in a node: AsyncFileCopierNode.cs

#10

Hi tonfilm, thankx a lot man, this is amazing! I just had a quick look, in middle of another project but I will try and apply all this to my plug in at the week end. Thank you very much because I think it’s EXACTLY what I was looking for! I will leave this thread open just a little more to see if anyone has any other way of doing it but will flag as solution at the week end. Thankx again

#11

Thankx again Tonfilm I tried to apply this code to my plug in but I am out of my depth here, and I can’t seem to get it working, I am new to c# and yeah task/threading seems a little too much. I will try and get help from a friend, however this is the solution I am pretty sure, just need to apply it properly. thankx.

#12

@evvvvil:Yo! Post your plugin code, so may be we can help ;)

cheers

#13

WICKED man: I surprised and surpassed my little evil self last night and I got it working in the end! Here is the VERY SIMPLE solution: If you have a function that takes a while and you want to make sure vvvv doesn’t STALL / HALT while your function is running then do this (let’s say your function is “DoSearch()”)

instead of your normal “DoSearch();” call, use this:

Task MyTask = new Task(DoSearch);
MyTask.Start();

THAT’S IT!!! THAT simple bro, essentially it starts a second thread on top of the main (UI) thread, and as this thread runs outside of the main UI thread which vvvv “runs on” then it all works fine without stopping vvvv from running.

TONFILM example, although very good, was also way too complicated for what I needed but it helped me figure this out. In the end I will flag this as solution, since if someone needs to know about how to stop your plug in from halting then the simple 2 line task thing will do the trick.

@ZEOS: Nice one, I am gonna add you on Skype in case I run into some more problems… It looks like you have loads of experience making vvvv plugins, this is my first, so you know, might need your help later. Thankx for getting involved bro I appreciate it. You in London by the way? (I am in London)