Great stuff! I miss the node for HandsDetector…
indeed, VL.MediaPipe 0.1.0-alpha is now up with parsers for all models! talked a bit about it at last nights meetup.
6 Likes
is it possible to use the live image from a webcam as a texture while tracking with MediaPipe?
you’d have to use SpoutCam as described here: GitHub - torinmb/mediapipe-touchdesigner: GPU Accelerated MediaPipe Plugin for TouchDesigner
1 Like
Hi, tanks for the nuget, it would be possible to select a single element from the various detectors as seen in the touchdesigner video? (Like the position of a specific finger or similar)
Yeah, just dig for the finger or joint you need:
The second example in the nuget paints squares on all fingers, but you can of course filter for the one you want. Each hand has 21 including the wrist.
I did this for Genuary yesterday and it is really cool:
What I found is, that one needs to have a bright room for it to be fast. Low light needed much more detection time while a bright scene was up to 5 times faster.
Also a cleaner image seems to detect faster, my GFs webcam is very noisy and that was slower and more jittery.
I drew points between the joints for visualisation first, then used them as emitters:
1 Like
VL.MediaPipe 0.1.2-alpha now comes with PoseJointByName and FingerJointByName nodes allow you to select the respective joints via enum istead of index.
4 Likes
Great, thanks!
VL.MediaPipe 0.3.0-alpha:
- breaking change: simplified working with Hands, FingerJoints providing nodes: SelectHand, SelectJoint
- updated to underlying mediapipe-touchdesigner 0.4.0 which adds new ImageEmbedder feature
3 Likes
This is super cool!
Ive had a look ad the Finger tracking data and its 2d only, it looks like you need to sample a depthmap to insert the Z-value for each joint. ( mediapipe/docs/solutions/hands.md at master · google/mediapipe · GitHub )
Is it possible to output the depthmap?
Edit: Never mind got it working more more or less, will post a patch
1 Like
The HandDetector returns Landmarks and WorldLandmarks the latter of which has non-zero values for Z. is that also what you found?
1 Like
yes not sure if im doing it right but it looks like you have to combine them. Landmarks + Worldlandmarks with aspect ratio correction. its matching but not perfect
2 Likes
Hello! I’m working on a project where I need to implement face recognition of a person’s name. I’m considering using Mediapipe for this task. Can anyone share their experience or advice on using Mediapipe for recognizing people’s names? I would be grateful for any guidance or code examples. Thank you!
What are the current incantations/versions needed for this to work with latest version of Gamma (6.4)?
Looks like mismatch with VL.CEF?
2024/05/28 18:52:08.278 [CRT] (Sys) 0 vvvv TypeLoadException: "Could not load type '_VL_CEF_.UdoRCqSpxMEMbo74f4XZWOInitializer' from assembly 'VL.CEF.vl, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null'."
TypeName: "_VL_CEF_.UdoRCqSpxMEMbo74f4XZWOInitializer"
StackTrace:
_VL_MediaPipe_.M2aPRBXAePbOKHj546pVLKInitializer { public virtual void CollectDependencies(VL.Core.CompilerServices.DependencyCollector collector) { ... } }
VL.AppServices.AppHostBase { private void Scan(System.Reflection.Assembly assembly, bool runUserCode) { ... } }
VL.AppServices.AppHostBase { internal void SetScope(System.Collections.Generic.IEnumerable<System.Reflection.Assembly> entryAssemblies, bool runUserCode) { ... } }
VL.AppServices.NodeFactory.NodeFactoryRegistryImpl { internal static System.Collections.Immutable.ImmutableArray<VL.Core.IVLNodeDescriptionFactory> CollectFactories(VL.Core.AppHost appHost, System.Collections.Generic.IEnumerable<System.Reflection.Assembly> assemblies) { ... } }
VL.Lang.Platforms.Roslyn.CompiledSymbols { private System.Collections.Immutable.ImmutableArray<VL.Lang.Symbols.SymbolSourceReference> <get_References>g__Compute|31_0() { ... } }
VL.Lang.Platforms.Roslyn.CompiledSymbols { public virtual System.Collections.Immutable.ImmutableArray<VL.Lang.Symbols.SymbolSourceReference> get_References() { ... } }
VL.Lang.Symbols.SymbolSourceReference+<GetAdhocReferences>d__16 { private virtual bool MoveNext() { ... } }
VL.Lang.Symbols.DocSymbols { private System.Collections.Immutable.ImmutableArray<VL.Lang.Symbols.SymbolSourceReference> <get_AllReferences>g__Compute|145_0() { ... } }
VL.Lang.Symbols.DocSymbols { public virtual System.Collections.Immutable.ImmutableArray<VL.Lang.Symbols.SymbolSourceReference> get_AllReferences() { ... } }
VL.Lang.Symbols.Layer { internal static void <CollectSymbolSources>g__DoCollect|26_0(System.Collections.Immutable.ImmutableArray<VL.Lang.Symbols.LayerEntry>.Builder result, VL.Lang.Symbols.ISymbolSource symbolSource, bool isFriend, System.Collections.Immutable.ImmutableQueue<VL.Lang.Symbols.SymbolSourceReference> currentPath, System.Collections.Generic.HashSet<VL.Lang.Symbols.ISymbolSource> seen, bool firstlevel) { ... } }
VL.Lang.Symbols.Layer { private static void CollectSymbolSources(System.Collections.Immutable.ImmutableArray<VL.Lang.Symbols.LayerEntry>.Builder result, VL.Lang.Symbols.ISymbolSource symbolSource, System.Collections.Generic.HashSet<VL.Lang.Symbols.ISymbolSource> toBeSkipped) { ... } }
VL.Lang.Symbols.Layer { public static VL.Lang.Symbols.Layer Document(VL.Lang.Symbols.DocSymbols document) { ... } }
VL.Lang.Symbols.PreCompilation { private void Initialize(VL.Lang.Symbols.PreCompilation previousCompilation, System.Threading.CancellationToken token, System.IProgress<VL.Model.LoadMessage> progress) { ... } }
VL.Lang.Symbols.PreCompilation { internal static VL.Lang.Symbols.PreCompilation Create(VL.Lang.Symbols.IPlatform platform, VL.Model.Internal.Solution solution, VL.Lang.Symbols.PreCompilation previous, VL.Model.CompileOptions options, System.Threading.CancellationToken token, System.IProgress<VL.Model.LoadMessage> progress) { ... } }
VL.Lang.Symbols.PreCompilation { internal VL.Lang.Symbols.PreCompilation WithSolution(VL.Model.Internal.Solution value, VL.Model.CompileOptions options, System.Threading.CancellationToken token, System.IProgress<VL.Model.LoadMessage> progress) { ... } }
VL.Lang.Symbols.PreCompilation { public VL.Lang.Symbols.PreCompilation WithSolution(VL.Model.Solution value, VL.Model.CompileOptions options, System.Threading.CancellationToken token, System.IProgress<VL.Model.LoadMessage> progress) { ... } }
VL.Model.VLSession+<>c__DisplayClass194_0 { internal VL.Lang.Symbols.PreCompilation <PrepareSolutionWithFreshSymbolsAsync>b__0() { ... } }
System.Threading.Tasks.Task`1 { internal virtual void InnerInvoke() { ... } }
System.Threading.ExecutionContext { internal static void RunFromThreadPoolDispatchLoop(System.Threading.Thread threadPoolThread, System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) { ... } }
System.Runtime.ExceptionServices.ExceptionDispatchInfo { public void Throw() { ... } }
System.Threading.ExecutionContext { internal static void RunFromThreadPoolDispatchLoop(System.Threading.Thread threadPoolThread, System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) { ... } }
System.Threading.Tasks.Task { private void ExecuteWithThreadLocal(System.Threading.Tasks.Task& currentTaskSlot, System.Threading.Thread threadPoolThread) { ... } }
System.Runtime.ExceptionServices.ExceptionDispatchInfo { public void Throw() { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void ThrowForNonSuccess(System.Threading.Tasks.Task task) { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task, System.Threading.Tasks.ConfigureAwaitOptions options) { ... } }
System.Runtime.CompilerServices.TaskAwaiter`1 { public TResult GetResult() { ... } }
VL.Model.VLSession+<PrepareSolutionWithFreshSymbolsAsync>d__194 { private virtual void MoveNext() { ... } }
System.Runtime.ExceptionServices.ExceptionDispatchInfo { public void Throw() { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void ThrowForNonSuccess(System.Threading.Tasks.Task task) { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task, System.Threading.Tasks.ConfigureAwaitOptions options) { ... } }
System.Runtime.CompilerServices.TaskAwaiter`1 { public TResult GetResult() { ... } }
VL.Model.VLSession+<UpdateCompilationAsync>d__192 { private virtual void MoveNext() { ... } }
Hey @beyon
I tested on gamma 6.4 and it works right after instsalling the nuget.
Also might need ther VL.CEF.Stride 0.5.3
If you see the duocument menue is red, click on the dependencies > missimg dependenciy > install the exact refrenced version.
Getting VL.MediaPipe itself working was easier by removing all related nugets and starting again.
For VL.CEF.Stride I had to specify same version as VL.CEF installed after installing VL.MediaPipe and not the exact referenced version.
So currently nuget install VL.CEF.Stride -Version 0.5.5 for VL.MediaPipe.0.4.0-alpha
