You may have heard that Microsoft is winding up its production of Kinects and that the responsibility for the tech is getting handed over to Orbbec. No? Read here:
Orbbec make a series of time-of-flight cameras and I was wondering if anyone’s had a chance to play with one.
The SDK can be seen at the bottom, but on 2nd viewing its a bit confusing → can we use a Kinect with the Orbbec SDK or can I use an Orbbec with Azure Kinect wrappers? Does this mean we need another Nuget?
H
Came here to basically ask the same question… has anybody made experiences yet with the Orbbec Femto Bolt camera, is there support for it already?
The camera is compatible with the Orbbec SDK for both Windows and Linux. This SDK also features an Azure Kinect Sensor SDK Wrapper, allowing software developed for the Azure Kinect development kit to be compatible with Orbbec cameras.
So I’d say yes one would work for both but that that orbbec one will need updating. Someone with an azure could check it out.
Hmm, it not going to be as simple as that.
The VL.Azure Kinect nuget uses K4A. When I tried switching it with the Orbbec version, Gamma crashed.
I cloned the VL.Azure Kinect repo to see if it would be as easy as pointing to the right dll, or changing the names of the functions in CSharp. In fact I think this needs to be recompiled to use with Microsoft.Azure.Kinect.Sensor, which is a .net wrapper.
Then the Orbbec K4A can be used as a native dependency, and probably with Azure Kinect Body.
If the project were being developed in c++ the only requirement would be to swap out the k4a files
can you explain in detail what exactly you did there?
what makes you think so?
If you have the device (Bolt or Mega), it will use a standard UVC protocol so it shouldn’t require additional drivers. Check the hardware is working by downloading the Orbbec Viewer. You may ave to try a few different cables and usb ports.
Download the compiled binaries from Orbbec’s K4A repo. Unzip it and in the bin folder you’ll see these files
You can check that these work by running k4aviewer.exe which should show the sensor running
(While using Gamma preview 0253) install VL.Azure>Kinect Nuget.
Go to the C:\Users\Igor\AppData\Local\vvvv\gamma-preview\nugets\VL.Devices.AzureKinect.1.4.1\runtimes\win-x64\native folder
and replace the depthengine_2_0.dll, k4a.dll and the k4arecord.dll files with those of the Orbbec ones downloaded.
Try to open a patch
Analysis from Windbg:
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
*** WARNING: Unable to verify checksum for C:\Program Files\vvvv\vvvv_gamma_5.3-0253-g8ddcaf050d\package-cache\VL.CoreLib_HDE_Plugin.2023.5.3-0253-g8ddcaf050d\VL.CoreLib.HDE.vl.dll
*** WARNING: Unable to verify checksum for C:\Program Files\vvvv\vvvv_gamma_5.3-0253-g8ddcaf050d\vvvv.exe
*** WARNING: Unable to verify checksum for C:\Program Files\vvvv\vvvv_gamma_5.3-0253-g8ddcaf050d\VL.AppServices.dll
*** WARNING: Unable to verify checksum for C:\Program Files\vvvv\vvvv_gamma_5.3-0253-g8ddcaf050d\packs\dependencies\Stride.Core.IO.dll
*** WARNING: Unable to verify checksum for C:\Program Files\vvvv\vvvv_gamma_5.3-0253-g8ddcaf050d\Stride.Core.dll
*** WARNING: Unable to verify checksum for C:\Users\Igor\AppData\Local\vvvv\gamma-preview\nugets\VL.Devices.AzureKinect.1.4.1\runtimes\win-x64\native\OrbbecSDK.dll
KEY_VALUES_STRING: 1
Key : Analysis.CPU.mSec
Value: 9530
Key : Analysis.Elapsed.mSec
Value: 15649
Key : Analysis.IO.Other.Mb
Value: 0
Key : Analysis.IO.Read.Mb
Value: 3
Key : Analysis.IO.Write.Mb
Value: 4
Key : Analysis.Init.CPU.mSec
Value: 18265
Key : Analysis.Init.Elapsed.mSec
Value: 147870
Key : Analysis.Memory.CommitPeak.Mb
Value: 16516
Key : CLR.Engine
Value: CORECLR
Key : CLR.Version
Value: 6.0.1122.52304
Key : Failure.Bucket
Value: HEAP_CORRUPTION_ACTIONABLE_BlockNotBusy_DOUBLE_FREE_80000003_ntdll.dll!RtlpFreeHeapInternal
Key : Failure.Hash
Value: {3cadf81c-0091-a5b2-56fb-a31043faa9c9}
Key : Timeline.OS.Boot.DeltaSec
Value: 320429
Key : Timeline.Process.Start.DeltaSec
Value: 191
Key : WER.OS.Branch
Value: vb_release
Key : WER.OS.Version
Value: 10.0.19041.1
Key : WER.Process.Version
Value: 2023.5.3.0
NTGLOBALFLAG: 0
PROCESS_BAM_CURRENT_THROTTLED: 0
PROCESS_BAM_PREVIOUS_THROTTLED: 0
APPLICATION_VERIFIER_FLAGS: 0
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007fffe27af202 (ntdll!RtlReportCriticalFailure+0x0000000000000056)
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 1
Parameter[0]: 0000000000000000
FAULTING_THREAD: 00003a08
PROCESS_NAME: vvvv.exe
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
EXCEPTION_CODE_STR: 80000003
EXCEPTION_PARAMETER1: 0000000000000000
ADDITIONAL_DEBUG_TEXT: Followup set based on attribute [Heap_Error_Type] from Frame:[0] on thread:[PSEUDO_THREAD] ; Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]
STACK_TEXT:
00000000`00000000 00000000`00000000 ntdll!RtlpFreeHeapInternal+0x0
STACK_COMMAND: !heap ; ** Pseudo Context ** ManagedPseudo ** Value: ffffffff ** ; kb
SYMBOL_NAME: ntdll!RtlpFreeHeapInternal+0
MODULE_NAME: ntdll
IMAGE_NAME: ntdll.dll
FAILURE_BUCKET_ID: HEAP_CORRUPTION_ACTIONABLE_BlockNotBusy_DOUBLE_FREE_80000003_ntdll.dll!RtlpFreeHeapInternal
OS_VERSION: 10.0.19041.1
BUILDLAB_STR: vb_release
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
IMAGE_VERSION: 10.0.19041.3570
FAILURE_ID_HASH: {3cadf81c-0091-a5b2-56fb-a31043faa9c9}
Followup: MachineOwner
---------
I’m complete ignorant to what any of it means.
two things:
Yes, its works fine.
Replacing just the k4a.dll, and k4arecord.dll and .pdb’s (+ OrbbecSDK.dll and depthengine_2_0.dll) worked successfully with the Azure Kinect Viewer.
Okay, an update.
I rebuilt the k4a from scratch using Orbbec’s repo. Which should produce a dll compatible with vl.and Femtos. However:
Error time -Femto Vl Kinect: EmitException: "error CS1705: Assembly 'Microsoft.Azure.Kinect.Sensor.Record' with identity 'Microsoft.Azure.Kinect.Sensor.Record, Version=1.4.1.0, Culture=neutral, PublicKeyToken=6f6471c4ba98fac7' uses 'Microsoft.Azure.Kinect.Sensor, Version=1.4.1.0, Culture=neutral, PublicKeyToken=6f6471c4ba98fac7' which has a higher version than referenced assembly 'Microsoft.Azure.Kinect.Sensor' with identity 'Microsoft.Azure.Kinect.Sensor, Version=1.4.0.0, Culture=neutral, PublicKeyToken=6f6471c4ba98fac7'"
StackTrace:
VL.Lang.Packages.PackageCompiler+<CompileCoreAsync>d__31 { private virtual void MoveNext() { ... } }
System.Runtime.ExceptionServices.ExceptionDispatchInfo { public void Throw() { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void ThrowForNonSuccess(System.Threading.Tasks.Task task) { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) { ... } }
System.Runtime.CompilerServices.TaskAwaiter`1 { public TResult GetResult() { ... } }
VL.Lang.Packages.PackageCompiler+<CompileAsync>d__29 { private virtual void MoveNext() { ... } }
System.Runtime.ExceptionServices.ExceptionDispatchInfo { public void Throw() { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void ThrowForNonSuccess(System.Threading.Tasks.Task task) { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) { ... } }
VL.Lang.Packages.PackageCompiler+<CompilePackagesAsync>d__28 { private virtual void MoveNext() { ... } }
System.Runtime.ExceptionServices.ExceptionDispatchInfo { public void Throw() { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void ThrowForNonSuccess(System.Threading.Tasks.Task task) { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) { ... } }
VL.Model.Internal.Solution+<RestoreAsync>d__14 { private virtual void MoveNext() { ... } }
System.Runtime.ExceptionServices.ExceptionDispatchInfo { public void Throw() { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void ThrowForNonSuccess(System.Threading.Tasks.Task task) { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) { ... } }
System.Runtime.CompilerServices.TaskAwaiter`1 { public TResult GetResult() { ... } }
VL.Model.Internal.Solution+<LoadDocumentAsync>d__11 { private virtual void MoveNext() { ... } }
System.Runtime.ExceptionServices.ExceptionDispatchInfo { public void Throw() { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void ThrowForNonSuccess(System.Threading.Tasks.Task task) { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) { ... } }
System.Runtime.CompilerServices.TaskAwaiter`1 { public TResult GetResult() { ... } }
VL.Model.Solution+<LoadDocumentAsync>d__53 { private virtual void MoveNext() { ... } }
System.Runtime.ExceptionServices.ExceptionDispatchInfo { public void Throw() { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void ThrowForNonSuccess(System.Threading.Tasks.Task task) { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) { ... } }
VL.HDE.PatchEditor.NavigationMenuProvider+<DoOpenDocumentsAsync>d__44 { private virtual void MoveNext() { ... } }
AggregateException: "One or more errors occurred. (error CS1705: Assembly 'Microsoft.Azure.Kinect.Sensor.Record' with identity 'Microsoft.Azure.Kinect.Sensor.Record, Version=1.4.1.0, Culture=neutral, PublicKeyToken=6f6471c4ba98fac7' uses 'Microsoft.Azure.Kinect.Sensor, Version=1.4.1.0, Culture=neutral, PublicKeyToken=6f6471c4ba98fac7' which has a higher version than referenced assembly 'Microsoft.Azure.Kinect.Sensor' with identity 'Microsoft.Azure.Kinect.Sensor, Version=1.4.0.0, Culture=neutral, PublicKeyToken=6f6471c4ba98fac7')"
InnerExceptions: "1"
StackTrace:
VL.HDE.PatchEditor.NavigationMenuProvider+<DoOpenDocumentsAsync>d__44 { private virtual void MoveNext() { ... } }
System.Runtime.ExceptionServices.ExceptionDispatchInfo { public void Throw() { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void ThrowForNonSuccess(System.Threading.Tasks.Task task) { ... } }
System.Runtime.CompilerServices.TaskAwaiter { private static void HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task task) { ... } }
VL.Model.VLSession+<>c__DisplayClass186_0+<<RunAndForget>b__0>d { private virtual void MoveNext() { ... } }
I don’t have a crash, but I get this, and Azure.Kinect patches won’t open. So I think there are a couple of gotchas:
another sanity check: instead of replacing files in the nugets runtimes\win-x64\native try removing them there and putting the new ones, including the additional OrbbecSDK.dll, next to vvvv.exe
but also indeed, for the c# wrapper we’re using the custom build mentioned here to get support for the recorder. though this shouldn’t interfere as this didn’t change the native binaries.
@joreg the “PlaybackExtensions” uses Microsoft.Azure.Kinect.Sensor.Record
have you tried this?
not sure what you’re referring to here!?
Yes short answer it goes bang.
Long answer, I pasted them in after a clean install while running an example. It showed some device info, but no pictures came through. When I closed that project and ran another Gamma crashed silently.
(Edit:)
Further
and…
Trying to compile the Orbbec k4a.dll against the ValdPerrformance version, and it complains about version.cs being missing. I’ll look again in the morning.
Hey everybody,
Are there any new findings on setting up the Femto Bolt for the AzureKinect SDK? Has anybody been able to use it with full functionality including the color image so far?
Hi @maxs
Since there was an update in the the Summer, we found that the Femto Bolt does work with the Azure Kinect nodes in gamma.
I don’t recall the exact details of how we got it working but it might have been as simple as a firmware update.
@ravazquez could you give us some insight here?
From the SDK (Windows) package install the sensor driver
From Orbbec’s K4A wrapper (Windows) copy the dlls from the bin folder to the VL AzureKinect nuget:
"...nugets\Vl.Devices.AzureKinect.1.5.0\runtimes\win-x64\native\"
Hope it helps @maxs
Thank you a lot @Hadasi !
Works perfectly with your description for the standard AzureKinect integration.
Have you by chance also tried to use the VL.Devices.AzureKinect.Body nuget with the Femto? Because there I get an exception if I try the same thing and exchange the dlls with the ones from the K4A wrapper.
Here is the exception:
Unexpected exception: result = K4A_RESULT_FAILED
Microsoft.Azure.Kinect.BodyTracking.Exceptions.AzureKinectBodyTrackingCreateException: result = K4A_RESULT_FAILED
at Microsoft.Azure.Kinect.BodyTracking.Tracker.Create(Calibration calibration, TrackerConfiguration configuration) in C:\source\src\csharp\Micro
soft.Azure.Kinect.BodyTracking\Tracker.cs:line 32 at VL_Devices_AzureKinect_Body.Devices.AzureKinect.BodyTracker_CsQK4hmPxTfL0XAw2o7C7A.<>c__DisplayClass3_1.b__0()
at System.Reactive.Linq.ObservableImpl.Using2._.Run(Using2 parent)
