Boygroup network error, stops listening to master

vvvv beta bug
1

Running 33.3 x86 on Win7 x64, boygrouped slave PCs will randomly show this error:

[Events.TMSortedEvent.Call: error occured in TCPIPNetwork.TMTCPIPNetwork.NetworkSyncCB](Events.TMSortedEvent.Call: error occured in TCPIPNetwork.TMTCPIPNetwork.NetworkSyncCB): Invalid pointer operation

and after a while will completely stop responding to the master. Sometimes they are still live and operate normally (UI etc.), other times vvvv froze completely. The NetworkStatus node on the master shows it as still sending data, but nothing coming back. On the slaves, no NetworkStatus activity is shown at all.

So far I’ve only seen this happen setting the master boygroup node to use UDP, I’m testing TCP now. The dynamic data being sent by the master is just the output of two LFOs, so size is not the problem, just two active links of nine total.

The errors show up about once an hour, with activity stopping roughly 12-36 hours later. This happens on two different slaves with completely different motherboards, NICs, etc.

2

Hmmm, why did the forum hyperlink that error message?

3

Tried moving to 32.1, but happens there also. Here’s the error, full log attached:

------------------------------------------------------------------------------
Exception log with detailed tech info. Generated on 11/24/2014 8:01:18 AM.
You may send it to the application vendor, helping him to understand what had happened.
 Application title: 199.165.243.171=S
 Application file: D:\Heliotrope\SW\vvvv\vvvv_45beta32.1_x86\vvvv.exe
------------------------------------------------------------------------------
Exception class: EInvalidPointer
Exception message: Invalid pointer operation.
Exception address: 0040AB62
------------------------------------------------------------------------------
Main thread ID = 2672
Exception thread ID = 2672
------------------------------------------------------------------------------
Exception stack
Stack list, generated 11/24/2014 8:01:18 AM
[0040AB5D](0040AB5D){vvvv.exe    } System.@DynArrayClear (Line 28554, "System.pas" + 45)
[004047BE](004047BE){vvvv.exe    } System.ErrorAt (Line 4349, "System.pas" + 3)
[00404802](00404802){vvvv.exe    } System.Error (Line 4360, "System.pas" + 1)
[0040AB5D](0040AB5D){vvvv.exe    } System.@DynArrayClear (Line 28554, "System.pas" + 45)
[00409EB5](00409EB5){vvvv.exe    } System.@FinalizeArray (Line 26241, "System.pas" + 138)
[0040AB54](0040AB54){vvvv.exe    } System.@DynArrayClear (Line 28545, "System.pas" + 36)
[00661EC1](00661EC1){vvvv.exe    } OSCUtils.{System.Generics.Collections}TList<System.TArray<System.Byte>>.DeleteRange (Line 873, "System.Generics.Collections.pas" + 24)
[00661648](00661648){vvvv.exe    } OSCUtils.{System.Generics.Collections}TList<System.TArray<System.Byte>>.SetCount (Line 606, "System.Generics.Collections.pas" + 6)
[006615E3](006615E3){vvvv.exe    } OSCUtils.{System.Generics.Collections}TList<System.TArray<System.Byte>>.SetCapacity (Line 595, "System.Generics.Collections.pas" + 2)
[00661A47](00661A47){vvvv.exe    } OSCUtils.{System.Generics.Collections}TList<System.TArray<System.Byte>>.Destroy (Line 683, "System.Generics.Collections.pas" + 1)
[00406794](00406794){vvvv.exe    } System.TObject.Free (Line 13025, "System.pas" + 1)
[00737833](00737833){vvvv.exe    } TCPIPNetwork.TMTCPIPNetwork.NetworkSyncCB (Line 522, "TCPIPNetwork.pas" + 101)
[0073784E](0073784E){vvvv.exe    } TCPIPNetwork.TMTCPIPNetwork.NetworkSyncCB (Line 527, "TCPIPNetwork.pas" + 106)
[0080B501](0080B501){vvvv.exe    } Events.TMEvent.Call (Line 403, "Events.pas" + 11)
[0080B820](0080B820){vvvv.exe    } Events.TMSortedEvent.Call (Line 547, "Events.pas" + 4)
[00805F20](00805F20){vvvv.exe    } MainLoop.TMMainLoop.DoTimer (Line 266, "MainLoop.pas" + 85)
[005DD70D](005DD70D){vvvv.exe    } IdleTimer.TMIdleTimer.AppIdle (Line 181, "IdleTimer.pas" + 46)
[0052C64D](0052C64D){vvvv.exe    } Vcl.Forms.TApplication.Idle (Line 10742, "Vcl.Forms.pas" + 7)
[0052B917](0052B917){vvvv.exe    } Vcl.Forms.TApplication.HandleMessage (Line 10194, "Vcl.Forms.pas" + 1)
[0052BC49](0052BC49){vvvv.exe    } Vcl.Forms.TApplication.Run (Line 10332, "Vcl.Forms.pas" + 26)
[00A4E3A5](00A4E3A5){vvvv.exe    } vvvv.vvvv (Line 42, "" + 7)
------------------------------------------------------------------------------
Call stack for main thread
Stack list, generated 11/24/2014 8:01:18 AM
[76F90C42](76F90C42){ntdll.dll   } ZwGetContextThread
------------------------------------------------------------------------------

boygroup_error.txt (25.3 kB)

4

This very simple boygroup patch threw the above error after about an hour and a half. So far I have seen this with 32.1 x86 and 33.3 x86, always with UDP - but at the much lower frame rate TCP runs at (roughly 1/15th), I just may not have run it on TCP long enough.

BGBug.v4p (14.0 kB)

5

hey mediadog,
sorry for your troubles.

we might have run into this delphi bug http://qc.embarcadero.com/wc/qcmain.aspx?d=122163 but still investigating.

thanks for your reports!

6

Hey gregsn - I always have my stuff log the output of Memory (Debug) and Memory (Debug ex9) every fifteen minutes, and those numbers stabilized quickly and then remained pretty constant:

41964.006933, 01:15:06  -  : vvvv 45 (c) 1997-dato vvvv.org
41964.006933, 01:15:06  -  :   30 fps running time    0 days  1 hrs 15 min  6 sec
41964.006933, 01:15:06  -  : phys : used= 14, total=   4194304 kb, avail=   4194304 kb
41964.006933, 01:15:06  -  : virt : used= 64, total=   2097024 kb, avail=    750288 kb
41964.006933, 01:15:06  -  : page : used=  0, total=   4194304 kb, avail=   4194304 kb
41964.006934, Texture Memory:  Auto: 2520  Manual: 0

And 40 hours later:

41965.715282, 1 day 18:15:06  -  : vvvv 45 (c) 1997-dato vvvv.org
41965.715282, 1 day 18:15:06  -  :   30 fps running time    1 days 18 hrs 15 min  6 sec
41965.715282, 1 day 18:15:06  -  : phys : used= 15, total=   4194304 kb, avail=   4194304 kb
41965.715282, 1 day 18:15:06  -  : virt : used= 65, total=   2097024 kb, avail=    740124 kb
41965.715282, 1 day 18:15:06  -  : page : used=  0, total=   4194304 kb, avail=   4194304 kb
41965.715283, Texture Memory:  Auto: 2520  Manual: 0

But then:

41965.715638, 1 day 18:15:37  *  : [Events.TMSortedEvent.Call: error occured in TCPIPNetwork.TMTCPIPNetwork.NetworkSyncCB](Events.TMSortedEvent.Call: error occured in TCPIPNetwork.TMTCPIPNetwork.NetworkSyncCB): Invalid pointer operation
41965.715638, 1 day 18:15:37  -  : To help us track this error down, enable the ExceptionDialog via the menu or starting vvvv with /showexceptions.
41965.715639, 
41965.717183, 1 day 18:17:50  *  : [Events.TMSortedEvent.Call: error occured in TCPIPNetwork.TMTCPIPNetwork.NetworkSyncCB](Events.TMSortedEvent.Call: error occured in TCPIPNetwork.TMTCPIPNetwork.NetworkSyncCB): Invalid pointer operation
41965.717183, 1 day 18:17:50  -  : To help us track this error down, enable the ExceptionDialog via the menu or starting vvvv with /showexceptions.
41965.717183, 
41965.720064, 1 day 18:21:59  *  : [Events.TMSortedEvent.Call: error occured in TCPIPNetwork.TMTCPIPNetwork.NetworkSyncCB](Events.TMSortedEvent.Call: error occured in TCPIPNetwork.TMTCPIPNetwork.NetworkSyncCB): Invalid pointer operation
41965.720064, 1 day 18:21:59  -  : To help us track this error down, enable the ExceptionDialog via the menu or starting vvvv with /showexceptions.

And a hang.

Is there any other data I could record that would help?

7

Also, I have not seen a single instance of this when using TCP instead of UDP, even after doing the reg hack to get the same rate as with UDP.

8

My TCP Boygroup test has now been running over four days without an error, so this looks to be UDP related.

9

I have a bigger project where I send a lot of data on UDP and was really wondering my ass off why it crashed irregularly every 24~ hours or so. So after reading this, checking and changing no freezes anymore. Hope this will be fixed! :)

10

@ethermammoth: what you did not mention specifically: did you also use a boygrouping setup or only use UDP nodes?

11

I only used UDP nodes without boygroup, sending some spreads over multiple UDP nodes with different ports to avoid syncing too much.

The only error from tty is this one:

[error occured in Pin._Prepare of UDP (Network Server)](error occured in Pin._Prepare of UDP (Network Server)): Invalid pointer operation

It occurs only rarely and is not followed by a crash - however i think since it hasnt happened since we changed the rate at which data is send (lowered it heavily) that it indicates to be related to the same issue. Also what happened when it really crashed vvvv, is that it froze the application without further feedback - so maybe a buffer running full. I also logged out RAM consumption, but it still got enough available.

Can upload some RAM / CPU / TTY logs if it helps.

12

@mediadog: If you still have access to the boygroup setup it would be very kind of you if you could do a test run with a latest alpha build and see if the crash still persists when running with UDP.

13

@Elias - will do ASAP and report back. Thanks!

14

It is running now, and after 24 hours none of the previous exceptions! I’l keep it churning for a few more days.

However, whenever the client first connects to the server, I do get one divide by zero exception. When I dismiss it, things then seem to run fine. The full stacktrace is attached here.

33.5-fault-slave-start.txt (23.8 kB)

15

After running the alpha for over four days, there were none of the previous problems!

However that divide-by-zero exception when the client first connects happens every time. I can probably trim down a test patch if needed, let me know.

16

If you have the time feel free to track it down and send the patch. But my guess is that it’s unrelated to boygrouping and more or less a common minor error on some plugin node taking a string as input surfacing only in alpha builds as they do have exception dialog enabled by default.

17

You were oh-so-right, a bare boygroup patch does not show the error. Awreet!