VICON and QUALISYS integration into PlusServer - almost done but remaining jitter issues

Development
1

Hi everyone,

We have been working on integrating Vicon Nexus into our 3D ultrasound acquisition workflow based on PlusServer. The goal is to obtain a Vicon integration behaving as closely as possible to the native devices (Optitrack, NDI, etc.) already available in Plus, which gives perfect 3D reconstructions.

We have implemented a working solution based on the Vicon DataStream SDK. Instead of developing a new Plus device directly, we chose the following architecture:

Vicon Nexus (DataStream SDK)
        ↓
Python bridge (echoCap library)
        ↓
OpenIGTLink Transform messages on port 18946
        ↓
PlusServer 
        ↓
VirtualMixer with ultrasound images sent on port 18944
        ↓
Tracked ultrasound images

The Python bridge runs asynchronously:

  • a dedicated thread continuously waits for new Vicon frames (GetFrame() in EServerPush mode),
  • each frame is converted into a 4×4 transform,
  • transforms are streamed to PlusServer through OpenIGTLink.

We have progressively improved the implementation:

  • switched from synchronous polling to asynchronous acquisition,
  • removed unnecessary processing from the acquisition loop,
  • experimented with different timestamp strategies (system timestamps, Vicon frame-based timestamps, no explicit timestamps),
  • investigated possible OpenIGTLink buffering effects,
  • verified that the Vicon tracking itself is stable (very low calibration residuals).

The situation is now the following:

  • Probe spatial calibration works very well (RMS < 3 mm using fiducials in 3D Slicer).
  • Temporal calibration (fCal) usually converges to a stable offset (~150-200ms), although it occasionally returns incorrect values.
  • However, 3D ultrasound reconstruction still exhibits noticeable jitter/waviness during scanning.
  • Sometimes the alignment is almost perfect, then progressively drifts before becoming correct again, suggesting that the image/transform synchronization is not always consistent.

An important observation is that:

  • the ultrasound images are acquired at 60 Hz,
  • Vicon streams at 100 Hz,
  • in 3D Slicer, despite a target acquisition frequency of 30Hz, output is around 22 Hz (similar to what we have using Optitrack and NDI, which give perfect 3D reconstructions, so not a direct issue).

With exactly the same ultrasound setup and reconstruction pipeline, using the native OptiTrack device in PlusServer produces excellent reconstructions without visible jitter.

This makes us think that there is still an architectural difference between our bridge and the native Plus implementation (buffering, timestamp handling, interpolation, transform repository, or another internal mechanism).

Our implementation is available on GitHub (in private for now), and we would be very happy to collaborate with anyone familiar with the Plus tracking pipeline.

Would someone from the PlusServer development team be interested in helping us review this implementation and identify what is still missing to obtain behavior comparable to the native OptiTrack device?

Any advice on how Plus internally handles transform timestamps, buffering, interpolation, or synchronization with the VirtualMixer would also be greatly appreciated.

Thank you very much for your time!

Best regards,

Aurélie Sarcher & Fabien Leboeuf