Seeking Guidance: IMU-based Tracking for Real-time 3D Ultrasound Reconstruction using Plus Toolkit and Slicer

Support
,
1

Dear Slicer and Plus Toolkit Community,

I am currently working on a project for real-time 3D ultrasound reconstruction. My work is largely guided by the excellent tutorials provided by the developers of 3D Slicer and the Plus Toolkit.

Project Background:

The unique feature of my project is the use of an IMU (Inertial Measurement Unit) to track the ultrasound probe. I understand that IMU-only tracking comes with significant challenges, such as noise and drift, compared to optical tracking systems. However, I have chosen to adopt IMU tracking due to the following reasons:

  1. Cost and Size Advantages: Using an IMU reduces both the cost and the physical footprint of the tracking system.
  2. Upcoming High-Performance IMU: There is a new IMU, compatible with Sony’s Spresense microcontroller board, that is expected to offer high precision and low noise (details here). My goal is to explore whether this new IMU can enable standalone tracking for ultrasound applications.

Current Challenges:

The primary issue I am facing is that the current Plus Server does not natively support the Spresense board, and there is no existing configuration file for it. I have started developing custom code to address this. Currently:

  • The Plus Server successfully recognizes the serial port (COM) from Spresense.
  • 3D Slicer listens on port 18944.
  • However, the expected Transform nodes are not automatically created in Slicer.

Seeking Advice:

At this stage, I am considering two potential approaches to move forward:

  1. Custom Serial Communication via Plus Server: Develop a sketch for Spresense to output IMU data in a format suitable for Plus Server to process and relay as Transform nodes to Slicer.
  2. Utilize ROS2 with Slicer-ROS2 Integration: Since Spresense supports microROS, I could implement ROS2-based communication to handle Transform data exchange directly with Slicer.

Environment:

I am using a Windows 10 64-bit machine for development. While I primarily work directly on Windows, I also have access to WSL2 and Docker if they could be advantageous for this project.

I would greatly appreciate your insights or recommendations on which approach might be more suitable, or if there are other paths I should consider.

Thank you in advance for your time and guidance.

Best regards,
Masatoshi Oba

2

Ultrasound image reconstruction using IMU has been studied extensively. There ultrasound probes (such as ClariUS, ExactVU) have built-in IMU for this reason.

Accuracy of the IMU is typically not a cause for concern, commonly available sensors are already accurate enough for determining orientation. ROS is extremely difficult to set up on Windows and would be a huge overkill if you just want to read from a serial line. Plus has several examples of hoe to read sensor data frome serial port (including IMUs).

The main challenge is that the rotation axis is unknown and unstable. Many solutions have been proposed for this: careful spinning (for endocavity probes, e.g., for prostate, follicle counting), pivoting (for linear probes), and mechanical constraining (stepper for transperineal prostate biopsies, and various jigs and gimbals).

Note that the main competition of IMU based tracking is not the expensive ($2k-$15k) optical or electromagnetic tracking; but inexpensive camera based tracking. You may get 6DOF pose with acceptable accuracy from even a single camera, but you can improve rhe results by using more cameras and/or tracking 2D barcodes.