How can I overlap live camera feed (2-D image) with the CT scan model for an Augmented Reality feel?

Support
1

We want to overlay the live camera feed with the CT scan model. The method to calculate the transformation matrix for accurate positioning has been worked out. However, we are struggling with the integration of both 2-D camera feed image and CT scan volume in the same window. Any help will be hugely appreciated.

2

You can acquire camera image using Plus toolkit, display the live image in 3D via a slice view, and set up renderer camera parameters to match the optical camera parameters. You can access the camera as described here.

3

Sorry it still seems foreign to me, but to step back a little,

  1. What config is needed on Plus toolkit if e.g. to broadcast the webcam view?
    I tried “PlusServer: Video for Windows video capture device” on PlusApp-2.8.0, it showed “Connection successful!”, but the red light constantly spit out error messages of

SERVER> AddFrameToBuffer: video compression mode MJPG(0x47504a4d): can’t grab| in :\D\PSNP64b\PlusLib\src\PlusDataCollection\VideoForWindows\vtkPlusWin32VideoSource2.cxx(519)

  1. How to display the live image in 3D via a slice view?

Thank you!!

4

It seems that your webcam only supports mjpg encoding, while Plus only supports uncompressed encodings only. Video for Windows interface is about 30 years old. It is a miracle that Windows still supports it at all.

I would recommend to connect to your webcam using Plus toolkit’s Microsoft Media Foundation or OpenCV imaging interfaces.

5

Thanks @lassoan!
I did get some local help and managed to modify the config file of “PlusServer:Video for Windows video capture device” based on “PlusServer: Optical marker tracker using MMF video” (since it is webcam based), basically to set up device properly like below

 <Device
  Id="VideoDevice" Type="MmfVideo"
  FrameSize="640 480"
  VideoFormat="YUY2"
  CaptureDeviceId="0" 
  <DataSources>
    <DataSource Type="Video" Id="Video" PortUsImageOrientation="MF" ImageType="RGB_COLOR" />
  </DataSources>
  <OutputChannels>
    <OutputChannel Id="VideoStream" VideoDataSourceId="Video" />

Once the server is launched on Plus, adding an IGTLConnector using OpenIGTLink module on Slicer will do the job of displaying the cam video in a view window.

This is probably one of many ways to make it work.

1 Like