Hello, I’m thinking of doing an extension that incorporates google’s tilt brush into Slicer. Has anyone done anything similar? Or thought of it?
Thanks!
Slicer can already display the scene and and interact with the nodes in virtual reality. It would be probably much easier to enable the Segment Editor paint tool work in virtual reality than integrating a completely new virtual reality framework.
That said, you probably don’t want to segment by free painting in 3D. Painting in 2D is already tedious and slow, and in 3D it would be much worse. However, a smoothing brush, or marker brush (to not paint new objects but mark suspicious regions for cleanup), or scissors could work well in virtual reality.
Yes we have been planning to integrate Segment Editor into SlicerVR for at least two years now (wow). The biggest obstacle currently is using SlicerVR with VTK9, which is far from trivial due to the major change in the way VTK9 handles external modules. Once that is solved, we need to do some work to be able to use a virtual user interface panel in VR, then integrating Segment Editor will be relatively easy (there are parts already done, but need to hammer out a few details).
Personally I am quite optimistic that we’ll have it done in a few months. It depends on two important things, however:
- Sorting out the build issue against VTK9. Maybe @jcfr can give an update as we have been in contact about this regularly, but not recently.
- The decision on a grant proposal in which the user interface panel is involved. We are supposed to hear about it “today, tomorrow, or early next week”.
I agree with @lassoan in that using a VR paintbrush is probably not a good idea to do segmentation from scratch because it is very hard to see the contours/borders even with a high quality volume rendering. Also agree that smoothing brush and touch-ups will be great. However, my suspicion is that we will only be able to assess the real advantages and best uses once we are actually able to try it. It will be quite a fun research and a good paper!
1 Like
I was thinking more of an annotation tool paint brush. So that you can paint inside the VR module for slicer. More of an artistic tool than a medical one. Any thoughts?
Slicer’s main focus is medical applications. If you are just looking for a virtual reality platform for artistic/entertainment purposes then probably gaming engines, such as Unity or Unreal will work better.
Hey! Thanks for the reply. I understand slicers main focus. My boss is wondering if this is possible. I am willing to put in the time to create an extension that does this. I am simply wondering if it is possible and hoping for a little direction.
It is not easy to give directions if the overall goal is not clear, but probably the big decision you need to make early on is if you want to represent content as image volume or polygonal mesh. Boolean operations (add/subtract) are trivial in an image volume but volume rendering is computationally more demanding and less rendering effects are available (lighting, physically-based rendering, special render passes, etc.).
I would probably opt for using polygonal mesh then. I suppose a simple explanation would be this, I want to:
- Be able to create a paint like substance at the head of one of the controllers
- Have those substances be modifiable using in VR UI. I believe you answered a question about getting 2D actors to work on a given plane for better rendering.
Does this help with the goal? Tiltbrush has their code available here: GitHub - googlevr/tilt-brush
I am wondering how useful this code would be to use in slicer? Using it as a basis to create something similar in slicer.
It is still nto clear what the goal is. Why do you want to do artistic drawing in Slicer? What you would like to achieve as an end goal? What problem you would like to solve?
Since Slicer uses VTK as rendering engine, which has very little in common with Unity, probably it would be hard to reuse anything directly, but you could learn a lot from studying the tilt-brush source code.
You’re absolutely right, we are wanting to be able to use 3D DICOM data with tiltbrush, but I might just be able to use tilbrush and modify it to load data into it as a sort of canvas. Thank you.
You can certainly load DICOM data into Slicer, segment it, and export it as a surface mesh, which TiltBrush should be able to load.
1 Like