Amira like segmentation - add to segment functionality

Support Feature requests
1

Dear 3D Slicer community,
a colleague of mine, with whom I often collaborate is doing manual segmentation of muscles in µCT scans of small arthropods. Due to the small size, automatic segmentation is often not feasible. So, they will end up doing the segmentation by hand. Muscles often have very complex topologies and the segmentation work is also part of the learning and understanding process of the anatomy of the animal.

They worked extensively with Amira before, which has a feature that allows for a quite efficient workflow:
The segmentation is done in something like a sandbox mode. They can segment some slices, and apply a fill between slices function, judge whether they like the result and then add the interpolated slices to a named segment that is currently selected using a shortcut. This way they can segment individual muscle strands, one after another and add them to a segment.

The desired workflow is as follows.

  1. select a segment in the segment editor
  2. turn on the sandbox mode (what is painted will not automatically part of the selected segment)
  3. manually draw in several slices
  4. apply a fill between slices function
  5. check the interpolation (at best in 3D), proceed with step 6 or undo step 4 or clear the “sandbox” content
  6. add the interpolated “sandbox” segmentation to the previously selected segment
  7. optionally repeat with step 3

I know that it is possible to achieve this in 3D Slicer by creating new segments and merging them using the boolean operators. However, this is not very efficient. They often do manual segmentations for several hours and reported that they were more than twice as fast if they used this workflow in Amira compared to using 3D Slicer.

Is there a way to implement such “sandbox mode - add to segment” functionality in 3D Slicer? I know that this might be a niche functionality, but on the other hand, the lack of such functionality is the only reason why my colleague is still drawn to using Amira over open source software.

For those who are wondering about how the results of such models usually look like, here is a figure of an open access article. This example is definitively on the lower end of the spectrum regarding the complexity and amount of reconstructed muscles.

Screenshot from 2022-11-02 12-12-53
Screenshot from 2022-11-02 12-12-531267×1003 111 KB

Is it possible to achieve this functionality using a python script? I mean the core components are there, they are just not efficiently accessible? I would of course offer my help if it is possible to include this functionality into the 3D Slicer Segment Editor or as an extension for 3D Slicer. However, I have to point out that I am not a software engineer and what I can offer is only a basic proficiency in Python, limited to simple scripts and data analysis.

Thank you for looking into this! I admire your work and always try to promote others to give 3D Slicer a try.

2

Mario,

Can you generate a short video (screencast) demonstrating the workflow you described above in Amira? A video would show the functionality better. If the functionality is something that is useful for the wider Slicer community, we can help with the implementation.

-Andinet

1 Like
3

+1 to the suggestion from @Andinet_Enquobahrie

Writing specialized segmentation tools and workflows is definitely something we want to support and hopefully all the hooks are available to implement what you need.

4

Thanks for the very positive replies! I uploaded a video screen cast to YouTube (unlisted):

Sorry, that it took so long, but you know how it is when you cannot simply run the software on your own computer…

I think that this kind of workflow could also be of interest for users in other subfields in the biomedical spectrum. Good scans are often expensive and difficult to make and the quality of the scans at high magnifications is often not good enough to achieve desirable results by using automated methods. Further, the segmentation is often not there to separate objects of different materials but in the case of our example objects of the same material (muscles) in a complex environment. I can imagine similar situations for example with neurons, blood vessels, etc. There one of the main objectives while segmenting is also understanding the topology of the objects. Also, I think the developers of Amira would not have put this as their default (or only?) method for manual segmentation, despite being less intuitive for new users, if it were not of interest for a wider spectrum of users.

I hope this helps to deliver a better understanding why this kind of workflow is so important to us and maybe others as well.

5

What you shown is already possible with Slicer, it might require couple more clicks than the Amira interface.

Each time you are segmenting a new muscle section:

  1. Set the modify other segments to “Allow Overlap”
  2. Create a new segment for the new section to be segmented.
  3. use the Draw tool to paint the few sections you want to interpolate in between. (optionally enable editable intensity range to have more control over voxel selection)
  4. After 4-5 such outlines, use the Fill between slices tool (again optionally with intensity range enabled).
  5. Repeat the steps 2-4 for each segment you are playing with.

Once done, use the logical operators with Add option to combine the segments into a single segment.

6

Here is a video of this:

If you segment multiple structures then you need to simply paint all those in the selected few slices.

I would also add that “Fill between slices” is a last resort tool, for cases when there is no contrast between the structure you want to segment and surrounding tissues (e.g., the user draws contours based on where he knows the boundary is, even though it is not really visible). For the most common case, when the segmented structure has slightly different intensity compared to surrounding, you can segment faster and more accurately using “Grow from seeds” effect.

1 Like
7

Dear responders,
I am well aware of the possibility to do it this way in 3D Slicer. My idea was to make this more efficient. In Amira this kind of work requires very little effort because you only need one shortcut to fill between slices and one to add your segmented slices to an already pre-selected segment.

My question is whether it is possible to create an equally efficient workflow in 3D Slicer. As you pointed out, the core functions are already present. Maybe this could be done by creating an extension that can be installed by users who desire this kind of segmentation workflow over the already existing one. “SegmentEditorExtraEffects” does modify the segmentation editor interface, which makes me think that this might be possible.

8

It requires 4 clicks in Slicer - 2 clicks to initialize, 2 clicks to apply (one click to go to Fill between slices effect, another o click Initialize or Update). It is also fully dynamic - it automatically updates segmentation in 3D as soon as you add any new segmented slices or you modify a slice. Amira’s tool does not seem to support this auto-update mode, only manual update, so to me Amira’s tool seems to be less convenient than Slicer’s.

Is the problem that for a new user it is hard to discover where to do those 2x2 clicks have to be made? Or you would prefer a keyboard shortcut instead of the 2 clicks? Or you want to disable auto-update by default and have a keyboard shortcut to manually update?

9

What I would like, and what should probably also be possible with simply having a script rather than an extension, is to have shortcuts that spare me that extra clicks.

I could imagine something like this:

Shortcut 1:

  • retrieve the name of the currently selected segment and store it in a variable named parent
  • creates a new segment with a name that is for sure not in use like “temp#123”
  • selects the newly created segment “temp#123”
  • turn on the allow-overlap mode

Shortcut 2:

  • fill between slices (maybe there is already a shortcut that I am currently not aware of)

Shortcut 3:

  • check whether the currently selected segment is named “temp#123” (safety mechanism to avoid merging other segments with a shortcut)
  • use the logic operator “Add” to combine the segment “temp#123” and the segment with the name that is stored in parent
  • select the segment with the name that is stored in parent

I really like the the auto updated preview that 3D Slicer has (as I usually don’t use Amira, I am not sure if it has this function, it could be simply not activated), so I have no intention to disable it, unless I run into hardware limitations.

I hope my intention for this question/suggestion/help-request got a bit clearer now. If somebody could help me write a script for creating and enabling such shortcuts this would be awesome.

10

The Fill between slices effect already does almost exactly this. Instead of trying to manage everything in a single segmentation (and come up with a segment name that is not used and switch to allow overlap, to prevent unwanted modifications), Slicer simply creates a new “Preview” segmentation (you can see that if you switch to Data module while “Fill between slices” or “Grow from seeds” or “Watershed” effect is active), so there is no interference between the manually created inputs and continuously automatically updated segmentation results.

  • Shortcut 1 = Click “Initialize” in “Fill between slices”
  • Shortcut 2 = automatic, no need to click anything (if you uncheck Auto-update then you need to click “Update” in “Fill between slices”)
  • Shortcut 3 = Click “Apply” in “Fill between slices”

The only inconvenience that I see of the current workflow is that you need to switch to the “Fill between slices” module to initialize/update/apply. You don’t need to use your mouse for this, as you can just hit the “space” key to switch between the last two effects, but this shortcut is not easily discoverable and still requires some mental effort.

Would it fulfill your needs if we had the “Apply” and “Cancel” buttons (and if auto-update is disabled then an “Update” button; maybe also a “Restart” button) would be available in the Segment Editor module when an auto-update effect was active?

Something like this:

image
image1083×1021 56.1 KB

Or maybe even better:

image
image1086×1076 54.4 KB

11

There is one problem with the solution you showed:

If the preview/sandbox segment is of the same segment as what has been previosly segmented, the fill between slices algorithm tries to incorporate everything instead of the not yet interpolated part.

Here is a screenshot of what I mean.

Screenshot from 2022-11-09 11-47-12
Screenshot from 2022-11-09 11-47-121920×1080 99.4 KB

I segmented one structure, filled between slices and began segmenting a new structure (of the same muscle that I intended to join later. As you can see the fill between slices algorithm made additions to the structure that I did not want it to make changes to (e.g. it prolonged the left muscle strand in the 3D view). Also, note the shape of the interpolated segment in the red view - it has sharp corners. This would not happen if I was working on a different segment.

I hope it gets clear what I mean.

12

It is a different segment then. You can merge segments later if needed.

Fill between slices interpolates between all visible slices, so if there are segments that you want to exclude from interpolation then you can hide them or temporarily move them to a different segmentation (you can drag-and-drop segments between segmentations in Data module).

It may be simpler not to exclude segments but keep “Fill between slices” active until you are done with all (or at least a group) of your segments. The only drawback of this approach may be that you need to always segment complete slices. Therefore, if you want to fix up the automatic filling result of only one segment in a slice then you need to paint all the segments that are visible in that slice.