Rotational Ultrasound

[shabopbop]

Hey all, I am pretty new to 3D slicer, so I apologize if someone has answered my question before. Essentially, I have taken a rotational ultrasound around a fixed point and I am trying to recreated a 3D model based on this alone. Does anyone know it is possible for 3D slicer can produce a model from something like this? My issue is that the program view my rotational scan as a horizontal scan (through one dimensional plane). I feel like if I could make 3D slicer fixate potentially on a pixel that I know the ultrasound rotates around and spit out a model as if it was reading the base of a cone… maybe it could work? Regardless, does anyone have any ideas/extensions/anything that could be of help? Also my ability to code things is pretty poor… ='(

Thank you in advance!

[lassoan (Andras Lasso)]

SlicerIGT extension’s Volume reconstruction module is developed for this purpose. See for example this demo:

Reconstruct 3D or 4D cardiac volumes from sparse 2D frame set

If your ultasound image sequence does not contain position and orientation information for each frame (which is usually the case) then you can use this script to add that information and reconstruct the volume.

The script assumes that the ultrasound image sequence is loaded as a 3D volume (each time point is a frame of the volume), so you may either need to load the ultrasound image with enabling Advanced option in DICOM module and choose the Scalar Volume reader; or modify the script sothat it uses a volume sequence as input (the script will be somewhat simpler, as you don’t need to create a new sequence, just modify volumes in the sequence).

[shabopbop]

Thank you for your help! I am still running into some issues. I don’t exactly know if it is because I am using a stack of png files instead of dicom, or if I am not utilizing the right combination of extensions, or if it is something else entirely. But, I am getting this error back when I try to implement the script you linked:

Mixed Content: The page at ‘@KitwareMedical/slicer-extensions-webapp’ was loaded over HTTPS, but requested an insecure image ‘http://www.slicer.org/slicerWiki/images/b/bc/BaselineFollowupSCANRegisteredCMFreg2.png’. This content should also be served over HTTPS.

Sometimes, it has caused slicer to crash entirely.

[lassoan (Andras Lasso)]

The Mixed Content:... messages are just warnings that the extension icons are stored on a non-https server. You can ignore them.

Starting from pngs should not be a problem at all. Then the script should be usable as is. Use the latest Slicer Stable Release and if you encounter any issues then let me know. Provide enough details that allows me to reproduce what you do.

[shabopbop]

I will try to outline what I have done so far so I can hopefully provide more insight to what I am doing. So, I upload my ultrasound .png’s into 3D Slicer with the ‘Add Data’ widget, and once loaded, I open the Python Console in the Slicer program and paste the script you posted in these responses. I have the extensions downloaded and enabled as well (SlicerIGSIO and SlicerIGT).

If I past the script alone without adding my images, it provides an attribute error (assuming because there is no image to obtain data from). And when I post the script after adding my images, my slicer program usually crashes. A couple times it gave an attribute error as well, to which I tried using DICOM files but my DICOM files aren’t compatible with Slicer at the moment.

I am sorry if this isn’t too helpful =( I can post images of what I am doing as well if that would be helpful in any way.

[lassoan (Andras Lasso)]

Does the application actually crash or just hangs (does not respond while performing computations)? If it actually crashes then most likely you run out of memory due to non-optimal reconstruction parameters.

You can run the code in smaller chunks to see what line does not succeed and if you get any error messages. If you get any error message then it is not enough to tell us that you got some error messages, but copy here the complete message. Maybe also attach a few screenshots (that allows us to see what kind of images you work with, what Slicer version you are using, what operating system, etc. and we might notice something unusual that could help us determine what needs to be done).

It could also help if you could share your data set (upload somewhere and post the link). You can use Crop volume module to cut off patient information from the images (if they are burned into the image corner).

[shabopbop]

Yeah, the application crashes fully where it is forcefully shut down and I have to reopen the application. I am using Slicer version 5.2.2 on windows 10. I am posting screenshots below, but my work flow is that we have a video recording of the US, I extracted all the frames from the video and made those frames into png files. Then I uploaded the files into slicer. Eventually, my images will be cropped in a way so that only the tissue will be present and none of the UI portions of the ultrasound program that was also recorded.

I ran the code in smaller chunks/ individual lines without any errors until I input line 79 (slicer.modules.volumereconstruction.logic().ReconstructVolumeFromSequence(volumeReconstructionNode)) where it crashes the program. I also notice that I am receiving notification that my scalar component is 0, which I do not know if this is something I need to worry about.

[lassoan (Andras Lasso)]

@shabopbop

I am receiving notification that my scalar component is 0

This can be the root cause of the error. Please provide the exact message.

Ankther possibility is that your run out of memory. This can happen because you do not crop your image and the full screen screenshot can be very large.

outputSpacingMm = imageSpacingMm * 10.0

You can also crop the image inside Slicer by modifying extent after setting it:

extent = inputFrameVolume.GetExtent()
extent[0] = 5
extent[1] = 510
extent[2] = 50
extent[3] = 400

It seems that you want to rotate around the horizontal axis, so you need to set the rotation axis to:

rotationAxis = [1, 0, 0]

[shabopbop]

So this is the output that I was referring to:

# Convert RGB/RGBA volume to grayscale volume
if inputFrameVolume.GetNumberOfScalarComponents() > 1:
    componentToExtract = 0
    print(f"Using scalar component {componentToExtract} of the image")
    extract = vtk.vtkImageExtractComponents()
    extract.SetInputData(inputFrameVolume)
    extract.SetComponents(componentToExtract)
    extract.Update()
    inputFrameVolume = extract.GetOutput()
Using scalar component 0 of the image

I will also try to crop the images before hand

Slicer finally didn’t crash on me! So here is the error from that line before:

>>> slicer.modules.volumereconstruction.logic().ReconstructVolumeFromSequence(volumeReconstructionNode)

[Qt] int __cdecl ctkVTKAbstractView::resumeRender(void) Cannot resume rendering, pause render count is already 0!

[lassoan (Andras Lasso)]

Using scalar component 0 of the image

This is not an error but a message that our script prints.

[Qt] int __cdecl ctkVTKAbstractView::resumeRender(void) Cannot resume rendering, pause render count is already 0!

This is a false alarm that is fixed in recent Slicer Preview Releases. You can ignore this message.

[shabopbop]

I started using the preview release and the script is working! I am getting closer to the image I am looking for and it is now rotation along the x axis. However I am stumped on how to align the axis along the center of the ultrasound image I want. It seems to be rotating along the bottom edge of the frame rather than the center (which makes a rainbow shape rather than a conical one).

I do want to thank you so much for the help thus far, there is absolutely no way I could have made it to this point without your help!

[lassoan (Andras Lasso)]

Nice progress.

@shabopbop

I am stumped on how to align the axis along the center of the ultrasound image

The center of rotation position is specified here:

centerOfRotationIJK = [(extent[0]+extent[1])/2.0, extent[2], 0]

extent[2] means the top of the image along the J axis. You can replace it by (extent[2]+extent[3])/2.0 if you want to rotate around the J axis center.

[shabopbop]

Hello,

Just wanted to give you an update/reaffirm that what you sent over worked for me! And it works pretty well too, so I am attaching an image of what I am making a model of. It is supposed to be they human eye with different layers out lined (retina, sclera, tumor tissue in this example below).

A couple things I did notice while working on it. After the image manipulation, the resolution of the resulting images did decrease a fair amount. I was just curious, but is this unavoidable?

Also, I saved these models as a mrml file. But when I try to open the model in Slicer, my program crashes soon after it tries to load the model. Do you have any potential reason why or if this can be fixed? I don’t think there are any error codes after the crash that I can find at least. I would post the file here so you had an example, but it seems like I can’t do so.

[pieper (Steve Pieper (Isomics, Inc.))]

These things (reduced resolution and crashing on reload) sound like bugs or maybe something related to your workflow that can be worked around. Either way, it would be great if you could find a way to reproduce them with sharable data so they can be tracked down and fixed.

[shabopbop]

I don’t know if I can reproduce the bugs in a way that can be shared. However, I may be able to send my mrml files through this avenue that I am trying. I will see if this works.

If not, I can share my png files and outline the steps I do to produce the 3d model and see if anyone in the community can find the bugs/ workflow issues.

tumor.mrml
tumor.seq.mrb

[shabopbop]

I was able to load a different file onto 3d slicer, I didn’t have to use the mrml/seq.mrb files like I initially thought! (used the nrrd file instead). The resolution decreasing issue still stands but I can work with what I have now. Thanks to all who have helped!

[lassoan (Andras Lasso)]

@shabopbop

After the image manipulation, the resolution of the resulting images did decrease a fair amount

You can choose any resolution that is sufficient for your purpose and your computer can still handle, by adjusting outputSpacingMm. If you decrease the spacing a lot then you may need to enable hole filling (because the slices you paste become thinner).

[nunno]

Hello Andras.

This is Josh. I am currently working on the same project as shabopbop, and we are encountering a issue that we are not sure how to resolve.

There are two parts.

1. When the flipper.py converts images from ijk to xyz dimension, certain region shows a misaligned center of axis rotation.

this shift is a result of flipping along a false axis (yellow) and it needs to be fixed to flip along the true axis (red).

I have identified that the value “2.12” from below determines the location of the axis.

#Set up frame geometry and rotation
#centerOfRotationIJK = [(extent[0]+extent[1])/2.0, extent[2], 0]
centerOfRotationIJK = [(newExtent[0]+newExtent[1])/2.0, newExtent[2]+newExtent[3]/2.12, 0]
rotationAxis = [1, 0, 0]
rotationDegreesPerFrame = 180.0/numberOfFrames

Here’s my question.
Is it possible to automate estimation for this value for each image before executing the image flip to fix the misalignment?

2. When annotating the segments, I noticed that anatomical structures before the image flip is much clearer compared to post flip. Is it possible to edit the flip code such that we can annotate the structure first on the ijk images, then flip the annotation along with the pixels?

[nunno]

Hello.

This is a follow up to the resolved thread “Rotational Ultrasound”

Rotational Ultrasound

Hello Andras. This is Josh. I am currently working on the same project as shabopbop, and we are encountering a issue that we are not sure how to resolve. There are two parts. When the flipper.py converts images from ijk to xyz dimension, certain region shows a misaligned center of axis rotation. [Screenshot 2023-11-10 at 2.19.35 AM] this shift is a result of flipping along a false axis (yellow) and it needs to be fixed to flip along the true axis (red). [Screenshot 2023-11-10 at 2.27.18…

Can someone help me with this?

Thank you.

Josh

[lassoan (Andras Lasso)]

@nunno

Is it possible to automate estimation for this value for each image before executing the image flip to fix the misalignment?

It seems that it could be quite easily annotated, for example by computing the center of gravity of the non-zero voxels in the purple region.

@nunno

When annotating the segments, I noticed that anatomical structures before the image flip is much clearer compared to post flip. Is it possible to edit the flip code such that we can annotate the structure first on the ijk images, then flip the annotation along with the pixels?

Yes, you can segment the 2D slices (before 3D reconstruction), export the segmentation to a labelmap image, and then reconstruct a 3D volume from this labelmap image the same way as you do from the ultrasound image.