I am working with microCT tiff stacks and currently it is 75GB when read into the software at full resolution. Following the general guideline of using a hardware with RAM 6x-10x that of the file is not attainable especially with the current global memory shortage. I have been fortunate enough to get my hands on an HPC with 800GB of RAM and thus am able to do basic segmentation using the thresholding tool. However, it still takes a quite a while to convert the segments into 3D so that I can verify the segmentation.
Things I’ve tried:
Loading at half-resolution and remove the extra structures using ROI, then load the ROI in full-resolution. This reduces my file size, but not by much.
If I don’t use any semi-automated segmenting tools, could I subset the object using ROI into sections to do my segmentation and then combine the segmentation from each section together afterwards? This way I will only have to work with 1/4 of the data at a time.
What other strategies or recommendations are there for segmenting with very large files?
That’s interesting, so you can’t really trim much of the data then? When I use this methods, I often get my dataset less than 50% of the original size.
If you need to work on the full dataset, yes, that’s the best solution. Maybe do more than 4. The important part is to make note of the threshold range you are using so that it remains consistent across the data.
But in the end, converting such segmentations into 3D models (if you are doing that) will be slow regardless. If you don’t need the 3D models, you might consider using Colorize Volume to color render the volume with your segmentations. However, at this point there are no commercial GPU out there that will fit your data, including those $40K ones (when you use RGBA rendering on full resolution 75GB dataset, you need a GPU with 300GB of RAM, which currently doesn’t exist).
If you tell us what your dataset, what you are segmenting, and what your end goal is we might be able to suggest other things. But at this point, I think you already figured out your options.
All of that can be done in Slicer with ImageStacks and CropVolume. Fiji doesn’t provide anything more, but causes more trouble (such as not recording the crop offset or correct image scale after resizing). I would not recommend using Fiji for this purpose.
Been a while since Ive needed to do it but FijiJ made very large microCT datasets workable for me, in the situations where I couldn’t perform any manipulation in Slicer. I seem to remember their virtual stack function being very useful.
The first part is the exactly the same functionality of ImageStacks, which provides many additional features not found in the Fiji, more convenient, and more importantly allows you to reimport the data correctly multiple times, without losing spatial relationship of the any subvolumes you might have created. You cannot do any of those in Fiji (not easily).
For workflow 2 if you want to load a dataset that’s beyond your computers memory size, you can increase the virtual memory of your own computer (which will be equivalent to using the virtual stack option) there.
@muratmaga and @arumiat, thank you both for such great and detailed suggestions. I have made some progress in my workflow by incorporating suggestions you made here and other posts in combination with my trial-and-error.
I’m going to recap the discussion here for future readers.
Objective / Use case
Segment medical device microCT scans that contain non-organic (plastic, metal, etc) and organic (blood clots) components/substances and quantify the statistics (esp. volume) of each segmented components.
Challenge
The image stacks are large (approximately 70GB-80GB) and the segments are non-continuous (scattered like stars in the night sky). Because of how large the model size is, the computer takes a long time to do a simple segment by threshold and “Show 3D” feature will crash the application even when running with ~800GB of RAM.
Current solution(s)
Create pseudo-ROI using “Draw tube” effect and “Scissors” effect (add/remove shapes) to create the final segment for pseudo-ROI.
Use this pseudo-ROI as a mask by selecting it in the “Editable area” dropdown menu of the “Masking” tab when any of the “Segment Editor” effects is selected.
