MONAI Label, CUDA Out of Memory during preprocessing: CPU preprocessing and scalable training

Hi everyone,

I’m using MONAI Label Radiology with 3D Slicer to train a custom 3D medical image segmentation model.

During dataset loading, the process stopped before training started with the following error:

torch.cuda.OutOfMemoryError:
CUDA out of memory.
Tried to allocate 1.45 GiB.

RuntimeError:
applying transform Spacingd

I’m currently using SmartCacheDataset with a training batch size of 1.

I suspect that preprocessing is being performed on the GPU, causing excessive VRAM consumption.

My proposed solution would be:

  1. Move preprocessing operations (particularly Spacingd) to the CPU by changing EnsureTyped(..., device=context.device) to device="cpu" in segmentation.py.

  2. Keep the actual model training on the GPU.

  3. Initially test with the standard Dataset implementation to verify that training can run successfully.

  4. Consider switching to PersistentDataset for disk-based caching instead of SmartCacheDataset.

  5. Optimize caching, batch size, and data-loading settings to support a progressively growing dataset.

Would this be a correct and recommended approach?

I would also appreciate clarification on two points:

  • Are any additional code modifications required to ensure that preprocessing runs on the CPU while training remains on the GPU, without affecting image-label alignment or segmentation accuracy?

  • How can I ensure that all eligible annotated images are eventually included in training, even when using a limited caching strategy?

My goal is to establish a stable and scalable training pipeline capable of handling an increasing number of annotated 3D images without exhausting GPU memory.

Any suggestions or example code would be greatly appreciated.

Thank you!

Memory consumptions in the models I am familiar with are affected by the model architecture choices (number of convolution layers, image dimensions, etc), not by sample size. If your training model works for 20 samples, it should work for 2000 samples as well.

That CUDA OOM reported number is not very large, it is only 1.5GB. When I run into OOMs that usually an order of magnitude larger than that. What GPU and model are you using?

Going back between CPU and GPU and loading/offloading data usually kills your training speed. It might be better if you run the preprocessing at once, save the output and have model run directly on the processed data, as opposed to do preprocessing on the fly.

If you put you architecture and code to an LLM, these are kinds of things that they are very good at optimizing or finding solution to.