Hi everyone,
I’m using MONAI Label Radiology with 3D Slicer to train a custom 3D medical image segmentation model.
During dataset loading, the process stopped before training started with the following error:
torch.cuda.OutOfMemoryError:
CUDA out of memory.
Tried to allocate 1.45 GiB.
RuntimeError:
applying transform Spacingd
I’m currently using SmartCacheDataset with a training batch size of 1.
I suspect that preprocessing is being performed on the GPU, causing excessive VRAM consumption.
My proposed solution would be:
-
Move preprocessing operations (particularly
Spacingd) to the CPU by changingEnsureTyped(..., device=context.device)todevice="cpu"insegmentation.py. -
Keep the actual model training on the GPU.
-
Initially test with the standard
Datasetimplementation to verify that training can run successfully. -
Consider switching to
PersistentDatasetfor disk-based caching instead ofSmartCacheDataset. -
Optimize caching, batch size, and data-loading settings to support a progressively growing dataset.
Would this be a correct and recommended approach?
I would also appreciate clarification on two points:
-
Are any additional code modifications required to ensure that preprocessing runs on the CPU while training remains on the GPU, without affecting image-label alignment or segmentation accuracy?
-
How can I ensure that all eligible annotated images are eventually included in training, even when using a limited caching strategy?
My goal is to establish a stable and scalable training pipeline capable of handling an increasing number of annotated 3D images without exhausting GPU memory.
Any suggestions or example code would be greatly appreciated.
Thank you!