We are encountering an issue while tracking a scapula in Autoscoper during an arm elevation task.
For one participant, Autoscoper consistently crashes when saving frames within specific ranges of motion. The crash occurs during tracking of the shoulder/scapula and is reproducible across multiple trials. We have attempted to process the same participant data on different computers and obtained the same result each time.
I found an older issue that appears to describe a very similar problem, although it does not seem to have been resolved:
Thanks for bringing this to our attention. My experience with SAM is limited to tracking the femur and tibia, and I have not encountered this issue. However, I know that other groups have used SAM to successfully track the scapula. @Cesar Did you have this issue with any of your data?
Does this happen when processing data from other participants, or just this one?
I remember running into a similar issue while using the optimization feature, but it was fixed fairly quickly. I haven’t seen this specific problem before. I wonder if the graphics card could be causing the crash. Alonso, are you running SAM on a remote computer?
Thank you for taking the time to investigate this issue.
We are currently running SAM on a local workstation. We’ve tested multiple computers and several different subjects, and it appears that approximately half of the subjects we upload to SAM encounter this issue.
The workstations we have been using for these trials have the following specifications:
Dell Precision 5860
Intel Xeon W3-2423
NVIDIA RTX 3080 Ti
I’d be happy to share the files with your team if you’d like to investigate further. However, the image files are too large to upload through this platform. Please let me know the best way to transfer them, and I’ll send them over.
Hi @John_Holtgrewe, I wanted to provide a brief update on the Autoscoper-CUDA crashes we previously reported. We have now reproduced the issue with multiple scapular models from different patients and CT scans. The crash is also reproducible across several computers with different hardware configurations, including RTX 3080 and 3080 Ti GPUs, different Intel processors, and at least 64 GB of RAM. This suggests that the issue is not isolated to a particular dataset or workstation.
Further testing suggests that the crash may be related to the size or extent of the scapular model. If we crop the model to approximately half of the scapula, excluding the inferior portion, the crash no longer occurs. However, this is not a practical solution for our application because excluding the inferior scapula removes landmarks that we use for tracking and reduces the reliability and capability of the autotracking function.
We have also been investigating the source of the crash on our side and have found the following:
Windows Reliability Monitor initially reported exception code 0xc0000409 in ucrtbase.dll. We therefore configured Windows Error Reporting to capture a full crash dump and analyzed it with WinDbg. The dump indicates that, following the relevant button click in Autoscoper, the application throws an unhandled native C++ exception (0xe06d7363). The C++ runtime then calls terminate() and abort(), producing the final FAST_FAIL_FATAL_APP_EXIT (0xc0000409). The first identifiable Autoscoper frame is autoscoper-CUDA+0x660c7, but without the corresponding debug symbols we cannot resolve it to a specific function.
I can provide the complete WinDbg output, crash dump, and additional reproduction details if helpful.
I’m not familiar with AutoScoper, but it sounds from your troubleshooting like the scapula model may just have too many points for the processing occurring and the memory problems are leading to the crash (running out of memory is probably the most common reason for Slicer crashes). For models, the typical best solution is decimation. For models with a high density of points, you can often reduce the number of points very dramatically (>90%) without much loss of detail. So, rather than cropping, perhaps you can decimate the model to stay within memory limits. Interactively, you can do this using the SurfaceToolbox module, and it is also possible via python/VTK if it needs to be integrated into the workflow more deeply.
I apologize for just now getting back to you. The size of the volumes possibly causing the crash is interesting. We have only worked with relatively smaller volumes (distal end of the femur and proximal end of tibia). Could you share the following details about the volumes that you are trying to load in: dimensions of the tiff stack (including number of slices), voxel size, and file size?
Yes, any additional reproduction details you can provide will be helpful with documenting this issue.
We followed @mikebind advice and decimated both the STL models and the TIFF stacks, but unfortunately, the issue persists. Thank you as well, Mike, we’ll report back as soon as we resolve it. I am adding the details of the data that we are using, but I am happy to share the volumes and images if you’d like.
