I’m using slicer to work with very large files. After disk activity stops, there is a (surprisingly) long delay before the volume shows up in viewers.
Disk time to load is a few minutes, the delay after that has been as long as 40 minutes.
One cpu core is busy during this long delay time.
I think it is the min/max calculation, but i’m not currently in a position to verify that.
Is there a way to skip the min/max calc on load, or are there bare-bones programmatic ways to load which don’t do any processing?
This was tried with and without compression. Compression is also a serious detriment in loading these, when used with a single data file this took hours to load, my memory is 7, but I didnt find my record of it. When using multiple compressed files, (one per slice) loading time was very long, but could be accomplished in one working day. I didnt time that, i started loading in the morning and it was ready at lunch time.
type: unsigned short
sizes: 6263 10186 2621
space directions: (0.0018,0,0) (0,0.0018,0) (0,0,0.0040)
kinds: domain domain domain
space origin:(-5.36155006705028, -9.68239733742919, -4.23750168693362)
data file: big.dat
I have noticed a delay between the end of the loading my large NRRD files (not compressed) and the appearance of slice views, which can be a minute or two (never really timed either). I always assumed those are overheads associated with generating MRML nodes etc. But I never tested with such large dataset, my large files are 10-20GB range. Yours is almost 400GB.
I’ve tested loading of a 6.4GB uncompressed NRRD file and only a few seconds passed after file loading finished:
While cold loading of the file (after computer restarted), resource monitor showed 100MB/sec disk activity for about 64 seconds. A few seconds after disk activity went down to zero, the image appeared.
While warm loading the file (loading it again right after it was loaded), resource monitor did not show any activity and image appeared in 6 seconds.
A few things that would be nice if you could do to diagnose what’s happening:
Disk activity light is not reliable source of information. If you use a resource monitor that shows the actual disk transfer then you will know if data is actually read. Note that if you load a recently loaded file then it may be retrieved from memory instead of re-reading from disk, so you cannot blindly trust resource monitors either.
Your image is stored as big endian, so each value has to be byte-swapped after loading. It should not cause significant delay but it may worth a try to save the image (as it is saved as little endian) and see if loading this image is any faster.
You may try to instantiate a simple NRRD reader and see if it makes loading any faster:
You can also try VerySleepy tool, which provides you function list and call stacks of methods that the application spends the most time with. For release-mode builds, without debug information, the data is not fully reliable but often gives useful hints about what the application does.
Instead of just watching the open file listing, use a tool that gives you real-time information about the amount of data transfer (such as Resource Monitor or Performance Monitor). You can then check if the disk activity makes sense (time x bandwidth should equal file size).