For a project was looking a way to add anatomic structures details (i.e. tissue histology) directly starting from a CT/MR, to be used for education, analysis and simulation using realtime 3d rendering.
In 3d Slicer you can segment your data to extract the different elements, and export them in various formats that can be used for realtime rendering.
I have seen beautiful CR examples such as this one
that I suspect it’s “just” using the traditional PBR pipeline using surface materials.
Meaning if you cut the model you didn’t find any detail inside.
So far to add volumetric details you can sculpt them as a volume using tools such as Freeform and next provide the volume itself some color. It’s a long process.
A better way will be using some color 3D brushes to paint directly in 3d and export 3D volumetric images.
But searching on I didn’t find any approach like that, for sure might be not so efficient in term of data sizes (volume model + volume textures).
Wanted to ask if someone has better information than me on this topic, to understand if it’s just a bad night idea, or there is some value in it.
Segmentation in Slicer is always 3D and volumetric. So instead of exporting your segmentation as models, you can use the MaskVolume feature to export them separate volumes, and then use the volume rendering to assign different transfer functions to each of these volumes. So that’s already doable easily.
Where it gets difficult is the multi-volume rendering in Slicer is not fully implemented (as far as I know). Depending on the view/angle, you will have rendering artifacts and such, but I think the more users request and demand it, more attention it will get.
From the cloud-like appearance of minor vessels you can tell that the image above is rendered using volume raycasting or path tracing. There is no surface extraction or segmentation and the same material is used for the entire volume: there is just a color map that makes lower intensity voxels darker red and higher intensity voxels brighter yellow; bones and vessels appear exactly the same, with the same red haze at the boundary due to partial volume effect.
I think the rendering is impressive partly because the image resolution is higher than average and because of the material/lighting model maybe a slightly more sophisticated than just a basic ambient/diffuse/specular reflective model: there seems to be some soft shadows created by screen space ambient occlusion or path tracing.
This is what you can get with a simple volume raycasting in Slicer on slightly lower resolution images:
Volume rendering already displays a “filled” 3D volumetric texture. If you can see this in all the semi-transparent regions or if you cut into it using the crop region.
@lassoan clear enough. In this way you can anyway color the volume only based on a single property the density.
My idea was if there is any standard where I will be able to load painted 3d images, with much more details such as from the longitudinal and cross section of muscle
You can volume render an RGB or RGBA volume as well. In this case the provided color and opacity values are used (they are not computed from a single scalar with transfer functions).
Painting/sculpting segmentations in virtual reality sound like extremely tedious to me (even just holding out your arms for a few minutes takes quite a bit of effort). However, quick review (with grab to move, pinch to zoom) and touch-up (smoothing, filling holes, removing excess) may be useful and we plan to add these features.
