If I may, I would like to ask the group if there is significant interest in future extensions to Slicer that would bridge the gap to photorealistic rendering?
In the ParaView world we’ve been doing work with NVidia’s IndeX and OptiX (volume and surface rendering respectively) and Intel’s OSPRay (both volume and surface) that give ParaView the ability to render photorealistic images. I would like to make the case for doing the same thing in Slicer.
So, do you think that photorealism be a useful tool medical visualization? If it was available what would you use it for and what controls and features would you want to see? Also, if Slicer isn’t the right platform, could you suggest others?
I will be interested in the seeing these features in the Slicer, especially if they are compatible with the existing volume rendering interface. I think Slicer is indeed the right platform for it. But I think a good photorealistic renderer also can benefit from some sort of a keyframe rendering to enable complex animations, which we currently seem to lack.
I would be very interested in this as well, as would colleagues here who use Slicer. I can’t speak for medical imaging, but would be very useful in invertebrate and vertebrate zoology and paleontology. If I have specific lighting or surfacing requirements, I usually wind up exporting models to Blender.
A key-framing animation setup would be a very welcome addition, but I’m also aware that it’s likely a major undertaking. As with photorealistic rendering, any time I need to do any animations, I likewise export models to Blender. Just for my uses, I would actually put some kind of key-framing animation setup for camera movement at a higher priority than photorealistic rendering. I’ve tried to use the endoscopy module before but I just can’t get it to do what I’d like it to do. (Problem between chair and keyboard, probably.)
I have no experience with OSPRay, so I’m not sure what it can do. Things I would like to see in a photorealistic rendering setup would include the ability to position multiple lights, different light types (point, directional, area), light color, enable/disable casting shadows, the ability to set some rudimentary surface shading parameters (a whole node-based shading system would be awesome but total overkill), etc. I do a lot of transparency shading to show internal structures, so rudimentary controls like that would be very useful.
I used to use ParaView for animations but have not checked it out in a while - if these are the directions they’re going in lately, I will definitely have to take a look. Exporting from Slicer to ParaView is super-easy.
I agree with @hherhold, if one is to be prioritized over one I think keyframe editor is a bigger issue for people who wants to do animations of their organism in Slicer.
In medical imaging, photorealistic volume rendering is advertised as “cinematic rendering”. There are mixed results in using it for diagnostics (complex lighting may improve depth perception and even visibility of some structures in certain circumstances, but it may also make it harder to see some details). However, more realistic rendering would be very welcome in medical education and in communication between patients and physicians. We (and I guess many other) users would be very interested in trying these techniques.
Same interface as regular volume rendering (presets, transfer function editor, quality/speed controls, etc.), shadows enable/disable, and light editing (probably lightkit presets, maybe tuning of position of key light and intensity of filling lights exposed on the GUI). We would also like to use multi-volume rendering.
For medical image visualization (human and animal) Slicer is probably the best platform for many reasons. The infrastructure is already in place, only little work is needed to expose new volume rendering features on the GUI. The main reason we have not considered adding OSPray support to Slicer yet is because of the long rendering time for good-quality images (see details below) - but if there is a chance for fast options then we should work together to make it available for users.
Current experience with OSPray:
With default settings in ParaView, speed is borderline tolerable but there are significant rendering artifacts compared to baseline GPU raycasting.
Increasing samples per pixel value removes many artifacts but surfaces still look quite messy at high-gradient areas (maybe sampling distance is not small enough? but I did not find how to set that in ParaView) and rendering is not interactive anymore (rendering may take 5-20 seconds).
OSPray (default settings: samples per pixel = 15):
I figure I’d bump this thread with some good looking volume renders. One from syngo-via, the other from Inviwo (I think). You will notice the shadows are not harsh, and over all they look a lot more realistic than those paraview examples. Another nice feature would be real time basic lighting for meshes also like when using segment editor.
