VTK (Slicer’s rendering library) added support for physically based rendering (PBR) in recent versions. See some examples in these blog posts: PBR basics, image-based lighting, coats. We exposed some of these features in the latest Slicer Preview Release.
PBR interpolation mode in Models module
A new PBR mode is added to Models module / Display / 3D Display / Advanced / Interpolation listbox. In PBR mode, the material properties are described with a diffuse reflection factor (overall brightness), metallic, and roughness factors.
Experimental image-based lighting setup feature was added to Lights module (in Sandbox extension). This can be used to specify an image that provides lighting to PBR models and this image can be seen reflected from smooth metallic surfaces.
Currently a single image of a hospital room can be chosen (by clicking the corresponding button), but users can experiment with adding different images by editing the module’s Python script.
This feature was added to the Lights module some time ago, but it is worth mentioning again here, because PBR models often don’t have darkening at the edges and this feature can help making edges more visible.
Example rendering of models with PBR interpolation, non-metallic objects with image-based lighting and ambient shadows:
I tried installing this morning for Windows preview release (2022-02-01) but the Sandbox extension could not be found. I did try the PBR interpolation and it looks great, but I’d like to see the image based lighting with it.
The Mac build doesn’t have this yet - seems to be a couple of days old?
Thanks for testing. A file was missing from SlicerSandbox that prevented the packaging. I’ve added the file now. If you don’t want to wait for tomorrow’s release, you can download the extension from here and add the Lights folder to the Modules → Additional module paths in your application settings.
I’m not sure what’s going on with the macOS build. @Sam_Horvath can you have a look?
In the models module, after I change the interpolation to PBR, and then I click on one of the presets, the settings change the way it looks under phong/flat/goraud shading (i.e., there are ambient/specular sliders etc). But when I try touch one of them, it reverts the way it looks in PBR (three sliders only).
I couldn’t try with the Lights module due to lack of Sandbox extension, but have a feeling that for PBR to work better, we might have to have the Lights module in the core perhaps (i.e., graduate out of Sandbox?)
Would it be possible to enable this in Segment Editor as well (maybe it is possible, I didn’t look to deep really).
If you click on it then the interpolation mode will be set to the mode that is specified in the preset. Currently, all 3 presets uses Gouraud interpolation, but unfortunately, there is a display bug: after clicking the preset, the interpolation checkbox still shows “PBR”.
When I added this feature I missed adding a file, so for one day the Sandbox extension was not built.
It would be awesome if someone would help with adding these features to the Slicer core. The Lights module cannot be added to the Slicer core as is, because settings are not preserved in the scene. We would need to decide where to store these settings (store lights in the view node, in a “lightkit” node, or add a separate node for each light), and implement standard GUI (make these view parameters available near existing view parameters), linking of properties (when view link is enabled and you change a view parameter in one view, apply it to all other views), etc. I also feel that without the ability to assign texture to each model, PBR rendering is still too limited and experimental.
PBR is somewhat supported for segmentation nodes: metallic and roughness value can only be specified for the entire segmentation (not for each segment) and we haven’t added GUI for this yet. It would not be hard to resolve these limitations, but for now you need to export to models to edit these properties.
Image-based lighting can be used for volume rendering, too, but I don’t think it is implemented in VTK (at least not in the default OpenGL2 rendering backend). @Chris_Rorden showed some nice examples here, which proves that it is possible (I think MatCaps is the same or very similar to what VTK calls image-based lighting).
PBR and image-based lighting (especially with ambient shadows) looks to be particularly useful for viewing intricate structures with lots of overhangs and such. I’ll be testing this out quite a bit in the next little while.
Speaking of ambient shadows, the size scale parameter seems somewhat arbitrary - my CT scans are typically of things ~ 1cm in size, and I basically just tweak this parameter until it looks good. I confess I have not looked this up in the VTK docs at all. Are there units associated with it or is it just a non-dimensional adjustment?
This is how it is intended to be used - you look at the image and adjust the slider.
The slider sets a “scene size” using an exponential exponential scale. This size is then used to set a distance threshold value (that is used to decide if shadow should be casted between two objects) and the radius value (that specifies the spread of the shadow).
The distance threshold value (called bias in the SSAO algorithm) is computed as pow(10, self.ssaoSizeScaleLog - 1). The power of 10 is used to allow covering a large range of size with a single slider. The -1 offset is added so that 0.0 slider value corresponds to 100 length unit size (millimeter by default), which works well as a default.
The spread value (called radius in the SSAO algorithm) is set to 100x the distance threshold value.
It might make sense to display the scene size or distance threshold value to the user to make this slider a bit less mysterious.
Gotcha, thanks. I might try tweaking this bias calculation a bit, as the default unit size for 0.0 might be a little large for insects. I get a lot of mach banding in shadows and some flicker artifacts, not sure if this is due to an overly large bias setting. That said, I haven’t tried it at all on larger things, so maybe this is just how it works until you get a good value figured out.
I’ve had a chance to work with PBR, Lights, and Ambient shadows quite a bit in the last week or so and I have a few thoughts on it.
The results I can get from tweaking the lights and various settings is nothing short of fantastic. I can get screen shots that are very similar to the level of shading and lighting I can achieve using Blender.
I would love to be able to use this to create publication quality plates. However, for my use case, it’s not useable for this, primarily because (as far as I know) I’m not able to save camera position and restore a view exactly as it was for a previous screen grab. My workflow is to export screen captures (or renders from Blender) and annotate them with lines and annotations using Illustrator. I’m currently in the midst of a large comparative study on insect morphology, and this will often require updating segmentations and figures, meaning I will have to update and re-export a view with the same camera position and lighting settings as previously, so I can re-import it into Illustrator and have my lines and labels where they were before. It would be great if there was a “scene save” so that I could reload exactly the same scene setup and position.
I know we’ve had discussions previously regarding things like key framing animation of the camera, saving various parameters for lights and other settings, and I fully realize that many of these things are substantial tasks. I just wanted to relay some initial feedback after a couple of weeks using PBR and lights. If others have suggestions on things like saving camera positions, etc I would be very happy to discuss.
In short, I think the output is stunning, in fact I’ve showed a few of my segmentations to others and they’ve universally given a “holy crap” reaction. Thanks very much for your work on it!
How about using sceneviews for this? The module might be in need of maintenance as it does not capture volume rendering state and I am not sure about the lighting, but it does capture camera position
Yes, I could do that, but as @muratmaga mentioned in another post, I’d wind up with a lot of separate mrml files, one for each figure. It would get cumbersome pretty quickly.
Images - in the works… The “holy crap” was more for the lighting and shading now available than for my images! We have a few VG Studio users here that continue to be very surprised at how capable Slicer is.
Scene Views feature have had lots of instability issues over the years. Now we have a good plan how to fix all that, but we’ll only do this rework after Slicer5 is released.
Until then, I would recommend to use Sequences to store state of selected nodes - such as the camera, maybe volume rendering properties. See a short tutorial here:
