How to include markup annotations in high-resolution screen capture

Hello Slicer Community,

I am currently developing an extension for high-resolution screen captures within Slicer. My goal is to programmatically capture images of a reconstructed scene that include markup annotations such as landmark names, angles, and distance measurements. However, I’ve encountered an issue where these markup labels do not appear in the off-screen rendering used for capturing the scene. My research points to the existence of a vtkMRMLDisplayableManagerGroup object that should be associated with my threeDView. This group should contain the displayable managers responsible for rendering the markup labels. How would I go about accessing this object so I can incorporate these markup annotations into the newly rendered scene?

Below is the core snippet of my screen capture logic:

    def runScreenCapture(self) -> None:
        if self.resolution and self.outputPath:
            vtk.vtkGraphicsFactory()
            gf = vtk.vtkGraphicsFactory()
            gf.SetOffScreenOnlyMode(1)
            gf.SetUseMesaClasses(1)
            rw = vtk.vtkRenderWindow()
            rw.SetOffScreenRendering(1)
            ren = vtk.vtkRenderer()
            rw.SetSize(self.resolution[0], self.resolution[1])

            lm = slicer.app.layoutManager()

            threeDViewWidget = lm.threeDWidget(self.threeDViewIndex)
            threeDView = threeDViewWidget.threeDView()

            renderers = threeDView.renderWindow().GetRenderers()
            ren3d = renderers.GetFirstRenderer()

            # Set the background color of the off-screen renderer to match the original
            backgroundColor = ren3d.GetBackground()
            ren.SetBackground(backgroundColor)

            camera = ren3d.GetActiveCamera()

            while ren3d:

                actors = ren3d.GetActors()
                for index in range(actors.GetNumberOfItems()):
                    actor = actors.GetItemAsObject(index)

                    actor_class_name = actor.GetClassName()  # Get the class name using VTK's method
                    # Alternatively, use Python's type function: actor_type = type(actor).__name__
                    print("Actor index:", index, "Class name:", actor_class_name)

                    property = actor.GetProperty()
                    # print("Actor Property:", property)
                    representation = property.GetRepresentation()

                    # vtkProperty defines three representation types:
                    # vtkProperty.VTK_POINTS, vtkProperty.VTK_WIREFRAME, vtkProperty.VTK_SURFACE
                    if representation == vtk.VTK_POINTS:
                        print("Actor index:", index, "is represented as points.")
                    elif representation == vtk.VTK_WIREFRAME:
                        print("Actor index:", index, "is represented as wireframe.")
                    elif representation == vtk.VTK_SURFACE:
                        print("Actor index:", index, "is represented as a surface.")
                    else:
                        print("Actor index:", index, "has an unknown representation.")

                    print("Actor index:", index, "Visibility -", actor.GetVisibility(), "|", isActorVisible(camera, actor))
                    if actor.GetVisibility():  # and isActorVisible(camera, actor):
                        ren.AddActor(actor)  # Add only visible actors

                lights = ren3d.GetLights()
                for index in range(lights.GetNumberOfItems()):
                    ren.AddLight(lights.GetItemAsObject(index))

                volumes = ren3d.GetVolumes()
                for index in range(volumes.GetNumberOfItems()):
                    ren.AddVolume(volumes.GetItemAsObject(index))

                ren3d = renderers.GetNextItem()

            ren.SetActiveCamera(camera)

            rw.AddRenderer(ren)
            rw.Render()

            wti = vtk.vtkWindowToImageFilter()
            wti.SetInput(rw)
            wti.Update()
            writer = vtk.vtkPNGWriter()
            writer.SetInputConnection(wti.GetOutputPort())
            writer.SetFileName(self.outputPath)
            writer.Update()
            writer.Write()
            i = wti.GetOutput()

Here is the desired output:

But my current code only outputs:

Hi @oothomas -

Thanks for reporting this - I spent some time looking but didn’t see an obvious solution. If others want to try the updated code below can be copy-pasted into the python console for easier testing.

I suspect the underlying issue has to do with the way the Markups text is displayed in an overlay so you can see it even when the point itself is hidden. This overlay mode may need to be copied over into the new temporary renderer or render window.

Perhaps @Sunderlandkyl or @lassoan has ideas?

class self:
  resolution = [500,500]
  outputPath = "/tmp/image.png"
  threeDViewIndex = 0

def runScreenCapture(self) -> None:
    if self.resolution and self.outputPath:
        vtk.vtkGraphicsFactory()
        gf = vtk.vtkGraphicsFactory()
        gf.SetOffScreenOnlyMode(1)
        gf.SetUseMesaClasses(1)
        global rw, ren
        rw = vtk.vtkRenderWindow()
        rw.SetOffScreenRendering(1)
        ren = vtk.vtkRenderer()
        rw.SetSize(self.resolution[0], self.resolution[1])

        lm = slicer.app.layoutManager()

        threeDViewWidget = lm.threeDWidget(self.threeDViewIndex)
        threeDView = threeDViewWidget.threeDView()

        renderers = threeDView.renderWindow().GetRenderers()
        rendererCount = renderers.GetNumberOfItems()
        ren3d = renderers.GetFirstRenderer()

        # Set the background color of the off-screen renderer to match the original
        ren.SetBackground(ren3d.GetBackground())
        ren.SetBackground2(ren3d.GetBackground2())
        ren.SetGradientBackground(True)

        camera = ren3d.GetActiveCamera()

        rendererCount = 1 # only use first renderer to get Slicer content
        for rendererIndex in range(rendererCount):
            ren3d = renderers.GetItemAsObject(rendererIndex)
            print(f"\n\nrenderer {rendererIndex}")
            print(f"Layer: {ren3d.GetLayer()}")

            actors = ren3d.GetActors()
            for index in range(actors.GetNumberOfItems()):
                actor = actors.GetItemAsObject(index)

                actor_class_name = actor.GetClassName()  # Get the class name using VTK's method
                # Alternatively, use Python's type function: actor_type = type(actor).__name__
                print("Actor index:", index, "Class name:", actor_class_name)

                property = actor.GetProperty()
                # print("Actor Property:", property)
                representation = property.GetRepresentation()

                # vtkProperty defines three representation types:
                # vtkProperty.VTK_POINTS, vtkProperty.VTK_WIREFRAME, vtkProperty.VTK_SURFACE
                if representation == vtk.VTK_POINTS:
                    print("Actor index:", index, "is represented as points.")
                elif representation == vtk.VTK_WIREFRAME:
                    print("Actor index:", index, "is represented as wireframe.")
                elif representation == vtk.VTK_SURFACE:
                    print("Actor index:", index, "is represented as a surface.")
                else:
                    print("Actor index:", index, "has an unknown representation.")

                #print("Actor index:", index, "Visibility -", actor.GetVisibility(), "|", isActorVisible(camera, actor))
                if actor.GetVisibility():  # and isActorVisible(camera, actor):
                    ren.AddActor(actor)  # Add only visible actors

            lights = ren3d.GetLights()
            for index in range(lights.GetNumberOfItems()):
                ren.AddLight(lights.GetItemAsObject(index))

            volumes = ren3d.GetVolumes()
            for index in range(volumes.GetNumberOfItems()):
                ren.AddVolume(volumes.GetItemAsObject(index))

            ren3d = renderers.GetNextItem()

        ren.SetActiveCamera(camera)

        rw.AddRenderer(ren)
        rw.Render()

        wti = vtk.vtkWindowToImageFilter()
        wti.SetInput(rw)
        wti.Update()
        writer = vtk.vtkPNGWriter()
        writer.SetInputConnection(wti.GetOutputPort())
        writer.SetFileName(self.outputPath)
        writer.Update()
        writer.Write()
        i = wti.GetOutput()

        global label
        label = qt.QLabel()
        label.size = qt.QSize(*self.resolution)
        label.show()
        pixmap = qt.QPixmap()
        pixmap.load(self.outputPath)
        label.setPixmap(pixmap)

slicer.modules.markups.logic().AddControlPoint(0,0,0)
slicer.app.processEvents()
runScreenCapture(self)

Should show something like this:

2 Likes

Thanks looking into this @pieper. High quality, 3D rendering of scenes is a common request we get from SlicerMorph users (e.g., for printed posters or journal covers). It will be great to have this feature fully available (including annotations).

@lassoan @Davide_Punzo your inputs will be much appreciated.

1 Like

Computing visibility of a markup control point is a complex operation, which has to be performed very quickly. Therefore we have to use a hardware picker, which captures the z buffer before overlay rendering and caches the result. Then in the overlay rendering step we determine which control points are occluded and only display the label for those points that are not occluded. This complex processing can break many ways.

So, I’m not surprised that the code above has issues. If you take care of initializing the render window the exact same way as it is done in the 3D viewer and copy all relevant renderers then there is a chance that some issues will go away. But the offscreen renderer may work a bit differently or may have different bugs or some features may be missing, so you may need to dig in deeper to fix the label display.

Alternatively, you could keep the render window as is, and also avoid offscreen rendering or switch to a software renderer (Mesa), by rendering the scene in NxN tiles (only the camera parameters are adjusted for each tile). Such logic is implemented in vtkRenderLargeImage, but only for one renderer.

2 Likes

I am not sure of your requirements, but another simple option, it could be to instantiate a 3D widget outside the view layout, e.g.: Script repository — 3D Slicer documentation . Then you could resize the full widget to get the desired high resolution. It is not an offscreen render (which is a cleaner solution), but you can still hide the widget in some way (e.g. adding it to a layout of another widget and set it to “lower”).

1 Like

But then rendering would be limited the size of the screen resolution, right? If that’s correct, that’s the issue.

Hi @pieper -

Thank you for adding an easy testing option. I had a feeling the fact that this was an overlay had something to do with the issue. For now, I have updated HiResScreenCapture and added a disclaimer to the tutorial noting that markup labels and annotations won’t be displayed. I’m looking forward to exploring this problem more so I will keep updating this thread with any progress I make.

1 Like

Thanks everyone for assisting with this. Your help is well appreciated. I will continue to probe the layout manager and associated objects to see if I can find anything useful. I want to learn more about how these overlays are handled. I’ve found nothing so far in the renderers in ‘threeDView’ so i’ll keep probing.

There are many invisible objects in the renderer, this sort of explains why that might be the case. I’ll check vtkRenderlargeImage to see if there’s another possible solution.

Thank you for explaining!

That’s a great idea!

You can resize the window to any size - does not matter how large your display is (I’ve tested it on Windows, I don’t see a reason why it would not work on other operating systems or window managers). So, a code snippet to render a 3D view at arbitrarily high resolution:

Set up a 3D view outside the main view layout:

# Switch to a layout that has a window that is not in the main window
layoutManager = slicer.app.layoutManager()
originalLayout = layoutManager.layout
layoutManager.setLayout(slicer.vtkMRMLLayoutNode.SlicerLayoutDualMonitorFourUpView)
# Maximize the 3D view within this layout
viewLogic = slicer.app.applicationLogic().GetViewLogicByLayoutName("1+")
viewNode = viewLogic.GetViewNode()
layoutManager.addMaximizedViewNode(viewNode)

Now set up visualization in the 3D view (e.g., drag-and-drop an image to show it with volume rendering).

# Resize the view
viewWidget = layoutManager.viewWidget(viewNode)
# Parent of the view widget is the frame, parent of the frame is the docking widget
layoutDockingWidget = viewWidget.parent().parent()
originalSize = layoutDockingWidget.size
layoutDockingWidget.resize(3000,3000)
# Capture the view
import ScreenCapture
cap = ScreenCapture.ScreenCaptureLogic()
cap.captureImageFromView(viewWidget.threeDView(), "c:/tmp/test.png")
# Restore original size and layout
layoutDockingWidget.resize(originalSize)
layoutManager.setLayout(originalLayout)

Since this is a regular 3D view, everything will just work - markups will appear the same way as in any other view, you can use it with screen capture module to capture videos, etc.

4 Likes

This worked real well, thank you both!

@oothomas you can revise your module to use this approach.

3 Likes

Thank you so much @Andras, this is very helpful!

While this seem to work for the annotation, on closer inspection, it doesn’t seem to do what we want to do, since it doesn’t seem to preserve the zoom in levels, centering etc in the 3D window, and adds a lot of padding to the image.
For example, this is what I have a in my 3D viewer:

and this is what I get out of the python code above.

It is an independent 3D view from the one that is in the 4-up layout, so zooming in another view will not affect it. You could set the camera in the detached view using your mouse; or you could copy the camera parameters to the detached view, etc. However, there could be still differences if certain nodes are selected to be shown in certain views. To address all these, you could add a custom layout (just a short xml string set in the layout manager, see examples here: Slicer/Libs/MRML/Logic/vtkMRMLLayoutLogic.cxx at bd2c929e54eefd196ffe3105f6b5235c14fa7099 · Slicer/Slicer · GitHub) that displays the selected 3D view in a separate viewport. After the screenshot is taken, you can switch back to the previous layout.

1 Like

Hi @lassoan,

Thanks for your previous help! I’m integrating what you’ve posted into HiResScreenCapture and have encountered a few issues:

1. Volume Rendering Not Displayed After making volumes visible, they now appear behind the bounding box instead of in front. Here’s what it looks like:

2. Missing Portions in High-Resolution Screenshots When taking screenshots at higher resolutions, some parts of the image are cut off. Here’s the expected vs. actual output:
Expected view:


Actual View:

3. Markup Scaling Issues Markups and annotations do not scale with resolution increases, making them nearly invisible at higher resolutions. How can we adjust the scaling for points, angles, and lines? Below is the function I’m using to scale annotations:

    def adjustMarkupSize(viewWidget, scaleFactor):
        # Retrieve the markup display node
        markupsNode = slicer.mrmlScene.GetFirstNodeByClass("vtkMRMLMarkupsDisplayNode")
        if markupsNode:
            # You can access specific properties of viewWidget if needed to further refine scaling
            # For example, you could adjust based on the widget's size or other relevant properties
            widgetSize = viewWidget.size
            scaleAdjustment = max(widgetSize.width(), widgetSize.height()) / 1  # Example scaling factor

            # Adjust size based on a scaling factor combined with widget dimensions
            originalMarkupSize = markupsNode.GetTextScale()
            newScale = originalMarkupSize * scaleFactor * scaleAdjustment
            markupsNode.SetTextScale(newScale)
            print(f"Adjusted markup text scale to: {newScale}")

            return originalMarkupSize

I haven’t implemented this annotation scaling yet. I plan to calculate a scale factor given the resolution specified by the user. Am I going in the right direction?

Code for Screen Capture : Here’s the complete function I’m using for screen capturing:

    def runScreenCapture(resolution, outputPath) -> None:
        if resolution and outputPath:
            layoutManager = slicer.app.layoutManager()
            originalLayout = layoutManager.layout
            originalViewNode = layoutManager.threeDWidget(0).mrmlViewNode()
            originalCamera = slicer.modules.cameras.logic().GetViewActiveCameraNode(originalViewNode)

            # Debugging: Print original camera settings
            print("Original Camera Settings:")
            print("Position:", originalCamera.GetPosition())
            print("Focal Point:", originalCamera.GetFocalPoint())
            print("View Up:", originalCamera.GetViewUp())

            # Set the layout to include the necessary view
            layoutManager.setLayout(slicer.vtkMRMLLayoutNode.SlicerLayoutDualMonitorFourUpView)
            viewLogic = slicer.app.applicationLogic().GetViewLogicByLayoutName("1+")
            viewNode = viewLogic.GetViewNode()
            layoutManager.addMaximizedViewNode(viewNode)
            newCamera = slicer.modules.cameras.logic().GetViewActiveCameraNode(viewNode)

            # Set and debug new camera settings
            newCamera.SetPosition(originalCamera.GetPosition())
            newCamera.SetFocalPoint(originalCamera.GetFocalPoint())
            newCamera.SetViewUp(originalCamera.GetViewUp())
            print("New Camera Settings Applied")

            # Ensure volume rendering is visible in the new view
            volumeRenderingLogic = slicer.modules.volumerendering.logic()
            volumeNode = slicer.mrmlScene.GetFirstNodeByClass("vtkMRMLVolumeNode")
            displayNode = volumeRenderingLogic.GetFirstVolumeRenderingDisplayNode(volumeNode)
            if displayNode:
                displayNode.SetVisibility(True)
                displayNode.SetViewNodeIDs([viewNode.GetID()])

            # Resize and capture the view
            viewWidget = layoutManager.viewWidget(viewNode)
            layoutDockingWidget = viewWidget.parent().parent()
            originalSize = layoutDockingWidget.size
            layoutDockingWidget.resize(resolution[0], resolution[1])

            # Force a redraw
            viewWidget.threeDView().forceRender()

            # Capture the view
            cap = ScreenCapture.ScreenCaptureLogic()
            cap.captureImageFromView(viewWidget.threeDView(), outputPath)

            # Restore original size and layout
            layoutDockingWidget.resize(originalSize.width(), originalSize.height())
            # layoutManager.setLayout(originalLayout)

            # reset volume rendering visibility
            if displayNode:
                displayNode.SetVisibility(True)
                displayNode.SetViewNodeIDs([originalViewNode.GetID()])

            print("Capture Completed")

            # Make all other view nodes except the original one invisible
            allViewNodes = slicer.mrmlScene.GetNodesByClass("vtkMRMLViewNode")
            allViewNodes.InitTraversal()
            viewNode = allViewNodes.GetNextItemAsObject()
            while viewNode:
                if viewNode.GetID() != originalViewNode.GetID():
                    slicer.mrmlScene.RemoveNode(viewNode)
                viewNode = allViewNodes.GetNextItemAsObject()

            # Refresh the layout to reflect changes
            layoutManager.layout = originalLayout

Could you suggest any improvements or point out what might be going wrong with these features?

Thanks for your continued help!

Best,
Oshane

There is indeed something wrong with the z buffer initialization sometimes, which makes the volume rendering not occlude the lines of the box.

As a workaround, you can nudge the view’s camera a bit to force a re-rendering: getNode('Camera_2').GetCamera().Dolly(1.001)

If you have a way to reproduce the behavior then please submit an issue to issues.slicer.org and reference it here.

It is because the magnified view has a different shape. You have to record the originalSize before you change layout.

You can change the screen scale factor to preserve relative size of markup control points and labels. This factor has not been exposed on the public API. I have added SetScreenScaleFactor method to the view node that allows you to change it (save the original value, set higher value, take the screenshot, and restore the original value). It will probably take a few days to get the update reviewed and merged into the Slicer Preview Release.

Hi Everyone,

I want to extend my heartfelt thanks to all of you for your invaluable assistance with this module thus far. Special thanks to Andras, Steve, and everyone else who helped resolve the issues during yesterday’s developer call.

I’m pleased to share the current version of the module:

HiResScreenCapture Module on GitHub

The module is functioning very well, and I’ve simplified the UI to enhance ease of use. I would greatly appreciate any additional suggestions or feedback you might have. I’m also eager to discuss how some of these features can be integrated into the screen capture module.

Thank you once again for your support.

Best regards,

Oshane


4 Likes

Thanks for joining the call and thanks for working on Slicer!

Thanks Oshane for making this available, I will test it out and report back!