You can jusr just the relevant parts of the image: setting up the display server and installing an extension. There are many other Slicer docker images for various purposes. You can find references to them in posts in this forum.
Unfortunately, we don’t have a single location where all the images are listed at, but we could create it now. @pieper, @muratmaga, @mau_igna_06 what do you think about adding links to all known Slicer docker (and singularity, etc.) images at SlicerDocker/README.rst at master · Slicer/SlicerDocker · GitHub? The earlier idea of having a single hierarchy of images may be unrealistic, but at least collecting links (with a 1-2 sentence description of each) at a single location should be feasible.