I’ve come across a serious limitation in our current database structure: the frame set (see also here) level is completely missing. We treat series as displayable set of frames, which is wrong for image modalities such as ultrasound and not always optimal for fluoroscopy and in many other cases. It is common to have dozens of ultrasound acquisitions in a single series that we currently cannot browse. We can switch to advanced mode and browser loadables (results of “Examine” step) but that’s very limited (e.g., no thumbnails) and slow.
I’m wondering if we should add a “FrameSet” table to the database. We would store each loadables item in a table, provide thumbnail and metadata for those and not for each series.
We can discuss this, but I agree we could implement a similar/enhanced UI of the default browser in the visual one with thumbnails.
Probably for the user knowing the concept of FrameSet is not important, so we could just provide a simple UI with thumbnails of the selected series where each new thumbnails/element comes either from:
differences in geometry of the instances, … etc… (i.e. the currrent examine operation)
I’m only thinking about improving the visual browser, as I expect that it can replace the old browser in the future.
Examine step provides the frame sets, so they are not two different things. I would not spend time with the old browser, we can leave it as is (show the list of series and generate framesets on-the-fly).
In the visual browser, the thumbnail list should always show the frameset list to the user, never the series list, so there is no need to ask the user. For this, we probably need a FrameSets table and the visual browser would display records of this table instead of the series table. The table would be populated by running “examine” for each imported (or updated) study automatically. We could store output of all plugins in the framelists table, but by default we would only show the results provided by the plugin with the highest confidence. We could add an option to the right-click menu of the thumbnail to choose outputs from a different plugin.
Yes, I think the browser should expose the logical structure of the acquisition as captured in the dicom data and support previews that make sense for the modality and use case. I think you should do what is needed for the visual browser.
Longer term we should rethink the database entirely, and define a new API that better refects framesets and other DICOM concepts that aren’t currently well represented.
In my experiments I found that the dicomDatabase can be completely replaced by a DICOMweb client so we should factor the database out from the plugins (which currently treat it as a global singleton). In this branch I made the database an optional argument for the ScalarVolumePlugin, and I think we can go a lot further to clean things up. This could be a good deliverable for some funding proposal.
the UI probably will need only minor refactoring (we could simply convert the series widget into a Frame sets one)
however for the logic (Examine step, which would be run as soon as we fetch the series and instances metadata), we would need some refactoring and probably move (at least part of) the plugins logic into CTK (C++).
Yes, the logic part is the difficult one. Unfortunately, it is not an option to move all the plugin logic to C++ because many people who have the domain knowledge to create plugins will provide it in their language of preference (now mostly Python, but soon also Rust, etc.).
It should not be a big issue, except we need to find a robust and fast way of running these plugins in parallel in the background. Multithreading in Python is rather limited, so we may need to run Python plugins in separate processes.
We will have to rework the plugin/database interface anyway to be able to run the plugins in the background. I’m not sure if DICOMweb REST API is optimal or sufficient, as we still need to store additional data, such as framesets. There might be some overhead due to HTTP and fully text-based communication, which might be perceivable when inspecting geometry of large series, such as high-resolution 4D CTs. But the API could be perhaps very similar to DICOMweb so that it would be trivial to use it with a DICOMweb server.
Yes, the database API should not assume DICOMweb, but it also should not assume DIMSE or filesystem either, but rather expose the metadata in a clean and efficient way, possibly pre-cached and possibly fetched on the fly.
Exposing the plugins to the C++ layer should not mean migrating the code to C++, it should mean providing a clean abstract C++ class that can be specialized to use the plugins in whatever language they come in. Running in threads or processes would be options to explore.
@David_Clunie gave a very useful information for implementation of framesets. IHE RAD TF-2 Transactions standard section 4.16.4.2.2.6.2 specifies a logic for determining framesets in a study. While the described logic may have some oversimplifying assumptions, it is good enough as a default implementation (and in the future we may implement plugin infrastructure for special cases).
Each DICOM image instance is a separate FrameSet, with two exceptions:
a Series of only single frame instances comprises a single FrameSet
the combined frames of a Concatenation comprise a single FrameSet (if Concatenations are supported, which they are not required to be)
This means that when a Series contains a mixture of single frame and multi-frame images, in order to preserve sequence of images within a Series:
each successive single frame image will be a separate FrameSet consisting of one frame
each multi-frame image will be a separate FrameSet (except for combined images of a Concatenation, if supported)
Furthermore, there will never be fewer FrameSets than Series (i.e., Series will not be combined).
The logic is simple enough that it could be possible to implement without a new database table. However, it would probably simplify the implementation if there was a 1-to-1 mapping between database table records and viewer items. Also, having a table would allow saving additional information into the database. For example, we could store how the user chose to load that frameset last time (what DICOM plugin, what interpretation), or we could use that to hide a frameset (instead of actually deleting it, thereby potentially corrupting the image series).
I’ve added an issue to track this feature request: