It should be available. In the save as dialog box, you should be able to change it back to .fcsv (in fact for me that’s the default for fiducials, mrk.json only is enabled for angles, planes, lines, and others).
Ah my bad. I got mixed up. I meant to say that it’d be great to still have the option to save curves as .fscv files.
The only time it makes sense to curves as fcsv, if you are using the control points as semi-landmarks in the analysis. Is that your use case?
And yes, it appears that ability to save curves as fcsv format also disappeared.
@smrolfe @lassoan @pieper is it possible to bring back the fcsv format for curves (both open and close), or should we approach this specifically in SlicerMorph?
We need to distinguish between:
- node saving/loading: lossless storage of all the information of a markups node, to be used when you save the scene
- node export/import: writing of some information of a markups node to a file that can be processed outside of Slicer; and reading markups stored in various formats
.fcsv or .csv format is perfectly fine for node export, but it is not suitable for node saving, as it cannot store all information that markups node can contain and the format is not flexible or extensible enough (you cannot easily add more metadata while keeping backward compatibility, you cannot store a variety of markups in a single file, etc.). Therefore, we’ll use json for node saving/loading.
However, users can very easily export markups to any other format of their choice, such as standard csv. If there file formats that seem to be preferred by many users, or used by other popular software then we can add importers/exporters for them in Slicer core.
@jmhuie Can you write a bit more about your application and how do you process exported data (what programming language or software do you use)?
We now have two storage nodes for markups: vtkMRMLMarkupsJsonStorageNode and vtkMRMLMarkupsFiducialStorageNode. File formats that are displayed in Save dialog are those that are offered by the current storage node. You can create a new storage node and assign it to the markups node to switch between fcsv and json but there is no GUI for this.
There could be many ways to allow users to more easily switch between fcsv and json (for example, merge the two storage nodes), but since fcsv is a lossy format for current markups (it cannot store all information that is in markup nodes), I don’t think we would want to keep using fcsv for saving fiducials. Csv import/export is certainly useful, so when we change default save format for fiducials to json then we’ll have to make csv/fcsv export conveniently available to users.
We already expose some lossy formats in the save dialog (e.g. STL of a model with scalars). Probably we should flag the lossy ones like fcsv but still give the option.
Slicer always offers saving in a lossless file format by default and we assume that users know what they are doing when they choose a different one. But of course users don’t always know the limitations of each format. There are some particularly sneaky errors, such as image directions lost when saving image as .vtk, or image axes orthogonalized when saving as nifti.
Maybe we should add a warning icon in the Save data dialog, which would be displayed when not all information of a node can be saved in the chosen format? Or show a warning at the end of the save operation (we wanted to implement display of more detailed error information anyway). Or maybe we should separate scene saving and data export more clearly (as it is done in other software, such as Gimp or Blender)? Although I have to admit that as a user I don’t like this forceful distinction between Dave and export in Gimp - I would just want to save the edited file by clicking “Save” instead of remembering that in this software I need to look for “Overwrite…”.
I like the ability to read the measurements, angles into SLicer with the new json format. However, I did probe our users when we discussed this initially a few months ago, and a lot people indicated they like the ease of reading fcsv into R (or other data analysis programs). So if possible, I would like the fcsv save as option to be enabled for curves in particular. I think it make sense to keep the json as the primary ‘lossless’ format, but give the option fcsv as a secondary format that user has to choose.
Another option separating Save As from Export that discussed briefly sometime ago was to implement Export As as a right-click action from the Data module.
To me the most consistent would be to flag the lossy file types, perhaps with a * or other warning icon when the type is selected for save (and a tooltip explaining the icon). This would help educate people who might choose to save surfaces in STL just because they have heard of it and not realize the limitations.
1 Like
Probably people feel more comfortable with tables because they know them already and they need to use them for computations anyway. While tables are great for data processing, they are not well suited for data archival, because if you store data in tables then you either need to use many tables cross-referencing each other; or have fewer tables, with highly redundant information. If you store data in a json file then you can create tables as temporary “views” of the data by a couple of lines of R code, but you need to educate users about how to do that.
But of course Slicer should push people too much to learn, even if that would be the right thing to do. Maybe a good solution would be to add a “Convert” section to markups module, which would allow converting markups to/from table nodes? In table export, there could be a few options about what columns to export and how. Table nodes can be already loaded and saved as standard csv files.
Do you know if users expect to see only control points or all curve point coordinates in the exported csv files?
I like the idea of providing an option to convert the Markups to Table nodes - that way people can see them directly in Slicer before saving. Maybe they can even do some simple plotting without having to leave Slicer. Would also be cool if we could create something like a MarkupStatistics that would create more sophisticated tables and plots.
We only need the control points. @smrolfe can chime in, but the curve resampling function in the curves would take a curve (and optioanlly a model) with a few control points, and would create a new curve by resampling specifed number of equidistant points on curve. These become the control points of the new curve, and that’s what people need to save and use in their analyses.
Ideas about the implementing additional summary tables and conversion tools is fine to continue to discuss, but for the time being we do need the fcsv save as option for curves for people to continue saving the way they used to.
Again, I think the default save mode should be json, with an option to choose to save as fcsv as format (perhaps with an indicator about lossiness).
I agree. I don’t see a need for all curve point coordinates when the resampling function can provide more/equidistant control points when needed. The control points should be sufficient.
How do your users use .fcsv files now?
- They cannot be parsed by standard csv readers because they have an invalid header (multiple lines of non comma-separated values). Do they manually remove the first few lines after each saving? Do they remove those lines programmatically in their analysis scripts?
- What do they do with the .fcsv curves after saving? They cannot load it back into Slicer as curves, just as fiducials.
- Is not a problem for them that they end up with as many files as curves? It is so much harder to manage, compare, process models if they are stored across many files. You can easily export many curves, labels, etc. into a single json file.
If your users really want direct csv export from Slicer then they would be probably much better off having a one-click export of standard .csv format (not .fcsv) in your module, rather than trying to make the generic Save data dialog work well for this specific use case.
I think for most other users it would be enough to add an Export/import section in Markups module, which could export/import markups to table and model nodes (similarly to how Segmentations can be imported/exported to labelmap volume and model nodes).
@jclauneuro I know that you use .fcsv files extensively, too. What do you think about switching to json? It would have many advantages, but we would want to ensure smooth transition and backward compatibility with old files.
1 Like
Reading the coordinates from fcsv into a dataframe in R is a single liner, which they loop over.
read.csv(file='F.fcsv', skip=2, header=T)[,2:4]
that’s what most people do.
What do they do with the .fcsv curves after saving? They cannot load it back into Slicer as curves, just as fiducials.
Most often, we only need the coordinates from the resampled curve. If they want to retain their original hand-drawn curve, they can save that as a json.
But this is already the default behavior for json format too! I get as many json’s to save as I have curve nodes.
By the way I am not opposing these changes, and I am sure we will find a good solution moving forward. However, not enabling the existing fcsv format cripples our current users. So, until we have a final solution, my suggestion is to re-enable (albeit temporarily) the fcsv format as a secondary file saving option.
The main issue is that vtkMRMLMarkupsFiducialStorageNode and vtkMRMLMarkupsJsonStorageNode are two different storage nodes, so you cannot switch between formats offered by them in Save data dialog (you need to delete one storage node and create the other type if you want to switch). Calling vtkMRMLMarkupsFiducialStorageNode from vtkMRMLMarkupsJsonStorageNode could be feasible, but not clear, would complicate things, and not something that we would want to maintain in the long term.
I think what your users would prefer is a convenient export option. You could export all the curves by a single click (you would not need to manually select/unselect node in save dialog, remember to set output folder, remember to set file format; then later remember to set file format back to json for saving). You could further simplify things for the users by doing resampling automatically during this export step (so users would not need to manually resample the curves). Doing resampling during export could be also beneficial because resampling is a destructive operation.
In the long term, everybody will win with switching to json. Reading part of a json file into an R dataframe may not be a one-liner, but should not be more than a couple of lines, and of course SlicerMorph could implement a small reader functions in Python and R to simplify things.
Saving multiple markups could be added as a subject hierarchy action to markups module’s plugin. It should be quite easy and it would be great if you could work on it, but eventually @jcfr or I will get to it, too.
I agree. Can the vtkMRMLMarkupsJsonStorageNode converted to vtkMRMLMarkupsFiducialStorageNode and exported as regular fcsv? Like right-click export as function? @smrolfe @pieper
Resampling may need tweaking of some parameters, and it can’t be done at bulk effectively. It is not destructive either, you can choose to output a new curve (although current default behavior is to overwrite the existing node).
No need to convert storage nodes. For export, you can create a temporary vtkMRMLMarkupsFiducialStorageNode, which does not interfere with the regular storage node that is used for saving (filename, format, and saved state are all preserved):
markupsNode = getNode('Curve1')
temporaryStorageNode = slicer.mrmlScene.AddNewNodeByClass("vtkMRMLMarkupsFiducialStorageNode")
temporaryStorageNode.SetFileName("c:/tmp/something.fcsv")
temporaryStorageNode.WriteData(markupsNode)
slicer.mrmlScene.RemoveNode(temporaryStorageNode)
I have been using the curves as semi-landmarks and exporting them as .fscv to be read and processed in R. I recognize the issue with .fscv not holding any data and trying to load those back into Slicer.
I do something along this line. I’ve written a short function in R that reads all of the .fscv files with a “XXXX_Genus_species.fscv” from a folder and converts it into an R friendly format. XXXX could represent any set of MarkupFiducials or curve with semi-landmarks. From there, it’s not really that bad to concatenate curves and points.
The AFIDS group seems to be on board with switching to json (they already convert fcsv to json). From Switching to json? · Issue #90 · afids/afids-validator · GitHub
This sounds great for us on the validator development end. With the majority of the code in Python, we are currently storing relevant information in an json object after reading in the fcsv file (skipping a few lines as someone in the discussion mentioned) as its easier for us to work with. A change in file format would likely be pretty simple to update once we know what keys to look for and would likely allow us to skip a few steps for users using Slicer5.
1 Like
Hi @lassoan thanks for looping our team in. I think the simple answer is that we will adapt to whatever specs are decided on for Slicer5, but ultimately we have interest in easily being able to convert between Slicer’s Markup file format and one that would be compatible with the Brain Imaging Data Structure (BIDS) specifications.
This may be a good time to try to determine if there is a point of convergence between the needs of both Slicer and BIDS for handling coordinates in .json files that might eliminate the need for any conversion tool. It may be better if I start a new thread about this since I would be veering quite off topic from the original thread.
1 Like