Failures saving a volume to path with % sign

jamesobutler · October 12, 2022, 6:55pm

I’m observing an issue with some file I/O when reading/writing metaimage mhd+raw data in a filepath location that contains a character such as “%”. I have seen users utilize this character for a directory name when describing a volume such as the concentration of a drug. This “%” character is also a valid character in the Windows file system.

Below is a python code snippet to observe the issue which can also be triggered when defining an output filepath for a metaimage using the Slicer Save dialog. You will see below that the output reports that the .mhd is written, but the corresponding raw file is not present. When writing the split file format for nrrd (nhdr+raw) is successful writing out both files.

The saving of the volume is successful with the single file format of .mha.

import os
import SampleData

volume_node = SampleData.SampleDataLogic().downloadMRHead()

output_filepath = os.path.join(os.getenv("USERPROFILE"), "Downloads", "2%concentration", "MyVolume.mhd")
slicer.util.saveNode(volume_node, output_filepath)
print(f"{output_filepath} exists?: {os.path.exists(output_filepath)}")
raw_filepath = os.path.splitext(output_filepath)[0] + '.raw'
print(f"{raw_filepath} exists?: {os.path.exists(raw_filepath)}")

output_filepath = os.path.join(os.getenv("USERPROFILE"), "Downloads", "2%concentration", "MyVolume.nhdr")
slicer.util.saveNode(volume_node, output_filepath)
print(f"{output_filepath} exists?: {os.path.exists(output_filepath)}")
raw_filepath = os.path.splitext(output_filepath)[0] + '.raw.gz'
print(f"{raw_filepath} exists?: {os.path.exists(raw_filepath)}")

Is there a workaround to support saving the split file format for metaimage to a filepath with a “%” character? Or does something need to be done to update to a newer version of the metaimage I/O library which might contain a bug fix?

jamesobutler · October 12, 2022, 7:01pm

It appears MetaIO has an issue written up for this and a PR trying to close it. @dzenanz Do you know who might be able to help move this along?

github.com/Kitware/MetaIO

Assumed string encoding for .mhd differs on platforms

opened 03:25PM - 10 Jan 19 UTC

codeling

Looking at the .mhd documentation, I see no explicit mention of any assumed enco…ding (or a setting for it). Yet when ElementDataFile points to a filename containing special characters, the character encoding is highly relevant for MetaIO to find the actual file. In my experiments (utilizing the MetaIO included in ITK), on Windows, an encoding of cp1252 is assumed, while on Linux, an encoding of utf-8 is expected. This means that when the filename given under ElementDataFile contains special characters, a separate .mhd file is required for Linux and Windows (and potentially more for other platforms I have not tested). What is thus required to make this consistent (and .mhd files with special characters transferrable between platforms), in my opinion, is to implement one of two options: 1. That .mhd files are required to have a specific encoding (utf-8 seems to be the logic choice), or 2. To have a separate entry specifying the encoding of the .mhd file Or am I missing something here, is there an encoding specification somewhere already?

github.com/Kitware/MetaIO

Make ElementDataFileName always use UTF-8 encoding

Kitware:master ← codeling:master

opened 12:34PM - 10 Aug 22 UTC

codeling

+82 -6

Attempt to fix https://github.com/Kitware/MetaIO/issues/68. Limitations: - O…nly handles the `ElementDataFileName` field in datasets of type metaImage; I have seen the same field being used in e.g. metaArray and metaFEMObject as well, but I cannot test these at the moment; as for other MET_STRING fields, I have no idea if any might need such handling as well, and if so what consequences this might have, therefore I didn't touch them. - Not sure if all possible cases are handled correctly; I only have limited ability to test, and I don't know all possibilites of how the code is used from VTK/ITK. There is for example a logic of determining the ElementDataFileName from the FileName in the ::Write method, which I don't fully get; or rather I don't get why it intermittently sets the ElementDataFileName to the full data file path, and only later cuts away the path again; because this makes it a bit tricky to handle if the path itself also contains special characters - one should not encode the full path immediately in utf-8, since later, the path is extracted again from the ElementDataFileName, and a comparisons to the non-utf-8 encoded version of the file path is done.... - The file name itself still has to be specified in local encoding (on Windows). - I considered adding a test case to testMeta3Image.cxx, but couldn't figure out how to set the file name properly, as source files are by default stored in utf-8 (but we would need to provide an ANSI filename on Windows) Maybe a solution using vtksys iostreams (https://gitlab.kitware.com/paraview/paraview/-/merge_requests/3850) would be preferrable? Though this would probably break backwards-compatibility, since all file names would then be expected to be in UTF-8 format I guess?

dzenanz · October 12, 2022, 9:50pm

Brad King (no profile on this forum) and @jcfr. They both commented on MetaIO#68.

todoooo · October 13, 2022, 1:00pm

Since % is not an extended character, I don’t see how the referenced MRs in MetaIO are relevant here.

Are you using a non-US keyboard? It is not clear to me what your Python code is demonstrating.

jamesobutler · October 13, 2022, 4:39pm

@todoooo I am using a regular US keyboard.

It appears relevant to the MetaIO issues because the ElementDataFile points to a filename with a special character (%) where the target filename is like the following:

“C:\Users\butlej30383\Downloads\2%concentration\MyVolume.mhd”

What my python code snippet is showing is that when saving the volume to this path, MetaIO only writes MyVolume.mhd, but does not write the corresponding MyVolume.raw (or MyVolume.zraw). MetaImage: M_WriteElementsData: file stream is fail after write

Things save correctly if I save with the single file format
“C:\Users\butlej30383\Downloads\2%concentration\MyVolume.mha”

The issue is when the separate ElementDataFile (.raw/.zraw) is to be a filepath that contains a special character such as %.

However when using the NRRD format, writing a MyVolume.nhdr is successful as it also writes the corresponding MyVolume.raw.gz. So NRRD is doing a better job handling this situation.

lassoan · October 13, 2022, 5:04pm

MetaIO supports One-Slice-Per-File Data Formats and it uses the % character to specify the slice number in the filename format string.

Due to the special meaning of the % character, currently MetaIO does not allow using this chracter in the path. This limitation should either be clearly documented or the implementation should be made more sophisticated (to somehow differentiate the % that refers to per-slice filename generation from simple % occurrences in the path; or by allowing completely disabling this extremely rarely used one-slice-per-file feature).

todoooo · October 14, 2022, 12:07am

It seems then that % in the filename should be treated with special meaning whereas % in the path name should be ignored.

As far as utf-8 encoding is concerned % is not a special character.

jamesobutler · October 17, 2022, 2:24pm

I’ve created the following issue to keep track of this problem:

github.com/Kitware/MetaIO

Unable to save data file to output filepath that contains a % character

opened 02:24PM - 17 Oct 22 UTC

jamesobutler

As originally posted about in https://discourse.slicer.org/t/failures-saving-a-v…olume-to-path-with-sign/25657 and more specifically in https://discourse.slicer.org/t/failures-saving-a-volume-to-path-with-sign/25657/6 thanks to @lassoan: > MetaIO supports [One-Slice-Per-File Data Formats 1](https://itk.org/Wiki/ITK/MetaIO/Documentation#Reading_DICOM_and_Other_One-Slice-Per-File_Data_Formats) and it uses the % character to specify the slice number in the filename format string. > > Due to the special meaning of the % character, currently [MetaIO does not allow using this chracter in the path](https://github.com/Kitware/MetaIO/blob/0b8c5284f3fe126968fd66c24639fd70b88e8788/src/metaImage.cxx#L1533). This limitation should either be clearly documented or the implementation should be made more sophisticated (to somehow differentiate the % that refers to per-slice filename generation from simple % occurrences in the path; or by allowing completely disabling this extremely rarely used one-slice-per-file feature). As such, % used in the basename could possibly still be treated with the special meaning for the filename format string for One-Slice-Per-File functionality however, % used elsewhere in the filepath should be supported. Last resort would be to document this limitation. The below code uses 3D Slicer to show that the mhd file is written for a volume output filepath that contains a "%", but the corresponding data file (.zraw) is not written. ```python import os import SampleData volume_node = SampleData.SampleDataLogic().downloadMRHead() output_filepath = os.path.join(os.getenv("USERPROFILE"), "Downloads", "2%concentration", "MyVolume.mhd") slicer.util.saveNode(volume_node, output_filepath) print(f"{output_filepath} exists?: {os.path.exists(output_filepath)}") raw_filepath = os.path.splitext(output_filepath)[0] + '.zraw' # default Slicer saves it compressed hence zraw and not raw print(f"{raw_filepath} exists?: {os.path.exists(raw_filepath)}") ```

Topic		Replies	Views
How to input and output segments in mhd format in 3D slicers. Support segmentation	3	350	September 28, 2023
.xml file output for .mhd file Support	2	14	August 29, 2024
Updating only header information for MetaImage/NRRD Development	5	858	December 24, 2020
Exporting labelmap in raw format Support	4	277	October 20, 2023
Different file formats unavailable Support	8	407	November 13, 2019

Failures saving a volume to path with % sign

Related topics