# DICOM database update is requested after new Slicer version, but appears to be impossible

**URL:** <https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821>\
**Category:** Support\
**Tags:** dicombrowser\
**Created:** [April 29, 2024, 9:49pm UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821 "2024-04-29T21:49:06Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![fedorov](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/fedorov/32/14_2.png) [@fedorov](https://discourse.slicer.org/u/fedorov)\
**Post date:** [April 29, 2024, 9:49pm UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/1 "2024-04-29T21:49:06Z")

</div>

I installed the latest nightly, and experienced this situation where DICOM Browser would request me to update the database, but trying to perform the update operation does not appear to have any effect.

Fortunately for me, I did not have a need to keep the database, and just created a new one, which showed up empty. After adding one study to that new database, my prior content re-appeared.

I do not know if this is expected behavior, and I could not find similar posts.

![2024-04-29_17-42-48](https://us1.discourse-cdn.com/flex002/uploads/slicer/original/3X/3/8/38b4d92fb010f7ce17487a80825f0cc1e5234b6d.gif)

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [April 29, 2024, 9:56pm UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/2 "2024-04-29T21:56:11Z")

</div>

That sounds like a regression. @Davide_Punzo @lassoan maybe related to the visual browser?

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [April 29, 2024, 10:08pm UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/3 "2024-04-29T22:08:00Z")

</div>

Until @Davide_Punzo can check this, the easiest is to create a new database.

---

<div class="post-metadata">

**Author:** ![Davide\_Punzo](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/davide_punzo/32/66104_2.png) [@Davide\_Punzo](https://discourse.slicer.org/u/Davide_Punzo)\
**Post date:** [April 29, 2024, 10:36pm UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/4 "2024-04-29T22:36:54Z")

</div>

Definitely it can be a regression from my last PR, thanks for reporting @fedorov !!  
Unfortunately the bug was not flagged either by the automated and manual tests. I would be curious to see if the database update works by using the new browser in your case. I clearly remember testing manually the feature, but I’m not sure that I tested it on both browsers.

Anyway, in any case, I’ll fix it asap, but I’m currently in vacation and I’ll be back on Monday (and I’ll make it a priority).

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [April 30, 2024, 2:04am UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/5 "2024-04-30T02:04:29Z")

</div>

Im wondering if we should keep offering this feature. It works by reimporting the DICOM files, which was OK when the database was just a cache of DICOM tags. However, the database now contains information that cannot be derived from the DICOM files (where was it retrieved from). So, we would need to improve the update process to preserve all these information; or not do it anymore (but automatically create a new empty database whenever the schema changes).

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [April 30, 2024, 11:35am UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/6 "2024-04-30T11:35:14Z")

</div>

I don’t like the idea of users losing their databases.

Schema changes should be very rare and they shouldn’t cause problems for users who don’t use new features. Maybe these extra fields should be stored separately if they are still subject to change.

---

<div class="post-metadata">

**Author:** ![Davide\_Punzo](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/davide_punzo/32/66104_2.png) [@Davide\_Punzo](https://discourse.slicer.org/u/Davide_Punzo)\
**Post date:** [April 30, 2024, 3:09pm UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/7 "2024-04-30T15:09:31Z")

</div>

The new field for the allowed/denied connections should not be an issue. For databases with the old schema when updated those fields would be simply empty and then the visual browser would use the server settings as default.

On the long term not sure what is the way to go. I generally agree with you, but at the same time losing the database at schema change would not be ideal as Steve pointed out

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [April 30, 2024, 7:10pm UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/8 "2024-04-30T19:10:36Z")

</div>

Thanks for working on this @Davide_Punzo and @lassoan 👍

On @fedorov 's original report, is there another bug that prevents him from updating the schema? Not clear to me why this would be in a loop when the schema update feature used to work.

---

<div class="post-metadata">

**Author:** ![Davide\_Punzo](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/davide_punzo/32/66104_2.png) [@Davide\_Punzo](https://discourse.slicer.org/u/Davide_Punzo)\
**Post date:** [April 30, 2024, 10:26pm UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/9 "2024-04-30T22:26:13Z")

</div>

Not sure, it should work (at least on the visual DICOM browser I know its working). Andrey also wrote that after adding a new study, the previous content re appeared. So probably the update was done successfully, but then something went wrong (maybe some update signals are broken).

On Monday I will investigate the issue on the default browser and I will let you know!

---

<div class="post-metadata">

**Author:** ![Davide\_Punzo](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/davide_punzo/32/66104_2.png) [@Davide\_Punzo](https://discourse.slicer.org/u/Davide_Punzo)\
**Post date:** [April 30, 2024, 10:40pm UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/10 "2024-04-30T22:40:27Z")

</div>

> [@lassoan](#):
>
> So, we would need to improve the update process to preserve all these information;

Ah yes this would be nice to implement

---

<div class="post-metadata">

**Author:** ![Davide\_Punzo](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/davide_punzo/32/66104_2.png) [@Davide\_Punzo](https://discourse.slicer.org/u/Davide_Punzo)\
**Post date:** [May 6, 2024, 8:30am UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/11 "2024-05-06T08:30:57Z")

</div>

@fedorov @pieper @lassoan I tried to reproduce this issue, but I could not.  
Specifically I have tried to update with latest Slicer a database created with Slicer-5.6.2-linux-amd64, i.e. DICOM database from version 0.7.0 to 0.8.1

Here a video:

@fedorov which OS are you using? @pieper could you please try to reproduce this on MacOS? and @lassoan could you try Windows as well, please?

we have also an automated test for this: [CTK/Libs/DICOM/Core/Testing/Cpp/ctkDICOMDatabaseTest3.cpp at master · commontk/CTK · GitHub](https://github.com/commontk/CTK/blob/master/Libs/DICOM/Core/Testing/Cpp/ctkDICOMDatabaseTest3.cpp)

but the CTK CI is run on linux.

---

<div class="post-metadata">

**Author:** ![cpinter](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/cpinter/32/7995_2.png) [@cpinter](https://discourse.slicer.org/u/cpinter)\
**Post date:** [May 6, 2024, 8:49am UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/12 "2024-05-06T08:49:51Z")

</div>

I did an update of a medium sized (~35 patients) database with the versions you mention yesterday on Windows (Slicer 5.7 from March 28), and it worked. The displayed name column content did not show during the update, but did do at the end.

I remember adding a ticket or commenting on one recently related to this (after doing some debugging, which showed a problem related with the displayed field generators - btw I implemented those back in the day), but can’t find it in CTK or Slicer. In any case, just letting know that the update worked for me too.

---

<div class="post-metadata">

**Author:** ![Davide\_Punzo](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/davide_punzo/32/66104_2.png) [@Davide\_Punzo](https://discourse.slicer.org/u/Davide_Punzo)\
**Post date:** [May 6, 2024, 9:10am UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/13 "2024-05-06T09:10:14Z")

</div>

> I did an update of a medium sized (~35 patients) database with the versions you mention yesterday on Windows (Slicer 5.7 from March 28), and it worked. The displayed name column content did not show during the update, but did do at the end.

ok perfect thanks for testing and let me know 🙂

> I remember adding a ticket or commenting on one recently related to this (after doing some debugging, which showed a problem related with the displayed field generators - btw I implemented those back in the day), but can’t find it in CTK or Slicer. In any case, just letting know that the update worked for me too.

ok if you find again the issue, please ping me, I will try to have a look!

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [May 6, 2024, 10:59am UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/14 "2024-05-06T10:59:59Z")

</div>

@Davide_Punzo it looks like the `Count` column is empty after the update. Can you test if this is a side-effect of updating the database or is it something that happens if you start with a fresh database (i.e. is the count populated in a new database).

---

<div class="post-metadata">

**Author:** ![fedorov](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/fedorov/32/14_2.png) [@fedorov](https://discourse.slicer.org/u/fedorov)\
**Post date:** [May 6, 2024, 3:03pm UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/15 "2024-05-06T15:03:17Z")

</div>

> [@Davide\_Punzo](#):
>
> @fedorov which OS are you using?

I am on macOS Sonoma 14.4.

---

<div class="post-metadata">

**Author:** ![Davide\_Punzo](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/davide_punzo/32/66104_2.png) [@Davide\_Punzo](https://discourse.slicer.org/u/Davide_Punzo)\
**Post date:** [May 7, 2024, 9:27am UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/16 "2024-05-07T09:27:41Z")

</div>

> @Davide_Punzo it looks like the `Count` column is empty after the update. Can you test if this is a side-effect of updating the database or is it something that happens if you start with a fresh database (i.e. is the count populated in a new database).

it is a general bug. I have investigated it and the issue is that:

1. in dcdeftag.h we have this definition:  
` #define DCM_SeriesInstanceUID DcmTagKey(0x0020, 0x000e)`
2. While in the metadata of that specific DICOM in my video the seriesIstanceUID tag is `"0020,000E"`
3. this create the issue in ctkDICOMDisplayedFieldGeneratorSeriesImageCountRule::getDisplayedFieldsForInstance:  
[CTK/Libs/DICOM/Core/ctkDICOMDisplayedFieldGeneratorSeriesImageCountRule.cpp at 7ed1da357b9e7e2462b4b764882612484c6fa051 · commontk/CTK · GitHub](https://github.com/commontk/CTK/blob/7ed1da357b9e7e2462b4b764882612484c6fa051/Libs/DICOM/Core/ctkDICOMDisplayedFieldGeneratorSeriesImageCountRule.cpp#L67)

because the return of `cachedTagsForInstance[dicomTagToString(DCM_SeriesInstanceUID)]` is an empty string (because `dicomTagToString(DCM_SeriesInstanceUID)` return “0020,000e” instead of “0020,000E”) and this creates issues when updating the tables.

I am not sure if this is a new regression, but I will check out differences in the CTK commits after Slicer version 5.6.2 and see if I can find where the regression happened.

@pieper @cpinter not sure what is the best way to fix the issue. Maybe we should fix the dicoms at loading/fetching time to have lowercase in the tags, instead of putting a lot of `if` exceptions everytime we use `dicomTagToString` to check lowercase vs uppercase

---

<div class="post-metadata">

**Author:** ![Davide\_Punzo](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/davide_punzo/32/66104_2.png) [@Davide\_Punzo](https://discourse.slicer.org/u/Davide_Punzo)\
**Post date:** [May 7, 2024, 9:32am UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/17 "2024-05-07T09:32:06Z")

</div>

> I am on macOS Sonoma 14.4.

ok, thanks. @fedorov @pieper would be possible, when you have time, to perfom the same test that I did in [DICOM database update is requested after new Slicer version, but appears to be impossible - #11 by Davide\_Punzo](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/11) on MacOS?

the idea would be to understand if it an issue at OS level (i.e. performing the same test I did on linux) or if it is something related to the database that Andrey tried to update (size, type of DICOM, version of the previous schema, etc…).

Finally, Andrey do you still have a copy of the database before updating it? and is it something that you can share?

---

<div class="post-metadata">

**Author:** ![Davide\_Punzo](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/davide_punzo/32/66104_2.png) [@Davide\_Punzo](https://discourse.slicer.org/u/Davide_Punzo)\
**Post date:** [May 7, 2024, 9:59am UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/18 "2024-05-07T09:59:32Z")

</div>

> [@Davide\_Punzo](#):
>
> > @Davide_Punzo it looks like the `Count` column is empty after the update. Can you test if this is a side-effect of updating the database or is it something that happens if you start with a fresh database (i.e. is the count populated in a new database).
> 
> it is a general bug. I have investigated it and the issue is that:
> 
> 1. in dcdeftag.h we have this definition:  
> ` #define DCM_SeriesInstanceUID DcmTagKey(0x0020, 0x000e)`
> 2. While in the metadata of that specific DICOM in my video the seriesIstanceUID tag is `"0020,000E"`
> 3. this create the issue in ctkDICOMDisplayedFieldGeneratorSeriesImageCountRule::getDisplayedFieldsForInstance:  
> [CTK/Libs/DICOM/Core/ctkDICOMDisplayedFieldGeneratorSeriesImageCountRule.cpp at 7ed1da357b9e7e2462b4b764882612484c6fa051 · commontk/CTK · GitHub](https://github.com/commontk/CTK/blob/7ed1da357b9e7e2462b4b764882612484c6fa051/Libs/DICOM/Core/ctkDICOMDisplayedFieldGeneratorSeriesImageCountRule.cpp#L67)
> 
> because the return of `cachedTagsForInstance[dicomTagToString(DCM_SeriesInstanceUID)]` is an empty string (because `dicomTagToString(DCM_SeriesInstanceUID)` return “0020,000e” instead of “0020,000E”) and this creates issues when updating the tables.
> 
> I am not sure if this is a new regression, but I will check out differences in the CTK commits after Slicer version 5.6.2 and see if I can find where the regression happened.
> 
> @pieper @cpinter not sure what is the best way to fix the issue. Maybe we should fix the dicoms at loading/fetching time to have lowercase in the tags, instead of putting a lot of `if` exceptions everytime we use `dicomTagToString` to check lowercase vs uppercase

@pieper I think the issue is this commit:

> <https://github.com/commontk/CTK/commit/84187713304e4ed5a457ee9758de1af4a22d8dbd>
>
> This generalizes the ctkDICOMDatabase code to ease hard-coded restrictions about…
> DICOM files always being on disk. Now the schema includes a \`URL\` field of the
> SQLite database so certain file-specific operations are only performed on filePaths
> while URL are handled independently. Entries in the \`Images\` table of the database
> may now have either a \`URL\` or a \`fileName\` or both and new accessors are provided
> to get \`URL\`s at the series and instance level.
> 
> Schema version is updated from version \`0.7.0\` to \`0.8.0\`.
> 
> Also this contains a fix to the \`ctkDICOMTagCache\` so that tags are always stored
> internally as uppercase hex, so a string like "0010,000d" will always be turned
> into "0010,000D" for storage in the database. Both forms can be used to query
> tags. This avoids the situation where some cache entries were duplicated because
> different code used either upper or lower case to refer to the same tag. Using
> only upper case should improve storage use and improve performance by avoiding
> unneeded cache misses.
> 
> Co-authored-by: Andras Lasso \<lasso@queensu.ca\>
> Co-authored-by: Davide Punzo \<punzodavide@hotmail.it\> 
> Co-authored-by: Jean-Christophe Fillion-Robin \<jchris.fillionr@kitware.com\>

I am not sure 100%, it could be a mix of things, but my investigation points to that. This because the count display works only up to Slicer version 5.6.2 which uses the dicom schema 0.7.0, i.e just before the linked commit which modifies the lower/upper case tags stuff too.

Adding .toUpper() in [CTK/Libs/DICOM/Core/ctkDICOMDisplayedFieldGeneratorAbstractRule.h at 7ed1da357b9e7e2462b4b764882612484c6fa051 · commontk/CTK · GitHub](https://github.com/commontk/CTK/blob/7ed1da357b9e7e2462b4b764882612484c6fa051/Libs/DICOM/Core/ctkDICOMDisplayedFieldGeneratorAbstractRule.h#L119) :

```auto

/// Utility function to convert a DICOM tag enum to string
  static QString dicomTagToString(const DcmTagKey& tag)
  {
    return QString("%1,%2").arg(tag.getGroup(),4,16,QLatin1Char('0')).arg(tag.getElement(),4,16,QLatin1Char('0')).toUpper();
  }

```

fix the issue, I have created a PR in CTK ([BUG: Convert lower case dicom tags by Punzo · Pull Request #1203 · commontk/CTK · GitHub](https://github.com/commontk/CTK/pull/1203)). But I am not sure if we need to apply the fix in any another part of CTK

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [May 7, 2024, 11:27am UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/19 "2024-05-07T11:27:02Z")

</div>

Ah, good catch Davide - yes, this mix of upper and lower case hex tags was a big pain. I think we need to be very consistent and use upper case everywhere.

---

<div class="post-metadata">

**Author:** ![Davide\_Punzo](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/davide_punzo/32/66104_2.png) [@Davide\_Punzo](https://discourse.slicer.org/u/Davide_Punzo)\
**Post date:** [May 7, 2024, 11:56am UTC](https://discourse.slicer.org/t/dicom-database-update-is-requested-after-new-slicer-version-but-appears-to-be-impossible/35821/20 "2024-05-07T11:56:06Z")

</div>

> Ah, good catch Davide - yes, this mix of upper and lower case hex tags was a big pain. I think we need to be very consistent and use upper case everywhere.

I agree. I had a look and I think the only missing one was this one.