# Slicer DICOM import performance

**URL:** <https://discourse.slicer.org/t/slicer-dicom-import-performance/922>\
**Category:** Support\
**Tags:** dicombrowser\
**Created:** [August 22, 2017, 9:49pm UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922 "2017-08-22T21:49:18Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![fedorov](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/fedorov/32/14_2.png) [@fedorov](https://discourse.slicer.org/u/fedorov)\
**Post date:** [August 22, 2017, 9:49pm UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/1 "2017-08-22T21:49:18Z")

</div>

Today I tried MITK, and I noticed that DICOM import operation is order of magnitude faster there than in Slicer.

I tested this specific dataset: [https://bit.ly/QIN-HN-137](https://bit.ly/QIN-HN-137) on the same Linux system with the latest MITK package (2016.11) and the latest Slicer nightly. In MITK import of this dataset takes seconds, and in Slicer it is over 20 seconds. Considering both platforms are using the common set of components from CTK, is this behavior expected?

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [August 22, 2017, 9:53pm UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/2 "2017-08-22T21:53:38Z")

</div>

Does it have a DICOMDIR file? (download from the link is really slow…)  
Is the import slower if you delete the DICOMDIR file?

---

<div class="post-metadata">

**Author:** ![fedorov](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/fedorov/32/14_2.png) [@fedorov](https://discourse.slicer.org/u/fedorov)\
**Post date:** [August 22, 2017, 9:55pm UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/3 "2017-08-22T21:55:37Z")

</div>

No, there is no DICOMDIR file.

---

<div class="post-metadata">

**Author:** ![fedorov](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/fedorov/32/14_2.png) [@fedorov](https://discourse.slicer.org/u/fedorov)\
**Post date:** [August 22, 2017, 9:59pm UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/4 "2017-08-22T21:59:52Z")

</div>

I forgot to mention - I chose “Add Link” option in Slicer while importing the dataset.

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [August 22, 2017, 10:35pm UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/5 "2017-08-22T22:35:26Z")

</div>

Yes, we should profile this. I suspect it’s either the tagCache using sqlite inefficiently or unneeded updates of the widget, but it would be good to know and fix.

---

<div class="post-metadata">

**Author:** ![fedorov](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/fedorov/32/14_2.png) [@fedorov](https://discourse.slicer.org/u/fedorov)\
**Post date:** [August 22, 2017, 10:45pm UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/6 "2017-08-22T22:45:25Z")

</div>

I logged an issue [https://issues.slicer.org/view.php?id=4420](https://issues.slicer.org/view.php?id=4420)

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [August 23, 2017, 3:40am UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/7 "2017-08-23T03:40:57Z")

</div>

@fedorov It takes 11 seconds to import the complete folder. Considering my hard drive is very efficient (SSD NVME) … I would expect it to be faster.

---

<div class="post-metadata">

**Author:** ![fedorov](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/fedorov/32/14_2.png) [@fedorov](https://discourse.slicer.org/u/fedorov)\
**Post date:** [August 23, 2017, 3:55am UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/8 "2017-08-23T03:55:14Z")

</div>

Both for mitk and Slicer?

Anyway, I reported the time comparison on the same Linux box.

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [August 23, 2017, 3:57am UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/9 "2017-08-23T03:57:07Z")

</div>

> [@fedorov](#):
>
> Both for mitk and Slicer?

Only Slicer, 11 seconds is to slow. I am profiling and experimenting with few approaches.

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [August 23, 2017, 4:06am UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/10 "2017-08-23T04:06:42Z")

</div>

I instrumented CTK to report some timing.

```auto
"DICOM indexer has successfully processed 556 files [8.12s]"

```

> <https://github.com/commontk/CTK/pull/739>

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [August 23, 2017, 5:25am UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/11 "2017-08-23T05:25:12Z")

</div>

By commenting out this line, import time is reduced by a factor 2.

> <https://github.com/commontk/CTK/blob/1f3eba2282b892cc0603cb29d31f7949459944f0/Libs/DICOM/Widgets/ctkDICOMBrowser.cpp#L428>

```auto
"DICOM indexer has successfully processed 556 files [3.39s]"

```

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [August 23, 2017, 1:17pm UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/12 "2017-08-23T13:17:39Z")

</div>

That is a significant improvement!

It would be useful to keep the possibility of cancelling the import, so instead of removing the line, we could add a check which would only allow processing of events if at least a few seconds has passed since the last processing.

---

<div class="post-metadata">

**Author:** ![fedorov](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/fedorov/32/14_2.png) [@fedorov](https://discourse.slicer.org/u/fedorov)\
**Post date:** [August 23, 2017, 1:26pm UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/13 "2017-08-23T13:26:59Z")

</div>

I agree, improvement of factor of 2 is great. If we could get to factor of 10 to be on par with CTK (and probably much closer to OsiriX), this would be perfect!

@jcfr thank you for investigating this issue! 👍

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [August 23, 2017, 5:35pm UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/14 "2017-08-23T17:35:01Z")

</div>

> [@lassoan](#):
>
> so instead of removing the line, we could add a check which would only allow processing of events if at least a few seconds has passed since the last processing.

I was able to successfully close the dialog without that line on both Qt4 and Qt5 on Linux. Can someone try on windows ?

This line was added back in 2013 … it doesn’t seem to be useful anymore

> If we could get to factor of 10 to be on par with CTK

For reference, MITK is not doing anything is special with CTKDICOM:

> <https://github.com/MITK/MITK/blob/cf6879c304025341d6e99c2aa13f6ee7f700dba0/Modules/DicomUI/src/QmitkDicomLocalStorageWidget.cpp#L67-L81>

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [August 23, 2017, 5:58pm UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/15 "2017-08-23T17:58:12Z")

</div>

I did some profiling and about 50% of the time is spent in precacheTags, which is the part of the code that allows DICOMPlugins to save header values that they want to have fast access to in the future. This is being done as an sqlite transaction per-dicom object, but maybe there is a more efficient way to do this. Perhaps the tags can be stored in memory and then pushed to the database in one transaction when the parsing is finished (ideally the database transaction would be done in the background, but I don’t think that’s possible with sqlite).

> <https://github.com/commontk/CTK/blob/master/Libs/DICOM/Core/ctkDICOMDatabase.cpp#L1262-L1286>

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [August 23, 2017, 9:21pm UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/16 "2017-08-23T21:21:16Z")

</div>

> [@jcfr](#):
>
> lassoan:
> 
> so instead of removing the line, we could add a check which would only allow processing of events if at least a few seconds has passed since the last processing.
> 
> I was able to successfully close the dialog without that line on both Qt4 and Qt5 on Linux. Can someone try on windows ?

I’ll try this on Windows with Qt4.

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [August 24, 2017, 2:34am UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/17 "2017-08-24T02:34:04Z")

</div>

Here is the latest PR: [https://github.com/commontk/CTK/pull/740](https://github.com/commontk/CTK/pull/740)

With this commit, I managed to go from 3.3s to 2.9s.

[https://github.com/commontk/CTK/pull/740/commits/1e92bdcb4fc829823d7ae65bfbdbb39ace748557](https://github.com/commontk/CTK/pull/740/commits/1e92bdcb4fc829823d7ae65bfbdbb39ace748557)

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [August 24, 2017, 4:38pm UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/18 "2017-08-24T16:38:24Z")

</div>

The changes work without problems on Windows with Qt4.

I’ve made some speed measurements on loading 6 CTs:

- With patched CTK of this PR: DICOM indexer has successfully processed 1554 files [255.59s]
- If I comment out this-\>precacheTags(sopInstanceUID) call in ctkDICOMDatabase: DICOM indexer has successfully processed 1554 files [9.78s] !!!

There are some good tips on improving SQLite insert speed:

> <https://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite>

  
It seems that the most significant improvement (to 85 inserts/sec to 23000 inserts/sec) by using transactions.

After some investigation I’ve found a bug in how transactions is set up for inserting tagcache values. Actually, a transaction is created for the DICOM patient database and not for tag cache!

**This change inctkDICOMDatabasePrivate::precacheTags decreases import time from 255 sec to 19 sec!**

Before:

```
this->beginTransaction();
  q->cacheTags(sopInstanceUIDs, tags, values);
  this->endTransaction();
```

After:

```
QSqlQuery transaction(this->TagCacheDatabase);
  transaction.prepare("BEGIN TRANSACTION");
  transaction.exec();
  q->cacheTags(sopInstanceUIDs, tags, values);
  QSqlQuery transaction2(this->TagCacheDatabase);
  transaction2.prepare("END TRANSACTION");
  transaction2.exec();
```

Probably if we start/end the transaction for all files at once then it would further improve the speed.

@jcfr Could you add this to your branch and test if using this change you get a significant speed improvement, too?

---

<div class="post-metadata">

**Author:** ![fedorov](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/fedorov/32/14_2.png) [@fedorov](https://discourse.slicer.org/u/fedorov)\
**Post date:** [August 24, 2017, 4:52pm UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/19 "2017-08-24T16:52:41Z")

</div>

This sounds really great!

I can test after Friday.

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [August 24, 2017, 5:04pm UTC](https://discourse.slicer.org/t/slicer-dicom-import-performance/922/20 "2017-08-24T17:04:15Z")

</div>

> [@lassoan](#):
>
> Could you add this to your branch and test if using this change you get a significant speed improvement, too?

With this change in place, I get the following stats:

```auto
"DICOM indexer has successfully processed 556 files [2.12s]"

```

[Next page](https://discourse.slicer.org/t/slicer-dicom-import-performance/922.md?page=2)