Slicer DICOM import performance

Support
1

Today I tried MITK, and I noticed that DICOM import operation is order of magnitude faster there than in Slicer.

I tested this specific dataset: https://bit.ly/QIN-HN-137 on the same Linux system with the latest MITK package (2016.11) and the latest Slicer nightly. In MITK import of this dataset takes seconds, and in Slicer it is over 20 seconds. Considering both platforms are using the common set of components from CTK, is this behavior expected?

1 Like
2

Does it have a DICOMDIR file? (download from the link is really slow…)
Is the import slower if you delete the DICOMDIR file?

3

No, there is no DICOMDIR file.

4

I forgot to mention - I chose “Add Link” option in Slicer while importing the dataset.

5

Yes, we should profile this. I suspect it’s either the tagCache using sqlite inefficiently or unneeded updates of the widget, but it would be good to know and fix.

6

I logged an issue https://issues.slicer.org/view.php?id=4420

7

@fedorov It takes 11 seconds to import the complete folder. Considering my hard drive is very efficient (SSD NVME) … I would expect it to be faster.

8

Both for mitk and Slicer?

Anyway, I reported the time comparison on the same Linux box.

9

Only Slicer, 11 seconds is to slow. I am profiling and experimenting with few approaches.

1 Like
10

I instrumented CTK to report some timing.

"DICOM indexer has successfully processed 556 files [8.12s]"
11

By commenting out this line, import time is reduced by a factor 2.

"DICOM indexer has successfully processed 556 files [3.39s]"
1 Like
12

That is a significant improvement!

It would be useful to keep the possibility of cancelling the import, so instead of removing the line, we could add a check which would only allow processing of events if at least a few seconds has passed since the last processing.

13

I agree, improvement of factor of 2 is great. If we could get to factor of 10 to be on par with CTK (and probably much closer to OsiriX), this would be perfect!

@jcfr thank you for investigating this issue! :+1:

14

I was able to successfully close the dialog without that line on both Qt4 and Qt5 on Linux. Can someone try on windows ?

This line was added back in 2013 … it doesn’t seem to be useful anymore

If we could get to factor of 10 to be on par with CTK

For reference, MITK is not doing anything is special with CTKDICOM:

15

I did some profiling and about 50% of the time is spent in precacheTags, which is the part of the code that allows DICOMPlugins to save header values that they want to have fast access to in the future. This is being done as an sqlite transaction per-dicom object, but maybe there is a more efficient way to do this. Perhaps the tags can be stored in memory and then pushed to the database in one transaction when the parsing is finished (ideally the database transaction would be done in the background, but I don’t think that’s possible with sqlite).

16

I’ll try this on Windows with Qt4.

1 Like
17

Here is the latest PR: https://github.com/commontk/CTK/pull/740

With this commit, I managed to go from 3.3s to 2.9s.

https://github.com/commontk/CTK/pull/740/commits/1e92bdcb4fc829823d7ae65bfbdbb39ace748557

1 Like
18

The changes work without problems on Windows with Qt4.

I’ve made some speed measurements on loading 6 CTs:

  • With patched CTK of this PR: DICOM indexer has successfully processed 1554 files [255.59s]
  • If I comment out this->precacheTags(sopInstanceUID) call in ctkDICOMDatabase: DICOM indexer has successfully processed 1554 files [9.78s] !!!

There are some good tips on improving SQLite insert speed:


It seems that the most significant improvement (to 85 inserts/sec to 23000 inserts/sec) by using transactions.

After some investigation I’ve found a bug in how transactions is set up for inserting tagcache values. Actually, a transaction is created for the DICOM patient database and not for tag cache!

This change inctkDICOMDatabasePrivate::precacheTags decreases import time from 255 sec to 19 sec!

Before:

  this->beginTransaction();
  q->cacheTags(sopInstanceUIDs, tags, values);
  this->endTransaction();

After:

  QSqlQuery transaction(this->TagCacheDatabase);
  transaction.prepare("BEGIN TRANSACTION");
  transaction.exec();
  q->cacheTags(sopInstanceUIDs, tags, values);
  QSqlQuery transaction2(this->TagCacheDatabase);
  transaction2.prepare("END TRANSACTION");
  transaction2.exec();

Probably if we start/end the transaction for all files at once then it would further improve the speed.

@jcfr Could you add this to your branch and test if using this change you get a significant speed improvement, too?

2 Likes
19

This sounds really great!

I can test after Friday.

20

With this change in place, I get the following stats:

"DICOM indexer has successfully processed 556 files [2.12s]"
3 Likes