# Extension download statistics with the new backend

**URL:** <https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819>\
**Category:** Support\
**Created:** [September 23, 2021, 5:11am UTC](https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819 "2021-09-23T05:11:08Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![muratmaga](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/muratmaga/32/3622_2.png) [@muratmaga](https://discourse.slicer.org/u/muratmaga)\
**Post date:** [September 23, 2021, 5:11am UTC](https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819/1 "2021-09-23T05:11:08Z")

</div>

I tried to download the extension download statistics using the stable, and got the error below. This used to work, so I assume this is related to change in extension server.

Going forward, what is the plan on obtaining extension stats?

```auto
Switch to module: "ExtensionStats"
Traceback (most recent call last):
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 579, in getExtensionSlicerRevisionAndDownloads
    item_rev_downloads[itemid] = [self.getItemById(url, itemid)['download'], slicerrevision]
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 553, in getItemById
    return self._call_midas_url(url, data)
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 515, in _call_midas_url
    response_data = response_dict['data']
KeyError: 'data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 153, in onApplyButton
    release_downloads = self.logic.getExtensionDownloadStats(extensionName)
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 645, in getExtensionDownloadStats
    rev_downloads = self.getExtensionSlicerRevisionAndDownloads(url, extensionName)
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 584, in getExtensionSlicerRevisionAndDownloads
    self.setStatus("Retrieving package info {0}/{1} for extension {2} - Error: {3} - ".format(idx+1, len(all_itemids), extensionName, str(e)))
UnboundLocalError: local variable 'e' referenced before assignment

```

---

<div class="post-metadata">

**Author:** ![muratmaga](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/muratmaga/32/3622_2.png) [@muratmaga](https://discourse.slicer.org/u/muratmaga)\
**Post date:** [September 23, 2021, 6:23pm UTC](https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819/2 "2021-09-23T18:23:03Z")

</div>

@jcfr  
To add to the report same thing is happening with the preview as well:

```auto
Switch to module: "ExtensionStats"
Traceback (most recent call last):
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 579, in getExtensionSlicerRevisionAndDownloads
    item_rev_downloads[itemid] = [self.getItemById(url, itemid)['download'], slicerrevision]
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 553, in getItemById
    return self._call_midas_url(url, data)
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 515, in _call_midas_url
    response_data = response_dict['data']
KeyError: 'data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 153, in onApplyButton
    release_downloads = self.logic.getExtensionDownloadStats(extensionName)
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 645, in getExtensionDownloadStats
    rev_downloads = self.getExtensionSlicerRevisionAndDownloads(url, extensionName)
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 584, in getExtensionSlicerRevisionAndDownloads
    self.setStatus("Retrieving package info {0}/{1} for extension {2} - Error: {3} - ".format(idx+1, len(all_itemids), extensionName, str(e)))
UnboundLocalError: local variable 'e' referenced before assignment

```

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [September 24, 2021, 5:07am UTC](https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819/3 "2021-09-24T05:07:01Z")

</div>

The Midas server does not exist anymore.

You can get download counts in a single json file from the new extensions server: [SPKC](https://slicer-packages.kitware.com/#collection/5f4474d0e1d8c75dfc70547e/folder/5f4474d0e1d8c75dfc705484)

We also plan to develop a tool to get more detailed statistics from web server logs, which will work retrospectively, so no download statistics information will be lost.

---

<div class="post-metadata">

**Author:** ![muratmaga](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/muratmaga/32/3622_2.png) [@muratmaga](https://discourse.slicer.org/u/muratmaga)\
**Post date:** [September 24, 2021, 5:24am UTC](https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819/4 "2021-09-24T05:24:00Z")

</div>

Are the numbers in the JSONs counts since the new extension manager? They kind of look low (at least for SlicerMorph).

---

<div class="post-metadata">

**Author:** ![muratmaga](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/muratmaga/32/3622_2.png) [@muratmaga](https://discourse.slicer.org/u/muratmaga)\
**Post date:** [October 25, 2021, 5:27pm UTC](https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819/5 "2021-10-25T17:27:15Z")

</div>

Is there a direct URL to obtain the JSON as a file?

The output cannot be saved, but only copy/pasted.

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [October 26, 2021, 3:44am UTC](https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819/6 "2021-10-26T03:44:54Z")

</div>

I would be interested in this, too. It is quite inconvenient to copy-paste that long text.

Also, it would be important to get a snapshot of the last extension statistics of the old server (that is created by DeveloperToolsForExtensions).

@jcfr can you help with these?

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [October 26, 2021, 2:41pm UTC](https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819/7 "2021-10-26T14:41:02Z")

</div>

Current stats can be retrieved using this REST end point:

[https://slicer-packages.kitware.com/api/v1#!/app/app\_getDownloadStats](https://slicer-packages.kitware.com/api/v1#!/app/app_getDownloadStats)

For example:

```auto
https://slicer-packages.kitware.com/api/v1/app/5f4474d0e1d8c75dfc705482/downloadstats

```

where `5f4474d0e1d8c75dfc705482` corresponds to the application id (aka `folderId` in Girder terminology):

 ![image](https://us1.discourse-cdn.com/flex002/uploads/slicer/original/3X/9/f/9f325f7d6b655f1599ae9dea7bd95a9490c124af.png)

---

<div class="post-metadata">

**Author:** ![muratmaga](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/muratmaga/32/3622_2.png) [@muratmaga](https://discourse.slicer.org/u/muratmaga)\
**Post date:** [October 26, 2021, 6:15pm UTC](https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819/8 "2021-10-26T18:15:54Z")

</div>

Thanks JC. I parsed the output from the link you provided and the numbers are very different from what I used to get out of Extension Download Statistics tool, even though it includes historical revisions (not just the 4.13 revisions) for SlicerMorph.

This is important for us, and I suspect other grant supported projects. Would it be possible to reconcile this with the older statistics?

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [October 26, 2021, 7:51pm UTC](https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819/9 "2021-10-26T19:51:27Z")

</div>

I’ve updated the ExtensionStats module in SlicerDeveloperToolsForExtensions:

- it retrieves download counts from Girder
- it loads Midas statistics from a CSV file and adds it to the numbers provided by Girder

The Midas statistics is collected in May 2021, therefore we miss about 3 months of downloads. If @jcfr can retrieve the latest snapshot from the old server and updates the [Midas csv file](https://github.com/Slicer/SlicerDeveloperToolsForExtensions/blob/master/ExtensionStats/Resources/ExtensionsDownloadStats-20210521.csv) then we should have accurate download counts.

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [October 26, 2021, 7:55pm UTC](https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819/10 "2021-10-26T19:55:21Z")

</div>

By the way, getting full statistics on all extensions is much faster now. **Using the old server it took 6-8 hours, now it takes about 2 seconds**. The main difference is that the new server stores the statistics as a single file that can be retrieved using one query, while on Midas we had to submit thousands of queries (one for each revision of each extension).

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [October 27, 2021, 1:57am UTC](https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819/11 "2021-10-27T01:57:19Z")

</div>

> [@lassoan](#):
>
> If @jcfr can retrieve the latest snapshot from the old server and updates the [Midas csv file](https://github.com/Slicer/SlicerDeveloperToolsForExtensions/blob/master/ExtensionStats/Resources/ExtensionsDownloadStats-20210521.csv) then we should have accurate download counts.

The script is currently running and I expect it will complete in few hours. I will then upload it to [https://github.com/Slicer/SlicerDeveloperToolsForExtensions/tree/master/ExtensionStats/Resources](https://github.com/Slicer/SlicerDeveloperToolsForExtensions/tree/master/ExtensionStats/Resources)

---

<div class="post-metadata">

**Author:** ![jcfr](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/jcfr/32/17825_2.png) [@jcfr](https://discourse.slicer.org/u/jcfr)\
**Post date:** [October 27, 2021, 4:31pm UTC](https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819/12 "2021-10-27T16:31:58Z")

</div>

> [@lassoan](#):
>
> The Midas statistics is collected in May 2021, therefore we miss about 3 months of downloads.

Historical extension download stats have been published.

> <https://github.com/Slicer/SlicerDeveloperToolsForExtensions/pull/18>
>
> See https://discourse.slicer.org/t/extension-download-statistics-with-the-new-ba…ckend/19819

---

<div class="post-metadata">

**Author:** ![muratmaga](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/muratmaga/32/3622_2.png) [@muratmaga](https://discourse.slicer.org/u/muratmaga)\
**Post date:** [October 27, 2021, 5:47pm UTC](https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819/13 "2021-10-27T17:47:45Z")

</div>

> [@lassoan](#):
>
> I’ve updated the ExtensionStats module in SlicerDeveloperToolsForExtensions:

I works great and really fast, thank you Andras (and JC as well).

There is a minor issue that table copy doesn’t function

```auto
Traceback (most recent call last):
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.13.0-2021-10-26/NA-MIC/Extensions-30334/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 144, in copyTableToClipboard
    tableText += table.GetColumn(columnIndex).GetValue(rowIndex)
TypeError: must be str, not int

```

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [October 27, 2021, 9:22pm UTC](https://discourse.slicer.org/t/extension-download-statistics-with-the-new-backend/19819/14 "2021-10-27T21:22:49Z")

</div>

> [@muratmaga](#):
>
> There is a minor issue that table copy doesn’t function

Good catch. I’ve fixed it now.