Extension download statistics with the new backend

Support
1

I tried to download the extension download statistics using the stable, and got the error below. This used to work, so I assume this is related to change in extension server.

Going forward, what is the plan on obtaining extension stats?

Switch to module:  "ExtensionStats"
Traceback (most recent call last):
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 579, in getExtensionSlicerRevisionAndDownloads
    item_rev_downloads[itemid] = [self.getItemById(url, itemid)['download'], slicerrevision]
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 553, in getItemById
    return self._call_midas_url(url, data)
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 515, in _call_midas_url
    response_data = response_dict['data']
KeyError: 'data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 153, in onApplyButton
    release_downloads = self.logic.getExtensionDownloadStats(extensionName)
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 645, in getExtensionDownloadStats
    rev_downloads = self.getExtensionSlicerRevisionAndDownloads(url, extensionName)
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 584, in getExtensionSlicerRevisionAndDownloads
    self.setStatus("Retrieving package info {0}/{1} for extension {2} - Error: {3} - ".format(idx+1, len(all_itemids), extensionName, str(e)))
UnboundLocalError: local variable 'e' referenced before assignment
2

@jcfr
To add to the report same thing is happening with the preview as well:

Switch to module:  "ExtensionStats"
Traceback (most recent call last):
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 579, in getExtensionSlicerRevisionAndDownloads
    item_rev_downloads[itemid] = [self.getItemById(url, itemid)['download'], slicerrevision]
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 553, in getItemById
    return self._call_midas_url(url, data)
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 515, in _call_midas_url
    response_data = response_dict['data']
KeyError: 'data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 153, in onApplyButton
    release_downloads = self.logic.getExtensionDownloadStats(extensionName)
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 645, in getExtensionDownloadStats
    rev_downloads = self.getExtensionSlicerRevisionAndDownloads(url, extensionName)
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 584, in getExtensionSlicerRevisionAndDownloads
    self.setStatus("Retrieving package info {0}/{1} for extension {2} - Error: {3} - ".format(idx+1, len(all_itemids), extensionName, str(e)))
UnboundLocalError: local variable 'e' referenced before assignment
3

The Midas server does not exist anymore.

You can get download counts in a single json file from the new extensions server: SPKC

We also plan to develop a tool to get more detailed statistics from web server logs, which will work retrospectively, so no download statistics information will be lost.

4

Are the numbers in the JSONs counts since the new extension manager? They kind of look low (at least for SlicerMorph).

5

Is there a direct URL to obtain the JSON as a file?

The output cannot be saved, but only copy/pasted.

6

I would be interested in this, too. It is quite inconvenient to copy-paste that long text.

Also, it would be important to get a snapshot of the last extension statistics of the old server (that is created by DeveloperToolsForExtensions).

@jcfr can you help with these?

7

Current stats can be retrieved using this REST end point:

https://slicer-packages.kitware.com/api/v1#!/app/app_getDownloadStats

For example:

https://slicer-packages.kitware.com/api/v1/app/5f4474d0e1d8c75dfc705482/downloadstats

where 5f4474d0e1d8c75dfc705482 corresponds to the application id (aka folderId in Girder terminology):

image

8

Thanks JC. I parsed the output from the link you provided and the numbers are very different from what I used to get out of Extension Download Statistics tool, even though it includes historical revisions (not just the 4.13 revisions) for SlicerMorph.

This is important for us, and I suspect other grant supported projects. Would it be possible to reconcile this with the older statistics?

9

I’ve updated the ExtensionStats module in SlicerDeveloperToolsForExtensions:

  • it retrieves download counts from Girder
  • it loads Midas statistics from a CSV file and adds it to the numbers provided by Girder

The Midas statistics is collected in May 2021, therefore we miss about 3 months of downloads. If @jcfr can retrieve the latest snapshot from the old server and updates the Midas csv file then we should have accurate download counts.

2 Likes
10

By the way, getting full statistics on all extensions is much faster now. Using the old server it took 6-8 hours, now it takes about 2 seconds. The main difference is that the new server stores the statistics as a single file that can be retrieved using one query, while on Midas we had to submit thousands of queries (one for each revision of each extension).

3 Likes
11

The script is currently running and I expect it will complete in few hours. I will then upload it to https://github.com/Slicer/SlicerDeveloperToolsForExtensions/tree/master/ExtensionStats/Resources

2 Likes
12

Historical extension download stats have been published.

13

I works great and really fast, thank you Andras (and JC as well).

There is a minor issue that table copy doesn’t function

Traceback (most recent call last):
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.13.0-2021-10-26/NA-MIC/Extensions-30334/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 144, in copyTableToClipboard
    tableText += table.GetColumn(columnIndex).GetValue(rowIndex)
TypeError: must be str, not int
14

Good catch. I’ve fixed it now.

1 Like