Extension download statistics with the new backend

I tried to download the extension download statistics using the stable, and got the error below. This used to work, so I assume this is related to change in extension server.

Going forward, what is the plan on obtaining extension stats?

Switch to module:  "ExtensionStats"
Traceback (most recent call last):
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 579, in getExtensionSlicerRevisionAndDownloads
    item_rev_downloads[itemid] = [self.getItemById(url, itemid)['download'], slicerrevision]
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 553, in getItemById
    return self._call_midas_url(url, data)
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 515, in _call_midas_url
    response_data = response_dict['data']
KeyError: 'data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 153, in onApplyButton
    release_downloads = self.logic.getExtensionDownloadStats(extensionName)
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 645, in getExtensionDownloadStats
    rev_downloads = self.getExtensionSlicerRevisionAndDownloads(url, extensionName)
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 584, in getExtensionSlicerRevisionAndDownloads
    self.setStatus("Retrieving package info {0}/{1} for extension {2} - Error: {3} - ".format(idx+1, len(all_itemids), extensionName, str(e)))
UnboundLocalError: local variable 'e' referenced before assignment

@jcfr
To add to the report same thing is happening with the preview as well:

Switch to module:  "ExtensionStats"
Traceback (most recent call last):
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 579, in getExtensionSlicerRevisionAndDownloads
    item_rev_downloads[itemid] = [self.getItemById(url, itemid)['download'], slicerrevision]
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 553, in getItemById
    return self._call_midas_url(url, data)
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 515, in _call_midas_url
    response_data = response_dict['data']
KeyError: 'data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 153, in onApplyButton
    release_downloads = self.logic.getExtensionDownloadStats(extensionName)
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 645, in getExtensionDownloadStats
    rev_downloads = self.getExtensionSlicerRevisionAndDownloads(url, extensionName)
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 584, in getExtensionSlicerRevisionAndDownloads
    self.setStatus("Retrieving package info {0}/{1} for extension {2} - Error: {3} - ".format(idx+1, len(all_itemids), extensionName, str(e)))
UnboundLocalError: local variable 'e' referenced before assignment

The Midas server does not exist anymore.

You can get download counts in a single json file from the new extensions server: SPKC

We also plan to develop a tool to get more detailed statistics from web server logs, which will work retrospectively, so no download statistics information will be lost.

Are the numbers in the JSONs counts since the new extension manager? They kind of look low (at least for SlicerMorph).

Is there a direct URL to obtain the JSON as a file?

The output cannot be saved, but only copy/pasted.

I would be interested in this, too. It is quite inconvenient to copy-paste that long text.

Also, it would be important to get a snapshot of the last extension statistics of the old server (that is created by DeveloperToolsForExtensions).

@jcfr can you help with these?

Current stats can be retrieved using this REST end point:

https://slicer-packages.kitware.com/api/v1#!/app/app_getDownloadStats

For example:

https://slicer-packages.kitware.com/api/v1/app/5f4474d0e1d8c75dfc705482/downloadstats

where 5f4474d0e1d8c75dfc705482 corresponds to the application id (aka folderId in Girder terminology):

Thanks JC. I parsed the output from the link you provided and the numbers are very different from what I used to get out of Extension Download Statistics tool, even though it includes historical revisions (not just the 4.13 revisions) for SlicerMorph.

This is important for us, and I suspect other grant supported projects. Would it be possible to reconcile this with the older statistics?

I’ve updated the ExtensionStats module in SlicerDeveloperToolsForExtensions:

  • it retrieves download counts from Girder
  • it loads Midas statistics from a CSV file and adds it to the numbers provided by Girder

The Midas statistics is collected in May 2021, therefore we miss about 3 months of downloads. If @jcfr can retrieve the latest snapshot from the old server and updates the Midas csv file then we should have accurate download counts.

2 Likes

By the way, getting full statistics on all extensions is much faster now. Using the old server it took 6-8 hours, now it takes about 2 seconds. The main difference is that the new server stores the statistics as a single file that can be retrieved using one query, while on Midas we had to submit thousands of queries (one for each revision of each extension).

3 Likes

The script is currently running and I expect it will complete in few hours. I will then upload it to https://github.com/Slicer/SlicerDeveloperToolsForExtensions/tree/master/ExtensionStats/Resources

2 Likes

Historical extension download stats have been published.

I works great and really fast, thank you Andras (and JC as well).

There is a minor issue that table copy doesn’t function

Traceback (most recent call last):
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.13.0-2021-10-26/NA-MIC/Extensions-30334/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 144, in copyTableToClipboard
    tableText += table.GetColumn(columnIndex).GetValue(rowIndex)
TypeError: must be str, not int

Good catch. I’ve fixed it now.

1 Like