Extension download statistics with the new backend

muratmaga · September 23, 2021, 5:11am

I tried to download the extension download statistics using the stable, and got the error below. This used to work, so I assume this is related to change in extension server.

Going forward, what is the plan on obtaining extension stats?

Switch to module:  "ExtensionStats"
Traceback (most recent call last):
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 579, in getExtensionSlicerRevisionAndDownloads
    item_rev_downloads[itemid] = [self.getItemById(url, itemid)['download'], slicerrevision]
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 553, in getItemById
    return self._call_midas_url(url, data)
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 515, in _call_midas_url
    response_data = response_dict['data']
KeyError: 'data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 153, in onApplyButton
    release_downloads = self.logic.getExtensionDownloadStats(extensionName)
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 645, in getExtensionDownloadStats
    rev_downloads = self.getExtensionSlicerRevisionAndDownloads(url, extensionName)
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.11.20210226/NA-MIC/Extensions-29738/DeveloperToolsForExtensions/lib/Slicer-4.11/qt-scripted-modules/ExtensionStats.py", line 584, in getExtensionSlicerRevisionAndDownloads
    self.setStatus("Retrieving package info {0}/{1} for extension {2} - Error: {3} - ".format(idx+1, len(all_itemids), extensionName, str(e)))
UnboundLocalError: local variable 'e' referenced before assignment

muratmaga · September 23, 2021, 6:23pm

@jcfr
To add to the report same thing is happening with the preview as well:

Switch to module:  "ExtensionStats"
Traceback (most recent call last):
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 579, in getExtensionSlicerRevisionAndDownloads
    item_rev_downloads[itemid] = [self.getItemById(url, itemid)['download'], slicerrevision]
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 553, in getItemById
    return self._call_midas_url(url, data)
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 515, in _call_midas_url
    response_data = response_dict['data']
KeyError: 'data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 153, in onApplyButton
    release_downloads = self.logic.getExtensionDownloadStats(extensionName)
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 645, in getExtensionDownloadStats
    rev_downloads = self.getExtensionSlicerRevisionAndDownloads(url, extensionName)
  File "C:/Users/amaga/AppData/Local/NA-MIC/Slicer 4.13.0-2021-09-20/NA-MIC/Extensions-30230/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 584, in getExtensionSlicerRevisionAndDownloads
    self.setStatus("Retrieving package info {0}/{1} for extension {2} - Error: {3} - ".format(idx+1, len(all_itemids), extensionName, str(e)))
UnboundLocalError: local variable 'e' referenced before assignment

lassoan · September 24, 2021, 5:07am

The Midas server does not exist anymore.

You can get download counts in a single json file from the new extensions server: SPKC

We also plan to develop a tool to get more detailed statistics from web server logs, which will work retrospectively, so no download statistics information will be lost.

muratmaga · September 24, 2021, 5:24am

Are the numbers in the JSONs counts since the new extension manager? They kind of look low (at least for SlicerMorph).

muratmaga · October 25, 2021, 5:27pm

Is there a direct URL to obtain the JSON as a file?

The output cannot be saved, but only copy/pasted.

lassoan · October 26, 2021, 3:44am

I would be interested in this, too. It is quite inconvenient to copy-paste that long text.

Also, it would be important to get a snapshot of the last extension statistics of the old server (that is created by DeveloperToolsForExtensions).

@jcfr can you help with these?

jcfr · October 26, 2021, 2:41pm

Current stats can be retrieved using this REST end point:

https://slicer-packages.kitware.com/api/v1#!/app/app_getDownloadStats

For example:

https://slicer-packages.kitware.com/api/v1/app/5f4474d0e1d8c75dfc705482/downloadstats

where 5f4474d0e1d8c75dfc705482 corresponds to the application id (aka folderId in Girder terminology):

muratmaga · October 26, 2021, 6:15pm

Thanks JC. I parsed the output from the link you provided and the numbers are very different from what I used to get out of Extension Download Statistics tool, even though it includes historical revisions (not just the 4.13 revisions) for SlicerMorph.

This is important for us, and I suspect other grant supported projects. Would it be possible to reconcile this with the older statistics?

lassoan · October 26, 2021, 7:51pm

I’ve updated the ExtensionStats module in SlicerDeveloperToolsForExtensions:

it retrieves download counts from Girder
it loads Midas statistics from a CSV file and adds it to the numbers provided by Girder

The Midas statistics is collected in May 2021, therefore we miss about 3 months of downloads. If @jcfr can retrieve the latest snapshot from the old server and updates the Midas csv file then we should have accurate download counts.

lassoan · October 26, 2021, 7:55pm

By the way, getting full statistics on all extensions is much faster now. Using the old server it took 6-8 hours, now it takes about 2 seconds. The main difference is that the new server stores the statistics as a single file that can be retrieved using one query, while on Midas we had to submit thousands of queries (one for each revision of each extension).

jcfr · October 27, 2021, 1:57am

The script is currently running and I expect it will complete in few hours. I will then upload it to https://github.com/Slicer/SlicerDeveloperToolsForExtensions/tree/master/ExtensionStats/Resources

jcfr · October 27, 2021, 4:31pm

Historical extension download stats have been published.

muratmaga · October 27, 2021, 5:47pm

I works great and really fast, thank you Andras (and JC as well).

There is a minor issue that table copy doesn’t function

Traceback (most recent call last):
  File "C:/Users/murat/AppData/Local/NA-MIC/Slicer 4.13.0-2021-10-26/NA-MIC/Extensions-30334/DeveloperToolsForExtensions/lib/Slicer-4.13/qt-scripted-modules/ExtensionStats.py", line 144, in copyTableToClipboard
    tableText += table.GetColumn(columnIndex).GetValue(rowIndex)
TypeError: must be str, not int

lassoan · October 27, 2021, 9:22pm

Good catch. I’ve fixed it now.

Topic		Replies	Views
Extension statistics module giving an error Support	1	168	August 16, 2020
Extension download statistics Development	3	476	June 7, 2019
Reliability of extension stats Support	7	235	October 16, 2023
Interpreting the results of ExtensionStats Support extensions-manager	4	551	July 11, 2017
Developer tools for extensions is failing for preview Support	2	256	January 11, 2022

Extension download statistics with the new backend

Related topics