Implementing PyTest-qt

Development
1

Are we able to implement PyTest-qt into our custom slicer app? I don’t know all of the discrepancies associated with the licensing but I would like to implement it to be able to run GUI based tests.

2

PyTest-qt summary:

The main usage is to use the qtbot fixture, responsible for handling qApp creation as needed and provides methods to simulate user interaction, like key presses and mouse clicks:

@jcfr @lassoan @pieper Do you have any thoughts about python testing of Slicer modules and their usage of simulated user interaction?

How does QtTesting fit into current recommendations?

Recording of macro using QtTesting should be considered experimental.
We suggest to implement workflow automation through Python scripts operating at a lower level by changing properties of MRML nodes and calling module logic functions

One of the above links has the above header. Does this mean it is not recommended to do this GUI level type of testing that uses simulated clicks and interactions? Slicer core appears to utilize some QtTesting in qMRMLCheckableNodeComboBoxEventTranslatorPlayerTest1.xml. I assume this is because it is a qMRML QObject rather than say a Slicer module.

Are Self-tests the Slicer recommended way for scripted loadable module python testing? Is the clickAndDrag method the recommended manner for synthetic input into Slicer modules, or is something like PyTest-qt as mentioned in the original post a possibility for Slicer scripted module testing?

1 Like
3

I would say yes. We have had good luck with the SelfTests and we are planning to extend it as part of our CZI project for automating tutorial localization, where we will support running the tests and generating screenshots in different languages.

The SelfTests probably supports most of what the PyTest-qt project provides, but if there are additional features we could look at either adding them or using PyTest-qt with PythonQt, which may not be too hard.

I agree with your statement @jamesobutler that the QtTesting infrastructure for recording and replaying events is probably more suited to unit testing of widgets; SelfTests have proven easier for complex operations because they are debuggable scripts as opposed to xml files of mouse coordinates and button clicks.

1 Like
4

Hi Steve, thanks for the response. I think I’m still a bit lost. I’m not sure what the difference is between SelfTests and unittests?

Futhermore, I am currently utilizing the ScripedLoadableModuleTest for testing, which (from my understanding) is based on unittest. I have some test cases, but they tend to be very lengthy and dont simulate user interactions very well, which I think is what I am looking for. Are you saying there is a way to do that with SelfTests?

5

Yes, there’s not really a crisp distinction but we use the term unit test to mean checking one widget or API in isolation as part of the development and build process where the SelfTest refers to scripting the full application, specifically the SelfTest means that it’s part of the installed application, meaning that any user could run a SelfTest, e.g. to test their own hardware or OS without needing any additional tooling.

In Slicer we have some modules that only exist to serve as SelfTests (you can look at these for examples). These are hidden by default but are turned on in Developer Mode. But as you’ve seen also any scripted module can include a class that inherits from ScriptedLoadableModuleTest to declare a set of tests that can be at whatever level the developer is able to provide. Both types are integrated with the testing procedures on the factory builds and contribute results to the dashboard and use the python unittest module internally. These can also be defined for extensions and results are posted to the dashboard.

Anyway, yes, with clickAndDrag and other methods you can script the user interface to simulate interactions with vtk render windows. For Qt windows we typically call clicked() or other signals directly.

1 Like
6

@hannahm Can you provide an example of a type of interaction that you might want to simulate? How were you thinking of doing this in PyTest-Qt and does the existing slicer.util.clickAndDrag or qpushbutton.click() type methods that exist in Slicer cover the same simulating user behavior?

7

I may be mistaken, but when using .clicked() or .click(), it doesn’t act exactly as a person would. For example, when clicking a button or editing a text box, the focus isn’t set on that widget, which may affect the behavior of the rest of the app.

I ran the following test on slicer 5.0.3 this morning

  1. pop the console window out of the main window
  2. load in a volume
  3. create a segmentation on any of the slices using draw
  4. confirm that the focus is on that slice viewer using slicer.util.mainWindow().focusWidget(). (I toggled the visibility to fully confirm)
  5. find the AddSegmentationButton (slicer.util.mainWindow().findChild(qt.QPushButton, 'AddSegmentButton')) and use .click() on it.
  6. Confirm that the mainWindows focusWidget is still the slice widget
  7. Repeat steps 5 and 6 but with .clicked() instead of .click()
  8. Manually click the ‘AddSegmentationButton’
  9. Confirm that the focusWidget of the mainWindow is now that push button

Additionally, writing the tests with the current SelfTests framework can be long and tedious to write and review and require developers to know exactly which flags are going to be set with clicked, which makes writing and reviewing less beginner friendly.

Edit:
Additionally x2, as far as I am aware, there is no way to test shortcut keys (ex, Ctrl+O to add new data). PyTest-qt seems to make that possible with the use of qtbot.keyClick()

I have also noticed many times while using SelfTests that quite a few tracebacks are logged that wouldnt be there had the test been manually run. I suspect this may be a similar issue and that the UI is not properly updating, although its just a suspicion I have.
For ex. ctkSliderWidget::setSingleStep() 0 is out of bounds. 0 10 1

8

So it appears the use of more Qt specific testing methods such as those in the QTest module is what you are looking for. It appears QTest is used in some Slicer C++ testing, but this module appears to not be included in the PythonQt wrapping? or maybe hidden in a namespace I haven’t found? It would likely only be included in Slicer local builds with Testing built.

9

There are definitely limitations to our current testing systems and any contributions are welcome! Anything that helps developers create more useful tests would be great.

1 Like
10

It should not be hard to expose QTest methods in Python:

  • Option A. Add QTest to PythonQt. You can ask PythonQt developers if they plan to add wrapping for QTest API this. If they don’t plan to do it then you can do it yourself (as it was done for adding QtMultimedia).
  • Option B. Add a Python-wrapped C++ tester class which calls QTest methods. Similarly how we added the qSlicerWebWidget to Slicer instead of adding the QWebengine to PythonQt.
1 Like
11

@lassoan

Is there a preferred method? I’m happy to do either

12

First try A then if that does not work out then do B.

1 Like
13

Improving PythonQt is definitely a good place to start. Note that the generator has some issues that may complicate things. Adding Slicer-specific subclasses may make sense in any case to handle things like VTK based views or application level flags like test mode or no-main-window mode. These would be qSlicer subjclasses like we have elsewhere in the code. So you may end up with A and some of B or just B.

2 Likes