Rotation in screen capture module specifies rotation axes relative to the current camera orientation. You can choose between the two meaningful rotation axes (vertical and horizontal - yaw and pitch). Roll would not be useful, as it would not reveal anything of the scene content (it would just rotate around the camera’s “up” vector).
You can make more complex camera animations by using Endoscopy module. You may also find Animator module (in SlicerMorph extension) useful. For completely custom camera or node movements, you can apply a transform or node you want to move and adjust the transformation matrix in a Python script.