Transforms module only uses 4x4 homogeneous transformation matrix for linear transforms. You can convert between these representations easily, for example by using vtk.vtkMath.QuaternionToMatrix3x3() and vtk.vtkMath.Matrix3x3ToQuaternion() methods.
Rotation angles in the Transforms module GUI simply allow incremental rotation along LR, AP, IS axes. The values displayed next to the sliders only show angle of the last rotation, therefore the displayed values do not represent the total rotation. See some more explanation at the end of this section in Transforms module documentation.