Panoramic image is useful if you are inside the patient and want to render the surroundings, which only happens in endoscopic AR, but even then, typically you want to overlay tumors or trajectories and not an entire 360 box.
What would you like to display over the patient? Burr hole position, drill/needle insertion trajectory, vertebral models, …? They are all just take up a very small part of the field of view, so panoramic 360 view is not relevant. Normally you add a tracking reference on the patient and display a 3D model anchored relative to that reference marker (but if you don’t need robust registration then markerless tracking works, too).
You can try the open-source surgical AR system developed by Etienne Leger (it integrates with Slicer, I think Slicer or maybe Ibis renders the overlay and it is sent to the AR tablet using OpenIGTLink): https://link.springer.com/article/10.1007/s11548-020-02155-6
There are some other earlier efforts related to this - some of the links may still be useful: https://www.slicer.org/wiki/Documentation/Labs/Augmented_Reality_and_Virtual_Reality_support