# Question about robotic simulation in VR

**URL:** <https://discourse.slicer.org/t/question-about-robotic-simulation-in-vr/21871>\
**Category:** Support\
**Tags:** virtual-reality, robotics\
**Created:** [February 9, 2022, 3:07pm UTC](https://discourse.slicer.org/t/question-about-robotic-simulation-in-vr/21871 "2022-02-09T15:07:25Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![user5](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/user5/32/13217_2.png) [@user5](https://discourse.slicer.org/u/user5)\
**Post date:** [February 9, 2022, 3:07pm UTC](https://discourse.slicer.org/t/question-about-robotic-simulation-in-vr/21871/1 "2022-02-09T15:07:25Z")

</div>

I am trying to use a robotic model to do needle insertion. It may include robotic simulation and transform. I would like to show and operate in SlicerVR which use the VR controller to control the robotic simulation and transform.  
Is SlicerVR able to perform these tasks?

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [February 9, 2022, 3:11pm UTC](https://discourse.slicer.org/t/question-about-robotic-simulation-in-vr/21871/2 "2022-02-09T15:11:18Z")

</div>

Yes, that should all be possible. SlicerVR shows you everything that’s in the 3D views, so you can set up OpenIGTLink, links to ROS, etc. For sure some amount of customization would be needed to make it more usable, but this should all be doable in python.

---

<div class="post-metadata">

**Author:** ![user5](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/user5/32/13217_2.png) [@user5](https://discourse.slicer.org/u/user5)\
**Post date:** [February 10, 2022, 2:57pm UTC](https://discourse.slicer.org/t/question-about-robotic-simulation-in-vr/21871/3 "2022-02-10T14:57:41Z")

</div>

If I want to make customization function in Python for SlicerVR, which file I should go to modify?

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [February 10, 2022, 3:03pm UTC](https://discourse.slicer.org/t/question-about-robotic-simulation-in-vr/21871/4 "2022-02-10T15:03:55Z")

</div>

It depends on what you are trying to accomplish with your customization. You’ll need to read the code to figure out.

---

<div class="post-metadata">

**Author:** ![user5](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/user5/32/13217_2.png) [@user5](https://discourse.slicer.org/u/user5)\
**Post date:** [February 10, 2022, 3:47pm UTC](https://discourse.slicer.org/t/question-about-robotic-simulation-in-vr/21871/5 "2022-02-10T15:47:33Z")

</div>

Which folder contain those code?  
Also, I would like to ask that does SlicerVR has any tutorial for beginner development?

---

<div class="post-metadata">

**Author:** ![pieper](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/pieper/32/8_2.png) [@pieper](https://discourse.slicer.org/u/pieper)\
**Post date:** [February 10, 2022, 4:22pm UTC](https://discourse.slicer.org/t/question-about-robotic-simulation-in-vr/21871/6 "2022-02-10T16:22:15Z")

</div>

This material is all very new, so no developer tutorials that I know of.

The VR code is here:

> **[SlicerVirtualReality/VirtualReality at master ·...](https://github.com/KitwareMedical/SlicerVirtualReality/tree/master/VirtualReality)**
>
> master/VirtualReality

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [February 10, 2022, 7:42pm UTC](https://discourse.slicer.org/t/question-about-robotic-simulation-in-vr/21871/7 "2022-02-10T19:42:59Z")

</div>

The great thing about SlicerVirtualReality is that behaviors in the scene can be defined using Python-scripting the same way in virtual reality as you would do it for the regular desktop experience. Typically the user just moves objects around and you add observers to those transforms (or directly to the controllers pose), and then you react to those changes any way you want (update some other transforms, show/hide object, change any other properties of any objects in the scene). You can find lots of code snippets for manipulating objects (that are all usable in virtual reality) in the [script repository](https://slicer.readthedocs.io/en/latest/developer_guide/script_repository.html).

Very soon (or maybe already) you can display Qt widgets in the immersive view, which will make it trivial to add buttons, sliders, etc. that can be manipulated without peeking out of the headset.

If you have questions about how to implement a particular behavior then let us know and we can give more specific instructions.

---

<div class="post-metadata">

**Author:** ![user5](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/user5/32/13217_2.png) [@user5](https://discourse.slicer.org/u/user5)\
**Post date:** [February 11, 2022, 6:00pm UTC](https://discourse.slicer.org/t/question-about-robotic-simulation-in-vr/21871/9 "2022-02-11T18:00:51Z")

</div>

I find that the file format in GitHub is .cxx which is different with the file format in 3D Slicer’s extension root folder which is .pyd. Can I directly use .cxx file to replace .pyd file?

---

<div class="post-metadata">

**Author:** ![lassoan](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/lassoan/32/13_2.png) [@lassoan](https://discourse.slicer.org/u/lassoan)\
**Post date:** [February 11, 2022, 7:19pm UTC](https://discourse.slicer.org/t/question-about-robotic-simulation-in-vr/21871/10 "2022-02-11T19:19:27Z")

</div>

You need to build the source cxx files (implemented in C++) to generate executable, Python-wrapped code.

---

<div class="post-metadata">

**Author:** ![user5](https://sea2.discourse-cdn.com/flex002/user_avatar/discourse.slicer.org/user5/32/13217_2.png) [@user5](https://discourse.slicer.org/u/user5)\
**Post date:** [February 16, 2022, 3:27am UTC](https://discourse.slicer.org/t/question-about-robotic-simulation-in-vr/21871/11 "2022-02-16T03:27:40Z")

</div>

I am already trying to modify the SlicerVR extension in visual studio 2017. I follow this link to build .pyd file.

> [Building a C++ extension for Python 3.5 on Windows - YouTube](https://www.youtube.com/watch?v=9q-LHP7cMfg&ab_channel=PyCon2014)

Since visual studio 2017 does not have VTK Modules, I have followed this link to import VTK Modules into visual studio 2017.

> [Day039 — How to include and link VTK, Boost and GLM in Visual Studio 2017 on Windows | by Jacky Tsang | Medium](https://jacky-ttt.medium.com/day039-how-to-include-and-link-vtk-boost-and-glm-in-visual-studio-2017-on-windows-2c611d63dd09)

But the build is failed.

 ![擷取11](https://us1.discourse-cdn.com/flex002/uploads/slicer/original/3X/a/b/abe9424c2a193774574687e8600e3798a2e6ec90.png)

Is it has anything that I am missing to add?