It is not trivial at all how to provide your existing code as context for LLMs.
Cursor is a company currently that does just this and it is currently valued at $30 billion. For me, VS Code works just as well as Cursor. So, it is not surprising that code generation works better inside the VS Code IDE.
For creating and developing a full module, it may be harder to do better than in an IDE. To make the first steps easier it may be enough to provide users templates, tutorials, a good agentic coding manifest (see some information for example here).
However, it could make sense to have a module in Slicer for creating and running code snippets to automate simple tasks (load some data, do some processing, set up visualization, export results, etc.). The module could automatically provide all the necessary context (where to look for code snippets, what data is in the scene, etc.). To facilitate this, we could:
- Develop a set of useful high-level functions, which would implement all the tasks that users frequently ask. I would guess that a few hundred small functions would provide all the necessary building blocks. The list of functions and their documentation would be used as context for the code generation.
- Set up model context protocol in Slicer that would allow LLMs to verify themselves by testing code in Slicer. There was a post about it not too long ago: MCP-Slicer: 3D Slicer Model Context Protocol Integration. The main difficulty is that you would want to run the LLM in a sandbox or you would really need to be careful about the API you expose.