Langchain: Query the complete 3D Slicer documentation, script repository and FAQ (PDF and HTML) with OpenAI LLM

This test implementation enables reading the complete 3D Slicer documentation, including the script repository, from a PDF file, and making it available for queries in an OpenAI LLM using Langchain.
The functionality is similar to ChatGPT, but the source is simply this single PDF document.

You’ll need to make a copy of the 3D Slicer readthedocs documentation as PDF (or modify the code to scrape it directly) and place it in one of your Google drive folders. In addition, you will need to add your OpenAI secret key to make this functional. It works great.

My highly intelligent prompt engineering :slight_smile: includes

prompts = [
          'What is 3D Slicer?',
          'Is 3D Slicer FDA-approved?',
          'Who is Ron Kirkinis?',
          'Who is Andras Lasso?',
          'Who is Steve Pieper?',
          'Who created 3D Slicer?',
          'What are the main use-cases for 3D Slicer?',
          'Does 3D Slicer support AI?',
          'What is an extension?',
           
          'How can I clone a node? Please provide Python code.',

          'please summarize this document',
          'please summarize chapters 1-12 of this document',
          'please summarize chapter 1 of this document',
          'please summarize chapter 2 of this document',
          'please summarize chapter 3 of this document',
          'please summarize chapter 4 of this document',
          'please summarize chapter 5 of this document',
          'please summarize chapter 6 of this document',
          'please summarize chapter 7 of this document',
          'please summarize chapter 8 of this document',
          'please summarize chapter 9 of this document',
          'please summarize chapter 10 of this document',
          'please summarize chapter 11 of this document',
          'please summarize chapter 12 of this document',
           
           ]

Here is the Colab notebook, please make your copies for testing.

3 Likes

The answers to my intelliprompts, summarizing the PDF and providing details:

The answer to ‘What is 3D Slicer?’ is:
3D Slicer is a software application for visualization and analysis of medical image computing datasets. All commonly used data sets are supported, such as images, segmentations, surfaces, annotations, transformations, etc., in 2D, 3D, and 4D. Visualization is available on desktop and in virtual reality. Analysis includes segmentation, registration, and various quantifications. It is also a research software platform, which allows researchers to quickly develop and evaluate new methods and distribute them to clinical users. 3D Slicer is completely free and there are no restrictions on how it is used, however, it is not approved for clinical use and the distributed application is intended for research use.

The answer to ‘Is 3D Slicer FDA-approved?’ is:
No, 3D Slicer is not FDA-approved. It is not intended for clinical use and is only intended for research use.

The answer to ‘Who is Ron Kirkinis?’ is:
Ron Kirkinis is one of the major contributors to the 3D Slicer core effort. He is listed as the Principal Investigator in the Acknowledgments section of the 3D Slicer Documentation.

The answer to ‘Who is Andras Lasso?’ is:
Andras Lasso is a benevolent dictator for life in the 3D Slicer community. He can integrate changes to keep the platform healthy and help interpret or address conflict related to the contribution guidelines.

The answer to ‘Who is Steve Pieper?’ is:
Steve Pieper is a Chief Architect for 3D Slicer. He is a major contributor to the core engineering of Slicer and is part of the Slicer team at Isomics. He is also an Associate Professor of Radiology at Harvard Medical School.

The answer to ‘Who created 3D Slicer?’ is:
3D Slicer was created by a international community of scientists from multiple fields, including engineering and biomedicine. Major contributors include Ron Kikinis, Steve Pieper, Jean-Christophe Fillion-Robin, Nicole Aucoin, Stephen Aylward, Andrey Fedorov, Noby Hata, Hans Johnson, Tina Kapur, Gabor Fichtinger, Andras Lasso, Csaba Pinter, Jim Miller, Sonia Pujol, Junichi Tokuda, Lauren O’Donnell, Andinet Enquobahrie, and Beatriz Paniagua. Groups contributing to the core engineering of Slicer in a major way include SPL, Isomics, Kitware, the University of Iowa, GE, and the Perk Lab, Queen’s University.

The answer to ‘What are the main use-cases for 3D Slicer?’ is:
3D Slicer is used for visualization and analysis of medical image computing datasets, prototyping and releasing products, and visualization of DICOM datasets and multi-frame datasets. It is also used for exploration of multivolume data, and to display CT or MRI volumes.

The answer to ‘Does 3D Slicer support AI?’ is:
No, 3D Slicer does not have any built-in support for AI.

The answer to ‘What is an extension?’ is:
An extension is a delivery package bundling together one or more 3DSlicer modules that can be added to 3DSlicer by installing the extension.

The answer to ‘How can I clone a node? Please provide Python code.’ is:

# Get a node from SampleData that we will clone

import SampleData

nodeToClone = SampleData.SampleDataLogic().downloadMRHead()

# Clone the node

shNode = slicer.vtkMRMLSubjectHierarchyNode.GetSubjectHierarchyNode(slicer.mrmlScene)

itemIDToClone = shNode.GetItemByDataNode(nodeToClone)

clonedItemID = slicer.modules.subjecthierarchy.logic().CloneSubjectHierarchyItem(shNode, itemIDToClone)

clonedNode = shNode.GetItemDataNode(clonedItemID)

The answer to ‘please summarize this document’ is:
This document is 3D Slicer Documentation, which contains two chapters. Chapter 1 is about 3D Slicer, and Chapter 2 covers getting started.

The answer to ‘please summarize chapters 1-12 of this document’ is:
Chapters 1-12 of this document provide information about 3D Slicer, such as how to get started and how to use the indices and tables.

The answer to ‘please summarize chapter 1 of this document’ is:
Chapter 1 of the 3D Slicer Documentation is entitled “About 3D Slicer” and gives an overview of the software and its features.

The answer to ‘please summarize chapter 2 of this document’ is:
Chapter 2 of the 3D Slicer Documentation is titled ‘Getting Started’ and provides an introduction to the 3D Slicer software. It covers topics related to how to install and use the software.

The answer to ‘please summarize chapter 3 of this document’ is:
Chapter 3 of the 3D Slicer Documentation covers how to get help with 3D Slicer.
It outlines different types of help, including online help, tutorials, mailing lists, and user groups.

The answer to ‘please summarize chapter 4 of this document’ is:
Chapter 4 of the 3D Slicer Documentation is about the User Interface.
It explains that data sets are stored in a data repository called the “scene” and each data set is
represented in the scene as a node. It also explains that Slicer contains over 100 built-in modules and
additional modules can be installed by using the Extensions Manager.

The answer to ‘please summarize chapter 5 of this document’ is:
Chapter 5 of the 3D Slicer Documentation covers data loading and saving, giving instructions on how to import, export, and manipulate data in 3D Slicer.

The answer to ‘please summarize chapter 6 of this document’ is:
Chapter 6 of the 3D Slicer Documentation is about Image Segmentation. It covers topics related to the segmentation of images using 3D Slicer.

The answer to ‘please summarize chapter 7 of this document’ is:
Chapter 7 of the 3D Slicer Documentation is titled “Registration” and covers topics related to registering images.

The answer to ‘please summarize chapter 8 of this document’ is:
Chapter 8 of this document covers the DICOM module of 3D Slicer. It provides an overview of the module and its features, which include importing, exporting, and network transferring of DICOM data, as well as querying, retrieving, sending, and receiving data sets. The module also supports DIMSE and DICOMweb networking protocols. It is funded by the National Institutes of Health and a Cancer Care Ontario research unit.

The answer to ‘please summarize chapter 9 of this document’ is:
I don’t know.

The answer to ‘please summarize chapter 10 of this document’ is:
Chapter 10 of the 3D Slicer Documentation outlines the various application settings available to the user.

The answer to ‘please summarize chapter 11 of this document’ is:
Chapter 11 of this document is titled Advanced Topics. It covers topics such as using 3D Slicer in research, developing Slicer modules, and creating Slicer packages.

The answer to ‘please summarize chapter 12 of this document’ is:
Chapter 12 of this document is about Indices and Tables. It includes genindex, modindex, and search as topics. It also has a Python Module Index and an Index.

2 Likes

Very nice, thanks for digging into this. It has huge potential, especially if the model could also incorporate all the discussions on this forum.

Can it also provide links to the source?

It provided some misleading information with great confidence, which may be misleading or even hurt us:

The answer to ‘Who is Steve Pieper?’ is: Steve Pieper is a Chief Architect for 3D Slicer. … He is also an Associate Professor of Radiology at Harvard Medical School.

The answer to ‘Does 3D Slicer support AI?’ is: No, 3D Slicer does not have any built-in support for AI.

I expect that soon chat interface will be a standard feature on websites and forums, replacing full-text search (in fact, it has been going on for a while, but the earlier free-text query implementations were so poor that they did not get much attention). Discourse already has some experimental plugin for allowing an AI bot to answer questions. This will of course make more sense when it will use a model that gives more weight to content that it finds on the forum.

The only source was the PDF document imported, but maybe “enriched” with OpenAI contents, so we’ll need to reconstruct why that happens.

You could import several different sources. See here:

1 Like

I have updated my langchain-based OPENAI LLM Jupyter notebook to version 2 by integrating an agent which queries:

  • 3D Slicer PDF
  • Script repository (simple Webpage)
  • FAQ (simple Webpage)

Select “Open with Colab notebook”
You just need to put in your OpenAI API key.

2 Likes

The latest 3D Slicer documentation in PDF format can be found when you click here:

https://slicer.readthedocs.io/en/latest/

There is a new version of the Colab Notebook querying 3D Slicer documentation available.

https://drive.google.com/drive/folders/1doNHe1nCkfIK7zhaIbhcfrIfe6eu4Nbm?usp=sharing

There is interaction with:

3D Slicer PDF from readthedocs
Script Repository
3D Slicer FAQ
3D Slicer as a platform for the automatic segmentation of real-time MRI data

A simple gradio interface is provided. And we have chat memory (follow up questions) :slight_smile:

1 Like

Also presented are the astounding 3D Slicer citation counts (as of the analysis @lassoan).
Could we have an automatic page for this analysis ?

User:
Please comment on the number of citations of 3D Slicer between 2006 and 2022.

Could anybody point me to a way to query all 3D Slicer extensions available (short descriptions)?

You can get information on all extensions for all Slicer versions from the ExtensionsIndex repository. You can get packages and download counts from the Extensions Catalog (Girder server) - see ExtensionStats. An easy way to download all extension repositories is to build the ExtensionsIndex.

1 Like