Document modules on readthedocs

I generate markdown code from the source tree, no html. I use mkdocs with
the material theme. Medics generates the html.

Sorry, mkdocs generates the html.

I see. In your case, the documentation toolchain is mkdocs and it outputs the html that is then pushed on GitHub.

Instead of maintaining a build machine yourself, you could have readthedocs taking care of running mkdocs for you. See http://docs.readthedocs.io/en/latest/builds.html#mkdocs

I’ll take a look at readthedocs and see if it adds value for my app.

How long has readthe docs been around and is it’s future as stable as
github?

Code base has been around for 7 years so far. See https://github.com/rtfd/readthedocs.org/graphs

Hosting has been around for 4 years.

From https://readthedocs.org/sustainability/ :

Read the Docs has grown substantially since its beginning as a weekend project.
Today, we:

    Serve over 20 million pages of documentation a month
    Have 10 servers and serve over 2 TB of documentation a month
    Host over 18,000 projects and support 25,000 users
    Are supported by two dedicated engineers

2 guys… I think I’ll stick with github.

Sounds good.

One more comment and I will let you make your final call …

Finally, note that in both case … if you want to serve documentation over https using a custom domain, you will have to setup your own server:

Custom domain is very important, as it allows you to change anything (hosting, documentation generator, etc.) without disrupting users or search indexing bots. It is impossible to know what services will still exist in 2-3 years or if there will be much better options you will want to switch to.

1 Like

it would be nice to start using readthedocs more seriously for Slicer core module documentation. People could just use their GitHub account to contribute (no more wiki signup troubles), we could keep the code and documentation tightly linked, etc.

Blender manual on ReadTheDocs is a good example of how it could look. It is a huge manual, so most probably we can use the same techniques, we would not need to invent anything new. Its source code is available here: https://developer.blender.org/diffusion/BM/

The main complication right now is that the documentation is generated from a separate branch. Keeping the documentation branch in sync with master is difficult because for example I want to merge changes of util.py from master, but some files are only present in the documentation branch.

Should we maintain code and documentation in same repository and branch?

Advantages of keeping code and documentation together:

  • Documentation and code would be tightly synchronized. We could ensure that even small details, programming examples, etc. are always accurate.
  • We could do code and documentation updates in the same pull request (we can better enforce consistency and quality, include documentation testing in continuous integration, etc).
  • Documentation and code is kept together in smaller projects anyway, such as in Slicer extensions. We would use the same operating mechanism for Slicer core and extension documentation.
  • We can generate user and API documentation from the same repository. User and developer documentation are often interlinked, so it is useful if we can generate them together.

Advantages of keeping documentation in separate repositories:

  • Smaller code repository size. Slicer source code size is currently about 110MB. Over time, documentation would probably make it about 50% larger (estimated based on Blender manual, which is 70MB: contains 1900 png files, 1200 rst files).
  • Access control and development process could be defined separately for source code and documentation.

I lean towards keeping code and documentation in one repository and branch. What do you think?
@jcfr @pieper @cpinter @ihnorton @fedorov

3 Likes

I’d strongly prefer having them in the same repository. If we have the documentation in the code then it also will be much harder to forget updating the documentation (you need to realize that it needs to be changed, find the wiki page, upload new screenshot to mediawiki, etc.).

About the disadvantages of the same repository approach, the only one I consider a problem is access. I guess then we’d need a place where anybody can suggest edits, and a person who’s responsible for migrating the suggestions. Neither seem to be an easy problem to solve, but the second one could happen the same way we handle PRs: anyone who has a minute reviewing/integrating does it. Any ideas about how to offer the documentation for editing for the wide public?

You can edit documentation on the web interface and when you submit your change then a pull request is automatically created. So, documentation changes can be managed the same way as source code change requests. This would require promoting GitHub to be the primary repository (which we plan to do anyway).

2 Likes

I like the idea of putting everything in the same repository. I think it’s just way simpler (and things are already complicated enough).

Repository size would be an issue if we host all images and movies in the repo. Look at the ProjectWeek repo which is already almost half a gig. At the same time I’d like a seamless process for adding media to the documentation.

Access control, actually sounds easier if we make all documentation changes pull requests.

Videos would probably not be updated that frequently, could be very big, and mostly accessed online (you could not print, would not want to bundle them with the manual anyway), so I think we should just add youtube links for videos (as it is done in Blender manual). We can enforce reduced-size screenshots and images through code reviews.

@jcfr It seems that we all lean towards trying to host documentation in the same repository and branch as the source code. If you agree, too, then it would be great if you could merge rst… branch to master and change readthedocs settings to generate documentation from master branch.

2 Likes

If we make the merge and finally have a “live” rst documentation (the current one I consider kindof dormant), then I volunteer to migrate some of the most important pages to rst.

1 Like

It is live (updated automatically within minutes) but generated from rst-user-and-dev-guide branch. You can start adding content in that branch.

Yes, thanks! I meant that it is a bit obscure and this is why nobody edits it. Once the doc is with the actual source code then editing it will be more coupled with the development process itself. I think this is the biggest issue right now.

One option for images is to store them in git-lfs. That way they are versioned, but don’t impact cloning or take up space in the “real” repository (when they are not used). See this comment.

This is a great idea.

I think git-lfs is still a niche and can cause problems at many places, but if it is only used for documentation files then probably the impact can be minimized. If somebody does not bother with setting up git-lfs then only documentation files would be missed.

We would need to do some experiments if it works with web uploads and if automatic rules can be created that prevent accidental uploads of large documentation files as non-lfs objects.

3 Likes

I use git-lfs for large datasets in the VTKExamples. No probkems
It does require a client side intsall to see the files.

2 Likes