readthedocs.org Documentation

Dev/Design/Apiv3

======================
API v3 design document
======================

This document describes the design,
some decisions already made and built (current Version 1 of APIv3)
and an implementation plan for next Versions of APIv3.

APIv3 will be designed to be easy to use and useful to perform read and write operations as the main two goals.

It will be based on Resources as APIv2 but considering the `Projectresource as the main one, from where most of the endpoint will be based on it.

.. contents:: :local: :backlinks: none :depth: 1

Goals -----

* Easy to use for our users (access most of resources by slug) * Useful to perform read and write operations * Authentication/Authorization

* Authentication based on scoped-tokens * Handle Authorization nicely using an abstraction layer

* Cover most useful cases:

* Integration on CI (check build status, trigger new build, etc) * Usage from public Sphinx/MkDocs extensions * Allow creation of flyout menu client-side * Simplify migration from other services (import projects, create multiple redirects, etc)

Non-Goals ---------

* Filter by arbitrary and useless fields

* "Builds with exit_code=1" * "Builds containingERRORon their output" * "Projects created after X datetime" * "Versions with tagpython"

Cover all the actions* available from the WebUI

Problems with APIv2 -------------------

There are several problem with our current APIv2 that we can list:

* No authentication * It's read-only * Not designed for slugs * Useful APIs not exposed (only for internal usage currently) * Error reporting is a mess * Relationships between API resources is not obvious * Footer API endpoint returns HTML

Implementation stages ---------------------

Version 1 +++++++++

The first implementation of APIv3 will cover the following aspects:

* Authentication

* all endpoints require authentication via Authorization:request header * detail endpoints are available for all authenticated users * only Project's maintainers can access listing endpoints * personalized listing

* Read and Write

* edit attributes from Version (only active and privacy_level) * trigger Build for a specific Version

* Accessible by slug

* Projects are accessed by slug* Versions are accessed byslug*/projects/endpoint is the main one and all of the other are nested under it * Builds are accessed byid, as exception to this rule * access all (active/non-active) Versions of a Project byslug* get latest Build for a Project (and Version) byslug* filter by relevant fields

* Proper status codes to report errors

* Browse-able endpoints

* browse is allowed hitting /api/v3/projects/as starting point * ability to navigate clicking on other resources under_links attribute

* Rate limited

Version 2 +++++++++

.. note::

This is currently implemented and live.

Second iteration will polish issues found from the first step, and add new endpoints to allow import a project and configure it without the needed of using the WebUI as a main goal.

After Version 2 is deployed, we will invite users that reach us as beta testers to receive more feedback and continue improving it by supporting more use cases.

This iteration will include:

* Minor changes to fields returned in the objects * Import Project endpoint * Edit Project attributes ("Settings" and "Advanced settings-Global settings" in the WebUI) * Trigger Build for default version * Allow CRUD for Redirect, Environment Variables and Notifications (WebHook and EmailHook) * Create/Delete a Project as subproject of another Project * Documentation

Version 3 +++++++++

Third iteration will implement granular permissions. Keeping in mind how Sphinx extension will use it:

sphinx-version-warning needs to get all active Versions of a Project* An extension that creates a landing page, will need all the subprojects of a Project*

To fulfill these requirements, this iteration will include:

* Scope-based authorization token

Version 4 +++++++++

* Specific endpoint for our flyout menu (returning JSON instead of HTML)

Out of roadmap --------------

These are some features that we may want to build at some point. Although, they are currently out of our near roadmap because they don't affect too many users, or are for internal usage only.

* CRUD for Domain * Add User as maintainer * Give access to a documentation page (objects.inv, /design/core.html) * Internal Build process

Nice to have ------------

* Request-IDheader *JSON minified by default_ (maybe with ?pretty=true) *JSON schema and validation_ with docs_

.. _JSON minified by default: https://geemus.gitbooks.io/http-api-design/content/en/responses/keep-json-minified-in-all-responses.html .. _JSON schema and validation: https://geemus.gitbooks.io/http-api-design/content/en/responses/keep-json-minified-in-all-responses.html .. _docs: https://geemus.gitbooks.io/http-api-design/content/en/artifacts/provide-human-readable-docs.html

---

`Dev/Design/Better Doc Urls Handling`

Better handling of docs URLs ============================

proxitois the component of our code base in charge of serving documentation to users and handling any other URLs from the user documentation domain.

The current implementation has some problems that are discussed in this document, and an alternative implementation is proposed to solve those problems.

Goals -----

* Simplifying our parsing logic for URLs * Removing reserved paths and ambiguities from URLs * Allow serving docs from a different prefix and subproject prefix.

Non-goals ---------

* Allowing fully arbitrary URL generation for projects, like changing the order of the elements or removing them.

Current implementation ----------------------

The current implementation is based on Django URLs trying to match a pattern that looks like a single project, a versioned project, or a subproject, this means that a couple of URLs are reserved, and won't resolve to the correct file if it exists (https://github.com/readthedocs/readthedocs.org/issues/8399, https://github.com/readthedocs/readthedocs.org/issues/2292), this usually happens with single version projects.

And to support custom URLs we are hacking into Django's urlconf to override it at runtime, this doesn't allow us to implement custom URLs for subprojects easily (https://github.com/readthedocs/readthedocs.org/pull/8327).

Alternative implementation --------------------------

Instead of trying to map a URL to a view, we first analyze the root project (given from the subdomain), and based on that we map each part of the URL to the current project and version.

This will allow us to reuse this code in our unresolver without the need to override the Django's urlconf at runtime, or guessing a project only by the structure of its URL.

Terminology:

Root project The project from where the documentation is served (usually the parent project of a subproject or translation). Current project The project that owns the current file being served (a subproject, a translation, etc). Requested file The final path to the file that we need to serve from the current project.

Look up process ---------------

Proxito will process all documentation requests from a single docs serve view, excluding/_ URLs.

This view then will process the current URL using the root project as follows:

- Check if the root project has translations (the project itself is a translation if isn't a single version project), and the first part is a language code and the second is a version.

- If the lang code doesn't match, we continue. - If the lang code matches, but the version doesn't, we return 404.

- Check if it has subprojects and the first part of the URL matches the subprojects prefix (projects), and if the second part of the URL matches a subproject alias.

- If the subproject prefix or the alias don't match, we continue. - If they match, we try to match the rest of the URL for translations/versions and single versions (i.e, we don't search for subprojects) and we use the subproject as the new root project.

- Check if the project is a single version. Here we just try to serve the rest of the URL as the file.

- Check if the first part of the URL is page, then this is apageredirect. Note that this is done after we have discarded the project being a single version project, since it doesn't makes sense to use that redirect with single version projects, and it could collide with the project having apage/ directory.

- 404 if none of the above rules match.

Custom URLs ~~~~~~~~~~~

We are using custom URLs mainly to serve the documentation from a different directory:

- deeplearning/nemo/user-guide/docs/$language/$version/$filename - deeplearning/nemo/user-guide/docs/$language/$version/$filename - deeplearning/frameworks/nvtx-plugins/user-guide/docs/$language/$version/$filename

We always keep the lang/version/filename order, do we need/want to support changing this order? Doesn't seem useful to do so.

So, what we need is have a way to specify a prefix only. We would have a prefix used for translations and another one used for subprojects. These prefixes will be set in the root project.

The look up order would be as follow:

- If the root project has a custom prefix, and the current URL matches that prefix, remove the prefix and follow the translations and single version look up process. We exclude subprojects from it, i.e, we don't check for{prefix}/projects. - If the root project has subprojects and a custom subprojects prefix (projectsby default), and if the current URL matches that prefix, and the next part of the URL matches a subproject alias, continue with the subproject look up process.

Examples --------

The next examples are organized in the following way:

- First there is a list of the projects involved, with their available versions. - The first project would be the root project. - The other projects will be related to the root project (their relationship is given by their name). - Next we will have a table of the requests, and their result.

Project with versions and translations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Projects:

- project (latest, 1.0) - project-es (latest, 1.0)

Requests: