`` or other well-defined standard tags) identifier found is returned. Embed endpoints --------------- This is the list of endpoints to be implemented in APIv3: .. http:get:: /api/v3/embed/ Returns the exact HTML content for a specific identifier (``id``). If no anchor identifier is specified the content of the first one returned. **Example request**: .. code:: bash $ curl https://app.readthedocs.org/api/v3/embed/?url=https://docs.readthedocs.io/en/latest/development/install.html#set-up-your-environment **Example response**: .. sourcecode:: json { "project": "docs", "version": "latest", "language": "en", "path": "development/install.html", "title": "Development Installation", "url": "https://docs.readthedocs.io/en/latest/install.html#set-up-your-environment", "id": "set-up-your-environment", "content": "
\n

Development InstallationΒΆ

\n ..." } :query url (required): Full URL for the documentation page with optional anchor identifier. .. http:get:: /api/v3/embed/metadata/ Returns all the available metadata for an specific page. .. note:: As it's not trivial to get the ``title`` associated with a particular ``id`` and it's not easy to get a nested list of identifiers, we may not implement this endpoint in initial version. The endpoint as-is, is mainly useful to explore/discover what are the identifiers available for a particular page --which is handy in the development process of a new tool that consumes the API. Because of this, we don't have too much traction to add it in the initial version. **Example request**: .. code:: bash $ curl https://app.readthedocs.org/api/v3/embed/metadata/?url=https://docs.readthedocs.io/en/latest/development/install.html **Example response**: .. sourcecode:: json { "identifiers": { "id": "set-up-your-environment", "url": "https://docs.readthedocs.io/en/latest/development/install.html#set-up-your-environment" "_links": { "embed": "https://docs.readthedocs.io/_/api/v3/embed/?url=https://docs.readthedocs.io/en/latest/development/install.html#set-up-your-environment" } }, { "id": "check-that-everything-works", "url": "https://docs.readthedocs.io/en/latest/development/install.html#check-that-everything-works" "_links": { "embed": "https://docs.readthedocs.io/_/api/v3/embed/?url=https://docs.readthedocs.io/en/latest/development/install.html#check-that-everything-works" } }, } :query url (required): Full URL for the documentation page Handle specific Sphinx cases ---------------------------- .. https://github.com/readthedocs/readthedocs.org/pull/8039#discussion_r640670085 We are currently handling some special cases for Sphinx due how it writes the HTML output structure. In some cases, we look for the HTML tag with the identifier requested but we return the ``.next()`` HTML tag or the ``.parent()`` tag instead of the *requested one*. Currently, we have identified that this happens for definition tags (``dl``, ``dt``, ``dd``) --but may be other cases we don't know yet. Sphinx adds the ``id=`` attribute to the ``dt`` tag, which contains only the title of the definition, but as a user, we are expecting the description of it. In the following example we will return the whole ``dl`` HTML tag instead of the HTML tag with the identifier ``id="term-name"`` as requested by the client, because otherwise the "Term definition for Term Name" content won't be included and the response would be useless. .. code:: html
Term Name
Term definition for Term Name
If the definition list (``dl``) has more than *one definition* it will return **only the term requested**. Considering the following example, with the request ``?url=glossary.html#term-name`` .. code:: html
...
Term Name
Term definition for Term Name
Term Unknown
Term definition for Term Unknown
...
It will return the whole ``dl`` with only the ``dt`` and ``dd`` for ``id`` requested: .. code:: html
Term Name
Term definition for Term Name
However, this assumptions may not apply to documentation pages built with a different doctool than Sphinx. For this reason, we need to communicate to the API that we want to handle this special cases in the backend. This will be done by appending a request GET argument to the Embed API endpoint: ``?doctool=sphinx&version=4.0.1&writer=html4``. In this case, the backend will known that has to deal with these special cases. .. note:: This leaves the door open to be able to support more special cases (e.g. for other doctools) without breaking the actual behavior. Support for external documents ------------------------------ When the ``?url=`` argument passed belongs to a documentation page not hosted on Read the Docs, the endpoint will do an external request to download the HTML file, parse it and return the content for the identifier requested. The whole logic should be the same, the only difference would be where the source HTML comes from. .. warning:: We should be careful with the URL received from the user because those may be internal URLs and we could be leaking some data. Example: ``?url=http://localhost/some-weird-endpoint`` or ``?url=http://169.254.169.254/latest/meta-data/`` (see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html). This is related to SSRF (https://en.wikipedia.org/wiki/Server-side_request_forgery). It doesn't seem to be a huge problem, but something to consider. Also, the endpoint may need to limit the requests per-external domain to avoid using our servers to take down another site. .. note:: Due to the potential security issues mentioned, we will start with an allowed list of domains for common Sphinx docs projects. Projects like Django and Python, where ``sphinx-hoverxref`` users might commonly want to embed from. We aren't planning to allow arbitrary HTML from any website. Handle project's domain changes ------------------------------- The proposed Embed APIv3 implementation only allows ``?url=`` argument to embed content from that page. That URL can be: * a URL for a project hosted under ``.readthedocs.io`` * a URL for a project with a custom domain In the first case, we can easily get the project's slug directly from the URL. However, in the second case we get the project's slug by querying our database for a ``Domain`` object with the full domain from the URL. Now, consider that all the links in the documentation page that uses Embed APIv3 are pointing to ``docs.example.com`` and the author decides to change the domain to be ``docs.newdomain.com``. At this point there are different possible scenarios: * The user creates a new ``Domain`` object with ``docs.newdomain.com`` as domain's name. In this case, old links will keep working because we still have the old ``Domain`` object in our database and we can use it to get the project's slug. * The user *deletes* the old ``Domain`` besides creating the new one. In this scenario, our query for a ``Domain`` with name ``docs.example.com`` to our database will fail. We will need to do a request to ``docs.example.com`` and check for a 3xx response status code and in that case, we can read the ``Location:`` HTTP header to find the new domain's name for the documentation. Once we have the new domain from the redirect response, we can query our database again to find out the project's slug. .. note:: We will follow up to 5 redirects to find out the project's domain. Embed APIv2 deprecation ----------------------- The v2 is currently widely used by projects using the ``sphinx-hoverxref`` extension. Because of that, we need to keep supporting it as-is for a long time. Next steps on this direction should be: * Add a note in the documentation mentioning this endpoint is deprecated * Promote the usage of the new Embed APIv3 * Migrate the ``sphinx-hoverxref`` extension to use the new endpoint Once we have done them, we could check our NGINX logs to find out if there are people still using APIv2, contact them and let them know that they have some months to migrate since the endpoint is deprecated and will be removed. Unanswered questions -------------------- * How do we distinguish between our APIv3 for resources (models in the database) from these "feature API endpoints"? --- ### Dev/Design/File Tree Diff Version file tree diff ====================== Goals ----- - Compare files from two versions to identify the files that have been added, removed, or modified. - Provide an API for this feature. - Integrate this feature to suggest redirects on files that were removed. - Integrate this feature to list the files that changed in a pull request. Non-goals --------- - Replace the `docdiff `__ feature from addons. That works on the client side, and it's good for comparing the content of files. Current problems ---------------- Currently, when a user opens a PRs, they need to manually search for the files of interest (new and modified files). We have a GitHub action that links to the root of the documentation preview, that helps a little, but it's not enough. When files are removed or renamed, users may not be aware that a redirect may be needed. We track 404s in our traffic analytics, but they don't keep track of the version, and it may be too late to add a redirect when users are already seeing a 404. In the past, we haven't implemented those features, because it's hard to map the source files to the generated files, since that depends on the build tool and configuration used by the project. Git providers may already offer a way to compare file trees, but again, they work on the source files, and not on the generated files. All hope was lost for having nice features like this, until now. Proposed solution ----------------- Since redirects and files of interest are related to the generated files, instead of working over the source files, we will work over the generated files, which we have access to. The key points of this feature are: - Get the diff of the file tree between two versions. - Expose that as an API. - Integrate that in PR previews. Diff between two versions ------------------------- Using a manifest ~~~~~~~~~~~~~~~~ We can create a manifest that contains the hashes and other important metadata of the files, we can save this manifest in storage or in the DB. When a build finishes, we generate this manifest for all HTML files, and store it. When we need to compare two versions, we can just compare the manifests. This doesn't require downloading the files, but it requires building a version to generate the manifest. The manifest will be a JSON object with the following structure: .. code:: json { "build": { "id": 1 }, "files": { "index.html": { "hash": "1234567890" }, "path/to/file.html": { "hash": "1234567890" } } } Using rclone ~~~~~~~~~~~~ .. note:: This solution won't be used in the final implementation, it's kept here for reference. We are already using ``rclone`` to speed up uploads to S3, ``rclone`` has a command (``rclone check``) to return the diff between two directories. For this, it uses the metadata of the files, like size and hash (it doesn't download the files). .. code:: console $ ls a changed.txt new.txt unchanged.txt $ ls b changed.txt deleted.txt unchanged.txt $ rclone check --combined=- /usr/src/app/checkouts/readthedocs.org/a /usr/src/app/checkouts/readthedocs.org/b + new.txt - deleted.txt = unchanged.txt * changed.txt The result is a list of files with a mark indicating if they were added, removed, or modified, or if they were unchanged. The result is easy to parse. There is no option to exclude the files that were unchanged when using ``--combined``, another option can be to output each type of change to a different file (``--missing-on-dst``, ``--missing-on-src``, ``--differ``). To start, we will only consider HTML files (``--include=*.html``). Changed files ------------- Listing the files that were added or deleted is straightforward, but when listing the files that were modified, we want to list files that had relevant changes only. For example, if the build injects some content that changes on every build (like a timestamp or commit), we don't want to list all files as modified. We have a couple of options to improve this list. Hashing the main content ~~~~~~~~~~~~~~~~~~~~~~~~ Timestamps and other metadata is usually added in the footer of the files, outside the main content. Instead of hashing the whole file, we can hash only the main content of the file, and use that hash to compare the files. This will allow us to better detect files that were modified in a meaningful way. Since we don't need a secure hash, we can use MD5, since it's built-in in Python. Lines changed between two files ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. note:: This solution won't be used in the final implementation, it's kept here for reference. In order to provide more useful information, we can sort the files by some metrics, like the number of lines that changed. Once we have the list of files that changed, we can use a tool like ``diff`` to get the lines that changed. This is useful to link to the most relevant files that changed in a PR. .. code:: console $ cat a.txt One Two Three Four Five $ cat b.txt Ore Three Four Five Six $ diff --side-by-side --suppress-common-lines a.txt b.txt One | Ore Two < > Six .. note:: Taken from https://stackoverflow.com/questions/27236891/diff-command-to-get-number-of-different-lines-only. The command will return only the lines that changed between the two files. We can just count the lines, or maybe even parse each symbol to check if the line was added or removed. Another alternative is to use the `difflib `__ module, the only downside is that it doesn't distinguish lines that were changed from lines that were added or removed. But maybe that's ok? Do we really need to know if a line was changed instead of added or removed? .. code:: python import difflib diff = difflib.ndiff(["one", "two", "three", "four"], ["ore", "three", "four", "five"]) print(list(diff)) # ['+ ore', '- one', '- two', ' three', ' four', '+ five'] A good thing of using Python is that we don't need to write the files to disk, and the result is easier to parse. Alternative metrics +++++++++++++++++++ .. note:: This solution won't be used in the final implementation, it's kept here for reference. Checking the number of lines that changed is a good metric, but it requires downloading the files. Another metric we could use is the size of the files, that can be obtained from the metadata (no need of downloading the files), The most a file size has changed, the most lines have likely been added or removed, this still leaves lines that changed with the same amount of characters as irrelevant in the listing. Storing results --------------- Doing a diff between two versions can be expensive, so we need to store the results. We can store the results in the DB (``VersionDiff``). The information to store would contain some information about the versions compared, the builds, and the diff itself. .. code:: python class VersionDiff(models.Model): version_a = models.ForeignKey( Version, on_delete=models.CASCADE, related_name="diff_a" ) version_b = models.ForeignKey( Version, on_delete=models.CASCADE, related_name="diff_b" ) build_a = models.ForeignKey(Build, on_delete=models.CASCADE, related_name="diff_a") build_b = models.ForeignKey(Build, on_delete=models.CASCADE, related_name="diff_b") diff = JSONField() The diff will be a JSON object with the files that were added, removed, or modified. With an structure like this: .. code:: json { "added": [{"file": "new.txt"}], "removed": [{"file": "deleted.txt"}], "modified": [{"file": "changed.txt", "lines": {"added": 1, "removed": 1}}] } The information is stored in a similar way that it will be returned by the API. Things important to note: - We need to take into consideration the diff of the latest successful builds only. If any of the builds from the stored diff don't match the latest successful build of any of the versions, we need to the diff again. - Once we have the diff between versions ``A`` and ``B``, we can infer the diff between ``B`` and ``A``. We can store that information as well, or just calculate it on the fly. - The list of files are objects, so we can store additional information in the future. - When a file has been modified, we also store the number of lines that changed. We could also show this for files that were added or removed. - If a project or version is deleted (or deactivated), we should delete the diff as well. - Using the DB to save this information will serve as the lock for the API, so we don't calculate the diff multiple times for the same versions. We could store the changed files sorted by the number of changes, or make that an option in the API, or just let the client sort the files as they see fit. API --- The initial diff operation can be expensive, so we may consider not exposing this feature to unauthenticated users. And a diff can only be done between versions of the same project that the user has access to. The endpoint will be: GET /api/v3/projects/{project_slug}/diff/?version_a={version_a}&version_b={version_b} And the response will be: .. code:: json { "version_a": {"id": 1, "build": {"id": 1}}, "version_b": {"id": 2, "build": {"id": 2}}, "diff": { "added": [{"file": "new.txt"}], "removed": [{"file": "deleted.txt"}], "modified": [{"file": "changed.txt", "lines": {"added": 1, "removed": 1}}] } } The version and build can be the full objects, or just the IDs and slugs. We will generate a lock on this request, to avoid multiple calls to the API for the same versions. We can reply with a ``202 Accepted`` if the diff is being calculated in another request. Integrations ------------ You may be thinking that once we have an API, it will be just a matter of calling that API from a GitHub action. Wrong! Doing the API call is easy, but knowing *when* to call it is hard. We need to call the API after the build has finished successfully, or we will be comparing the files of an incomplete or stale build. Luckily, we have a webhook that tells us when a build has finished successfully. But, we don't want users to have to implement the integration by themselves. We could: - Use this as an opportunity to explore using GitHub Apps. - Request additional permissions in our existing OAuth2 integration (``project`` scope). Probably not a good idea. - Expose this feature in the dashboard for now, and use our GitHub action to simply link to the dashboard. Maybe don't even expose the API to the public, just use it internally. - Use a custom `repository dispatch event `__ to trigger the action from our webhook. This requires the user to do some additional setup, and for our webhooks to support custom headers. - Hit the API repeatedly from the GitHub action until the diff is ready. This is not ideal, some build may take a long time, and the action may time out. - Expose this feature in the addons API only, which will hit the service when a user views the PR preview. Initial implementation ---------------------- For the initial implementation, we will: - Generate a manifest of all HTML files from the versions that we want to compare. This will be done at the end of the build. - Generate the hash based on the main content of the file, not the whole file. - MD5 will be the hashing algorithm used. - Only expose the files that were added, removed, or modified (HTML files only). The number of lines that changed won't be exposed. - Don't store the results in the DB, we can store the results in a next iteration. - Expose this feature only via the addons feature. - Allow to diff an external version against the version that points to the default branch/tag of the project only. - Use a feature flag to enable this feature on projects. Other features that are not mentioned here, like exposing the number of lines that changed, or a public API, will not be implemented in the initial version, and may be considered in the future (and their implementation is subject to change). Possible issues --------------- In the case that we use a manifest, hashing the contents of the files may add some overhead to the build. In the case that we use ``rclone``, even if we don't download files from S3, we are still making calls to S3, and AWS charges for those calls. But since we are doing this on demand, and we can cache the results, we can minimize the costs (maybe is not that much). ``rclone check`` returns only the list of files that changed, if we want to make additional checks over those files, we will need to make additional calls to S3. We should also just check a X number of files, we don't want to run a diff of thousands of files, and also a limit on the size of the files. Future improvements and ideas ----------------------------- - Detect moved files. This will imply checking the hashes of deleted and added files, if that same hash of a file that was deleted matches one from a file that was added, we have a move. In case we use rclone, since we don't have access to those hashes after rclone is run, we would need to re-fetch that metadata from S3. Could be a feature request for rclone. - Detect changes in sections of HTML files. We could reuse the code we have for search indexing. - Expand to other file types - Allow doing a diff between versions of different projects - Allow to configure how the main content of the file is detected (like a CSS selector). - Allow to configure content that should be ignored when hashing the file (like a CSS selector). --- ### Dev/Design/Flyout Redesign =============================== Flyout Redesign Design Document =============================== .. warning:: This document detailed an initial idea that was not implemented in the end. Lot of things have changed since this document was written. A different approach is being implemented as part of the work done on the new addons client at https://github.com/readthedocs/readthedocs-client This document describes the design of a new "flyout API" to replace our existing `footer_html` endpoint in APIv2. The Read the Docs theme uses this API to put an updated version selector with active versions into the lower left menu. On other themes, this appears as a floating menu in the lower right. .. contents:: Contents :local: :backlinks: none :depth: 1 Issues with the existing API ------------------------------ .. figure:: ../../_static/images/design-docs/flyout/flyout-expanded.png :align: right :figwidth: 300px :target: ../../_static/images/design-docs/flyout/flyout-expanded.png Read the Docs with an expanded flyout menu in the lower left corner * The largest problem with the existing ``/api/v2/footer_html`` endpoint is that it returns a blob of HTML. This limits the ability to use this data in ways other than to generate our exact flyout. For example, Sphinx themes other than our own cannot consume this data easily and there's no real integration point for MkDocs at all. * Due to how the URLs in the version selector are generated, this API endpoint can be fairly expensive in the worst case for projects with many versions. As it stands now, this API accounts for approximately 15% of the time taken on our webservers at about 70ms for an average request (P95 ~= 150ms, P99 ~= 235ms). * The current API is a combination of data about a project that contains information on the live versions, translations, sub- and super-projects, and links to various other things like downloads, the repository on the VCS, and builds for the project on RTD itself. Some of this data never changes (links to the project and builds on RTD itself) and are effectively static while some of it could change after the documentation is built (translations, active versions). Overlap with existing API endpoints ----------------------------------- There is already significant overlap between the APIv2 ``footer_html`` call with the existing APIv3 for a project (eg. ``https://app.readthedocs.org/api/v3/projects/docs/``). The project API already returns much of the data we want, but some things like other active versions, translations, and downloads would require additional API calls or options. These options already partially implemented via the `DRF Flex Fields `_ module (eg. ``https://app.readthedocs.org/api/v3/projects/docs/?expand=active_versions``). Currently, there are separate API endpoints for translations and downloads, but ideally all the data needed to generate a version selector would be available from a single API. While this is a good approach, it will not be without issues. It's likely that some database queries and especially URL generation will need to be optimized for this to not perform worse than the existing footer API. JavaScript integration ---------------------- Our current footer API is requested from our embedded document JavaScript (``readthedocs-doc-embed.js``) which is placed on pages during the build process. Currently, this is a static file that offers no integration points for theme maintainers or project authors. There is also some static data about the project injected into the HTML of the generated documentation (``READTHEDOCS_DATA``) which has some basic data about the project and the specific build being served. This includes data such as the project and version slugs, the build date, and theme. One of the goals of this project is to give theme authors and project maintainers integration points where they can control some aspects of Read the Docs in a repeatable, deterministic way. In order to do this, we need to have JavaScript constants and functions that these maintainers can set. This proposal suggests creating a new ``readthedocs`` global variable like so: .. code-block:: javascript // This first line helps handle the case where the project/theme's overrides // are executed before the readthedocs global is defined. window.readthedocs = window.readthedocs || {}; window.readthedocs.data = JSON.parse(document.getElementById('READTHEDOCS_DATA').innerHTML); if (!window.readthedocs.integration_version_selector) { // Set the default flyout menu code // But only if the user hasn't overridden it already window.readthedocs.integration_version_selector = function (project) { // Existing code we use to create the flyout menu // Currently, this is in `core/static-src/core/js/doc-embed/footer.js:injectFooter` // The `project` variable is the result of the v3 project API for the current project }; } When Read the Docs goes to create the flyout menu, it will call this new ``readthedocs.integration_version_selector`` method which will either be our default flyout menu method or the overridden method from the theme or project. This gives us a baseline of how to add these kinds of integrations. There are other possible integrations that users may want in the future such as: * Control how RTD's search overrides Sphinx's built-in search. This could be used to remove RTD's search overrides entirely or to just change how they take effect. * Control how RTD's built-in project analytics sends data. * There could be an override to how the project's data is retrieved from the API. This could allow not even getting project/version/translation data at all to save an API call for the project. Disabling the flyout entirely ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ One nice aspect of this integration is that for projects or themes that want to completely disable the version selector, this could be done by having JS like this: .. code-block:: javascript window.readthedocs = window.readthedocs || {}; window.readthedocs.integration_version_selector = function () {}; An alternative would be a way for JS projects to define constants that affects how RTD works. This could be something like: .. code-block:: javascript window.readthedocs = window.readthedocs || {}; window.readthedocs.customizations = {disable_custom_search: true, disable_version_selector: true}; .. figure:: ../../_static/images/design-docs/flyout/flask-versions-mockup.png :align: right :figwidth: 300px :target: ../../_static/images/design-docs/flyout/flask-versions-mockup.png Flask documentation with a mockup of a custom version selector on the left sidebar Implementation steps -------------------- These are the steps that need to be taken to replace our existing footer API v2. As much as possible, these steps have been setup so they can be done and rolled out independently so they don't need to be completed all at once. * Make the changes to APIv3 to allow requesting translations, sub- and super-projects, and downloads. * Create a feature flag that will make projects use the new APIv3 instead of APIv2. Set that feature flag on our own projects. * Modify our embedded document JavaScript to use a new ``readthedocs`` global variable. If this new feature flag is set, instead of calling the APIv2, the APIv3 will be called and then ``readthedocs.integration_version_selector`` will be called with the results. * If all goes well, remove the feature flag and make APIv3 the default and deprecate APIv2. Future changes not in this rollout ---------------------------------- * Removing the old ``READTHEDOCS_DATA`` variable is not part of this implementation. This global will continue to be available but removing it will cause some projects to break for sure. * This proposal doesn't involve creating an integration point to control custom search. That could happen at a later date. * This proposal doesn't rework how the version selector looks either on the RTD Sphinx theme or on other themes by default. Any restyling can be done independently of this. --- ### Dev/Design/Future Builder Future builder ============== .. contents:: :local: :depth: 2 This document is a continuation of Santos' work about "`Explicit Builders`_". It builds on top of that document some extra features and makes some decisions about the final goal, proposing a clear direction to move forward with intermediate steps keeping backward and forward compatibility. .. _Explicit Builders: https://github.com/readthedocs/readthedocs.org/pull/8103/ .. note:: A lot of things have changed since this document was written. We have had multiple discussions where we already took some decisions and discarded some of the ideas/details proposed here. The document was merged as-is without a cleaned up and there could be some inconsistencies. Note that ``build.jobs`` and ``build.commands`` are already implemented *without defining a contract* yet, and with small differences from the idea described here. Please, refer to the following links to read more about all the discussions we already had: - Public discussions: - https://github.com/readthedocs/readthedocs.org/issues/9062 - https://github.com/readthedocs/readthedocs.org/issues/1083 - https://github.com/readthedocs/readthedocs.org/issues/9063 - https://github.com/readthedocs/readthedocs.org/issues/9088 - Private discussions: - https://github.com/readthedocs/meta/discussions/9 - https://github.com/readthedocs/meta/discussions/14 - https://github.com/readthedocs/meta/discussions/17 Goals ----- * Keep the current builder working as-is * Keep backward and forward (with intermediate steps) compatibility * Define a clear support for newbie, intermediate and advanced users * Allow users to override a command, run pre/post hook commands or define all commands by themselves * Remove the Read the Docs requirement of having access to the build process * Translate our current magic at build time to a defined contract with the user * Provide a way to add a command argument without implementing it as a config file (e.g. ``fail_on_warning``) * Define a path forward towards supporting other tools * Re-write all ``readthedocs-sphinx-ext`` features to post-processsing HTML features * Reduce complexity maintained by Read the Docs' core team * Make Read the Docs responsible for Sphinx support and delegate other tools to the community * Eventually support upload pre-build docs * Allow us to add a feature with a defined contract without worry about breaking old builds * Introduce ``build.builder: 2`` config (does not install pre-defined packages) for these new features * Motivate users to migrate to ``v2`` to finally deprecate this magic by educating users Steps ran by the builder ------------------------ Read the Docs currently controls all the build process. Users are only allowed to modify very limited behavior by using a ``.readthedocs.yaml`` file. This drove us to implement features like ``sphinx.fail_on_warning``, ``submodules``, among others, at a high implementation and maintenance cost to the core team. Besides, this hasn't been enough for more advanced users that require more control over these commands. This document proposes to clearly define the steps the builder ran and allow users to override them depending on their needings: - Newbie user / simple platform usage: Read the Docs controls all the commands (current builder) - Intermediate user: ability to override one or more commands plus running pre/post hooks - Advanced user: controls *all the commands* executed by the builder The steps identified so far are: #. Checkout #. Expose project data via environment variables (\*) #. Create environment (virtualenv / conda) #. Install dependencies #. Build documentation #. Generate defined contract (``metadata.yaml``) #. Post-process HTML (\*) #. Upload to storage (\*) Steps marked with *(\*)* are managed by Read the Docs and can't be overwritten. Defined contract ---------------- Projects building on Read the Docs must provide a ``metadata.yaml`` file after running their last command. This file contains all the data required by Read the Docs to be able to add its integrations. If this file is not provided or malformed, Read the Docs will fail the build and stop the process communicating to the user that there was a problem with the ``metadata.yaml`` and we require them to fix the problem. .. note:: There is no restriction about how this file is generated (e.g. generated with Python, Bash, statically uploaded to the repository, etc) Read the Docs does not have control over it and it's only responsible for generating it when building with Sphinx. The following is an example of a ``metadata.yaml`` that is generated by Read the Docs when building Sphinx documentation: .. code:: yaml # metadata.yaml version: 1 tool: name: sphinx version: 3.5.1 builder: html readthedocs: html_output: ./_build/html/ pdf_output: ./_build/pdf/myproject.pdf epub_output: ./_build/pdf/myproject.epub search: enabled: true css_identifier: #search-form > input[name="q"] analytics: false flyout: false canonical: docs.myproject.com language: en .. warning:: The ``metadata.yaml`` contract is not defined yet. This is just an example of what we could expect from it to be able to add our integrations. Config file ----------- As we mentioned, we want all users to use the same config file and have a clear way to override commands as they need. This will be done by using the current ``.readthedocs.yaml`` file that we already have by adding two new keys: ``build.jobs`` and ``build.commands``. If neither ``build.jobs`` or ``build.commands`` are present in the config file, Read the Docs will execute the builder we currently support without modification, keeping compatibility with all projects already building successfully. When users make usage of ``jobs:`` or ``commands:`` keys we are not responsible for them in case they fail. In these cases, we only check for a ``metadata.yaml`` file and run our code to add the integrations. ``build.jobs`` ~~~~~~~~~~~~~~ It allows users to execute one or multiple pre/post hooks and/or overwrite one or multiple commands. These are some examples where this is useful: - User wants to pass an extra argument to ``sphinx-build`` - Project requires to execute a command *before* building - User has a personal/private PyPI URL - Install project with `pip install -e` (see https://github.com/readthedocs/readthedocs.org/issues/6243) - Disable git shallow clone (see https://github.com/readthedocs/readthedocs.org/issues/5989) - Call `pip install` with `--constraint` (see https://github.com/readthedocs/readthedocs.org/issues/7258) - Do something _before_ install (see https://github.com/readthedocs/readthedocs.org/issues/6662) - Use a conda lock file to create the environment (see https://github.com/readthedocs/readthedocs.org/issues/7772) - Run a check after the build is done (e.g. ``sphinx-build -W -b linkcheck . _build/html``) - Create virtualenv with ``--system-site-packages`` - etc .. code:: yaml # .readthedocs.yaml build: builder: 2 jobs: pre_checkout: checkout: git clone --branch main https://github.com/readthedocs/readthedocs.org post_checkout: pre_create_environment: create_environment: python -m virtualenv venv post_create_environment: pre_install: install: pip install -r requirements.txt post_install: pre_build: build: html: sphinx-build -T -j auto -E -b html -d _build/doctrees -D language=en . _build/html pdf: latexmk -r latexmkrc -pdf -f -dvi- -ps- -jobname=test-builds -interaction=nonstopmode epub: sphinx -T -j auto -b epub -d _build/doctrees -D language=en . _build/epub post_build: pre_metadata: metadata: ./metadata_sphinx.py post_medatada: .. note:: *All these commands* are executed passing all the exposed environment variables. If the user only provides a subset of these jobs, we ran our default commands if the user does not provide them (see :ref:`design/future-builder:Steps ran by the builder`). For example, the following YAML is enough when the project requires running Doxygen as a pre-build step: .. code:: yaml # .readthedocs.yaml build: builder: 2 jobs: # https://breathe.readthedocs.io/en/latest/readthedocs.html#generating-doxygen-xml-files pre_build: cd ../doxygen; doxygen ``build.commands`` ~~~~~~~~~~~~~~~~~~ It allows users to have full control over the commands executed in the build process. These are some examples where this is useful: - project with a custom build process that does map ours - specific requirements that we can't/want to cover as a general rule - build documentation with a different tool than Sphinx .. code:: yaml # .readthedocs.yaml build: builder: 2 commands: - git clone --branch main https://github.com/readthedocs/readthedocs.org - pip install -r requirements.txt - sphinx-build -T -j auto -E -b html -d _build/doctrees -D language=en . _build/html - ./metadata.py Intermediate steps for rollout ------------------------------ #. Remove all the exposed data in the ``conf.py.tmpl`` file and move it to ``metadata.yaml`` #. Define structure required for ``metadata.yaml`` as contract #. Define the environment variables required (e.g. some from ``html_context``) and execute all commands with them #. Build documentation using this contract #. Leave ``readthedocs-sphinx-ext`` as the only package installed and extension install in ``conf.py.tmpl`` #. Add ``build.builder: 2`` config without any *magic* #. Build everything needed to support ``build.jobs`` and ``build.commands`` keys #. Write guides about how to use the new keys #. Re-write ``readthedocs-sphinx-ext`` features to post-process HTML features Final notes ----------- - The migration path from ``v1`` to ``v2`` will require users to explicitly specify their requirements (we don't install pre-defined packages anymore) - We probably not want to support ``build.jobs`` on ``v1`` to reduce core team's time maintaining that code without the ability to update it due to projects randomly breaking. - We would be able to start building documentation using new tools without having to *integrate them*. - Building on Read the Docs with a new tool will require: - the user to execute a different set of commands by overriding the defaults. - the project/build/user to expose a ``metadata.yaml`` with the contract that Read the Docs expects. - none, some or all the integrations will be added to the HTML output (these have to be implemented at Read the Docs core) - We are not responsible for extra formats (e.g. PDF, ePub, etc) on other tools. - Focus on support Sphinx with nice integrations made in a tool-agnostic way that can be re-used. - Removing the manipulation of ``conf.py.tmpl`` does not require us to implement the same manipulation for projects using the new potential feature ``sphinx.yaml`` file. --- ### Dev/Design/In Doc Search Ui In-doc search UI ================ Giving readers the ability to easily search the information that they are looking for is important for us. We have already upgraded to the latest version of `Elasticsearch`_ and we plan to implement `search as you type` feature for all the documentations hosted by us. It will be designed to provide instant results as soon as the user starts typing in the search bar with a clean and minimal frontend. This design document aims to provides the details of it. This is a GSoC'19 project. .. warning:: This design document details future features that are **not yet implemented**. To discuss this document, please get in touch in the `issue tracker`_. The final result may look something like this: .. figure:: /_static/images/design-docs/in-doc-search-ui/in-doc-search-ui-demo.gif :align: center Short demo Goals And non-Goals ------------------- Project goals +++++++++++++ * Support a search-as-you-type/autocomplete interface. * Support across all (or virtually all) Sphinx themes. * Support for the JavaScript user experience down to IE11 or graceful degradation where we can't support it. * Project maintainers should have a way to opt-in/opt-out of this feature. * (Optional) Project maintainers should have the flexibility to change some of the styles using custom CSS and JS files. Non-goals +++++++++ * For the initial release, we are targeting only Sphinx documentations as we don't index MkDocs documentations to our Elasticsearch index. Existing search implementation ------------------------------ We have a detailed documentation explaining the underlying architecture of our search backend and how we index documents to our Elasticsearch index. You can read about it :doc:`here `. Proposed architecture for in-doc search UI ------------------------------------------ Frontend ++++++++ Technologies ~~~~~~~~~~~~ Frontend is to designed in a theme agnostics way. For that, we explored various libraries which may be of use but none of them fits our needs. So, we might be using vanilla JavaScript for this purpose. This will provide us some advantages over using any third party library: * Better control over the DOM. * Performance benefits. Proposed architecture ~~~~~~~~~~~~~~~~~~~~~ We plan to select the search bar, which is present in every theme, and use the `querySelector()`_ method of JavaScript. Then add an event listener to it to listen for the changes and fire a search query to our backend as soon as there is any change. Our backend will then return the suggestions, which will be shown to the user in a clean and minimal UI. We will be using `document.createElement()`_ and `node.removeChild()`_ method provided by JavaScript as we don't want empty `
` hanging out in the DOM. We have a few ways to include the required JavaScript and CSS files in all the projects: * Add CSS into `readthedocs-doc-embed.css` and JS into `readthedocs-doc-embed.js` and it will get included. * Package the in-doc search into it's own self-contained CSS and JS files and include them in a similar manner to `readthedocs-doc-embed.*`. * It might be possible to package up the in-doc CSS/JS as a sphinx extension. This might be nice because then it's easy to enable it on a per-project basis. When we are ready to roll it out to a wider audience, we can make a decision to just turn it on for everybody (put it in `here`_) or we could enable it as an opt-in feature like the `404 extension`_. UI/UX ~~~~~ We have two ways which can be used to show suggestions to the user. * Show suggestions below the search bar. * Open a full page search interface when the user click on search field. Backend +++++++ We have a few options to support `search as you type` feature, but we need to decide that which option would be best for our use-case. Edge NGram Tokenizer ~~~~~~~~~~~~~~~~~~~~ * Pros * More effective than Completion Suggester when it comes to autocompleting words that can appear in any order. * It is considerable fast because most of the work is being done at index time, hence the time taken for autocompletion is reduced. * Supports highlighting of the matching terms. * Cons * Requires greater disk space. Completion suggester ~~~~~~~~~~~~~~~~~~~~ * Pros * Really fast as it is optimized for speed. * Does not require large disk space. * Cons * Matching always starts at the beginning of the text. So, for example, "Hel" will match "Hello, World" but not "World Hello". * Highlighting of the matching words is not supported. * According to the official docs for Completion Suggester, fast lookups are costly to build and are stored in-memory. Milestones ---------- +-----------------------------------------------------------------------------------+------------------+ | Milestone | Due Date | +===================================================================================+==================+ | A local implementation of the project. | 12th June, 2019 | +-----------------------------------------------------------------------------------+------------------+ | In-doc search on a test project hosted on Read the Docs using the RTD Search API. | 20th June, 2019 | +-----------------------------------------------------------------------------------+------------------+ | In-doc search on docs.readthedocs.io. | 20th June, 2019 | +-----------------------------------------------------------------------------------+------------------+ | Friendly user trial where users can add this on their own docs. | 5th July, 2019 | +-----------------------------------------------------------------------------------+------------------+ | Additional UX testing on the top-10 Sphinx themes. | 15th July, 2019 | +-----------------------------------------------------------------------------------+------------------+ | Finalize the UI. | 25th July, 2019 | +-----------------------------------------------------------------------------------+------------------+ | Improve the search backend for efficient and fast search results. | 10th August, 2019| +-----------------------------------------------------------------------------------+------------------+ Open questions ++++++++++++++ * Should we rely on jQuery, any third party library or pure vanilla JavaScript? * Are the subprojects to be searched? * Is our existing Search API is sufficient? * Should we go for edge ngrams or completion suggester? .. _issue tracker: https://github.com/readthedocs/readthedocs.org/issues .. _Elasticsearch: https://www.elastic.co/products/elasticsearch .. _querySelector(): https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelector .. _document.createElement(): https://developer.mozilla.org/en-US/docs/Web/API/Document/createElement .. _node.removeChild(): https://developer.mozilla.org/en-US/docs/Web/API/Node/removeChild .. _here: https://github.com/readthedocs/readthedocs.org/blob/9ca5858e859dea0759d913e8db70a623d62d6a16/readthedocs/doc_builder/templates/doc_builder/conf.py.tmpl#L135-L142 .. _404 extension : https://github.com/readthedocs/sphinx-notfound-page --- ### Dev/Design/Index Design documents ================ This is where we outline the design of major parts of our project. Generally this is only available for features that have been build in the recent past, but we hope to write more of them over time. .. warning:: These documents may not match the final implementation, or may be out of date. .. toctree:: :maxdepth: 1 :glob: * --- ### Dev/Design/New Notifications System Notification system: a new approach after lot of discussions ============================================================ Notifications have been a recurrent topic in the last years. We have talked about different problems and solution's approaches during these years. However, due to the complexity of the change, and without having a clear path, it has been hard to prioritize. We've written a lot about the problems and potential solutions for the current notification system. This is a non-complete list of them just for reference: * https://github.com/readthedocs/readthedocs.org/issues/4226 * https://github.com/readthedocs/readthedocs.org/issues/3399 * https://github.com/readthedocs/meta/discussions/126 * https://github.com/readthedocs/readthedocs.org/issues/5909 * https://github.com/readthedocs/readthedocs.org/issues/9110 * https://github.com/readthedocs/readthedocs.org/issues/9279 * https://github.com/readthedocs/readthedocs.org/issues/4235 At the offsite in Portland, Anthony and myself were able to talk deeply about this and wrote a bunch of thoughts in a Google Doc. We had pretty similar ideas and we thought we were solving most of the problems we identified already. I read all of these issues and all the discussions I found and wrote this document that summarizes my proposal: *create a new notification system that we can customize and expand as we need in the future*: * A Django model to store the notifications' data * API endpoints to retrieve the notifications for a particular resource (User, Build, Project, Organization) * Frontend code to display them (*outside the scope of this document*) Goals ----- * Keep raising exceptions for errors from the build process * Ability to add non-error notifications from the build process * Add extra metadata associated to the notification: icon, header, body, etc * Support different types of notifications (e.g. error, warning, note, tip) * Reuse the new notification system for product updates (e.g. new features, deprecated config keys) * Message content lives on Python classes that can be translated and formatted with objects (e.g. Build, Project) * Message could have richer content (e.g. HTML code) to generate links and emphasis * Notifications have trackable state (e.g. unread (default)=never shown, read=shown, dismissed=don't show again, cancelled=auto-removed after user action) * An object (e.g. Build, Organization) can have more than 1 notification attached * Remove hardcoded notifications from the templates * Notifications can be attached to Project, Organization, Build and User models * Specific notifications can be shown under the user's bell icon * Easy way to cleanup notification on status changes (e.g. subscription failure notification is auto-deleted after CC updated) * Notifications attached to Organization/Project disappear for all the users once they are dismissed by anyone Non-goals --------- * Create new Build "state" or "status" option for these fields * Implement the new notification in the old dashboard * Define front-end code implementation * Replace email or webhook notifications Small notes and other considerations ------------------------------------ * Django message system is not enough for this purpose. * Use a new model to store all the required data (expandable in the future) * How do we handle translations? We should use ``_("This is the message shown to the user")`` in Python code and return the proper translation when they are read. * Reduce complexity on ``Build`` object (remove ``Build.status`` and ``Build.error`` fields among others). * Since the ``Build`` object could have more than 1 notification, when showing them, we will sort them by importance: errors, warnings, note, tip. * In case we need a pretty specific order, we can add an extra field for that, but it adds unnecessary complexity at this point. * For those notifications that are attached to the ``Project`` or ``Organization``, should it be shown to all the members even if they don't have admin permissions? If yes, this is good because all of them will be notified but only some of them will be able to take an action. If no, non-admin users won't see the notification and won't be able to communicate this to the admins * Notification could be attached to a ``BuildCommand`` in case we want to display a specific message on a command itself. We don't know how useful this will be, but it's something we can consider in the future. * Notification preferences: what kind of notifications I want to see in my own bell icon? - Build errors - Build tips - Product updates - Blog post news - Organization updates - Project updates Implementation ideas -------------------- This section shows all the classes and models involved for the notification system as well as some already known use-cases. .. note:: Accessing the database from the build process Builders doesn't have access to the database due to security reasons. We had solved this limitation by creating an API endpoint the builder hits once they need to interact with the database to get a ``Project``, ``Version`` and ``Build`` resources, create a ``BuildCommand`` resource, etc. Besides, the build process is capable to trigger Celery tasks that are useful for managing more complex logic that also require accessing from and writing to the database. Currently, ``readthedocs.doc_builder.director.Director`` and ``readthedocs.doc_builder.environments.DockerBuildEnvironment`` have access to the API client and can use it to create the ``Notification`` resources. I plan to use the same pattern to create ``Notification`` resources by hitting the API from the director or the build environment. In case we require hitting the API from other places, we will need to pass the API client instance to those other classes as well. ``Message`` class definition ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This class encapsulates the content of the notification (e.g. header, body, icon, etc) --the message it's shown to the uer--, and some helper logic to return in the API response. .. code-block:: python class Message: def __init__(self): header = str body = str icon = str icon_style = str(SOLID, DUOTONE) type = str(ERROR, WARNING, NOTE, TIP) def get_display_icon(self): if self.icon: return self.icon if self.type == ERROR: return "fa-exclamation" if self.type == WARNING: return "fa-triangle-exclamation" Definition of notifications to display to users ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This constant defines all the possible notifications to be displayed to the user. Each notification has to be defined here using the ``Message`` class previously defined. .. code-block:: python NOTIFICATION_MESSAGES = { "generic-with-build-id": Message( header=_("Unknown problem"), # Note the message receives the instance it's attached to # and could be use it to inject related data body=_( """ There was a problem with Read the Docs while building your documentation. Please try again later. If this problem persists, report this error to us with your build id ({instance[pk]}). """, type=ERROR, ), ), "build-os-required": Message( header=_("Invalid configuration"), body=_( """ The configuration key "build.os" is required to build your documentation. Read more. """, type=ERROR, ), ), "cancelled-by-user": Message( header=_("User action"), body=_( """ Build cancelled by the user. """, type=ERROR, ), ), "os-ubuntu-18.04-deprecated": Message( header=_("Deprecated OS selected"), body=_( """ Ubuntu 18.04 is deprecated and will be removed soon. Update your .readthedocs.yaml to use a newer image. """, type=TIP, ), ), } ``Notification`` model definition ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This class is the representation of a notification attached to an resource (e.g. User, Build, etc) in the database. It contains an identifier (``message_id``) pointing to one of the messages defined in the previous section (key in constant ``NOTIFICATION_MESSAGES``). .. code-block:: python import textwrap from django.utils.translation import gettext_noop as _ class Notification(TimeStampedModel): # Message identifier message_id = models.CharField(max_length=128) # UNREAD: the notification was not shown to the user # READ: the notification was shown # DISMISSED: the notification was shown and the user dismissed it # CANCELLED: removed automatically because the user has done the action required (e.g. paid the subscription) state = models.CharField( choices=[UNREAD, READ, DISMISSED, CANCELLED], default=UNREAD, db_index=True, ) # Makes the notification impossible to dismiss (useful for Build notifications) dismissable = models.BooleanField(default=False) # Show the notification under the bell icon for the user news = models.BooleanField(default=False, help_text="Show under bell icon") # Notification attached to # # Uses ContentType for this. # https://docs.djangoproject.com/en/4.2/ref/contrib/contenttypes/#generic-relations # attached_to_content_type = models.ForeignKey(ContentType, on_delete=models.CASCADE) attached_to_id = models.PositiveIntegerField() attached_to = GenericForeignKey("attached_to_content_type", "attached_to_id") # If we don't want to use ContentType, we could define all the potential models # the notification could be attached to # # organization = models.ForeignKey(Organization, null=True, blank=True, default=None) # project = models.ForeignKey(Project, null=True, blank=True, default=None) # build = models.ForeignKey(Build, null=True, blank=True, default=None) # user = models.ForeignKey(User, null=True, blank=True, default=None) def get_display_message(self): return textwrap.dedent( NOTIFICATION_MESSAGES.get(self.message_id).format( instance=self.attached_to, # Build, Project, Organization, User ) ) Attach error ``Notification`` during the build process ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ During the build, we will keep raising exceptions to both things: - stop the build process immediately - communicate back to the ``doc_builder.director.Director`` class the build failed. The director is the one in charge of creating the error ``Notification``, in a similar way it currently works now. The only difference is that instead of saving the error under ``Build.error`` as it currently works now, it will create a ``Notification`` object and attach it to the particular ``Build``. Note the director does not have access to the DB, so it will need to create/associate the object via an API endpoint/Celery task. Example of how the exception ``BuildCancelled`` creates an error ``Notification``: .. code-block:: python class UpdateDocsTask(...): def on_failure(self): self.data.api_client.build(self.data.build["id"]).notifications.post( { "message_id": "cancelled-by-user", # Override default fields if required "type": WARNING, } ) Attach non-error ``Notification`` during the build process ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ During the build, we will be able attach non-error notifications with the following pattern: - check something in particular (e.g. using a deprecated key in ``readthedocs.yaml``) - create a non-error ``Notification`` and attach it to the particular ``Build`` object .. code-block:: python class DockerBuildEnvironment(...): def check_deprecated_os_image(self): if self.config.build.os == "ubuntu-18.04": self.api_client.build(self.data.build["id"]).notifications.post( { "message_id": "os-ubuntu-18.04-deprecated", } ) Show a ``Notification`` under the user's bell icon ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If we want to show a notification on a user's profile, we can create the notification as follows, maybe from a simple script ran in the Django shell's console after publishing a blog post: .. code-block:: python users_to_show_notification = User.objects.filter(...) for user in users_to_show_notification: Notification.objects.create( message_id="blog-post-beta-addons", dismissable=True, news=True, attached_to=User, attached_to_id=user.id, ) Remove notification on status change ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When we show a notification for an unpaid subscription, we want to remove it once the user has updated and paid the subscription. We can do this with the following code: .. code-block:: python @handler("customer.subscription.updated", "customer.subscription.deleted") def subscription_updated_event(event): if subscription.status == ACTIVE: organization = Organization.objects.get(slug="read-the-docs") Notification.objects.filter( message_id="subscription-update-your-cc-details", state__in=[UNREAD, READ], attached_to=Organization, attached_to_id=organization.id, ).update(state=CANCELLED) API definition -------------- I will follows the same pattern we have on APIv3 that uses nested endpoints. This means that we will add a ``/notifications/`` postfix to most of the resource endpoints where we want to be able to attach/list notifications. Notifications list ~~~~~~~~~~~~~~~~~~ .. http:get:: /api/v3/users/(str:user_username)/notifications/ Retrieve a list of all the notifications for this user. .. http:get:: /api/v3/projects/(str:project_slug)/notifications/ Retrieve a list of all the notifications for this project. .. http:get:: /api/v3/organizations/(str:organization_slug)/notifications/ Retrieve a list of all the notifications for this organization. .. http:get:: /api/v3/projects/(str:project_slug)/builds/(int:build_id)/notifications/ Retrieve a list of all the notifications for this build. **Example response**: .. sourcecode:: json { "count": 25, "next": "/api/v3/projects/pip/builds/12345/notifications/?unread=true&sort=type&limit=10&offset=10", "previous": null, "results": [ { "message_id": "cancelled-by-user", "state": "unread", "dismissable": false, "news": false, "attached_to": "build", "message": { "header": "User action", "body": "Build cancelled by the user.", "type": "error", "icon": "fa-exclamation", "icon_style": "duotone", } } ] } :query boolean unread: return only unread notifications :query string type: filter notifications by type (``error``, ``note``, ``tip``) :query string sort: sort the notifications (``type``, ``date`` (default)) Notification create ~~~~~~~~~~~~~~~~~~~ .. http:post:: /api/v3/projects/(str:project_slug)/builds/(int:build_id)/notifications/ Create a notification for the resource. In this example, for a ``Build`` resource. **Example request**: .. sourcecode:: json { "message_id": "cancelled-by-user", "type": "error", "state": "unread", "dismissable": false, "news": false, } .. note:: Similar API endpoints will be created for each of the resources we want to attach a ``Notification`` (e.g. ``User``, ``Organization``, etc) Notification update ~~~~~~~~~~~~~~~~~~~ .. http:patch:: /api/v3/projects/(str:project_slug)/builds/(int:build_id)/notifications/(int:notification_id)/ Update an existing notification. Mainly used to change the state from the front-end. **Example request**: .. sourcecode:: json { "state": "read", } .. note:: Similar API endpoints will be created for each of the resources we want to attach a ``Notification`` (e.g. ``User``, ``Organization``, etc) Backward compatibility ---------------------- It's not strictly required, but if we want, we could extract the current notification logic from: * Django templates * "Don't want ``setup.py`` called?" * ``build.image`` config key is deprecated * Configuration file is required * ``build.commands`` is a beta feature * ``Build.error`` fields * Build cancelled by user * Unknown exception * ``build.os`` is not found * No config file * No checkout revision * Failed when cloning the repository * etc and iterate over all the ``Build`` objects to create a ``Notification`` object for each of them. I'm not planning to implement the "new notification system" in the old templates. It doesn't make sense to spend time in them since we are deprecating them. Old builds will keep using the current notification approach based on ``build.error`` field. New builds won't have ``build.error`` anymore and they will use the new notification system on ext-theme. --- ### Dev/Design/New Search Api New search API ============== Goals ----- - Allow to configure search at the API level, instead of having the options in the database. - Allow to search a group of projects/versions at the same time. - Bring the same syntax to the dashboard search. Syntax ------ The parameters will be given in the query using the ``key:value`` syntax. Inspired by `GitHub `__ and other services. Currently the values from all parameters don't include spaces, so surrounding the value with quotes won't be supported (``key:"value"``). To avoid interpreting a query as a parameter, an escape character can be put in place, for example ``project\:docs`` won't be interpreted as a parameter, but as the search term ``project:docs``. This is only necessary if the query includes a valid parameter, unknown parameters (``foo:bar``) don't require escaping. All other tokens that don't match a valid parameter, will be join to form the final search term. Parameters ---------- project: Indicates the project and version to includes results from (this doesn't include subprojects). If the version isn't provided, the default version is used. Examples: - ``project:docs/latest`` - ``project:docs`` It can be one or more project parameters. At least one is required. If the user doesn't have permission over one version or if the version doesn't exist, we don't include results from that version. We don't fail the search, this is so users can use one endpoint for all their users, without worrying about what permissions each user has or updating it after a version or project has been deleted. The ``/`` is used as separator, but it could be any other character that isn't present in the slug of a version or project. ``:`` was considered (``project:docs:latest``), but it could be hard to read since ``:`` is already used to separate the key from the value. subprojects: This allows to specify from what project exactly we are going to return subprojects from, and also include the version we are going to try to match. This includes the parent project in the results. As the ``project`` parameter, the version can be optional, and defaults to the default version of the parent project. user: Include results from projects the given user has access to. The only supported value is ``@me``, which is an alias for the current user. Including subprojects ~~~~~~~~~~~~~~~~~~~~~ Now that we are returning results only from the given projects, we need an easy way to include results from subprojects. Some ideas for implementing this feature are: ``include-subprojects:true`` This doesn't make it clear from what projects we are going to include subprojects from. We could make it so it returns subprojects for all projects. Users will probably use this with one project only. ``subprojects:project/version`` (inclusive) This allows to specify from what project exactly we are going to return subprojects from, and also include the version we are going to try to match. This includes the parent project in the results. As the ``project`` parameter, the version can be optional, and defaults to the default version of the parent project. ``subprojects:project/version`` (exclusive) This is the same as the above, but it doesn't include the parent project in the results. If we want to include the results from the project, then the query will be ``project:project/latest subprojects:project/latest``. Is this useful? The second option was chosen, since that's the current behavior of our search when searching on a project with subprojects, and avoids having to repeat the project if the user wants to include it in the search too. Cache ----- Since the request could be attached to more than one project. We will return all the list of projects for the cache tags, this is ``project1, project1:version, project2, project2:version``. CORS ---- Since the request could be attached to more than one project. we can't make the decision if we should enable CORS or not on a given request from the middleware easily, so we won't allow cross site requests when using the new API for now. We would need to refactor our CORS code, so every view can decide if CORS should be allowed or not, for this case, cross site requests will be allowed only if all versions of the final search are public, another alternative could be to always allow cross site requests, but when a request is cross site, we only return results from public versions. Analytics --------- We will record the same query for each project that was used in the final search. Response -------- The response will be similar to the old one, but will include extra information about the search, like the projects, versions, and the query that were used in the final search. And the ``version``, ``project``, and ``project_alias`` attributes will now be objects. We could just reuse the old response too, since the only breaking changes would be the attributes now being objects, and we aren't adding any new information to those objects (yet). But also, re-using the current serializers shouldn't be a problem either. .. code-block:: json { "count": 1, "next": null, "previous": null, "projects": [ { "slug": "docs", "versions": [ { "slug": "latest" } ] } ], "query": "The final query used in the search", "results": [ { "type": "page", "project": { "slug": "docs", "alias": null }, "version": { "slug": "latest" }, "title": "Main Features", "path": "/en/latest/features.html", "domain": "https://docs.readthedocs.io", "highlights": { "title": [] }, "blocks": [ { "type": "section", "id": "full-text-search", "title": "Full-Text Search", "content": "We provide search across all the projects that we host. This actually comes in two different search experiences: dashboard search on the Read the Docs dashboard and in-doc search on documentation sites, using your own theme and our search results. We offer a number of search features: Search across subprojects Search results land on the exact content you were looking for Search across projects you have access to (available on Read the Docs Business) A full range of search operators including exact matching and excluding phrases. Learn more about Server Side Search.", "highlights": { "title": [ "Full-Text Search" ], "content": [] } }, { "type": "domain", "role": "http:post", "name": "/api/v3/projects/", "id": "post--api-v3-projects-", "content": "Import a project under authenticated user. Example request: BashPython$ curl \\ -X POST \\ -H \"Authorization: Token \" https://app.readthedocs.org/api/v3/projects/ \\ -H \"Content-Type: application/json\" \\ -d @body.json import requests import json URL = 'https://app.readthedocs.org/api/v3/projects/' TOKEN = '' HEADERS = {'Authorization': f'token {TOKEN}'} data = json.load(open('body.json', 'rb')) response = requests.post( URL, json=data, headers=HEADERS, ) print(response.json()) The content of body.json is like, { \"name\": \"Test Project\", \"repository\": { \"url\": \"https://github.com/readthedocs/template\", \"type\": \"git\" }, \"homepage\": \"http://template.readthedocs.io/\", \"programming_language\": \"py\", \"language\": \"es\" } Example response: See Project details Note Read the Docs Business, also accepts", "highlights": { "name": [], "content": [ ", json=data, headers=HEADERS, ) print(response.json()) The content of body.json is like, "name": "Test" ] } } ] } ] } Examples -------- - ``project:docs project:dev/latest test``: search for ``test`` in the default version of the ``docs`` project, and in the latest version of the ``dev`` project. - ``a project:docs/stable search term``: search for ``a search term`` in the stable version of the ``docs`` project. - ``project:docs project\:project/version``: search for ``project::project/version`` in the default version of the ``docs`` project. - ``search``: invalid, at least one project is required. Dashboard search ---------------- This is the search feature that you can access from the readthedocs.org/readthedocs.com domains. We have two types: Project scoped search: Search files and versions of the current project only. Global search: Search files and versions of all projects in .org, and only the projects the user has access to in .com. Global search also allows to search projects by name/description. This search also allows you to see the number of results from other projects/versions/sphinx domains (facets). Project scoped search ~~~~~~~~~~~~~~~~~~~~~ Here the new syntax won't have effect, since we are searching for the files of one project only! Another approach could be linking to the global search with ``project:{project.slug}`` filled in the query. Global search (projects) ~~~~~~~~~~~~~~~~~~~~~~~~ We can keep the project search as is, without using the new syntax (since it doesn't make sense there). Global search (files) ~~~~~~~~~~~~~~~~~~~~~ Using the same syntax from the API will be allowed, by default it will search all projects in .org, and all projects the user has access to in .com. Another approach could be to allow filtering by user on .org, this is ``user:stsewd`` or ``user:@me`` so a user can search all their projects easily. We could allow just ``@me`` to start. Facets ~~~~~~ We will support only the ``projects`` facet to start. We can keep the facets, but they would be a little different, since with the new syntax we need to specify a project in order to search for a version, i.e, we can't search all ``latest`` versions of all projects. By default we will use/show the ``project`` facet, and after the user has filtered by a project, we will use/show the ``version`` facet. If the user searches more than one project, things get complicated, should we keep showing the ``version`` facet? If clicked, should we change the version on all the projects? If that is too complicated to explain/implement, we should be fine by just supporting the ``project`` facet for now. Backwards compatibility ~~~~~~~~~~~~~~~~~~~~~~~ We should be able to keep the old URLs working in the global search, but we could also just ignore the old syntax, or transform the old syntax to the new one and redirect the user to it, for example ``?q=test&project=docs&version=latest`` would be transformed to ``?q=test project:docs/latest``. Future features --------------- - Allow searching on several versions of the same project (the API response is prepared to support this). - Allow searching on all versions of a project easily, with a syntax like ``project:docs/*`` or ``project:docs/@all``. - Allow specify the type of search: - Multi match (query as is) - Simple query string (allows using the ES query syntax) - Fuzzy search (same as multi match, but with fuzziness) - Add the ``org`` filter, so users can search by all projects that belong to an organization. We would show results of the default versions of each project. --- ### Dev/Design/New Sphinx Guides Proposed contents for new Sphinx guides ======================================= .. note:: This work is in progress, see discussion on `this Sphinx issue `_ and the pull requests linked at the end. The two main objectives are: - Contributing a good Sphinx tutorial for beginners. This should introduce the readers to all the various Sphinx major features in a pedagogical way, and be mostly focused on Markdown using MyST. We would try to find a place for it in the official Sphinx documentation. - Write a new narrative tutorial for Read the Docs that complements the existing guides and offers a cohesive story of how to use the service. Sphinx tutorial --------------- Appendixes are optional, i.e.Β not required to follow the tutorial, but highly recommended. #. The Sphinx way - Preliminary section giving an overview of what Sphinx is, how it works, how reStructuredText and Markdown/MyST are related to it, some terminology (toctree, builders), what can be done with it. #. About this tutorial - A section explaining the approach of the tutorial, as well as how to download the result of each section for closer inspection or for skipping parts of it. #. Getting started #. Creating our project - Present a fictitious goal for a documentation project - Create a blank ``README.md`` to introduce the most basic elements of Markdown (headings and paragraph text) #. Installing Sphinx and cookiecutter in a new development environment - Install Python (or miniforge) - Create a virtual environment (and/or conda environment) - Activate our virtual environment (it will always be the first step) - Install Sphinx inside the virtual environment - Check that ``sphinx-build --help`` works (yay!) #. Creating the documentation layout - Apply our cookiecutter to create a minimal ``docs/`` directory (similar to what ``sphinx-quickstart`` does, but with source and build separation by default, project release 0.1, English language, and a MyST index, if at all) [1]_ - Check that the correct files are created (yay!) #. Appendix: Using version control - Install git (we will not use it during the tutorial) - Add a proper ``.gitignore`` file (copied from gitignore.io) - Create the first commit for the project (yay!) #. First steps to document our project using Sphinx #. Converting our documentation to local HTML - Create (or minimally tweak) ``index.md`` - Build the HTML output using ``sphinx-build -b -W html doc doc/_build/html`` [2]_ - Navigate to ``doc/_build/html`` and launch an HTTP server (``python -m http.server``) - Open http://localhost:8000 in a web browser, and see the HTML documentation (yay!) #. Converting our documentation to other formats - Build PseudoXML using ``make pseudoxml`` - Build Text using ``make text`` - See how the various formats change the output (yay!) #. Appendix: Simplify documentation building by using Make [3]_ - Install Make (nothing is needed on Windows, `make.bat` is standalone) - Add more content to ``index.md`` - Build HTML doing ``cd doc && make html`` - Observe that the HTML docs have changed (yay!) #. Appendix: PDF without LaTeX using rinoh (beta) #. Customizing Sphinx configuration #. Changing the HTML theme - Install https://pypi.org/project/furo/ - Change the ``html_theme`` in ``conf.py`` - Rebuild the HTML documentation and observe that the theme has changed (yay!) #. Changing the PDF appearance - Add a ``latex_theme`` and set it to ``howto`` - Rebuild ``make latexpdf`` - Check that the appearance changed (yay!) #. Enable an extension - Add a string to the ``extensions`` list in ``conf.py`` for ``sphinx.ext.duration`` - Rebuild the HTML docs ``make html`` and notice that now the times are printed (yay!) #. Writing narrative documentation with Sphinx - First focus on ``index.md``, diving more into Markdown and mentioning Semantic Line Breaks. - Then add another ``.md`` file to teach how ``toctree`` works. - Then continue introducing elements of the syntax to add pictures, cross-references, and the like. #. Describing code in Sphinx - Explain the Python domain as part of narrative documentation to interleave code with text, include doctests, and justify the usefulness of the next section. #. Autogenerating documentation from code in Sphinx #. Deploying a Sphinx project online - A bit of background on the options: GitHub/GitLab Pages, custom server, Netlify, Read the Docs - Make reference to Read the Docs tutorial #. Appendix: Using Jupyter notebooks inside Sphinx #. Appendix: Understanding the docutils document tree #. Appendix: Where to go from here - Refer the user to the Sphinx, reST and MyST references, prominent projects already using Sphinx, compilations of themes and extensions, the development documentation. .. note - Looks like MathJax is enabled by default now? Can't see a reference in the docstrings Read the Docs tutorial ---------------------- #. The Read the Docs way #. Getting started #. Preparing our project on GitHub - Fork a starter GitHub repository (something like `our demo template `_, as a starting point that helps mimicking the `sphinx-quickstart` or `cookiecutter` step without having to checkout the code locally) #. Importing our project to Read the Docs - Sign up with GitHub on RTD - Import the project (don't "Edit advanced project options", we will do this later) - The project is created on RTD - Browse "builds", open the build live logs, wait a couple of minutes, open the docs (yay!) #. Basic configuration changes - Add a description, homepage, and tags - Configure your email for build failure notification (until we turn them on by default) - Enable "build pull requests for this project" in the advanced settings - Edit a file from the GitHub UI as part of a new branch, and open a pull request - See the RTD check on the GitHub PR UI, wait a few minutes, open result (yay!) #. Customizing the build process - Use `.readthedocs.yaml` (rather than the web UI) to customize build formats, change build requirements and Python version, enable fail-on-warnings #. Versioning documentation - Explain how to manage versions on RTD: create release branches, activate the corresponding version, browse them in the version selector, selectively build versions - Intermediate topics: hide versions, create Automation Rules #. Getting insights from your projects - Move around the project, explore results in Traffic Analytics - Play around with server-side search, explore results in Search Analytics #. Managing translations #. Where to go from here - Reference our existing guides, prominent projects already using RTD, domain configuration, our support form, our contributing documentation Possible new how-to Guides -------------------------- Some ideas for extra guides on specific topics, still for beginners but more problem-oriented documents, covering a wide range of use cases: - How to turn a bunch of Markdown files into a Sphinx project - How to turn a bunch of Jupyter notebooks into a Sphinx project - How to localize an existing Sphinx project - How to customize the appearance of the HTML output of a Sphinx project - How to convert existing reStructuredText documentation to Markdown - How to use Doxygen autogenerated documentation inside a Sphinx project - How to keep a changelog of your project Reference --------- All the references should be external: the Sphinx reference, the MyST and reST syntax specs, and so forth. .. [1] Similar to https://github.com/sphinx-contrib/cookiecutter, but only for the `docs/` directory? This way it can be less opinionated about everything else .. [2] At first I considered β€œmake mode”, but the current maintainers don’t know much about its original intent (see `my comment here `__ and the discussion after it) .. [3] There have been attempts at creating a `sphinx` command, see `this pull request `__ --- ### Dev/Design/Organizations Organizations ============= Currently we don't support organizations in the community site (a way to group different projects), we only support individual accounts. Several integrations that we support like GitHub and Bitbucket have organizations, where users group their repositories and manage them in groups rather than individually. Why move organizations in the community site? --------------------------------------------- We support organizations in the commercial site, having no organizations in the community site makes the code maintenance difficult for Read the Docs developers. Having organizations in the community site will make the differences between both more easy to manage. Users from the community site can have organizations in external sites from where we import their projects (like GitHub, Gitlab). Currently users have all projects from different organizations in their account. Having not a clear way to group/separate those. We are going to first move the code, and after that enable the feature on the community site. How are we going to support organizations? ------------------------------------------ Currently only users can own projects in the community site. With organizations this is going to change to: Users and organizations can own projects. With this, the migration process would be straightforward for the community site. For the commercial site we are only to allow organizations to own projects for now (since the we have only subscriptions per organizations). What features of organizations are we going to support? ------------------------------------------------------- We have the following features in the commercial site that we don't have on the community site: - Owners - Teams - Permissions - Subscriptions Owners should be included to represent owners of the current organization. Teams, this is also handy to manage access to different projects under the same organization. Permissions, currently we have two type of permissions for teams: admin and read only. Read only permissions doesn't make sense in the community site since we only support public projects/versions (we do support private versions now, but we are planning to remove those). So, we should only support admin permissions for teams. Subscriptions, this is only valid for the corporate site, since we don't charge for use in the community site. How to migrate current projects ------------------------------- Since we are not replacing the current implementation, we don't need to migrate current projects from the community site nor from the corporate site. How to migrate the organizations app ------------------------------------ The migration can be split in: #. Remove/simplify code from the organizations app on the corporate site. #. Isolate/separate models and code that isn't going to be moved. #. Start by moving the models, managers, and figure out how to handle migrations. #. Move the rest of the code as needed. #. Activate organizations app on the community site. #. Integrate the code from the community site to the new code. #. UI changes We should start by removing unused features and dead code from the organizations in the corporate site, and simplify existing code if possible (some of this was already done). Isolate/separate the models to be moved from the ones that aren't going to be moved. We should move the models that aren't going to me moved to another app. - Plan - PlanFeature - Subscription This app can be named *subscriptions*. We can get around the table names and migrations by setting the explicitly the table name to ``organizations_``, and doing a fake migration. Following suggestions in https://stackoverflow.com/questions/48860227/moving-multiple-models-from-one-django-app-to-another, that way we avoid having any downtime during the migration and any inconvenient caused from renaming the tables manually. Code related to subscriptions should be moved out from the organizations app. After that, it should be easier to move the organizations *app* (or part of it) to the community site (and no changes to table names would be required). We start by moving the models. - Organization - OrganizationOwner - Team - TeamInvite - TeamMember Migrations aren't moved, since all current migrations depend on other models that aren't going to be moved. In the community site we run an initial migration, for the corporate site we run a fake migration. The migrations left from the commercial site can be removed after that. For managers and querysets that depend on subscriptions, we can use our pattern to make overridable classes (inheriting from ``SettingsOverrideObject``). Templates, urls, views, forms, notifications, signals, tasks can be moved later (we just need to make use of the models from the ``readthedocs.organizations`` module). If we decide to integrate organizations in the community site, we can add/move the UI elements and enable the app. After the app is moved, we can move more code that depends on organizations to the community site. Namespace --------- Currently we use the project's slug as namespace, in the commercial site we use the combination of ``organization.slug`` + ``project.slug`` as namespace, since in the corporate site we don't care so much about a unique namespace between all users, but a unique namespace per organization. For the community site probably this approach isn't the best, since we always serve docs publicly from ``slug.readthedocs.io``. And most of the users don't have a custom domain. The corporate site will use ``organization.slug`` + ``project.slug`` as slug, And the community site will always use ``project.slug`` as slug, even if the project belongs to an organization. We need to refactor the way we get the namespace to be more easy to manage in both sites. Future Changes -------------- Changes that aren't needed immediately after the migration, but that should be done: - UI for organizations in the community site. - Add new endpoints to the API (v3 only). - Make the relationship between the models ``Organization`` and ``Project`` one to many (currently many to many). --- ### Dev/Design/Pr Builder Design of pull request builder ============================== Background ---------- This will focus on automatically building documentation for Pull Requests on Read the Docs projects. This is one of the most requested feature of Read the Docs. This document will serve as a design document for discussing how to implement this features. Scope ----- - Making Pull Requests work like temporary ``Version`` - Excluding PR Versions from Elasticsearch Indexing - Adding a ``PR Builds`` Tab in the Project Dashboard - Updating the Footer API - Adding Warning Banner to Docs - Serving PR Docs - Excluding PR Versions from Search Engines - Receiving ``pull_request`` webhook event from Github - Fetching data from pull requests - Storing PR Version build Data - Creating PR Versions when a pull request is opened and Triggering a build - Triggering Builds on new commits on a PR - Status reporting to Github Fetching data from pull requests -------------------------------- We already get Pull request events from Github webhooks. We can utilize that to fetch data from pull requests. when a ``pull_request`` event is triggered we can fetch the data of that pull request. We can fetch the pull request by doing something similar to travis-ci. ie: ``git fetch origin +refs/pull//merge:`` Modeling pull requests as a type of version ------------------------------------------- Pull requests can be Treated as a Type of Temporary ``Version``. We might consider adding a ``VERSION_TYPES`` to the ``Version`` model. - If we go with ``VERSION_TYPES`` we can add something like ``pull_request`` alongside Tag and Branch. We should add ``Version`` and ``Build`` Model Managers for PR and Regular Versions and Builds. The proposed names for PR and Regular Version and Build Managers are ``external`` and ``internal``. We can then use ``Version.internal.all()`` to get all regular versions, ``Version.external.all()`` to get all PR versions. We can then use ``Build.internal.all()`` to get all regular version builds, ``Build.external.all()`` to get all PR version builds. Excluding PR versions from Elasticsearch indexing ------------------------------------------------- We should exclude to PR Versions from being Indexed to Elasticsearch. We need to update the queryset to exclude PR Versions. Adding a PR builds tab in the project dashboard ----------------------------------------------- We can add a Tab in the project dashboard that will listout the PR Builds of that project. We can name it ``PR Builds``. Creating versions for pull requests ----------------------------------- If the Github webhook event is ``pull_request`` and action is ``opened``, this means a pull request was opened in the projects repository. We can create a ``Version`` from the Payload data and trigger a initial build for the version. A version will be created whenever RTD receives an event like this. Triggering build for new commits in a pull request -------------------------------------------------- We might want to trigger a new build for the PR version if there is a new commit on the PR. If the Github webhook event is ``pull_request`` and action is ``synchronize``, this means a new commit was added to the pull request. Status reporting to GitHub -------------------------- We could send build status reports to Github. We could send if the build was Successful or Failed. We can also send the build URL. By this we could show if the build passed or failed on Github something like travis-ci does. As we already have the ``repo:status`` scope on our OAuth App, we can send the status report to Github using the Github Status API. Sending the status report would be something like this: .. http:post:: /repos/:owner/:repo/statuses/:sha .. code:: json { "state": "success", "target_url": "", "description": "The build succeeded!", "context": "continuous-documentation/read-the-docs" } Storing pull request docs ------------------------- We need to think about how and where to store data after a PR Version build is finished. We can store the data in a blob storage. Excluding PR versions from search engines ----------------------------------------- We should Exclude the PR Versions from Search Engines, because it might cause problems for RTD users. As users might land to a pull request doc but not the original Project Docs. This will cause confusion for the users. Serving PR docs --------------- We need to think about how we want to serve the PR Docs. - We could serve the PR Docs from another Domain. - We could serve the PR Docs using ```` namespace on the same Domain. - Using ``pr-`` as the version slug ``https://.readthedocs.io//pr-/`` - Using ``pr`` subdomain ``https://pr..readthedocs.io//`` Updating the Footer API ----------------------- We need to update the Footer API to reflect the changes. We might want to have a way to show that if this is a PR Build on the Footer. - For regular project docs we should remove the PR Versions from the version list of the Footer. - We might want to send ``is_pr`` data with the Footer API response. Adding warning banner to Docs ----------------------------- We need to add a warning banner to the PR Version Docs to let the users know that this is a Draft/PR version. We can use a sphinx extension that we will force to install on the PR Versions to add the warning banner. Related issues -------------- - `Autobuild Docs for Pull Requests`_ - `Add travis-ci style pull request builder`_ .. _Autobuild Docs for Pull Requests: https://github.com/readthedocs/readthedocs.org/issues/5684 .. _Add travis-ci style pull request builder: https://github.com/readthedocs/readthedocs.org/issues/1340 --- ### Dev/Design/Privacy Levels Privacy levels ============== This document describes how to handle and unify privacy levels on the community and commercial version of Read the Docs. Current state ------------- Currently, we have three privacy levels for projects and versions: #. Public #. Private #. Protected (currently hidden) These levels of privacy aren't clear and bring confusion to our users. Also, the private level doesn't makes sense on the community site, since we only support public projects. Places where we use the privacy levels are: - On serving docs - Footer - Dashboard Project level privacy --------------------- Project level privacy was meant to control the dashboard visibility. This privacy level brings to confusion when users want to make a version public. We should remove all the project privacy levels. For the community site the dashboard would be always visible, and for the commercial site, the dashboard would be always hidden. The project privacy level is also used to serve the ``404.html`` page, show ``robots.txt``, and show ``sitemap.xml``. The privacy level from versions should be used instead. Some other ideas about keeping the privacy level is to dictate the default version level of new versions, but removing all other logic related to this privacy level. This can be (or is going to be) possible with automation rules, so we can just remove the field. Version level privacy --------------------- Version level privacy is mainly used to restrict access to documentation. For public level, everyone can access to the documentation. For private level, only users that are maintainers or that belong to a team with access (for the commercial site) can access to the documentation. The protected privacy level was meant to hide versions from listings and search. For the community site these versions are treated like public versions, and on the commercial site they are treated like private. The protected privacy level is currently hidden. To keep the behavior of hiding versions from listings and search, a new field should be added to the Version model and forms: ``hidden`` (`#5321 `__). The privacy level (public or private) would be respected to determine access to the documentation. For the community site, the privacy level would be public and can't be changed. The default privacy level of new versions for the commercial site would be ``private`` (this is the ``DEFAULT_PRIVACY_LEVEL`` setting). Footer ------ The footer is used to display not hidden versions that the current user has access to. For the community site no changes are required on the footer. For the commercial site we use the project level privacy to decide if show or not links to the project's dashboard: downloads, project home, and builds. Given that the project privacy level would be removed (and the dashboard is always under login), those links would never be shown, except for admin users (owners or from a team with admin access) since they are the only ones allowed to make changes on the project. Overview -------- For the community site: - The project's dashboard is visible to all users. - All versions are always public. - The footer shows links to the project's dashboard (build, downloads, home) to all users. - Only versions with ``hidden = False`` are listed on the footer and appear on search results. - If a project has a `404.html` file on the default version, it's served. - If a project has a ``robots.txt`` file on the default version, it's served. - A ``sitemap.xml`` file is always served. For the commercial site: - The project's dashboard is visible to only users that have read permission over the project. - The footer shows links to the project's dashboard (build, downloads, home) to only admin users. - Only versions with ``hidden = False`` are listed on the footer and appear on search results. - If a project has a ``404.html`` file on the default version, it's served if the user has permission over that version. - If a project has a ``robots.txt`` file on the default version, it's served if the user has permission over that version. - A ``sitemap.xml`` file is served if the user has at least one public version. And it will only list public versions. Migration --------- To differentiate between allowing or not privacy levels, we need to add a setting ``RTD_ALLOW_PRIVACY_LEVELS`` (``False`` by default). For the community and commercial site, we need to: - Remove/change code that depends on the project's privacy level. Use the global setting ``RTD_ALLOW_PRIVACY_LEVELS`` and default version's privacy level instead. - Display robots.txt - Serve 404.html page - Display sitemap.xml - Querysets - Remove `Project.privacy_level` field - Migrate all protected versions to have the attribute ``hidden = True`` (data migration), and set their privacy level to public for the community site and private for the commercial site. - Change all querysets used to list versions on the footer and on search to use the ``hidden`` attribute. - Update docs For the community site: - Hide all privacy level related settings from the version form. - Don't expose privacy levels on API v3. - Mark all versions as public. For the commercial site: - Always hide the dashboard - Show links to the dashboard (downloads, builds, project home) on the footer only to admin users. Upgrade path overview --------------------- Community site ############## The default privacy level for the community site is public for versions and the dashboard is always public. Public project (community) ~~~~~~~~~~~~~~~~~~~~~~~~~~ - Public version: Normal use case, no changes required. - Protected version: Users didn't want to list this version on the footer, but also not deactivate it. We can do a data migration of those versions to the new ``hidden`` setting and make them public. - Private version: Users didn't want to show this version to their users yet or they were testing something. This can be solved with the pull request builder feature and the ``hidden`` setting. We migrate those to public with the ``hidden`` setting. If we are worried about leaking anything from the version, we can email users before doing the change. Protected project (community) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Protected projects are not listed publicly. Probably users were hosting a WIP project, or personal public project. A public project should work for them, as we are removing listing all projects publicly (except for search). The migration path for versions of protected projects is the same as a public project. Private project (community) ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Probably these users want to use our enterprise solution instead. Or they were hosting a personal project. The migration path for versions of private projects is the same as a public project. If we are worried about leaking anything from the dashboard or build page, we can email users before doing the change. Commercial site ############### The default privacy level for the commercial site is private for versions and the dashboard is show only to admin users. Private project (commercial) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Private version: Normal usa case, not changes required. - Protected version: Users didn't want to list this version on the footer, but also not deactivate it. This can be solved by using the new ``hidden`` setting. We can do a data migration of those versions to the new ``hidden`` setting and make them private. - Public version: User has private code, but want to make public their docs. No changes required. Protected project (commercial) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ I can't think of a use case for protected projects, since they aren't listed publicly on the commercial site. The migration path for versions of protected projects is the same as a private project. Public project (commercial) ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Currently we show links back to project dashboard if the project is public, which probably users shouldn't see. With the implementation of this design doc, public versions don't have links to the project dashboard (except for admin users) and the dashboard is always under login. - Private versions: Users under the organization can see links to the dashboard. Not changes required. - Protected versions: Users under the organization can see links to the dashboard. We can do a data migration of those versions to the new ``hidden`` setting and make them private. - Public versions: All users can see links to the dashboard. Probably they have an open source project, but they still want to manage access using the same teams of the organization. Not changes are required. A breaking change here is: users outside the organization would not be able to see the dashboard of the project. --- ### Dev/Design/Redirects Improving redirects =================== Redirects are a core feature of Read the Docs, they allow users to keep old URLs working when they rename or move a page. The current implementation lacks some features and has some undefined/undocumented behaviors. .. contents:: :local: :depth: 3 Goals ----- - Improve the user experience when creating redirects. - Improve the current implementation without big breaking changes. Non-goals --------- - Replicate every feature of other services without having a clear use case for them. - Improve the performance of redirects. This can be discussed in an issue or pull request. Performance should be considered when implementing new improvements. - Allow importing redirects. We should push users to use our API instead. - Allow specifying redirects in the RTD config file. We have had several discussions around this, but we haven't reached a consensus. Current implementation ---------------------- We have five types of redirects: Prefix redirect: Allows to redirect all the URLs that start with a prefix to a new URL using the default version and language of the project. For example: a prefix redirect with the value ``/prefix/`` will redirect ``/prefix/foo/bar`` to ``/en/latest/foo/bar``. They are basically the same as an exact redirect with a wildcard at the end. They are a shortcut for a redirect like: From: ``/prefix/$rest`` To: ``/en/latest/`` Or maybe we could use a prefix redirect to replace the exact redirect with a wildcard? Page redirect: Allows to redirect a single page to a new URL using the current version and language. For example: a page redirect with the value ``/old/page.html`` will redirect ``/en/latest/old/page.html`` to ``/en/latest/new/page.html``. Cross domain redirects are not allowed in page redirects. They apply to all versions, if you want it to apply only to a specific version you can use an exact redirect. A whole directory can't be redirected with a page redirect, an exact redirect with a wildcard at the end needs to be used instead. A page redirect on a single version project is the same as an exact redirect. Exact redirect: Allows to redirect an exact URL to a new URL, it allows a wildcard at the end to redirect. For example: an exact redirect with the value ``/en/latest/page.html`` will redirect ``/en/latest/page.html`` to the new URL. If an exact redirect with the value ``/en/latest/dir/$rest`` is created, it will redirect all paths that start with ``/en/latest/dir/``, the rest of the path will be added to the new URL automatically. - Cross domain redirects are allowed in exact redirects. - They apply to all versions. - A wildcard is allowed at the end of the URL. - If a wildcard is used, the rest of the path will be added to the new URL automatically. Sphinx HTMLDir to HTML: Allows to redirect clean-URLs to HTML URLs. Useful in case a project changed the style of their URLs. They apply to all projects, not just Sphinx projects. Sphinx HTML to HTMLDir: Allows to redirect HTML URLs to clean-URLs. Useful in case a project changed the style of their URLs. They apply to all projects, not just Sphinx projects. How other services implement redirects -------------------------------------- - Gitbook implementation is very basic, they only allow page redirects. https://docs.gitbook.com/integrations/git-sync/content-configuration#redirects - Cloudflare pages allow to capture placeholders and one wildcard (in any part of the URL). They also allow you to set the status code of the redirect, and redirects can be specific in a ``_redirects`` file. https://developers.cloudflare.com/pages/platform/redirects/ They have a limit of 2100 redirects. In case of multiple matches, the topmost redirect will be used. - Netlify allows to capture placeholders and a wildcard (only allowed at the end). They also allow you to set the status code of the redirect, and redirects can be specific in a ``_redirects`` file. - Forced redirects - Match query arguments - Match by country/language and cookies - Per-domain and protocol redirects - In case of multiple matches, the topmost redirect will be used. - Rewrites, serve a different file without redirecting. https://docs.netlify.com/routing/redirects/ - GitLab pages supports the same syntax as Netlify, and supports a subset of their features: - ``_redirects`` config file - Status codes - Rewrites - Wildcards (splats) - Placeholders https://docs.gitlab.com/ee/user/project/pages/redirects.html Improvements ------------ General improvements ~~~~~~~~~~~~~~~~~~~~ The following improvements will be applied to all types of redirects. - Allow choosing the status code of the redirect. We already have a field for this, but it's not exposed to users. - Allow to explicitly define the order of redirects. This will be similar to the automation rules feature, where users can reorder the rules so the most specific ones are first. We currently rely on the implicit order of the redirects (updated_at). - Allow to disable redirects. It's useful when testing redirects, or when debugging a problem. Instead of having to re-create the redirect, we can just disable it and re-enable it later. - Allow to add a short description. It's useful to document why the redirect was created. Don't run redirects on domains from pull request previews ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We currently run redirects on domains from pull request previews, this is a problem when moving a whole project to a new domain. We don't the need to run redirects on external domains, they should be treated as temporary domains. Normalize paths with trailing slashes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Currently, if users want to redirect a path with a trailing slash and without it, they need to create two separate redirects (``/page/`` and ``/page``). We can simplify this by normalizing the path before matching it, or before saving it. For example: From: ``/page/`` To: ``/new/page`` The from path will be normalized to ``/page``, and the filename to match will also be normalized before matching it. This is similar to what Netlify does: https://docs.netlify.com/routing/redirects/redirect-options/#trailing-slash. Page and exact redirects without a wildcard at the end will be normalized, all other redirects need to be matched as is. This makes it impossible to match a path with a trailing slash. Use ``*`` and ``:splat`` for wildcards ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Currently we are using ``$rest`` at the end of the ``From URL`` to indicate that the rest of the path should be added to the target URL. A similar feature is implemented in other services using ``*`` and ``:splat``. Instead of using ``$rest`` in the URL for the suffix wildcard, we now will use ``*``, and ``:splat`` as a placeholder in the target URL to be more consistent with other services. Existing redirects can be migrated automatically. Explicit ``:splat`` placeholder ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Explicitly place the ``:splat`` placeholder in the target URL, instead of adding it automatically. Some times users want to redirect to a different path, we have been adding a query parameter in the target URL to prevent the old path from being added in the final path. For example ``/new/path/?_=``. Instead of adding the path automatically, users have to add the ``:splat`` placeholder in the target URL. For example: From: ``/old/path/*`` To: ``/new/path/:splat`` From: ``/old/path/*`` To: ``/new/path/?page=:splat&foo=bar`` Improving page redirects ~~~~~~~~~~~~~~~~~~~~~~~~ - Allow to redirect to external domains. This can be useful to apply a redirect of a well known path in all versions to another domain. For example, ``/security/`` to a their security policy page in another domain. This new feature isn't strictly needed, but it will be useful to simplify the explanation of the feature (one less restriction to explain). Example: From: ``/security/`` To: ``https://example.com/security/`` - Allow a wildcard at the end of the from path. This will allow users to migrate a whole directory to a new path without having to create an exact redirect for each version. Similar to exact redirects, users need to add the ``:splat`` placeholder explicitly. This means that page redirects are the same as exact redirects, with the only difference that they apply to all versions. Example: From: ``/old/path/*`` To: ``/new/path/:splat`` Merge prefix redirects with exact redirects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Prefix redirects are the same as exact redirects with a wildcard at the end. We will migrate all prefix redirects to exact redirects with a wildcard at the end. For example: From: ``/prefix/`` Will be migrated to: From: ``/prefix/*`` To: ``/en/latest/:splat`` Where ``/en/latest`` is the default version and language of the project. For single version projects, the redirect will be: From: ``/prefix/*`` To: ``/:splat`` Improving Sphinx redirects ~~~~~~~~~~~~~~~~~~~~~~~~~~ These redirects are useful, but we should rename them to something more general, since they apply to all types of projects, not just Sphinx projects. Proposed names: - HTML URL to clean URL redirect (``file.html`` to ``file/``) - Clean URL to HTML URL redirect (``file/`` to ``file.html``) Other ideas to improve redirects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The following improvements will not be implemented in the first iteration. - Run forced redirects before built-in redirects. We currently run built-in redirects before forced redirects, this is a problem when moving a whole project to a new domain. For example, a forced redirect like ``/$rest``, won't work for the root URL of the project, since ``/`` will first redirect to ``/en/latest/``. But shouldn't be a real problem, since users will still need to handle the ``/en/latest/file/`` paths. - Run redirects on the edge. Cloudflare allow us to create redirects on the edge, but they have some limitations around the number of redirect rules that can be created. And they will be useful for forced exact redirects only, since we can't match a redirect based on the response of the origin server. - Merge all redirects into a single type. This may simplify the implementation, but it will make it harder to explain the feature to users. And to replace some redirects we need to implement some new features. - Placeholders. I haven't seen users requesting this feature. We can consider adding it in the future. Maybe we can expose the current language and version as placeholders. - Per-protocol redirects. We should push users to always use HTTPS. - Allow a prefix wildcard. We currently only allow a suffix wildcard, adding support for a prefix wildcard should be easy. But do users need this feature? - Per-domain redirects. The main problem that originated this request was that we were applying redirects on external domains, if we stop doing that, there is no need for this feature. We can also try to improve how our built-in redirects work (specially our canonical domain redirect). Allow matching query arguments ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ We can do this in three ways: - At the DB level with some restrictions. If done at the DB level, we would need to have a different field with just the path, and other with the query arguments normalized and sorted. For example, if we have a redirect with the value ``/foo?blue=1&yellow=2&red=3``, if would be normalized in the DB as ``/foo`` and ``blue=1&red=3&yellow=2``. This implies that the URL to be matched must have the exact same query arguments, it can't have more or less. I believe the implementation described here is the same being used by Netlify, since they have that same restriction. If the URL contains other parameters in addition to or instead of id, the request doesn't match that rule. https://docs.netlify.com/routing/redirects/redirect-options/#query-parameters - At the DB level using a JSONField. All query arguments will be saved normalized as a dictionary. When matching the URL, we will need to normalize the query arguments, and use some a combination of ``has_keys`` and ``contained_by`` to match the exact number of query arguments. - At the Python level. If done at the DB level, we would need to have a different field with just the path, and other with query arguments. The matching of the path would be done at the DB level, and the matching of the query arguments would be done at the Python level. Here we can be more flexible, allowing any query arguments in the matched URL. We had some performance problems in the past, but I believe it was mainly due to the use of regex instead of using string operations. And matching the path is still done at the DB level. We could limit the number of redirects that can be created with query arguments, or the number of redirects in general. We have had only one user requesting this feature, so this is not a priority. Migration --------- Most of the proposed improvements are backwards compatible, and just need a data migration to normalize existing redirects. For the exception of adding the ``$rest`` placeholder in the target URL explicitly, that needs users to re-learn how this feature works, i.e, they may be expecting to have the path added automatically in the target URL. We can create a small blog post explaining the changes. --- ### Dev/Design/Refactor Remote Repository ==================================== Refactor ``RemoteRepository`` object ==================================== This document describes the current usage of ``RemoteRepository`` objects and proposes a new normalized modeling. Goals ===== * De-duplicate data stored in our database. * Save only one ``RemoteRepository`` per GitHub repository. * Use an intermediate table between ``RemoteRepository`` and ``User`` to store associated remote data for the specific user. * Make this model usable from our SSO implementation (adding ``remote_id`` field in ``Remote`` objects). * Use Post ``JSONField`` to store associated ``json`` remote data. * Make ``Project`` connect directly to ``RemoteRepository`` without being linked to a specific ``User``. * Do not disconnect ``Project`` and ``RemoteRepository`` when a user delete/disconnects their account. Non-goals ========= * Keep ``RemoteRepository`` in sync with GitHub repositories. * Delete ``RemoteRepository`` objects deleted from GitHub. * Listen to GitHub events to detect ``full_name`` changes and update our objects. .. note:: We may need/want some of these non-goals in the future. They are just outside the scope of this document. Current implementation ====================== When a user connect their account to a social account, we create a * ``allauth.socialaccount.models.SocialAccount`` * basic information (provider, last login, etc) * provider's specific data saved in a JSON under ``extra_data`` * ``allauthsocialaccount.models.SocialToken`` * token to hit the API on behalf the user We *don't create* any ``RemoteRepository`` at this point. They are created when the user jumps into "Import Project" page and hit the circled arrows. It triggers ``sync_remote_repostories`` task in background that updates or creates ``RemoteRepositories``, but **it does not delete** them (after `#7183`_ and `#7310`_ got merged, they will be deleted). One ``RemoteRepository`` is created per repository the ``User`` has access to. .. note:: In corporate, we are automatically syncing ``RemoteRepository`` and ``RemoteOganization`` at signup (foreground) and login (background) via a signal. We should eventually move these to community. .. _sync_remote_repositoies: https://github.com/readthedocs/readthedocs.org/blob/56253cb786945c9fe53a034a4433f10672ae8a4f/readthedocs/oauth/tasks.py#L25-L43 .. _#7183: https://github.com/readthedocs/readthedocs.org/pull/7183 .. _#7310: https://github.com/readthedocs/readthedocs.org/pull/7310 Where ``RemoteRepository`` is used? =================================== * List of available repositories to import under "Import Project" * Show a "+", "External Arrow" or a "Lock" sign next to the element in the list * +: it's available to be imported * External Arrow: the repository is already imported (see `RemoteRepository.matches`_ method) * Lock: user doesn't have (admin) permissions to import this repository (uses ``RemoteRepository.private`` and ``RemoteRepository.admin``) * Avatar URL in the list of project available to import * `Update webhook`_ when user clicks "Resync webhook" from the :guilabel:`Admin` > :guilabel:`Integrations` tab * `Send build status`_ when building Pull Requests .. _RemoteRepository.matches: https://github.com/readthedocs/readthedocs.org/blob/56253cb786945c9fe53a034a4433f10672ae8a4f/readthedocs/oauth/models.py#L182-L204 .. _Update webhook: https://github.com/readthedocs/readthedocs.org/blob/56253cb786945c9fe53a034a4433f10672ae8a4f/readthedocs/oauth/utils.py#L26-L62 .. _Send build status: https://github.com/readthedocs/readthedocs.org/blob/56253cb786945c9fe53a034a4433f10672ae8a4f/readthedocs/projects/tasks.py#L1852-L1956 New normalized implementation ============================= The ``ManyToMany`` relation ``RemoteRepository.users`` will be changed to be ``ManyToMany(through='RemoteRelation')`` `to add extra fields in the relation`_ that are specific only for the User. Allows us to have *only one* ``RemoteRepository`` per GitHub repository with multiple relationships to ``User``. .. _to add extra fields in the relation: https://docs.djangoproject.com/en/2.2/topics/db/models/#extra-fields-on-many-to-many-relationships With this modeling, we can avoid the disconnection ``Project`` and ``RemoteRepository`` only by removing the ``RemoteRelation``. .. note:: All the points mentioned in the previous section may need to be adapted to use the new normalized modeling. However, it may be only field renaming or small query changes over new fields. Use this modeling for SSO ------------------------- We can get the list of ``Project`` where a user as access: .. code-block:: python admin_remote_repositories = RemoteRepository.objects.filter( users__contains=request.user, users__remoterelation__admin=True, # False for read-only access ) Project.objects.filter(remote_repository__in=admin_remote_repositories) Rollout plan ============ Due the constraints we have in the ``RemoteRepository`` table and its size, we can't just do the data migration at the same time of the deploy. Because of this we need to be more creative here and find a way to re-sync the data from VCS providers, while the site continue working. To achieve this, we thought on following this steps: 1. modify all the Python code to use the new modeling in .org and .com (will help us to find out bugs locally in an easier way) 1. QA this locally with test data 1. enable Django signal to re-sync RemoteRepository on login async (we already have this in .com). New active users will have updated data immediately 1. spin up a new instance with the new refactored code 1. run migrations to create a new table for `RemoteRepository` 1. re-sync everything from VCS providers into the new table for 1-week or so 1. dump-n-load `Project - RemoteRepository` relations 1. create a migration to use the new table with synced data 1. deploy new code once the sync is finished See these issues for more context: * https://github.com/readthedocs/readthedocs.org/pull/7536#issuecomment-724102640 * https://github.com/readthedocs/readthedocs.org/pull/7675#issuecomment-732756118 --- ### Dev/Design/Secure Api Access From Builders Secure API access from builders =============================== Goals ----- - Provide a secure way for builders to access the API. - Limit the access of the tokens to the minimum required. Non-goals --------- - Migrate builds to use API V3 - Implement this mechanism in API V3 - Expose it to users All these changes can be made in the future, if needed. Current state ------------- Currently, we access the API V2 from the builders using the credentials of the "builder" user. This user is a superuser, it has access to all projects, write access to the API, access to restricted endpoints, and restricted fields. The credentials are hardcoded in our settings file, so if there is a vulnerability that allows users to have access to the settings file, the attacker will have access to the credentials of the "builder" user, giving them full access to the API and all projects. Proposed solution ----------------- Instead of using the credential of a super user to access the API, we will create a temporal token attached to a project, and one of the owners of the project. This way this token will have access to the given project only for a limited period of time. This token will be generated from the webs, and passed to the builders via the celery task, where it can be used to access the API. Once the build has finished, this token will be revoked. Technical implementation ------------------------ We will use the rest-knox_ package, this package is recommended by the DRF documentation, since the default token implementation of DRF is very basic, some relevant features of knox are: - Support for several tokens per user. - Tokens are stored in a hashed format in the database. We don't have access the tokens after they are created. - Tokens can have an expiration date. - Tokens can be created with a prefix (rtd_xxx) (unreleased) - Support for custom token model (unreleased) We won't expose the token creation view directly, since we can create the tokens from the webs, and this isn't exposed to users. The view to revoke the token will be exposed, since we need it to revoke the token once the build has finished. From the API, we just need to add the proper permission and authentication classes to the views we want to support. To differentiate from a normal user and a token authed user, we will have access to the token via the ``request.auth`` attribute in the API views, this will also be used to get the attached projects to filter the querysets. The knox package allows us to provide our own token model, this will be useful to add our own fields to the token model. Fields like the projects attached to the token, or access to all projects the user has access to, etc. .. _rest-knox: https://james1345.github.io/django-rest-knox/ Flow ---- The flow of creation and usage of the token will be: - Create a token from the webs when a build is triggered. The triggered project will be attached to the token, if the build was triggered by a user, that user will be attached to the token, otherwise the token will be attached to one of the owners of the project. - The token will be created with an expiration date of 3 hours, this should be enough for the build to finish. We could also make this dynamic depending of the project. - Pass the token to the builder via the celery task. - Pass the token to all places where the API is used. - Revoke the token when the build has finished. This is done by hitting the revoke endpoint. - In case the revoke endpoint fails, the token will expire in 3 hours. Why attach tokens to users? --------------------------- Attaching tokens to users will ease the implementation, since we can reuse the code from knox package. Attaching tokens to projects only is possible, but it will require to manage the authentication manually. This is since Knox requires a user to be attached to the token, and this user is used in their ``TokenAuthentication`` class. An alternative is to use the DRF API key package, which doesn't require a user, but then if we wanted to extend this functionality to our normal APIs, we will have to implement the authentication manually. Keeping backwards compatibility ------------------------------- Access to write API V2 is restricted to superusers, and was used only from the builders. So we don't need to keep backwards compatibility for authed requests, but we need to keep the old implementation working while we deploy the new one. Possible issues --------------- Some of the features that we may need are not released yet, we need the custom token model feature, specially. There is a race condition when using the token, and the user that is attached to that token is removed from the project. This is, if the user is removed while the build is running, the builders won't be able to access the API. We could avoid this by not relying on the user attached to the token, only on the projects attached to it (this would be for our build APIs only). Alternative implementation with Django REST Framework API Key ------------------------------------------------------------- Instead of using knox, we can use `DRF API key`_, it has the same features as knox, with the exception of: - It is only used for authorization, it can't be used for authentication (or it can't be out of the box). - It doesn't expose views to revoke the tokens (but this should be easy to manually implement) - Changing the behaviour of some things require sub-classing instead of defining settings. - It supports several token models (not just one like knox). - All features that we need are already released. The implementation will be very similar to the one described for knox, with the exception that tokens won't be attached to users, but just a project. And we won't be needing to handle authentication, since the token itself will grant access to the projects. To avoid breaking builders, we need to be able to make the old and the new implementation work together, this is, allow authentication and handle tokens at the same time. This means passing valid user credentials together with the token, this "feature" can be removed in the next deploy (with knox we also need to handle both implementations, but it doesn't require passing credentials with the token, since it also handles authentication). .. _DRF API key: https://florimondmanca.github.io/djangorestframework-api-key/ Decision -------- Because the required features from knox are not released yet, we have decided to use DRF API key instead. Future work ----------- This work can be extended to API V3, and be exposed to users in the future. We only need to take into consideration that the token model will be shared by both, API V2 and API V3 if using knox, if we use API key, we can have different token models for each use case. --- ### Dev/Design/Sphinx Jquery sphinxcontrib-jquery ==================== jQuery will be removed from Sphinx 6.0.0. We can expect 6.0.0 to ship in late 2022. .. seealso:: * https://github.com/sphinx-doc/sphinx/issues/7405 * https://github.com/sphinx-doc/sphinx/issues/10070 * https://github.com/readthedocs/sphinx-hoverxref/issues/160 * https://github.com/readthedocs/sphinx_rtd_theme/issues/1253 * https://github.com/pydata/pydata-sphinx-theme/issues/764 This is a "request for comments" for a community-owned Sphinx extension that bundles jQuery. Overview -------- Comment deadline: November 1st, 2022 Package-name: ``sphinxcontrib-jquery`` Python package: ``sphinxcontrib.jquery`` Dependencies: Python 3+, Sphinx 1.8+ (or perhaps no lower bound?) Ownership: Read the Docs core team will implement the initial releases of an otherwise community-owned package that lives in https://github.com/sphinx-contrib/jquery Functionality: sphinxcontrib-jquery is a Sphinx extension that provides a simple mechanism for other Sphinx extensions and themes to ensure that jQuery is included into the HTML build outputs and loaded in the HTML DOM itself. More specifically, the extension ensures that jQuery is loaded exactly once no matter how many themes and extensions that request to include jQuery nor the version of Sphinx. Scope: This extension assumes that it's enough to provide a single version of jQuery for all of its dependent extensions and themes. As the name implies, this extension is built to handle jQuery only. It's not a general asset manager and it's not looking to do dependency resolution of jQuery versions. Usage ----- The primary users of this package are **theme and extension developers** and **documentation project owners**. Theme and extension developers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The following 2 steps need to be completed: #. A Sphinx theme or extension should depend on the python package ``sphinxcontrib-jquery``. #. In your extension's or theme's ``setup(app)``, call ``app.setup_extension("sphinxcontrib.jquery")``. In addition to this, we recommend extension and theme developers to log to the browser's ``console.error`` in case jQuery isn't found. The log message could for instance say:: if (typeof $ == "undefined") console.error(" depends on sphinxcontrib-jquery. Please ensure that .setup(app) is called or add 'sphinxcontrib-jquery' to your conf.py extensions setting.") Documentation project owners ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you are depending on a theme or extension that did not itself address the removal of jQuery from Sphinx 6, you can patch up your project like this: #. Add ``sphinxcontrib-jquery`` to your installed dependencies. #. Add ``sphinxcontrib.jquery`` to your ``extensions`` setting in ``conf.py``. Calling ``app.setup_extension("sphinxcontrib.jquery")`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When a Sphinx theme or extension calls `setup_extension() `_, a call to ``sphinxcontrib.jquery.setup(app)`` will happen. Adding ``sphinxcontrib.jquery`` to a documentation project's ``conf.extensions`` will also call ``sphinxcontrib.jquery.setup(app)`` (at most once). In ``sphinxcontrib.jquery.setup(app)``, jQuery is added. The default behaviour is to detect the Sphinx version and include jQuery via `app.add_js_file `__ when Sphinx is from version 6 and up. jQuery is added at most once. Config value: ``jquery_force_enable`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When setting ``jquery_force_enable=True``, jQuery is added no matter the Sphinx version, but at most once. This is useful if you want to handle alternative conditions for adding jQuery. .. warning:: If you set ``jquery_force_enable=True``, you most likely should also add ``Sphinx>=6`` to your theme's/extension's dependencies since versions before this already bundles jQuery! jQuery version and inclusion ---------------------------- jQuery should be shipped together with the Python package and not be referenced from a CDN. Sphinx has kept relatively up to date with jQuery, and this package intends to follow. The most recently bundled jQuery version was v3.5.1 and only two releases have happened since: 3.6.0 and 3.6.1. The 3.6.0 release had a very small backwards incompatibility which illustrates how harmless these upgrades are for the general purpose Sphinx package. Therefore, we propose to start the release of ``sphinxcontrib-jquery`` at 3.5.1 (the currently shipped version) and subsequently release 3.6.1 in an update. This will give users that need 3.5.1 a choice of a lower version. The bundled jQuery version will be NPM pre-minified and distributed together with the PyPI package. The minified jQuery JS file is ultimately included by calling `app.add_js_file `__, which is passed the following arguments: .. code:: python app.add_js_file( get_jquery_url_path(), loading_method="defer", priority=200, integrity="sha256-{}".format(get_jquery_sha256_checksum()), ) .. note:: It's possible to include jQuery in other ways, but this ultimately doesn't require this extension and is therefore not supported. --- ### Dev/Design/System Packages Allow installation of system packages ===================================== Currently we don't allow executing arbitrary commands in the build process. The more common use case is to install extra dependencies. .. contents:: :local: :depth: 3 Current status -------------- There is a workaround when using Sphinx to run arbitrary commands, this is executing the commands inside the ``conf.py`` file. There isn't a workaround for MkDocs, but this problem is more common in Sphinx, since users need to install some extra dependencies in order to use autodoc or build Jupyter Notebooks. However, installation of some dependencies require root access, or are easier to install using ``apt``. Most of the CI services allow to use ``apt`` or execute any command with ``sudo``, so users are more familiar with that workflow. Some users use Conda instead of pip to install dependencies in order to avoid these problems, but not all pip users are familiar with Conda, or want to migrate to Conda just to use Read the Docs. Security concerns ----------------- Builds are run in a Docker container, but the app controlling that container lives in the same server. Allowing to execute arbitrary commands with super user privileges may introduce some security issues. Exposing ``apt install`` ------------------------ For the previous reasons we won't allow to execute arbitrary commands with root (yet), but instead allow only to install extra packages using ``apt``. We would expose this through the config file. Users will provide a list of packages to install, and under the hook we would run: - ``apt update -y`` - ``apt install -y {packages}`` These commands will be run before the Python setup step and after the clone step. .. note:: All package names must be validated to avoid injection of extra options (like ``-v``). Using ``docker exec`` --------------------- Currently we use ``docker exec`` to execute commands in a running container. This command also allows to pass a user which is used to run the commands (`#8058`_). We can run the ``apt`` commands in our current containers using a super user momentarily. .. _#8058: https://github.com/readthedocs/readthedocs.org/pull/8058 Config file ----------- The config file can add an additional mapping ``build.apt_packages`` to a list of packages to install. .. code-block:: yaml version: 2 build: apt_packages: - cmatrix - mysql-server .. note:: Other names that were considered were: - ``build.packages`` - ``build.extra_packages`` - ``build.system_packages`` These were rejected to avoid confusion with existing keys, and to be explicit about the type of package. Possible problems ----------------- - Some users may require to pass some additional flags or install from a ppa. - Some packages may require some additional setup after installation. Other possible solutions ------------------------ - We can allow to run the containers as root doing something similar to what Travis does: They have one tool to convert the config file to a shell script (travis-build_), and another that spins a docker container, executes that shell script and streams the logs back (travis-worker_). .. _travis-build: https://github.com/travis-ci/travis-build .. _travis-worker: https://github.com/travis-ci/worker - A similar solution could be implemented using `AWS Lambda`_. .. NOTE: Haven't done much research around this, but I remember David mentioned this a time ago. .. _AWS Lambda: https://aws.amazon.com/lambda/ This of course would require a large amount of work, but may be useful for the future. --- ### Dev/Design/Telemetry Collect data about builds ========================= We may want to take some decisions in the future about deprecations and supported versions. Right now we don't have data about the usage of packages and their versions on Read the Docs to be able to make an informed decision. .. contents:: :local: :depth: 3 Tools ----- Kibana: - https://www.elastic.co/kibana - We can import data from ES. - Cloud service provided by Elastic. Superset: - https://superset.apache.org/ - We can import data from several DBs (including postgres and ES). - Easy to setup locally, but doesn't look like there is cloud provider for it. Metabase: - https://www.metabase.com/ - We can import data from several DBs (including postgres). - Cloud service provided by Metabase. Summary: We have several tools that can inspect data form a postgres DB, and we also have ``Kibana`` that works *only* with ElasticSearch. The data to be collected can be saved in a postgres or ES database. Currently, we are making use of Metabase to get other information, so it's probably the right choice for this task. Data to be collected -------------------- The following data can be collected after installing all dependencies. Configuration file ~~~~~~~~~~~~~~~~~~ We are saving the config file in our database, but to save some space we are saving it only if it's different than the one from a previous build (if it's the same we save a reference to it). The config file being saved isn't the original one used by the user, but the result of merging it with its default values. We may also want to have the original config file, so we know which settings users are using. PIP packages ~~~~~~~~~~~~ We can get a json with all and root dependencies with ``pip list``. This will allow us to have the name of the packages and their versions used in the build. .. code-block:: $ pip list --pre --local --format json | jq # and $ pip list --pre --not-required --local --format json | jq [ { "name": "requests-mock", "version": "1.8.0" }, { "name": "requests-toolbelt", "version": "0.9.1" }, { "name": "rstcheck", "version": "3.3.1" }, { "name": "selectolax", "version": "0.2.10" }, { "name": "slumber", "version": "0.7.1" }, { "name": "sphinx-autobuild", "version": "2020.9.1" }, { "name": "sphinx-hoverxref", "version": "0.5b1" }, ] With the ``--not-required`` option, pip will list only the root dependencies. Conda packages ~~~~~~~~~~~~~~ We can get a json with all dependencies with ``conda list --json``. That command gets all the root dependencies and their dependencies (there is no way to list only the root dependencies), so we may be collecting some noise, but we can use ``pip list`` as a secondary source. .. code-block:: $ conda list --json --name conda-env [ { "base_url": "https://conda.anaconda.org/conda-forge", "build_number": 0, "build_string": "py_0", "channel": "conda-forge", "dist_name": "alabaster-0.7.12-py_0", "name": "alabaster", "platform": "noarch", "version": "0.7.12" }, { "base_url": "https://conda.anaconda.org/conda-forge", "build_number": 0, "build_string": "pyh9f0ad1d_0", "channel": "conda-forge", "dist_name": "asn1crypto-1.4.0-pyh9f0ad1d_0", "name": "asn1crypto", "platform": "noarch", "version": "1.4.0" }, { "base_url": "https://conda.anaconda.org/conda-forge", "build_number": 3, "build_string": "3", "channel": "conda-forge", "dist_name": "python-3.5.4-3", "name": "python", "platform": "linux-64", "version": "3.5.4" } ] APT packages ~~~~~~~~~~~~ We can get the list from the config file, or we can list the packages installed with ``dpkg --get-selections``. That command would list all pre-installed packages as well, so we may be getting some noise. .. code-block:: console $ dpkg --get-selections adduser install apt install base-files install base-passwd install bash install binutils install binutils-common:amd64 install binutils-x86-64-linux-gnu install bsdutils install build-essential install We can get the installed version with: .. code-block:: console $ dpkg --status python3 Package: python3 Status: install ok installed Priority: optional Section: python Installed-Size: 189 Maintainer: Ubuntu Developers Architecture: amd64 Multi-Arch: allowed Source: python3-defaults Version: 3.8.2-0ubuntu2 Replaces: python3-minimal (<< 3.1.2-2) Provides: python3-profiler Depends: python3.8 (>= 3.8.2-1~), libpython3-stdlib (= 3.8.2-0ubuntu2) Pre-Depends: python3-minimal (= 3.8.2-0ubuntu2) Suggests: python3-doc (>= 3.8.2-0ubuntu2), python3-tk (>= 3.8.2-1~), python3-venv (>= 3.8.2-0ubuntu2) Description: interactive high-level object-oriented language (default python3 version) Python, the high-level, interactive object oriented language, includes an extensive class library with lots of goodies for network programming, system administration, sounds and graphics. . This package is a dependency package, which depends on Debian's default Python 3 version (currently v3.8). Homepage: https://www.python.org/ Original-Maintainer: Matthias Klose Or with .. code-block:: console $ apt-cache policy python3 Installed: 3.8.2-0ubuntu2 Candidate: 3.8.2-0ubuntu2 Version table: *** 3.8.2-0ubuntu2 500 500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages 100 /var/lib/dpkg/status Python ~~~~~~ We can get the Python version from the config file when using a Python environment, and from the ``conda list`` output when using a Conda environment. OS ~~ We can infer the OS version from the build image used in the config file, but since it changes with time, we can get it from the OS itself: .. code-block:: $ lsb_release --description Description: Ubuntu 18.04.5 LTS # or $ cat /etc/issue Ubuntu 18.04.5 LTS \n \l Format ~~~~~~ The final information to be saved would consist of: - organization: the organization id/slug - project: the project id/slug - version: the version id/slug - build: the build id, date, length, status. - user_config: Original user config file - final_config: Final configuration used (merged with defaults) - packages.pip: List of pip packages with name and version - packages.conda: List of conda packages with name, channel, and version - packages.apt: List of apt packages - python: Python version used - os: Operating system used .. code-block:: json { "organization": { "id": 1, "slug": "org" }, "project": { "id": 2, "slug": "docs" }, "version": { "id": 1, "slug": "latest" }, "build": { "id": 3, "date/start": "2021-04-20-...", "length": "00:06:34", "status": "normal", "success": true, "commit": "abcd1234" }, "config": { "user": {}, "final": {} }, "packages": { "pip": [{ "name": "sphinx", "version": "3.4.5" }], "pip_all": [ { "name": "sphinx", "version": "3.4.5" }, { "name": "docutils", "version": "0.16.0" } ], "conda": [{ "name": "sphinx", "channel": "conda-forge", "version": "0.1" }], "apt": [{ "name": "python3-dev", "version": "3.8.2-0ubuntu2" }], }, "python": "3.7", "os": "ubuntu-18.04.5" } Storage ------- All this information can be collected after the build has finished, and we can store it in a dedicated database (telemetry), using Django's models. Since this information isn't sensitive, we should be fine saving this data even if the project/version is deleted. As we don't care about historical data, we can save the information per-version and from their latest build only. And delete old data if it grows too much. Should we make heavy use of JSON fields? Or try to avoid nesting structures as possible? Like config.user/config.final vs user_config/final_config. Or having several fields in our model instead of just one big json field? ---