memray Documentation

python3 -m pip install memray

export CPPFLAGS="-I$(brew --prefix lz4)/include" LDFLAGS="-L$(brew --prefix lz4)/lib -Wl,-rpath,$(brew --prefix lz4)/lib"

export MACOSX_DEPLOYMENT_TARGET=10.14

export MACOSX_DEPLOYMENT_TARGET=11.0

git clone [email protected]:bloomberg/memray.git memray
cd memray
python3 -m venv ../memray-env/  # just an example, put this wherever you want
source ../memray-env/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -e . --group test --group extra

prek install

/* Detailed source-code truncated for AI context efficiency. */

python3 -m memray run my_script.py

python3 -m memray run -m my_module

memray run my_script.py
memray run -m my_module

memray flamegraph memray-my_script.py.2369.bin

pytest --memray tests/

/* Detailed source-code truncated for AI context efficiency. */

@pytest.mark.limit_memory("24 MB")
def test_foobar():
    # do some stuff that allocates memory

memray run --native my_script.py

memray run --live my_script.py

memray run --live -m my_module

import memray

with memray.Tracker("output_file.bin"):
    print("Allocations will be tracked until the with block ends")

Signed-Off-By: Random J. Developer <[email protected]>

.. autoattribute:: memray.FileFormat.ALL_ALLOCATIONS
   :annotation:

Record every allocation that the tracked process performs. This is the
default format. The produced capture files may be very large if the process
performs many allocations. This is the only format that allows detecting
:doc:`temporary allocations </temporary_allocations>` or using the
:doc:`stats reporter <stats>`.

.. autoattribute:: memray.FileFormat.AGGREGATED_ALLOCATIONS
   :annotation:

For every location where the tracked process performed any allocations, the
capture file includes a count of:

- How many allocations at that location had not yet been deallocated when
  the process reached its heap memory high water mark
- How many bytes had been allocated at that location and not yet
  deallocated when the process reached its heap memory high water mark
- How many allocations at that location were leaked (i.e. not deallocated
  before tracking stopped)
- How many bytes were leaked by allocations at that location

You cannot find :doc:`temporary allocations </temporary_allocations>` using
this capture file format, since finding temporary allocations requires
knowing when each allocation was deallocated, and that information is lost
by the aggregation. You also cannot use the :doc:`stats reporter <stats>`
with this capture file format, because it needs to see every allocation's
size to compute its statistics.

Additionally, if the process is killed before tracking ends (for instance,
by the Linux OOM killer), then no useful information is ever written to the
capture file, because aggregation was still happening inside the process
when it died.

If you can live with these limitations, then ``AGGREGATED_ALLOCATIONS``
results in much smaller capture files that can be used seamlessly with most
reporters.

memray attach [options]

A flame graph may contain the text of source lines from the files that were
executing when memory was allocated. If you share a flame graph with
someone, you may be sharing source code with them. Don't share a Memray
flame graph with anyone you wouldn't share the underlying source code with.
Note that a malicious capture file could be crafted to include lines from
arbitrary files on your system!

These heuristics are best-effort and may be wrong in either direction.
A Python file can still contain hardcoded credentials, and a world-readable
file can still contain secrets. Even with ``--confidential-files=all``,
file names, function names, and line numbers are always included.

The Python allocator doesn't necessarily release memory to the system
when Python objects are deallocated and these can still appear as
"leaks". When you use the ``--leaks`` option, you should usually
also run your application with the ``PYTHONMALLOC=malloc``
environment variable set. See our documentation on :doc:`python
allocators ` for details.

Temporal flame graphs contain much more data than the default non-temporal
flame graphs, so they're slower to generate, and the generated HTML files
are also larger. They can't be generated from :ref:`aggregated capture
files <aggregated capture files>`, which don't contain the necessary data
about allocations over time. They also can't be used for finding
:doc:`temporary allocations </temporary_allocations>`.

memray flamegraph [options] <results>

--leaks : @replace
    Enables :ref:`memory-leaks-view`, where memory that was not deallocated is displayed, instead of peak memory
    usage.

--split-threads : @replace
    Enables :ref:`split-threads-view`, where each thread can be displayed separately. Allocations on the same source
    line across different threads are not merged, if this flag is passed.

When installing Memray with ``pip`` you need to install it with the
Python interpreter you intend to run your profiled application with. In
this example we're installing it for use with ``python3``:

.. code:: shell

    python3 -m pip install memray

Using the CLI
-------------

You can invoke Memray the following way:

.. code:: shell

  python3 -m memray

Or alternatively through the ``memray`` script:

.. code:: shell

  memray

The downside to using the ``memray`` script is that it's not immediately
clear what Python interpreter will be used to execute Memray. If you're using
a virtual environment that's not a problem because you know exactly what interpreter is
in use, but otherwise you need to be careful to ensure that ``memray`` is
running with the interpreter you meant to use.

Profiling with Memray should be done in two steps:

1. Run the application to track allocations and deallocations and save the results
2. Generate the desired report from the captured results

Running the Analysis
--------------------

To run memray on the ``example.py`` script, use :doc:`the run subcommand <run>`.

.. code:: shell

  memray run example.py

This will execute the script and track its memory allocations, displaying the name of the file where results are being recorded with a message like:

.. code:: text

  Writing profile results into memray-example.py.4131.bin

Generating a Flame Graph
------------------------

To generate a flame graph displaying memory usage across the application, we can run ``memray flamegraph`` and specify
the results file:

.. code:: shell

  memray flamegraph memray-example.py.4131.bin

This will generate the ``memray-flamegraph-example.py.4131.html`` file in the current directory. See the :doc:`flamegraph`
documentation which explains how to interpret flame graphs.

Next Steps
----------

The "Hands-on Tutorial" section of our sidebar includes :doc:`a set of lessons <tutorials/index>` you can use to
practice working with Memray by debugging example Python applications with surprising memory allocation behavior. You
can also try Memray out on our :doc:`example applications <examples/README>`.

If you instead want to jump directly into debugging one of your own applications, the "Concepts" section of our sidebar
gives background information to help you use Memray more effectively. Reading about :doc:`the run subcommand <run>` will
tell you what options to use for debugging memory leaks, or for seeing the native stack traces corresponding to
allocations. Interpreting the generated memory profiles will be much easier if you understand :doc:`the Python
allocators <python_allocators>` as well as :doc:`some general memory concepts <memory>`.

If you find any bugs, you can `file a bug report`_. If you aren't sure whether something is a bug or expected behavior,
or if you want to suggest an idea or discuss things with the maintainers, you should `start a discussion`_ instead.

Good luck, and happy debugging!

.. _file a bug report: https://github.com/bloomberg/memray/issues/new?&labels=bug&template=---bug-report.yaml
.. _start a discussion: https://github.com/bloomberg/memray/discussions/new/choose

---

### Index (`docs/index.rst`)

.. toctree::
   :hidden:

   overview
   getting_started

.. toctree::
   :hidden:
   :caption: Hands-on Tutorial

   tutorials/index
   tutorials/1
   tutorials/2
   tutorials/3
   tutorials/additional_features

.. toctree::
   :hidden:
   :caption: Concepts

   run
   python_allocators
   memory
   temporary_allocations
   attach
   native_mode
   examples/README
   api
   jupyter_magic
   performance

.. toctree::
   :hidden:
   :caption: Reporters

   live
   summary
   flamegraph
   table
   tree
   stats
   transform

.. toctree::
   :hidden:
   :caption: Project Information

   supported_environments
   changelog
   licenses

---

### Jupyter Magic (`docs/jupyter_magic.rst`)

.. _Jupyter integration:

Jupyter Integration
===================

We provide an IPython extension that adds a new Jupyter cell magic. This lets
you create Memray flame graphs directly in Jupyter notebooks.

memray_flamegraph
-----------------

To load our IPython plugin, you simply need to run::

    %load_ext memray

Once it's loaded, you'll have access to the ``%%memray_flamegraph`` cell magic.
You can fill a Jupyter cell with ``%%memray_flamegraph`` on its own line,
followed by some code whose memory usage you want to profile. Memray will run
that cell's code, tracking its memory allocations, and then display a flame
graph directly in Jupyter for you to analyze.

.. note::

    The flame graph is written to the current working directory, so Jupyter
    won't be able to find it if you've changed directories.

It's also possible to provide arguments on the ``%%memray_flamegraph`` line.
For instance, ``%%memray_flamegraph --trace-python-allocators --leaks`` would
let you look for memory not freed by the code in the cell::

    %%memray_flamegraph --trace-python-allocators --leaks
    def a():
        return "a" * 10_000

    def bc():
        return "bc" * 10_000

    x = a() + bc()

Arguments
---------

.. argparse::
   :ref: memray._ipython.flamegraph.argument_parser
   :prog: %%memray_flamegraph

---

### Licenses (`docs/licenses.rst`)

Licenses
========

Memray utilizes several other open source libraries under the hood. These are
the licenses of those libraries.

libbacktrace
------------

.. code-block:: text

    Copyright (C) 2012-2016 Free Software Foundation, Inc.

    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions are
    met:

        (1) Redistributions of source code must retain the above copyright
        notice, this list of conditions and the following disclaimer.

        (2) Redistributions in binary form must reproduce the above copyright
        notice, this list of conditions and the following disclaimer in
        the documentation and/or other materials provided with the
        distribution.

        (3) The name of the author may not be used to
        endorse or promote products derived from this software without
        specific prior written permission.

    THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY EXPRESS OR
    IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
    WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
    DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT,
    INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
    (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
    SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
    HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
    STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
    IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
    POSSIBILITY OF SUCH DAMAGE.

lz4_stream
----------

.. code-block:: text

    Copyright (c) 2015, Kasper Laudrup <[email protected]>
    All rights reserved.

    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions are
    met:

    1. Redistributions of source code must retain the above copyright
    notice, this list of conditions and the following disclaimer.

    2. Redistributions in binary form must reproduce the above copyright
    notice, this list of conditions and the following disclaimer in the
    documentation and/or other materials provided with the distribution.

    3. Neither the name of the copyright holder nor the names of its
    contributors may be used to endorse or promote products derived from
    this software without specific prior written permission.

    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
    A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
    HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
    SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
    LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
    DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
    (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

bootstrap
---------

.. code-block:: text

    The MIT License (MIT)

    Copyright (c) 2011-2025 The Bootstrap Authors

    Permission is hereby granted, free of charge, to any person obtaining a copy
    of this software and associated documentation files (the "Software"), to deal
    in the Software without restriction, including without limitation the rights
    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    copies of the Software, and to permit persons to whom the Software is
    furnished to do so, subject to the following conditions:

    The above copyright notice and this permission notice shall be included in
    all copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    THE SOFTWARE.

d3
--

.. code-block:: text

    Copyright 2010-2023 Mike Bostock

    Permission to use, copy, modify, and/or distribute this software for any purpose
    with or without fee is hereby granted, provided that the above copyright notice
    and this permission notice appear in all copies.

    THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
    REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
    FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
    INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
    OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
    TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF
    THIS SOFTWARE.

d3-flame-graph
--------------

.. code-block:: text

                                 Apache License
                           Version 2.0, January 2004
                        http://www.apache.org/licenses/

   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

   1. Definitions.

      "License" shall mean the terms and conditions for use, reproduction,
      and distribution as defined by Sections 1 through 9 of this document.

      "Licensor" shall mean the copyright owner or entity authorized by
      the copyright owner that is granting the License.

      "Legal Entity" shall mean the union of the acting entity and all
      other entities that control, are controlled by, or are under common
      control with that entity. For the purposes of this definition,
      "control" means (i) the power, direct or indirect, to cause the
      direction or management of such entity, whether by contract or
      otherwise, or (ii) ownership of fifty percent (50%) or more of the
      outstanding shares, or (iii) beneficial ownership of such entity.

      "You" (or "Your") shall mean an individual or Legal Entity
      exercising permissions granted by this License.

      "Source" form shall mean the preferred form for making modifications,
      including but not limited to software source code, documentation
      source, and configuration files.

      "Object" form shall mean any form resulting from mechanical
      transformation or translation of a Source form, including but
      not limited to compiled object code, generated documentation,
      and conversions to other media types.

      "Work" shall mean the work of authorship, whether in Source or
      Object form, made available under the License, as indicated by a
      copyright notice that is included in or attached to the work
      (an example is provided in the Appendix below).

      "Derivative Works" shall mean any work, whether in Source or Object
      form, that is based on (or derived from) the Work and for which the
      editorial revisions, annotations, elaborations, or other modifications
      represent, as a whole, an original work of authorship. For the purposes
      of this License, Derivative Works shall not include works that remain
      separable from, or merely link (or bind by name) to the interfaces of,
      the Work and Derivative Works thereof.

      "Contribution" shall mean any work of authorship, including
      the original version of the Work and any modifications or additions
      to that Work or Derivative Works thereof, that is intentionally
      submitted to Licensor for inclusion in the Work by the copyright owner
      or by an individual or Legal Entity authorized to submit on behalf of
      the copyright owner. For the purposes of this definition, "submitted"
      means any form of electronic, verbal, or written communication sent
      to the Licensor or its representatives, including but not limited to
      communication on electronic mailing lists, source code control systems,
      and issue tracking systems that are managed by, or on behalf of, the
      Licensor for the purpose of discussing and improving the Work, but
      excluding communication that is conspicuously marked or otherwise
      designated in writing by the copyright owner as "Not a Contribution."

      "Contributor" shall mean Licensor and any individual or Legal Entity
      on behalf of whom a Contribution has been received by Licensor and
      subsequently incorporated within the Work.

   2. Grant of Copyright License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      copyright license to reproduce, prepare Derivative Works of,
      publicly display, publicly perform, sublicense, and distribute the
      Work and such Derivative Works in Source or Object form.

   3. Grant of Patent License. Subject to the terms and conditions of
      this License, each Contributor hereby grants to You a perpetual,
      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
      (except as stated in this section) patent license to make, have made,
      use, offer to sell, sell, import, and otherwise transfer the Work,
      where such license applies only to those patent claims licensable
      by such Contributor that are necessarily infringed by their
      Contribution(s) alone or by combination of their Contribution(s)
      with the Work to which such Contribution(s) was submitted. If You
      institute patent litigation against any entity (including a
      cross-claim or counterclaim in a lawsuit) alleging that the Work
      or a Contribution incorporated within the Work constitutes direct
      or contributory patent infringement, then any patent licenses
      granted to You under this License for that Work shall terminate
      as of the date such litigation is filed.

   4. Redistribution. You may reproduce and distribute copies of the
      Work or Derivative Works thereof in any medium, with or without
      modifications, and in Source or Object form, provided that You
      meet the following conditions:

      (a) You must give any other recipients of the Work or
          Derivative Works a copy of this License; and

      (b) You must cause any modified files to carry prominent notices
          stating that You changed the files; and

      (c) You must retain, in the Source form of any Derivative Works
          that You distribute, all copyright, patent, trademark, and
          attribution notices from the Source form of the Work,
          excluding those notices that do not pertain to any part of
          the Derivative Works; and

      (d) If the Work includes a "NOTICE" text file as part of its
          distribution, then any Derivative Works that You distribute must
          include a readable copy of the attribution notices contained
          within such NOTICE file, excluding those notices that do not
          pertain to any part of the Derivative Works, in at least one
          of the following places: within a NOTICE text file distributed
          as part of the Derivative Works; within the Source form or
          documentation, if provided along with the Derivative Works; or,
          within a display generated by the Derivative Works, if and
          wherever such third-party notices normally appear. The contents
          of the NOTICE file are for informational purposes only and
          do not modify the License. You may add Your own attribution
          notices within Derivative Works that You distribute, alongside
          or as an addendum to the NOTICE text from the Work, provided
          that such additional attribution notices cannot be construed
          as modifying the License.

      You may add Your own copyright statement to Your modifications and
      may provide additional or different license terms and conditions
      for use, reproduction, or distribution of Your modifications, or
      for any such Derivative Works as a whole, provided Your use,
      reproduction, and distribution of the Work otherwise complies with
      the conditions stated in this License.

   5. Submission of Contributions. Unless You explicitly state otherwise,
      any Contribution intentionally submitted for inclusion in the Work
      by You to the Licensor shall be under the terms and conditions of
      this License, without any additional terms or conditions.
      Notwithstanding the above, nothing herein shall supersede or modify
      the terms of any separate license agreement you may have executed
      with Licensor regarding such Contributions.

   6. Trademarks. This License does not grant permission to use the trade
      names, trademarks, service marks, or product names of the Licensor,
      except as required for reasonable and customary use in describing the
      origin of the Work and reproducing the content of the NOTICE file.

   7. Disclaimer of Warranty. Unless required by applicable law or
      agreed to in writing, Licensor provides the Work (and each
      Contributor provides its Contributions) on an "AS IS" BASIS,
      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
      implied, including, without limitation, any warranties or conditions
      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
      PARTICULAR PURPOSE. You are solely responsible for determining the
      appropriateness of using or redistributing the Work and assume any
      risks associated with Your exercise of permissions under this License.

   8. Limitation of Liability. In no event and under no legal theory,
      whether in tort (including negligence), contract, or otherwise,
      unless required by applicable law (such as deliberate and grossly
      negligent acts) or agreed to in writing, shall any Contributor be
      liable to You for damages, including any direct, indirect, special,
      incidental, or consequential damages of any character arising as a
      result of this License or out of the use or inability to use the
      Work (including but not limited to damages for loss of goodwill,
      work stoppage, computer failure or malfunction, or any and all
      other commercial damages or losses), even if such Contributor
      has been advised of the possibility of such damages.

   9. Accepting Warranty or Additional Liability. While redistributing
      the Work or Derivative Works thereof, You may choose to offer,
      and charge a fee for, acceptance of support, warranty, indemnity,
      or other liability obligations and/or rights consistent with this
      License. However, in accepting such obligations, You may act only
      on Your own behalf and on Your sole responsibility, not on behalf
      of any other Contributor, and only if You agree to indemnify,
      defend, and hold each Contributor harmless for any liability
      incurred by, or claims asserted against, such Contributor by reason
      of your accepting any such warranty or additional liability.

   END OF TERMS AND CONDITIONS

   APPENDIX: How to apply the Apache License to your work.

      To apply the Apache License to your work, attach the following
      boilerplate notice, with the fields enclosed by brackets "{}"
      replaced with your own identifying information. (Don't include
      the brackets!)  The text should be enclosed in the appropriate
      comment syntax for the file format. We also recommend that a
      file or class name and description of purpose be included on the
      same "printed page" as the copyright notice for easier
      identification within third-party archives.

   Copyright {yyyy} {name of copyright owner}

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

d3-scale-chromatic
------------------

.. code-block:: text

    Copyright 2010-2024 Mike Bostock

    Permission to use, copy, modify, and/or distribute this software for any purpose
    with or without fee is hereby granted, provided that the above copyright notice
    and this permission notice appear in all copies.

    THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
    REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
    FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
    INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
    OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
    TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF
    THIS SOFTWARE.

    Apache-Style Software License for ColorBrewer software and ColorBrewer Color Schemes

    Copyright 2002 Cynthia Brewer, Mark Harrower, and The Pennsylvania State University

    Licensed under the Apache License, Version 2.0 (the "License"); you may not use
    this file except in compliance with the License. You may obtain a copy of the
    License at

    http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software distributed
    under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
    CONDITIONS OF ANY KIND, either express or implied. See the License for the
    specific language governing permissions and limitations under the License.

@floating-ui/dom
----------------

.. code-block:: text

    The MIT License (MIT)
    Copyright (c) 2021 Floating UI contributors

    Permission is hereby granted, free of charge, to any person obtaining
    a copy of this software and associated documentation files (the
    "Software"), to deal in the Software without restriction, including without
    limitation the rights to use, copy, modify, merge, publish, distribute,
    sublicense, and/or sell copies of the Software, and to permit persons to
    whom the Software is furnished to do so, subject to the following
    conditions:

    The above copyright notice and this permission notice shall be included in
    all copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
    FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
    DEALINGS IN THE SOFTWARE.

datatables.net and datatables.net-bs4
-------------------------------------

.. code-block:: text

    The MIT License (MIT)

    Copyright SpryMedia Limited and other contributors
    http://datatables.net

    Permission is hereby granted, free of charge, to any person obtaining a copy
    of this software and associated documentation files (the "Software"), to deal
    in the Software without restriction, including without limitation the rights
    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    copies of the Software, and to permit persons to whom the Software is
    furnished to do so, subject to the following conditions:

    The above copyright notice and this permission notice shall be included in
    all copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    THE SOFTWARE.

jquery
------

.. code-block:: text

    Copyright OpenJS Foundation and other contributors, https://openjsf.org/

    Permission is hereby granted, free of charge, to any person obtaining
    a copy of this software and associated documentation files (the
    "Software"), to deal in the Software without restriction, including
    without limitation the rights to use, copy, modify, merge, publish,
    distribute, sublicense, and/or sell copies of the Software, and to
    permit persons to whom the Software is furnished to do so, subject to
    the following conditions:

    The above copyright notice and this permission notice shall be
    included in all copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
    EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
    MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
    NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
    LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
    OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
    WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

plotly.js-dist-min
------------------

.. code-block:: text

    MIT License

    Copyright (c) 2016-2024 Plotly Technologies Inc.

    Permission is hereby granted, free of charge, to any person obtaining a copy
    of this software and associated documentation files (the "Software"), to deal
    in the Software without restriction, including without limitation the rights
    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    copies of the Software, and to permit persons to whom the Software is
    furnished to do so, subject to the following conditions:

    The above copyright notice and this permission notice shall be included in
    all copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
    THE SOFTWARE.

popper.js
---------

.. code-block:: text

    MIT License

    Copyright (c) 2021 Floating UI contributors

    Permission is hereby granted, free of charge, to any person obtaining a copy
    of this software and associated documentation files (the "Software"), to deal
    in the Software without restriction, including without limitation the rights
    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    copies of the Software, and to permit persons to whom the Software is
    furnished to do so, subject to the following conditions:

    The above copyright notice and this permission notice shall be included in all
    copies or substantial portions of the Software.

    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    SOFTWARE.

---

### Live (`docs/live.rst`)

Live Reporting
==============

Overview
--------

Memray supports presenting a "live" view for observing the heap memory usage of a running Python program.
This provides insight into how memory is allocated as the program executes.

It is especially helpful for investigating the behavior of a program that has different allocation patterns during
different stages of the program's execution. It lets you see what the peak heap usage has been and how much memory is
being allocated by various parts of the program. The information is presented in a tabular format, showing how much of
the in-use memory each function directly allocated ("own memory"), how much was cumulatively allocated by that function
and everything it called ("total memory"), and the cumulative count of not yet freed allocations performed by that
function and everything it called ("allocation count").

Usage
-----

To use live mode, you can specify the program to be profiled in live mode using ``run --live``:

.. code:: shell-session

  $ memray run --live application.py

Immediately Memray will start your process in the background and will connect a TUI to it in the foreground. The TUI will
display the heap's current allocations every time it takes a snapshot, in a tabular format.

.. image:: _static/images/live_running.png

This view presents the following information:

- What the PID is of the running process.
- What command was used to invoke the program.
- The current heap size and the peak heap size seen so far.
- How long the program has been running.
- How many snapshots of the memory (referred to as samples) have been presented.
- A plot of the heap size over time.

Once the program has exited, there will be a message presented in the live view, stating that "Remote has disconnected".

.. image:: _static/images/live_disconnected.png

Sorting results
---------------

The results are displayed in descending order of total memory allocated by a function and the subfunctions called by
it. You can change the ordering with the following keyboard shortcuts:

- t (default): Sort by total memory
- o: Sort by own memory
- a: Sort by allocation count

In most terminals you can also click the "Sort by Total", "Sort by Own", and "Sort by Allocations" buttons on the
footer.

The sorted column's heading is underlined.

Viewing different threads
-------------------------

By default, the ``live`` command will present allocations from all of the program's threads. You can look at individual
threads of the program by first pressing ``m`` (to disable thread merging), and then pressing the greater than and less
than keys, ``<`` and ``>``, to switch between threads. In most terminals you can also click the "Unmerge Threads",
"Previous Thread", and "Next Thread" buttons on the footer.

.. image:: _static/images/live_different_thread.png

Using with native tracking
--------------------------

It is possible to use :ref:`native tracking` along with the live mode. This can be achieved by passing ``--native``
to the ``run`` command.

.. code:: shell-session

  $ memray run --live --native application.py

Remote mode
-----------

Remote mode allows you to run the program to be traced in one process and the tui in another. To use remote mode, you can
specify the program to be profiled in live mode using ``run --live-remote``:

.. code:: shell-session

  $ memray run --live-remote application.py
  Run 'memray live <port>' in another shell to see live results

As you can see, Memray is now waiting on a connection to the live tracking server. You can now attach to
the server using running the ``live`` command in a separate shell.

.. code:: shell-session

  $ # Run this in a different shell:
  $ memray live <port>

This command will connect to the server that was started in the previous command, when given the requested port number.
Once the connection is established, the program will start executing. All memory allocations and deallocations
throughout the program will be tracked.

The program being tracked will present its output on the shell running the ``run --live-remote`` command. In the shell running
the ``live`` command, information about the program will be presented with the regular TUI of live mode.


Using a different port
----------------------

It is possible to make ``run --live-remote`` start the server on a user-specified port, using the ``--live-port`` argument.

.. code:: shell-session

  $ memray run --live-remote --live-port 12345 application.py
  Run 'memray live 12345' in another shell to see live results

.. important::

  Due to the syntax of the command line arguments of memray, make sure that you pass any options intended for the
  ``run`` command *before* your script/module. Otherwise, they will be treated as arguments for the script and will not
  be used by Memray.

  For example, the following invocation will pass ``--live-port 12345`` to ``application.py``,
  instead of having it be used by ``memray run``:

  .. code:: shell-session

    $ memray run --live-remote application.py --live-port 12345
    Run 'memray live 60125' in another shell to see live results

---

### Manpage (`docs/manpage.rst`)

:orphan:

Overview
========

.. argparse::
   :ref: memray.commands.get_argument_parser
   :manpage:
   :nosubcommands:

   Memray can track memory allocations in Python code, in native extension modules, and in the Python
   interpreter itself. It can generate several different types of reports to help you analyze the captured
   memory usage data. While commonly used as a CLI tool, it can also be used as a library to perform more
   fine-grained profiling tasks.

   Most commonly you will use the ``memray run`` subcommand to create a capture file, and then use a reporter
   like the ``memray flamegraph`` subcommand to analyze it.

   .. note::

       This manual page only documents usage of the Memray subcommands that can be invoked from the command line.
       See `<https://bloomberg.github.io/memray/overview.html>`_ for the full Memray documentation, which
       includes advice for interpreting Memray reports, example programs, API documentation, information about
       integrating Memray into Jupyter notebooks and pytest test suites, explanations to help you understand how
       Python uses memory and how Memray gathers information about memory usage, and more.

RUN SUB-COMMAND
---------------

.. argparse::
   :ref: memray.commands.get_argument_parser
   :path: run
   :nodefaultconst:
   :noepilog:

FLAMEGRAPH SUB-COMMAND
----------------------

.. argparse::
   :ref: memray.commands.get_argument_parser
   :path: flamegraph
   :nodefaultconst:
   :noepilog:

TABLE SUB-COMMAND
-----------------

.. argparse::
   :ref: memray.commands.get_argument_parser
   :path: table
   :nodefaultconst:
   :noepilog:

LIVE SUB-COMMAND
----------------

.. argparse::
   :ref: memray.commands.get_argument_parser
   :path: live
   :nodefaultconst:
   :noepilog:

TREE SUB-COMMAND
----------------

.. argparse::
   :ref: memray.commands.get_argument_parser
   :path: tree
   :nodefaultconst:
   :noepilog:

PARSE SUB-COMMAND
-----------------

.. argparse::
   :ref: memray.commands.get_argument_parser
   :path: parse
   :nodefaultconst:
   :noepilog:

SUMMARY SUB-COMMAND
-------------------

.. argparse::
   :ref: memray.commands.get_argument_parser
   :path: summary
   :nodefaultconst:
   :noepilog:

STATS SUB-COMMAND
-----------------

.. argparse::
   :ref: memray.commands.get_argument_parser
   :path: stats
   :nodefaultconst:
   :noepilog:

TRANSFORM SUB-COMMAND
---------------------

.. argparse::
   :ref: memray.commands.get_argument_parser
   :path: transform
   :nodefaultconst:
   :noepilog:

ATTACH SUB-COMMAND
------------------

.. argparse::
   :ref: memray.commands.get_argument_parser
   :path: attach
   :nodefaultconst:
   :noepilog:

DETACH SUB-COMMAND
------------------

.. argparse::
   :ref: memray.commands.get_argument_parser
   :path: detach
   :nodefaultconst:
   :noepilog:

---

### Memory (`docs/memory.rst`)

.. _memory overview:

Memory overview
===============

Heap memory vs resident memory
------------------------------

When you are profiling memory usage, you will often see two different metrics:
heap memory and resident memory.

Resident memory corresponds to the memory that is allocated currently in your
main memory or RAM. Generally speaking, programs will store most of the memory
they are using in main memory but there are some caveats you need to be aware of
if you want to make sense of how your programs are using memory. Resident memory
is a metric that is **not independent** of the other programs that are running
concurrently and what's happening in your machine. This means that **two
identical runs of your program can have very different resident memory
measurements**.  If the OS determines that other programs or
tasks have higher priority than the one you are analyzing, it may move some of the
memory used by the program to swap space. This means that resident memory usage
in your program may decrease (and increase later) even if you don't free or
allocate any memory yourself.

This makes resident memory a tricky metric to make sense of. On the one hand it is a
good indicator of what may be causing your machine to run out of memory.
Ultimately this is the memory that is limited by the amount of RAM (and
swap space) you have. But on the other hand the value of resident memory
associated with a given program is dependent on all the other programs that are
running concurrently so it may be a bit difficult to properly diagnose why it
decreases or increases.

Memory is lazily allocated
--------------------------

What makes it even more complicated to properly relate heap memory and resident
memory is that memory is lazily allocated by the OS. For example, if you call
one the system allocator APIs (``malloc`` for example), it may return instantly
without actually allocating memory at all. It still will give you a pointer to a
chunk of memory that you can use, but memory will only be allocated when you write to that pointer.
``malloc`` promises you the memory chunk but you only get it for real when you
really need it. This means that heap memory will increase as soon as the
allocator API is called but resident memory will only increase once you actually
write to that memory.

For instance, consider this code: ::

    import time
    import numpy
    time.sleep(1)
    big_array = numpy.empty(1_000_000)
    time.sleep(1)
    big_array[:] = 42.0
    time.sleep(1)

If you run ``memray`` against this code and generate a flamegraph, you will see
the following plot:

.. image:: _static/images/rss_vs_heap.png

As you can see in the plot, the line for the heap size increases first
(corresponding to the call to ``numpy.empty``) but the resident size does not
increase inmediately. Instead, the resident size only increases after we have
populated the whole array with floating point numbers. Is only at this moment
when the OS will actually allocate the necessary memory pages to satisfy our
initial request. Notice that this happens when memory is **written** to the
array, so a memory profiler **won't be able to tell you what makes the resident
size grow** as it doesn't have visibility into when pages are actually assigned.

Memory is not freed immediately
-------------------------------

Another thing that makes difficult to relate heap memory and resident memory is
that memory is not freed immediately after it is no longer needed. This is
because the system allocator may not release the memory to the OS when it is no
longer needed. This means that once you call a deallocator API (``free`` for
example), the implementation may not free the memory for real until later. This means
that you may see the heap size decrease but the resident memory size may
not decrease yet.

For instance, consider this code: ::

    import time
    import numpy
    time.sleep(1)
    big_array = numpy.empty(1_000_000)
    time.sleep(1)
    big_array[:] = 42.0
    time.sleep(1)
    del big_array
    time.sleep(1)

If you run ``memray`` against this code and generate a flamegraph, you will see the following the following plot:

.. image:: _static/images/rss_vs_heap_no_free.png

As you can see in the plot, the line for the heap size decreases after we delete
the array (corresponding to the call to ``del``) but the resident size does not
decrease inmediately. Instead, the resident size will only decrease (not shown
in this plot) after the system allocator determines that it is a good idea to free
the memory pages.  Notice that this happens when pages are released, so a
memory profiler **won't be able to tell you what makes the resident size
decrease** as it doesn't have visibility when pages are actually unmapped.

Memory is shared
----------------

Another thing that makes it difficult to relate heap memory and resident memory is
that memory is shared. This means that the same memory pages can be used by
different processes. This happens for instance when you fork a process. When you
fork a process, the child process will initially share the same memory pages with the
parent process. This means that the resident memory size that the child process
requires will not increase until copy on write (COW) is triggered. You can read
more about COW in the `Wikipedia page <https://en.wikipedia.org/wiki/Copy-on-write>`_.

Memory can be fragmented
------------------------

Another thing that makes it difficult to relate heap memory and resident memory is
that memory can be fragmented. This means that the memory that is allocated by
the system allocator may be spread around the address space in different
fragments. This means that the resident memory size will increase/decrease in
unpredictable ways. This happens because the system allocator may not be able to
reuse memory that has been freed before.

Memory fragmentation results in seemingly unnecessary requests to the OS
for more memory. Even when the sum of the space already available for the memory
allocator is large enough to satisfy a memory allocation request, it's possible
no individual fragment (or set of contiguous fragments) is large enough to satisfy that
memory allocation request.  Memory fragmentation is caused by a combination of
the allocation strategy used by the allocator you are using, the sizes and
alignments of the internal structures, and the memory allocation
behavior of your application.

Detecting fragmentation is a very difficult task because it depends on the system
allocator that you are using. If you are using GLIBC's ``malloc`` for example,
you can use the ``malloc_stats`` API to get information about the memory
allocator. This API will give you information about the number of free chunks
and the total size of the free chunks. If you see that the number of free chunks
is large but the total size of the free chunks is small, then you may be
suffering from memory fragmentation. You can read more about this in the `man
page <https://man7.org/linux/man-pages/man3/malloc_stats.3.html>`_.

Although this API must be called from native code, you can use the `ctypes module <https://docs.python.org/3/library/ctypes.html>`_
to call it from Python. For example, you can use the following code to call it from Python: ::

    import ctypes
    libc = ctypes.CDLL("libc.so.6")
    libc.malloc_stats.restype = None
    libc.malloc_stats()

Another option is to use GLIBC's ``malloc_info`` API. This API will give you
information about the memory allocator in a format that is easier to parse from
programs. As with the other API, you can use the `ctypes module <https://docs.python.org/3/library/ctypes.html>`_
to call it from Python. ::

    import ctypes

    class MallInfo(ctypes.Structure):
        _fields_ = [
            (name, ctypes.c_int)
            for name in (
                "arena",
                "ordblks",
                "smblks",
                "hblks",
                "hblkhd",
                "usmblks",
                "fsmblks",
                "uordblks",
                "fordblks",
                "keepcost",
            )
        ]


    libc = ctypes.CDLL("libc.so.6")
    mallinfo = libc.mallinfo
    mallinfo.argtypes = []
    mallinfo.restype = MallInfo

    info = mallinfo()
    fields = [(name, getattr(info, name)) for name, _ in info._fields_]
    print("Malloc info:")
    for name, value in fields:
        print(f"- {name}: {value}")

---

### Native Mode (`docs/native_mode.rst`)

Symbolic information in native mode
===================================

.. highlight:: shell-session

.. important::

   For the best native mode experience, we recommend running your program on
   Linux using an interpreter and libraries built with as much debugging
   information as possible.

When passing the ``--native`` flag to the :doc:`the run subcommand <run>`,
Memray will collect information about the native call stack leading to each
allocation, and it will dump it to the result file. This information is in
a raw form where every call is identified by a number called the "instruction
pointer" or "program counter". The instruction pointer is a number that helps
running programs identify what CPU instruction needs to be executed next. After
fetching each instruction the instruction pointer is incremented, and holds the
memory address of ("points to") the next instruction to be executed. Processors
usually fetch instructions sequentially from memory, but control transfer
instructions change the sequence by placing a new value in the instruction
pointer. These include branches (sometimes called jumps), subroutine calls, and
returns. A transfer that is performed only if some condition is true allows the
computer to follow a different sequence under different conditions.

When creating reports, Memray needs to convert these instruction pointers into
human readable information, like the function name, the file name where that
function is defined, and the line number within that file corresponding to the
instruction. This process is known as **symbolification**.

How Memray resolves symbols
---------------------------

There are two different approaches Memray uses to symbolify an instruction
pointer:

* Use the `DWARF <https://dwarfstd.org/>`_ debugging information embedded in
  the executable or shared library containing the instruction. This process
  will provide function names, file names, and line numbers, and will also be
  able to resolve inlined functions. This method is used whenever debugging
  information is available.

* Use the symbol table information embedded in the executable or shared library
  containing the instruction. This is a suboptimal method that will **only**
  provide function names, not file names or line numbers. It can also be
  unreliable as the symbol table may not contain entries for every function.

.. attention::

   To reduce tracking overhead, Memray delays symbolification **until reports
   are being generated**. This means that when a report is being generated,
   Memray needs to read information from the interpreter executable that was
   used to run the tracked application, and from the shared libraries that were
   loaded into it. Reports must be generated **on the same machine that the
   application ran on**, because the same versions of the interpreter and
   shared libraries need to be available for inspection. Failing to do this
   will result in symbolification errors or incorrect reports.

If Memray is able to resolve file names, line numbers, and inline functions,
it can hide some of the Python interpreter's internal functions which don't add
much information to the report. If there is no debug information available then
the produced flame graphs will look more noisy and be harder to read, because
we won't be able to reliably detect these uninteresting functions. In these
images you can compare two flame-graphs, one with debug information and one
without debug information, produced from the same capture file:

.. list-table::

    * - .. figure:: _static/images/native_mode_no_debug.png

           Flamegraph in native mode produced without debug information.

      - .. figure:: _static/images/native_mode_debug.png

           Flamegraph in native mode produced with debug information.

You can see that file names are reported as ``<unknown>`` and line numbers as
``0`` when debugging information is not available.

On Linux, you can check whether your binaries have DWARF debug information
available by running::

    $ readelf -S ./a.out | grep debug

For example, for checking if the Python interpreter has DWARF debug information
available::

    $ readelf -S $(which python) | grep debug

    [26] .debug_aranges    PROGBITS         0000000000000000  00001057
    [27] .debug_info       PROGBITS         0000000000000000  00001087
    [28] .debug_abbrev     PROGBITS         0000000000000000  000010de
    [29] .debug_line       PROGBITS         0000000000000000  0000111a
    [30] .debug_str        PROGBITS         0000000000000000  00001177
    [31] .debug_macro      PROGBITS         0000000000000000  00003eb1


.. _mac symbolification:

Symbolification in macOS
------------------------

.. caution::

    Because most macOS binaries for Python don't include debug
    information, **reports produced in macOS can be much less accurate**.

On Mac OS X there was a decision to not have the linker include all of the
debug information into the executables and libraries it builds. Instead, the
compiler generates the DWARF debug information in the ``.s`` files, the
assembler outputs it in the ``.o`` files, and the linker includes a "debug map"
in the executable or shared library which tells debug info users where all of
the symbols were relocated during the link. There is a tool called ``dsymutil``
that can read the debug map, load the DWARF information from the ``.o`` files,
relocate all the addresses and then output a single binary with all the DWARF
information at their final, linked addresses. This final binary is called
a ``dSYM`` bundle and is normally placed alongside every executable.

This process has some advantages, but unfortunately in the Python world neither
redistributors of Python interpreters nor library maintainers generally package
debugging information with the binaries. This means that **most macOS binaries
don't have debug information inside**, so in general **native mode
symbolification will not work well** in macOS.

Memray will fall back to symbol table analysis if it can't find any debug
information in the binary, but when producing reports we won't be able to
identify where the different symbols came from, making flame graphs very
verbose and hard to read.

Adding debugging information to your native extension

$ # Sanity check: ensure that the object files are still around
$ dsymutil -s  src/memray/_memray.cpython-310-darwin.so | grep OSO | head -n 1
[  9431] 000d39a1 66 (N_OSO        ) 00     0001   0000000062fb8052 'memray/build/temp.macosx-12.5-arm64-cpython-310/src/memray/_memray.o'

$ ls memray/build/temp.macosx-12.5-arm64-cpython-310/src/memray/_memray.o
.rw-r--r-- 3.5M pgalindo3 16 Aug 12:32 memray/build/temp.macosx-12.5-arm64-cpython-310/src/memray/_memray.o

$ # Then generate a dSYM bundle with the debug information:
$ dsymutil src/memray/_memray.cpython-310-darwin.so

$ export DEBUGINFOD_PROGRESS=1

$ export DEBUGINFOD_VERBOSE=1

$ export DEBUGINFOD_TIMEOUT=10       # seconds
$ export DEBUGINFOD_MAXSIZE=1000000  # bytes

$ unset DEBUGINFOD_URLS

* CPU: use 2 logical CPUs: 17,35
* Perf event: Maximum sample rate: 1 per second
* ASLR: Full randomization
* Linux scheduler: Isolated CPUs (2/36): 17,35
* Linux scheduler: RCU disabled on CPUs (2/36): 17,35
* CPU Frequency: 0-16,18-34=min=1200 MHz, max=3000 MHz; 17,35=min=max=3000 MHz
* CPU scaling governor (intel_pstate): performance
* Turbo Boost (intel_pstate): Turbo Boost disabled
* IRQ affinity: Default IRQ affinity: CPU 0-16,18-34


Source of the overhead
----------------------

The overhead of running Memray comes from two places:

- A `Python profile function <https://docs.python.org/3/library/sys.html#sys.setprofile>`_ implemented in
  C++. This function is called every time a Python function is called or returns
  and is used to register the function calls so Memray can reconstruct the
  stack trace of every allocation. Although the overhead is very, very small, it
  adds up. This means that the more your application calls Python
  functions the bigger the overhead will be.
- The allocation registering code. This is the main source of the overhead.
  Every time your application makes a memory allocation or deallocation,
  Memray needs to register it. This means that the more frequently your application
  allocates memory, the bigger the overhead will be. Tracking Python allocators
  will greatly increase this overhead because the Python allocator is used for
  small, short lived objects which means that it's generally called with very high
  frequency.

Results
-------

The following plots show the overhead of running Memray over different test
cases using violin plots. Each row corresponds to the result of a given test.
For a given row, the continuos vertical line in the middle corresponds to the
**mean** while the two dashed vertical vars at the left and right of the mean
correspond to the **10% and 90% quantiles**.  The shaded area represents the
distribution of the data, with thicker areas indicating a higher density of data
points. The **ALL** row corresponds to the distribution of all the results across all the
benchmarks combined.

Default options

The following results correspond to running Memray with ``--trace-python-allocators``.

.. image:: _static/images/plot_python_allocators.png
   :align: center

Native traces
~~~~~~~~~~~~~

The following results correspond to running Memray with ``--native``.

.. image:: _static/images/plot_native.png
   :align: center

Python allocators + Native traces

memray run [options] file.py [args]
memray run [options] -m module [args]

Memray supports tracking native C/C++ functions as well as Python functions. This can be especially useful
when profiling libraries that have extension modules (such as ``numpy`` or ``pandas``) as this
gives a holistic vision of how much memory is allocated by the extension and how much is allocated by Python itself.

For instance, consider the Mandelbrot example from the :ref:`example-applications` section with native tracking
disabled. Some of the most important allocations happen when operating on NumPy arrays:

.. image:: _static/images/mandelbrot_operation_non_native.png

Here, we can see that the allocation happens when doing some math on NumPy arrays but unfortunately this doesn't inform us
of what exact operation is allocating memory or how temporaries are being used. We also don't know if the memory was
allocated by NumPy or by the interpreter itself. By using the native tracking mode with Memray we can get a much richer report:

.. image:: _static/images/mandelbrot_operation_native.png

In this native report, we can see all the internal C calls that are underneath. We can see that the memory allocation
happens when the NumPy arrays are being added, due to ``PyNumber_Add`` appearing in the stack trace. Based on
``PyNumber_Multiply`` not appearing in the stack trace, we can conclude that the temporary array created by NumPy is
immediately freed (or that it didn't need to allocate memory in the first place, perhaps because it could reuse some
already allocated memory).

.. tip::
    Memray will also include *inlined* functions and *macros* when native tracking is enabled.

.. caution::
    Activating native tracking has a moderate impact on performance as every instruction pointer in the call stack needs
    to be resolved whenever an allocation happens. This effect is more noticeable the more allocations the traced
    application performs.

Check the :doc:`section on native symbolification <native_mode>` for more
information on how to obtain the best reports with native information, and on
how to debug problems in reports with native information.

Usage
~~~~~

To activate native tracking, you need to provide the ``--native`` argument when using the ``run`` subcommand:

.. code:: shell

  memray run --native example.py

This will add native stack information to the result file, which any reporter will automatically use.

.. important::
   When generating reports for result files that contain native frames, the report needs to be generated **on the same
   machine** where the result file was generated. This is because the shared libraries that were loaded by the process
   need to be inspected by Memray to get the correct symbol names.

When reporters display native information they will normally use a different color for the Python frames than the native
frames. This can also be distinguished by looking at the file name in a frame, since Python frames will generally come
from source files with a ``.py`` extension.

Python allocator tracking
-------------------------

Memray normally tracks allocation and deallocation requests made to the system
allocator, but by default it won't see individual Python objects being created.
That's because the Python interpreter normally uses its own memory pools for
creating most objects, only making calls to the system allocator as needed to
grow or shrink its memory pools. Our documentation on :doc:`python allocators
` describes this memory pooling in greater detail. This
behavior speeds the Python interpreter up, and by extension speeds up profiling
with Memray, while still allowing Memray to show you each place where your
program needs to acquire more memory.

You can ask Memray to show you each individual object being created and
destroyed, instead, by providing the ``--trace-python-allocators`` argument to
the ``run`` subcommand. This records a lot more data and makes profiling much
slower. It will show you all allocations, even ones that don't result in your
program requesting more memory from the system because the interpreter already
had memory available for reuse. It can be useful in some cases, though,
especially when tracking down memory leaks.

.. note::
  This acts also as an alternative way to run with ``PYTHONMALLOC=malloc`` but
  in a way that allows distinguishing allocations made by using the system
  allocator directly and ones made by using the Python allocator.

.. code:: shell

  memray run --trace-python-allocators example.py

.. caution::
  Tracking the Python allocators will result in much larger report files and
  slower profiling due to the larger amount of data that needs to be collected.

.. _Live tracking:

Live tracking
-------------

Overview

You can run a program in live mode using ``run --live``:

.. code:: shell

  memray run --live application.py

Immediately Memray will start your application in the background and will run a TUI in the foreground that you can use
to analyze your application's memory usage. If you don't want to run your program in the background, you can instead
use ``run --live-remote``:

.. code:: shell

  memray run --live-remote application.py

In this mode, Memray will choose an unused port, bind to it, and display a message saying:

.. code:: text

  Run 'memray live <port>' in another shell to see live results

It will wait for you to run:

.. code:: shell

   memray live <port>

in another terminal window to attach to it. Regardless of whether you choose to use one terminal or two, the resulting
TUI is exactly the same. See :doc:`live` for details on how to interpret and control the TUI.


.. _Tracking across forks:

Tracking across forks
---------------------

Overview

To activate tracking through forks, you need to provide the ``--follow-fork`` argument to the ``run`` subcommand:

.. code:: shell

  memray run --follow-fork example.py

In this mode, each time the process forks, a new output file will be created for the new child process, with the new
child's process ID appended to the original capture file's name. The capture files for child processes are exactly like
any other capture file, and can be fed into any reporter of your choosing.

.. note::

  ``--follow-fork`` mode can only be used with an output file. It is incompatible with ``--live``
  mode and ``--live-remote`` mode, since the TUI can't be attached to multiple processes at once.

Losing capture files after OOM Errors
-------------------------------------

When a process runs out of memory, this commonly causes an Out Of Memory error,
or "OOM Error". That causes the process to be killed by its operating system.
Within orchestrations like Kubernetes the termination of the main process might
immediately lead to the destruction of the container and the loss of the files
that Memray uses to collect its results.

When running ``memray run myprogram.py`` a capture file gets created on the file
system, but the entire file system will be thrown away as soon as the
orchestration cleans up the container. If the program exits unexpectedly,
perhaps because the kernel kills it due to an OOM error, the orchestration might
throw away the capture file before you ever get a chance to use it. Since Memray
is often used to chase memory leaks, this condition might happen more often than
you'd like.

Since Memray is running in the same process as your application, it has no way
to prevent this data loss (by sending it over the network, for example) because
any work it does will be terminated when the process crashes.

Instead of directly calling ``memray run myprogram.py`` you can wrap it in
a script that will run Memray and run post-processing operations on the capture
file. That way, the container won't be destroyed by the orchestration until
after the wrapper script exits, rather than being destroyed as soon as the
Python script being tracked by Memray exits.

.. code-block:: shell

  memray run --output /tmp/capture.bin myprogram.py
  echo "Program finished"
  # Do your post-processing here. This example just logs a summary of what's
  # in the capture file, but you might want to generate reports from it and
  # copy them over the network to some persistent storage, for instance.
  memray summary /tmp/capture.bin

.. _aggregated capture files:

Aggregated capture files
------------------------

If you supply the ``--aggregate`` argument to ``memray run``, it will write
much smaller capture files. Instead of containing information about every
individual allocation performed by the tracked program, a capture file produced
using ``--aggregate`` will contain some statistics about the process's
allocations aggregated by the location where the allocation happened.

Specifically, for every location where the tracked process performed any
allocations, an aggregated capture file includes a count of:

- How many allocations at that location had not yet been deallocated when the
  process reached its heap memory high water mark
- How many bytes had been allocated at that location and not yet deallocated
  when the process reached its heap memory high water mark
- How many allocations at that location were leaked (i.e. not deallocated
  before tracking stopped)
- How many bytes were leaked by allocations at that location

These counts provide enough information to generate flame graphs. In fact, this
information is enough to run most of our reporters, with just a few exceptions:

- You cannot find :doc:`temporary allocations </temporary_allocations>` using
  this capture file format, since finding temporary allocations requires
  knowing when each individual allocation was deallocated.
- You cannot use the :doc:`stats reporter <stats>` with this capture file
  format, because it needs to see each individual allocation's size.
- You cannot use ``--aggregate`` with :ref:`live tracking <live tracking>`,
  since the live TUI needs to see each allocation as it happens.

Also, note that if the process is killed before tracking ends (for instance, by
the Linux OOM killer), then the process will die before it finishes calculating
its statistics, and so no useful information is ever written to the capture
file. With the default file format, the capture file is usually still usable
even if the process crashes or is killed, but with the aggregated file format
it is not.

If you can live with these limitations, then using ``--aggregate`` results in
much smaller capture files that can be used seamlessly with most reporters.

.. _buffered file io:

Buffered file I/O
-----------------

By default, Memray maps its capture file into memory, letting it write to the
file as if it were a normal memory buffer. This makes Memray faster and more
resilient to process crashes (including death to the OOM killer), but some
filesystems don't support shared writable ``mmap`` (or the ``posix_fallocate``
function Memray uses to grow the file). If you see an error about ``mmap`` or
``posix_fallocate`` when writing a capture file, and you can't move the capture
file to a more standard file system, you can pass the ``--buffered-file-io``
argument to ``memray run`` instead:

.. code-block:: shell

  memray run --buffered-file-io example.py

With this option, Memray buffers captured allocations in memory and writes them
out with ordinary ``write`` calls, flushing whenever the buffer fills. This
works on more filesystems, but is slower and less resilient to crashes and
other unclean exits.

CLI Reference
-------------

.. argparse::
   :ref: memray.commands.get_argument_parser
   :path: run
   :prog: memray

---

### Stats (`docs/stats.rst`)

Stats Reporter
==============

The stats reporter generates high level statistics about the tracked process's
memory allocations.

.. image:: _static/images/stats_example.png

The output includes the following:

* Total number of allocations performed

* Total amount of memory allocated

* Histogram displaying the distribution of allocation sizes. The y-axis data (size) is logarithmic.

* Distribution of allocation types (e.g. *MALLOC*, *CALLOC*, *MMAP*, etc.)

* Stack trace and **size** of the top 'n' largest allocating locations by size (*default: 5*, configurable with the ``-n`` command line param)

* Stack trace and **count** of the top 'n' largest allocating locations by number of allocations (*default: 5*, configurable with the ``-n`` command line param)

* (for JSON output only) Metadata about the tracked process

Basic Usage
-----------

The general form of the ``stats`` subcommand is:

.. code:: shell

    memray stats [options] <results>

The only argument the ``stats`` subcommand requires is the capture file
previously generated using :doc:`the run subcommand <run>`.

The output will be printed directly to the standard output of the terminal.

JSON Output
-----------

If you supply the ``--json`` flag, the ``stats`` subcommand will write its
output to a JSON file, rather than to the terminal. Like other commands that
output to files, the default output file name is based on the name of your
capture file, but it can be overridden with the ``-o`` / ``--output`` option.
By default Memray will refuse to overwrite an existing file, but you can force
it to by supplying the ``-f`` / ``--force`` option.

Note that new fields may be added to the JSON output over time, though we'll
try to avoid removing existing fields.

CLI Reference
-------------

.. argparse::
   :ref: memray.commands.get_argument_parser
   :path: stats
   :prog: memray

---

### Summary (`docs/summary.rst`)

Summary Reporter
================

The summary reporter provides a quick overview of allocated memory at the time
when the tracked process's memory usage was at its peak. It displays a table
showing how much of that peak memory each function directly allocated ("own
memory"), how much was cumulatively allocated by that function and everything
it called ("total memory"), and the cumulative count of not yet freed
allocations performed by that function and everything it called ("allocation
count"). Allocations for different threads are aggregated together and shown in
the same table.

.. image:: _static/images/summary_example.png

Basic Usage
-----------

The general form of the ``summary`` subcommand is:

.. code:: shell

    memray summary [options] <results>

The only argument the ``summary`` subcommand requires is the capture file
previously generated using :doc:`the run subcommand <run>`.

The output will be printed directly to standard output. If standard output is
a terminal, the output will be colorized.

CLI Reference
-------------

.. argparse::
   :ref: memray.commands.get_argument_parser
   :path: summary
   :prog: memray

---

### Supported Environments (`docs/supported_environments.rst`)

Supported environments
======================

Supported Python interpreters
-----------------------------

Only CPython is supported.

Supported Python versions
-------------------------

Every Python version that hasn't reached end of life is supported, and older
versions may remain supported for a while.

Currently that's Python 3.9 through 3.15.

Supported operating systems
---------------------------

You will have the best Memray experience on Linux.

We also support macOS 10.14 or newer on Intel Macs, and macOS 11 or newer on
Apple Silicon Macs. We cannot support older macOS versions, as they don't
provide a C++17 compatible runtime. Although all
features work on macOS, the way that macOS applications and Python libraries
are typically distributed often results in subpar native stacks on Mac. See
:ref:`the native mode documentation <mac symbolification>` for details on these
shortcomings.

We are unlikely to ever support Windows. While the basic technique that Memray
uses to detect memory allocations is possible on Windows, much of the library
would need to be rewritten to support non-POSIX platforms, and none of the
current maintainers have the expertise to do so. We do test in WSL, however.

Supported CPU architectures
---------------------------

For Linux, we test on ``i686``, ``x86-64``, and ``aarch64``. Pre-built wheels
are available on PyPI.

For macOS, we test on ``x86-64`` and ``arm64`` - so, both Intel and Apple
Silicon Macs. Pre-built wheels are available for both architectures.

Supported runtime environments
------------------------------

We require a C++17 runtime. As noted above, macOS 10.14 or higher is required
for a C++17 runtime on Mac.

On Linux we support glibc and musl libc. Other libc's have not been tested, and
issues are likely. We support platforms compatible with the ``manylinux2014``
specification.

Known issues and limitations
----------------------------

* Native stack traces on macOS are often difficult to read, and may be missing
  function calls. See :ref:`the native mode documentation <mac
  symbolification>`.
* We support :ref:`tracking across forks `, but can't
  track across an ``exec``, so if the tracked child process calls `an os.exec
  function`_, even to start a new Python interpreter, we will not be able to
  report the allocations performed in the new process. Notably, the default
  `multiprocessing start method`_ on macOS is "spawn", which leverages
  ``exec``.
* Cython_ functions will not be included in the Python stacks we report, even
  if the Cython module is built with profiling_ support. You'll need to use
  :ref:`native tracking` to see what's happening inside Cython modules.
* We have experimental support for the ``greenlet`` library, which may lead to
  incorrect stacks being reported if :doc:`the Memray API <api>` is used to
  start tracking in one thread while another thread is already making use of
  the Greenlet library.

.. _an os.exec function: https://docs.python.org/3/library/os.html#os.execl
.. _multiprocessing start method: https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
.. _Cython: http://docs.cython.org/en/latest/
.. _profiling: http://docs.cython.org/en/latest/src/tutorial/profiling_tutorial.html

---

### Table (`docs/table.rst`)

Table Reporter
==============

The table reporter generates an HTML document showing a simple tabular view of
the allocations contributing to the tracked process's peak memory usage. Each
source line that contributed to that peak memory usage is given a row in the
generated table, showing the amount of memory it allocated, the type of
allocator it used, and the number of allocations it was responsible for.

.. image:: _static/images/table_example.png


The table can be sorted by each column and searched in the search field. The columns show the following data:

- Thread ID: thread where the allocation happened
- Size: total amount of memory used by all of these allocations
- Allocator: allocator or deallocator function which acquired the memory
- Allocations: total number of allocations performed by this entry
- Location: function name, file and line of the allocation or "???" if unknown

Basic Usage
-----------

The general form of the ``table`` subcommand is:

.. code:: shell

    memray table [options] <results>

The only argument the ``table`` subcommand requires is the capture file
previously generated using :doc:`the run subcommand <run>`.


The output file will be named as ``memray-table-.html`` unless the ``-o`` argument was
specified to override the default name.


CLI Reference
-------------

.. argparse::
   :ref: memray.commands.get_argument_parser
   :path: table
   :prog: memray

---

### Temporary Allocations (`docs/temporary_allocations.rst`)

Temporary allocations
=====================

Some reporters accept the ``--temporary-allocation-threshold=THRESHOLD`` and
``--temporary-allocations`` options. When one of these options is used, the
reporter will show where temporary allocations happened.

What are temporary allocations?
-------------------------------

We consider a memory allocation "temporary" if there are at most ``THRESHOLD``
other allocations performed between when it is allocated and when it is
deallocated. When the threshold is 0 an allocation is considered temporary
only if it is immediately deallocated. When the threshold is 1 an allocation
will be detected as temporary even if 1 other allocation occurs before it is
deallocated.

Seeing where temporary allocations are being performed can help you identify
areas of your code base that frequently perform small allocations and
deallocations. Sometimes these can be avoided by doing one big allocation
instead, or by using a memory pool.

The ``--temporary-allocations`` option behaves as though you passed
``--temporary-allocation-threshold=1``.  We think this is the most interesting
default threshold, because it lets you detect when elements are sequentially
added to a container. That's because growing a container is often performed by
allocating a new, larger buffer for the container, then copying over every
element from the old buffer and deallocating it. When iteratively adding new
elements to a container, each resize results in 1 allocation for a new buffer
before the previously allocated buffer can be freed, and so these buffers
wouldn't be seen as temporary with a threshold of 0.

Detecting inefficient allocation patterns
-----------------------------------------

Temporary allocation detection can help detect inefficient allocation patterns.
For example, consider the following code:

.. code-block:: python

    def foo(n):
        x = []
        for _ in range(n):
            x.append(None)
        return x

    foo(1_000_000)

If we run this code and check the output of the :doc:`the summary reporter `

.. code-block:: shell-session

    $ memray run -fo test.bin example.py
    $ memray summary test.bin --temporary-allocations
    ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┓
    ┃                      ┃     <Total ┃      Total ┃            ┃ Own Memory ┃ Allocation ┃
    ┃ Location             ┃    Memory> ┃   Memory % ┃ Own Memory ┃          % ┃      Count ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━┩
    │ foo at example.py    │   72.486MB │     99.93% │   72.486MB │     99.93% │         78 │
    │ ...                  │            │            │            │            │            │
    └──────────────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘

we can see that our function ``foo()`` is responsible for making 78 allocations
which cumulatively allocate 72.48MB. This happens because the list needs to grow
as we append elements to it.

If we change how the list is built

.. code-block:: python

    def foo(n):
        return [None] * n

    foo(1_000_000)

and run the same commands as before

.. code-block:: shell-session

    $ memray run -fo test.bin example2.py
    $ memray summary test.bin --temporary-allocations
    ┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┓
    ┃                      ┃     <Total ┃      Total ┃            ┃ Own Memory ┃ Allocation ┃
    ┃ Location             ┃    Memory> ┃   Memory % ┃ Own Memory ┃          % ┃      Count ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━┩
    │ foo at example2.py   │    7.629MB │     99.29% │    7.629MB │     99.29% │          1 │
    │ ...                  │            │            │            │            │            │
    └──────────────────────┴────────────┴────────────┴────────────┴────────────┴────────────┘

we can see that ``foo()`` only made 1 allocation, and the total amount of memory
it allocates has been reduced by around 90%. This is because ``[None] * n``
knows how many elements will be present in the final result and allocates
a single chunk of memory large enough to hold all the elements right from the
start, instead of starting off with a small buffer and then repeatedly growing
it as needed.

---

### Transform (`docs/transform.rst`)

Transform Reporter
==================

The transform reporter is used to convert a Memray capture file into a format
that can be consumed by other tools.

Basic Usage
-----------

The general form of the ``transform`` subcommand is:

.. code:: shell

    memray transform <format> [options] <results>

The ``transform`` subcommand requires two arguments: the format you want the
output file to be in, and the location of the capture file previously generated
using :doc:`the run subcommand <run>`.

The output file will be named ``memray-<format>-.<format_suffix>``
unless the ``-o`` argument is used to override the default name.

Available formats
-----------------

These are the formats that you can use as the *format* parameter to the
``transform`` subcommand.

gprof2dot

+-----------------------------+
|           filename          |
|        function_name        |
|           total %           |
|           (self %)          |
|        num_allocs ×         |
+-----------------------------+

percentage %
caller -------------------> callee

This format allows you to produce a `comma separated values
<https://en.wikipedia.org/wiki/Comma-separated_values>`_ file with all the
allocations that contributed to the process's memory high water mark. This can
be very useful to analyze the information using other data analysis tools and
libraries such as `pandas <https://pandas.pydata.org>`_.

Every row in the CSV file represents a call stack where memory that contributed
to the process's memory high water mark was allocated.

The available columns are:

* ``allocator``: the name of the allocator that performed the allocations.
* ``num_allocations``: the number of different allocations performed at this
  location by this thread and not deallocated before the high water mark.
* ``size``: the total size in bytes of the allocations performed at this
  location by this thread and not deallocated before the high water mark.
* ``tid``: the thread id of the thread that performed the allocations.
* ``thread_name``: that thread's id and name, such as ``0x3 (worker)``.
* ``stack_trace``: the stack trace of the allocations. The stack trace is
  represented as a ``|`` separated list of stack frames (most recent call
  first), where each stack frame in the list has the following format:
  ``<function_name>;<file_name>;<line_number>``.

Unlike memray's human-readable reporters, the ``csv`` format does not merge
allocations across threads. Two threads that allocate memory at the same call
stack produce two separate rows, each carrying that thread's own ``tid`` and
``thread_name``, so the output can be grouped or filtered by thread with other
data analysis tools.

speedscope

memray tree [options] <results>

cd docs/tutorials/

python3 -m venv .venv

source .venv/bin/activate

python3 -m pip install -r requirements-tutorial.txt

pytest

pytest tests/test_exercise_1.py

python -m http.server 3000

cd docs/tutorials/exercise_1/

memray run fibonacci.py

memray flamegraph memray-fibonacci.py.<run-id>.bin

The run id will change each time you run the command.

memray run exercise_1/fibonacci.py

memray flamegraph exercise_1/memray-fibonacci.py.<run-id>.bin

The run id will change each time you run the command.

def fibonacci(length):
    # edge cases
    if length < 1:
        return []
    if length == 1:
        return [1]
    if length == 2:
        return [1, 1]

    output = [1, 1]

    for i in range(length - 2):
        output.append(output[i] + output[i + 1])  # <- Here!

    return output

*Toggle to see the sample solution*

def fibonacci(length):
    # edge cases
    if length < 1:
        return
    if length == 1:
        yield 1
        return

    left = right = 1
    yield left
    yield right

    for _ in range(length - 2):
        left, right = right, left + right
        yield right

Sometimes it is better to do all the calculations up front. Generators are far more memory
efficient than lists, but iterating over generators is slightly slower than iterating over
lists, and generators can only be iterated over once. The solution with the best trade-offs will
vary from case to case.

*Hint 1*

@functools.cache
def factorial_plus(self, n: int) -> int:
    return n * self.factorial_plus(n - 1) + self.inc if n else 1 + self.inc

overrides = {'input_encoding': 'ascii',
             'output_encoding': 'latin-1'}
output = publish_string(..., settings_overrides=overrides)

_`body`
    ``parts['body']`` is equivalent to parts['fragment_'].  It is
    *not* equivalent to parts['html_body_'].

_`body_prefix`
    ``parts['body_prefix']`` contains::

        </head>
        
        

    and, if applicable::

        
        ...
        

_`body_pre_docinfo`
    ``parts['body_pre_docinfo]`` contains (as applicable)::

        <h1 class="title">...</h1>
        <h2 class="subtitle" id="...">...</h2>

_`body_suffix`
    ``parts['body_suffix']`` contains::

        

    (the end-tag for ````), the footer division
    if applicable::

        
        ...
        

    and::

        
        </html>

_`docinfo`
    ``parts['docinfo']`` contains the document bibliographic data, the
    docinfo field list rendered as a table.

_`footer`
    ``parts['footer']`` contains the document footer content, meant to
    appear at the bottom of a web page, or repeated at the bottom of
    every printed page.

_`fragment`
    ``parts['fragment']`` contains the document body (*not* the HTML
    ````).  In other words, it contains the entire document,
    less the document title, subtitle, docinfo, header, and footer.

_`head`
    ``parts['head']`` contains ``<meta ... />`` tags and the document
    ``<title>...</title>``.

_`head_prefix`
    ``parts['head_prefix']`` contains the XML declaration, the DOCTYPE
    declaration, the ``<html ...>`` start tag and the ``<head>`` start
    tag.

_`header`
    ``parts['header']`` contains the document header content, meant to
    appear at the top of a web page, or repeated at the top of every
    printed page.

_`html_body`
    ``parts['html_body']`` contains the HTML ```` content, less
    the ```` and ```` tags themselves.

_`html_head`
    ``parts['html_head']`` contains the HTML ``<head>`` content, less
    the stylesheet link and the ``<head>`` and ``</head>`` tags
    themselves.  Since ``publish_parts`` returns Unicode strings and
    does not know about the output encoding, the "Content-Type" meta
    tag's "charset" value is left unresolved, as "%s"::

        <meta http-equiv="Content-Type" content="text/html; charset=%s" />

    The interpolation should be done by client code.

_`html_prolog`
    ``parts['html_prolog]`` contains the XML declaration and the
    doctype declaration.  The XML declaration's "encoding" attribute's
    value is left unresolved, as "%s"::

        <?xml version="1.0" encoding="%s" ?>

    The interpolation should be done by client code.

_`html_subtitle`
    ``parts['html_subtitle']`` contains the document subtitle,
    including the enclosing ``<h2 class="subtitle">`` & ``</h2>``
    tags.

_`html_title`
    ``parts['html_title']`` contains the document title, including the
    enclosing ``<h1 class="title">`` & ``</h1>`` tags.

_`meta`
    ``parts['meta']`` contains all ``<meta ... />`` tags.

_`stylesheet`
    ``parts['stylesheet']`` contains the embedded stylesheet or
    stylesheet link.

_`subtitle`
    ``parts['subtitle']`` contains the document subtitle text and any
    inline markup.  It does not include the enclosing ``<h2>`` &
    ``</h2>`` tags.

_`title`
    ``parts['title']`` contains the document title text and any inline
    markup.  It does not include the enclosing ``<h1>`` & ``</h1>``
    tags.


PEP/HTML Writer

The S5/HTML writer provides the same parts as the `HTML4 writer`_.


HTML5 Writer
````````````

The HTML5 writer provides the same parts as the `HTML4 writer`_.
However, it uses semantic HTML5 elements for the document, header and
footer.


Parts Provided by the LaTeX2e Writer
------------------------------------

See the template files for examples how these parts can be combined
into a valid LaTeX document.

abstract
    ``parts['abstract']`` contains the formatted content of the
    'abstract' docinfo field.

body
    ``parts['body']`` contains the document's content. In other words, it
    contains the entire document, except the document title, subtitle, and
    docinfo.

    This part can be included into another LaTeX document body using the
    ``\input{}`` command.

body_pre_docinfo
    ``parts['body_pre_docinfo]`` contains the ``\maketitle`` command.

dedication
    ``parts['dedication']`` contains the formatted content of the
    'dedication' docinfo field.

docinfo
    ``parts['docinfo']`` contains the document bibliographic data, the
    docinfo field list rendered as a table.

    With ``--use-latex-docinfo`` 'author', 'organization', 'contact',
    'address' and 'date' info is moved to titledata.

    'dedication' and 'abstract' are always moved to separate parts.

fallbacks
    ``parts['fallbacks']`` contains fallback definitions for
    Docutils-specific commands and environments.

head_prefix
    ``parts['head_prefix']`` contains the declaration of
    documentclass and document options.

latex_preamble
    ``parts['latex_preamble']`` contains the argument of the
    ``--latex-preamble`` option.

pdfsetup
     ``parts['pdfsetup']`` contains the PDF properties
     ("hyperref" package setup).

requirements
    ``parts['requirements']`` contains required packages and setup
    before the stylesheet inclusion.

stylesheet
    ``parts['stylesheet']`` contains the embedded stylesheet(s) or
    stylesheet loading command(s).

subtitle
    ``parts['subtitle']`` contains the document subtitle text and any
    inline markup.

title
    ``parts['title']`` contains the document title text and any inline
    markup.

titledata
    ``parts['titledata]`` contains the combined title data in
    ``\title``, ``\author``, and ``\data`` macros.

    With ``--use-latex-docinfo``, this includes the 'author',
    'organization', 'contact', 'address' and 'date' docinfo items.

---

### Benchmarks Benchmarking Cases Docutils Data Docs Api Runtime Settings (`benchmarks/benchmarking/cases/docutils_data/docs/api/runtime-settings.txt`)

===========================
 Docutils Runtime Settings
===========================

:Author: David Goodger, Günter Milde
:Contact: [email protected]
:Date: $Date$
:Revision: $Revision$
:Copyright: This document has been placed in the public domain.

.. contents::


Introduction
============

Docutils runtime settings are assembled from several sources:

* Settings specifications of the selected components_,
* `configuration files`_ (if enabled), and
* command-line options (if enabled).

Docutils overlays default and explicitly specified values from these
sources such that settings behave the way we want and expect them to
behave.


Settings priority
=================

The sources are overlaid in the following order (later sources
overwrite earlier ones):

1. Defaults specified in the `settings_spec`__ and
   `settings_defaults`__ attributes for each component_.

   __ SettingsSpec.settings_spec_
   __ SettingsSpec.settings_defaults_

2. Defaults specified in the `settings_default_overrides`__ attribute
   for each component_.

   __ SettingsSpec.settings_default_overrides_

3. Settings specified in the `settings_overrides`__ parameter of the
   `convenience functions`_ resp. the `settings_overrides` attribute of
   a `Publisher`_ instance.

   __ `settings_overrides parameter`_

4. Settings specified in `active sections`_ of the `configuration files`_
   in the order described in `Configuration File Sections & Entries`_
   (if enabled).

5. Command line options (if enabled).

For details see the ``docutils/__init__.py``, ``docutils/core.py``, and
``docutils.frontend.py`` modules and the implementation description in
`Runtime Settings Processing`_.


.. _SettingsSpec:

SettingsSpec base class
=======================

.. note::
   Implementation details will change with the move to replace the
   deprecated optparse_ module with argparse_.

The `docutils.SettingsSpec` base class is inherited by Docutils
components_ and `frontend.OptionParser`.
It defines the following six **attributes**:

.. _SettingsSpec.settings_spec:

`settings_spec`
   a sequence of

   1. option group title (string or None)

   2. description (string or None)

   3. option tuples with

      a) help text
      b) options string(s)
      c) dictionary with keyword arguments for `OptionParser.add_option()`_
         and an optional "validator", a `frontend.validate_*()` function
         that processes the values (e.g. convert to other data types).

   For examples, see the source of ``frontend.OptionParser.settings_spec``
   or the `settings_spec` attributes of the Docutils components_.

   .. _SettingsSpec.settings_defaults:

`settings_defaults`
   for purely programmatic settings
   (not accessible from command line and configuration files).

   .. _SettingsSpec.settings_default_overrides:

`settings_default_overrides`
   to override defaults for settings
   defined in other components' `setting_specs`.

`relative_path_settings`
   listing settings containing filesystem paths.

   .. _active sections:

`config_section`
   the configuration file section specific to this
   component.

`config_section_dependencies`
   lists configuration files sections
   that should also be read (before the `config_section`).

The last two attributes define which configuration file sections are
"active". See also `Configuration File Sections & Entries`_.


Glossary
========

.. _component:

components
----------

Docutils front-ends and applications combine a selection of
*components* of the `Docutils Project Model`_.

All components inherit the `SettingsSpec`_ base class.
This means that all instances of ``readers.Reader``, ``parsers.Parser``, and
``writers.Writer`` are also instances of ``docutils.SettingsSpec``.

For the determination of runtime settings, ``frontend.OptionParser`` and
the `settings_spec parameter`_ in application settings specifications
are treated as components as well.


.. _convenience function:

convenience functions
---------------------

Applications usually deploy Docutils by one of the
`Publisher convenience functions`_.

All convenience functions accept the following optional parameters:

.. _settings parameter:

`settings`
  a ``frontend.Values`` instance.
  If present, it must be complete.

  No further runtime settings processing is done and the
  following parameters have no effect.

  .. _settings_spec parameter:

`settings_spec`
  a `SettingsSpec`_ subclass or instance containing the settings
  specification for the "Application" itself.
  The instance is added to the components_ (after the generic
  settings, parser, reader, and writer).

  .. _settings_overrides parameter:

`settings_overrides`
  a dictionary which is used to update the
  defaults of the components' settings specifications.

  .. _config_section parameter:

`config_section`
  the name of an application-specific
  `configuration file section`_ for this application.

  Can be specified instead of a `settings_spec` (a new SettingsSpec_
  instance that just defines a configuration section will be created)
  or in addition to a `settings_spec`
  (overriding its `config_section` attribute).


settings_spec
-------------

The name ``settings_spec`` may refer to

a) an instance of the SettingsSpec_ class, or
b) the data structure `SettingsSpec.settings_spec`_ which is used to
   store settings details.


.. References:

.. _Publisher: publisher.html
.. _Publisher convenience functions:
    publisher.html#publisher-convenience-functions
.. _front-end tools: ../user/tools.html
.. _configuration files:
.. _Docutils Configuration: ../user/config.html#configuration-files
.. _configuration file section:
.. _Configuration File Sections & Entries:
    ../user/config.html#configuration-file-sections-entries
.. _Docutils Project Model: ../peps/pep-0258.html#docutils-project-model
.. _Reader: ../peps/pep-0258.html#reader
.. _Runtime Settings Processing: ../dev/runtime-settings-processing.html

.. _optparse: https://docs.python.org/dev/library/optparse.html
.. _argparse: https://docs.python.org/dev/library/argparse.html
.. _OptionParser.add_option():
    https://docs.python.org/dev/library/optparse.html
    #optparse.OptionParser.add_option

---

### Benchmarks Benchmarking Cases Docutils Data Docs Api Transforms (`benchmarks/benchmarking/cases/docutils_data/docs/api/transforms.txt`)

=====================
 Docutils Transforms
=====================

:Author: David Goodger
:Contact: [email protected]
:Revision: $Revision$
:Date: $Date$
:Copyright: This document has been placed in the public domain.


.. contents::

Transforms change the document tree in-place, add to the tree, or prune it.
Transforms resolve references and footnote numbers, process interpreted
text, and do other context-sensitive processing. Each transform is a
subclass of ``docutils.transforms.Transform``.

There are `transforms added by components`_, others (e.g.
``parts.Contents``) are added by the parser, if a corresponding directive is
found in the document.

To add a transform, components (objects inheriting from
Docutils.Component like Readers, Parsers, Writers, Input, Output) overwrite
the ``get_transforms()`` method of their base class. After the Reader has
finished processing, the Publisher calls
``Transformer.populate_from_components()`` with a list of components and all
transforms returned by the component's ``get_transforms()`` method are
stored in a `transformer object` attached to the document tree.


For more about transforms and the Transformer object, see also `PEP
258`_. (The ``default_transforms()`` attribute of component classes mentioned
there is deprecated. Use the ``get_transforms()`` method instead.)

.. _PEP 258: ../peps/pep-0258.html#transformer


Transforms Listed in Priority Order
===================================

Transform classes each have a default_priority attribute which is used by
the Transformer to apply transforms in order (low to high). The default
priority can be overridden when adding transforms to the Transformer object.


==============================  ============================  ========
Transform: module.Class         Added By                      Priority
==============================  ============================  ========
misc.class                      "class" (d/p)                 210

references.Substitutions        standalone (r), pep (r)       220

references.PropagateTargets     standalone (r), pep (r)       260

frontmatter.DocTitle            standalone (r)                320

frontmatter.DocInfo             standalone (r)                340

frontmatter.SectSubTitle        standalone (r)                350

peps.Headers                    pep (r)                       360

peps.Contents                   pep (r)                       380

universal.StripClasses...       Writer (w)                    420

references.AnonymousHyperlinks  standalone (r), pep (r)       440

references.IndirectHyperlinks   standalone (r), pep (r)       460

peps.TargetNotes                pep (r)                       520

references.TargetNotes          peps.TargetNotes (t/p)        0

misc.CallBack                   peps.TargetNotes (t/p)        1

references.TargetNotes          "target-notes" (d/p)          540

references.Footnotes            standalone (r), pep (r)       620

references.ExternalTargets      standalone (r), pep (r)       640

references.InternalTargets      standalone (r), pep (r)       660

parts.SectNum                   "sectnum" (d/p)               710

parts.Contents                  "contents" (d/p),             720
                                peps.Contents (t/p)

universal.StripComments         Reader (r)                    740

peps.PEPZero                    peps.Headers (t/p)            760

components.Filter               *not used*                    780

universal.Decorations           Reader (r)                    820

misc.Transitions                standalone (r), pep (r)       830

universal.ExposeInternals       Reader (r)                    840

references.DanglingReferences   standalone (r), pep (r)       850

universal.SmartQuotes           Parser                        855

universal.Messages              Writer (w)                    860

universal.FilterMessages        Writer (w)                    870

universal.TestMessages          DocutilsTestSupport           880

writer_aux.Compound             *not used, to be removed*     910

writer_aux.Admonitions          html4css1 (w),                920
                                latex2e (w)

misc.CallBack                   n/a                           990
==============================  ============================  ========

Key:

* (r): Reader
* (w): Writer
* (d): Directive
* (t): Transform
* (/p): Via a "pending" node


Transform Priority Range Categories
===================================

====  ====  ================================================
 Priority
----------  ------------------------------------------------
From   To   Category
====  ====  ================================================
   0    99  immediate execution (added by another transform)
 100   199  very early (non-standard)
 200   299  very early
 300   399  early
 400   699  main
 700   799  late
 800   899  very late
 900   999  very late (non-standard)
====  ====  ================================================


Transforms added by components
===============================


readers.Reader:
  | universal.Decorations,
  | universal.ExposeInternals,
  | universal.StripComments

readers.ReReader:
  None

readers.standalone.Reader:
  | references.Substitutions,
  | references.PropagateTargets,
  | frontmatter.DocTitle,
  | frontmatter.SectionSubTitle,
  | frontmatter.DocInfo,
  | references.AnonymousHyperlinks,
  | references.IndirectHyperlinks,
  | references.Footnotes,
  | references.ExternalTargets,
  | references.InternalTargets,
  | references.DanglingReferences,
  | misc.Transitions

readers.pep.Reader:
  | references.Substitutions,
  | references.PropagateTargets,
  | references.AnonymousHyperlinks,
  | references.IndirectHyperlinks,
  | references.Footnotes,
  | references.ExternalTargets,
  | references.InternalTargets,
  | references.DanglingReferences,
  | misc.Transitions,
  | peps.Headers,
  | peps.Contents,
  | peps.TargetNotes

parsers.rst.Parser
  universal.SmartQuotes

writers.Writer:
  | universal.Messages,
  | universal.FilterMessages,
  | universal.StripClassesAndElements

writers.UnfilteredWriter
  None

writers.latex2e.Writer
  writer_aux.Admonitions

writers.html4css1.Writer:
  writer_aux.Admonitions

writers.odf_odt.Writer:
  removes references.DanglingReferences

---

### Benchmarks Benchmarking Cases Docutils Data Docs Dev Distributing (`benchmarks/benchmarking/cases/docutils_data/docs/dev/distributing.txt`)

===============================
 Docutils_ Distributor's Guide
===============================

:Author: Lea Wiemann
:Contact: [email protected]
:Revision: $Revision$
:Date: $Date$
:Copyright: This document has been placed in the public domain.

.. _Docutils: https://docutils.sourceforge.io/

.. contents::

This document describes how to create packages of Docutils (e.g. for
shipping with a Linux distribution).  If you have any questions,
please direct them to the Docutils-develop_ mailing list.

First, please download the most current `release tarball`_ and unpack
it.

.. _Docutils-develop: ../user/mailing-lists.html#docutils-develop
.. _release tarball: https://docutils.sourceforge.io/#download


Dependencies
============

Docutils has the following dependencies:

* Python 3.7 or later is required.
  Use ">= Python 3.7" in the dependencies.

* Docutils may optionally make use of the PIL (`Python Imaging
  Library`_ or Pillow_).  If PIL is present, it is automatically
  detected by Docutils.

* Docutils recommends the `Pygments`_ syntax hightlighter. If available, it
  is used for highlighting the content of `code directives`_ and roles as
  well as included source code files (with the "code" option to the include_
  directive).

* Docutils can use the `recommonmark`_ parser to parse input in
  the Markdown format (new in 0.17).

.. _Python Imaging Library:
    https://en.wikipedia.org/wiki/Python_Imaging_Library
.. _Pillow: https://pypi.org/project/Pillow/
.. _Pygments: https://pygments.org/
.. _recommonmark: https://pypi.org/project/recommonmark/

.. _code directives: ../ref/rst/directives.html#code
.. _include: ../ref/rst/directives.html#include


Python Files
============

The Docutils Python files must be installed into the
``site-packages/`` directory of Python.  Running ``python setup.py
install`` should do the trick, but if you want to place the files
yourself, you can just install the ``docutils/`` directory of the
Docutils tarball to ``/usr/lib/python/site-packages/docutils/``.  In
this case you should also compile the Python files to ``.pyc`` and/or
``.pyo`` files so that Docutils doesn't need to be recompiled every
time it's executed.


Executables
===========

The executable front-end tools are located in the ``tools/`` directory
of the Docutils tarball.

The ``rst2*.py`` tools are intended for end-users. You should install them
to ``/usr/bin/``.  You do not need to change the names (e.g. to
``docutils-rst2html.py``) because the ``rst2`` prefix is unique.


Documentation
=============

The documentation should be generated using ``buildhtml.py``.  To
generate HTML for all documentation files, go to the ``tools/``
directory and run::

    # Place html4css1.css in base directory.
    cp ../docutils/writers/html4css1/html4css1.css ..
    ./buildhtml.py --stylesheet-path=../html4css1.css ..

Then install the following files to ``/usr/share/doc/docutils/`` (or
wherever you install documentation):

* All ``.html`` and ``.txt`` files in the base directory.

* The ``docs/`` directory.

  Do not install the contents of the ``docs/`` directory directly to
  ``/usr/share/doc/docutils/``; it's incomplete and would contain
  invalid references!

* The ``licenses/`` directory.

* ``html4css1.css`` in the base directory.


Removing the ``.txt`` Files
---------------------------

If you are tight with disk space, you can remove all ``.txt`` files in
the tree except for:

* those in the ``licenses/`` directory because they have not been
  processed to HTML and

* ``user/rst/cheatsheet.txt`` and ``user/rst/demo.txt``, which should
  be readable in source form.

Before you remove the ``.txt`` files you should rerun ``buildhtml.py``
with the ``--no-source-link`` switch to avoid broken references to the
source files.


Other Files
===========

You may want to install the Emacs-Lisp files
``tools/editors/emacs/*.el`` into the appropriate directory.


Configuration File
==================

It is possible to have a system-wide configuration file at
``/etc/docutils.conf``.  However, this is usually not necessary.  You
should *not* install ``tools/docutils.conf`` into ``/etc/``.


Tests
=====

While you probably do not need to ship the tests with your
distribution, you can test your package by installing it and then
running ``alltests.py`` from the ``tests/`` directory of the Docutils
tarball.

For more information on testing, view the `Docutils Testing`_ page.

.. _Docutils Testing: https://docutils.sourceforge.io/docs/dev/testing.html

---

### Benchmarks Benchmarking Cases Docutils Data Docs Dev Enthought Plan (`benchmarks/benchmarking/cases/docutils_data/docs/dev/enthought-plan.txt`)

===========================================
 Plan for Enthought API Documentation Tool
===========================================

:Author: David Goodger
:Contact: [email protected]
:Date: $Date$
:Revision: $Revision$
:Copyright: 2004 by `Enthought, Inc. <http://www.enthought.com>`_
:License: `Enthought License`_ (BSD-style)

.. _Enthought License: https://docutils.sourceforge.io/licenses/enthought.txt

This document should be read in conjunction with the `Enthought API
Documentation Tool RFP`__ prepared by Janet Swisher.

__ enthought-rfp.html

.. contents::
.. sectnum::


Introduction
============

In March 2004 at I met Eric Jones, president and CTO of `Enthought,
Inc.`_, at `PyCon 2004`_ in Washington DC.  He told me that Enthought
was using reStructuredText_ for source code documentation, but they
had some issues.  He asked if I'd be interested in doing some work on
a customized API documentation tool.  Shortly after PyCon, Janet
Swisher, Enthought's senior technical writer, contacted me to work out
details.  Some email, a trip to Austin in May, and plenty of Texas
hospitality later, we had a project.  This document will record the
details, milestones, and evolution of the project.

In a nutshell, Enthought is sponsoring the implementation of an open
source API documentation tool that meets their needs.  Fortuitously,
their needs coincide well with the "Python Source Reader" description
in `PEP 258`_.  In other words, Enthought is funding some significant
improvements to Docutils, improvements that were planned but never
implemented due to time and other constraints.  The implementation
will take place gradually over several months, on a part-time basis.

This is an ideal example of cooperation between a corporation and an
open-source project.  The corporation, the project, I personally, and
the community all benefit.  Enthought, whose commitment to open source
is also evidenced by their sponsorship of SciPy_, benefits by
obtaining a useful piece of software, much more quickly than would
have been possible without their support.  Docutils benefits directly
from the implementation of one of its core subsystems.  I benefit from
the funding, which allows me to justify the long hours to my wife and
family.  All the corporations, projects, and individuals that make up
the community will benefit from the end result, which will be great.

All that's left now is to actually do the work!

.. _PyCon 2004: http://pycon.org/dc2004/
.. _reStructuredText: https://docutils.sourceforge.io/rst.html
.. _SciPy: http://www.scipy.org/


Development Plan
================

1. Analyze prior art, most notably Epydoc_ and HappyDoc_, to see how
   they do what they do.  I have no desire to reinvent wheels
   unnecessarily.  I want to take the best ideas from each tool,
   combined with the outline in `PEP 258`_ (which will evolve), and
   build at least the foundation of the definitive Python
   auto-documentation tool.

   .. _Epydoc: http://epydoc.sourceforge.net/
   .. _HappyDoc: http://happydoc.sourceforge.net/
   .. _PEP 258:
      https://docutils.sourceforge.io/docs/peps/pep-0258.html#python-source-reader

2. Decide on a base platform.  The best way to achieve Enthought's
   goals in a reasonable time frame may be to extend Epydoc or
   HappyDoc.  Or it may be necessary to start fresh.

3. Extend the reStructuredText parser.  See `Proposed Changes to
   reStructuredText`_ below.

4. Depending on the base platform chosen, build or extend the
   docstring & doc comment extraction tool.  This may be the biggest
   part of the project, but I won't be able to break it down into
   details until more is known.


Repository
==========

If possible, all software and documentation files will be stored in
the Subversion repository of Docutils and/or the base project, which
are all publicly-available via anonymous pserver access.

The Docutils project is very open about granting Subversion write
access; so far, everyone who asked has been given access.  Any
Enthought staff member who would like Subversion write access will get
it.

If either Epydoc or HappyDoc is chosen as the base platform, I will
ask the project's administrator for CVS access for myself and any
Enthought staff member who wants it.  If sufficient access is not
granted -- although I doubt that there would be any problem -- we may
have to begin a fork, which could be hosted on SourceForge, on
Enthought's Subversion server, or anywhere else deemed appropriate.


Copyright & License
===================

Most existing Docutils files have been placed in the public domain, as
follows::

    :Copyright: This document has been placed in the public domain.

This is in conjunction with the "Public Domain Dedication" section of
COPYING.txt__.

__ https://docutils.sourceforge.io/COPYING.html

The code and documentation originating from Enthought funding will
have Enthought's copyright and license declaration.  While I will try
to keep Enthought-specific code and documentation separate from the
existing files, there will inevitably be cases where it makes the most
sense to extend existing files.

I propose the following:

1. New files related to this Enthought-funded work will be identified
   with the following field-list headers::

       :Copyright: 2004 by Enthought, Inc.
       :License: Enthought License (BSD Style)

   The license field text will be linked to the license file itself.

2. For significant or major changes to an existing file (more than 10%
   change), the headers shall change as follows (for example)::

       :Copyright: 2001-2004 by David Goodger
       :Copyright: 2004 by Enthought, Inc.
       :License: BSD-style

   If the Enthought-funded portion becomes greater than the previously
   existing portion, Enthought's copyright line will be shown first.

3. In cases of insignificant or minor changes to an existing file
   (less than 10% change), the public domain status shall remain
   unchanged.

A section describing all of this will be added to the Docutils
`COPYING`__ instructions file.

If another project is chosen as the base project, similar changes
would be made to their files, subject to negotiation.

__ https://docutils.sourceforge.io/COPYING.html


Proposed Changes to reStructuredText
====================================

Doc Comment Syntax
------------------

The "traits" construct is implemented as dictionaries, where
standalone strings would be Python syntax errors.  Therefore traits
require documentation in comments.  We also need a way to
differentiate between ordinary "internal" comments and documentation
comments (doc comments).

Javadoc uses the following syntax for doc comments::

    /**
     * The first line of a multi-line doc comment begins with a slash
     * and *two* asterisks.  The doc comment ends normally.
     */

Python doesn't have multi-line comments; only single-line.  A similar
convention in Python might look like this::

    ##
    # The first line of a doc comment begins with *two* hash marks.
    # The doc comment ends with the first non-comment line.
    'data' : AnyValue,

    ## The double-hash-marks could occur on the first line of text,
    #  saving a line in the source.
    'data' : AnyValue,

How to indicate the end of the doc comment? ::

    ##
    # The first line of a doc comment begins with *two* hash marks.
    # The doc comment ends with the first non-comment line, or another
    # double-hash-mark.
    ##
    # This is an ordinary, internal, non-doc comment.
    'data' : AnyValue,

    ## First line of a doc comment, terse syntax.
    #  Second (and last) line.  Ends here: ##
    # This is an ordinary, internal, non-doc comment.
    'data' : AnyValue,

Or do we even need to worry about this case?  A simple blank line
could be used::

    ## First line of a doc comment, terse syntax.
    #  Second (and last) line.  Ends with a blank line.

    # This is an ordinary, internal, non-doc comment.
    'data' : AnyValue,

Other possibilities::

    #" Instead of double-hash-marks, we could use a hash mark and a
    #  quotation mark to begin the doc comment.
    'data' : AnyValue,

    ## We could require double-hash-marks on every line.  This has the
    ## added benefit of delimiting the *end* of the doc comment, as
    ## well as working well with line wrapping in Emacs
    ## ("fill-paragraph" command).
    # Ordinary non-doc comment.
    'data' : AnyValue,

    #" A hash mark and a quotation mark on each line looks funny, and
    #" it doesn't work well with line wrapping in Emacs.
    'data' : AnyValue,

These styles (repeated on each line) work well with line wrapping in
Emacs::

    ##  #>  #|  #-  #%  #!  #*

These styles do *not* work well with line wrapping in Emacs::

    #"  #'  #:  #)  #.  #/  #@  #$  #^  #=  #+  #_  #~

The style of doc comment indicator used could be a runtime, global
and/or per-module setting.  That may add more complexity than it's
worth though.


Recommendation

# This is an ordinary non-doc comment.

#* This is a documentation comment, with an asterisk after the
#* hash marks on every line.
'data' : AnyValue,

# This is an ordinary non-doc comment.

## This is a documentation comment, with double-hash-marks on
## every line.
'data' : AnyValue,

Enthought's Traits system has switched to a metaclass base, and traits
are now defined via ordinary attributes.  Therefore doc comments are
no longer absolutely necessary; attribute docstrings will suffice.
Doc comments may still be desirable though, since they allow
documentation to precede the thing being documented.


Docstring Density & Whitespace Minimization
-------------------------------------------

One problem with extensively documented classes & functions, is that
there is a lot of screen space wasted on whitespace.  Here's some
current Enthought code (from lib/cp/fluids/gassmann.py)::

    def max_gas(temperature, pressure, api, specific_gravity=.56):
        """
        Computes the maximum dissolved gas in oil using Batzle and
        Wang (1992).

        Parameters
        ----------
        temperature : sequence
            Temperature in degrees Celsius
        pressure : sequence
            Pressure in MPa
        api : sequence
            Stock tank oil API
        specific_gravity : sequence
            Specific gravity of gas at STP, default is .56

        Returns
        -------
        max_gor : sequence
            Maximum dissolved gas in liters/liter

        Description
        -----------
        This estimate is based on equations given by Mavko, Mukerji,
        and Dvorkin, (1998, pp. 218-219, or 2003, p. 236) obtained
        originally from Batzle and Wang (1992).
        """
        code...

The docstring is 24 lines long.

Rather than using subsections, field lists (which exist now) can save
6 lines::

    def max_gas(temperature, pressure, api, specific_gravity=.56):
        """
        Computes the maximum dissolved gas in oil using Batzle and
        Wang (1992).

        :Parameters:
            temperature : sequence
                Temperature in degrees Celsius
            pressure : sequence
                Pressure in MPa
            api : sequence
                Stock tank oil API
            specific_gravity : sequence
                Specific gravity of gas at STP, default is .56
        :Returns:
            max_gor : sequence
                Maximum dissolved gas in liters/liter
        :Description: This estimate is based on equations given by
            Mavko, Mukerji, and Dvorkin, (1998, pp. 218-219, or 2003,
            p. 236) obtained originally from Batzle and Wang (1992).
        """
        code...

As with the "Description" field above, field bodies may begin on the
same line as the field name, which also saves space.

The output for field lists is typically a table structure.  For
example:

    :Parameters:
        temperature : sequence
            Temperature in degrees Celsius
        pressure : sequence
            Pressure in MPa
        api : sequence
            Stock tank oil API
        specific_gravity : sequence
            Specific gravity of gas at STP, default is .56
    :Returns:
        max_gor : sequence
            Maximum dissolved gas in liters/liter
    :Description:
        This estimate is based on equations given by Mavko,
        Mukerji, and Dvorkin, (1998, pp. 218-219, or 2003, p. 236)
        obtained originally from Batzle and Wang (1992).

But the definition lists describing the parameters and return values
are still wasteful of space.  There are a lot of half-filled lines.

Definition lists are currently defined as::

    term : classifier
        definition

Where the classifier part is optional.  Ideas for improvements:

1. We could allow multiple classifiers::

       term : classifier one : two : three ...
           definition

2. We could allow the definition on the same line as the term, using
   some embedded/inline markup:

   * "--" could be used, but only in limited and well-known contexts::

         term -- definition

     This is the syntax used by StructuredText (one of
     reStructuredText's predecessors).  It was not adopted for
     reStructuredText because it is ambiguous -- people often use "--"
     in their text, as I just did.  But given a constrained context,
     the ambiguity would be acceptable (or would it?).  That context
     would be: in docstrings, within a field list, perhaps only with
     certain well-defined field names (parameters, returns).

   * The "constrained context" above isn't really enough to make the
     ambiguity acceptable.  Instead, a slightly more verbose but far
     less ambiguous syntax is possible::

         term === definition

     This syntax has advantages.  Equals signs lend themselves to the
     connotation of "definition".  And whereas one or two equals signs
     are commonly used in program code, three equals signs in a row
     have no conflicting meanings that I know of.  (Update: there
     *are* uses out there.)

   The problem with this approach is that using inline markup for
   structure is inherently ambiguous in reStructuredText.  For
   example, writing *about* definition lists would be difficult::

       ``term === definition`` is an example of a compact definition list item

   The parser checks for structural markup before it does inline
   markup processing.  But the "===" should be protected by its inline
   literal context.

3. We could allow the definition on the same line as the term, using
   structural markup.  A variation on bullet lists would work well::

       : term :: definition
       : another term :: and a definition that
         wraps across lines

   Some ambiguity remains::

       : term ``containing :: double colons`` :: definition

   But the likelihood of such cases is negligible, and they can be
   covered in the documentation.

   Other possibilities for the definition delimiter include::

       : term : classifier -- definition
       : term : classifier --- definition
       : term : classifier : : definition
       : term : classifier === definition

The third idea currently has the best chance of being adopted and
implemented.


Recommendation

def max_gas(temperature, pressure, api, specific_gravity=.56):
    """
    Computes the maximum dissolved gas in oil using Batzle and
    Wang (1992).

    :Parameters:
        : temperature : sequence :: Temperature in degrees Celsius
        : pressure : sequence :: Pressure in MPa
        : api : sequence :: Stock tank oil API
        : specific_gravity : sequence :: Specific gravity of gas at
          STP, default is .56
    :Returns:
        : max_gor : sequence :: Maximum dissolved gas in liters/liter
    :Description: This estimate is based on equations given by
        Mavko, Mukerji, and Dvorkin, (1998, pp. 218-219, or 2003,
        p. 236) obtained originally from Batzle and Wang (1992).
    """
    code...

My *favorite* language is Python_.

.. _Python: https://www.python.org/

[uninteresting HTML code removed]
<body>
<div class="document">
<p>My <em>favorite</em> language is <a class="reference" href="https://www.python.org/">Python</a>.</p>
</div>
</body>
</html>

$ quicktest.py test.txt
<document source="test.txt">
    <paragraph>
        My
        <emphasis>
            favorite
         language is
        <reference name="Python" refname="python">
            Python
        .
    <target ids="python" names="python" refuri="https://www.python.org/">

$ rst2pseudoxml.py test.txt
<document source="test.txt">
    <paragraph>
        My
        <emphasis>
            favorite
         language is
        <reference name="Python" **refuri="https://www.python.org/"**>
            Python
        .
    <target ids="python" names="python" ``refuri="https://www.python.org/"``>

$ rst2html.py --link-stylesheet test.txt
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="https://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.3.10: https://docutils.sourceforge.io/" />
<title></title>
<link rel="stylesheet" href="../docutils/writers/html4css1/html4css1.css" type="text/css" />
</head>
<body>
<div class="document">
<p>My <em>favorite</em> language is <a class="reference" href="https://www.python.org/">Python</a>.</p>
</div>
</body>
</html>

document = docutils.core.publish_doctree(...)

output = docutils.core.publish_from_doctree(document, ...)

We reject: kings, presidents, and voting.  We believe in: rough
consensus and running code.

The number of contributors [to] projects is strongly and inversely
correlated with the number of hoops each project makes a user go
through to contribute.

Open source dictates that you lose a certain amount of control
over your codebase, and that's okay with us.

Before a new feature, an API change, or a complex, disruptive, or
controversial bug fix can be checked in to the core or into a
maintenance branch, it must undergo review.  These are the criteria:

* The branch must be complete, and include full documentation and
  tests.

* There should ideally be one branch merge commit per feature or
  change.  In other words, each branch merge should represent a
  coherent change set.

* The code must be stable and uncontroversial.  Moving targets and
  features under debate are not ready to be merged.

* The code must work.  The test suite must complete with no failures.
  See `Docutils Testing`_.

The review process will ensure that at least one other set of eyeballs
& brains sees the code before it enters the core.  In addition to the
above, the general `Check-ins`_ policy (below) also applies.

.. _Docutils testing: testing.html


Check-ins
---------

Changes or additions to the Docutils core and maintenance branches
carry a commitment to the Docutils user community.  Developers must be
prepared to fix and maintain any code they have committed.

The Docutils core (``trunk/docutils`` directory) and maintenance
branches should always be kept in a stable state (usable and as
problem-free as possible).  All changes to the Docutils core or
maintenance branches must be in `good shape`_, usable_, documented_,
tested_, and `reasonably complete`_. Starting with version 1.0, they must
also comply with the `backwards compatibility policy`_.

* _`Good shape` means that the code is clean, readable, and free of
  junk code (unused legacy code; by analogy to "junk DNA").

* _`Usable` means that the code does what it claims to do.  An "XYZ
  Writer" should produce reasonable XYZ output.

* _`Documented`: The more complete the documentation the better.
  Modules & files must be at least minimally documented internally.
  `Docutils Front-End Tools`_ should have a new section for any
  front-end tool that is added.  `Docutils Configuration Files`_
  should be modified with any settings/options defined.  For any
  non-trivial change, the HISTORY.txt_ file should be updated.

* _`Tested` means that unit and/or functional tests, that catch all
  bugs fixed and/or cover all new functionality, have been added to
  the test suite.  These tests must be checked by running the test
  suite under all supported Python versions, and the entire test suite
  must pass.  See `Docutils Testing`_.

* _`Reasonably complete` means that the code must handle all input.
  Here "handle" means that no input can cause the code to fail (cause
  an exception, or silently and incorrectly produce nothing).
  "Reasonably complete" does not mean "finished" (no work left to be
  done).  For example, a writer must handle every standard element
  from the Docutils document model; for unimplemented elements, it
  must *at the very least* warn that "Output for element X is not yet
  implemented in writer Y".

If you really want to check code directly into the Docutils core,
you can, but you must ensure that it fulfills the above criteria
first.  People will start to use it and they will expect it to work!
If there are any issues with your code, or if you only have time for
gradual development, you should put it on a branch or in the sandbox
first.  It's easy to move code over to the Docutils core once it's
complete.

It is the responsibility and obligation of all developers to keep the
Docutils core and maintenance branches stable.  If a commit is made to
the core or maintenance branch which breaks any test, the solution is
simply to revert the change.  This is not vindictive; it's practical.
We revert first, and discuss later.

Docutils will pursue an open and trusting policy for as long as
possible, and deal with any aberrations if (and hopefully not when)
they happen.  We'd rather see a torrent of loose contributions than
just a trickle of perfect-as-they-stand changes.  The occasional
mistake is easy to fix.  That's what version control is for!

.. _Docutils Front-End Tools: ../user/tools.html
.. _Docutils Configuration Files: ../user/config.html
.. _HISTORY.txt: ../../HISTORY.txt


.. _`Version Numbering`:

Version Identification
======================

The state of development of the current Docutils codebase is stored in
two forms: the sequence `docutils.__version_info__`_ and the
`PEP 440`_ conformant text string `docutils.__version__`_.
See also the `Docutils Release Procedure`_

.. _Docutils Release Procedure: release.html#version-numbers


``docutils.__version_info__``
-----------------------------

``docutils.__version_info__`` is an instance of ``docutils.VersionInfo``
based on collections.namedtuple_. It is modelled on `sys.version_info`_
and has the following attributes:

major : non-negative integer
    **Major releases** (x.0, e.g. 1.0) will be rare, and will
    represent major changes in API, functionality, or commitment.  The
    major number will be bumped to 1 when the project is
    feature-complete, and may be incremented later if there is a major
    change in the design or API.  When Docutils reaches version 1.0,
    the major APIs will be considered frozen.
    For details, see the `backwards compatibility policy`_.

minor : non-negative integer
    Releases that change the minor number (x.y, e.g. 0.5) will be
    **feature releases**; new features from the `Docutils core`_ will
    be included.

micro : non-negative integer
    Releases that change the micro number (x.y.z, e.g. 0.4.1) will be
    **bug-fix releases**.  No new features will be introduced in these
    releases; only bug fixes will be included.

    The micro number is omitted from `docutils.__version__`_ when it
    equals zero.

_`releaselevel` : text string
    The release level indicates the `development status`_ (or phase)
    of the project's codebase:

    =============  ==========  ===============================================
    Release Level  Label [#]_  Description
    =============  ==========  ===============================================
    alpha          ``a``       Reserved for use after major experimental
                               changes, to indicate an unstable codebase.

    beta           ``b``       Indicates active development, between releases.

    candidate      ``rc``      Release candidate: indicates that the
                               codebase is ready to release unless
                               significant bugs emerge.

    final                      Indicates an official project release.
    =============  ==========  ===============================================

    .. [#] The labels are used in the `docutils.__version__`_ pre-release
       segment.

    .. _development status:
       https://en.wikipedia.org/wiki/Software_release_life_cycle

_`serial` : non-negative integer
    The serial number is zero for final releases and incremented
    whenever a new pre-release is begun.

_`release` : boolean
    True for official releases and pre-releases, False during
    development.

* One of *{major, minor, micro, serial}* is incremented after each
  release, and the lower-order numbers are reset to 0.

* The default state of the repository during active development is
  release level = "beta", serial = 0, release = False.

``docutils.__version_info__`` can be used to test for a minimally
required version, e.g. ::

    docutils.__version_info__ >= (0, 13)

is True for all versions after ``"0.13"``.

.. _collections.namedtuple:
   https://docs.python.org/3/library/collections.html#collections.namedtuple
.. _sys.version_info:
   https://docs.python.org/3/library/sys.html#sys.version_info

``docutils.__version__``
------------------------

The text string ``docutils.__version__`` is a human readable,
`PEP 440`_-conforming version specifier.  For version comparison
operations, use `docutils.__version_info__`_.

``docutils.__version__`` takes the following form::

    "<major>.<minor>[.<micro>][<releaselevel>[<serial>]][.dev]"
     <--- release segment ---><-- pre-release segment -><- development ->

* The *pre-release segment* contains a label representing the
  releaselevel_ ("a", "b", or "rc") and eventually a serial_ number
  (omitted, if zero).

* The *development segment* is ``".dev"`` during active development
  (release_ == False) and omitted for official releases and pre-releases.

Examples of ``docutils.__version__`` identifiers, over the course of
normal development (without branches), in ascending order:

==============================  =============================
Release Level                   Version Identifier
==============================  =============================
final (release)                 0.14
beta (development) [#dev]_      0.15b.dev
beta (release)  [#skip]_        0.15b
candidate 1 (dev.)              0.15rc1.dev
candidate 1 (release)           0.15rc1
candidate 2 (dev.) [#skip]_     0.15rc2.dev
candidate 2 (release) [#skip]_  0.15rc2
...
final (release)                 0.15
beta (development) [#dev]_      0.16b.dev
==============================  =============================

.. [#dev] Default active development state between releases.
.. [#skip] These steps may be skipped.

.. _PEP 440: https://peps.python.org/pep-0440/

Policy History
--------------

* Prior to version 0.4, Docutils didn't have an official version
  numbering policy, and micro releases contained both bug fixes and
  new features.

* An earlier version of this policy was adopted in October 2005, and
  took effect with Docutils version 0.4.

* This policy was updated in June 2017 for Docutils version 0.14. See
  `Feature Request #50`_ and the `discussion on docutils-devel`__ from
  May 28 to June 20 2017.

  .. _Feature Request #50:
     https://sourceforge.net/p/docutils/feature-requests/50/
  __ https://sourceforge.net/p/docutils/mailman/message/35903816/


Backwards Compatibility Policy
==============================

.. note:: The backwards compatibility policy outlined below is a stub.

Docutils' backwards compatibility policy follows the rules for Python in
:PEP:`387`.

* The scope of the public API is laid out at the start of the `backwards
  compatibility rules`_.

* The rules for `making incompatible changes`_ apply.

A majority of projects depends on Docutils indirectly, via the Sphinx_
document processor.

* Sphinx developers should be given the chance to fix or work around a
  DeprecationWarning_ in the Sphinx development version before a new
  Docutils version is released. Otherwise, use a PendingDeprecationWarning_.

Changes that may affect end-users (e.g. by requiring changes to the
configuration file or potentially breaking custom style sheets) should be
announced with a FutureWarning_.

.. _backwards compatibility rules:
   https://peps.python.org/pep-0387/#backwards-compatibility-rules
.. _making incompatible changes:
   https://peps.python.org/pep-0387/#making-incompatible-changes
.. _Sphinx: https://www.sphinx-doc.org/
.. _DeprecationWarning:
   https://docs.python.org/3/library/exceptions.html#DeprecationWarning
.. _PendingDeprecationWarning:
   https://docs.python.org/3/library/exceptions.html#PendingDeprecationWarning
.. _FutureWarning:
   https://docs.python.org/3/library/exceptions.html#FutureWarning


Snapshots
=========

Snapshot tarballs can be downloaded from the repository (see the "download
snapshot" button in the head of the code listing table).

* the `Docutils core`_, representing the current cutting-edge state of
  development;

* the `sandbox directory`_ with contributed projects and extensions from
  `the Sandbox`_;

.. * maintenance branches, for bug fixes;

   TODO: do we have active maintenance branches?
   (the only branch looking like a maintenance branch is
   https://sourceforge.net/p/docutils/code/HEAD/tree/branches/docutils-0.4)

* `development branches`_, representing ongoing development efforts to bring
  new features into Docutils.

.. _Docutils core:
   https://sourceforge.net/p/docutils/code/HEAD/tree/trunk/docutils
.. _development branches:
   https://sourceforge.net/p/docutils/code/HEAD/tree/branches/


Setting Up For Docutils Development
===================================

When making changes to the code, testing_ is a must.  The code should
be run to verify that it produces the expected results, and the entire
test suite should be run too.  The modified Docutils code has to be
accessible to Python for the tests to have any meaning.
See `editable installs`_ for ways to keep the Docutils code
accessible during development.

.. _testing: tested_
.. _editable installs: repository.html#editable-installs


Mailing Lists
=============

Developers are recommended to subscribe to all `Docutils mailing
lists`_.

.. _Docutils mailing lists: ../user/mailing-lists.html


The Wiki
========

There is a development wiki at http://docutils.python-hosting.com/ as
a scratchpad for transient notes.  Please use the repository for
permanent document storage.

Extensions and Related Projects
===============================

The Sandbox
-----------

The `sandbox directory`_ is a place to play around, to try out and
share ideas.  It's a part of the Subversion repository but it isn't
distributed as part of Docutils releases.  Feel free to check in code
to the sandbox; that way people can try it out but you won't have to
worry about it working 100% error-free, as is the goal of the Docutils
core.  A project-specific subdirectory should be created for each new
project.  Any developer who wants to play in the sandbox may do so,
but project directories are recommended over personal directories,
which discourage collaboration.  It's OK to make a mess in the
sandbox!  But please, play nice.

Please update the `sandbox README`_ file with links and a brief
description of your work.

In order to minimize the work necessary for others to install and try
out new, experimental components, the following sandbox directory
structure is recommended::

    sandbox/
        project_name/ # For a collaborative project.
            README.txt  # Describe the requirements, purpose/goals, usage,
                        # and list the maintainers.
            docs/
                ...
            component.py    # The component is a single module.
                            # *OR* (but *not* both)
            component/      # The component is a package.
                __init__.py  # Contains the Reader/Writer class.
                other1.py    # Other modules and data files used
                data.txt     # by this component.
                ...
            test/       # Test suite.
                ...
            tools/      # For front ends etc.
                ...
            setup.py    # Install the component code and tools/ files
                        # into the right places.
        userid/       # For *temporary* personal space.

Some sandbox projects are destined to move to the Docutils core once
completed.  Others, such as add-ons to Docutils or applications of
Docutils, may graduate to become `parallel projects`_.

.. _sandbox README: https://docutils.sourceforge.io/sandbox/README.html
.. _sandbox directory:
   https://sourceforge.net/p/docutils/code/HEAD/tree/trunk/sandbox/


.. _parallel project:

Parallel Projects
-----------------

Parallel projects contain useful code that is not central to the
functioning of Docutils.  Examples are specialized add-ons or
plug-ins, and applications of Docutils.  They use Docutils, but
Docutils does not require their presence to function.

An official parallel project will have its own directory beside (or
parallel to) the main ``docutils`` directory in the Subversion
repository.  It can have its own web page in the
docutils.sourceforge.io domain, its own file releases and
downloadable snapshots, and even a mailing list if that proves useful.
However, an official parallel project has implications: it is expected
to be maintained and continue to work with changes to the core
Docutils.

A parallel project requires a project leader, who must commit to
coordinate and maintain the implementation:

* Answer questions from users and developers.
* Review suggestions, bug reports, and patches.
* Monitor changes and ensure the quality of the code and
  documentation.
* Coordinate with Docutils to ensure interoperability.
* Put together official project releases.

Of course, related projects may be created independently of Docutils.
The advantage of a parallel project is that the SourceForge
environment and the developer and user communities are already
established.  Core Docutils developers are available for consultation
and may contribute to the parallel project.  It's easier to keep the
projects in sync when there are changes made to the core Docutils
code.

Other related projects
----------------------

Many related but independent projects are listed in the Docutils
`link list`_. If you want your project to appear there, drop a note at
the Docutils-develop_ mailing list.

.. _link list: https://docutils.sourceforge.io/docs/user/links.html
.. _docutils-develop: docs/user/mailing-lists.html#docutils-develop


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End:

---

### Benchmarks Benchmarking Cases Docutils Data Docs Dev Pysource (`benchmarks/benchmarking/cases/docutils_data/docs/dev/pysource.txt`)

======================
 Python Source Reader
======================
:Author: David Goodger
:Contact: [email protected]
:Revision: $Revision$
:Date: $Date$
:Copyright: This document has been placed in the public domain.

This document explores issues around extracting and processing
docstrings from Python modules.

For definitive element hierarchy details, see the "Python Plaintext
Document Interface DTD" XML document type definition, pysource.dtd_
(which modifies the generic docutils.dtd_).  Descriptions below list
'DTD elements' (XML 'generic identifiers' or tag names) corresponding
to syntax constructs.


.. contents::


Model
=====

The Python Source Reader ("PySource") model that's evolving in my mind
goes something like this:

1. Extract the docstring/namespace [#]_ tree from the module(s) and/or
   package(s).

   .. [#] See `Docstring Extractor`_ below.

2. Run the parser on each docstring in turn, producing a forest of
   doctrees (per nodes.py).

3. Join the docstring trees together into a single tree, running
   transforms:

   - merge hyperlinks
   - merge namespaces
   - create various sections like "Module Attributes", "Functions",
     "Classes", "Class Attributes", etc.; see pysource.dtd_
   - convert the above special sections to ordinary doctree nodes

4. Run transforms on the combined doctree.  Examples: resolving
   cross-references/hyperlinks (including interpreted text on Python
   identifiers); footnote auto-numbering; first field list ->
   bibliographic elements.

   (Or should step 4's transforms come before step 3?)

5. Pass the resulting unified tree to the writer/builder.

I've had trouble reconciling the roles of input parser and output
writer with the idea of modes ("readers" or "directors").  Does the
mode govern the transformation of the input, the output, or both?
Perhaps the mode should be split into two.

For example, say the source of our input is a Python module.  Our
"input mode" should be the "Python Source Reader".  It discovers (from
``__docformat__``) that the input parser is "reStructuredText".  If we
want HTML, we'll specify the "HTML" output formatter.  But there's a
piece missing.  What *kind* or *style* of HTML output do we want?
PyDoc-style, LibRefMan style, etc.  (many people will want to specify
and control their own style).  Is the output style specific to a
particular output format (XML, HTML, etc.)?  Is the style specific to
the input mode?  Or can/should they be independent?

I envision interaction between the input parser, an "input mode" , and
the output formatter.  The same intermediate data format would be used
between each of these, being transformed as it progresses.


Docstring Extractor
===================

We need code that scans a parsed Python module, and returns an ordered
tree containing the names, docstrings (including attribute and
additional docstrings), and additional info (in parentheses below) of
all of the following objects:

- packages
- modules
- module attributes (+ values)
- classes (+ inheritance)
- class attributes (+ values)
- instance attributes (+ values)
- methods (+ formal parameters & defaults)
- functions (+ formal parameters & defaults)

(Extract comments too?  For example, comments at the start of a module
would be a good place for bibliographic field lists.)

In order to evaluate interpreted text cross-references, namespaces for
each of the above will also be required.

See python-dev/docstring-develop thread "AST mining", started on
2001-08-14.


Interpreted Text
================

DTD elements: package, module, class, method, function,
module_attribute, class_attribute, instance_attribute, variable,
parameter, type, exception_class, warning_class.

To classify identifiers explicitly, the role is given along with the
identifier in either prefix or suffix form::

    Use :method:`Keeper.storedata` to store the object's data in
    `Keeper.data`:instance_attribute:.

The role may be one of 'package', 'module', 'class', 'method',
'function', 'module_attribute', 'class_attribute',
'instance_attribute', 'variable', 'parameter', 'type',
'exception_class', 'exception', 'warning_class', or 'warning'.  Other
roles may be defined.

.. _pysource.dtd: pysource.dtd
.. _docutils.dtd: ../ref/docutils.dtd



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   fill-column: 70
   End:

---

### Benchmarks Benchmarking Cases Docutils Data Docs Dev Release (`benchmarks/benchmarking/cases/docutils_data/docs/dev/release.txt`)

=============================
 Docutils_ Release Procedure
=============================

:Authors: David Goodger; Lea Wiemann; open to all Docutils developers
:Contact: [email protected]
:Date: $Date$
:Revision: $Revision$
:Copyright: This document has been placed in the public domain.

.. _Docutils: https://docutils.sourceforge.io/

Releasing (post 2020)
---------------------

* Announce the upcoming release on docutils-develop list.

  Consider **feature freeze** or/and **check-in freeze** .

* Update RELEASE-NOTES.txt add section ``Release <version>``.

  Consult HISTORY.txt for important changes.

* Change HISTORY.txt title ``Changes Since `` to ``Release <version>``.

* Set new version with ``sandbox/infrastructure/set_version.sh <version>``

  Check what was changed with version control system by ``set_version.sh``

  Run tests ::

    export PYTHONWARNINGS=default
    python3 test/alltests.py

  or use tox.
    
  ``export PYTHONWARNINGS=default`` prints DeprecationWarnings in python3.

* Generate universal wheel and source-distribution::

    python3 setup.py sdist
    python3 setup.py bdist_wheel --universal

* Upload universal wheel and source to test.pypi::

    python3 -m twine upload --repository-url https://test.pypi.org/legacy/ dist/*

  Test in venv ::

    python3 -m venv du3 ; cd du3
    export PYTHONPATH= ; . bin/activate

    python -m pip install --index-url https://test.pypi.org/simple/ --no-deps docutils

    cp -Lr ../docutils-code/docutils/test .
    python test/alltests.py

    python -m pip uninstall docutils
    deactivate ; cd .. ; rm -r du3

* Commit changes ... the changed version number.

* tag 0.## (Note: only directory docutils is copied)::

    svn copy svn+ssh://[email protected]/p/docutils/code/trunk/docutils \
             svn+ssh://[email protected]/p/docutils/code/tags/docutils-0.## \
             -m "tagging release 0.##"

* Update your source directory. 
* Rebuild universal wheel and source-distribution ::

    python3 setup.py sdist
    python3 setup.py bdist_wheel --universal

* Now upload to pypi::

    python3 -m twine upload  dist/docutils-0.##*

* Remove previous package from local cache::

    find .cache/pip/wheels -name docutils\*whl -exec rm -v -i {} \;

* and test::

    python3 -m venv du3 ; cd du3
    export PYTHONPATH= ; . bin/activate

    pip install --no-deps docutils
    cp -Lr ../docutils-code/docutils/test .
    python test/alltests.py

    deactivate ; cd .. ; rm -r du3

* Notify to docutils-developer and user.

* upload source and generated html to sf-htdocs/0.## ::

    mkdir tmp1
    cd tmp1
    tar xzvf ../dist/docutils-0.##.tar.gz
    cd docutils-0.##/
    tools/buildhtml.py .
    find . -name \*.pyc -exec rm -v {} \;
    find . -name __pycache__ -exec rmdir -v {} \;
    rm -r docutils.egg-info
    rsync -e ssh -r -t ./ web.sourceforge.net:/home/project-web/docutils/htdocs/0.##

* Check web/index.txt for necessary corrections.
* Run sandbox/infrastructure/docutils-update.local to update web-content.
* Release to sourceforge.

  - Upload tar.gz and 0.16 release notes to sourceforge.
  - Select docutils-0.16.tar.gz as default for all OS.  

* set_version 0.#.#+1b.dev
* test with py3
* docutils/HISTORY.txt: add title "Changes Since 0.##"
* run sandbox/infrastructure/docutils-update.local


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End:

---

### Benchmarks Benchmarking Cases Docutils Data Docs Dev Repository (`benchmarks/benchmarking/cases/docutils_data/docs/dev/repository.txt`)

=====================================
 The Docutils_ Version Repository
=====================================

:Author: Lea Wiemann, Docutils developers
:Contact: [email protected]
:Revision: $Revision$
:Date: $Date$
:Copyright: This document has been placed in the public domain.

.. _Docutils: https://docutils.sourceforge.io/

.. admonition:: Quick Instructions

   To get a checkout of the Docutils source tree (with the
   sandboxes) with SVN_, type ::

       svn checkout https://svn.code.sf.net/p/docutils/code/trunk docutils-code

   Users of Git_ can clone a mirror of the docutils repository with ::

      git clone git://repo.or.cz/docutils.git

   If you are going to commit changes to the repository, please read
   the **whole document**, especially the section "`Information for
   Developers`_"!

Docutils uses a Subversion_ (SVN) repository located at
``docutils.svn.sourceforge.net``.

While Unix and Mac OS X users will probably prefer the standard
Subversion command line interface, Windows user may want to try
TortoiseSVN_, a convenient explorer extension.  The instructions apply
analogously.

There is a Git_ mirror at http://repo.or.cz/docutils.git providing
`web access`_ and the base for `creating a local Git clone`_.
[#github-mirrors]_

For the project policy on repository use (check-in requirements,
branching, etc.), please see the `Docutils Project Policies`__.

__ policies.html#subversion-repository

.. _SVN:
.. _Subversion: https://subversion.apache.org/
.. _TortoiseSVN: https://tortoisesvn.net/
.. _SourceForge.net: https://sourceforge.net/
.. _Git: http://git-scm.com/

.. contents::


Accessing the Repository
========================

Web Access
----------

The repository can be browsed and examined via the web at
https://sourceforge.net/p/docutils/code.

Alternatively, use the web interface at http://repo.or.cz/docutils.git.
[#github-mirrors]_

.. [#github-mirrors] There are also 3rd-party mirrors and forks at
   GitHub, some of them orphaned. At the time of this writing (2021-11-03),
   https://github.com/live-clones/docutils/tree/master/docutils
   provides an hourly updated clone.

Repository Access Methods
-------------------------

To get a checkout, first determine the root of the repository depending
on your preferred protocol:

anonymous access: (read only)
    Subversion_: ``https://svn.code.sf.net/p/docutils/code``

    Git_: ``git://repo.or.cz/docutils.git``

`developer access`_: (read and write)
    ``svn+ssh://@svn.code.sf.net/p/docutils/code``

Checking Out the Repository
---------------------------

.. _creating a local Git clone:

Git_ users can clone a mirror of the docutils repository with ::

      git clone git://repo.or.cz/docutils.git

and proceed according to the `Git documentation`_.
Developer access (read and write) is possible with `git svn`_.

.. _Git documentation: https://git.wiki.kernel.org/index.php/GitDocumentation
.. _git svn: https://git.wiki.kernel.org/index.php/Git-svn

Subversion_ users can use the following commands
(substitute your preferred repository root for ROOT):

* To check out only the current main source tree of Docutils, type ::

    svn checkout ROOT/trunk/docutils

* To check out everything (main tree, sandboxes, web site, and parallel
  projects), type ::

    svn checkout ROOT/trunk docutils

  This will create a working copy of the whole trunk in a new directory
  called ``docutils``.

Note that you probably do *not* want to check out the ROOT itself
(without "/trunk"), because then you'd end up fetching the whole
Docutils tree for every branch and tag over and over again.

To update your working copy later on, ``cd`` into the working copy and
type ::

    svn update

Switching the Repository Root
-----------------------------

If you changed your mind and want to use a different repository root,
``cd`` into your working copy and type::

    svn switch --relocate OLDROOT NEWROOT


Editable installs
=================

There are several ways to ensure that edits to the Docutils code are
picked up by Python.
We'll assume that the Docutils "trunk" is checked out under the
``~/projects/`` directory.

1. Do an `editable install`__ with pip_::

     python3 -m pip install -e ~/projects/docutils/docutils

   __ https://pip.pypa.io/en/stable/cli/pip_install/#editable-installs

2. Install in `development mode`__ with setuptools_.

   __ https://setuptools.pypa.io/en/latest/userguide/development_mode.html
      #development-mode

   .. _install manually:

3. Install "manually".

   Ensure that the "docutils" package is in ``sys.path`` by
   one of the following actions:

   * Set the ``PYTHONPATH`` environment variable so that Python
     picks up your local working copy of the code.

     For the bash shell, add this to your ``~/.profile``::

         PYTHONPATH=$HOME/projects/docutils/docutils
         export PYTHONPATH

     The first line points to the directory containing the ``docutils``
     package.  The second line exports the environment variable.

   * Create a symlink to the docutils package directory somewhere in the
     module search path (``sys.path``), e.g., ::

         ln -s ~/projects/docutils/docutils \
               /usr/local/lib/python3.9/dist-packages/

   * Use a `path configuration file`__.

     __ https://docs.python.org/library/site.html

   Optionally, add some or all `front-end tools`_
   to the binary search path, e.g.,
   add the ``tools`` directory to the ``PATH`` variable::

         PATH=$PATH:$HOME/projects/docutils/docutils/tools
         export PATH

   or link idividual front-end tools to a suitable place
   in the binary path::

         ln -s ~/projects/docutils/docutils/tools/docutils-cli.py \
               /usr/local/bin/docutils

5. Reinstall Docutils after any change::

       python3 setup.py install

   .. CAUTION::

      This method is **not** recommended for day-to-day development;
      it's too easy to forget.  Confusion inevitably ensues.

      If you install Docutils this way, Python will always pick up the
      last-installed copy of the code.  If you ever forget to
      reinstall the "docutils" package, Python won't see your latest
      changes.

A useful addition to the ``docutils`` top-level directory in branches
and alternate copies of the code is a ``set-PATHS`` file
containing the following lines::

    # source this file
    export PYTHONPATH=$PWD:$PWD
    export PATH=$PWD/tools:$PATH

Open a shell for this branch, ``cd`` to the ``docutils`` top-level
directory, and "source" this file.  For example, using the bash
shell::

    $ cd some-branch/docutils
    $ . set-PATHS

.. _pip: https://pypi.org/project/pip/
.. _setuptools: https://pypi.org/project/setuptools/
.. _front-end tools: ../user/tools.html


.. _developer access:

Information for Developers
==========================

If you would like to have write access to the repository, register
with SourceForge.net_ and send your SourceForge.net
user names to [email protected].
(Note that there may be a delay of several hours until you can commit
changes to the repository.)

Sourceforge SVN access is documented `here`__

__ https://sourceforge.net/p/forge/documentation/svn/


Ensure any changes comply with the `Docutils Project Policies`_
before `checking in`_,

.. _Docutils Project Policies: policies.html
.. _checking in: policies.html#check-ins


Setting Up Your Subversion Client For Development
-------------------------------------------------

Before committing changes to the repository, please ensure that the
following lines are contained (and uncommented) in your local
~/.subversion/config file, so that new files are added with the
correct properties set::

    [miscellany]
    # For your convenience:
    global-ignores = ... *.pyc ...
    # For correct properties:
    enable-auto-props = yes

    [auto-props]
    *.py = svn:eol-style=native;svn:keywords=Author Date Id Revision
    *.txt = svn:eol-style=native;svn:keywords=Author Date Id Revision
    *.html = svn:eol-style=native;svn:keywords=Author Date Id Revision
    *.xml = svn:eol-style=native;svn:keywords=Author Date Id Revision
    *.tex = svn:eol-style=native;svn:keywords=Author Date Id Revision
    *.css = svn:eol-style=native;svn:keywords=Author Date Id Revision
    *.patch = svn:eol-style=native
    *.sh = svn:eol-style=native;svn:executable;svn:keywords=Author Date Id Revision
    *.png = svn:mime-type=image/png
    *.jpg = svn:mime-type=image/jpeg
    *.gif = svn:mime-type=image/gif


Repository Layout
=================

The following tree shows the repository layout::

    docutils/
    |-- branches/
    |   |-- branch1/
    |   |   |-- docutils/
    |   |   |-- sandbox/
    |   |   `-- web/
    |   `-- branch2/
    |       |-- docutils/
    |       |-- sandbox/
    |       `-- web/
    |-- tags/
    |   |-- tag1/
    |   |   |-- docutils/
    |   |   |-- sandbox/
    |   |   `-- web/
    |   `-- tag2/
    |       |-- docutils/
    |       |-- sandbox/
    |       `-- web/
    `-- trunk/
        |-- docutils/
        |-- sandbox/
        `-- web/

The main source tree lives at ``docutils/trunk/docutils/``, next to
the sandboxes (``docutils/trunk/sandbox/``) and the web site files
(``docutils/trunk/web/``).

``docutils/branches/`` and ``docutils/tags/`` contain (shallow) copies
of either the whole trunk or only the main source tree
(``docutils/trunk/docutils``).

---

### Benchmarks Benchmarking Cases Docutils Data Docs Dev Runtime Settings Processing (`benchmarks/benchmarking/cases/docutils_data/docs/dev/runtime-settings-processing.txt`)

=============================
 Runtime Settings Processing
=============================

:Author: David Goodger, Günter Milde
:Contact: [email protected]
:Date: $Date$
:Revision: $Revision$
:Copyright: This document has been placed in the public domain.

:Abstract: A detailled description of Docutil's settings processing
           framework.

.. contents::


The ``docutils/__init__.py``, ``docutils/core.py``, and
``docutils.frontend.py`` modules are described.
Following along with the actual code is recommended.

See `Docutils Runtime Settings`_ for a high-level description of the core
API and `Docutils Configuration`_ for a description of the individual
settings.

.. note::
   This document is informal.
   It describes the state in Docutils 0.18.1.
   Implementation details will change with the move to replace the
   deprecated optparse_ module with argparse_.


Settings priority
=================

Docutils overlays default and explicitly specified values from various
sources such that settings behave the way we want and expect them to
behave.

The souces are overlaid in the following order (later sources
overwrite earlier ones):

1. Defaults specified in `settings_spec`__ and
   `settings_defaults`__ attributes for each component_.
   (details__)

   __ ../api/runtime-settings.html#settingsspec-settings-spec
   __ ../api/runtime-settings.html#settingsspec-settings-defaults
   __ `OptionParser.populate_from_components()`_

2. Defaults specified in `settings_default_overrides`__ attribute for
   each component_.
   (details__)

   __ ../api/runtime-settings.html#settingsspec-settings-default-overrides
   __ component.settings_default_overrides_

3. Settings specified in the `settings_overrides parameter`_ of the
   `convenience functions`_ rsp. the `settings_overrides` attribute of
   a `core.Publisher` instance.
   (details__)

   __ OptionParser.defaults_

4. If enabled, settings specified in `active sections`_ of the
   `configuration files`_ in the order described in
   `Configuration File Sections & Entries`_. (details__)

   See also `Configuration File Sections & Entries`_.

   __ `OptionParser.get_standard_config_settings()`_

5. If enabled, command line arguments (details__).

   __ `Publisher.process_command_line()`_


Settings assigned to the `settings parameter`_ of the
`convenience functions`_ or the ``Publisher.settings`` attribute
are used **instead of** the above sources
(see below for details for `command-line tools`__ and
`other applications`__).

__ `publisher.publish()`_
__ `publisher.process_programmatic_settings()`_

.. _command-line tools:

Runtime settings processing for command-line tools
==================================================

The command-line `front-end tools`_ usually import and call
the `convenience function`_ ``docutils.core.publish_cmdline()``.

1. ``docutils.core.publish_cmdline()`` creates a `Publisher`_ instance::

       publisher = core.Publisher(…, settings)

   eventually sets the components_ from the respective names, and calls ::

       publisher.publish(argv, …, settings_spec,
                         settings_overrides, config_section, …)

   .. _publisher.publish():

2. If `publisher.settings` is None, ``publisher.publish()`` calls::

       publisher.process_command_line(…,
           settings_spec, config_section, **defaults)

   with `defaults` taken from `publisher.settings_overrides`.

   If `publisher.settings` is defined, steps 3 to 5 are skipped.

3. ``publisher.process_command_line()`` calls::

       optpar = publisher.setup_option_parser(…,
                    settings_spec, config_section, **defaults)

   .. _publisher.setup_option_parser():

4. ``publisher.setup_option_parser()``

   - merges the value of the `config_section parameter`_ into
     `settings_spec` and

   - creates an `OptionParser` instance ::

        optpar = docutils.frontend.OptionParser(components, defaults)

     with `components` the tuple of the `SettingsSpec`_ instances
     ``(publisher.parser, publisher.reader, publisher.writer, settings_spec)``

   .. _OptionParser.populate_from_components():

5. The `OptionParser` instance prepends itself to the `components` tuple
   and calls ``self.populate_from_components(components)``, which updates
   the attribute ``self.defaults`` in two steps:

   a) For each component passed, ``component.settings_spec`` is processed
      and ``component.settings_defaults`` is applied.

      .. _component.settings_default_overrides:

   b) In a second loop, for each component
      ``component.settings_default_overrides`` is applied. This way,
      ``component.settings_default_overrides`` can override the default
      settings of any other component.

   .. _OptionParser.defaults:

6. Back in ``OptionParser.__init__()``, ``self.defaults`` is updated with
   the `defaults` argument passed to ``OptionParser(…)`` in step 5.

   This means that the `settings_overrides` argument of the
   `convenience functions`_ has priority over all
   ``SettingsSpec.settings_spec`` defaults.

   .. _OptionParser.get_standard_config_settings():

7. If configuration files are enabled,
   ``self.get_standard_config_settings()`` is called.

   This reads the Docutils `configuration files`_, and returns a
   dictionary of settings in `active sections`_ which is used to update
   ``optpar.defaults``. So configuration file settings have priority over
   all software-defined defaults.

   .. _Publisher.process_command_line():

8. ``publisher.process_command_line()`` calls ``optpar.parse_args()``.
   The OptionParser parses all command line options and returns a
   `docutils.frontend.Values` object.
   This is assigned to ``publisher.settings``.
   So command-line options have priority over configuration file
   settings.

9. The `` and `<destination>` command-line arguments
   are also parsed, and assigned to ``publisher.settings._source``
   and ``publisher.settings._destination`` respectively.

10. ``publisher.publish()`` calls ``publisher.set_io()`` with no arguments.
    If either ``publisher.source`` or ``publisher.destination`` are not
    set, the corresponding ``publisher.set_source()`` and
    ``publisher.set_destination()`` are called:

    ``publisher.set_source()``
      checks for a ``source_path`` argument, and if there is none (which
      is the case for command-line use), it is taken from
      ``publisher.settings._source``.  ``publisher.source`` is set by
      instantiating a ``publisher.source_class`` object.
      For command-line front-end tools, the default
      ``publisher.source_class`` (i.e. ``docutils.io.FileInput``)
      is used.

    ``publisher.set_destination()``
      does the same job for the destination. (the default
      ``publisher.destination_class`` is ``docutils.io.FileOutput``).

    .. _accessing the runtime settings:

11. ``publisher.publish()`` passes ``publisher.settings`` to the reader_
    component's ``read()`` method.

12. The reader component creates a new `document root node`__.
    ``nodes.document.__init__()`` adds the settings to the internal
    attributes.

    __ ../ref/doctree.html#document

    All components acting on the Document Tree are handed the
    ``document`` root node and can access the runtime settings as
    ``document.settings``.


Runtime settings processing for other applications
==================================================

The `convenience functions`_ , ``core.publish_file()``,
``core.publish_string()``, or ``core.publish_parts()`` do not parse the
command line for settings.

1. The convenience functions call the generic programmatic interface
   function ``core.publish_programmatically()`` that creates a
   `core.Publisher` instance ::

       pub = core.Publisher(…, settings)

   eventually sets the components_ from the respective names, and calls ::

       publisher.process_programmatic_settings(
           settings_spec, settings_overrides, config_section)

   .. _publisher.process_programmatic_settings():

2. If `publisher.settings` is None,
   ``publisher.process_programmatic_settings()`` calls::

       publisher.get_settings(settings_spec, config_section, **defaults)

   with `defaults` taken from `publisher.settings_overrides`.

   If `publisher.settings` is defined, the following steps are skipped.

3. ``publisher.get_settings()`` calls::

       optpar = publisher.setup_option_parser(…,
                    settings_spec, config_section, **defaults)

4. The OptionParser instance determines setting defaults
   as described in `steps 4 to 7`__ in the previous section.

   __ `publisher.setup_option_parser()`_

5. Back in ``publisher.get_settings()``, the ``frontend.Values`` instance
   returned by ``optpar.get_default_values()`` is stored in
   ``publisher.settings``.

6. ``publish_programmatically()`` continues with setting
   ``publisher.source`` and ``publisher.destination``.

7. Finally, ``publisher.publish()`` is called. As ``publisher.settings``
   is not None, no further command line processing takes place.

8. All components acting on the Document Tree are handed the
   ``document`` root node and can access the runtime settings as
   ``document.settings`` (cf. `steps 11 and 12`__ in the previous section).

   __ `accessing the runtime settings`_


.. References:

.. _optparse: https://docs.python.org/dev/library/optparse.html
.. _argparse: https://docs.python.org/dev/library/argparse.html

.. _Docutils Runtime Settings:
   ../api/runtime-settings.html
.. _active sections:
   ../api/runtime-settings.html#active-sections
.. _SettingsSpec:
   ../api/runtime-settings.html#settingsspec
.. _component:
.. _components:
    ../api/runtime-settings.html#components
.. _application settings specifications:
.. _convenience function:
.. _convenience functions:
    ../api/runtime-settings.html#convenience-functions
.. _settings_overrides parameter:
    ../api/runtime-settings.html#settings-overrides-parameter
.. _settings parameter:
   ../api/runtime-settings.html#settings-parameter
.. _config_section parameter:
   ../api/runtime-settings.html#config-section-parameter

.. _Publisher convenience functions:
    ../api/publisher.html#publisher-convenience-functions
.. _front-end tools: ../user/tools.html
.. _configuration file:
.. _configuration files:
.. _Docutils Configuration: ../user/config.html#configuration-files
.. _Configuration File Sections & Entries:
    ../user/config.html#configuration-file-sections-entries
.. _Docutils Project Model: ../peps/pep-0258.html#docutils-project-model
.. _Publisher: ../peps/pep-0258.html#publisher
.. _Reader: ../peps/pep-0258.html#reader

---

### Benchmarks Benchmarking Cases Docutils Data Docs Dev Semantics (`benchmarks/benchmarking/cases/docutils_data/docs/dev/semantics.txt`)

=====================
 Docstring Semantics
=====================
:Author: David Goodger
:Contact: [email protected]
:Revision: $Revision$
:Date: $Date$
:Copyright: This document has been placed in the public domain.

These are notes for a possible future PEP providing the final piece of
the Python docstring puzzle: docstring semantics or documentation
methodology.  `PEP 257`_, Docstring Conventions, sketches out some
guidelines, but does not get into methodology details.

I haven't explored documentation methodology more because, in my
opinion, it is a completely separate issue from syntax, and it's even
more controversial than syntax.  Nobody wants to be told how to lay
out their documentation, a la JavaDoc_.  I think the JavaDoc way is
butt-ugly, but it *is* an established standard for the Java world.
Any standard documentation methodology has to be formal enough to be
useful but remain light enough to be usable.  If the methodology is
too strict, too heavy, or too ugly, many/most will not want to use it.

I think a standard methodology could benefit the Python community, but
it would be a hard sell.  A PEP would be the place to start.  For most
human-readable documentation needs, the free-form text approach is
adequate.  We'd only need a formal methodology if we want to extract
the parameters into a data dictionary, index, or summary of some kind.


PythonDoc
=========

(Not to be confused with Daniel Larsson's pythondoc_ project.)

A Python version of the JavaDoc_ semantics (not syntax).  A set of
conventions which are understood by the Docutils.  What JavaDoc has
done is to establish a syntax that enables a certain documentation
methodology, or standard *semantics*.  JavaDoc is not just syntax; it
prescribes a methodology.

- Use field lists or definition lists for "tagged blocks".  By this I
  mean that field lists can be used similarly to JavaDoc's ``@tag``
  syntax.  That's actually one of the motivators behind field lists.
  For example, we could have::

      """
      :Parameters:
          - `lines`: a list of one-line strings without newlines.
          - `until_blank`: Stop collecting at the first blank line if
            true (1).
          - `strip_indent`: Strip common leading indent if true (1,
            default).

      :Return:
          - a list of indented lines with minimum indent removed;
          - the amount of the indent;
          - whether or not the block finished with a blank line or at
            the end of `lines`.
      """

  This is taken straight out of docutils/statemachine.py, in which I
  experimented with a simple documentation methodology.  Another
  variation I've thought of exploits the Grouch_-compatible
  "classifier" element of definition lists.  For example::

      :Parameters:
          `lines` : [string]
              List of one-line strings without newlines.
          `until_blank` : boolean
              Stop collecting at the first blank line if true (1).
          `strip_indent` : boolean
              Strip common leading indent if true (1, default).

- Field lists could even be used in a one-to-one correspondence with
  JavaDoc ``@tags``, although I doubt if I'd recommend it.  Several
  ports of JavaDoc's ``@tag`` methodology exist in Python, most
  recently Ed Loper's "epydoc_".


Other Ideas
===========

- Can we extract comments from parsed modules?  Could be handy for
  documenting function/method parameters::

      def method(self,
                 source,        # path of input file
                 dest           # path of output file
                ):

  This would save having to repeat parameter names in the docstring.

  Idea from Mark Hammond's 1998-06-23 Doc-SIG post, "Re: [Doc-SIG]
  Documentation tool":

      it would be quite hard to add a new param to this method without
      realising you should document it

- Frederic Giacometti's `iPhrase Python documentation conventions`_ is
  an attachment to his Doc-SIG post of 2001-05-30.


.. _PEP 257: https://peps.python.org/pep-0257
.. _JavaDoc: http://java.sun.com/j2se/javadoc/
.. _pythondoc: http://starship.python.net/crew/danilo/pythondoc/
.. _Grouch: http://www.mems-exchange.org/software/grouch/
.. _epydoc: http://epydoc.sourceforge.net/
.. _iPhrase Python documentation conventions:
   https://mail.python.org/pipermail/doc-sig/2001-May/001840.html


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End:

---

### Benchmarks Benchmarking Cases Docutils Data Docs Dev Testing (`benchmarks/benchmarking/cases/docutils_data/docs/dev/testing.txt`)

===================
 Docutils_ Testing
===================

:Authors: Lea Wiemann <[email protected]>;
          David Goodger <[email protected]>;
          Docutils developers <[email protected]>
:Revision: $Revision$
:Date: $Date$
:Copyright: This document has been placed in the public domain.

.. _Docutils: https://docutils.sourceforge.io/

.. contents::

When adding new functionality (or fixing bugs), be sure to add test
cases to the test suite.  Practise test-first programming; it's fun,
it's addictive, and it works!

This document describes how to run the Docutils test suite, how the
tests are organized and how to add new tests or modify existing tests.


Running the Test Suite
======================

Before checking in any changes, run the entire Docutils test suite to
be sure that you haven't broken anything.  From a shell do [#]_::

    cd docutils/test
    python -u alltests.py

Before `checking in`__ changes to the Docutils core, run the tests on
all `supported Python versions`_ (see below for details).
In a pinch, the edge cases should cover most of it.

__ policies.html#check-ins

.. note::
   Due to incompatible customization of the standard unittest_
   framework, the test suite does not work with popular test frameworks
   like pytest_ or nose_.

   .. _unittest: https://docs.python.org/3/library/unittest.html
   .. _pytest: https://pypi.org/project/pytest/
   .. _nose: https://pypi.org/project/nose3/

   .. cf. https://sourceforge.net/p/docutils/feature-requests/81/

.. [#] When using the `Python launcher for Windows`__, make sure to
   specify a Python version, e.g., ``py -3.9 -u alltests.py`` for
   Python 3.9.

   __ https://docs.python.org/3/using/windows.html#python-launcher-for-windows

   .. cf. https://sourceforge.net/p/docutils/bugs/434/


.. _Python versions:

Testing across multiple Python versions
---------------------------------------

A Docutils release has a commitment to support a minimum Python version [#]_
and beyond. Before a release is cut, tests must pass in all
`supported versions`_.

You can use `tox`_ to test with all supported versions in one go.
From the shell::

    cd docutils
    tox

To test a specific version, use the ``pyNN`` environment. For example::

    tox -e py37

`pyenv`_ can be installed and configured (see `installing pyenv`_) to
get multiple Python versions::

    # assuming your system runs 3.9.x
    pyenv install 3.7.12
    pyenv install 3.8.12
    pyenv install 3.10.1
    pyenv global system 3.7.12 3.8.12 3.10.1

    # reset your shims
    rm -rf ~/.pyenv/shims && pyenv rehash

This will give you ``python3.7`` through ``python3.10``.
Then run::

    python3.7 -u alltests.py
    [...]
    python3.10 -u alltests.py

.. [#] Good resources covering the differences between Python versions
   are the `What's New` documents (`What's New in Python 3.10`__ and
   similar).

__ https://docs.python.org/3/whatsnew/3.10.html


.. _supported versions:
.. _supported Python versions: ../../README.html#requirements
.. _pyenv: https://github.com/yyuu/pyenv
.. _installing pyenv: https://github.com/yyuu/pyenv#installation
.. _tox: https://pypi.org/project/tox/


Unit Tests
==========

Unit tests test single functions or modules (i.e. whitebox testing).

If you are implementing a new feature, be sure to write a test case
covering its functionality.  It happens very frequently that your
implementation (or even only a part of it) doesn't work with an older
(or even newer) Python version, and the only reliable way to detect
those cases is using tests.

Often, it's easier to write the test first and then implement the
functionality required to make the test pass.


Writing New Tests
-----------------

When writing new tests, it very often helps to see how a similar test
is implemented.  For example, the files in the
``test_parsers/test_rst/`` directory all look very similar.  So when
adding a test, you don't have to reinvent the wheel.

If there is no similar test, you can write a new test from scratch
using Python's ``unittest`` module.  For an example, please have a
look at the following imaginary ``test_square.py``::

    #! /usr/bin/env python

    # $Id$
    # Author: Your Name <[email protected]>
    # Copyright: This module has been placed in the public domain.

    """
    Test module for docutils.square.
    """

    import unittest
    import docutils.square


    class SquareTest(unittest.TestCase):

        def test_square(self):
            self.assertEqual(docutils.square.square(0), 0)
            self.assertEqual(docutils.square.square(5), 25)
            self.assertEqual(docutils.square.square(7), 49)

        def test_square_root(self):
            self.assertEqual(docutils.square.sqrt(49), 7)
            self.assertEqual(docutils.square.sqrt(0), 0)
            self.assertRaises(docutils.square.SquareRootError,
                              docutils.square.sqrt, 20)


    if __name__ == '__main__':
        unittest.main()

For more details on how to write tests, please refer to the
documentation of the ``unittest`` module.

.. Note::

   Unit tests and functional test should generally set ::

     settings_overrides['_disable_config'] = True

   in order to be independent on the users local configuration.

.. _functional:

Functional Tests
================

The directory ``test/functional/`` contains data for functional tests.

Performing functional testing means testing the Docutils system as a
whole (i.e. blackbox testing).


Directory Structure
-------------------

+ ``functional/`` The main data directory.

  + ``input/`` The input files.

    - ``some_test.txt``, for example.

  + ``output/`` The actual output.

    - ``some_test.html``, for example.

  + ``expected/`` The expected output.

    - ``some_test.html``, for example.

  + ``tests/`` The config files for processing the input files.

    - ``some_test.py``, for example.

    - ``_default.py``, the `default configuration file`_.


The Testing Process
-------------------

When running ``test_functional.py``, all config files in
``functional/tests/`` are processed.  (Config files whose names begin
with an underscore are ignored.)  The current working directory is
always Docutils' main test directory (``test/``).

For example, ``functional/tests/some_test.py`` could read like this::

    # Source and destination file names.
    test_source = "some_test.txt"
    test_destination = "some_test.html"

    # Keyword parameters passed to publish_file.
    reader_name = "standalone"
    parser_name = "rst"
    writer_name = "html"
    settings_overrides['output-encoding'] = 'utf-8'
    # Relative to main ``test/`` directory.
    settings_overrides['stylesheet_path'] = '../docutils/writers/html4css1/html4css1.css'

The two variables ``test_source`` and ``test_destination`` contain the
input file name (relative to ``functional/input/``) and the output
file name (relative to ``functional/output/`` and
``functional/expected/``).  Note that the file names can be chosen
arbitrarily.  However, the file names in ``functional/output/`` *must*
match the file names in ``functional/expected/``.

If defined, ``_test_more`` must be a function with the following
signature::

    def _test_more(expected_dir, output_dir, test_case, parameters):

This function is called from the test case to perform tests beyond the
simple comparison of expected and actual output files.

``test_source`` and ``test_destination`` are removed from the
namespace, as are all variables whose names begin with an underscore
("_").  The remaining names are passed as keyword arguments to
``docutils.core.publish_file``, so you can set reader, parser, writer
and anything else you want to configure.  Note that
``settings_overrides`` is already initialized as a dictionary *before*
the execution of the config file.


Creating New Tests
------------------

In order to create a new test, put the input test file into
``functional/input/``.  Then create a config file in
``functional/tests/`` which sets at least input and output file names,
reader, parser and writer.

Now run ``test_functional.py``.  The test will fail, of course,
because you do not have an expected output yet.  However, an output
file will have been generated in ``functional/output/``.  Check this
output file for validity [#]_ and correctness.  Then copy the file to
``functional/expected/``.

If you rerun ``test_functional.py`` now, it should pass.

If you run ``test_functional.py`` later and the actual output doesn't
match the expected output anymore, the test will fail.

If this is the case and you made an intentional change, check the
actual output for validity and correctness, copy it to
``functional/expected/`` (overwriting the old expected output), and
commit the change.

.. [#] The validity of `Docutils XML` can be tested with
   ``xmllint <document-referencing-local-Docutils-DTD>.xml --valid --noout``.

   .. note: the ``--dtdvalid`` and ``--nonet`` options did not help override
     a reference to the PUBLIC "docutils.dtd" if there is a local version
     on the system (e.g. /usr/share/xml/docutils/docutils.dtd in Debian).


.. _default configuration file:

The Default Configuration File
------------------------------

The file ``functional/tests/_default.py`` contains default settings.
It is executed just before the actual configuration files, which has
the same effect as if the contents of ``_default.py`` were prepended
to every configuration file.

---

### Benchmarks Benchmarking Cases Docutils Data Docs Dev Todo (`benchmarks/benchmarking/cases/docutils_data/docs/dev/todo.txt`)

======================
 Docutils_ To Do List
======================

:Author: David Goodger (with input from many); open to all Docutils
         developers
:Contact: [email protected]
:Date: $Date$
:Revision: $Revision$
:Copyright: This document has been placed in the public domain.

.. _Docutils: https://docutils.sourceforge.io/

.. contents::


Priority items are marked with "@" symbols.  The more @s, the higher
the priority.  Items in question form (containing "?") are ideas which
require more thought and debate; they are potential to-do's.

Many of these items are awaiting champions.  If you see something
you'd like to tackle, please do!
Please see also the Bugs_ document for a list of bugs in Docutils.

.. _bugs: ../../BUGS.html


Minimum Requirements for Python Standard Library Candidacy
==========================================================

Below are action items that must be added and issues that must be
addressed before Docutils can be considered suitable to be proposed
for inclusion in the Python standard library.

Many of these are now handled by Sphinx_

* Support for `document splitting`_.  May require some major code
  rework.

* Support for subdocuments (see `large documents`_).

* `Object numbering and object references`_.

* `Nested inline markup`_.

* `Python Source Reader`_.

* The HTML writer needs to be rewritten (or a second HTML writer
  added) to allow for custom classes, and for arbitrary splitting
  (stack-based?).

* Documentation_ of the architecture.  Other docs too.

* Plugin support.

* Suitability for `Python module documentation
  <https://docutils.sourceforge.io/sandbox/README.html#documenting-python>`_.

.. _Sphinx: http://www.sphinx-doc.org/

Repository
==========

Move to a Git repository.

* This is a long standing `feature request`__
  (with pointers to Sphinx issues and discussion).

  __ https://sourceforge.net/p/docutils/feature-requests/58/

* From a `post by David Goodger`__

    An absolute requirement, for me, is that such a change be complete.
    We can't lose any data or have to refer to the old system as an
    "archive". So all the SVN history, all branches, and the full sandbox
    need to be converted at the same time.

  __ https://sourceforge.net/p/docutils/mailman/message/31878077/

Convert with reposurgeon_?

  If you are doing a full import rather than gatewaying, reposurgeon is
  probably what you want. It has been tested against a lot of large, old,
  nasty repositories and is thus known to be robust in the presence of
  repository malformations (a property regularly checked by a test suite
  that is a rogue's gallery of Subversion botches).

  -- `Git Wiki`__

The comprehensive `Reposurgeon documentation`_ comes with
`a guide to repository conversion`__
as well as info about `reading Subversion repositories`__.
Converting from an SVN dump file is faster than from a checkout.

.. _reposurgeon: http://www.catb.org/esr/reposurgeon/
.. _reposurgeon documentation:
    http://www.catb.org/esr/reposurgeon/repository-editing.html
__ https://git.wiki.kernel.org/index.php/
   Interfaces,_frontends,_and_tools#Subversion
__ http://www.catb.org/esr/reposurgeon/repository-editing.html#conversion   
__ http://www.catb.org/esr/reposurgeon/repository-editing.html
   #_reading_subversion_repositories   


Adam Turner wrote a conversion Makefile and ``.lift`` scripts that
downloads the repo from SF with rsync, converts it to a SVN mirror and
finally to Git, splitting sandbox, prest, and web from docutils.


Sourceforge supports multiple Git repositories per project, so we can
switch the version control system independent of the decision on an
eventual switch of the host.
Cf. https://sourceforge.net/p/forge/documentation/Git/


General
=======

Miscellaneous
-------------

Code cleanup and modernization:
  Use flake8_? See also the configuration in `<../../tox.ini>`__.

  Check and solve issue from  :PEP:`290` - Code Migration and Modernization.
  (Covers issues up to Python 2.4, is there an equivalent for more recent
  modernizations?)

  Ensure `backwards compatibility`_!

  .. _flake8: https://pypi.org/project/flake8/
  .. _backwards compatibility: policies.html#backwards-compatibility-policy

* Encoding of command line arguments can only be guessed:

  * try UTF-8/strict first, then try the locale's encoding with
    strict error handling, then ASCII/replace?

    UTF-8 is almost 100% safe to try first; false positives are rare,
    The locale's encoding with strict error handling may be a
    reasonable compromise, but any error would indicate that the
    locale's encoding is inappropriate.  The only safe fallback is
    ASCII/replace.

  * Do not decode argv before option parsing but individual string
    values?

    +1  Allows for separate command-line vs. filesystem encodings,
        respectively to keep file names encoded.
    +1  Allows to configure command-line encoding in a config file,
    -1  More complicated.

  Cf. <http://thread.gmane.org/gmane.text.docutils.user/2890/focus=2957>.

* Improve handling on Windows:

  - Get graphical installer.
  - Make rst2html.py an .exe file using py2exe.

* .. _GUI:

  The user interface is very difficult to use for most Windows users;
  you can't really expect them to use the command line.  We need some
  kind of GUI that can launch rst2html.py, and save the HTML output to
  a file, and launch a browser.  What's important is that we get
  settings to work with the GUI.  So we need some way to dynamically
  generate a list of settings for the GUI.  The current settings_spec
  for OptionParser doesn't seem to be usable for this for the
  following reasons:

  - It's biased toward the command line -- there are *two* options for
    one boolean setting.

  - You cannot have both a one-line description and a longer
    description for tooltips/help-texts.

  - It doesn't provide hints for the input type.  You cannot easily
    infer the type of a setting from its validator, because any
    component can add new validators.  In fact, it may be necessary to
    have both a hint about the input type (e.g. string) and a
    validator (valid ID), or it may be necessary to have a different
    set of choices for the CLI (1, INFO, 2, ...) and for the GUI
    (INFO, WARNING, ...).

  - It's coupled to the OptionParser.  We want to be able to change
    the underlying system without breaking everything.

  - It's a bunch of primitive structures.  We want an extensible (thus
    object-oriented) interface.

  So we probably need to create a class for storing all the settings,
  and auto-generate the OptionParser data from that.

  I talked to Stephan Deibel about getting Docutils integrated into
  Wing IDE.  He said it's possible, and he'd be willing to help.
  There's a scripting interface to Wing, which we'd use.  We can
  dynamically generate a list of preferences and not worry too much
  about the rendering (from what I understood); Wing's whole GUI is
  dynamic anyway.  The interface could be made usable for other GUIs.
  For example, we could try to get option support for DocFactory.  //
  FW

* Allow different report levels for STDERR and system_messages inside
  the document?

* Change the docutils-update script (in sandbox/infrastructure), to
  support arbitrary branch snapshots.

* Move some general-interest sandboxes out of individuals'
  directories, into subprojects?

* Add option for file (and URL) access restriction to make Docutils
  usable in Wikis and similar applications.

  2005-03-21: added ``file_insertion_enabled`` & ``raw_enabled``
  settings.  These partially solve the problem, allowing or disabling
  **all** file accesses, but not limited access.

* Configuration_ file handling needs discussion:

  - There should be some error checking on the contents of config
    files.  How much checking should be done?  How loudly should
    Docutils complain if it encounters an error/problem?

  - Docutils doesn't complain when it doesn't find a configuration
    file supplied with the ``--config`` option.  Should it?  (If yes,
    error or warning?)

* Internationalization:

  - I18n needs refactoring, the language dictionaries are difficult to
    maintain.  Maybe have a look at gettext or similar tools.

    (This would make a nice Google Summer of Code project)

  - Language modules: in accented languages it may be useful to have
    both accented and unaccented entries in the
    ``bibliographic_fields`` mapping for versatility.

  - Add a "--strict-language" option & setting: no English fallback
    for language-dependent features.

    Make this the default for output (as opposed to input)?
    Throw an error with a helpful message, e.g.

      Default "contents" title for language %s missing, please specify
      an explicit title.

    or

     "attention" title for language %s missing, please use a generic
     admonition with explicit title.

  - Add internationalization to _`footer boilerplate text` (resulting
    from "--generator", "--source-link", and "--date" etc.), allowing
    translations.


* Add validation?  See http://pytrex.sourceforge.net, RELAX NG, pyRXP.

* In ``docutils.readers.get_reader_class`` (& ``parsers`` &
  ``writers`` too), should we be importing "standalone" or
  "docutils.readers.standalone"?  (This would avoid importing
  top-level modules if the module name is not in docutils/readers.
  Potential nastiness.)

* Perhaps store a _`name-to-id mapping file`?  This could be stored
  permanently, read by subsequent processing runs, and updated with
  new entries.  ("Persistent ID mapping"?)

* Perhaps the ``Component.supports`` method should deal with
  individual features ("meta" etc.) instead of formats ("html" etc.)?
  Currently, it is not used at all.

  Do we need it at all?  Or rather let the writers just ignore some
  nodes (like we already do for "class" values)?

  The current implementation of the framework also leads to bug
  `bug #241`__ "doctree-based publishing != publish_string".
  The "components.Filter" transform is run by publish_doctree(). When
  filtering based on the output format, it should be run by
  publish_from_doctree() instead because only then the writer is
  known.

  So we need to either remove or fix the framework.

  __ https://sourceforge.net/p/docutils/bugs/241/


* Think about _`large documents` made up of multiple subdocument
  files.  Issues: continuity (`persistent sequences`_ above),
  cross-references (`name-to-id mapping file`_ above and `targets in
  other documents`_ below), splitting (`document splitting`_ below).

  When writing a book, the author probably wants to split it up into
  files, perhaps one per chapter (but perhaps even more detailed).
  However, we'd like to be able to have references from one chapter to
  another, and have continuous numbering (pages and chapters, as
  applicable).  Of course, none of this is implemented yet.  There has
  been some thought put into some aspects; see `the "include"
  directive`__ and the `Reference Merging`_ transform below.

  When I was working with SGML in Japan, we had a system where there
  was a top-level coordinating file, book.sgml, which contained the
  top-level structure of a book: the  element, containing the
  book <title> and empty component elements (, <chapter>,
  <appendix>, etc.), each with filename attributes pointing to the
  actual source for the component.  Something like this::

      
      <title>Title of the Book</title>
      
      <chapter inrefid="ch01"></chapter>
      <chapter inrefid="ch02"></chapter>
      <chapter inrefid="ch03"></chapter>
      <appendix inrefid="ap01"></appendix>
      

  (The "inrefid" attribute stood for "insertion reference ID".)

  The processing system would process each component separately, but
  it would recognize and use the book file to coordinate chapter and
  page numbering, and keep a persistent ID to (title, page number)
  mapping database for cross-references.  Docutils could use a similar
  system for large-scale, multipart documents.

  __ ../ref/rst/directives.html#including-an-external-document-fragment

  Aahz's idea:

      First the ToC::

          .. ToC-list::
              Introduction.txt
              Objects.txt
              Data.txt
              Control.txt

      Then a sample use::

          .. include:: ToC.txt

          As I said earlier in chapter :chapter:`Objects.txt`, the
          reference count gets increased every time a binding is made.

      Which produces::

          As I said earlier in chapter 2, the
          reference count gets increased every time a binding is made.

      The ToC in this form doesn't even need to be references to actual
      reST documents; I'm simply doing it that way for a minimum of
      future-proofing, in case I do want to add the ability to pick up
      references within external chapters.

  Perhaps, instead of ToC (which would overload the "contents"
  directive concept already in use), we could use "manifest".  A
  "manifest" directive might associate local reference names with
  files::

      .. manifest::
         intro: Introduction.txt
         objects: Objects.txt
         data: Data.txt
         control: Control.txt

  Then the sample becomes::

      .. include:: manifest.txt

      As I said earlier in chapter :chapter:`objects`, the
      reference count gets increased every time a binding is made.

* Add support for _`multiple output files` and _`generic data
  handling`:

  It should be possible for a component to **emit or reference** data
  to be either **included or referenced** in the output document.
  Examples of such data are stylesheets or images.

  For this, we need a "data" object which stores the data either
  inline or by referring to a file.  The Docutils framework is
  responsible for either:

  * storing the data in the appropriate location (e.g. in the
    directory of the output file, or in a user-specified directory)
    and providing the paths of the stored files to the writer, *or*

  * providing the data itself to the writer so that it can be embedded
    in the output document.

  This approach decouples data handling from the data source (which
  can either be embedded or referenced) and the destination (which can
  either be embedded or referenced as well).

  See <http://article.gmane.org/gmane.text.docutils.devel/3631>.

* Add testing for Docutils' front end tools?

* Publisher: "Ordinary setup" shouldn't require specific ordering; at
  the very least, there ought to be error checking higher up in the
  call chain.  [Aahz]

  ``Publisher.get_settings`` requires that all components be set up
  before it's called.  Perhaps the I/O *objects* shouldn't be set, but
  I/O *classes*.  Then options are set up (``.set_options``), and
  ``Publisher.set_io`` (or equivalent code) is called with source &
  destination paths, creating the I/O objects.

  Perhaps I/O objects shouldn't be instantiated until required.  For
  split output, the Writer may be called multiple times, once for each
  doctree, and each doctree should have a separate Output object (with
  a different path).  Is the "Builder" pattern applicable here?

* Perhaps I/O objects should become full-fledged components (i.e.
  subclasses of ``docutils.Component``, as are Readers, Parsers, and
  Writers now), and thus have associated option/setting specs and
  transforms.

* Multiple file I/O suggestion from Michael Hudson: use a file-like
  object or something you can iterate over to get file-like objects.

* Add an "--input-language" option & setting?  Specify a different
  language module for input (bibliographic fields, directives) than
  for output.  The "--language" option would set both input & output
  languages.

* Auto-generate reference tables for language-dependent features?
  Could be generated from the source modules.  A special command-line
  option could be added to Docutils front ends to do this.  (Idea from
  Engelbert Gruber.)

* Enable feedback of some kind from internal decisions, such as
  reporting the successful input encoding.  Modify runtime settings?
  System message?  Simple stderr output?

* Rationalize Writer settings (HTML/LaTeX/PEP) -- share settings.

* Add an "--include file" command-line option (config setting too?),
  equivalent to ".. include:: file" as the first line of the doc text?
  Especially useful for character entity sets, text transform specs,
  boilerplate, etc.

* Support "include" as embedded inline-compatible directive in substitution
  definitions, e.g. ::

    .. |version| include:: version.txt

    This document describes version |version| of ...

  (cf. Grzegorz Adam Hankiewicz's post from 2014-10-01 in docutils-devel)

* Add an ``:optional: <replacement text>`` option to the "include"
  directive? This would not throw an error for a missing file, instead a
  warning is given and ``<replacement text>`` is used instead. It would be
  the responsibility of the author to ensure the missing file does not lead
  to problems later in the document.

  Use cases:

  + Standard rST syntax to replace Sphinx's "literalinclude"::

      .. include:: blah.cpp
         :literal:
         :optional: file ``blah.cpp`` not found

  + Variable content taken from a file, e.g.

    version.txt::

       .. |version| replace:: 3.1

    optionally used as::

       .. include:: version.txt
          :optional: .. |version| replace:: unknown

       This document describes version |version| of ...

    (cf. Grzegorz Adam Hankiewicz's post from 2014-10-01 in docutils-devel)

* Parameterize the Reporter object or class?  See the `2004-02-18
  "rest checking and source path"`_ thread.

  .. _2004-02-18 "rest checking and source path":
     http://thread.gmane.org/gmane.text.docutils.user/1112

* Add a "disable_transforms" setting? Would allow for easy syntax
  checking. Where ("null" writer, generic, parser(s))?
  Cf. the `2004-02-18 "rest checking and source path"`_ thread.

* Add a generic meta-stylesheet mechanism?  An external file could
  associate style names ("class" attributes) with specific elements.
  Could be generalized to arbitrary output attributes; useful for HTML
  & XMLs.  Aahz implemented something like this in
  sandbox/aahz/Effective/EffMap.py.

* .. _classes for table cells:

  William Dode suggested that table cells be assigned "class"
  attributes by columns, so that stylesheets can affect text
  alignment.  Unfortunately, there doesn't seem to be a way (in HTML
  at least) to leverage the "colspec" elements (HTML "col" tags) by
  adding classes to them.  The resulting HTML is very verbose::

      111
      222
      ...

  At the very least, it should be an option.  People who don't use it
  shouldn't be penalized by increases in their HTML file sizes.

  Table rows could also be assigned classes (like odd/even).  That
  would be easier to implement.

  How should it be implemented?

  * There could be writer options (column classes & row classes) with
    standard values.

  * The table directive could grow some options.  Something like
    ":cell-classes: col1 col2 col3" (either must match the number of
    columns, or repeat to fill?)  and ":row-classes: odd even" (repeat
    to fill; body rows only, or header rows too?).

  Probably per-table directive options are best.  The "class" values
  could be used by any writer, and applying such classes to all tables
  in a document with writer options is too broad.

  See also the `table_styling Sphinx extension`_ which defines

  :widths: also in Docutils core (but different implementation)
  :column-alignment: Sets per-column text alignment
  :column-wrapping:  Sets per-column text wrapping
  :column-dividers:  Add dividers between columns
  :column-classes:   Add per-column css classes.
  :header-columns:   Specify number of “stub” columns

  .. _table_styling Sphinx extension: https://pythonhosted.org/cloud_sptheme/
                                      lib/cloud_sptheme.ext.table_styling.html

* Add file-specific settings support to config files, like::

      [file index.txt]
      compact-lists: no

  Is this even possible?  Should the criterion be the name of the
  input file or the output file?  Alternative (more explicit) syntax::

      [source_file index.txt]
      ...

      [dest_file index.html]
      ...

  Or rather allow settings configuration from the rst source file
  (see misc.settings_ directive)?

* The "validator" support added to OptionParser is very similar to
  "traits_" in SciPy_.  Perhaps something could be done with them?
  (Had I known about traits when I was implementing docutils.frontend,
  I may have used them instead of rolling my own.)

  .. _traits: http://code.enthought.com/traits/
  .. _SciPy: http://www.scipy.org/

* tools/buildhtml.py: Extend the --prune option ("prune" config
  setting) to accept file names (generic path) in addition to
  directories (e.g. --prune=docs/user/rst/cheatsheet.txt, which should
  *not* be converted to HTML).

* Add support for _`plugins`.

* _`Config directories`: Currently, ~/.docutils, ./docutils.conf/, &
  /etc/docutils.conf are read as configuration_ files.  Proposal: allow
  ~/.docutils to be a a configuration *directory*, along with
  /etc/docutils/ and ./docutils.conf/.  Within these directories,
  check for config.txt files.  We can also have subdirectories here,
  for plugins, S5 themes, components (readers/writers/parsers) etc.

  Docutils will continue to support configuration files for backwards
  compatibility.

* Add support for document decorations other than headers & footers?
  For example, top/bottom/side navigation bars for web pages.  Generic
  decorations?

  Seems like a bad idea as long as it isn't independent from the output
  format (for example, navigation bars are only useful for web pages).

* docutils_update: Check for a ``Makefile`` in a directory, and run
  ``make`` if found?  This would allow for variant processing on
  specific source files, such as running rst2s5.py instead of
  rst2html.py.

* Add a "disable table of contents" setting?  The S5 writer could set
  it as a default.  Rationale:

      The ``contents`` (table of contents) directive must not be used
      [in S5/HTML documents].  It changes the CSS class of headings
      and they won't show up correctly in the screen presentation.

      -- `Easy Slide Shows With reStructuredText & S5
      <../user/slide-shows.html>`_

  Analogue to the ``sectnum_xform`` setting, it could be used by the
  latex writer to switch to a LaTeX generated ToC (currently, the latex
  writer calls it "use_latex_toc").

object numbering and object references
--------------------------------------

For equations, tables & figures.

These would be the equivalent of DocBook's "formal" elements.

In LaTeX, automatic counters are implemented for sections, equations and
floats (figures, tables) (configurable via stylesheets or in the
latex-preamble). Objects can be given `reference names`_ with the
``\label{<refname}`` command, ``\ref{<refname>}`` inserts the
corresponding number.

No such mechanism exists in HTML.

* We need _`persistent sequences`, similar to chapter and footnote
  numbers. See `OpenOffice.org XML`_ "fields".

  - Should the sequences be automatic or manual (user-specifyable)?

* It is already possible to give `reference names`_ to objects via
  internal hyperlink targets or the "name" directive option::

      .. _figure name:

      .. figure:: image.png

  or ::

      .. figure:: image.png
         :name: figure name

  Improve the mapping of "phrase references" to IDs/labels with Literal
  transcription (i.e. ü -> ue, ß -> ss, å -> aa) instead of just
  stripping the accents and other non-ASCII chars. See also the feature
  request `allow more characters when transforming "names" to "ids"`__.

  A "table" directive has been implemented, supporting table titles.

  Perhaps the name could derive from the title/caption?

  .. _reference names: ../ref/rst/restructuredtext.html#reference-names
  __ https://sourceforge.net/p/docutils/feature-requests/66/

* We need syntax for object references.  Cf. `OpenOffice.org XML`_
  "reference fields":

  - Parameterized substitutions are too complicated
    (cf. `or not to do`: `object references`_)

  - An interpreted text approach is simpler and better::

      See Figure :ref:`figure name` and Equation :ref:`eq:identity`.

  - "equation", "figure", and "page" roles could generate appropriate
    boilerplate text::

        See :figure:`figure name` on :page:`figure name`.

    See `Interpreted Text`_ below.

    Reference boilerplate could be specified in the document
    (defaulting to nothing)::

        .. fignum::
           :prefix-ref: "Figure "
           :prefix-caption: "Fig. "
           :suffix-caption: :

    The position of the role (prefix or suffix) could also be utilized

  .. _OpenOffice.org XML: http://xml.openoffice.org/
  .. _object references: rst/alternatives.html#object-references

See also the `Modified rst2html
<http://www.loria.fr/~rougier/coding/article/rst2html.py>`__
by Nicolas Rougier for a sample implementation.


Documentation
=============

User Docs
---------

* Add a FAQ entry about using Docutils (with reStructuredText) on a
  server and that it's terribly slow.  See the first paragraphs in
  <http://article.gmane.org/gmane.text.docutils.user/1584>.

* Add document about what Docutils has previously been used for
  (web/use-cases.txt?).

* Improve index in docs/user/config.txt.


Developer Docs
--------------

* Improve the internal module documentation (docstrings in the code).
  Specific deficiencies listed below.

  - docutils.parsers.rst.states.State.build_table: data structure
    required (including StringList).

  - docutils.parsers.rst.states: more complete documentation of parser
    internals.

* docs/ref/doctree.txt: DTD element structural relationships,
  semantics, and attributes.  In progress; element descriptions to be
  completed.

* Document the ``pending`` elements, how they're generated and what
  they do.

* Document the transforms_ (perhaps in docstrings?): how they're used,
  what they do, dependencies & order considerations.

* Document the HTML classes used by html4css1.py.

* Write an overview of the Docutils architecture, as an introduction
  for developers.  What connects to what, why, and how.  Either update
  PEP 258 (see PEPs_ below) or as a separate doc.

* Give information about unit tests.  Maybe as a howto?

* Document the docutils.nodes APIs.

* Complete the docs/api/publisher.txt docs.


How-Tos
-------

* Creating Docutils Writers

* Creating Docutils Readers

* Creating Docutils Transforms_

* Creating Docutils Parsers

* Using Docutils as a Library


PEPs
----

* Complete PEP 258 Docutils Design Specification.

  - Fill in the blanks in API details.

  - Specify the nodes.py internal data structure implementation?

        [Tibs:] Eventually we need to have direct documentation in
        there on how it all hangs together - the DTD is not enough
        (indeed, is it still meant to be correct?  [Yes, it is.
        --DG]).

* Rework PEP 257, separating style from spec from tools, wrt Docutils?
  See Doc-SIG from 2001-06-19/20.


Python Source Reader
====================

General:

* Analyze Tony Ibbs' PySource code.

* Analyze Doug Hellmann's HappyDoc project.

* Investigate how POD handles literate programming.

* Take the best ideas and integrate them into Docutils.

Miscellaneous ideas:

* Ask Python-dev for opinions (GvR for a pronouncement) on special
  variables (__author__, __version__, etc.): convenience vs. namespace
  pollution.  Ask opinions on whether or not Docutils should recognize
  & use them.

* If we can detect that a comment block begins with ``##``, a la
  JavaDoc, it might be useful to indicate interspersed section headers
  & explanatory text in a module.  For example::

      """Module docstring."""

      ##
      # Constants
      # =========

      a = 1
      b = 2

      ##
      # Exception Classes
      # =================

      class MyException(Exception): pass

      # etc.

* Should standalone strings also become (module/class) docstrings?
  Under what conditions?  We want to prevent arbitrary strings from
  becoming docstrings of prior attribute assignments etc.  Assume
  that there must be no blank lines between attributes and attribute
  docstrings?  (Use lineno of NEWLINE token.)

  Triple-quotes are sometimes used for multi-line comments (such as
  commenting out blocks of code).  How to reconcile?

* HappyDoc's idea of using comment blocks when there's no docstring
  may be useful to get around the conflict between `additional
  docstrings`_ and ``from __future__ import`` for module docstrings.
  A module could begin like this::

      #!/usr/bin/env python
      # :Author: Me
      # :Copyright: whatever

      """This is the public module docstring (``__doc__``)."""

      # More docs, in comments.
      # All comments at the beginning of a module could be
      # accumulated as docstrings.
      # We can't have another docstring here, because of the
      # ``__future__`` statement.

      from __future__ import division

  Using the JavaDoc convention of a doc-comment block beginning with
  ``##`` is useful though.  It allows doc-comments and implementation
  comments.

  .. _additional docstrings:
     ../peps/pep-0258.html#additional-docstrings

* HappyDoc uses an initial comment block to set "parser configuration
  values".  Do the same thing for Docutils, to set runtime settings on
  a per-module basis?  I.e.::

      # Docutils:setting=value

  Could be used to turn on/off function parameter comment recognition
  & other marginal features.  Could be used as a general mechanism to
  augment config files and command-line options (but which takes
  precedence?).

* Multi-file output should be divisible at arbitrary level.

* Support all forms of ``import`` statements:

  - ``import module``: listed as "module"
  - ``import module as alias``: "alias (module)"
  - ``from module import identifier``: "identifier (from module)"
  - ``from module import identifier as alias``: "alias (identifier
    from module)"
  - ``from module import *``: "all identifiers (``*``) from module"

* Have links to colorized Python source files from API docs?  And
  vice-versa: backlinks from the colorized source files to the API
  docs!

* In summaries, use the first *sentence* of a docstring if the first
  line is not followed by a blank line.


reStructuredText Parser
=======================

Also see the `... Or Not To Do?`__ list.

__ rst/alternatives.html#or-not-to-do

Bugs
----

* A container directive with ``:class:`` option gets the spurious
  class value "class".

Misc
----

* Another list problem::

      * foo
            * bar
            * baz

  This ends up as a definition list.  This is more of a usability
  issue.

* This case is probably meant to be a nested list, but it ends up as a
  list inside a block-quote without an error message::

      - foo

       - bar

  It should probably just be an error.

  The problem with this is that you don't notice easily in HTML that
  it's not a nested list but a block-quote -- there's not much of a
  visual difference.

* Treat enumerated lists that are not arabic and consist of only one
  item in a single line as ordinary paragraphs.  See
  <http://article.gmane.org/gmane.text.docutils.user/2635>.

* The citation syntax could use some improvements.  See
  <http://thread.gmane.org/gmane.text.docutils.user/2499> (and the
  sub-thread at
  <http://thread.gmane.org/gmane.text.docutils.user/2499/focus=3028>,
  and the follow-ups at
  <http://thread.gmane.org/gmane.text.docutils.user/3087>,
  <http://thread.gmane.org/gmane.text.docutils.user/3110>,
  <http://thread.gmane.org/gmane.text.docutils.user/3114>),
  <http://thread.gmane.org/gmane.text.docutils.user/2443>,
  <http://thread.gmane.org/gmane.text.docutils.user/2715>,
  <http://thread.gmane.org/gmane.text.docutils.user/3027>,
  <http://thread.gmane.org/gmane.text.docutils.user/3120>,
  <http://thread.gmane.org/gmane.text.docutils.user/3253>.

* The current list-recognition logic has too many false positives, as
  in ::

      * Aorta
      * V. cava superior
      * V. cava inferior

  Here ``V.`` is recognized as an enumerator, which leads to
  confusion.  We need to find a solution that resolves such problems
  without complicating the spec to much.

  See <http://thread.gmane.org/gmane.text.docutils.user/2524>.

* Add indirect links via citation references & footnote references.
  Example::

      `Goodger (2005)`_ is helpful.

      .. _Goodger (2005): [goodger2005]_
      .. [goodger2005] citation text

  See <http://thread.gmane.org/gmane.text.docutils.user/2499>.

* Complain about bad URI characters
  (http://article.gmane.org/gmane.text.docutils.user/2046) and
  disallow internal whitespace
  (http://article.gmane.org/gmane.text.docutils.user/2214).

* Create ``info``-level system messages for unnecessarily
  backslash-escaped characters (as in ``"\something"``, rendered as
  "something") to allow checking for errors which silently slipped
  through.

* Add (functional) tests for untested roles.

* Add test for ":figwidth: image" option of "figure" directive.  (Test
  code needs to check if PIL is available on the system.)

* Add support for CJK double-width whitespace (indentation) &
  punctuation characters (markup; e.g. double-width "*", "-", "+")?

* Add motivation sections for constructs in spec.

* Support generic hyperlink references to _`targets in other
  documents`?  Not in an HTML-centric way, though (it's trivial to say
  ``https://www.example.org/doc#name``, and useless in non-HTML
  contexts).  XLink/XPointer?  ``.. baseref::``?  See Doc-SIG
  2001-08-10.

* Implement the header row separator modification to table.el.  (Wrote
  to Takaaki Ota & the table.el mailing list on 2001-08-12, suggesting
  support for "=====" header rows.  On 2001-08-17 he replied, saying
  he'd put it on his to-do list, but "don't hold your breath".)

* Fix the parser's indentation handling to conform with the stricter
  definition in the spec.  (Explicit markup blocks should be strict or
  forgiving?)

  .. XXX What does this mean?  Can you elaborate, David?

* Make the parser modular.  Allow syntax constructs to be added or
  disabled at run-time.  Subclassing is probably not enough because it
  makes it difficult to apply multiple extensions.

* Generalize the "doctest block" construct (which is overly
  Python-centric) to other interactive sessions?  "Doctest block"
  could be renamed to "I/O block" or "interactive block", and each of
  these could also be recognized as such by the parser:

  - Shell sessions::

        $ cat example1.txt
        A block beginning with a "$ " prompt is interpreted as a shell
        session interactive block.  As with Doctest blocks, the
        interactive block ends with the first blank line, and wouldn't
        have to be indented.

  - Root shell sessions::

        # cat example2.txt
        A block beginning with a "# " prompt is interpreted as a root
        shell session (the user is or has to be logged in as root)
        interactive block.  Again, the block ends with a blank line.

  Other standard (and unambiguous) interactive session prompts could
  easily be added (such as "> " for WinDOS).

  Tony Ibbs spoke out against this idea (2002-06-14 Doc-SIG thread
  "docutils feedback").

* Add support for pragma (syntax-altering) directives.

  Some pragma directives could be local-scope unless explicitly
  specified as global/pragma using ":global:" options.

* Support whitespace in angle-bracketed standalone URLs according to
  Appendix E ("Recommendations for Delimiting URI in Context") of `RFC
  2396`_.

  .. _RFC 2396: https://www.rfc-editor.org/rfc/rfc2396.txt

* Use the vertical spacing of the source text to determine the
  corresponding vertical spacing of the output?

* [From Mark Nodine]  For cells in simple tables that comprise a
  single line, the justification can be inferred according to the
  following rules:

  1. If the text begins at the leftmost column of the cell,
     then left justification, ELSE
  2. If the text begins at the rightmost column of the cell,
     then right justification, ELSE
  3. Center justification.

  The onus is on the author to make the text unambiguous by adding
  blank columns as necessary.  There should be a parser setting to
  turn off justification-recognition (normally on would be fine).

  Decimal justification?

  All this shouldn't be done automatically.  Only when it's requested
  by the user, e.g. with something like this::

      .. table::
         :auto-indent:

         (Table goes here.)

  Otherwise it will break existing documents.

* Generate a warning or info message for paragraphs which should have
  been lists, like this one::

      1. line one
      3. line two

* Generalize the "target-notes" directive into a command-line option
  somehow?  See docutils-develop 2003-02-13.

* Allow a "::"-only paragraph (first line, actually) to introduce a
  _`literal block without a blank line`?  (Idea from Paul Moore.) ::

      ::
          This is a literal block

  Is indentation enough to make the separation between a paragraph
  which contains just a ``::`` and the literal text unambiguous?
  (There's one problem with this concession: If one wants a definition
  list item which defines the term "::", we'd have to escape it.)  It
  would only be reasonable to apply it to "::"-only paragraphs though.
  I think the blank line is visually necessary if there's text before
  the "::"::

      The text in this paragraph needs separation
      from the literal block following::
          This doesn't look right.

* Add new syntax for _`nested inline markup`?  Or extend the parser to
  parse nested inline markup somehow?  See the `collected notes
  <rst/alternatives.html#nested-inline-markup>`__.

* Drop the backticks from embedded URIs with omitted reference text?
  Should the angle brackets be kept in the output or not? ::

      <file_name>_

  Probably not worth the trouble.

* How about a syntax for alternative hyperlink behavior, such as "open
  in a new window" (as in HTML's ``<a target="_blank">``)?

  The MoinMoin wiki uses a caret ("^") at the beginning of the URL
  ("^" is not a legal URI character).  That could work for both inline
  and explicit targets::

      The `reference docs <^url>`__ may be handy.

      .. _name: ^url

  This may be too specific to HTML.  It hasn't been requested very
  often either.

* Add an option to add URI schemes at runtime.

* _`Segmented lists`::

      : segment : segment : segment
      : segment : segment : very long
        segment
      : segment : segment : segment

  The initial colon (":") can be thought of as a type of bullet

  We could even have segment titles::

      :: title  : title   : title
      : segment : segment : segment
      : segment : segment : segment

  This would correspond well to DocBook's SegmentedList.  Output could
  be tabular or "name: value" pairs, as described in DocBook's docs.

* Enable grid _`tables inside XML comments`, where "``--``" ends comments.

  Implementation possibilities:

  1. Make the table syntax characters into "table" directive options.
     This is the most flexible but most difficult, and we probably
     don't need that much flexibility.

  2. Substitute "~" for "-" with a specialized directive option
     (e.g. ":tildes:").

  3. Make the standard table syntax recognize "~" as well as "-", even
     without a directive option.  Individual tables would have to be
     internally consistent.

  4. Allow Unicode box characters for table markup
     (`feature request [6]`_)

  Directive options are preferable to configuration settings, because
  tables are document-specific.  A pragma directive would be another
  approach, to set the syntax once for a whole document.

  Unicode box character markup would kill two birds with one stone.

  In the meantime, the list-table_ directive is a good replacement for
  grid tables inside XML comments.

  .. _feature request [6]:
      https://sourceforge.net/p/docutils/feature-requests/6
  .. _list-table: ../ref/rst/directives.html#list-table


* Generalize docinfo contents (bibliographic fields): remove specific
  fields, and have only a single generic "field"?

* _`Line numbers` and "source" in system messages:

  - Add "source" and "line" keyword arguments to all Reporter calls?
    This would require passing source/line arguments along all
    intermediate functions (where currently only `line` is used).

    Or rather specify "line" only if actually needed?

    Currently, `document.reporter` uses a state machine instance to
    determine the "source" and "line" info from
    `statemachine.input_lines` if not given explicitly. Except for
    special cases, the "line" argument is not needed because,
    `document.statemachine` keeps record of the current line number.

  - For system messages generated after the parsing is completed (i.e. by
    transforms or the writer) "line" info must be present in the doctree
    elements.

    Elements' .line assignments should be checked.  (Assign to .source
    too?  Add a set_info method?  To what?)

    The "source" (and line number in the source) can either be added
    explicitly to the elements or determined from the “raw” line
    number by `document.statemachine.get_source_and_line`.

  - Some line numbers in elements are not being set properly
    (explicitly), just implicitly/automatically.  See rev. 1.74 of
    docutils/parsers/rst/states.py for an example of how to set.

  - The line numbers of definition list items are wrong::

        $ rst2pseudoxml.py --expose-internal-attribute line
        1
          2
          3

        5
          6
          7

        <document source="<stdin>">
            <definition_list>
                <definition_list_item internal:line="3">
                    <term>
                        1
                    <definition>
                        
                            2
                            3
                <definition_list_item internal:line="6">
                    <term>
                        5
                    <definition>
                        
                            6
                            7

* .. _none source:

  Quite a few nodes are getting a "None" source attribute as well.  In
  particular, see the bodies of definition lists.


Adaptable file extensions
-------------------------

Questions
`````````
/* Detailed source-code truncated for AI context efficiency. */
`````````

How about using ".*" to indicate "choose the most appropriate filename
extension"?  For example::

    .. _Another Document: another.*

* My point about using ``.*`` is that any other mechanism inside reST
  leads to too many ambiguities in reading reST documents; at least
  with ``.*`` it's clear that some kind of substitution is going on.

  --- Aahz

* What is to be done for output formats that don't *have* hyperlinks?
  For example, LaTeX targeted at print.  Hyperlinks may be "called
  out", as footnotes with explicit URLs.  (Don't convert the links.)

  But then there's also LaTeX targeted at PDFs, which *can* have
  links.  Perhaps a runtime setting for "*" could explicitly provide
  the extension, defaulting to the output file's extension.

* If this handles images also, how to differentiate between document
  and image links?  Element context (within "image")?  Which image
  extension to use for which document format? For HTML output, there
  is no reliable way of determining which extension to use (svg, png,
  jpg, jpeg, gif, ...).

  Should the system check for existing files?  No, not practical (the
  image files may be not available when the document is processed to HTML).

  Mailing list threads: `Images in both HTML and LaTeX`__ (especially
  `this summary of Lea's objections`__).

  __ https://sourceforge.net/p/docutils/mailman/docutils-users/thread/40BAA4B7.5020801%40python.org/#msg6918066
  __ https://sourceforge.net/p/docutils/mailman/message/6918089/

Chris Liechti suggests a new ``:link:`` role in `more-universal
links?`__::

    .. role:: link(rewrite)
       :transform: .txt|.html

  and then to use it::

    for more information see :link:`README.txt`

  it would be useful if it supported an additional option
  ``:format: html`` so that separate rules for each format can be
  defined. (like the "raw" role)

__ https://sourceforge.net/p/docutils/mailman/message/6919484/


Idea from Jim Fulton: an external lookup table of targets:

    I would like to specify the extension (e.g. .txt) [in the
    source, rather than ``filename.*``], but tell the converter to
    change references to the files anticipating that the files will
    be converted too.

    For example::

      .. _Another Document: another.txt

      rst2html.py --convert-links "another.txt bar.txt" foo.txt

    That is, name the files for which extensions should be converted.

    Note that I want to refer to original files in the original text
    (another.txt rather than another.*) because I want the
    unconverted text to stand on its own.

    Note that in most cases, people will be able to use globs::

      rst2html.py --convert-link-extensions-for "`echo *.txt`" foo.txt

    It might be nice to be able to use multiple arguments, as in::

      rst2html.py --convert-link-extensions-for *.txt -- foo.txt

    > Handle documents only, or objects (images, etc.) also?

    No, documents only, but there really is no need for guesswork.
    Just get the file names as command-line arguments.  EIBTI
    [explicit is better than implicit].

In `Patch #169`__ `Hyperlink extension rewriting`, John L. Clark
suggests command line options that map to-be-changed file extensions, e.g.::

     rst2html --map-extension rst html --map-extension jpg png \
        input-filename.rst

__ https://sourceforge.net/p/docutils/patches/169/

  Specifying the mapping as regular expressions would make this
  approach more generic and easier to implement (use ``re.replace``
  and refer to the "re" module's documentation instead of coding and
  documenting a home-grown extraction and mapping procedure).


Math Markup
-----------

Since Docutils 0.8, a "math" role and directive using LaTeX math
syntax as input format is part of reStructuredText.

Open issues:

* Use a "Transform" for math format conversions as extensively discussed in
  the "math directive issues" thread in May 2008
  (http://osdir.com/ml/text.docutils.devel/2008-05/threads.html)?

* Generic `math-output setting`_ (currently specific to HTML).
  (List of math-output preferences?)

* Try to be compatible with `Math support in Sphinx`_?

  * The ``:label:`` option selects a label for the equation, by which it
    can be cross-referenced, and causes an equation number to be issued.
    In Docutils, the option ``:name:`` sets the label.
    Equation numbering is not implemented yet.

  * Option ``:nowrap:`` prevents wrapping of the given math in a
    math environment (you have to specify the math environment in the
    content).

  .. _Math support in Sphinx: http://sphinx.pocoo.org/ext/math.html

* Equation numbering and references. (see the section on
  `object numbering and object references` for equations,
  formal tables, and images.)

.. _math-output setting: ../user/config.html#math-output


alternative input formats

Which equation environments should be supported by the math directive?

* one line:

  + numbered: `equation`
  + unnumbered: `equation*`

* multiline (test for ``\\`` outside of a nested environment
  (e.g. `array` or `cases`)

  + numbered: `align` (number every line)

    (To give one common number to all lines, put them in a `split`
    environment. Docutils then places it in an `equation` environment.)

  + unnumbered: `align*`

  + Sphinx math also supports `gather` (checking for blank lines in
    the content). Docutils puts content blocks separated by blank
    lines in separate math-block doctree nodes. (The only difference of
    `gather` to two consecutive "normal" environments seems to be that
    page-breaks between the two are prevented.)

See http://www.math.uiuc.edu/~hildebr/tex/displays.html.


HTML output
`````````
/* Detailed source-code truncated for AI context efficiency. */

* Special class argument to individually place figures?

  Example::

    .. figure:: foo.pdf
       :class: place-here-if-possible place-top place-bottom

  would be written as ``\figure[htb]{...}`` with
  the optional args:

  :H: place-here
  :h: place-here-if-possible
  :t: place-top
  :b: place-bottom
  :p: place-on-extra-page

  Alternative: class value = "place-" + optional arg, e.g. ``:class:
  place-htb``.

Footnotes
`````````

+ True footnotes with LaTeX auto-numbering (as option ``--latex-footnotes``)
  (also for target-footnotes):
  Write ``\footnote{<footnote content>}`` at the place of the
  ``<footnote_reference>`` node.

+ Open questions:

  - Load hyperref_ with option "hyperfootnotes" and/or
    package footnotebackref_ or leave this to the user?

  - Consider cases where LaTeX does not support footnotes
    (inside tables__, headings__, caption, ...).
    Use ftnxtra_, tabularx_, tabulary_, longtable_?

    __ http://www.tex.ac.uk/cgi-bin/texfaq2html?label=footintab
    __ http://www.tex.ac.uk/cgi-bin/texfaq2html?label=ftnsect

  - Consider `multiple footnote refs to common footnote text`__.

    KOMA-script classes and the KOMA scrextend_ package provide
    ``\footref`` that can be used for additional references to a
    ``\label``-ed footnote. Since 2021-05-01, ``\footref`` is provided
    by the LaTeX core, too.

    __ http://www.tex.ac.uk/cgi-bin/texfaq2html?label=multfoot

  - Consider numbered vs. symbolic footnotes.

+ document customization (links to how-to and packages)

.. Footnote packages at CTAN (www.ctan.org/pkg/):

   :footnote: provides a "savenotes" environment which collects all
              footnotes and emits them at ``end{savenotes}``
              (texlive-latex-recommended)

   :ftnxtra_: fixes the issue of footnote inside \caption{},
             tabular environment and \section{} like commands.

   :footnotebackref_: bidirectional links to/from footnote mark to
                      footnote text.

.. Footnote Discussion:

   `German tutorial
   <http://www2.informatik.hu-berlin.de/~ahamann/studies/footnotes.pdf>`__

   `wikibooks: footnote workarounds
   <https://en.wikibooks.org/wiki/LaTeX/Footnotes_and_Margin_Notes#Common_problems_and_workarounds>`__

.. _footnotebackref: https://www.ctan.org/pkg/footnotebackref
.. _ftnxtra: https://www.ctan.org/pkg/ftnxtra
.. _hyperref: https://www.ctan.org/pkg/hyperref
.. _longtable: https://www.ctan.org/pkg/longtable
.. _scrextend: https://www.ctan.org/pkg/longtable
.. _tabularx: https://www.ctan.org/pkg/tabularx


Other LaTeX constructs and packages instead of re-implementations

/* Detailed source-code truncated for AI context efficiency. */

The `LyX <http://www.lyx.org>`_ document processor has a comprehensive
Unicode to LaTeX conversion feature with a file called ``unicodesymbols``
that lists LaTeX counterparts for a wide range of Unicode characters.

* Use this in the LaTeXTranslator?
  Think of copyright issues!

* The "ucs" package has many translations in ...doc/latex/ucs/config/

* The bibstuff_ tool ships a `latex_codec` Python module!

.. _bibstuff: http://code.google.com/p/bibstuff/


XeTeX writer
````````````

* Glyphs missing in the font are left out in the PDF without warning
  (e.g. ⇔ left-right double arrow in the functional test output).

* Disable word-wrap (hyphenation) in literal text locally with
  ``providecommand{\nohyphenation}{\addfontfeatures{HyphenChar=None}}``?


problematic URLs

\href{https://www.w3.org/XML/Schema^^dev}{\fragileURLi}

From http://article.gmane.org/gmane.text.docutils.devel/3429/

The problem is that since we have a default value, we have to
differentiate between adding another stylesheet and replacing the
default.  I suggest that the existing --stylesheet & --stylesheet-path
options keep their semantics to replace the existing settings.  We
could introduce new --add-stylesheet & --add-stylesheet-path options,
which accumulate; further --stylesheet/--stylesheet-path options would
clear these lists.  The stylesheet or stylesheet_path setting (only
one may be set), plus the added_stylesheets and added_stylesheet_paths
settings, describe the combined styles.

For example, this run will have only one custom stylesheet:

    rstpep2html.py --stylesheet-path custom.css ...

This run will use the default stylesheet, and the custom one:

    rstpep2html.py --add-stylesheet-path custom.css ...

This run will use the default stylesheet, a custom local stylesheet,
and an external stylesheet:

    rstpep2html.py --add-stylesheet-path custom.css \
        --add-stylesheet https://www.example.org/external.css ...

This run will use only the second custom stylesheet:

    rstpep2html.py --add-stylesheet-path custom.css \
        --stylesheet-path second.css ...




Front-End Tools
===============

* Parameterize help text & defaults somehow?  Perhaps a callback?  Or
  initialize ``settings_spec`` in ``__init__`` or ``init_options``?

* Disable common options that don't apply?
  (This should now be easier with ``frontend.filter_settings_spec``.)

* Add ``--section-numbering`` command line option.  The "sectnum"
  directive should override the ``--no-section-numbering`` command
  line option then.

* Implement the following suggestions from clig.dev?

     Display output on success, but keep it brief.
     provide a --quiet option to suppress all non-essential output.
   
     Consider chaining several args as input and use --output
     (or redirection) for output.
   
     -- https://clig.dev/#help

.. _partial parsing:
   https://docs.python.org/3/library/argparse.html#partial-parsing

.. _configuration: ../user/config.html
.. _transforms: ../api/transforms.html



..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End:

---

### Benchmarks Benchmarking Cases Docutils Data Docs Dev Website (`benchmarks/benchmarking/cases/docutils_data/docs/dev/website.txt`)

===================
 Docutils Web Site
===================

:Author: David Goodger; open to all Docutils developers
:Contact: [email protected]
:Date: $Date$
:Revision: $Revision$
:Copyright: This document has been placed in the public domain.

The Docutils web site, <https://docutils.sourceforge.io/>, is
maintained by the ``docutils-update.local`` script, run by project
maintainers on their local machines.  The script
will process any .txt file which is newer than the corresponding .html
file in the local copy of the project's web directory and upload the changes
to the web site at SourceForge.

..  .. old instructions, for cron job:

    The Docutils web site, <https://docutils.sourceforge.io/>, is
    maintained automatically by the ``docutils-update`` script, run as an
    hourly cron job on shell.berlios.de (by user "wiemann").  The script
    will process any .txt file which is newer than the corresponding .html
    file in the project's web directory on shell.berlios.de
    (``/home/groups/docutils/htdocs/aux/htdocs/``) and upload the changes
    to the web site at SourceForge.

Please **do not** add any generated .html files to the Docutils
repository.  They will be generated automatically after a one-time
setup (`described below`__).

__ `Adding .txt Files`_

The docutils-update.local__ script is located at
``sandbox/infrastructure/docutils-update.local``.

__ https://docutils.sourceforge.io/sandbox/infrastructure/docutils-update.local

If you want to share files via the web, you can upload them using the
uploaddocutils.sh__ script
(``sandbox/infrastructure/uploaddocutils.sh``).

__ https://docutils.sourceforge.io/sandbox/infrastructure/uploaddocutils.sh


Setting Up
==========

(TBA)

.. hint::
  Anyone with checkin privileges can be a web-site maintainer. You need to
  set up the directories for a local website build.

  The procedure for that was on the docutils-devel list a while ago.


Adding .txt Files
=================

User/Contributor
----------------

When adding a new .txt file that should be converted to HTML:

#. Edit sandbox/infrastructure/htmlfiles.lst, and add the .html file
   corresponding to the new .txt file (please keep the sorted order).

#. Commit the edited version to the SVN repository.

Maintainer
----------

#. If there are new directories in the SVN, allow the update script to run
   once to create the directories in the filesystem before preparing for
   HTML processing.

#. Run the sandbox/infrastructure/update-htmlfiles shell script to generate
   .html files::

      cd <DOCUTILS-ROOT>/docutils/
      sandbox/infrastructure/update-htmlfiles \
      sandbox/infrastructure/htmlfiles.lst

   (Maybe this should become part of docutils-update.local.)


Removing Files & Directories
============================

#. Remove from SVN

#. Remove to-be-generated HTML files from
   ``sandbox/infrastructure/htmlfiles.lst``.

#. Removing files and directories from SVN will not trigger their removal
   from the web site.  Files and directories must be manually removed from
   sourceforge.net (under ``/home/project-web/docutils/htdocs/``).


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   End:

---

### Benchmarks Benchmarking Cases Docutils Data Docs Dev Rst Alternatives (`benchmarks/benchmarking/cases/docutils_data/docs/dev/rst/alternatives.txt`)

==================================================
 A Record of reStructuredText Syntax Alternatives
==================================================

:Author: David Goodger
:Contact: [email protected]
:Revision: $Revision$
:Date: $Date$
:Copyright: This document has been placed in the public domain.

The following are ideas, alternatives, and justifications that were
considered for reStructuredText syntax, which did not originate with
Setext_ or StructuredText_.  For an analysis of constructs which *did*
originate with StructuredText or Setext, please see `Problems With
StructuredText`_.  See the `reStructuredText Markup Specification`_
for full details of the established syntax.

The ideas are divided into sections:

* Implemented_: already done.  The issues and alternatives are
  recorded here for posterity.

* `Not Implemented`_: these ideas won't be implemented.

* Tabled_: these ideas should be revisited in the future.

* `To Do`_: these ideas should be implemented.  They're just waiting
  for a champion to resolve issues and get them done.

* `... Or Not To Do?`_: possible but questionable.  These probably
  won't be implemented, but you never know.

.. _Setext: https://docutils.sourceforge.io/mirror/setext.html
.. _StructuredText: https://zopestructuredtext.readthedocs.org/
.. _Problems with StructuredText: problems.html
.. _reStructuredText Markup Specification:
   ../../ref/rst/restructuredtext.html


.. contents::

-------------
 Implemented
-------------

Field Lists
===========

Prior to the syntax for field lists being finalized, several
alternatives were proposed.

1. Unadorned RFC822_ everywhere::

       Author: Me
       Version: 1

   Advantages: clean, precedent (RFC822-compliant).  Disadvantage:
   ambiguous (these paragraphs are a prime example).

   Conclusion: rejected.

2. Special case: use unadorned RFC822_ for the very first or very last
   text block of a document::

       """
       Author: Me
       Version: 1

       The rest of the document...
       """

   Advantages: clean, precedent (RFC822-compliant).  Disadvantages:
   special case, flat (unnested) field lists only, still ambiguous::

       """
       Usage: cmdname [options] arg1 arg2 ...

       We obviously *don't* want the like above to be interpreted as a
       field list item.  Or do we?
       """

   Conclusion: rejected for the general case, accepted for specific
   contexts (PEPs, email).

3. Use a directive::

       .. fields::

          Author: Me
          Version: 1

   Advantages: explicit and unambiguous, RFC822-compliant.
   Disadvantage: cumbersome.

   Conclusion: rejected for the general case (but such a directive
   could certainly be written).

4. Use Javadoc-style::

       @Author: Me
       @Version: 1
       @param a: integer

   Advantages: unambiguous, precedent, flexible.  Disadvantages:
   non-intuitive, ugly, not RFC822-compliant.

   Conclusion: rejected.

5. Use leading colons::

       :Author: Me
       :Version: 1

   Advantages: unambiguous, obvious (*almost* RFC822-compliant),
   flexible, perhaps even elegant.  Disadvantages: no precedent, not
   quite RFC822-compliant.

   Conclusion: accepted!

6. Use double colons::

       Author:: Me
       Version:: 1

   Advantages: unambiguous, obvious? (*almost* RFC822-compliant),
   flexible, similar to syntax already used for literal blocks and
   directives.  Disadvantages: no precedent, not quite
   RFC822-compliant, similar to syntax already used for literal blocks
   and directives.

   Conclusion: rejected because of the syntax similarity & conflicts.

Why is RFC822 compliance important?  It's a universal Internet
standard, and super obvious.  Also, I'd like to support the PEP format
(ulterior motive: get PEPs to use reStructuredText as their standard).
But it *would* be easy to get used to an alternative (easy even to
convert PEPs; probably harder to convert python-deviants ;-).

Unfortunately, without well-defined context (such as in email headers:
RFC822 only applies before any blank lines), the RFC822 format is
ambiguous.  It is very common in ordinary text.  To implement field
lists unambiguously, we need explicit syntax.

The following question was posed in a footnote:

   Should "bibliographic field lists" be defined at the parser level,
   or at the DPS transformation level?  In other words, are they
   reStructuredText-specific, or would they also be applicable to
   another (many/every other?) syntax?

The answer is that bibliographic fields are a
reStructuredText-specific markup convention.  Other syntaxes may
implement the bibliographic elements explicitly.  For example, there
would be no need for such a transformation for an XML-based markup
syntax.

.. _RFC822: https://www.rfc-editor.org/rfc/rfc822.txt


Interpreted Text "Roles"
========================

The original purpose of interpreted text was as a mechanism for
descriptive markup, to describe the nature or role of a word or
phrase.  For example, in XML we could say "<function>len</function>"
to mark up "len" as a function.  It is envisaged that within Python
docstrings (inline documentation in Python module source files, the
primary market for reStructuredText) the role of a piece of
interpreted text can be inferred implicitly from the context of the
docstring within the program source.  For other applications, however,
the role may have to be indicated explicitly.

Interpreted text is enclosed in single backquotes (`).

1. Initially, it was proposed that an explicit role could be indicated
   as a word or phrase within the enclosing backquotes:

   - As a prefix, separated by a colon and whitespace::

         `role: interpreted text`

   - As a suffix, separated by whitespace and a colon::

         `interpreted text :role`

   There are problems with the initial approach:

   - There could be ambiguity with interpreted text containing colons.
     For example, an index entry of "Mission: Impossible" would
     require a backslash-escaped colon.

   - The explicit role is descriptive markup, not content, and will
     not be visible in the processed output.  Putting it inside the
     backquotes doesn't feel right; the *role* isn't being quoted.

2. Tony Ibbs suggested that the role be placed outside the
   backquotes::

       role:`prefix` or `suffix`:role

   This removes the embedded-colons ambiguity, but limits the role
   identifier to be a single word (whitespace would be illegal).
   Since roles are not meant to be visible after processing, the lack
   of whitespace support is not important.

   The suggested syntax remains ambiguous with respect to ratios and
   some writing styles.  For example, suppose there is a "signal"
   identifier, and we write::

       ...calculate the `signal`:noise ratio.

   "noise" looks like a role.

3. As an improvement on #2, we can bracket the role with colons::

       :role:`prefix` or `suffix`:role:

   This syntax is similar to that of field lists, which is fine since
   both are doing similar things: describing.

   This is the syntax chosen for reStructuredText.

4. Another alternative is two colons instead of one::

       role::`prefix` or `suffix`::role

   But this is used for analogies ("A:B::C:D": "A is to B as C is to
   D").

   Both alternative #2 and #4 lack delimiters on both sides of the
   role, making it difficult to parse (by the reader).

5. Some kind of bracketing could be used:

   - Parentheses::

         (role)`prefix` or `suffix`(role)

   - Braces::

         {role}`prefix` or `suffix`{role}

   - Square brackets::

         [role]`prefix` or `suffix`[role]

   - Angle brackets::

         <role>`prefix` or `suffix`<role>

     (The overlap of \*ML tags with angle brackets would be too
     confusing and precludes their use.)

Syntax #3 was chosen for reStructuredText.


Comments
========

A problem with comments (actually, with all indented constructs) is
that they cannot be followed by an indented block -- a block quote --
without swallowing it up.

I thought that perhaps comments should be one-liners only.  But would
this mean that footnotes, hyperlink targets, and directives must then
also be one-liners?  Not a good solution.

Tony Ibbs suggested a "comment" directive.  I added that we could
limit a comment to a single text block, and that a "multi-block
comment" could use "comment-start" and "comment-end" directives.  This
would remove the indentation incompatibility.  A "comment" directive
automatically suggests "footnote" and (hyperlink) "target" directives
as well.  This could go on forever!  Bad choice.

Garth Kidd suggested that an "empty comment", a ".." explicit markup
start with nothing on the first line (except possibly whitespace) and
a blank line immediately following, could serve as an "unindent".  An
empty comment does **not** swallow up indented blocks following it,
so block quotes are safe.  "A tiny but practical wart."  Accepted.


Anonymous Hyperlinks
====================

Alan Jaffray came up with this idea, along with the following syntax::

    Search the `Python DOC-SIG mailing list archives`{}_.

    .. _: https://mail.python.org/pipermail/doc-sig/

The idea is sound and useful.  I suggested a "double underscore"
syntax::

    Search the `Python DOC-SIG mailing list archives`__.

    .. __: https://mail.python.org/pipermail/doc-sig/

But perhaps single underscores are okay?  The syntax looks better, but
the hyperlink itself doesn't explicitly say "anonymous"::

    Search the `Python DOC-SIG mailing list archives`_.

    .. _: https://mail.python.org/pipermail/doc-sig/

Mixing anonymous and named hyperlinks becomes confusing.  The order of
targets is not significant for named hyperlinks, but it is for
anonymous hyperlinks::

    Hyperlinks: anonymous_, named_, and another anonymous_.

    .. _named: named
    .. _: anonymous1
    .. _: anonymous2

Without the extra syntax of double underscores, determining which
hyperlink references are anonymous may be difficult.  We'd have to
check which references don't have corresponding targets, and match
those up with anonymous targets.  Keeping to a simple consistent
ordering (as with auto-numbered footnotes) seems simplest.

reStructuredText will use the explicit double-underscore syntax for
anonymous hyperlinks.  An alternative (see `Reworking Explicit Markup
(Round 1)`_ below) for the somewhat awkward ".. __:" syntax is "__"::

    An anonymous__ reference.

    __ http://anonymous


Reworking Explicit Markup (Round 1)
===================================

Alan Jaffray came up with the idea of `anonymous hyperlinks`_, added
to reStructuredText.  Subsequently it was asserted that hyperlinks
(especially anonymous hyperlinks) would play an increasingly important
role in reStructuredText documents, and therefore they require a
simpler and more concise syntax.  This prompted a review of the
current and proposed explicit markup syntaxes with regards to
improving usability.

1. Original syntax::

       .. _blah:                     internal hyperlink target
       .. _blah: http://somewhere    external hyperlink target
       .. _blah: blahblah_           indirect hyperlink target
       .. __:                        anonymous internal target
       .. __: http://somewhere       anonymous external target
       .. __: blahblah_              anonymous indirect target
       .. [blah] http://somewhere    footnote
       .. blah:: http://somewhere    directive
       .. blah: http://somewhere     comment

   .. Note::

      The comment text was intentionally made to look like a hyperlink
      target.

   Origins:

   * Except for the colon (a delimiter necessary to allow for
     phrase-links), hyperlink target ``.. _blah:`` comes from Setext.
   * Comment syntax from Setext.
   * Footnote syntax from StructuredText ("named links").
   * Directives and anonymous hyperlinks original to reStructuredText.

   Advantages:

   + Consistent explicit markup indicator: "..".
   + Consistent hyperlink syntax: ".. _" & ":".

   Disadvantages:

   - Anonymous target markup is awkward: ".. __:".
   - The explicit markup indicator ("..") is excessively overloaded?
   - Comment text is limited (can't look like a footnote, hyperlink,
     or directive).  But this is probably not important.

2. Alan Jaffray's proposed syntax #1::

       __ _blah                      internal hyperlink target
       __ blah: http://somewhere     external hyperlink target
       __ blah: blahblah_            indirect hyperlink target
       __                            anonymous internal target
       __ http://somewhere           anonymous external target
       __ blahblah_                  anonymous indirect target
       __ [blah] http://somewhere    footnote
       .. blah:: http://somewhere    directive
       .. blah: http://somewhere     comment

   The hyperlink-connoted underscores have become first-level syntax.

   Advantages:

   + Anonymous targets are simpler.
   + All hyperlink targets are one character shorter.

   Disadvantages:

   - Inconsistent internal hyperlink targets.  Unlike all other named
     hyperlink targets, there's no colon.  There's an extra leading
     underscore, but we can't drop it because without it, "blah" looks
     like a relative URI.  Unless we restore the colon::

         __ blah:                      internal hyperlink target

   - Obtrusive markup?

3. Alan Jaffray's proposed syntax #2::

       .. _blah                      internal hyperlink target
       .. blah: http://somewhere     external hyperlink target
       .. blah: blahblah_            indirect hyperlink target
       ..                            anonymous internal target
       .. http://somewhere           anonymous external target
       .. blahblah_                  anonymous indirect target
       .. [blah] http://somewhere    footnote
       !! blah: http://somewhere     directive
       ## blah: http://somewhere     comment

   Leading underscores have been (almost) replaced by "..", while
   comments and directives have gained their own syntax.

   Advantages:

   + Anonymous hyperlinks are simpler.
   + Unique syntax for comments.  Connotation of "comment" from
     some programming languages (including our favorite).
   + Unique syntax for directives.  Connotation of "action!".

   Disadvantages:

   - Inconsistent internal hyperlink targets.  Again, unlike all other
     named hyperlink targets, there's no colon.  There's a leading
     underscore, matching the trailing underscores of references,
     which no other hyperlink targets have.  We can't drop that one
     leading underscore though: without it, "blah" looks like a
     relative URI.  Again, unless we restore the colon::

         .. blah:                      internal hyperlink target

   - All (except for internal) hyperlink targets lack their leading
     underscores, losing the "hyperlink" connotation.

   - Obtrusive syntax for comments.  Alternatives::

         ;; blah: http://somewhere
            (also comment syntax in Lisp & others)
         ,, blah: http://somewhere
            ("comma comma": sounds like "comment"!)

   - Iffy syntax for directives.  Alternatives?

4. Tony Ibbs' proposed syntax::

       .. _blah:                     internal hyperlink target
       .. _blah: http://somewhere    external hyperlink target
       .. _blah: blahblah_           indirect hyperlink target
       ..                            anonymous internal target
       .. http://somewhere           anonymous external target
       .. blahblah_                  anonymous indirect target
       .. [blah] http://somewhere    footnote
       .. blah:: http://somewhere    directive
       .. blah: http://somewhere     comment

   This is the same as the current syntax, except for anonymous
   targets which drop their "__: ".

   Advantage:

   + Anonymous targets are simpler.

   Disadvantages:

   - Anonymous targets lack their leading underscores, losing the
     "hyperlink" connotation.
   - Anonymous targets are almost indistinguishable from comments.
     (Better to know "up front".)

5. David Goodger's proposed syntax: Perhaps going back to one of
   Alan's earlier suggestions might be the best solution.  How about
   simply adding "__ " as a synonym for ".. __: " in the original
   syntax?  These would become equivalent::

       .. __:                        anonymous internal target
       .. __: http://somewhere       anonymous external target
       .. __: blahblah_              anonymous indirect target

       __                            anonymous internal target
       __ http://somewhere           anonymous external target
       __ blahblah_                  anonymous indirect target

Alternative 5 has been adopted.


Backquotes in Phrase-Links
==========================

[From a 2001-06-05 Doc-SIG post in reply to questions from Doug
Hellmann.]

The first draft of the spec, posted to the Doc-SIG in November 2000,
used square brackets for phrase-links.  I changed my mind because:

1. In the first draft, I had already decided on single-backquotes for
   inline literal text.

2. However, I wanted to minimize the necessity for backslash escapes,
   for example when quoting Python repr-equivalent syntax that uses
   backquotes.

3. The processing of identifiers (function/method/attribute/module
   etc. names) into hyperlinks is a useful feature.  PyDoc recognizes
   identifiers heuristically, but it doesn't take much imagination to
   come up with counter-examples where PyDoc's heuristics would result
   in embarrassing failure.  I wanted to do it deterministically, and
   that called for syntax.  I called this construct "interpreted
   text".

4. Leveraging off the ``*emphasis*/**strong**`` syntax, lead to the
   idea of using double-backquotes as syntax.

5. I worked out some rules for inline markup recognition.

6. In combination with #5, double backquotes lent themselves to inline
   literals, neatly satisfying #2, minimizing backslash escapes.  In
   fact, the spec says that no interpretation of any kind is done
   within double-backquote inline literal text; backslashes do *no*
   escaping within literal text.

7. Single backquotes are then freed up for interpreted text.

8. I already had square brackets required for footnote references.

9. Since interpreted text will typically turn into hyperlinks, it was
   a natural fit to use backquotes as the phrase-quoting syntax for
   trailing-underscore hyperlinks.

The original inspiration for the trailing underscore hyperlink syntax
was Setext.  But for phrases Setext used a very cumbersome
``underscores_between_words_like_this_`` syntax.

The underscores can be viewed as if they were right-pointing arrows:
``-->``.  So ``hyperlink_`` points away from the reference, and
``.. _hyperlink:`` points toward the target.


Substitution Mechanism
======================

Substitutions arose out of a Doc-SIG thread begun on 2001-10-28 by
Alan Jaffray, "reStructuredText inline markup".  It reminded me of a
missing piece of the reStructuredText puzzle, first referred to in my
contribution to "Documentation markup & processing / PEPs" (Doc-SIG
2001-06-21).

Substitutions allow the power and flexibility of directives to be
shared by inline text.  They are a way to allow arbitrarily complex
inline objects, while keeping the details out of the flow of text.
They are the equivalent of SGML/XML's named entities.  For example, an
inline image (using reference syntax alternative 4d (vertical bars)
and definition alternative 3, the alternatives chosen for inclusion in
the spec)::

    The |biohazard| symbol must be used on containers used to dispose
    of medical waste.

    .. |biohazard| image:: biohazard.png
       [height=20 width=20]

The ``|biohazard|`` substitution reference will be replaced in-line by
whatever the ``.. |biohazard|`` substitution definition generates (in
this case, an image).  A substitution definition contains the
substitution text bracketed with vertical bars, followed by a an
embedded inline-compatible directive, such as "image".  A transform is
required to complete the substitution.

Syntax alternatives for the reference:

1. Use the existing interpreted text syntax, with a predefined role
   such as "sub"::

       The `biohazard`:sub: symbol...

   Advantages: existing syntax, explicit.  Disadvantages: verbose,
   obtrusive.

2. Use a variant of the interpreted text syntax, with a new suffix
   akin to the underscore in phrase-link references::

       (a) `name`@
       (b) `name`#
       (c) `name`&
       (d) `name`/
       (e) `name`<
       (f) `name`::
       (g) `name`:


   Due to incompatibility with other constructs and ordinary text
   usage, (f) and (g) are not possible.

3. Use interpreted text syntax with a fixed internal format::

       (a) `:name:`
       (b) `name:`
       (c) `name::`
       (d) `::name::`
       (e) `%name%`
       (f) `#name#`
       (g) `/name/`
       (h) `&name&`
       (i) `|name|`
       (j) `[name]`
       (k) `<name>`
       (l) `&name;`
       (m) `'name'`

   To avoid ML confusion (k) and (l) are definitely out.  Square
   brackets (j) won't work in the target (the substitution definition
   would be indistinguishable from a footnote).

   The ```/name/``` syntax (g) is reminiscent of "s/find/sub"
   substitution syntax in ed-like languages.  However, it may have a
   misleading association with regexps, and looks like an absolute
   POSIX path.  (i) is visually equivalent and lacking the
   connotations.

   A disadvantage of all of these is that they limit interpreted text,
   albeit only slightly.

4. Use specialized syntax, something new::

       (a) #name#
       (b) @name@
       (c) /name/
       (d) |name|
       (e) <<name>>
       (f) //name//
       (g) ||name||
       (h) ^name^
       (i) [[name]]
       (j) ~name~
       (k) !name!
       (l) =name=
       (m) ?name?
       (n) >name<

   "#" (a) and "@" (b) are obtrusive.  "/" (c) without backquotes
   looks just like a POSIX path; it is likely for such usage to appear
   in text.

   "|" (d) and "^" (h) are feasible.

5. Redefine the trailing underscore syntax.  See definition syntax
   alternative 4, below.

Syntax alternatives for the definition:

1. Use the existing directive syntax, with a predefined directive such
   as "sub".  It contains a further embedded directive resolving to an
   inline-compatible object::

       .. sub:: biohazard
          .. image:: biohazard.png
             [height=20 width=20]

       .. sub:: parrot
          That bird wouldn't *voom* if you put 10,000,000 volts
          through it!

   The advantages and disadvantages are the same as in inline
   alternative 1.

2. Use syntax as in #1, but with an embedded directivecompressed::

       .. sub:: biohazard image:: biohazard.png
          [height=20 width=20]

   This is a bit better than alternative 1, but still too much.

3. Use a variant of directive syntax, incorporating the substitution
   text, obviating the need for a special "sub" directive name.  If we
   assume reference alternative 4d (vertical bars), the matching
   definition would look like this::

       .. |biohazard| image:: biohazard.png
          [height=20 width=20]

4. (Suggested by Alan Jaffray on Doc-SIG from 2001-11-06.)

   Instead of adding new syntax, redefine the trailing underscore
   syntax to mean "substitution reference" instead of "hyperlink
   reference".  Alan's example::

       I had lunch with Jonathan_ today.  We talked about Zope_.

       .. _Jonathan: lj [user=jhl]
       .. _Zope: https://www.zope.dev/

   A problem with the proposed syntax is that URIs which look like
   simple reference names (alphanum plus ".", "-", "_") would be
   indistinguishable from substitution directive names.  A more
   consistent syntax would be::

       I had lunch with Jonathan_ today.  We talked about Zope_.

       .. _Jonathan: lj:: user=jhl
       .. _Zope: https://www.zope.dev/

   (``::`` after ``.. _Jonathan: lj``.)

   The "Zope" target is a simple external hyperlink, but the
   "Jonathan" target contains a directive.  Alan proposed is that the
   reference text be replaced by whatever the referenced directive
   (the "directive target") produces.  A directive reference becomes a
   hyperlink reference if the contents of the directive target resolve
   to a hyperlink.  If the directive target resolves to an icon, the
   reference is replaced by an inline icon.  If the directive target
   resolves to a hyperlink, the directive reference becomes a
   hyperlink reference.

   This seems too indirect and complicated for easy comprehension.

   The reference in the text will sometimes become a link, sometimes
   not.  Sometimes the reference text will remain, sometimes not.  We
   don't know *at the reference*::

       This is a `hyperlink reference`_; its text will remain.
       This is an `inline icon`_; its text will disappear.

   That's a problem.

The syntax that has been incorporated into the spec and parser is
reference alternative 4d with definition alternative 3::

    The |biohazard| symbol...

    .. |biohazard| image:: biohazard.png
       [height=20 width=20]

We can also combine substitution references with hyperlink references,
by appending a "_" (named hyperlink reference) or "__" (anonymous
hyperlink reference) suffix to the substitution reference.  This
allows us to click on an image-link::

    The |biohazard|_ symbol...

    .. |biohazard| image:: biohazard.png
       [height=20 width=20]
    .. _biohazard: https://www.cdc.gov/

There have been several suggestions for the naming of these
constructs, originally called "substitution references" and
"substitutions".

1. Candidate names for the reference construct:

   (a) substitution reference
   (b) tagging reference
   (c) inline directive reference
   (d) directive reference
   (e) indirect inline directive reference
   (f) inline directive placeholder
   (g) inline directive insertion reference
   (h) directive insertion reference
   (i) insertion reference
   (j) directive macro reference
   (k) macro reference
   (l) substitution directive reference

2. Candidate names for the definition construct:

   (a) substitution
   (b) substitution directive
   (c) tag
   (d) tagged directive
   (e) directive target
   (f) inline directive
   (g) inline directive definition
   (h) referenced directive
   (i) indirect directive
   (j) indirect directive definition
   (k) directive definition
   (l) indirect inline directive
   (m) named directive definition
   (n) inline directive insertion definition
   (o) directive insertion definition
   (p) insertion definition
   (q) insertion directive
   (r) substitution definition
   (s) directive macro definition
   (t) macro definition
   (u) substitution directive definition
   (v) substitution definition

"Inline directive reference" (1c) seems to be an appropriate term at
first, but the term "inline" is redundant in the case of the
reference.  Its counterpart "inline directive definition" (2g) is
awkward, because the directive definition itself is not inline.

"Directive reference" (1d) and "directive definition" (2k) are too
vague.  "Directive definition" could be used to refer to any
directive, not just those used for inline substitutions.

One meaning of the term "macro" (1k, 2s, 2t) is too
programming-language-specific.  Also, macros are typically simple text
substitution mechanisms: the text is substituted first and evaluated
later.  reStructuredText substitution definitions are evaluated in
place at parse time and substituted afterwards.

"Insertion" (1h, 1i, 2n-2q) is almost right, but it implies that
something new is getting added rather than one construct being
replaced by another.

Which brings us back to "substitution".  The overall best names are
"substitution reference" (1a) and "substitution definition" (2v).  A
long way to go to add one word!


Inline External Targets
=======================

Currently reStructuredText has two hyperlink syntax variations:

* Named hyperlinks::

      This is a named reference_ of one word ("reference").  Here is
      a `phrase reference`_.  Phrase references may even cross `line
      boundaries`_.

      .. _reference: https://www.example.org/reference/
      .. _phrase reference: https://www.example.org/phrase_reference/
      .. _line boundaries: https://www.example.org/line_boundaries/

  + Advantages:

    - The plaintext is readable.
    - Each target may be reused multiple times (e.g., just write
      ``"reference_"`` again).
    - No synchronized ordering of references and targets is necessary.

  + Disadvantages:

    - The reference text must be repeated as target names; could lead
      to mistakes.
    - The target URLs may be located far from the references, and hard
      to find in the plaintext.

* Anonymous hyperlinks (in current reStructuredText)::

      This is an anonymous reference__.  Here is an anonymous
      `phrase reference`__.  Phrase references may even cross `line
      boundaries`__.

      __ https://www.example.org/reference/
      __ https://www.example.org/phrase_reference/
      __ https://www.example.org/line_boundaries/

  + Advantages:

    - The plaintext is readable.
    - The reference text does not have to be repeated.

  + Disadvantages:

    - References and targets must be kept in sync.
    - Targets cannot be reused.
    - The target URLs may be located far from the references.

For comparison and historical background, StructuredText also has two
syntaxes for hyperlinks:

* First, ``"reference text":URL``::

      This is a "reference":https://www.example.org/reference/
      of one word ("reference").  Here is a "phrase
      reference":https://www.example.org/phrase_reference/.

* Second, ``"reference text", https://example.org/absolute_URL``::

      This is a "reference", https://www.example.org/reference/
      of one word ("reference").  Here is a "phrase reference",
      https://www.example.org/phrase_reference/.

Both syntaxes share advantages and disadvantages:

+ Advantages:

  - The target is specified immediately adjacent to the reference.

+ Disadvantages:

  - Poor plaintext readability.
  - Targets cannot be reused.
  - Both syntaxes use double quotes, common in ordinary text.
  - In the first syntax, the URL and the last word are stuck
    together, exacerbating the line wrap problem.
  - The second syntax is too magical; text could easily be written
    that way by accident (although only absolute URLs are recognized
    here, perhaps because of the potential for ambiguity).

A new type of "inline external hyperlink" has been proposed.

1. On 2002-06-28, Simon Budig proposed__ a new syntax for
   reStructuredText hyperlinks::

       This is a reference_(https://www.example.org/reference/) of one
       word ("reference").  Here is a `phrase
       reference`_(https://www.example.org/phrase_reference/).  Are
       these examples, (single-underscore), named?  If so, `anonymous
       references`__(https://www.example.org/anonymous/) using two
       underscores would probably be preferable.

   __ https://mail.python.org/pipermail/doc-sig/2002-June/002648.html

   The syntax, advantages, and disadvantages are similar to those of
   StructuredText.

   + Advantages:

     - The target is specified immediately adjacent to the reference.

   + Disadvantages:

     - Poor plaintext readability.
     - Targets cannot be reused (unless named, but the semantics are
       unclear).

   + Problems:

     - The ``"`ref`_(URL)"`` syntax forces the last word of the
       reference text to be joined to the URL, making a potentially
       very long word that can't be wrapped (URLs can be very long).
       The reference and the URL should be separate.  This is a
       symptom of the following point:

     - The syntax produces a single compound construct made up of two
       equally important parts, *with syntax in the middle*, *between*
       the reference and the target.  This is unprecedented in
       reStructuredText.

     - The "inline hyperlink" text is *not* a named reference (there's
       no lookup by name), so it shouldn't look like one.

     - According to the IETF standards RFC 2396 and RFC 2732,
       parentheses are legal URI characters and curly braces are legal
       email characters, making their use prohibitively difficult.

     - The named/anonymous semantics are unclear.

2. After an analysis__ of the syntax of (1) above, we came up with the
   following compromise syntax::

       This is an anonymous reference__
       __<https://www.example.org/reference/> of one word
       ("reference").  Here is a `phrase reference`__
       __<https://www.example.org/phrase_reference/>.  `Named
       references`_ _<https://www.example.org/anonymous/> use single
       underscores.

   __ https://mail.python.org/pipermail/doc-sig/2002-July/002670.html

   The syntax builds on that of the existing "inline internal
   targets": ``an _`inline internal target`.``

   + Advantages:

     - The target is specified immediately adjacent to the reference,
       improving maintainability:

       - References and targets are easily kept in sync.
       - The reference text does not have to be repeated.

     - The construct is executed in two parts: references identical to
       existing references, and targets that are new but not too big a
       stretch from current syntax.

     - There's overwhelming precedent for quoting URLs with angle
       brackets [#]_.

   + Disadvantages:

     - Poor plaintext readability.
     - Lots of "line noise".
     - Targets cannot be reused (unless named; see below).

   To alleviate the readability issue slightly, we could allow the
   target to appear later, such as after the end of the sentence::

       This is a named reference__ of one word ("reference").
       __<https://www.example.org/reference/>  Here is a `phrase
       reference`__.  __<https://www.example.org/phrase_reference/>

   Problem: this could only work for one reference at a time
   (reference/target pairs must be proximate [refA trgA refB trgB],
   not interleaved [refA refB trgA trgB] or nested [refA refB trgB
   trgA]).  This variation is too problematic; references and inline
   external targets will have to be kept immediately adjacent (see (3)
   below).

   The ``"reference__ __<target>"`` syntax is actually for "anonymous
   inline external targets", emphasized by the double underscores.  It
   follows that single trailing and leading underscores would lead to
   *implicitly named* inline external targets.  This would allow the
   reuse of targets by name.  So after ``"reference_ _<target>"``,
   another ``"reference_"`` would point to the same target.

   .. [#]
      From RFC 2396 (URI syntax):

          The angle-bracket "<" and ">" and double-quote (")
          characters are excluded [from URIs] because they are often
          used as the delimiters around URI in text documents and
          protocol fields.

          Using <> angle brackets around each URI is especially
          recommended as a delimiting style for URI that contain
          whitespace.

      From RFC 822 (email headers):

          Angle brackets ("<" and ">") are generally used to indicate
          the presence of a one machine-usable reference (e.g.,
          delimiting mailboxes), possibly including source-routing to
          the machine.

3. If it is best for references and inline external targets to be
   immediately adjacent, then they might as well be integrated.
   Here's an alternative syntax embedding the target URL in the
   reference::

       This is an anonymous `reference <https://www.example.org
       /reference/>`__ of one word ("reference").  Here is a `phrase
       reference <https://www.example.org/phrase_reference/>`__.

   Advantages and disadvantages are similar to those in (2).
   Readability is still an issue, but the syntax is a bit less
   heavyweight (reduced line noise).  Backquotes are required, even
   for one-word references; the target URL is included within the
   reference text, forcing a phrase context.

   We'll call this variant "embedded URIs".

   Problem: how to refer to a title like "HTML Anchors: <a>" (which
   ends with an HTML/SGML/XML tag)?  We could either require more
   syntax on the target (like ``"`reference text
   __<https://example.org/>`__"``), or require the odd conflicting
   title to be escaped (like ``"`HTML Anchors: \<a>`__"``).  The
   latter seems preferable, and not too onerous.

   Similarly to (2) above, a single trailing underscore would convert
   the reference & inline external target from anonymous to implicitly
   named, allowing reuse of targets by name.

   I think this is the least objectionable of the syntax alternatives.

Other syntax variations have been proposed (by Brett Cannon and Benja
Fallenstein)::

    `phrase reference`->https://www.example.org

    `phrase reference`@https://www.example.org

    `phrase reference`__ ->https://www.example.org

    `phrase reference` [-> https://www.example.org]

    `phrase reference`__ [-> https://www.example.org]

    `phrase reference` <https://www.example.org>_

None of these variations are clearly superior to #3 above.  Some have
problems that exclude their use.

With any kind of inline external target syntax it comes down to the
conflict between maintainability and plaintext readability.  I don't
see a major problem with reStructuredText's maintainability, and I
don't want to sacrifice plaintext readability to "improve" it.

The proponents of inline external targets want them for easily
maintainable web pages.  The arguments go something like this:

- Named hyperlinks are difficult to maintain because the reference
  text is duplicated as the target name.

  To which I said, "So use anonymous hyperlinks."

- Anonymous hyperlinks are difficult to maintain because the
  references and targets have to be kept in sync.

  "So keep the targets close to the references, grouped after each
  paragraph.  Maintenance is trivial."

- But targets grouped after paragraphs break the flow of text.

  "Surely less than URLs embedded in the text!  And if the intent is
  to produce web pages, not readable plaintext, then who cares about
  the flow of text?"

Many participants have voiced their objections to the proposed syntax:

    Garth Kidd: "I strongly prefer the current way of doing it.
    Inline is spectactularly messy, IMHO."

    Tony Ibbs: "I vehemently agree... that the inline alternatives
    being suggested look messy - there are/were good reasons they've
    been taken out...  I don't believe I would gain from the new
    syntaxes."

    Paul Moore: "I agree as well.  The proposed syntax is far too
    punctuation-heavy, and any of the alternatives discussed are
    ambiguous or too subtle."

Others have voiced their support:

    fantasai: "I agree with Simon.  In many cases, though certainly
    not in all, I find parenthesizing the url in plain text flows
    better than relegating it to a footnote."

    Ken Manheimer: "I'd like to weigh in requesting some kind of easy,
    direct inline reference link."

(Interesting that those *against* the proposal have been using
reStructuredText for a while, and those *for* the proposal are either
new to the list ["fantasai", background unknown] or longtime
StructuredText users [Ken Manheimer].)

I was initially ambivalent/against the proposed "inline external
targets".  I value reStructuredText's readability very highly, and
although the proposed syntax offers convenience, I don't know if the
convenience is worth the cost in ugliness.  Does the proposed syntax
compromise readability too much, or should the choice be left up to
the author?  Perhaps if the syntax is *allowed* but its use strongly
*discouraged*, for aesthetic/readability reasons?

After a great deal of thought and much input from users, I've decided
that there are reasonable use cases for this construct.  The
documentation should strongly caution against its use in most
situations, recommending independent block-level targets instead.
Syntax #3 above ("embedded URIs") will be used.


Doctree Representation of Transitions
=====================================

(Although not reStructuredText-specific, this section fits best in
this document.)

Having added the "horizontal rule" construct to the `reStructuredText
Markup Specification`_, a decision had to be made as to how to reflect
the construct in the implementation of the document tree.  Given this
source::

    Document
    ========

    Paragraph 1

    --------

    Paragraph 2

The horizontal rule indicates a "transition" (in prose terms) or the
start of a new "division".  Before implementation, the parsed document
tree would be::

    <document>
        <section names="document">
            <title>
                Document
            
                Paragraph 1
            --------               <--- error here
            
                Paragraph 2

There are several possibilities for the implementation:

1. Implement horizontal rules as "divisions" or segments.  A
   "division" is a title-less, non-hierarchical section.  The first
   try at an implementation looked like this::

       <document>
           <section names="document">
               <title>
                   Document
               
                   Paragraph 1
               
                   
                       Paragraph 2

   But the two paragraphs are really at the same level; they shouldn't
   appear to be at different levels.  There's really an invisible
   "first division".  The horizontal rule splits the document body
   into two segments, which should be treated uniformly.

2. Treating "divisions" uniformly brings us to the second
   possibility::

       <document>
           <section names="document">
               <title>
                   Document
               
                   
                       Paragraph 1
               
                   
                       Paragraph 2

   With this change, documents and sections will directly contain
   divisions and sections, but not body elements.  Only divisions will
   directly contain body elements.  Even without a horizontal rule
   anywhere, the body elements of a document or section would be
   contained within a division element.  This makes the document tree
   deeper.  This is similar to the way HTML_ treats document contents:
   grouped within a ```` element.

3. Implement them as "transitions", empty elements::

       <document>
           <section names="document">
               <title>
                   Document
               
                   Paragraph 1
               
               
                   Paragraph 2

   A transition would be a "point element", not containing anything,
   only identifying a point within the document structure.  This keeps
   the document tree flatter, but the idea of a "point element" like
   "transition" smells bad.  A transition isn't a thing itself, it's
   the space between two divisions.  However, transitions are a
   practical solution.

Solution 3 was chosen for incorporation into the document tree model.

.. _HTML: https://www.w3.org/MarkUp/


Syntax for Line Blocks
======================

* An early idea: How about a literal-block-like prefix, perhaps
  "``;;``"?  (It is, after all, a *semi-literal* literal block, no?)
  Example::

      Take it away, Eric the Orchestra Leader!  ;;

          A one, two, a one two three four

          Half a bee, philosophically,
          must, *ipso facto*, half not be.
          But half the bee has got to be,
          *vis a vis* its entity.  D'you see?

          But can a bee be said to be
          or not to be an entire bee,
          when half the bee is not a bee,
          due to some ancient injury?

          Singing...

  Kinda lame.

* Another idea: in an ordinary paragraph, if the first line ends with
  a backslash (escaping the newline), interpret the entire paragraph
  as a verse block?  For example::

      Add just one backslash\
      And this paragraph becomes
      An awful haiku

  (Awful, and arguably invalid, since in Japanese the word "haiku"
  contains three syllables not two.)

  This idea was superseded by the rules for escaped whitespace, useful
  for `character-level inline markup`_.

* In a `2004-02-22 docutils-develop message`__, Jarno Elonen proposed
  a "plain list" syntax (and also provided a patch)::

       | John Doe
       | President, SuperDuper Corp.
       | [email protected]

  __ https://thread.gmane.org/gmane.text.docutils.devel/1187

  This syntax is very natural.  However, these "plain lists" seem very
  similar to line blocks, and I see so little intrinsic "list-ness"
  that I'm loathe to add a new object.  I used the term "blurbs" to
  remove the "list" connotation from the originally proposed name.
  Perhaps line blocks could be refined to add the two properties they
  currently lack:

  A) long lines wrap nicely
  B) HTML output doesn't look like program code in non-CSS web
     browsers

  (A) is an issue of all 3 aspects of Docutils: syntax (construct
  behaviour), internal representation, and output.  (B) is partly an
  issue of internal representation but mostly of output.

ReStructuredText will redefine line blocks with the "|"-quoting
syntax.  The following is my current thinking.


Syntax
------

Perhaps line block syntax like this would do::

     | M6: James Bond
     | MIB: Mr. J.
     | IMF: not decided yet, but probably one of the following:
     |   Ethan Hunt
     |   Jim Phelps
     |   Claire Phelps
     | CIA: Lea Leiter

Note that the "nested" list does not have nested syntax (the "|" are
not further indented); the leading whitespace would still be
significant somehow (more below).  As for long lines in the input,
this could suffice::

     | John Doe
     | Founder, President, Chief Executive Officer, Cook, Bottle
       Washer, and All-Round Great Guy
     | SuperDuper Corp.
     | [email protected]

The lack of "|" on the third line indicates that it's a continuation
of the second line, wrapped.

I don't see much point in allowing arbitrary nested content.  Multiple
paragraphs or bullet lists inside a "blurb" doesn't make sense to me.
Simple nested line blocks should suffice.


Internal Representation
-----------------------

Line blocks are currently represented as text blobs as follows::

     <!ELEMENT line_block %text.model;>
     <!ATTLIST line_block
         %basic.atts;
         %fixedspace.att;>

Instead, we could represent each line by a separate element::

     <!ELEMENT line_block (line+)>
     <!ATTLIST line_block %basic.atts;>

     <!ELEMENT line %text.model;>
     <!ATTLIST line %basic.atts;>

We'd keep the significance of the leading whitespace of each line
either by converting it to non-breaking spaces at output, or with a
per-line margin.  Non-breaking spaces are simpler (for HTML, anyway)
but kludgey, and wouldn't support indented long lines that wrap.  But
should inter-word whitespace (i.e., not leading whitespace) be
preserved?  Currently it is preserved in line blocks.

Representing a more complex line block may be tricky::

     | But can a bee be said to be
     |     or not to be an entire bee,
     |         when half the bee is not a bee,
     |             due to some ancient injury?

Perhaps the representation could allow for nested line blocks::

     <!ELEMENT line_block (line | line_block)+>

With this model, leading whitespace would no longer be significant.
Instead, left margins are implied by the nesting.  The example above
could be represented as follows::

     <line_block>
         <line>
             But can a bee be said to be
         <line_block>
             <line>
                  or not to be an entire bee,
             <line_block>
                 <line>
                     when half the bee is not a bee,
                 <line_block>
                     <line>
                         due to some ancient injury?

I wasn't sure what to do about even more complex line blocks::

     |     Indented
     | Not indented
     |   Indented a bit
     |     A bit more
     |  Only one space

How should that be parsed and nested?  Should the first line have
the same nesting level (== indentation in the output) as the fourth
line, or the same as the last line?  Mark Nodine suggested that such
line blocks be parsed similarly to complexly-nested block quotes,
which seems reasonable.  In the example above, this would result in
the nesting of first line matching the last line's nesting.  In
other words, the nesting would be relative to neighboring lines
only.


Output
------

In HTML, line blocks are currently output as "" blocks, which
gives us significant whitespace and line breaks, but doesn't allow
long lines to wrap and causes monospaced output without stylesheets.
Instead, we could output "" elements parallelling the
representation above, where each nested  would
have an increased left margin (specified in the stylesheet).

Jarno suggested the following HTML output::

    
       First, top level line
       &nbsp;
          Second, once nested
          Third, once nested
          ...
       
       ...
    

The ```` and ``&nbsp;`` are meant to support non-CSS and
non-graphical browsers.  I understand the case for "br", but I'm not
so sure about hidden "&nbsp;".  I question how much effort should be
put toward supporting non-graphical and especially non-CSS browsers,
at least for html4css1.py output.

Should the lines themselves be ```` or ````?  I don't like
mixing inline and block-level elements.


Implementation Plan
-------------------

We'll leave the old implementation in place (via the "line-block"
directive only) until all Writers have been updated to support the new
syntax & implementation.  The "line-block" directive can then be
updated to use the new internal representation, and its documentation
will be updated to recommend the new syntax.


List-Driven Tables
==================

The original idea came from Dylan Jay:

    ... to use a two level bulleted list with something to
    indicate it should be rendered as a table ...

It's an interesting idea.  It could be implemented in as a directive
which transforms a uniform two-level list into a table.  Using a
directive would allow the author to explicitly set the table's
orientation (by column or by row), the presence of row headers, etc.

Alternatives:

1. (Implemented in Docutils 0.3.8).

   Bullet-list-tables might look like this::

       .. list-table::

          * - Treat
            - Quantity
            - Description
          * - Albatross!
            - 299
            - On a stick!
          * - Crunchy Frog!
            - 1499
            - If we took the bones out, it wouldn't be crunchy,
              now would it?
          * - Gannet Ripple!
            - 199
            - On a stick!

   This list must be written in two levels.  This wouldn't work::

       .. list-table::

          * Treat
          * Albatross!
          * Gannet!
          * Crunchy Frog!

          * Quantity
          * 299
          * 199
          * 1499

          * Description
          * On a stick!
          * On a stick!
          * If we took the bones out...

   The above is a single list of 12 items.  The blank lines are not
   significant to the markup.  We'd have to explicitly specify how
   many columns or rows to use, which isn't a good idea.

2. Beni Cherniavsky suggested a field list alternative.  It could look
   like this::

       .. field-list-table::
          :headrows: 1

          - :treat: Treat
            :quantity: Quantity
            :descr: Description

          - :treat: Albatross!
            :quantity: 299
            :descr: On a stick!

          - :treat: Crunchy Frog!
            :quantity: 1499
            :descr: If we took the bones out, it wouldn't be
                    crunchy, now would it?

   Column order is determined from the order of fields in the first
   row.  Field order in all other rows is ignored.  As a side-effect,
   this allows trivial re-arrangement of columns.  By using named
   fields, it becomes possible to omit fields in some rows without
   losing track of things, which is important for spans.

3. An alternative to two-level bullet lists would be to use enumerated
   lists for the table cells::

       .. list-table::

           * 1. Treat
             2. Quantity
             3. Description
           * 1. Albatross!
             2. 299
             3. On a stick!
           * 1. Crunchy Frog!
             2. 1499
             3. If we took the bones out, it wouldn't be crunchy,
                now would it?

   That provides better correspondence between cells in the same
   column than does bullet-list syntax, but not as good as field list
   syntax.  I think that were only field-list-tables available, a lot
   of users would use the equivalent degenerate case::

       .. field-list-table::
           - :1: Treat
             :2: Quantity
             :3: Description
           ...

4. Another natural variant is to allow a description list with field
   lists as descriptions::

       .. list-table::
           :headrows: 1

           Treat
               :quantity: Quantity
               :descr: Description
           Albatross!
               :quantity: 299
               :descr: On a stick!
           Crunchy Frog!
               :quantity: 1499
               :descr: If we took the bones out, it wouldn't be
                       crunchy, now would it?

   This would make the whole first column a header column ("stub").
   It's limited to a single column and a single paragraph fitting on
   one source line.  Also it wouldn't allow for empty cells or row
   spans in the first column.  But these are limitations that we could
   live with, like those of simple tables.

The List-driven table feature could be done in many ways.  Each user
will have their preferred usage.  Perhaps a single "list-table"
directive could handle them all, depending on which options and
content are present.

Issues:

* How to indicate that there's 1 header row?  Perhaps two lists?  ::

      .. list-table::

         + - Treat
           - Quantity
           - Description

         * - Albatross!
           - 299
           - On a stick!

  This is probably too subtle though.  Better would be a directive
  option, like ``:headrows: 1``.  An early suggestion for the header
  row(s) was to use a directive option::

      .. field-list-table::
         :header:
             - :treat: Treat
               :quantity: Quantity
               :descr: Description
         - :treat: Albatross!
           :quantity: 299
           :descr: On a stick!

  But the table data is at two levels and looks inconsistent.

  In general, we cannot extract the header row from field lists' field
  names because field names cannot contain everything one might put in
  a table cell.  A separate header row also allows shorter field names
  and doesn't force one to rewrite the whole table when the header
  text changes.  But for simpler cases, we can offer a ":header:
  fields" option, which does extract header cells from field names::

      .. field-list-table::
          :header: fields

          - :Treat: Albatross!
            :Quantity: 299
            :Description: On a stick!

* How to indicate the column widths?  A directive option? ::

      .. list-table::
         :widths: 15 10 35

  Automatic defaults from the text used?

* How to handle row and/or column spans?

  In a field list, column-spans can be indicated by specifying the
  first and last fields, separated by space-dash-space or ellipsis::

      - :foo - baz: quuux
      - :foo ... baz: quuux

  Commas were proposed for column spans::

      - :foo, bar: quux

  But non-adjacent columns become problematic.  Should we report an
  error, or duplicate the value into each span of adjacent columns (as
  was suggested)?  The latter suggestion is appealing but may be too
  clever.  Best perhaps to simply specify the two ends.

  It was suggested that comma syntax should be allowed, too, in order
  to allow the user to avoid trouble when changing the column order.
  But changing the column order of a table with spans is not trivial;
  we shouldn't make it easier to mess up.

  One possible syntax for row-spans is to simply treat any row where a
  field is missing as a row-span from the last row where it appeared.
  Leaving a field empty would still be possible by writing a field
  with empty content.  But this is too implicit.

  Another way would be to require an explicit continuation marker
  (``...``/``-"-``/``"``?) in all but the first row of a spanned
  field.  Empty comments could work ("..").  If implemented, the same
  marker could also be supported in simple tables, which lack
  row-spanning abilities.

  Explicit markup like ":rowspan:" and ":colspan:" was also suggested.

  Sometimes in a table, the first header row contains spans.  It may
  be necessary to provide a way to specify the column field names
  independently of data rows.  A directive option would do it.

* We could specify "column-wise" or "row-wise" ordering, with the same
  markup structure.  For example, with definition data::

      .. list-table::
         :column-wise:

         Treat
             - Albatross!
             - Crunchy Frog!
         Quantity
             - 299
             - 1499
         Description
             - On a stick!
             - If we took the bones out, it wouldn't be
               crunchy, now would it?

* A syntax for _`stubs in grid tables` is easy to imagine::

      +------------------------++------------+----------+
      | Header row, column 1   || Header 2   | Header 3 |
      +========================++============+==========+
      | body row 1, column 1   || column 2   | column 3 |
      +------------------------++------------+----------+

  Or this idea from Nick Moffitt::

      +-----+---+---+
      | XOR # T | F |
      +=====+===+===+
      |   T # F | T |
      +-----+---+---+
      |   F # T | F |
      +-----+---+---+


Auto-Enumerated Lists
=====================

Implemented 2005-03-24: combination of variation 1 & 2.

The advantage of auto-numbered enumerated lists would be similar to
that of auto-numbered footnotes: lists could be written and rearranged
without having to manually renumber them.  The disadvantages are also
the same: input and output wouldn't match exactly; the markup may be
ugly or confusing (depending on which alternative is chosen).

1. Use the "#" symbol.  Example::

       #. Item 1.
       #. Item 2.
       #. Item 3.

   Advantages: simple, explicit.  Disadvantage: enumeration sequence
   cannot be specified (limited to arabic numerals); ugly.

2. As a variation on #1, first initialize the enumeration sequence?
   For example::

       a) Item a.
       #) Item b.
       #) Item c.

   Advantages: simple, explicit, any enumeration sequence possible.
   Disadvantages: ugly; perhaps confusing with mixed concrete/abstract
   enumerators.

3. Alternative suggested by Fred Bremmer, from experience with MoinMoin::

       1. Item 1.
       1. Item 2.
       1. Item 3.

   Advantages: enumeration sequence is explicit (could be multiple
   "a." or "(I)" tokens).  Disadvantages: perhaps confusing; otherwise
   erroneous input (e.g., a duplicate item "1.") would pass silently,
   either causing a problem later in the list (if no blank lines
   between items) or creating two lists (with blanks).

   Take this input for example::

       1. Item 1.

       1. Unintentional duplicate of item 1.

       2. Item 2.

   Currently the parser will produce two list, "1" and "1,2" (no
   warnings, because of the presence of blank lines).  Using Fred's
   notation, the current behavior is "1,1,2 -> 1 1,2" (without blank
   lines between items, it would be "1,1,2 -> 1 [WARNING] 1,2").  What
   should the behavior be with auto-numbering?

   Fred has produced a patch__, whose initial behavior is as follows::

       1,1,1   -> 1,2,3
       1,2,2   -> 1,2,3
       3,3,3   -> 3,4,5
       1,2,2,3 -> 1,2,3 [WARNING] 3
       1,1,2   -> 1,2 [WARNING] 2

   (After the "[WARNING]", the "3" would begin a new list.)

   I have mixed feelings about adding this functionality to the spec &
   parser.  It would certainly be useful to some users (myself
   included; I often have to renumber lists).  Perhaps it's too
   clever, asking the parser to guess too much.  What if you *do* want
   three one-item lists in a row, each beginning with "1."?  You'd
   have to use empty comments to force breaks.  Also, I question
   whether "1,2,2 -> 1,2,3" is optimal behavior.

   In response, Fred came up with "a stricter and more explicit rule
   [which] would be to only auto-number silently if *all* the
   enumerators of a list were identical".  In that case::

       1,1,1   -> 1,2,3
       1,2,2   -> 1,2 [WARNING] 2
       3,3,3   -> 3,4,5
       1,2,2,3 -> 1,2 [WARNING] 2,3
       1,1,2   -> 1,2 [WARNING] 2

   Should any start-value be allowed ("3,3,3"), or should
   auto-numbered lists be limited to begin with ordinal-1 ("1", "A",
   "a", "I", or "i")?

   __ https://sourceforge.net/tracker/index.php?func=detail&aid=548802
      &group_id=38414&atid=422032

4. Alternative proposed by Tony Ibbs::

       #1. First item.
       #3. Aha - I edited this in later.
       #2. Second item.

   The initial proposal required unique enumerators within a list, but
   this limits the convenience of a feature of already limited
   applicability and convenience.  Not a useful requirement; dropped.

   Instead, simply prepend a "#" to a standard list enumerator to
   indicate auto-enumeration.  The numbers (or letters) of the
   enumerators themselves are not significant, except:

   - as a sequence indicator (arabic, roman, alphabetic; upper/lower),

   - and perhaps as a start value (first list item).

   Advantages: explicit, any enumeration sequence possible.
   Disadvantages: a bit ugly.


Adjacent citation references
============================

A special case for inline markup was proposed and implemented:
multiple citation references could be joined into one::

   [cite1]_[cite2]_ instead of requiring [cite1]_ [cite2]_

However, this was rejected as an unwarranted exception to the rules
for inline markup.
(The main motivation for the proposal, grouping citations in the latex writer,
was implemented by recognising the second group in the example above and
transforming it into ``\cite{cite1,cite2}``.)


Inline markup recognition
=========================

Implemented 2011-12-05 (version 0.9):
Extended `inline markup recognition rules`_.

Non-ASCII whitespace, punctuation characters and "international" quotes are
allowed around inline markup (based on `Unicode categories`_). The rules for
ASCII characters were not changed.

Rejected alternatives:

a) Use `Unicode categories`_ for all chars (ASCII or not)

   +1  comprehensible, standards based,
   -1  many "false positives" need escaping,
   -1  not backwards compatible.

b) full backwards compatibility

   :Pi: only before start-string
   :Pf: only behind end-string
   :Po: "conservative" sorting of other punctuation:

        :``.,;!?\\``: Close
        :``¡¿``:   Open

   +1  backwards compatible,
   +1  logical extension of the existing rules,
   -1  exception list for "other" punctuation needed,
   -1  rules even more complicated,
   -1  not clear how to sort "other" punctuation that is currently not
       recognized,
   -2  international quoting convention like
       »German ›angular‹ quotes« not recognized.

.. _Inline markup recognition rules:
   ../../ref/rst/restructuredtext.html#inline-markup-recognition-rules
.. _Unicode categories:
   https://www.unicode.org/Public/5.1.0/ucd/UCD.html#General_Category_Values


-----------------
 Not Implemented
-----------------

Reworking Footnotes
===================

As a further wrinkle (see `Reworking Explicit Markup (Round 1)`_
above), in the wee hours of 2002-02-28 I posted several ideas for
changes to footnote syntax:

    - Change footnote syntax from ``.. [1]`` to ``_[1]``? ...
    - Differentiate (with new DTD elements) author-date "citations"
      (``[GVR2002]``) from numbered footnotes? ...
    - Render footnote references as superscripts without "[]"? ...

These ideas are all related, and suggest changes in the
reStructuredText syntax as well as the docutils tree model.

The footnote has been used for both true footnotes (asides expanding
on points or defining terms) and for citations (references to external
works).  Rather than dealing with one amalgam construct, we could
separate the current footnote concept into strict footnotes and
citations.  Citations could be interpreted and treated differently
from footnotes.  Footnotes would be limited to numerical labels:
manual ("1") and auto-numbered (anonymous "#", named "#label").

The footnote is the only explicit markup construct (starts with ".. ")
that directly translates to a visible body element.  I've always been
a little bit uncomfortable with the ".. " marker for footnotes because
of this; ".. " has a connotation of "special", but footnotes aren't
especially "special".  Printed texts often put footnotes at the bottom
of the page where the reference occurs (thus "foot note").  Some HTML
designs would leave footnotes to be rendered the same positions where
they're defined.  Other online and printed designs will gather
footnotes into a section near the end of the document, converting them
to "endnotes" (perhaps using a directive in our case); but this
"special processing" is not an intrinsic property of the footnote
itself, but a decision made by the document author or processing
system.

Citations are almost invariably collected in a section at the end of a
document or section.  Citations "disappear" from where they are
defined and are magically reinserted at some well-defined point.
There's more of a connection to the "special" connotation of the ".. "
syntax.  The point at which the list of citations is inserted could be
defined manually by a directive (e.g., ".. citations::"), and/or have
default behavior (e.g., a section automatically inserted at the end of
the document) that might be influenced by options to the Writer.

Syntax proposals:

+ Footnotes:

  - Current syntax::

        .. [1] Footnote 1
        .. [#] Auto-numbered footnote.
        .. [#label] Auto-labeled footnote.

  - The syntax proposed in the original 2002-02-28 Doc-SIG post:
    remove the ".. ", prefix a "_"::

        _[1] Footnote 1
        _[#] Auto-numbered footnote.
        _[#label] Auto-labeled footnote.

    The leading underscore syntax (earlier dropped because
    ``.. _[1]:`` was too verbose) is a useful reminder that footnotes
    are hyperlink targets.

  - Minimal syntax: remove the ".. [" and "]", prefix a "_", and
    suffix a "."::

        _1. Footnote 1.
        _#. Auto-numbered footnote.
        _#label. Auto-labeled footnote.

                 ``_1.``, ``_#.``, and ``_#label.`` are markers,
                 like list markers.

    Footnotes could be rendered something like this in HTML

        | 1. This is a footnote.  The brackets could be dropped
        |    from the label, and a vertical bar could set them
        |    off from the rest of the document in the HTML.

    Two-way hyperlinks on the footnote marker ("1." above) would also
    help to differentiate footnotes from enumerated lists.

    If converted to endnotes (by a directive/transform), a horizontal
    half-line might be used instead.  Page-oriented output formats
    would typically use the horizontal line for true footnotes.

+ Footnote references:

  - Current syntax::

        [1]_, [#]_, [#label]_

  - Minimal syntax to match the minimal footnote syntax above::

        1_, #_, #label_

    As a consequence, pure-numeric hyperlink references would not be
    possible; they'd be interpreted as footnote references.

+ Citation references: no change is proposed from the current footnote
  reference syntax::

      [GVR2001]_

+ Citations:

  - Current syntax (footnote syntax)::

        .. [GVR2001] Python Documentation; van Rossum, Drake, et al.;
           https://www.python.org/doc/

  - Possible new syntax::

        _[GVR2001] Python Documentation; van Rossum, Drake, et al.;
                   https://www.python.org/doc/

        _[DJG2002]
            Docutils: Python Documentation Utilities project; Goodger
            et al.; https://docutils.sourceforge.io/

    Without the ".. " marker, subsequent lines would either have to
    align as in one of the above, or we'd have to allow loose
    alignment (I'd rather not)::

        _[GVR2001] Python Documentation; van Rossum, Drake, et al.;
            https://www.python.org/doc/

I proposed adopting the "minimal" syntax for footnotes and footnote
references, and adding citations and citation references to
reStructuredText's repertoire.  The current footnote syntax for
citations is better than the alternatives given.

From a reply by Tony Ibbs on 2002-03-01:

    However, I think easier with examples, so let's create one::

        Fans of Terry Pratchett are perhaps more likely to use
        footnotes [1]_ in their own writings than other people
        [2]_.  Of course, in *general*, one only sees footnotes
        in academic or technical writing - it's use in fiction
        and letter writing is not normally considered good
        style [4]_, particularly in emails (not a medium that
        lends itself to footnotes).

        .. [1] That is, little bits of referenced text at the
           bottom of the page.
        .. [2] Because Terry himself does, of course [3]_.
        .. [3] Although he has the distinction of being
           *funny* when he does it, and his fans don't always
           achieve that aim.
        .. [4] Presumably because it detracts from linear
           reading of the text - this is, of course, the point.

    and look at it with the second syntax proposal::

        Fans of Terry Pratchett are perhaps more likely to use
        footnotes [1]_ in their own writings than other people
        [2]_.  Of course, in *general*, one only sees footnotes
        in academic or technical writing - it's use in fiction
        and letter writing is not normally considered good
        style [4]_, particularly in emails (not a medium that
        lends itself to footnotes).

        _[1] That is, little bits of referenced text at the
             bottom of the page.
        _[2] Because Terry himself does, of course [3]_.
        _[3] Although he has the distinction of being
             *funny* when he does it, and his fans don't always
             achieve that aim.
        _[4] Presumably because it detracts from linear
             reading of the text - this is, of course, the point.

    (I note here that if I have gotten the indentation of the
    footnotes themselves correct, this is clearly not as nice.  And if
    the indentation should be to the left margin instead, I like that
    even less).

    and the third (new) proposal::

        Fans of Terry Pratchett are perhaps more likely to use
        footnotes 1_ in their own writings than other people
        2_.  Of course, in *general*, one only sees footnotes
        in academic or technical writing - it's use in fiction
        and letter writing is not normally considered good
        style 4_, particularly in emails (not a medium that
        lends itself to footnotes).

        _1. That is, little bits of referenced text at the
            bottom of the page.
        _2. Because Terry himself does, of course 3_.
        _3. Although he has the distinction of being
            *funny* when he does it, and his fans don't always
            achieve that aim.
        _4. Presumably because it detracts from linear
            reading of the text - this is, of course, the point.

    I think I don't, in practice, mind the targets too much (the use
    of a dot after the number helps a lot here), but I do have a
    problem with the body text, in that I don't naturally separate out
    the footnotes as different than the rest of the text - instead I
    keep wondering why there are numbers interspered in the text.  The
    use of brackets around the numbers ([ and ]) made me somehow parse
    the footnote references as "odd" - i.e., not part of the body text
    - and thus both easier to skip, and also (paradoxically) easier to
    pick out so that I could follow them.

    Thus, for the moment (and as always susceptable to argument), I'd
    say -1 on the new form of footnote reference (i.e., I much prefer
    the existing ``[1]_`` over the proposed ``1_``), and ambivalent
    over the proposed target change.

    That leaves David's problem of wanting to distinguish footnotes
    and citations - and the only thing I can propose there is that
    footnotes are numeric or # and citations are not (which, as a
    human being, I can probably cope with!).

From a reply by Paul Moore on 2002-03-01:

    I think the current footnote syntax ``[1]_`` is *exactly* the
    right balance of distinctness vs unobtrusiveness.  I very
    definitely don't think this should change.

    On the target change, it doesn't matter much to me.

From a further reply by Tony Ibbs on 2002-03-01, referring to the
"[1]" form and actual usage in email:

    Clearly this is a form people are used to, and thus we should
    consider it strongly (in the same way that the usage of ``*..*``
    to mean emphasis was taken partly from email practise).

    Equally clearly, there is something "magical" for people in the
    use of a similar form (i.e., ``[1]``) for both footnote reference
    and footnote target - it seems natural to keep them similar.

    ...

    I think that this established plaintext usage leads me to strongly
    believe we should retain square brackets at both ends of a
    footnote.  The markup of the reference end (a single trailing
    underscore) seems about as minimal as we can get away with.  The
    markup of the target end depends on how one envisages the thing -
    if ".." means "I am a target" (as I tend to see it), then that's
    good, but one can also argue that the "_[1]" syntax has a neat
    symmetry with the footnote reference itself, if one wishes (in
    which case ".." presumably means "hidden/special" as David seems
    to think, which is why one needs a ".." *and* a leading underline
    for hyperlink targets.

Given the persuading arguments voiced, we'll leave footnote & footnote
reference syntax alone.  Except that these discussions gave rise to
the "auto-symbol footnote" concept, which has been added.  Citations
and citation references have also been added.


Syntax for Questions & Answers
==============================

Implement as a generic two-column marked list?  As a standalone
(non-directive) construct?  (Is the markup ambiguous?)  Add support to
parts.contents?

New elements would be required.  Perhaps::

    <!ELEMENT question_list (question_list_item+)>
    <!ATTLIST question_list
        numbering  (none | local | global)
                            #IMPLIED
        start     NUMBER    #IMPLIED>
    <!ELEMENT question_list_item (question, answer*)>
    <!ELEMENT question %text.model;>
    <!ELEMENT answer (%body.elements;)+>

Originally I thought of implementing a Q&A list with special syntax::

    Q: What am I?

    A: You are a question-and-answer
       list.

    Q: What are you?

    A: I am the omniscient "we".

Where each "Q" and "A" could also be numbered (e.g., "Q1").  However,
a simple enumerated or bulleted list will do just fine for syntax.  A
directive could treat the list specially; e.g. the first paragraph
could be treated as a question, the remainder as the answer (multiple
answers could be represented by nested lists).  Without special
syntax, this directive becomes low priority.

As described in the FAQ__, no special syntax or directive is needed
for this application.

__ https://docutils.sourceforge.io/FAQ.html
   #how-can-i-mark-up-a-faq-or-other-list-of-questions-answers


--------
 Tabled
--------

Reworking Explicit Markup (Round 2)
===================================

See `Reworking Explicit Markup (Round 1)`_ for an earlier discussion.

In April 2004, a new thread becan on docutils-develop: `Inconsistency
in RST markup`__.  Several arguments were made; the first argument
begat later arguments.  Below, the arguments are paraphrased "in
quotes", with responses.

__ https://thread.gmane.org/gmane.text.docutils.devel/1386

1. References and targets take this form::

       targetname_

       .. _targetname: stuff

   But footnotes, "which generate links just like targets do", are
   written as::

       [1]_

       .. [1] stuff

   "Footnotes should be written as"::

       [1]_

       .. _[1]: stuff

   But they're not the same type of animal.  That's not a "footnote
   target", it's a *footnote*.  Being a target is not a footnote's
   primary purpose (an arguable point).  It just happens to grow a
   target automatically, for convenience.  Just as a section title::

       Title
       =====

   isn't a "title target", it's a *title*, which happens to grow a
   target automatically.  The consistency is there, it's just deeper
   than at first glance.

   Also, ".. [1]" was chosen for footnote syntax because it closely
   resembles one form of actual footnote rendering.  ".. _[1]:" is too
   verbose; excessive punctuation is required to get the job done.

   For more of the reasoning behind the syntax, see `Problems With
   StructuredText (Hyperlinks) `__ and
   `Reworking Footnotes`_.

2. "I expect directives to also look like ``.. this:`` [one colon]
   because that also closely parallels the link and footnote target
   markup."

   There are good reasons for the two-colon syntax:

       Two colons are used after the directive type for these reasons:

       - Two colons are distinctive, and unlikely to be used in common
         text.

       - Two colons avoids clashes with common comment text like::

             .. Danger: modify at your own risk!

       - If an implementation of reStructuredText does not recognize a
         directive (i.e., the directive-handler is not installed), a
         level-3 (error) system message is generated, and the entire
         directive block (including the directive itself) will be
         included as a literal block.  Thus "::" is a natural choice.

       -- `restructuredtext.html#directives
       <../../ref/rst/restructuredtext.html#directives>`__

   The last reason is not particularly compelling; it's more of a
   convenient coincidence or mnemonic.

3. "Comments always seemed too easy.  I almost never write comments.
   I'd have no problem writing '.. comment:' in front of my comments.
   In fact, it would probably be more readable, as comments *should*
   be set off strongly, because they are very different from normal
   text."

   Many people do use comments though, and some applications of
   reStructuredText require it.  For example, all reStructuredText
   PEPs (and this document!) have an Emacs stanza at the bottom, in a
   comment.  Having to write ".. comment::" would be very obtrusive.

   Comments *should* be dirt-easy to do.  It should be easy to
   "comment out" a block of text.  Comments in programming languages
   and other markup languages are invariably easy.

   Any author is welcome to preface their comments with "Comment:" or
   "Do Not Print" or "Note to Editor" or anything they like.  A
   "comment" directive could easily be implemented.  It might be
   confused with admonition directives, like "note" and "caution"
   though.  In unrelated (and unpublished and unfinished) work, adding
   a "comment" directive as a true document element was considered::

       If structure is necessary, we could use a "comment" directive
       (to avoid nonsensical DTD changes, the "comment" directive
       could produce an untitled topic element).

4. "One of the goals of reStructuredText is to be *readable* by people
   who don't know it.  This construction violates that: it is not at
   all obvious to the uninitiated that text marked by '..' is a
   comment.  On the other hand, '.. comment:' would be totally
   transparent."

   Totally transparent, perhaps, but also very obtrusive.  Another of
   `reStructuredText's goals`_ is to be unobtrusive, and
   ".. comment::" would violate that.  The goals of reStructuredText
   are many, and they conflict.  Determining the right set of goals
   and finding solutions that best fit is done on a case-by-case
   basis.

   Even readability is has two aspects.  Being readable without any
   prior knowledge is one.  Being as easily read in raw form as in
   processed form is the other.  ".." may not contribute to the former
   aspect, but ".. comment::" would certainly detract from the latter.

   .. _author's note:
   .. _reStructuredText's goals: ../../ref/rst/introduction.html#goals

5. "Recently I sent someone an rst document, and they got confused; I
   had to explain to them that '..' marks comments, *unless* it's a
   directive, etc..."

   The explanation of directives *is* roundabout, defining comments in
   terms of not being other things.  That's definitely a wart.

6. "Under the current system, a mistyped directive (with ':' instead
   of '::') will be silently ignored.  This is an error that could
   easily go unnoticed."

   A parser option/setting like "--comments-on-stderr" would help.

7. "I'd prefer to see double-dot-space / command / double-colon as the
   standard Docutils markup-marker.  It's unusual enough to avoid
   being accidentally used.  Everything that starts with a double-dot
   should end with a double-colon."

   That would increase the punctuation verbosity of some constructs
   considerably.

8. Edward Loper proposed the following plan for backwards
   compatibility:

       1. ".. foo" will generate a deprecation warning to stderr, and
          nothing in the output (no system messages).
       2. ".. foo: bar" will be treated as a directive foo.  If there
          is no foo directive, then do the normal error output.
       3. ".. foo:: bar" will generate a deprecation warning to
          stderr, and be treated as a directive.  Or leave it valid?

       So some existing documents might start printing deprecation
       warnings, but the only existing documents that would *break*
       would be ones that say something like::

           .. warning: this should be a comment

       instead of::

           .. warning:: this should be a comment

       Here, we're trading fairly common a silent error (directive
       falsely treated as a comment) for a fairly uncommon explicitly
       flagged error (comment falsely treated as directive).  To make
       things even easier, we could add a sentence to the
       unknown-directive error.  Something like "If you intended to
       create a comment, please use '.. comment:' instead".

On one hand, I understand and sympathize with the points raised.  On
the other hand, I think the current syntax strikes the right balance
(but I acknowledge a possible lack of objectivity).  On the gripping
hand, the comment and directive syntax has become well established, so
even if it's a wart, it may be a wart we have to live with.

Making any of these changes would cause a lot of breakage or at least
deprecation warnings.  I'm not sure the benefit is worth the cost.

For now, we'll treat this as an unresolved legacy issue.


-------
 To Do
-------

Nested Inline Markup
====================

These are collected notes on a long-discussed issue.  The original
mailing list messages should be referred to for details.

* In a 2001-10-31 discussion I wrote:

      Try, for example, `Ed Loper's 2001-03-21 post`_, which details
      some rules for nested inline markup. I think the complexity is
      prohibitive for the marginal benefit. (And if you can understand
      that tree without going mad, you're a better man than I. ;-)

      Inline markup is already fragile. Allowing nested inline markup
      would only be asking for trouble IMHO. If it proves absolutely
      necessary, it can be added later. The rules for what can appear
      inside what must be well thought out first though.

      .. _Ed Loper's 2001-03-21 post:
         https://mail.python.org/pipermail/doc-sig/2001-March/001487.html

      -- https://mail.python.org/pipermail/doc-sig/2001-October/002354.html

* In a 2001-11-09 Doc-SIG post, I wrote:

      The problem is that in the
      what-you-see-is-more-or-less-what-you-get markup language that
      is reStructuredText, the symbols used for inline markup ("*",
      "**", "`", "``", etc.) may preclude nesting.

  I've rethought this position.  Nested markup is not precluded, just
  tricky.  People and software parse "double and 'single' quotes" all
  the time.  Continuing,

      I've thought over how we might implement nested inline
      markup. The first algorithm ("first identify the outer inline
      markup as we do now, then recursively scan for nested inline
      markup") won't work; counterexamples were given in my `last post
      <https://mail.python.org/pipermail/doc-sig/2001-November/002363.html>`__.

      The second algorithm makes my head hurt::

          while 1:
              scan for start-string
              if found:
                  push on stack
                  scan for start or end string
                  if new start string found:
                      recurse
                  elif matching end string found:
                      pop stack
                  elif non-matching end string found:
                      if its a markup error:
                          generate warning
                      elif the initial start-string was misinterpreted:
                          # e.g. in this case: ***strong** in emphasis*
                          restart with the other interpretation
                          # but it might be several layers back ...
              ...

      This is similar to how the parser does section title
      recognition, but sections are much more regular and
      deterministic.

      Bottom line is, I don't think the benefits are worth the effort,
      even if it is possible. I'm not going to try to write the code,
      at least not now. If somebody codes up a consistent, working,
      general solution, I'll be happy to consider it.

      -- https://mail.python.org/pipermail/doc-sig/2001-November/002388.html

* In a `2003-05-06 Docutils-Users post`__ Paul Tremblay proposed a new
  syntax to allow for easier nesting.  It eventually evolved into
  this::

      :role:[inline text]

  The duplication with the existing interpreted text syntax is
  problematic though.

  __ https://article.gmane.org/gmane.text.docutils.user/317

* Could the parser be extended to parse nested interpreted text? ::

      :emphasis:`Some emphasized text with :strong:`some more
      emphasized text` in it and **perhaps** :reference:`a link``

* In a `2003-06-18 Docutils-Develop post`__, Mark Nodine reported on
  his implementation of a form of nested inline markup in his
  Perl-based parser (unpublished).  He brought up some interesting
  ideas.  The implementation was flawed, however, by the change in
  semantics required for backslash escapes.

  __ https://article.gmane.org/gmane.text.docutils.devel/795

* Docutils-develop threads between David Abrahams, David Goodger, and
  Mark Nodine (beginning 2004-01-16__ and 2004-01-19__) hashed out
  many of the details of a potentially successful implementation, as
  described below.  David Abrahams checked in code to the "nesting"
  branch of CVS, awaiting thorough review.

  __ https://thread.gmane.org/gmane.text.docutils.devel/1102
  __ https://thread.gmane.org/gmane.text.docutils.devel/1125

It may be possible to accomplish nested inline markup in general with
a more powerful inline markup parser.  There may be some issues, but
I'm not averse to the idea of nested inline markup in general.  I just
don't have the time or inclination to write a new parser now.  Of
course, a good patch would be welcome!

I envisage something like this.  Explicit-role interpreted text must
be nestable.  Prefix-based is probably preferred, since suffix-based
will look like inline literals::

    ``text`:role1:`:role2:

But it can be disambiguated, so it ought to be left up to the author::

    `\ `text`:role1:`:role2:

In addition, other forms of inline markup may be nested if
unambiguous::

    *emphasized ``literal`` and |substitution ref| and link_*

IOW, the parser ought to be as permissive as possible.


Index Entries & Indexes
=======================

Were I writing a book with an index, I guess I'd need two
different kinds of index targets: inline/implicit and
out-of-line/explicit.  For example::

    In this `paragraph`:index:, several words are being
    `marked`:index: inline as implicit `index`:index:
    entries.

    .. index:: markup
    .. index:: syntax

    The explicit index directives above would refer to
    this paragraph.  It might also make sense to allow multiple
    entries in an ``index`` directive:

    .. index::
        markup
        syntax

The words "paragraph", "marked", and "index" would become index
entries pointing at the words in the first paragraph.  The index
entry words appear verbatim in the text.  (Don't worry about the
ugly ":index:" part; if indexing is the only/main application of
interpreted text in your documents, it can be implicit and
omitted.)  The two directives provide manual indexing, where the
index entry words ("markup" and "syntax") do not appear in the
main text.  We could combine the two directives into one::

    .. index:: markup; syntax

Semicolons instead of commas because commas could *be* part of the
index target, like::

    .. index:: van Rossum, Guido

Another reason for index directives is because other inline markup
wouldn't be possible within inline index targets.

Sometimes index entries have multiple levels.  Given::

    .. index:: statement syntax: expression statements

In a hypothetical index, combined with other entries, it might
look like this::

    statement syntax
        expression statements ..... 56
        assignment ................ 57
        simple statements ......... 58
        compound statements ....... 60

Inline multi-level index targets could be done too.  Perhaps
something like::

    When dealing with `expression statements <statement syntax:>`,
    we must remember ...

The opposite sense could also be possible::

    When dealing with `index entries <:multi-level>`, there are
    many permutations to consider.

Also "see / see also" index entries.

Given::

    Here's a paragraph.

    .. index:: paragraph

(The "index" directive above actually targets the *preceding*
object.)  The directive should produce something like this XML::

    
    
    Here's a paragraph.
    

This kind of content model would also allow true inline
index-entries::

    Here's a `paragraph`:index:.

If the "index" role were the default for the application, it could be
dropped::

    Here's a `paragraph`.

Both of these would result in this XML::

    
    Here's a paragraph.
    


from 2002-06-24 docutils-develop posts
--------------------------------------

    If all of your index entries will appear verbatim in the text,
    this should be sufficient.  If not (e.g., if you want "Van Rossum,
    Guido" in the index but "Guido van Rossum" in the text), we'll
    have to figure out a supplemental mechanism, perhaps using
    substitutions.

I've thought a bit more on this, and I came up with two possibilities:

1. Using interpreted text, embed the index entry text within the
   interpreted text::

       ... by `Guido van Rossum [Van Rossum, Guido]` ...

   The problem with this is obvious: the text becomes cluttered and
   hard to read.  The processed output would drop the text in
   brackets, which goes against the spirit of interpreted text.

2. Use substitutions::

       ... by |Guido van Rossum| ...

       .. |Guido van Rossum| index:: Van Rossum, Guido

   A problem with this is that each substitution definition must have
   a unique name.  A subsequent ``.. |Guido van Rossum| index:: BDFL``
   would be illegal.  Some kind of anonymous substitution definition
   mechanism would be required, but I think that's going too far.

Both of these alternatives are flawed.  Any other ideas?


-------------------
 ... Or Not To Do?
-------------------

This is the realm of the possible but questionably probable.  These
ideas are kept here as a record of what has been proposed, for
posterity and in case any of them prove to be useful.


Compound Enumerated Lists
=========================

Allow for compound enumerators, such as "1.1." or "1.a." or "1(a)", to
allow for nested enumerated lists without indentation?


Indented Lists
==============

Allow for variant styles by interpreting indented lists as if they
weren't indented?  For example, currently the list below will be
parsed as a list within a block quote::

    paragraph

      * list item 1
      * list item 2

But a lot of people seem to write that way, and HTML browsers make it
look as if that's the way it should be.  The parser could check the
contents of block quotes, and if they contain only a single list,
remove the block quote wrapper.  There would be two problems:

1. What if we actually *do* want a list inside a block quote?

2. What if such a list comes immediately after an indented construct,
   such as a literal block?

Both could be solved using empty comments (problem 2 already exists
for a block quote after a literal block).  But that's a hack.

Perhaps a runtime setting, allowing or disabling this convenience,
would be appropriate.  But that raises issues too:

    User A, who writes lists indented (and their config file is set up
    to allow it), sends a file to user B, who doesn't (and their
    config file disables indented lists).  The result of processing by
    the two users will be different.

It may seem minor, but it adds ambiguity to the parser, which is bad.

See the `Doc-SIG discussion starting 2001-04-18`__ with Ed Loper's
"Structuring: a summary; and an attempt at EBNF", item 4 (and
follow-ups, here__ and here__).  Also `docutils-users, 2003-02-17`__
and `beginning 2003-08-04`__.

__ https://mail.python.org/pipermail/doc-sig/2001-April/001776.html
__ https://mail.python.org/pipermail/doc-sig/2001-April/001789.html
__ https://mail.python.org/pipermail/doc-sig/2001-April/001793.html
__ https://sourceforge.net/mailarchive/message.php?msg_id=3838913
__ https://sf.net/mailarchive/forum.php?thread_id=2957175&forum_id=11444


Sloppy Indentation of List Items
================================

Perhaps the indentation shouldn't be so strict.  Currently, this is
required::

    1. First line,
       second line.

Anything wrong with this? ::

    1. First line,
     second line.

Problem? ::

    1. First para.

       Block quote.  (no good: requires some indent relative to first
       para)

     Second Para.

    2. Have to carefully define where the literal block ends::

         Literal block

       Literal block?

Hmm...  Non-strict indentation isn't such a good idea.


Lazy Indentation of List Items
==============================

Another approach: Going back to the first draft of reStructuredText
(2000-11-27 post to Doc-SIG)::

    - This is the fourth item of the main list (no blank line above).
    The second line of this item is not indented relative to the
    bullet, which precludes it from having a second paragraph.

Change that to *require* a blank line above and below, to reduce
ambiguity.  This "loosening" may be added later, once the parser's
been nailed down.  However, a serious drawback of this approach is to
limit the content of each list item to a single paragraph.


David's Idea for Lazy Indentation
---------------------------------

Consider a paragraph in a word processor.  It is a single logical line
of text which ends with a newline, soft-wrapped arbitrarily at the
right edge of the page or screen.  We can think of a plaintext
paragraph in the same way, as a single logical line of text, ending
with two newlines (a blank line) instead of one, and which may contain
arbitrary line breaks (newlines) where it was accidentally
hard-wrapped by an application.  We can compensate for the accidental
hard-wrapping by "unwrapping" every unindented second and subsequent
line.  The indentation of the first line of a paragraph or list item
would determine the indentation for the entire element.  Blank lines
would be required between list items when using lazy indentation.

The following example shows the lazy indentation of multiple body
elements::

    - This is the first paragraph
    of the first list item.

      Here is the second paragraph
    of the first list item.

    - This is the first paragraph
    of the second list item.

      Here is the second paragraph
    of the second list item.

A more complex example shows the limitations of lazy indentation::

    - This is the first paragraph
    of the first list item.

      Next is a definition list item:

      Term
          Definition.  The indentation of the term is
    required, as is the indentation of the definition's
    first line.

          When the definition extends to more than
    one line, lazy indentation may occur.  (This is the second
    paragraph of the definition.)

    - This is the first paragraph
    of the second list item.

      - Here is the first paragraph of
    the first item of a nested list.

      So this paragraph would be outside of the nested list,
    but inside the second list item of the outer list.

    But this paragraph is not part of the list at all.

And the ambiguity remains::

    - Look at the hyphen at the beginning of the next line
    - is it a second list item marker, or a dash in the text?

    Similarly, we may want to refer to numbers inside enumerated
    lists:

    1. How many socks in a pair? There are
    2. How many pants in a pair? Exactly
    1. Go figure.

Literal blocks and block quotes would still require consistent
indentation for all their lines.  For block quotes, we might be able
to get away with only requiring that the first line of each contained
element be indented.  For example::

    Here's a paragraph.

        This is a paragraph inside a block quote.
    Second and subsequent lines need not be indented at all.

        - A bullet list inside
    the block quote.

          Second paragraph of the
    bullet list inside the block quote.

Although feasible, this form of lazy indentation has problems.  The
document structure and hierarchy is not obvious from the indentation,
making the source plaintext difficult to read.  This will also make
keeping track of the indentation while writing difficult and
error-prone.  However, these problems may be acceptable for Wikis and
email mode, where we may be able to rely on less complex structure
(few nested lists, for example).


Multiple Roles in Interpreted Text
==================================

In reStructuredText, inline markup cannot be nested (yet; `see
above`__).  This also applies to interpreted text.  In order to
simultaneously combine multiple roles for a single piece of text, a
syntax extension would be necessary.  Ideas:

1. Initial idea::

       `interpreted text`:role1,role2:

2. Suggested by Jason Diamond::

       `interpreted text`:role1:role2:

If a document is so complex as to require nested inline markup,
perhaps another markup system should be considered.  By design,
reStructuredText does not have the flexibility of XML.

__ `Nested Inline Markup`_


Parameterized Interpreted Text
==============================

In some cases it may be expedient to pass parameters to interpreted
text, analogous to function calls.  Ideas:

1. Parameterize the interpreted text role itself (suggested by Jason
   Diamond)::

       `interpreted text`:role1(foo=bar):

   Positional parameters could also be supported::

       `CSS`:acronym(Cascading Style Sheets): is used for HTML, and
       `CSS`:acronym(Content Scrambling System): is used for DVDs.

   Technical problem: current interpreted text syntax does not
   recognize roles containing whitespace.  Design problem: this smells
   like programming language syntax, but reStructuredText is not a
   programming language.

2. Put the parameters inside the interpreted text::

       `CSS (Cascading Style Sheets)`:acronym: is used for HTML, and
       `CSS (Content Scrambling System)`:acronym: is used for DVDs.

   Although this could be defined on an individual basis (per role),
   we ought to have a standard.  Hyperlinks with embedded URIs already
   use angle brackets; perhaps they could be used here too::

       `CSS <Cascading Style Sheets>`:acronym: is used for HTML, and
       `CSS <Content Scrambling System>`:acronym: is used for DVDs.

   Do angle brackets connote URLs too much for this to be acceptable?
   How about the "tag" connotation -- does it save them or doom them?

3. `Nested inline markup`_ could prove useful here::

       `CSS :def:`Cascading Style Sheets``:acronym: is used for HTML,
       and `CSS :def:`Content Scrambling System``:acronym: is used for
       DVDs.

   Inline markup roles could even define the default roles of nested
   inline markup, allowing this cleaner syntax::

       `CSS `Cascading Style Sheets``:acronym: is used for HTML, and
       `CSS `Content Scrambling System``:acronym: is used for DVDs.

Does this push inline markup too far?  Readability becomes a serious
issue.  Substitutions may provide a better alternative (at the expense
of verbosity and duplication) by pulling the details out of the text
flow::

    |CSS| is used for HTML, and |CSS-DVD| is used for DVDs.

    .. |CSS| acronym:: Cascading Style Sheets
    .. |CSS-DVD| acronym:: Content Scrambling System
       :text: CSS

----------------------------------------------------------------------

This whole idea may be going beyond the scope of reStructuredText.
Documents requiring this functionality may be better off using XML or
another markup system.

This argument comes up regularly when pushing the envelope of
reStructuredText syntax.  I think it's a useful argument in that it
provides a check on creeping featurism.  In many cases, the resulting
verbosity produces such unreadable plaintext that there's a natural
desire *not* to use it unless absolutely necessary.  It's a matter of
finding the right balance.


Syntax for Interpreted Text Role Bindings
=========================================

The following syntax (idea from Jeffrey C. Jacobs) could be used to
associate directives with roles::

    .. :rewrite: class:: rewrite

    `She wore ribbons in her hair and it lay with streaks of
    grey`:rewrite:

The syntax is similar to that of substitution declarations, and the
directive/role association may resolve implementation issues.  The
semantics, ramifications, and implementation details would need to be
worked out.

The example above would implement the "rewrite" role as adding a
``class="rewrite"`` attribute to the interpreted text ("inline"
element).  The stylesheet would then pick up on the "class" attribute
to do the actual formatting.

The advantage of the new syntax would be flexibility.  Uses other than
"class" may present themselves.  The disadvantage is complexity:
having to implement new syntax for a relatively specialized operation,
and having new semantics in existing directives ("class::" would do
something different).

The `"role" directive`__ has been implemented.

__ ../../ref/rst/directives.html#role


Character Processing
====================

Several people have suggested adding some form of character processing
to reStructuredText:

* Some sort of automated replacement of ASCII sequences:

  - ``--`` to em-dash (or ``--`` to en-dash, and ``---`` to em-dash).
  - Convert quotes to curly quote entities.  (Essentially impossible
    for HTML?  Unnecessary for TeX.)
  - Various forms of ``:-)`` to smiley icons.
  - ``"\ "`` to &nbsp;.  Problem with line-wrapping though: it could
    end up escaping the newline.
  - Escaped newlines to .
  - Escaped period or quote or dash as a disappearing catalyst to
    allow character-level inline markup?

* XML-style character entities, such as "&copy;" for the copyright
  symbol.

Docutils has no need of a character entity subsystem.  Supporting
Unicode and text encodings, character entities should be directly
represented in the text: a copyright symbol should be represented by
the copyright symbol character.  If this is not possible in an
authoring environment, a pre-processing stage can be added, or a table
of substitution definitions can be devised.

A "unicode" directive has been implemented to allow direct
specification of esoteric characters.  In combination with the
substitution construct, "include" files defining common sets of
character entities can be defined and used.  `A set of character
entity set definition files have been defined`__ (`tarball`__).
There's also `a description and instructions for use`__.

__ https://docutils.sourceforge.io/tmp/charents/
__ https://docutils.sourceforge.io/tmp/charents.tgz
__ https://docutils.sourceforge.io/tmp/charents/README.html

To allow for `character-level inline markup`_, a limited form of
character processing has been added to the spec and parser: escaped
whitespace characters are removed from the processed document.  Any
further character processing will be of this functional type, rather
than of the character-encoding type.

.. _character-level inline markup:
   ../../ref/rst/restructuredtext.html#character-level-inline-markup

* Directive idea::

      .. text-replace:: "pattern" "replacement"

  - Support Unicode "U+XXXX" codes.
  - Support regexps, perhaps with alternative "regexp-replace"
    directive.
  - Flags for regexps; ":flags:" option, or individuals.
  - Specifically, should the default be case-sensistive or
    -insensitive?


Page Or Line Breaks
===================

* Should ^L (or something else in reST) be defined to mean
  force/suggest page breaks in whatever output we have?

  A "break" or "page-break" directive would be easy to add.  A new
  doctree element would be required though (perhaps "break").  The
  final behavior would be up to the Writer.  The directive argument
  could be one of page/column/recto/verso for added flexibility.

  Currently ^L (Python's ``\f``) characters are treated as whitespace.
  They're converted to single spaces, actually, as are vertical tabs
  (^K, Python's ``\v``).  It would be possible to recognize form feeds
  as markup, but it requires some thought and discussion first.  Are
  there any downsides?  Many editing environments do not allow the
  insertion of control characters.  Will it cause any harm?  It would
  be useful as a shorthand for the directive.

  It's common practice to use ^L before Emacs "Local Variables"
  lists::

      ^L
      ..
         Local Variables:
         mode: indented-text
         indent-tabs-mode: nil
         sentence-end-double-space: t
         fill-column: 70
         End:

  These are already present in many PEPs and Docutils project
  documents.  From the Emacs manual (info):

      A "local variables list" goes near the end of the file, in the
      last page.  (It is often best to put it on a page by itself.)

  It would be unfortunate if this construct caused a final blank page
  to be generated (for those Writers that recognize the page breaks).
  We'll have to add a transform that looks for a "break" plus zero or
  more comments at the end of a document, and removes them.

  Probably a bad idea because there is no such thing as a page in a
  generic document format.

* Could the "break" concept above be extended to inline forms?
  E.g. "^L" in the middle of a sentence could cause a line break.
  Only recognize it at the end of a line (i.e., ``\f\n``)?

  Or is formfeed inappropriate?  Perhaps vertical tab (``\v``), but
  even that's a stretch.  Can't use carriage returns, since they're
  commonly used for line endings.

  Probably a bad idea as well because we do not want to use control
  characters for well-readable and well-writable markup, and after all
  we have the line block syntax for line breaks.


Superscript Markup
==================

Add ``^superscript^`` inline markup?  The only common non-markup uses
of "^" I can think of are as short hand for "superscript" itself and
for describing control characters ("^C to cancel").  The former
supports the proposed syntax, and it could be argued that the latter
ought to be literal text anyhow (e.g. "``^C`` to cancel").

However, superscripts are seldom ne