## File: docs/building.rst .. include:: header.rst .. contents:: Table of contents :depth: 2 :backlinks: none downloading and building ------------------------ To download the latest version of libtorrent, clone the `github repository`__. __ https://github.com/arvidn/libtorrent The build systems supported "out of the box" in libtorrent are boost-build cmake. If you still can't build after following these instructions, you can usually get help in the ``#libtorrent`` IRC channel on ``irc.freenode.net``. .. warning:: A common mistake when building and linking against libtorrent is to build with one set of configuration options (#defines) and link against it using a different set of configuration options. Since libtorrent has some code in header files, that code will not be compatible with the built library if they see different configurations. Always make sure that the same TORRENT_* and BOOST_* macros are defined when you link against libtorrent as when you build it. The simplest way to see the full list of macros defined is to build libtorrent with ``-n -a`` switches added to ``b2`` command line, which output all compiler switches. Boost-build supports propagating configuration options to dependencies. building from git ----------------- To build libtorrent from git you need to clone the libtorrent repository from github. Note that the git repository depends on other git repositories via submodules, which also need to be initialized and updated. If you downloaded a release `tarball`__, you can skip this section. __ https://github.com/arvidn/libtorrent/releases/latest :: git clone --recurse-submodules https://github.com/arvidn/libtorrent.git building with boost build ------------------------- The primary reason to use boost-build is that it will automatically build the dependent boost libraries with the correct compiler settings, in order to ensure that the build targets are link compatible (see `boost guidelines`__ for some details on this issue). __ https://boost.org/more/separate_compilation.html Since BBv2 will build the boost libraries for you, you need the full boost source package. Having boost installed via some package system is usually not enough (and even if it is enough, the necessary environment variables are usually not set by the package installer). If you want to build against an installed copy of boost, you can skip directly to step 3 (assuming you also have boost build installed). build commands ~~~~~~~~~~~~~~ Linux:: sudo apt install libboost-tools-dev libboost-dev libboost-system-dev echo "using gcc ;" >>~/user-config.jam b2 crypto=openssl release Mac OS:: brew install boost-build boost openssl@3 echo "using darwin ;" >>~/user-config.jam b2 crypto=openssl release Windows (assuming the boost package is saved to ``C:\boost_1_69_0``):: set BOOST_ROOT=c:\boost_1_69_0 set BOOST_BUILD_PATH=%BOOST_ROOT%\tools\build (cd %BOOST_ROOT% && .\bootstrap.bat) echo using msvc ; >>%HOMEDRIVE%%HOMEPATH%\user-config.jam %BOOST_ROOT%\b2.exe --hash release docker file ~~~~~~~~~~~ A Docker file is available that's used to build and run the fuzzers, at OSS-Fuzz__. __ https://github.com/google/oss-fuzz/tree/master/projects/libtorrent Step 1: Download boost ~~~~~~~~~~~~~~~~~~~~~~ If you want to build against boost installed on your system, you can skip this strep. Just make sure to have `BOOST_ROOT` unset for the `b2` invocation. You'll find boost here__. __ https://www.boost.org/users/download/#live Extract the archive to some directory where you want it. For the sake of this guide, let's assume you extract the package to ``c:\boost_1_69_0``. You'll need at least version 1.67 of the boost library in order to build libtorrent. Step 2: Setup BBv2 ~~~~~~~~~~~~~~~~~~ If you have installed ``boost-build`` via a package manager, you can skip this step. If not, you need to build boost build from the boost source package. First you need to build ``b2``. You do this by opening a terminal (In windows, run ``cmd``). Change directory to ``c:\boost_1_69_0\tools\build``. Then run the script called ``bootstrap.bat`` or ``bootstrap.sh`` on a Unix system. This will build ``b2`` and place it in a directory ``src/engine/bin.``. Copy the ``b2.exe`` (or ``b2`` on a Unix system) to a place that's in you shell's ``PATH``. On Linux systems a place commonly used may be ``/usr/local/bin`` or on Windows ``c:\windows`` (you can also add directories to the search paths by modifying the environment variable called ``PATH``). Now you have ``b2`` installed. ``b2`` can be considered an interpreter that the boost-build system is implemented on. So boost-build uses ``b2``. So, to complete the installation you need to make two more things. You need to set the environment variable ``BOOST_BUILD_PATH``. This is the path that tells ``b2`` where it can find boost-build, your configuration file and all the toolsets (descriptions used by boost-build to know how to use different compilers on different platforms). Assuming the boost install path above, set it to ``c:\boost_1_69_0\tools\build``. To set an environment variable in windows, type for example:: set BOOST_BUILD_PATH=c:\boost_1_69_0\tools\build\v2 In a terminal window. The last thing to do is to configure which compiler(s) to use. Create a file ``user-config.jam`` in your home directory. Depending on your platform and which compiler you're using, you should add a line for each compiler and compiler version you have installed on your system that you want to be able to use with BBv2. For example, if you're using Microsoft Visual Studio 14.2 (2019), just add a line:: using msvc : 14.2 ; If you use GCC, add the line:: using gcc ; If you have more than one version of GCC installed, you can add the command line used to invoke g++ after the version number, like this:: using gcc : 6.0 : g++-6 ; using gcc : 7.0 : g++-7 ; Another toolset worth mentioning is the ``darwin`` toolset (for macOS). From Tiger (10.4) macOS comes with both GCC 3.3 and GCC 4.0. Then you can use the following toolsets:: using darwin : 3.3 : g++-3.3 ; using darwin : 4.0 : g++-4.0 ; Note that the spaces around the semi-colons and colons are important! Also see the `boost-build documentation`_. .. _`boost-build documentation`: https://www.bfgroup.xyz/b2/manual/main/index.html Step 3: Building libtorrent ~~~~~~~~~~~~~~~~~~~~~~~~~~~ When building libtorrent, boost is either picked up from system installed locations or from a boost source package, if the ``BOOST_ROOT`` environment variable is set pointing to one. If you're building boost from source, set ``BOOST_ROOT`` to your boost directory, e.g. ``c:\boost_1_69_0``. Then the only thing left is simply to invoke ``b2``. If you want to specify a specific toolset to use (compiler) you can just add that to the command line. For example:: b2 msvc-14.2 b2 gcc-7.0 b2 darwin-4.0 .. note:: If the environment variable ``BOOST_ROOT`` is not set, the Jamfile will attempt to link against "installed" boost libraries. i.e. assume the headers and libraries are available in default search paths. In this case it's critical that you build your project with the same version of C++ and the same build flags as the system libraries were built with. .. note:: Also see the `Visual Studio versions`_. .. _`Visual Studio versions`: https://en.wikipedia.org/wiki/Microsoft_Visual_C%2B%2B#Internal_version_numbering To build different versions you can also just add the name of the build variant. Some default build variants in BBv2 are ``release``, ``debug``, ``profile``. You can build libtorrent as a DLL too, by typing ``link=shared``, or ``link=static`` to build a static library. If you want to explicitly say how to link against the runtime library, you can set the ``runtime-link`` feature on the command line, either to ``shared`` or ``static``. Most operating systems will only allow linking shared against the runtime, but on windows you can do both. Example:: b2 msvc-14.2 variant=release link=static runtime-link=static debug-symbols=on .. note:: When building on windows, the path boost-build puts targets in may be too long. If you get an error message like: "The input line is long", try to pass --hash on the ``b2`` command line. .. warning:: If you link statically to the runtime library, you cannot build libtorrent as a shared library (DLL), since you will get separate heaps in the library and in the client application. It will result in crashes and possibly link errors. .. note:: When building on Solaris, you may have to specify ``stdlib=sun-stlport`` on the b2 command line. The build targets are put in a directory called bin, and under it they are sorted in directories depending on the toolset and build variant used. To build the examples, just change directory to the examples directory and invoke ``b2`` from there. To build and run the tests, go to the test directory and run ``b2``. Note that if you're building on windows using the ``msvc`` toolset, you cannot run it from a cygwin terminal, you'll have to run it from a ``cmd`` terminal. The same goes for cygwin, if you're building with gcc in cygwin you'll have to run it from a cygwin terminal. Also, make sure the paths are correct in the different environments. In cygwin, the paths (``BOOST_BUILD_PATH`` and ``BOOST_ROOT``) should be in the typical Unix-format (e.g. ``/cygdrive/c/boost_1_69_0``). In the windows environment, they should have the typical windows format (``c:/boost_1_69_0``). .. note:: In Jamfiles, spaces are separators. It's typically easiest to avoid spaces in path names. If you want spaces in your paths, make sure to quote them with double quotes ("). The ``Jamfile`` will define ``NDEBUG`` when it's building a release build. For more build configuration flags see `Build configurations`_. Jamfile will look in some default directory for the openssl headers and libraries. On macOS, it will look for the homebrew openssl package. On Windows, it will look in ``C:\OpenSSL-Win32``, or ``C:\OpenSSL-Win64`` if compiling in 64-bit. To customize the library path and include path for openssl, set the features ``openssl-lib`` and ``openssl-include`` respectively. The option to link with wolfSSL (by setting the ``crypto`` feature to ``wolfssl``), requires a custom build of wolfSSL using the following options: ``--enable-asio --enable-sni --enable-nginx``. To customize the library path and include path for wolfSSL, set the features ``wolfssl-lib`` and ``wolfssl-include`` respectively. To disable linking against any SSL library, set the ``crypto`` build feature to ``built-in``. This will use an embedded version if SHA-1. Build features ~~~~~~~~~~~~~~ +--------------------------+----------------------------------------------------+ | boost build feature | values | +==========================+====================================================+ | ``cxxstd`` | The version of C++ to use, e.g. ``11``, ``14``, | | | ``17``, ``20``. The C++ version *may* affect the | | | libtorrent ABI (the ambition is to avoid that). | +--------------------------+----------------------------------------------------+ | ``boost-link`` | * ``static`` - links statically against the boost | | | libraries. | | | * ``shared`` - links dynamically against the boost | | | libraries. | +--------------------------+----------------------------------------------------+ | ``openssl-lib`` | can be used to specify the directory where libssl | | | and libcrypto are installed (or the windows | | | counterparts). | +--------------------------+----------------------------------------------------+ | ``openssl-include`` | can be used to specify the include directory where | | | the openssl headers are installed. | +--------------------------+----------------------------------------------------+ | ``logging`` | * ``off`` - logging alerts disabled. The | | | reason to disable logging is to keep the binary | | | size low where that matters. | | | * ``on`` - default. logging alerts available, | | | still need to be enabled by the alert mask. | +--------------------------+----------------------------------------------------+ | ``lto`` | * ``on`` - enables link time optimization, also | | | known as whole program optimization. | +--------------------------+----------------------------------------------------+ | ``alert-msg`` | * ``on`` - (default) return human readable | | | messages from the ``alert::message()`` call. | | | * ``off`` - Always return empty strings from | | | ``alert::message()``, and save binary size. | +--------------------------+----------------------------------------------------+ | ``dht`` | * ``on`` - build with DHT support | | | * ``off`` - build without DHT support. | +--------------------------+----------------------------------------------------+ | ``asserts`` | * ``auto`` - asserts are on if in debug mode | | | * ``on`` - asserts are on, even in release mode | | | * ``off`` - asserts are disabled | | | * ``production`` - assertion failures are logged | | | to ``asserts.log`` in the current working | | | directory, but won't abort the process. | | | The file they are logged to can be customized | | | by setting the global pointer ``extern char | | | const* libtorrent_assert_log`` to a different | | | filename. | | | * ``system`` use the libc assert macro | +--------------------------+----------------------------------------------------+ | ``encryption`` | * ``on`` - encrypted bittorrent connections | | | enabled. (Message Stream encryption).(default) | | | * ``off`` - turns off support for encrypted | | | connections. The shipped public domain SHA-1 | | | implementation is used. | +--------------------------+----------------------------------------------------+ | ``mutable-torrents`` | * ``on`` - mutable torrents are supported | | | (`BEP 38`_) (default). | | | * ``off`` - mutable torrents are not supported. | +--------------------------+----------------------------------------------------+ | ``crypto`` | * ``openssl`` - (default) links against openssl | | | and libcrypto to use for SHA-1 hashing. | | | This also enables HTTPS-tracker support and | | | support for bittorrent over SSL. | | | * ``built-in`` - (default) uses built-in SHA-1 | | | implementation. In macOS/iOS it uses | | | CommonCrypto SHA-1 implementation. | | | * ``wolfssl`` - links against wolfssl to use it | | | for SHA-1 hashing and HTTPS tracker support. | | | * ``libcrypto`` - links against libcrypto | | | to use the SHA-1 implementation. (no SSL support)| | | * ``gcrypt`` - links against libgcrypt | | | to use the SHA-1 implementation. (no SSL support)| +--------------------------+----------------------------------------------------+ | ``openssl-version`` | This can be used on windows to link against the | | | special OpenSSL library names used on windows | | | prior to OpenSSL 1.1. | | | | | | * ``1.1`` - link against the normal openssl | | | library name. (default) | | | * ``pre1.1`` - link against the old windows names | | | (i.e. ``ssleay32`` and ``libeay32``. | +--------------------------+----------------------------------------------------+ | ``link`` | * ``static`` - builds libtorrent as a static | | | library (.a / .lib) | | | * ``shared`` - builds libtorrent as a shared | | | library (.so / .dll). | +--------------------------+----------------------------------------------------+ | ``runtime-link`` | * ``static`` - links statically against the | | | run-time library (if available on your | | | platform). | | | * ``shared`` - link dynamically against the | | | run-time library (default). | +--------------------------+----------------------------------------------------+ | ``variant`` | * ``debug`` - builds libtorrent with debug | | | information and invariant checks. | | | * ``release`` - builds libtorrent in release mode | | | without invariant checks and with optimization. | | | * ``profile`` - builds libtorrent with profile | | | information. | +--------------------------+----------------------------------------------------+ | ``invariant-checks`` | This setting only affects debug builds (where | | | ``NDEBUG`` is not defined). It defaults to ``on``. | | | | | | * ``on`` - internal invariant checks are enabled. | | | * ``off`` - internal invariant checks are | | | disabled. The resulting executable will run | | | faster than a regular debug build. | | | * ``full`` - turns on extra expensive invariant | | | checks. | +--------------------------+----------------------------------------------------+ | ``debug-symbols`` | * ``on`` - default for debug builds. This setting | | | is useful for building release builds with | | | symbols. | | | * ``off`` - default for release builds. | +--------------------------+----------------------------------------------------+ | ``deprecated-functions`` | Enables or disables deprecated functionality by | | | specifying the API version to build. | | | Defaults to version 2, options are: | | | | | | * ``1`` libtorrent-1.1 | | | * ``2`` libtorrent-1.2 | | | * ``3`` libtorrent-2.0 | | | * ``4`` libtorrent-2.1 | | | | | | For backwards compatibility, it's also possible | | | to specify: | | | | | | * ``on`` - Includes deprecated functions from the | | | oldest supported version. | | | * ``off`` - Disables all deprecated functions. | | | Same as specifying the newest version. | +--------------------------+----------------------------------------------------+ | ``i2p`` | * ``on`` - default. build with I2P support | | | * ``off`` - build without I2P support | +--------------------------+----------------------------------------------------+ | ``profile-calls`` | * ``off`` - default. No additional call profiling. | | | * ``on`` - Enable logging of stack traces of | | | calls into libtorrent that are blocking. On | | | session shutdown, a file ``blocking_calls.txt`` | | | is written with stack traces of blocking calls | | | ordered by the number of them. | +--------------------------+----------------------------------------------------+ | ``utp-log`` | * ``off`` - default. Do not print verbose uTP | | | log. | | | * ``on`` - Print verbose uTP log, used to debug | | | the uTP implementation. | +--------------------------+----------------------------------------------------+ | ``picker-debugging`` | * ``off`` - default. no extra invariant checks in | | | piece picker. | | | * ``on`` - include additional invariant checks in | | | piece picker. Used for testing the piece picker. | +--------------------------+----------------------------------------------------+ | ``extensions`` | * ``on`` - enable extensions to the bittorrent | | | protocol.(default) | | | * ``off`` - disable bittorrent extensions. | +--------------------------+----------------------------------------------------+ | ``streaming`` | * ``on`` - enable streaming functionality. i.e. | | | ``set_piece_deadline()``. (default) | | | * ``off`` - disable streaming functionality. | +--------------------------+----------------------------------------------------+ | ``super-seeding`` | * ``on`` - enable super seeding feature. (default) | | | * ``off`` - disable super seeding feature | +--------------------------+----------------------------------------------------+ | ``share-mode`` | * ``on`` - enable share-mode feature. (default) | | | * ``off`` - disable share-mode feature | +--------------------------+----------------------------------------------------+ | ``predictive-pieces`` | * ``on`` - enable predictive piece announce | | | feature. i.e. | | | settings_pack::predictive_piece_announce | | | (default) | | | * ``off`` - disable feature. | +--------------------------+----------------------------------------------------+ | ``fpic`` | * ``off`` - default. Build without specifying | | | ``-fPIC``. | | | * ``on`` - Force build with ``-fPIC`` (useful for | | | building a static library to be linked into a | | | shared library). | +--------------------------+----------------------------------------------------+ | ``mmap-disk-io`` | * ``on`` - default. Enable mmap disk storage (if | | | available. | | | * ``off`` - disable mmap storage, and fall back to | | | single-threaded, portable file operations. | +--------------------------+----------------------------------------------------+ The ``variant`` feature is *implicit*, which means you don't need to specify the name of the feature, just the value. When building the example client on windows, you need to build with ``link=static`` otherwise you may get unresolved external symbols for some boost.program-options symbols. For more information, see the `Boost build v2 documentation`__, or more specifically `the section on built-in features`__. __ https://www.bfgroup.xyz/b2/manual/main/index.html __ https://www.boost.org/doc/libs/latest/tools/build/doc/html/index.html#b2.overview.builtins.features Step 4: Installing libtorrent ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To install libtorrent run ``b2`` with the ``install`` target:: b2 install --prefix=/usr/local Change the value of the ``--prefix`` argument to install it in a different location. Custom build flags ~~~~~~~~~~~~~~~~~~ Custom build flags can be passed to the command line via the ``cflags``, ``cxxflags`` and ``linkflags`` features. When specifying custom flags, make sure to build everything from scratch, to not accidentally mix incompatible flags. Example:: b2 cxxflags=-msse4.1 Custom flags can also be configured in the toolset, in ``~/user-config.jam``, ``Jamroot.jam`` or ``project-config.jam``. Example:: using gcc : sse41 : g++ : -msse4.1 ; Cross compiling ~~~~~~~~~~~~~~~ To cross compile libtorrent, configure a new toolset for ``b2`` to use. Toolsets can be configured in ``~/user-config.jam``, ``Jamroot.jam`` or ``project-config.jam``. The last two live in the libtorrent root directory. A toolset configuration is in this form: .. parsed-literal:: using *toolset* : *version* : *command-line* : *features* ; Toolset is essentially the family of compiler you're setting up, choose from `this list`__. __ https://www.bfgroup.xyz/b2/manual/main/index.html#b2.reference.tools.compilers Perhaps the most common ones would be ``gcc``, ``clang``, ``msvc`` and ``darwin`` (Apple's version of clang). The version can be left empty to be auto configured, or a custom name can be used to identify this toolset. The *command-line* is what to execute to run the compiler. This is also an opportunity to insert a call to ``ccache`` for example. *features* are boost-build features. Typical features to set here are ````, ```` and ````. For the ``gcc`` toolset, the ```` can be set to specify which tool to use to create a static library/archive. This is especially handy when cross compiling. Here's an example toolset for cross compiling for ARM Linux:: using gcc : arm : arm-linux-gnueabihf-g++ : arm-linux-gnueabihf-ar ; To build using this toolset, specify ``gcc-arm`` as the toolset on the ``b2`` command line. For example:: b2 toolset=gcc-arm building with cmake ------------------- First of all, you need to install ``cmake``. Additionally you need a build system to actually schedule builds, for example ``ninja``. Step 1: Generating the build system ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Create a build directory for out-of-source build inside the libtorrent root directory:: mkdir build and ``cd`` there:: cd build Run ``cmake`` in the build directory, like this:: cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=17 -G Ninja .. The ``CMAKE_CXX_STANDARD`` has to be at least 17, but you may want to raise it to ``20`` if your project use a newer version of the C++ standard. .. warning:: The detection of boost sometimes fail in subtle ways. If you have the ``BOOST_ROOT`` environment variable set, it may find the pre-built system libraries, but use the header files from your source package. To avoid this, invoke ``cmake`` with ``BOOST_ROOT`` set to an empty string: ``BOOST_ROOT="" cmake ...``. Other build options are: +-----------------------+---------------------------------------------------+ | ``BUILD_SHARED_LIBS`` | Defaults ``ON``. Builds libtorrent as a shared | | | library. | +-----------------------+---------------------------------------------------+ | ``static_runtime`` | Defaults ``OFF``. Link libtorrent statically | | | against the runtime libraries. | +-----------------------+---------------------------------------------------+ | ``build_tests`` | Defaults ``OFF``. Also build the libtorrent | | | tests. | +-----------------------+---------------------------------------------------+ | ``build_examples`` | Defaults ``OFF``. Also build the examples in the | | | examples directory. | +-----------------------+---------------------------------------------------+ | ``build_tools`` | Defaults ``OFF``. Also build the tools in the | | | tools directory. | +-----------------------+---------------------------------------------------+ | ``python-bindings`` | Defaults ``OFF``. Also build the python bindings | | | in bindings/python directory. | +-----------------------+---------------------------------------------------+ | ``encryption`` | Defaults ``ON``. Support trackers and bittorrent | | | over TLS, and obfuscated bittorrent connections. | +-----------------------+---------------------------------------------------+ Options are set on the ``cmake`` command line with the ``-D`` option or later on using ``ccmake`` or ``cmake-gui`` applications. ``cmake`` run outputs a summary of all available options and their current values. Step 2: Building libtorrent ~~~~~~~~~~~~~~~~~~~~~~~~~~~ In the terminal, run:: ninja If you enabled test in the configuration step, to run them, run:: ctest building with VCPKG ------------------- You can download and install libtorrent using the vcpkg_ dependency manager:: git clone https://github.com/Microsoft/vcpkg.git cd vcpkg ./bootstrap-vcpkg.sh ./vcpkg integrate install ./vcpkg install libtorrent .. _vcpkg: https://github.com/Microsoft/vcpkg/ The libtorrent port in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please `create an issue or pull request`__ on the vcpkg repository. __ https://github.com/Microsoft/vcpkg build configurations -------------------- By default libtorrent is built In debug mode, and will have pretty expensive invariant checks and asserts built into it. If you want to disable such checks (you want to do that in a release build) you can see the table below for which defines you can use to control the build. Make sure to define the same macros in your own code that compiles and links with libtorrent. +----------------------------------------+-------------------------------------------------+ | macro | description | +========================================+=================================================+ | ``NDEBUG`` | If you define this macro, all asserts, | | | invariant checks and general debug code will be | | | removed. Since there is quite a lot of code in | | | in header files in libtorrent, it may be | | | important to define the symbol consistently | | | across compilation units, including the clients | | | files. Potential problems is different | | | compilation units having different views of | | | structs and class layouts and sizes. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_DISABLE_LOGGING`` | This macro will disable support for logging | | | alerts, like log_alert, torrent_log_alert and | | | peer_log_alert. With this build flag, you | | | cannot enable those alerts. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_DISABLE_ALERT_MSG`` | Human readable messages returned from the alert | | | ``message()`` member functions will return | | | empty strings. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_DISABLE_SUPERSEEDING`` | This macro will disable support for super | | | seeding. The settings will exist, but will not | | | have an effect, when this macro is defined. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_DISABLE_SHARE_MODE`` | This macro will disable support for share-mode. | | | i.e. the mode to maximize upload/download | | | ratio for a torrent. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_DISABLE_MUTABLE_TORRENTS`` | Disables mutable torrent support (`BEP 38`_) | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_DISABLE_STREAMING`` | Disables set_piece_deadline() and associated | | | functionality. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_DISABLE_PREDICTIVE_PIECES`` | Disables | | | settings_pack::predictive_piece_announce | | | feature. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_LINKING_SHARED`` | If this is defined when including the | | | libtorrent headers, the classes and functions | | | will be tagged with ``__declspec(dllimport)`` | | | on msvc and default visibility on GCC 4 and | | | later. Set this in your project if you're | | | linking against libtorrent as a shared library. | | | (This is set by the Jamfile when | | | ``link=shared`` is set). | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_BUILDING_SHARED`` | If this is defined, the functions and classes | | | in libtorrent are marked with | | | ``__declspec(dllexport)`` on msvc, or with | | | default visibility on GCC 4 and later. This | | | should be defined when building libtorrent as | | | a shared library. (This is set by the Jamfile | | | when ``link=shared`` is set). | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_DISABLE_DHT`` | If this is defined, the support for trackerless | | | torrents will be disabled. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_DISABLE_ENCRYPTION`` | This will disable any encryption support and | | | the dependencies of a crypto library. | | | Encryption support is the peer connection | | | encrypted supported by clients such as | | | uTorrent, Azureus and KTorrent. | | | If this is not defined, either | | | ``TORRENT_USE_LIBCRYPTO`` or | | | ``TORRENT_USE_LIBGCRYPT`` must be defined. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_DISABLE_EXTENSIONS`` | When defined, libtorrent plugin support is | | | disabled along with support for the extension | | | handshake (BEP 10). | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_USE_INVARIANT_CHECKS`` | If defined to non-zero, this will enable | | | internal invariant checks in libtorrent. | | | The invariant checks can sometimes | | | be quite expensive, they typically don't scale | | | very well. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_EXPENSIVE_INVARIANT_CHECKS`` | This will enable extra expensive invariant | | | checks. Useful for finding particular bugs | | | or for running before releases. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_ABI_VERSION`` | The ABI version to support in the build. | | | A lower number means to support deprecated | | | functions, a higher number means newer versions | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_PRODUCTION_ASSERTS`` | Define to either 0 or 1. Enables assert logging | | | in release builds. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_USE_ASSERTS`` | Define as 0 to disable asserts unconditionally. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_USE_SYSTEM_ASSERTS`` | Uses the libc assert macro rather then the | | | custom one. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_HAVE_MMAP`` | Define as 0 to disable mmap support. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_USE_OPENSSL`` | Link against ``libssl`` for SSL support. Must | | | be combined with ``TORRENT_USE_LIBCRYPTO`` | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_USE_GNUTLS`` | Link against ``libgnutls`` for SSL support. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_USE_LIBCRYPTO`` | Link against ``libcrypto`` for SHA-1 support | | | and other hashing algorithms. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_USE_LIBGCRYPT`` | Link against ``libgcrypt`` for SHA-1 support | | | and other hashing algorithms. | +----------------------------------------+-------------------------------------------------+ | ``TORRENT_SSL_PEERS`` | Define to enable support for SSL torrents, | | | peers are connected over authenticated SSL | | | streams. | +----------------------------------------+-------------------------------------------------+ .. _`BEP 38`: https://www.bittorrent.org/beps/bep_0038.html If you experience that libtorrent uses unreasonable amounts of CPU, it will definitely help to define ``NDEBUG``, since it will remove the invariant checks within the library. building openssl for windows ---------------------------- To build openssl for windows with Visual Studio 7.1 (2003) execute the following commands in a command shell:: perl Configure VC-WIN32 --prefix="c:/openssl call ms\do_nasm call "C:\Program Files\Microsoft Visual Studio .NET 2003\vc7\bin\vcvars32.bat" nmake -f ms\nt.mak copy inc32\openssl "C:\Program Files\Microsoft Visual Studio .NET 2003\vc7\include\" copy out32\libeay32.lib "C:\Program Files\Microsoft Visual Studio .NET 2003\vc7\lib" copy out32\ssleay32.lib "C:\Program Files\Microsoft Visual Studio .NET 2003\vc7\lib" This will also install the headers and library files in the visual studio directories to be picked up by libtorrent. list of macros -------------- The following is a list of defines that libtorrent is built with: ``BOOST_ALL_NO_LIB``, ``BOOST_ASIO_ENABLE_CANCELIO``, ``BOOST_ASIO_HAS_STD_CHRONO``, ``BOOST_MULTI_INDEX_DISABLE_SERIALIZATION``, ``BOOST_NO_DEPRECATED``, ``BOOST_SYSTEM_NO_DEPRECATED`` Make sure you define the same at compile time for your code to avoid any runtime errors and other issues. These might change in the future, so it's always best to verify these every time you upgrade to a new version of libtorrent. The simplest way to see the full list of macros defined is to build libtorrent with ``-n -a`` switches added to ``b2`` command line:: b2 -n -a toolset=msvc-14.2 link=static runtime-link=static boost-link=static variant=release This will output all compiler switches, including defines (such as ``-DBOOST_ASIO_ENABLE_CANCELIO``). --- ## File: docs/client_test.rst =========================== client_test example program =========================== .. include:: header.rst Client test is a, more or less, complete bittorrent client. It lacks most settings and you can't start or stop torrents once you've started it. All the settings are hard coded. The command line arguments are:: client_test ... You can start any number of torrent downloads/seeds via the command line. If one argument starts with ``http://`` it is interpreted as a tracker announce url, and it expects an info-hash as the next argument. The info-hash has to be hex-encoded. For example: ``2410d4554d5ed856d69f426c38791673c59f4418``. If you pass an announce url and info-hash, a torrent-less download is started. It relies on that at least one peer on the tracker is running a libtorrent based client and has the metadata (.torrent file). The metadata extension in libtorrent will then download it from that peer (or from those peers if more than one). While running, the ``client_test`` sample will look something like this: .. image:: img/screenshot.png :class: screenshot :target: img/screenshot.png The commands available in the client are: * ``q`` quits the client (there will be a delay while the client waits for tracker responses) * ``l`` toggle log. Will display the log at the bottom, informing about tracker and peer events. * ``i`` toggles torrent info. Will show the peer list for each torrent. * ``d`` toggle download info. Will show the block list for each torrent, showing downloaded and requested blocks. * ``p`` pause all torrents. * ``u`` resume all torrents. * ``r`` force tracker reannounce for all torrents. * ``f`` toggle show file progress. Displays a list of all files and the download progress for each file. The list at the bottom (shown if you press ``d``) shows which blocks has been requested from which peer. The green background means that it has been downloaded. It shows that fast peers will prefer to request whole pieces instead of downloading parts of pieces. It may make it easier to determine which peer that sent the corrupt data if a piece fails the hash test. --- ## File: docs/contributing.rst .. include:: header.rst .. contents:: Table of contents :depth: 2 :backlinks: none contributing to libtorrent ========================== There are several ways to contribute to libtorrent at various levels. Any help is much appreciated. If you're interested in something libtorrent related that's not enumerated on this page, please contact arvid@libtorrent.org or the `mailing list`_. .. _`mailing list`: https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss 1. Testing This is not just limited to finding bugs and ways to reproduce crashes, but also sub-optimal behavior is certain scenarios and finding ways to reproduce those. Please report any issue to the bug tracker at `github`_. New features that need testing are streaming (``set_piece_deadline()``), the different choking algorithms (like the rate-based choker). Additional fuzzers are also always welcome. Find a libtorrent interface that's not already covered by a fuzzer (see the ``fuzzers`` directory in the root) and add a new fuzzer to it. Alternatively, improve an existing fuzzer by producing inputs that gets coverage deeper in to libtorrent. .. _`github`: https://github.com/arvidn/libtorrent/issues 2. Documentation Finding typos or outdated sections in the documentation. Contributing documentation based on your own experience and experimentation with the library or with BitTorrent in general. Non-reference documentation is very much welcome as well, higher level descriptions on how to configure libtorrent for various situations for instance. The reference documentation for libtorrent is generated from the header files. Each heading in the online documentation has a short-cut link to file a new issue against the documentation. For updates, please submit a `pull request`_. All documentation is in restructured text (rst_). All documentation is spell checked with hunspell which can be invoked via ``make spell-check`` in the docs directory. If words are missing, please add them to ``docs/hunspell/libtorrent.dic`` 3. Code Contributing code for new features or bug-fixes is highly welcome. If you're interested in adding a feature but not sure where to start, please contact the `mailing list`_ or ``#libtorrent`` @ ``irc.freenode.net``. For proposed fixes or updates, please submit a `pull request`_. New features might be better support for integrating with other services, new choking algorithms, seeding policies, ports to new platforms etc. For an overview of the internals of libtorrent, see the hacking_ page. For outstanding things to do, see the `todo list`_. .. _hacking: hacking.html .. _`pull request`: https://github.com/arvidn/libtorrent .. _`todo list`: todo.html .. _rst: https://docutils.sourceforge.io/rst.html --- ## File: docs/dht_extensions.rst Mainline DHT extensions ======================= .. include:: header.rst libtorrent implements a few extensions to the Mainline DHT protocol. get_peers response ------------------ libtorrent always responds with ``nodes`` to a get_peers request. If it has peers for the specified info-hash, it will return ``values`` as well. This is because just because some peer announced to us, doesn't mean that we are among the 8 closest nodes of the info hash. libtorrent also keeps traversing nodes using get_peers until it has found the 8 closest ones, and then announces to those nodes. forward compatibility --------------------- In order to support future DHT messages, any message which is not recognized but has either an ``info_hash`` or ``target`` argument is interpreted as find node for that target. i.e. it returns nodes. This allows future messages to be properly forwarded by clients that don't understand them instead of being blocked. client identification --------------------- In each DHT packet, an extra key is inserted named "v". This is a string describing the client and version used. This can help a lot when debugging and finding errors in client implementations. The string is encoded as four characters, two characters describing the client and two characters interpreted as a binary number describing the client version. Currently known clients: +---------------+--------+ | uTorrent | ``UT`` | +---------------+--------+ | libtorrent | ``LT`` | +---------------+--------+ | MooPolice | ``MP`` | +---------------+--------+ | GetRight | ``GR`` | +---------------+--------+ IPv6 support ------------ **This extension is superseded by** `BEP 32`_. .. _`BEP 32`: https://www.bittorrent.org/beps/bep_0032.html The DHT messages that don't support IPv6 are the ``nodes`` replies. They encode all the contacts as 6 bytes packed together in sequence in a string. The problem is that IPv6 endpoints cannot be encoded as 6 bytes, but needs 18 bytes. The extension libtorrent applies is to add another key, called ``nodes2``. ``nodes2`` may be present in replies that contains a ``nodes`` key. It is encoded as a list of strings. Each string represents one contact and is encoded as 20 bytes node-id and then a variable length encoded IP address (6 bytes in IPv4 case and 18 bytes in IPv6 case). As an optimization, libtorrent does not include the extra key in case there are only IPv4 nodes present. --- ## File: docs/dht_rss.rst ====================================== BitTorrent extension for DHT RSS feeds ====================================== .. include:: header.rst .. contents:: Table of contents :depth: 2 :backlinks: none This proposal has been superseded by the dht_put_ feature. This may still be implemented on top of that. .. _dht_put: dht_store.html This is a proposal for an extension to the BitTorrent DHT to allow for decentralized RSS feed like functionality. The intention is to allow the creation of repositories of torrents where only a single identity has the authority to add new content. For this repository to be robust against network failures and resilient to attacks at the source. The target ID under which the repository is stored in the DHT, is the SHA-1 hash of a feed name and the 512 bit public key. This private key in this pair MUST be used to sign every item stored in the repository. Every message that contain signed items MUST also include this key, to allow the receiver to verify the key itself against the target ID as well as the validity of the signatures of the items. Every recipient of a message with feed items in it MUST verify both the validity of the public key against the target ID it is stored under, as well as the validity of the signatures of each individual item. As with normal DHT announces, the write-token mechanism is used to prevent IP spoof attacks. terminology ----------- In this document, a *storage node* refers to the node in the DHT to which an item is being announce. A *subscribing node* refers to a node which makes look ups in the DHT to find the storage nodes, to request items from them. linked lists ------------ Items are chained together in a general singly linked list. A linked list does not necessarily contain RSS items, and no RSS related items are mandatory. However, RSS items will be used as examples in this BEP:: key = SHA1(name + key) +---------+ | head | key = SHA1(bencode(item)) | +---------+ +---------+ | | next |-------->| item | key = SHA1(bencode(item)) | | key | | +---------+ +---------+ | | name | | | next |------->| item | | | seq | | | key | | +---------+ | | ... | | | ... | | | next |--->0 | +---------+ | +---------+ | | key | | sig | | sig | | | ... | +---------+ +---------+ | +---------+ | sig | +---------+ The ``next`` pointer is at least 20 byte ID in the DHT key space pointing to where the next item in the list is announced. The list is terminated with an ID of all zeros. The ID an items is announced to is determined by the SHA1 hash of the bencoded representation of the item itself. This contains all fields in the item, except the signature. The only mandatory fields in an item are ``next``, ``key`` and ``sig``. The ``key`` field MUST match the public key of the list head node. The ``sig`` field MUST be the signature of the bencoded representation of ``item`` or ``head`` (whichever is included in the message). All subscribers MUST verify that the item is announced under the correct DHT key and MUST verify the signature is valid and MUST verify the public key is the same as the list-head. If a node fails any of these checks, it must be ignored and the chain of items considered terminated. Each item holds a bencoded dictionary with arbitrary keys, except two mandatory keys: ``next`` and ``key``. The signature ``sig`` is transferred outside of this dictionary and is the signature of all of it. An implementation should store any arbitrary keys that are announced to an item, within reasonable restriction such as nesting, size and numeric range of integers. skip lists ---------- The ``next`` key stored in the list head and the items is a string of at least length 20 bytes, it may be any length divisible by 20. Each 20 bytes are the ID of the next item in the list, the item 2 hops away, 4 hops away, 8 hops away, and so on. For simplicity, only the first ID (1 hop) in the ``next`` field is illustrated above. A publisher of an item SHOULD include as many IDs in the ``next`` field as the remaining size of the list warrants, within reason. These skip lists allow for parallelized lookups of items and also makes it more efficient to search for specific items. It also mitigates breaking lists missing some items. Figure of the skip list in the first list item:: n Item0 Item1 Item2 Item3 Item4 Item5 Item6 Item7 Item8 Item9 Item10 0 O-----> 20 O------------> 40 O--------------------------> 60 O------------------------------------------------------> *n* refers to the byte offset into the ``next`` field. list-head --------- The list head item is special in that it can be updated, without changing its DHT key. This is required to prepend new items to the linked list. To authenticate that only the original publisher can update the head, the whole linked list head is signed. In order to avoid a malicious node to overwrite the list head with an old version, the sequence number ``seq`` must be monotonically increasing for each update, and a node hosting the list node MUST not downgrade a list head from a higher sequence number to a lower one, only upgrade. The list head's DHT key (which it is announced to) MUST be the SHA1 hash of the name (``n``) and ``key`` fields concatenated. Any node MUST reject any list head which is announced under any other ID. messages -------- These are the messages to deal with linked lists. The ``id`` field in these messages has the same semantics as the standard DHT messages, i.e. the node ID of the node sending the message, to maintain the structure of the DHT network. The ``token`` field also has the same semantics as the standard DHT message ``get_peers`` and ``announce_peer``, when requesting an item and to write an item respectively. ``nodes`` and ``nodes6`` has the same semantics as in its ``get_peers`` response. requesting items ................ This message can be used to request both a list head and a list item. When requesting a list head, the ``n`` (name) field MUST be specified. When requesting a list item the ``n`` field is not required. .. parsed-literal:: { "a": { "id": *<20 byte ID of sending node>*, "key": *<64 byte public curve25519 key for this list>*, "n": ** "target": ** }, "q": "get_item", "t": **, "y": "q", } When requesting a list-head the ``target`` MUST always be SHA-1(*feed_name* + *public_key*). ``target`` is the target node ID the item was written to. The ``n`` field is the name of the list. If specified, It MUST be UTF-8 encoded string and it MUST match the name of the feed in the receiving node. request item response ..................... This is the format of a response of a list head: .. parsed-literal:: { "r": { "head": { "key": *<64 byte public curve25519 key for this list>*, "next": *<20 bytes item ID>*, "n": **, "seq": ** }, "sig": **, "id": *<20 byte id of sending node>*, "token": **, "nodes": **, "nodes6": ** }, "t": **, "y": "r", } This is the format of a response of a list item: .. parsed-literal:: { "r": { "item": { "key": *<64 byte public curve25519 key for this list>*, "next": *<20 bytes item ID>*, ... }, "sig": **, "id": *<20 byte id of sending node>*, "token": **, "nodes": **, "nodes6": ** }, "t": **, "y": "r", } A client receiving a ``get_item`` response MUST verify the signature in the ``sig`` field against the bencoded representation of the ``item`` field, using the ``key`` as the public key. The ``key`` MUST match the public key of the feed. The ``item`` dictionary MAY contain arbitrary keys, and all keys MUST be stored for items. announcing items ................ The message format for announcing a list head: .. parsed-literal:: { "a": { "head": { "key": *<64 byte public curve25519 key for this list>*, "next": *<20 bytes item ID>*, "n": **, "seq": ** }, "sig": **, "id": *<20 byte node-id of origin node>*, "target": **, "token": ** }, "y": "q", "q": "announce_item", "t": ** } The message format for announcing a list item: .. parsed-literal:: { "a": { "item": { "key": *<64 byte public curve25519 key for this list>*, "next": *<20 bytes item ID>*, ... }, "sig": **, "id": *<20 byte node-id of origin node>*, "target": **, "token": ** }, "y": "q", "q": "announce_item", "t": ** } A storage node MAY reject items and heads whose bencoded representation is greater than 1024 bytes. re-announcing ------------- In order to keep feeds alive, subscriber nodes SHOULD help out in announcing items they have downloaded to the DHT. Every subscriber node SHOULD store items in long term storage, across sessions, in order to keep items alive for as long as possible, with as few sources as possible. Subscribers to a feed SHOULD also announce items that they know of, to the feed. Since a feed may have many subscribers and many items, subscribers should re-announce items according to the following algorithm. .. parsed-literal:: 1. pick one random item (*i*) from the local repository (except items already announced this round) 2. If all items in the local repository have been announced 2.1 terminate 3. look up item *i* in the DHT 4. If fewer than 8 nodes returned the item 4.1 announce *i* to the DHT 4.2 goto 1 This ensures a balanced load on the DHT while still keeping items alive timeouts -------- Items SHOULD be announced to the DHT every 30 minutes. A storage node MAY time out an item after 60 minutes of no one announcing it. A storing node MAY extend the timeout when it receives a request for it. Since items are immutable, the data doesn't go stale. Therefore it doesn't matter if the storing node no longer is in the set of the 8 closest nodes. RSS feeds --------- For RSS feeds, following keys are mandatory in the list item's ``item`` dictionary. ih The torrent's info hash size The size (in bytes) of all files the torrent n name of the torrent example ....... This is an example of an ``announce_item`` message: .. parsed-literal:: { "a": { "item": { "key": "6bc1de5443d1a7c536cdf69433ac4a7163d3c63e2f9c92d 78f6011cf63dbcd5b638bbc2119cdad0c57e4c61bc69ba5e2c08 b918c2db8d1848cf514bd9958d307", "info-hash": "7ea94c240691311dc0916a2a91eb7c3db2c6f3e4", "size": 24315329, "n": "my stuff", "next": "c68f29156404e8e0aas8761ef5236bcagf7f8f2e" } "sig": ** "id": "b46989156404e8e0acdb751ef553b210ef77822e", "target": "b4692ef0005639e86d7165bf378474107bf3a762" "token": "23ba" }, "y": "q", "q": "announce_item", "t": "a421" } Strings are printed in hex for printability, but actual encoding is binary. Note that ``target`` is in fact SHA1 hash of the same data the signature ``sig`` is the signature of, i.e.:: d9:info-hash20:7ea94c240691311dc0916a2a91eb7c3db2c6f3e43:key64:6bc1de5443d1 a7c536cdf69433ac4a7163d3c63e2f9c92d78f6011cf63dbcd5b638bbc2119cdad0c57e4c61 bc69ba5e2c08b918c2db8d1848cf514bd9958d3071:n8:my stuff4:next20:c68f29156404 e8e0aas8761ef5236bcagf7f8f2e4:sizei24315329ee (note that binary data is printed as hex) RSS feed URI scheme -------------------- The proposed URI scheme for DHT feeds is: .. parsed-literal:: magnet:?xt=btfd:** &dn= ** Note that a difference from regular torrent magnet links is the **btfd** versus **btih** used in regular magnet links to torrents. The *feed name* is mandatory since it is used in the request and when calculating the target ID. rationale --------- The reason to use curve25519_ instead of, for instance, RSA is compactness. According to https://cr.yp.to/, curve25519 is free from patent claims and there are open implementations in both C and Java. .. _curve25519: https://cr.yp.to/ecdh.html --- ## File: docs/dht_sec.rst .. include:: header.rst .. contents:: Table of contents :depth: 2 :backlinks: none BitTorrent DHT security extension --------------------------------- The purpose of this extension is to make it harder to launch a few specific attacks against the BitTorrent DHT and also to make it harder to snoop the network. Specifically the attack this extension intends to make harder is launching 8 or more DHT nodes which node-IDs selected close to a specific target info-hash, in order to become the main nodes hosting peers for it. Currently this is very easy to do and lets the attacker not only see all the traffic related to this specific info-hash but also block access to it by other peers. The proposed guard against this is to enforce restrictions on which node-ID a node can choose, based on its external IP address. considerations -------------- One straight forward scheme to tie the node ID to an IP would be to hash the IP and force the node ID to share the prefix of that hash. One main draw back of this approach is that an entities control over the DHT key space grows linearly with its control over the IP address space. In order to successfully launch an attack, you just need to find 8 IPs whose hash will be *closest* to the target info-hash. Given the current size of the DHT, that is quite likely to be possible by anyone in control of a /8 IP block. The size of the DHT is approximately 8.4 million nodes. This is estimated by observing that a typical routing table typically has about 20 of its top routing table buckets full. That means the key space is dense enough to contain 8 nodes for every combination of the 20 top bits of node IDs. ``2^20 * 8 = 8388608`` By controlling that many IP addresses, an attacker could snoop any info-hash. By controlling 8 times that many IP addresses, an attacker could actually take over any info-hash. With IPv4, snooping would require a /8 IP block, giving access to 16.7 million IPs. Another problem with hashing the IP is that multiple users behind a NAT are forced to run their DHT nodes on the same node ID. Node ID restriction ------------------- In order to avoid the number node IDs controlled to grow linearly by the number of IPs, as well as allowing more than one node ID per external IP, the node ID can be restricted at each class level of the IP. Another important property of the restriction put on node IDs is that the distribution of the IDs remain uniform. This is why CRC32C (Castagnoli) was chosen as the hash function. The expression to calculate a valid ID prefix (from an IPv4 address) is:: crc32c((ip & 0x030f3fff) | (r << 29)) And for an IPv6 address (``ip`` is the high 64 bits of the address):: crc32c((ip & 0x0103070f1f3f7fff) | (r << 61)) ``r`` is a random number in the range [0, 7]. The resulting integer, representing the masked IP address is supposed to be big-endian before hashed. The "|" operator means bit-wise OR. The details of implementing this is to evaluate the expression, store the result in a big-endian 64 bit integer and hash those 8 bytes with CRC32C. The first (most significant) 21 bits of the node ID used in the DHT MUST match the first 21 bits of the resulting hash. The last byte of the hash MUST match the random number (``r``) used to generate the hash. .. image:: img/ip_id_v4.png :class: bw .. image:: img/ip_id_v6.png :class: bw Example code code for calculating a valid node ID:: uint8_t* ip; // our external IPv4 or IPv6 address (network byte order) int num_octets; // the number of octets to consider in ip (4 or 8) uint8_t node_id[20]; // resulting node ID uint8_t v4_mask[] = { 0x03, 0x0f, 0x3f, 0xff }; uint8_t v6_mask[] = { 0x01, 0x03, 0x07, 0x0f, 0x1f, 0x3f, 0x7f, 0xff }; uint8_t* mask = num_octets == 4 ? v4_mask : v6_mask; for (int i = 0; i < num_octets; ++i) ip[i] &= mask[i]; uint32_t rand = std::rand() & 0xff; uint8_t r = rand & 0x7; ip[0] |= r << 5; uint32_t crc = 0; crc = crc32c(crc, ip, num_octets); // only take the top 21 bits from crc node_id[0] = (crc >> 24) & 0xff; node_id[1] = (crc >> 16) & 0xff; node_id[2] = ((crc >> 8) & 0xf8) | (std::rand() & 0x7); for (int i = 3; i < 19; ++i) node_id[i] = std::rand(); node_id[19] = rand; test vectors: .. parsed-literal:: IP rand example node ID ============ ===== ========================================== 124.31.75.21 1 **5fbfbf** f10c5d6a4ec8a88e4c6ab4c28b95eee4 **01** 21.75.31.124 86 **5a3ce9** c14e7a08645677bbd1cfe7d8f956d532 **56** 65.23.51.170 22 **a5d432** 20bc8f112a3d426c84764f8c2a1150e6 **16** 84.124.73.14 65 **1b0321** dd1bb1fe518101ceef99462b947a01ff **41** 43.213.53.83 90 **e56f6c** bf5b7c4be0237986d5243b87aa6d5130 **5a** The bold parts of the node ID are the important parts. The rest are random numbers. The last bold number of each row has only its most significant bit pulled from the CRC32C function. The lower 3 bits are random. bootstrapping ------------- In order to set ones initial node ID, the external IP needs to be known. This is not a trivial problem. With this extension, *all* DHT responses SHOULD include a *top-level* field called ``ip``, containing a compact binary representation of the requester's IP and port. That is big-endian IP followed by 2 bytes of big-endian port. The IP portion is the same byte sequence used to verify the node ID. It is important that the ``ip`` field is in the top level dictionary. Nodes that enforce the node-ID will respond with an error message ("y": "e", "e": { ... }), whereas a node that supports this extension but without enforcing it will respond with a normal reply ("y": "r", "r": { ... }). A DHT node which receives an ``ip`` result in a request SHOULD consider restarting its DHT node with a new node ID, taking this IP into account. Since a single node can not be trusted, there should be some mechanism to determine whether or not the node has a correct understanding of its external IP or not. This could be done by voting, or only restart the DHT once at least a certain number of nodes, from separate searches, tells you your node ID is incorrect. rationale --------- The choice of using CRC32C instead of a more traditional cryptographic hash function is justified primarily of these reasons: 1. it is a fast function 2. produces well distributed results 3. there is no need for the hash function to be one-way (the input set is so small that any hash function could be reversed). 4. CRC32C (Castagnoli) is supported in hardware by SSE 4.2, which can significantly speed up computation There are primarily two tests run on SHA-1 and CRC32C to establish the distribution of results. The first one is the number of bits in the output set that contain every possible combination of bits. The CRC32C function has a longer such prefix in its output than SHA-1. This means nodes will still have well uniformly distributed IDs, even when IP addresses in use are not uniformly distributed. The following graph illustrate a few different hash functions with regard to this property. .. image:: img/complete_bit_prefixes.png :class: bw This test takes into account IP addresses that are not globally routable, i.e. reserved for local networks, multicast and other things. It also takes into account that some /8 blocks are not in use by end-users and extremely unlikely to ever run a DHT node. This makes the results likely to be very similar to what we would see in the wild. These results indicate that CRC32C provides the best uniformity in the results in terms of bit prefixes where all possibilities are represented, and that no more than 21 bits should be used from the result. If more than 21 bits were to be used, there would be certain node IDs that would be impossible to have, which would make routing sub-optimal. The second test is more of a sanity test for the uniform distribution property. The target space (32 bit integer) is divided up into 1000 buckets. Every valid IP and ``r`` input is run through the algorithm and the result is put in the bucket it falls in. The expectation is that each bucket has roughly an equal number of results falling into it. The following graph shows the resulting histogram, comparing SHA-1 and CRC32C. .. image:: img/hash_distribution.png :class: bw The source code for these tests can be found here_. .. _here: https://github.com/arvidn/hash_complete_prefix The reason to use CRC32C instead of the CRC32 implemented by zlib is that Intel CPUs have hardware support for the CRC32C calculations. The input being exactly 4 bytes is also deliberate, to make it fit in a single instruction. enforcement ----------- Once enforced, write tokens from peers whose node ID does not match its external IP should be considered dropped. In other words, a peer that uses a non-matching ID MUST never be used to store information on, regardless of which request. In the original DHT specification only ``announce_peer`` stores data in the network, but any future extension which stores data in the network SHOULD use the same restriction. Any peer on a local network address is exempt from this node ID verification. This includes the following IP blocks: 10.0.0.0/8 reserved for local networks 172.16.0.0/12 reserved for local networks 192.168.0.0/16 reserved for local networks 169.254.0.0/16 reserved for self-assigned IPs 127.0.0.0/8 reserved for loopback backwards compatibility and transition -------------------------------------- During some transition period, this restriction should not be enforced, and peers whose node ID does not match this formula relative to their external IP should not be blocked. Requests from peers whose node ID does not match their external IP should always be serviced, even after the transition period. The attack this protects from is storing data on an attacker's node, not servicing an attackers request. forward compatibility --------------------- If the total size of the DHT grows to the point where the inherent size limit in this proposal is too small, the modulus constants can be updated in a new proposal, and another transition period where both sets of modulus constants are accepted. --- ## File: docs/dht_store.rst ============================================ BitTorrent extension for arbitrary DHT store ============================================ .. include:: header.rst .. contents:: Table of contents :depth: 2 :backlinks: none This is a proposal for an extension to the BitTorrent DHT to allow storing and retrieving of arbitrary data. It supports both storing *immutable* items, where the key is the SHA-1 hash of the data itself, and *mutable* items, where the key is the public key of the key pair used to sign the data. There are two new proposed messages, ``put`` and ``get``. terminology ----------- In this document, a *storage node* refers to the node in the DHT to which an item is being announced and stored on. A *requesting node* refers to a node which makes look-ups in the DHT to find the storage nodes, to request items from them, and possibly re-announce those items to keep them alive. messages -------- The proposed new messages ``get`` and ``put`` are similar to the existing ``get_peers`` and ``announce_peer``. Responses to ``get`` should always include ``nodes`` and ``nodes6``. Those fields have the same semantics as in its ``get_peers`` response. It should also include a write token, ``token``, with the same semantics as int ``get_peers``. The write token MAY be tied specifically to the key which ``get`` requested. i.e. the ``token`` can only be used to store values under that one key. It may also be tied to the node ID and IP address of the requesting node. The ``id`` field in these messages has the same semantics as the standard DHT messages, i.e. the node ID of the node sending the message, to maintain the structure of the DHT network. The ``token`` field also has the same semantics as the standard DHT message ``get_peers`` and ``announce_peer``, when requesting an item and to write an item respectively. The ``k`` field is the 32 byte ed25519 public key, which the signature can be authenticated with. When looking up a mutable item, the ``target`` field MUST be the SHA-1 hash of this key concatenated with the ``salt``, if present. The distinction between storing mutable and immutable items is the inclusion of a public key, a sequence number, signature and an optional salt (``k``, ``seq``, ``sig`` and ``salt``). ``get`` requests for mutable items and immutable items cannot be distinguished from each other. An implementation can either store mutable and immutable items in the same hash table internally, or in separate ones and potentially do two lookups for ``get`` requests. The ``v`` field is the *value* to be stored. It is allowed to be any bencoded type (list, dict, string or integer). When it's being hashed (for verifying its signature or to calculate its key), its flattened, bencoded, form is used. It is important to use the verbatim bencoded representation as it appeared in the message. decoding and then re-encoding bencoded structures is not necessarily an identity operation. Storing nodes MAY reject ``put`` requests where the bencoded form of ``v`` is longer than 1000 bytes. In other words, it's not safe to assume storing more than 1000 bytes will succeed. immutable items --------------- Immutable items are stored under their SHA-1 hash, and since they cannot be modified, there is no need to authenticate the origin of them. This makes immutable items simple. A node making a lookup SHOULD verify the data it receives from the network, to verify that its hash matches the target that was looked up. put message ........... Request: .. parsed-literal:: { "a": { "id": *<20 byte id of sending node (string)>*, "v": ** }, "t": **, "y": "q", "q": "put" } Response: .. parsed-literal:: { "r": { "id": *<20 byte id of sending node (string)>* }, "t": **, "y": "r", } get message ........... Request: .. parsed-literal:: { "a": { "id": *<20 byte id of sending node (string)>*, "target": **, }, "t": **, "y": "q", "q": "get" } Response: .. parsed-literal:: { "r": { "id": *<20 byte id of sending node (string)>*, "token": **, "v": **, "nodes": **, "nodes6": ** }, "t": **, "y": "r", } mutable items ------------- Mutable items can be updated, without changing their DHT keys. To authenticate that only the original publisher can update an item, it is signed by a private key generated by the original publisher. The target ID mutable items are stored under is the SHA-1 hash of the public key (as it appears in the ``put`` message). In order to avoid a malicious node to overwrite the list head with an old version, the sequence number ``seq`` must be monotonically increasing for each update, and a node hosting the list node MUST not downgrade a list head from a higher sequence number to a lower one, only upgrade. The sequence number SHOULD not exceed ``MAX_INT64``, (i.e. ``0x7fffffffffffffff``. A client MAY reject any message with a sequence number exceeding this. A client MAY also reject any message with a negative sequence number. The signature is a 64 byte ed25519 signature of the bencoded sequence number concatenated with the ``v`` key. e.g. something like this:: 3:seqi4e1:v12:Hello world! If the ``salt`` key is present and non-empty, the salt string must be included in what's signed. Note that if ``salt`` is specified and an empty string, it is as if it was not specified and nothing in addition to the sequence number and the data is signed. The salt string may not be longer than 64 bytes. When a salt is included in what is signed, the key ``salt`` with the value of the key is prepended in its bencoded form. For example, if ``salt`` is "foobar", the buffer to be signed is:: 4:salt6:foobar3:seqi4e1:v12:Hello world! put message ........... Request: .. parsed-literal:: { "a": { "cas": **, "id": *<20 byte id of sending node (string)>*, "k": **, "salt": ** "seq": **, "sig": **, "token": **, "v": ** }, "t": **, "y": "q", "q": "put" } Storing nodes receiving a ``put`` request where ``seq`` is lower than or equal to what's already stored on the node, MUST reject the request. If the sequence number is equal, and the value is also the same, the node SHOULD reset its timeout counter. If the sequence number in the ``put`` message is lower than the sequence number associated with the currently stored value, the storing node MAY return an error message with code 302 (see error codes below). Note that this request does not contain a target hash. The target hash under which this blob is stored is implied by the ``k`` argument. The key is the SHA-1 hash of the key (``k``). In order to support a single key being used to store separate items in the DHT, an optional ``salt`` can be specified in the ``put`` request of mutable items. If the salt entry is not present, it can be assumed to be an empty string, and its semantics should be identical as specifying a salt key with an empty string. The salt can be any binary string (but probably most conveniently a hash of something). This string is appended to the key, as specified in the ``k`` field, when calculating the key to store the blob under (i.e. the key ``get`` requests specify to retrieve this data). This lets a single entity, with a single key, publish any number of unrelated items, with a single key that readers can verify. This is useful if the publisher doesn't know ahead of time how many different items are to be published. It can distribute a single public key for users to authenticate the published blobs. Note that the salt is not returned in the response to a ``get`` request. This is intentional. When issuing a ``get`` request for an item is expected to know what the salt is (because it is part of what the target ID that is being looked up is derived from). There is no need to repeat it back for bystanders to see. CAS ... CAS is short for *compare and swap*, it has similar semantics as CAS CPU instructions. It is used to avoid race conditions when multiple nodes are writing to the same slot in the DHT. The ``cas`` field is optional. If present it specifies the sequence number of the data blob being overwritten by the put. When present, the storing node MUST compare this number to the current sequence number it has stored under this key. Only if the ``cas`` matches the stored sequence number is the put performed. If it mismatches, the store fails and an error is returned. See errors_ below. The ``cas`` field only applies to mutable puts. If there is no current value, the ``cas`` field SHOULD be ignored. When sending a ``put`` request to a node that did not return any data for the ``get``, the ``cas`` field SHOULD NOT be included. response ........ Response: .. parsed-literal:: { "r": { "id": *<20 byte id of sending node (string)>* }, "t": **, "y": "r", } errors ...... If the store fails for any reason an error message is returned instead of the message template above, i.e. one where "y" is "e" and "e" is a tuple of [error-code, message]). Failures include ``cas`` mismatches and the sequence number is outdated. The error message (as specified by BEP5_) looks like this: .. _BEP5: https://www.bittorrent.org/beps/bep_0005.html .. parsed-literal:: { "e": [ **, ** ], "t": **, "y": "e", } In addition to the error codes defined in BEP5_, this specification defines some additional error codes. +------------+-----------------------------+ | error-code | description | +============+=============================+ | 205 | message (``v`` field) | | | too big. | +------------+-----------------------------+ | 206 | invalid signature | +------------+-----------------------------+ | 207 | salt (``salt`` field) | | | too big. | +------------+-----------------------------+ | 301 | the CAS hash mismatched, | | | re-read value and try | | | again. | +------------+-----------------------------+ | 302 | sequence number less than | | | current. | +------------+-----------------------------+ An implementation MUST emit 301 errors if the cas mismatches. This is a critical feature in synchronization of multiple agents sharing an immutable item. get message ........... Request: .. parsed-literal:: { "a": { "id": *<20 byte id of sending node (string)>*, "target:" *<20 byte SHA-1 hash of public key and salt (string)>* }, "t": **, "y": "q", "q": "get" } Response: .. parsed-literal:: { "r": { "id": *<20 byte id of sending node (string)>*, "k": **, "nodes": **, "nodes6": **, "seq": **, "sig": **, "token": **, "v": ** }, "t": **, "y": "r", } signature verification ---------------------- In order to make it maximally difficult to attack the bencoding parser, signing and verification of the value and sequence number should be done as follows: 1. encode value and sequence number separately 2. concatenate ("4:salt" *length-of-salt* ":" *salt*) "3:seqi" *seq* "e1:v" *len* ":" and the encoded value. sequence number 1 of value "Hello World!" would be converted to: "3:seqi1e1:v12:Hello World!". In this way it is not possible to convince a node that part of the length is actually part of the sequence number even if the parser contains certain bugs. Furthermore it is not possible to have a verification failure if a bencoding serializer alters the order of entries in the dictionary. The salt is in parenthesis because it is optional. It is only prepended if a non-empty salt is specified in the ``put`` request. 3. sign or verify the concatenated string On the storage node, the signature MUST be verified before accepting the store command. The data MUST be stored under the SHA-1 hash of the public key (as it appears in the bencoded dict) and the salt (if present). On the requesting nodes, the key they get back from a ``get`` request MUST be verified to hash to the target ID the lookup was made for, as well as verifying the signature. If any of these fail, the response SHOULD be considered invalid. expiration ---------- Without re-announcement, these items MAY expire in 2 hours. In order to keep items alive, they SHOULD be re-announced once an hour. Any node that's interested in keeping a blob in the DHT alive may announce it. It would simply repeat the signature for a mutable put without having the private key. test vectors ------------ test 1 (mutable) ................ value:: 12:Hello World! buffer being signed:: 3:seqi1e1:v12:Hello World! public key:: 77ff84905a91936367c01360803104f92432fcd904a43511876df5cdf3e7e548 private key:: e06d3183d14159228433ed599221b80bd0a5ce8352e4bdf0262f76786ef1c74d b7e7a9fea2c0eb269d61e3b38e450a22e754941ac78479d6c54e1faf6037881d **target ID**:: 4a533d47ec9c7d95b1ad75f576cffc641853b750 **signature**:: 305ac8aeb6c9c151fa120f120ea2cfb923564e11552d06a5d856091e5e853cff 1260d3f39e4999684aa92eb73ffd136e6f4f3ecbfda0ce53a1608ecd7ae21f01 test 2 (mutable with salt) .......................... value:: 12:Hello World! salt:: foobar buffer being signed:: 4:salt6:foobar3:seqi1e1:v12:Hello World! public key:: 77ff84905a91936367c01360803104f92432fcd904a43511876df5cdf3e7e548 private key:: e06d3183d14159228433ed599221b80bd0a5ce8352e4bdf0262f76786ef1c74d b7e7a9fea2c0eb269d61e3b38e450a22e754941ac78479d6c54e1faf6037881d **target ID**:: 411eba73b6f087ca51a3795d9c8c938d365e32c1 **signature**:: 6834284b6b24c3204eb2fea824d82f88883a3d95e8b4a21b8c0ded553d17d17d df9a8a7104b1258f30bed3787e6cb896fca78c58f8e03b5f18f14951a87d9a08 test 3 (immutable) .................. value:: 12:Hello World! **target ID**:: e5f96f6f38320f0f33959cb4d3d656452117aadb resources --------- Libraries that implement ed25519 DSA: * NaCl_ * libsodium_ * `nightcracker's ed25519`_ .. _NaCl: https://nacl.cr.yp.to/ .. _libsodium: https://github.com/jedisct1/libsodium .. _`nightcracker's ed25519`: https://github.com/nightcracker/ed25519 --- ## File: docs/examples.rst .. include:: header.rst .. contents:: Table of contents :depth: 2 :backlinks: none examples ======== Except for the example programs in this manual, there's also a bigger example of a (little bit) more complete client, ``client_test``. There are separate instructions for how to use it here__ if you'd like to try it. __ client_test.html simple client ------------- This is a simple client. It doesn't have much output to keep it simple: .. include:: ../examples/simple_client.cpp :code: c++ :tab-width: 2 :start-after: */ make_torrent ------------ Shows how to create a torrent from a directory tree: .. include:: ../examples/make_torrent.cpp :code: c++ :tab-width: 2 :start-after: */ dump_torrent ------------ This is an example of a program that will take a torrent-file as a parameter and print information about it to std out: .. include:: ../examples/dump_torrent.cpp :code: c++ :tab-width: 2 :start-after: */ --- ## File: docs/extension_protocol.rst :Author: Arvid Norberg, arvid@libtorrent.org Ludvig Strigeus, ludde@utorrent.com extension protocol for bittorrent ================================= The intention of this protocol is to provide a simple and thin transport for extensions to the bittorrent protocol. Supporting this protocol makes it easy to add new extensions without interfering with the standard bittorrent protocol or clients that don't support this extension or the one you want to add. To advertise to other clients that you support, one bit from the reserved bytes is used. The bit selected for the extension protocol is bit 20 from the right (counting starts at 0). So (reserved_byte[5] & 0x10) is the expression to use for checking if the client supports extended messaging. Once support for the protocol is established, the client is supposed to support 1 new message: +------------------------+----+ |name | id | +========================+====+ |``extended`` | 20 | +------------------------+----+ This message is sent as any other bittorrent message, with a 4 byte length prefix and a single byte identifying the message (the single byte being 20 in this case). At the start of the payload of the message, is a single byte message identifier. This identifier can refer to different extension messages and only one ID is specified, 0. If the ID is 0, the message is a handshake message which is described below. The layout of a general ``extended`` message follows (including the message headers used by the bittorrent protocol): +----------+---------------------------------------------------------+ | size | description | +==========+=========================================================+ | uint32_t | length prefix. Specifies the number of bytes for the | | | entire message. (big-endian) | +----------+---------------------------------------------------------+ | uint8_t | bittorrent message ID, = 20 | +----------+---------------------------------------------------------+ | uint8_t | extended message ID. 0 = handshake, >0 = extended | | | message as specified by the handshake. | +----------+---------------------------------------------------------+ handshake message ----------------- The payload of the handshake message is a bencoded dictionary. All items in the dictionary are optional. Any unknown names should be ignored by the client. All parts of the dictionary are case sensitive. This is the defined item in the dictionary: +-------+-----------------------------------------------------------+ | name | description | +=======+===========================================================+ | m | Dictionary of supported extension messages which maps | | | names of extensions to an extended message ID for each | | | extension message. The only requirement on these IDs | | | is that no extension message share the same one. Setting | | | an extension number to zero means that the extension is | | | not supported/disabled. The client should ignore any | | | extension names it doesn't recognize. | | | | | | The extension message IDs are the IDs used to send the | | | extension messages to the peer sending this handshake. | | | i.e. The IDs are local to this particular peer. | +-------+-----------------------------------------------------------+ Here are some other items that an implementation may choose to support: +--------+-----------------------------------------------------------+ | name | description | +========+===========================================================+ | p | Local TCP listen port. Allows each side to learn about | | | the TCP port number of the other side. Note that there is | | | no need for the receiving side of the connection to send | | | this extension message, since its port number is already | | | known. | +--------+-----------------------------------------------------------+ | v | Client name and version (as a utf-8 string). | | | This is a much more reliable way of identifying the | | | client than relying on the peer id encoding. | +--------+-----------------------------------------------------------+ | yourip | A string containing the compact representation of the ip | | | address this peer sees you as. i.e. this is the | | | receiver's external ip address (no port is included). | | | This may be either an IPv4 (4 bytes) or an IPv6 | | | (16 bytes) address. | +--------+-----------------------------------------------------------+ | ipv6 | If this peer has an IPv6 interface, this is the compact | | | representation of that address (16 bytes). The client may | | | prefer to connect back via the IPv6 address. | +--------+-----------------------------------------------------------+ | ipv4 | If this peer has an IPv4 interface, this is the compact | | | representation of that address (4 bytes). The client may | | | prefer to connect back via this interface. | +--------+-----------------------------------------------------------+ | reqq | An integer, the number of outstanding request messages | | | this client supports without dropping any. The default in | | | in libtorrent is 250. | +--------+-----------------------------------------------------------+ The handshake dictionary could also include extended handshake information, such as support for encrypted headers or anything imaginable. An example of what the payload of a handshake message could look like: +------------------------------------------------------+ | Dictionary | +===================+==================================+ | ``m`` | +--------------------------+ | | | | Dictionary | | | | +======================+===+ | | | | ``LT_metadata`` | 1 | | | | +----------------------+---+ | | | | ``ut_pex`` | 2 | | | | +----------------------+---+ | | | | +-------------------+----------------------------------+ | ``p`` | 6881 | +-------------------+----------------------------------+ | ``v`` | "uTorrent 1.2" | +-------------------+----------------------------------+ and in the encoded form: ``d1:md11:LT_metadatai1e6:ut_pexi2ee1:pi6881e1:v12:uTorrent 1.2e`` To make sure the extension names do not collide by mistake, they should be prefixed with the two (or one) character code that is used to identify the client that introduced the extension. This applies for both the names of extension messages, and for any additional information put inside the top-level dictionary. All one and two byte identifiers are invalid to use unless defined by this specification. This message should be sent immediately after the standard bittorrent handshake to any peer that supports this extension protocol. It is valid to send the handshake message more than once during the lifetime of a connection, the sending client should not be disconnected. An implementation may choose to ignore the subsequent handshake messages (or parts of them). Subsequent handshake messages can be used to enable/disable extensions without restarting the connection. If a peer supports changing extensions at run time, it should note that the ``m`` dictionary is additive. It's enough that it contains the actual *CHANGES* to the extension list. To disable the support for ``LT_metadata`` at run-time, without affecting any other extensions, this message should be sent: ``d11:LT_metadatai0ee``. As specified above, the value 0 is used to turn off an extension. The extension IDs must be stored for every peer, because every peer may have different IDs for the same extension. This specification, deliberately, does not specify any extensions such as peer-exchange or metadata exchange. This protocol is merely a transport for the actual extensions to the bittorrent protocol and the extensions named in the example above (such as ``p``) are just examples of possible extensions. rationale --------- The reason why the extension messages' IDs would be defined in the handshake is to avoid having a global registry of message IDs. Instead the names of the extension messages requires unique names, which is much easier to do without a global registry. The convention is to use a two letter prefix on the extension message names, the prefix would identify the client first implementing the extension message. e.g. ``LT_metadata`` is implemented by libtorrent, and hence it has the ``LT`` prefix. If the client supporting the extensions can decide which numbers the messages it receives will have, it means they are constants within that client. i.e. they can be used in ``switch`` statements. It's easy for the other end to store an array with the ID's we expect for each message and use that for lookups each time it sends an extension message. The reason for having a dictionary instead of having an array (using implicitly assigned index numbers to the extensions) is that if a client want to disable some extensions, the ID numbers would change, and it wouldn't be able to use constants (and hence, not use them in a ``switch``). If the messages IDs would map directly to bittorrent message IDs, It would also make it possible to map extensions in the handshake to existing extensions with fixed message IDs. The reasoning behind having a single byte as extended message identifier is to follow the bittorrent spec. with its single byte message identifiers. It is also considered to be enough. It won't limit the total number of extensions, only the number of extensions used simultaneously. The reason for using single byte identifiers for the standardized handshake identifiers is 1) The mainline DHT uses single byte identifiers. 2) Saves bandwidth. The only advantage of longer messages is that it makes the protocol more readable for a human, but the BT protocol wasn't designed to be a human readable protocol, so why bother. extensions ========== These extensions all operates within the `extension protocol`_. The name of the extension is the name used in the extension-list packets, and the payload is the data in the extended message (not counting the length-prefix, message-id nor extension-id). .. _`extension protocol`: extension_protocol.html Note that since this protocol relies on one of the reserved bits in the handshake, it may be incompatible with future versions of the mainline bittorrent client. These are the extensions that are currently implemented. metadata from peers ------------------- Extension name: "LT_metadata" .. note:: This extension is deprecated in favor of the more widely supported ``ut_metadata`` extension, see `BEP 9`_. The point with this extension is that you don't have to distribute the metadata (.torrent-file) separately. The metadata can be distributed through the bittorrent swarm. The only thing you need to download such a torrent is the tracker url and the info-hash of the torrent. It works by assuming that the initial seeder has the metadata and that the metadata will propagate through the network as more peers join. There are three kinds of messages in the metadata extension. These packets are put as payload to the extension message. The three packets are: * request metadata * metadata * don't have metadata request metadata: +-----------+---------------+----------------------------------------+ | size | name | description | +===========+===============+========================================+ | uint8_t | msg_type | Determines the kind of message this is | | | | 0 means *request metadata* | +-----------+---------------+----------------------------------------+ | uint8_t | start | The start of the metadata block that | | | | is requested. It is given in 256ths | | | | of the total size of the metadata, | | | | since the requesting client don't know | | | | the size of the metadata. | +-----------+---------------+----------------------------------------+ | uint8_t | size | The size of the metadata block that is | | | | requested. This is also given in | | | | 256ths of the total size of the | | | | metadata. The size is given as size-1. | | | | That means that if this field is set | | | | 0, the request wants one 256:th of the | | | | metadata. | +-----------+---------------+----------------------------------------+ metadata: +-----------+---------------+----------------------------------------+ | size | name | description | +===========+===============+========================================+ | uint8_t | msg_type | 1 means *metadata* | +-----------+---------------+----------------------------------------+ | int32_t | total_size | The total size of the metadata, given | | | | in number of bytes. | +-----------+---------------+----------------------------------------+ | int32_t | offset | The offset of where the metadata block | | | | in this message belongs in the final | | | | metadata. This is given in bytes. | +-----------+---------------+----------------------------------------+ | uint8_t[] | metadata | The actual metadata block. The size of | | | | this part is given implicit by the | | | | length prefix in the bittorrent | | | | protocol packet. | +-----------+---------------+----------------------------------------+ Don't have metadata: +-----------+---------------+----------------------------------------+ | size | name | description | +===========+===============+========================================+ | uint8_t | msg_type | 2 means "I don't have metadata". | | | | This message is sent as a reply to a | | | | metadata request if the the client | | | | doesn't have any metadata. | +-----------+---------------+----------------------------------------+ .. _`BEP 9`: https://www.bittorrent.org/beps/bep_0009.html dont_have --------- Extension name: "lt_donthave" The ``dont_have`` extension message is used to tell peers that the client no longer has a specific piece. The extension message should be advertised in the ``m`` dictionary as ``lt_donthave``. The message format mimics the regular ``HAVE`` bittorrent message. Just like all extension messages, the first 2 bytes in the message itself are 20 (the bittorrent extension message) and the message ID assigned to this extension in the ``m`` dictionary in the handshake. +-----------+---------------+----------------------------------------+ | size | name | description | +===========+===============+========================================+ | uint32_t | piece | index of the piece the peer no longer | | | | has. | +-----------+---------------+----------------------------------------+ The length of this message (including the extension message prefix) is 6 bytes, i.e. one byte longer than the normal ``HAVE`` message, because of the extension message wrapping.