## File: docs/adding-docs.md Adding a documentation may look like a daunting task but once you get the hang of it, it's actually quite simple. Don't hesitate to ask for help [in Discord](https://discord.gg/PRyKn3Vbay) if you ever get stuck. **Note:** please read the [contributing guidelines](../.github/CONTRIBUTING.md) before submitting a new documentation. 1. Create a subclass of `Docs::UrlScraper` or `Docs::FileScraper` in the `lib/docs/scrapers/` directory. Its name should be the [PascalCase](http://api.rubyonrails.org/classes/String.html#method-i-camelize) equivalent of the filename (e.g. `my_doc` → `MyDoc`) 2. Add the appropriate class attributes and filter options (see the [Scraper Reference](./scraper-reference.md) page). 3. Check that the scraper is listed in `thor docs:list`. 4. Create filters specific to the scraper in the `lib/docs/filters/[my_doc]/` directory and add them to the class's [filter stacks](./scraper-reference.md#filter-stacks). You may create any number of filters but will need at least the following two: * A [`CleanHtml`](./filter-reference.md#cleanhtmlfilter) filter whose task is to clean the HTML markup (e.g. adding `id` attributes to headings) and remove everything superfluous and/or nonessential. * An [`Entries`](./filter-reference.md#entriesfilter) filter whose task is to determine the pages' metadata (the list of entries, each with a name, type and path). The [Filter Reference](./filter-reference.md) page has all the details about filters. 5. Using the `thor docs:page [my_doc] [path]` command, check that the scraper works properly. Files will appear in the `public/docs/[my_doc]/` directory (but not inside the app as the command doesn't touch the index). `path` in this case refers to either the remote path (if using `UrlScraper`) or the local path (if using `FileScraper`). 6. Generate the full documentation using the `thor docs:generate [my_doc] --force` command. Additionally, you can use the `--verbose` option to see which files are being created/updated/deleted (useful to see what changed since the last run), and the `--debug` option to see which URLs are being requested and added to the queue (useful to pin down which page adds unwanted URLs to the queue). 7. Start the server, open the app, enable the documentation, and see how everything plays out. 8. Tweak the scraper/filters and repeat 5) and 6) until the pages and metadata are ok. Only the first run downloads the pages; the ones after that are served from the [response cache](./scraper-reference.md#response-cache), until you run `thor docs:clean`. 9. To customize the pages' styling, create an SCSS file in the `assets/stylesheets/pages/` directory and import it in `application.css.scss`. Both the file and CSS class should be named `_[type]` where [type] is equal to the scraper's `type` attribute (documentations with the same type share the same custom CSS and JS). Setting the type to `simple` will apply the general styling rules in `assets/stylesheets/pages/_simple.scss`, which can be used for documentations where little to no CSS changes are needed. 10. To add syntax highlighting or execute custom JavaScript on the pages, create a file in the `assets/javascripts/views/pages/` directory (take a look at the other files to see how it works). 11. Add the documentation's icon in the `public/icons/docs/[my_doc]/` directory, in both 16x16 and 32x32-pixels formats. The icon spritesheet is automatically generated when you (re)start your local DevDocs instance. 12. Add the documentation's copyright details to `options[:attribution]`. This is the data shown in the table on the [about](https://devdocs.io/about) page, and is ordered alphabetically. Please see an existing scraper for the typesetting. 13. Ensure `thor updates:check [my_doc]` shows the correct latest version. If the documentation includes more than a few hundreds pages and is available for download, try to scrape it locally (e.g. using `FileScraper`). It'll make the development process much faster and avoids putting too much load on the source site. (It's not a problem if your scraper is coupled to your local setup, just explain how it works in your pull request.) Finally, try to document your scraper and filters' behavior as much as possible using comments (e.g. why some URLs are ignored, HTML markup removed, metadata that way, etc.). It'll make updating the documentation much easier. --- ## File: docs/file-scrapers.md # File Scraper Reference This lists the docs that use `FileScraper` and instructions for building some of them. If you open a PR to update one of these docs, please add/fix the instructions. ## Dart Click the “API docs” link under the “Stable channel” header on https://www.dartlang.org/tools/sdk/archive. Rename the expanded ZIP to `dart~2` and put it in `docs/` Or run the following commands in your terminal: ```sh curl https://storage.googleapis.com/dart-archive/channels/stable/release/$RELEASE/api-docs/dartdocs-gen-api.zip > dartApi.zip; \ unzip dartApi.zip; mv gen-dartdocs docs/dart~$VERSION ``` ## date-fns ```sh git clone https://github.com/date-fns/date-fns docs/date_fns cd docs/date_fns git checkout v2.29.2 yarn install node scripts/build/docs.js ls tmp/docs.json ``` ## Django Go to https://docs.djangoproject.com/, select the version from the bubble in the bottom-right corner, then download the HTML version from the sidebar. ```sh mkdir --parent docs/django\~$VERSION/; \ curl https://media.djangoproject.com/docs/django-docs-$VERSION-en.zip | \ bsdtar --extract --file - --directory=docs/django\~$VERSION/ ``` ## Elisp Go to https://www.gnu.org/software/emacs/manual/elisp.html, download the HTML tarball and extract its content in `docs/elisp` or run the following command: ```sh mkdir docs/elisp \ && curl curl https://www.gnu.org/software/emacs/manual/elisp.html_node.tar.gz | \ tar --extract --gzip --strip-components=1 --directory=docs/elisp ``` ## Erlang Go to https://www.erlang.org/downloads and download the HTML documentation file. ```ah mkdir --parent docs/erlang\~$VERSION/; \ curl -L https://github.com/erlang/otp/releases/download/OTP-$RELEASE/otp_doc_html_$RELEASE.tar.gz | \ bsdtar --extract --file - --directory=docs/erlang\~$VERSION/ ``` ## es-toolkit ```sh git clone --branch v$RELEASE https://github.com/toss/es-toolkit docs/es_toolkit ``` ## Gnu ### Bash Go to https://www.gnu.org/software/bash/manual/, download the HTML tar file (with one web page per node) and extract its content in `docs/bash` or run the following command: ```sh mkdir docs/bash \ && curl https://www.gnu.org/software/bash/manual/bash.html_node.tar.gz | \ tar --extract --gzip --directory=docs/bash ``` ### GCC Go to https://gcc.gnu.org/onlinedocs/ and download the HTML tarball of GCC Manual and GCC CPP manual or run the following commands to download the tarballs: ```sh # GCC manual mkdir docs/gcc~${VERSION}; \ curl https://gcc.gnu.org/onlinedocs/gcc-$RELEASE/gcc-html.tar.gz | \ tar --extract --gzip --strip-components=1 --directory=docs/gcc~${VERSION} # GCC CPP manual mkdir docs/gcc~${VERSION}_cpp; \ curl https://gcc.gnu.org/onlinedocs/gcc-$RELEASE/cpp-html.tar.gz | \ tar --extract --gzip --strip-components=1 --directory=docs/gcc~${VERSION}_cpp ``` ### GNU Fortran Go to https://gcc.gnu.org/onlinedocs/ and download the HTML tarball of Fortran manual or run the following commands to download the tarball: ```sh mkdir docs/gnu_fortran~$VERSION; \ curl https://gcc.gnu.org/onlinedocs/gcc-$RELEASE/gfortran-html.tar.gz | \ tar --extract --gzip --strip-components=1 --directory=docs/gnu_fortran~$VERSION ``` ## GNU Make Go to https://www.gnu.org/software/make/manual/, download the HTML tarball and extract its content in `docs/gnu_make` or run the following command: ```sh mkdir docs/gnu_make \ && curl https://www.gnu.org/software/make/manual/make.html_node.tar.gz | \ tar --extract --gzip --strip-components=1 --directory=docs/gnu_make ``` ## Gnuplot The most recent release can be found near the bottom of https://sourceforge.net/p/gnuplot/gnuplot-main/ref/master/tags/ ```sh DEVDOCS_ROOT=/path/to/devdocs mkdir gnuplot-src $DEVDOCS_ROOT/docs/gnuplot git clone -b $RELEASE --depth 1 https://git.code.sf.net/p/gnuplot/gnuplot-main ./gnuplot-src cd gnuplot-src/ ./prepare ./configure cd docs/ make nofigures.tex latex2html -html 5.0,math -split 4 -link 8 -long_titles 5 -dir $DEVDOCS_ROOT/docs/gnuplot -ascii_mode -no_auto_link nofigures.tex ``` To install `latex2html` on macOS: `brew install basictex latex2html`, then edit `/usr/local/Cellar/latex2html/2019.2/l2hconf.pm` to include the path to LaTeX: On line 21 (approximately): ``` # Give the paths to latex and dvips on your system: # $LATEX = '/Library/TeX/texbin/latex'; # LaTeX $PDFLATEX = '/Library/TeX/texbin/pdflatex'; # pdfLaTeX $LUALATEX = '/Library/TeX/texbin/lualatex'; # LuaLaTeX $DVILUALATEX = '/Library/TeX/texbin/dvilualatex'; # dviLuaLaTeX $DVIPS = '/Library/TeX/texbin/dvips'; # dvips $DVIPNG = ''; # dvipng $PDFTOCAIRO = '/usr/local/bin/pdf2svg'; # pdf to svg converter $PDFCROP = ''; # pdfcrop $GS = '/usr/local/opt/ghostscript/bin/gs'; # GhostScript ``` ## Man ```sh wget --recursive --no-parent https://man7.org/linux/man-pages/ mv man7.org/linux/man-pages/ docs/man/ ``` ## NumPy ```sh mkdir --parent docs/numpy~$VERSION/; \ curl https://numpy.org/doc/$VERSION/numpy-html.zip | \ bsdtar --extract --file=- --directory=docs/numpy~$VERSION/ ``` ## OpenGL ```sh cd docs/ git clone https://github.com/KhronosGroup/OpenGL-Refpages.git ln -s OpenGL-Refpages/gl4/html/ opengl~4 ln -s OpenGL-Refpages/gl2.1/xhtml/ opengl~2.1 ``` ## OpenJDK Search 'Openjdk' in https://www.debian.org/distrib/packages, find the `openjdk-$VERSION-doc` package, download it, extract it with `dpkg -x $PACKAGE ./` and move `./usr/share/doc/openjdk-16-jre-headless/api/` to `path/to/devdocs/docs/openjdk~$VERSION` ```sh curl -O http://ftp.at.debian.org/debian/pool/main/o/openjdk-25/openjdk-25-doc_25+36-1_all.deb tar xf openjdk-25-doc_25+36-1_all.deb tar xf data.tar.xz mv ./usr/share/doc/openjdk-25-jre-headless/api/ docs/openjdk~25 ``` If you use or have access to a Debian-based GNU/Linux distribution you can run the following command: ```sh apt download openjdk-$VERSION-doc dpkg -x $PACKAGE ./ # previous command makes a directory called 'usr' in the current directory mv ./usr/share/doc/openjdk-16-jre-headless/api/ docs/openjdk~$VERSION ``` ## Pandas From the home directory; `devdocs`, execute below: ```sh curl https://pandas.pydata.org/docs/pandas.zip -o tmp.zip && unzip tmp.zip -d docs/pandas~3 && rm tmp.zip ``` ## PHP Click the link under the "Many HTML files" column on https://www.php.net/download-docs.php, extract the tarball, change its name to `php` and put it in `docs/`. Or run the following commands in your terminal: ```sh curl https://www.php.net/distributions/manual/php_manual_en.tar.gz | tar xz; mv php-chunked-xhtml/ docs/php/ ``` ## Python 3.6+ ```sh mkdir docs/python~$VERSION cd docs/python~$VERSION curl -L https://docs.python.org/$VERSION/archives/python-$RELEASE-docs-html.tar.bz2 | \ tar xj --strip-components=1 ``` ## Python < 3.6 ```sh mkdir docs/python~$VERSION cd docs/python~$VERSION curl -L https://docs.python.org/ftp/python/doc/$RELEASE/python-$RELEASE-docs-html.tar.bz2 | \ tar xj --strip-components=1 ``` ## R ```bash sudo dnf install bzip2-devel sudo dnf install gcc-gfortran sudo dnf install libcurl-devel sudo dnf install texinfo sudo dnf install xz-devel DEVDOCSROOT=docs/r RLATEST=https://cran.r-project.org/src/base/R-latest.tar.gz # or /R-${VERSION::1}/R-$VERSION.tar.gz RSOURCEDIR=${TMPDIR:-/tmp}/R/latest RBUILDDIR=${TMPDIR:-/tmp}/R/build mkdir -p "$RSOURCEDIR" "$RBUILDDIR" "$DEVDOCSROOT" # Download, configure, and build with static HTML pages curl "$RLATEST" | tar -C "$RSOURCEDIR" -xzf - --strip-components=1 (cd "$RBUILDDIR" && "$RSOURCEDIR/configure" --enable-prebuilt-html --with-recommended-packages --disable-byte-compiled-packages --disable-shared --disable-java --with-readline=no --with-x=no) make _R_HELP_LINKS_TO_TOPICS_=FALSE -C "$RBUILDDIR" # Export all html documentation built − global, and per-package cp -r "$RBUILDDIR/doc" "$DEVDOCSROOT/" ls -d "$RBUILDDIR"/library/*/html | while read orig; do dest="$DEVDOCSROOT${orig#$RBUILDDIR}" mkdir -p "$dest" && cp -r "$orig"/* "$dest/" done ``` ## RDoc ### Nokogiri ### Ruby / Minitest ```sh git clone https://github.com/minitest/minitest cd minitest/ echo -e "source 'https://rubygems.org'\n\ngem 'hoe'\ngem 'rdoc', '< 7'" > Gemfile bundle install bundle exec rake docs cp -r docs $DEVDOCS/docs/minitest ``` ### Ruby / Rack ```sh git clone https://github.com/rack/rack cd rack/ sed -i 's/gem "rdoc"/gem "rdoc", "<7"/' Gemfile bundle install bundle exec rdoc cp -r doc $DEVDOCS/docs/rack ``` ### Ruby on Rails * Run `git clone --branch v$RELEASE --depth 7 https://github.com/rails/rails.git && cd rails` * Open `railties/lib/rails/api/task.rb` and comment out any code related to sdoc (`configure_sdoc`) * Run `bundle config set --local without 'db job'` (in the Rails directory) * Run `bundle install && bundle exec rake rdoc` (in the Rails directory) * Run `cd guides && bundle exec rake guides:generate:html && cd ..` * Run `cp -r guides/output html/guides` * Run `cp -r html $DEVDOCS/docs/rails~$VERSION` ### Ruby Download the tarball of Ruby from https://www.ruby-lang.org/en/downloads/, extract it, run `./configure && make html` in your terminal (while your are in the ruby directory) and move `.ext/html` to `path/to/devdocs/docs/ruby~$VERSION/`. Or run the following commands in your terminal: ```sh curl https://cache.ruby-lang.org/pub/ruby/$VERSION/ruby-$RELEASE.tar.gz > ruby.tar; \ tar -xf ruby.tar; cd ruby-$RELEASE; ./configure && make html; mv .ext/html path/to/devdocs/docs/ruby~$VERSION ``` To generate the htmls file you have to run `make` command but it does not install Ruby in your system, only generates html files so you have not to worry about cleaning or removing a new Ruby installation. ## Scala See `lib/docs/scrapers/scala.rb` ## SQLite Download the docs from https://sqlite.org/download.html, unzip it, and rename it to `docs/sqlite` ```sh curl https://sqlite.org/2022/sqlite-doc-3400000.zip | bsdtar --extract --file - --directory=docs/sqlite/ --strip-components=1 ``` ## Three.js Download the docs from https://github.com/mrdoob/three.js/tree/dev/files or run the following commands in your terminal: Make sure to set the version per the release tag (e.g. r160). Note that the r prefix is already included, only the version number is needed. ```sh curl https://codeload.github.com/mrdoob/three.js/tar.gz/refs/tags/r${VERSION} > threejs.tar.gz tar -xzf threejs.tar.gz mkdir -p docs/threejs~${VERSION} mv three.js-r${VERSION}/list.json tmp/list.json mv three.js-r${VERSION}/docs/* docs/threejs~${VERSION}/ rm -rf three.js-r${VERSION}/ rm threejs.tar.gz ``` ## PowerShell ```sh curl -L 'https://github.com/MicrosoftDocs/PowerShell-Docs/archive/refs/heads/main.zip' | bsdtar --extract --file - --directory=docs/powershell/ --strip-components=2 ``` --- ## File: docs/filter-reference.md **Table of contents:** * [Overview](#overview) * [Instance methods](#instance-methods) * [Core filters](#core-filters) * [Custom filters](#custom-filters) - [CleanHtmlFilter](#cleanhtmlfilter) - [EntriesFilter](#entriesfilter) ## Overview Filters use the [HTML::Pipeline](https://github.com/jch/html-pipeline) library. They take an HTML string or [Nokogiri](http://nokogiri.org/) node as input, optionally perform modifications and/or extract information from it, and then outputs the result. Together they form a pipeline where each filter hands its output to the next filter's input. Every documentation page passes through this pipeline before being copied on the local filesystem. Filters are subclasses of the [`Docs::Filter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/core/filter.rb) class and require a `call` method. A basic implementation looks like this: ```ruby module Docs class CustomFilter < Filter def call doc end end end ``` Filters which manipulate the Nokogiri node object (`doc` and related methods) are _HTML filters_ and must not manipulate the HTML string (`html`). Vice-versa, filters which manipulate the string representation of the document are _text filters_ and must not manipulate the Nokogiri node object. The two types are divided into two stacks within the scrapers. These stacks are then combined into a pipeline that calls the HTML filters before the text filters (more details [here](./scraper-reference.md#filter-stacks)). This is to avoid parsing the document multiple times. The `call` method must return either `doc` or `html`, depending on the type of filter. ## Instance methods * `doc` [Nokogiri::XML::Node] The Nokogiri representation of the container element. See [Nokogiri's API docs](http://www.rubydoc.info/github/sparklemotion/nokogiri/Nokogiri/XML/Node) for the list of available methods. * `html` [String] The string representation of the container element. * `context` [Hash] **(frozen)** The scraper's `options` along with a few additional keys: `:base_url`, `:root_url`, `:root_page` and `:url`. * `result` [Hash] Used to store the page's metadata and pass back information to the scraper. Possible keys: - `:path` — the page's normalized path - `:store_path` — the path where the page will be stored (equal to `:path` with `.html` at the end) - `:internal_urls` — the list of distinct internal URLs found within the page - `:entries` — the [`Entry`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/core/models/entry.rb) objects to add to the index * `css`, `at_css`, `xpath`, `at_xpath` Shortcuts for `doc.css`, `doc.xpath`, etc. * `base_url`, `current_url`, `root_url` [Docs::URL] Shortcuts for `context[:base_url]`, `context[:url]`, and `context[:root_url]` respectively. * `root_path` [String] Shortcut for `context[:root_path]`. * `subpath` [String] The sub-path from the base URL of the current URL. _Example: if `base_url` equals `example.com/docs` and `current_url` equals `example.com/docs/file?raw`, the returned value is `/file`._ * `slug` [String] The `subpath` removed of any leading slash or `.html` extension. _Example: if `subpath` equals `/dir/file.html`, the returned value is `dir/file`._ * `root_page?` [Boolean] Returns `true` if the current page is the root page. * `initial_page?` [Boolean] Returns `true` if the current page is the root page or its subpath is one of the scraper's `initial_paths`. ## Core filters * [`ContainerFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/container.rb) — changes the root node of the document (remove everything outside) * [`CleanHtmlFilter`](https://github.com/freeCodeCamp/devdocs/blob/main/lib/docs/filters/core/clean_html.rb) — removes HTML comments, `