## File: Readme.md # Open Library [](https://github.com/internetarchive/openlibrary/graphs/contributors) [Open Library](https://openlibrary.org) is an open, editable library catalog, building towards a web page for every book ever published. ## Getting Started **Please read the [Contributing guide](https://docs.openlibrary.org/developers/CONTRIBUTING.html) thoroughly before contributing.** It covers community norms, contribution standards, and how to work respectfully with other contributors. Then follow the **[Quick Start](https://docs.openlibrary.org/developers/quick-start.html)** to set up your environment and make your first contribution. For full developer documentation, see [docs.openlibrary.org](https://docs.openlibrary.org/). To volunteer in non-development roles, visit [openlibrary.org/volunteer](https://openlibrary.org/volunteer). ## License All source code published here is available under the terms of the [GNU Affero General Public License, version 3](https://www.gnu.org/licenses/agpl-3.0.html). --- ## File: conf/twa/README.md # TWA Trusted Web Activity is a new way to open your web-app content such as your Progressive Web App (PWA) from your Android app using a protocol based on Custom Tabs. ## Bubblewrap CLI Bubblewrap is a Command Line Interface (CLI) that helps developers to create a Project for an Android application that launches an existing Progressive Web App (PWA) using a Trusted Web Activity (TWA). ## Setting up the Environment When running Bubblewrap for the first time, it will offer to automatically download and install external dependencies. ## Quickstart Guide ### Installing Bubblewrap ```shell npm i --no-audit -g @bubblewrap/cli ``` ### Initializing an Android Project Generate an Android project from an existing Web Manifest: ```shell bubblewrap init --manifest https://openlibrary.org/static/manifest.json ``` It will also ask you for the details needed to generate a signing key, used to sign the app before uploading to the Play Store. ### Building the Android Project ```shell bubblewrap build ``` --- ## File: docker/README.md # Docker Guide This guide covers everything Docker-related for local Open Library development. For the fastest path to your first contribution, start with the [Quick Start](https://docs.openlibrary.org/developers/quick-start.html) guide. ## Install Docker **Linux** users (including those using a VM) can install `Docker Engine` or `Docker Desktop` (see the [Docker FAQ](https://docs.docker.com/desktop/faqs/linuxfaqs/) to determine which is right for you). - Docker Engine: **Windows** and **macOS** users should use `Docker Desktop`. - Docker Desktop: For **macs** using an `M1 Chip`, please make sure that Docker Desktop Preferences `General / Use Docker Compose V2` is checked, enabling Docker to run without requiring `Rosetta 2`. ### Test Docker Before continuing, ensure Docker is correctly configured by running the `hello-world` container. ```sh docker run hello-world ``` The output should include a `Hello from Docker!` message. If Docker is unable to pull the `hello-world:latest` image from Docker Hub, try disabling your VPN if one is installed. Linux users, note the lack of `sudo` before `docker run hello-world`. See the [Linux post-installation instructions](https://docs.docker.com/engine/install/linux-postinstall/) if you see a permission denied error. ## Prepare your system > [!IMPORTANT] > **Windows** users should see [Fix line endings, symlinks, and git submodules](https://docs.openlibrary.org/developers/tools/git.html#fix-line-endings-symlinks-and-git-submodules-only-for-windows-users-not-using-a-linux-vm). Skipping those steps will likely prevent the containers from coming up. We recommend adjusting Docker's resource settings to use at least 4GB of RAM and 2GB of swap. The defaults of 2GB RAM and 1GB swap are likely to cause the error `Killed` when running `build-assets` to build JS dependencies. ## Build the project To build the project, `cd` into the project root directory (where `compose.yaml` is located) and run: ```sh docker compose build ``` The build process may take more than 15 minutes on older hardware or slower networks. If it times out and fails, simply re-run the command. If you encounter any build errors, please check the [Troubleshooting](#troubleshooting) section below or [browse our issues tagged with docker](https://github.com/internetarchive/openlibrary/issues?q=is%3Aissue%20label%3A%22Module%3A%20Docker%22). ## Run the app ```sh docker compose up # Ctrl-C to stop docker compose up -d # or, start in detached (silent) mode ``` You will know the environment is done loading when the text stops furiously scrolling by and you see the following repeating every few seconds on the console: ```log infobase_1 | 172.19.0.5:45716 - - [16/Feb/2023 16:54:10] "HTTP/1.1 GET /openlibrary.org/log/2023-02-16:0" - 200 OK infobase_1 | 172.19.0.5:45730 - - [16/Feb/2023 16:54:15] "HTTP/1.1 GET /openlibrary.org/log/2023-02-16:0" - 200 OK infobase_1 | 172.19.0.5:41790 - - [16/Feb/2023 16:54:20] "HTTP/1.1 GET /openlibrary.org/log/2023-02-16:0" - 200 OK ``` At this point, visit and you should see the Open Library banner image with "development version" emblazoned upon it to signify that it's running from your local development server. This exposes the following ports: | Port | Service | | ---- | ---------------------- | | 8080 | Open Library main site | | 7000 | Infobase | | 8983 | Solr | | 7075 | Cover store | For example, to access Solr admin, go to ## Code Updates > [!TIP] > When the Open Library code within your local docker image falls behind the code upstream, you can typically `git pull` the latest changes from GitHub without having to rebuild all your docker image(s). These instructions are for contributors who have **already** built Open Library using Docker, but may have made or pulled changes to their codebase that conflict with what's baked into the image. Building the Open Library project can take a while and most of its images don't change often, so we only want to rebuild them when necessary. When a docker image is created, its dependencies (`pip`, `npm`, `apt-get`) are frozen in time and will be inherited by any container based from it. Occasionally, there will be cases when changes we pull or make to our codebase will require changes to our docker environment -- for instance, the addition of a python dependency, npm package, or changes to our js/vue/css that require assets to be rebuilt. In most of these cases, because the `openlibrary` repo is volume mounted, we can use the pattern `docker compose run --rm home ` to spin up a temporary container with instructions that run build commands to ensure our environment has all the necessary dependencies. For instance, if changes have been made to: | Change | Fix | | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **`pip` python package dependencies** (e.g. `requirements.txt`) | rebuild just the `home` image with: `docker compose build home` | | **`npm` packages** | run `docker compose run --rm home npm install --no-audit` (see [#2032](https://github.com/internetarchive/openlibrary/issues/2032) for why) | | **`js`, `vue`, `css`** and other static front-end assets | perform a complete rebuild of assets using `docker compose run --rm home npm run build-assets`, or follow the [Frontend Guide](https://docs.openlibrary.org/developers/frontend/frontend-guide.html) to rebuild only certain asset types. | > [!WARNING] > In the off-chance you find yourself needing to edit core dependencies like `apt-get` or networking, you will most likely need to do a [full rebuild](#fully-resetting-your-environment). If you are making this sort of change, you will know exactly what you are doing. See [Developing the Dockerfile](#developing-the-dockerfile). > [!TIP] > In addition to using `docker compose run --rm home ` to spin up a temporary container to produce an artifact, you can also use `docker compose exec -it home ` to run the fix directly on the running container. This may also allow you to test one-off changes that might otherwise require a [full rebuild](#fully-resetting-your-environment). Use caution with this approach as you may end up breaking your container and causing a restart loop. ## Teardown commands ```sh cd path/to/your/cloned/openlibrary # stop the app (if started in detached mode) docker compose down # start specific service docker compose up --no-deps -d solr # remove all volumes (i.e. reset all databases); perform WHILE RUNNING docker compose stop docker compose rm -v ``` ## Fully Resetting Your Environment Been away for a while? Are you getting strange errors you weren't getting before? Sometimes changes are made to the docker configs which could cause your local environment to break. To do a full reset, run the following script from the root of the repo: [`scripts/fully_reset_environment.sh`](https://github.com/internetarchive/openlibrary/blob/master/scripts/fully_reset_environment.sh) ```sh bash scripts/fully_reset_environment.sh ``` ## Troubleshooting ### "openlibrary-web-1 | exec docker/ol-web-start.sh: no such file or directory" OR "/usr/bin/env: 'python\r': No such file or directory" These errors may appear in different containers or with different file names, but look for file-not-found type errors where `\r` is in the filename, or there's `exec`, a path that seems to exist, and a `no such file or directory` error, e.g. `openlibrary-web-1 | exec docker/ol-web-start.sh: no such file or directory` The likely cause here is that text files were given CRLF line endings during cloning+checkout. See [Fix line endings, symlinks, and git submodules](https://docs.openlibrary.org/developers/tools/git.html#fix-line-endings-symlinks-and-git-submodules-only-for-windows-users-not-using-a-linux-vm). Note: after fixing this error, be on the lookout for another error that will appear almost immediately after running `docker compose up` that mentions something about the `role` of `openlibrary` not existing. Read on for how to address that. ### An error similar to: FATAL: role "openlibrary" does not exist Look for errors that appear very soon after running `docker compose up` that come from the `openlibrary-db-1` container when it first starts mentioning something about the `role` of `openlibrary` not existing. In this case, try the steps in [Fully Resetting Your Environment](#fully-resetting-your-environment). Note: please update this README with the exact wording of the error if you run into it. ### "OSError: [Errno 12] Cannot allocate memory: '/openlibrary/openlibrary/core'" `OSError: [Errno 12] Cannot allocate memory:` could occur in conjunction with `/openlibrary/openlibrary/core` or any number of files. Simply try increasing free RAM or increasing swap/page file/virtual memory for your operating system. ### "No module named 'infogami'" The following should populate the target of the `infogami` symbolic link (i.e. `vendor/infogami/`): ```sh cd path/to/your/cloned/openlibrary git submodule init; git submodule sync; git submodule update ``` Windows users may need to see [Fix line endings, symlinks, and git submodules](https://docs.openlibrary.org/developers/tools/git.html#fix-line-endings-symlinks-and-git-submodules-only-for-windows-users-not-using-a-linux-vm). ### "no configuration file provided: not found" when running `docker compose ` Ensure you're running `docker compose` commands from within the `local-openlibrary-dev-directory`. ### ConnectionError: HTTPConnectionPool(host='solr', port=8983) The full error is something like (line breaks added): ```log /openlibrary/openlibrary/templates/home/index.html: error in processing template: ConnectionError: HTTPConnectionPool(host='solr', port=8983): Max retries exceeded with url: /solr/openlibrary/select (Caused by NameResolutionError(": Failed to resolve 'solr' ([Errno -2] Name or service not known)")) (falling back to default template) ``` The following should get everything running again: ```sh docker compose down docker container ls -a # If you see any openlibrary containers here, remove them with `docker rm -f NAME` docker network ls # If you see any open library networks here, remove them with `docker network rm NAME` docker compose up # or docker compose up -d ``` If you're curious and want to understand what happened, and why the above likely fixes it, first, verify the `solr` container is running (e.g. `docker ps | grep solr`, and then look for something like `openlibrary-solr-1` that isn't `solr-updater`.) If the `solr` container isn't running, simply start it with `docker compose up solr` (or `docker compose up -d solr`) and that should fix it. If `solr` is running, verify too that you can also connect to solr at . If you can't, something else is likely wrong. If the `solr` container is running and the error persists, one cause seems to be that the containers sometimes become disconnected from `openlibrary_webnet` (though this could happen with `openlibrary_dbnet` too). `openlibrary-web-1`/`web` should be connected to both `openlibrary_webnet` and `openlibrary_dbnet`, but when this problem occurs, instead only one is connected. E.g.: ```sh docker container inspect --format '{{.NetworkSettings.Networks}}' openlibrary-web-1 # output: map[openlibrary_dbnet:0xc00037c1c0] ``` Because you've read this far, you can now directly fix the problem without removing the containers and networks. Simply reconnect the container to the network: ```sh docker network connect openlibrary_webnet openlibrary-web-1 # or `openlibrary_dbnet` as the case may be. docker container inspect --format '{{.NetworkSettings.Networks}}' openlibrary-web-1 # output: map[openlibrary_dbnet:0xc00016c460 openlibrary_webnet:0xc00016c540] ``` 11874/hotfix/add-selinux-relabel-bind-mount-option-on-selinux-compose No restart is required. If `webnet` no longer exists, recreating it _should_ fix things: `docker network create openlibrary_webnet`. To understand a bit more about what's going on here, there are docker networks configured in `compose.yaml`. The containers should be able to resolve one another based on the container names (e.g. `web` and `solr`), assuming `compose.yaml` has them on the same netork. For more, see [Networking in Compose](https://docs.docker.com/compose/networking/). ### Permission denied errors on SELinux SELinux implements [Mandatory Access Controls (MAC)](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/virtualization_security_guide/sect-virtualization_security_guide-svirt-mac). Even if normal file permissions (user/group/other) would allow access, SELinux can still block resulting in permission denied errors. By adding the `z` or `Z` [bind mount options for Docker/Podman](https://docs.docker.com/engine/storage/bind-mounts/#configure-the-selinux-label) this allows the container process to access the files, by changing the MAC label. `compose.selinux.yaml` contains these label overrides to allow the containers to run with Docker or Podman within SELinux enabled systems. `podman-compose -f compose.yaml -f compose.override.yaml -f compose.selinux.yaml up -d` ## Technical notes master No restart is required. If `webnet` no longer exists, recreating it _should_ fix things: `docker network create openlibrary_webnet`. To understand a bit more about what's going on here, there are docker networks configured in `compose.yaml`. The containers should be able to resolve one another based on the container names (e.g. `web` and `solr`), assuming `compose.yaml` has them on the same network. For more, see [Networking in Compose](https://docs.docker.com/compose/networking/). ## Useful Runtime Commands See Docker's docs for more: ```sh # Read a service's logs (replace `web` with service name) docker compose logs web # Show all logs (onetime) docker compose logs -f --tail=10 web # Show last 10 lines and follow # Analyze a container docker compose exec web bash # Launch terminal in `web` service # Run tests docker compose run --rm home make test # Install Node.js modules (if you get an error running tests) # Important: npm jobs need to be run inside the Docker environment. docker compose run --rm home npm install --no-audit # build JS/CSS assets: docker compose run --rm home npm run build-assets ``` ```sh # Launch a temporary container and run tests docker compose run --rm home make test # Run Open Library using a local copy of Infogami for development docker compose down && \ docker compose -f compose.yaml -f compose.override.yaml -f compose.infogami-local.yaml up -d && \ docker compose logs -f --tail=10 web # In your browser, navigate to http://localhost:8080 # To test Open Library on another version of Python, modify Dockerfile.olbase and then # rebuild olbase (see above) and oldev (`docker compose build`) ``` ## Developing the Dockerfile If you need to make changes to the dependencies in Dockerfile.olbase, rebuild it with: ```sh docker build -t openlibrary/olbase:latest -f docker/Dockerfile.olbase . # 30+ min (Win10Home/Dec 2018) ``` This image is automatically rebuilt on deploy by ol-home0 and is pushed to at . If you're making changes you think might affect Docker Hub, you can create a branch starting with `docker-test`, e.g. `docker-test-py2py3` (no weird chars), to trigger a build in docker hub at e.g. `openlibrary/olbase:docker-test-py2py3`. ## Debugging and Profiling the Docker Image See [Debugging and Performance Profiling](https://docs.openlibrary.org/advanced/debugging-and-performance-profiling.html) for more information on how to attach a debugger when running in the Docker Container. --- ## File: openlibrary/catalog/README.md # Catalog This contains code that was originally part of early standalone import and catalog management tools which have now been integrated into Open Library. * `add_book` contains the main code used when books are imported into Open Library via `/api/import`. * `marc` contains current and some legacy code used to parse binary and XML MARC records for import and display. * `utils` contains an assortment of helper methods, many of which are legacy and unused. Current [openlibrary/solr](../../openlibrary/solr) code still makes use of some parts. * `get_ia.py` contains `get_marc_record_from_ia()` which is the main method used to read MARC records stored on archive.org. --- ## File: openlibrary/components/dev/README.md This folder contains content exclusively for local development purposes. It does not impact production builds. --- ## File: openlibrary/components/README.md Files in the root of this directory are compiled and added to `/static/build/components/`. These components can be rendered from template files like so: ```html $:render_component('HelloWorld', attrs=dict(name="Jimmy")) ``` The building of these files happens on `make components`. Every top-level `.vue` file is built at once into its own `ol-{ComponentName}.js` bundle. To rebuild all components automatically on change during local development, run: ```shell script npm run watch:components ``` Inside Docker this becomes `docker compose run --rm home npm run watch:components`. ## Live-reloading dev server The Vue components follow the following structure: ``` openlibrary/components/{MainComponent}.vue -- The entrypoint to the component openlibrary/components/{MainComponent}/... -- Any sub components, utils, etc. ``` **Outside the docker environment**, run: This way you can have a completely isolated component with hot reloading and easy to access without clicking to the exact page on the localhost you want to use the component on. ```shell script npm install --no-audit COMPONENT="HelloWorld" npm run serve # Or COMPONENT="LibraryExplorer" npm run serve # Then open http://localhost:5173/?ol_base=openlibrary.org ``` Changing `HelloWorld` to be the name of the main component you want to work on. For apps that are configured for it (like `LibraryExplorer` and `MergeUI`), when run in this mode, the vue server will make requests to production openlibrary.org for data like books, search results, covers, etc. You can configure where it fetches data from by setting url parameters on the running app, eg `?ol_base=http://localhost:8080`. See `openlibrary/components/configs.js` for all the available url parameters. ## Caveats - Vue is currently included with each component, so rendering multiple components per page results in very large load sizes - JSON attributes currently don't work - If Vue is embedded within a `
` input elements created by vue won't be picked up on form submission. This seems to be related to Vue's use of shadow dom. For a workaround, see the code in [#5093](https://github.com/internetarchive/openlibrary/pull/5093). ## Examples For an example of using Vue on existing pages see `openlibrary/components/IdentifiersInput.vue` --- ## File: openlibrary/coverstore/README.md # Coverstore README ## Where are covers archived? * Covers 0 - 7,139,999 are stored in `zip` files within items https://archive.org/download/olcovers1 - https://archive.org/download/olcovers713 in the https://archive.org/details/ol_exports collection * Covers 8,000,000 - 8,819,999 live in `tar` files within the https://archive.org/details/covers_0008 item * Covers 8,820,000 - 8,829,999 live in a `zip` file also in the https://archive.org/details/covers_0008 item ## Warnings As of 2022-11 there were 5,692,598 unarchived covers on ol-covers0 and archival hadn't occurred since 2014-11-29. This 5.7M number is sufficiently large that running `/openlibrary/openlibrary/coverstore/archive.py` `archive()` is still hanging after 5 minutes when trying to query for all unarchived covers. As a result, it is recommended to adjust the cover query for unarchived items within archive.py to batch using some limit e.g. 1000. Also note that an initial `id` is specified (which is the last known successfully archived ID in `2014-11-29`): ``` covers = _db.select('cover', where='archived=$f and id>6708293', order='id', vars={'f': False}, limit=1000) ``` # How to run Covers Archival First, `ssh -A ol-covers0` and run `docker exec -it openlibrary_covers_1 bash`. Next, launch a python terminal and run: ``` from openlibrary.coverstore import config from openlibrary.coverstore.server import load_config from openlibrary.coverstore import archive load_config("/olsystem/etc/coverstore.yml") archive.archive(test=False) ``` # How it works As of 2022-11, the way coverstore works is that new covers that are uploaded to Open Library go into `/1/var/lib/openlibrary/coverstore/localdisk/` within a directory named `/YYYY/MM/DD/`. A record for each cover (and its size variants) is recorded within the `cover` table of the `coverstore` psql db located on `ol-db1`. At some (presumably advantageous if) regular interval, as the `localdisk` fills, the files can undergo archival, a process whereby covers are compressed and bundled into tar archives which are moved into the `/1/var/lib/openlibrary/coverstore/items/` directory within folders called "staging items" (e.g. `covers_0007`). The database reference to these covers' filename paths are updated accordingly by the `archive.py` script. Mek speculates that when coverstore attempts to look up a cover, its entry is looked up in the DB and if the filename is a tar, coverstore first looks on disk for a "staging item" folder within the staging directory `/1/var/lib/openlibrary/coverstore/items/` and if no such "staging item" exists, the staging item is assumed to have been uploaded as an archive.org item having the same name (and thus redirects/resolves its request via archive.org). # State of Cover Archival As of 2022-11 there are 5,692,598 unarchived covers on `ol-covers0` and we're starting to run short on space. Specifically, cover archives haven't been happening since ~2014-11-29, as we can see from the following brutally slow query: ``` coverstore=# select id, olid, filename, last_modified from cover where archived=true order by id desc limit 1; id | olid | filename | last_modified ---------+-------------+--------------------------------------+---------------------------- 7315539 | OL25645665M | covers_0007_31.tar:1849729536:247493 | 2014-11-29 22:34:37.329315 ``` In the previous query, we see that the last cover (id #7,315,539) was archived on `2014-11-29` and resides within a tar `covers_0007_31.tar`. Coverstore assumes this tar resolves to an `item` folder called `covers_0007`, either staged on disk within `/1/var/lib/openlibrary/coverstore/items/` or on archive.org/details/covers_0007. In this case, at the time of writing, this item was still staged on disk. As far as Mek can tell, staged items presumably get manually uploaded to archive.org under an item having the same name. The item name itself (e.g. `coverd_0007`) is a combination of the prefix `covers` and the code `web.numify("%010d.jpg" % cover.id)[:4]` where, in this case, `cover.id` is `7315539`. The `"%010d"` format parameter pads the `cover.id` with leading 0's until it is 10 digits long and then the [:4] takes the first 4 digits of this padded number. Anything lower than `cover.id` 1,000,000 will thus be in `covers_0000` and from there the next 1M will be in `covers_0002` and so on. In total, this scheme allows for just under 10B covers before it breaks, which is a sufficiently unlikely number to hit! 2022-12-03: Anand says: "The cover id is considered to be 10 digits, 4 digits go to items, 2 digits go to tar file and the remaining 4 go to the filename." **NB**: We identified **unarchived** covers (denoted with `archived=false` within the `covers` table) prior to `2014-11-29` but early tests suggest the archive process may not have been ironed out and standardized before this date, and so we decided to use the latest successful archival date to resume our archival efforts. ## Archival Process **Recipe for moving one batch of 10k covers at a time into tars on archive.org.** 1. On ol-covers0 docker container, run archive.py on ~10k items to create a new partial of unarchived covers, starting at stable ID 8M (e.g. `covers_0008_00`) ``` from openlibrary.coverstore import config from openlibrary.coverstore.server import load_config from openlibrary.coverstore import archive load_config("/olsystem/etc/coverstore.yml") archive.archive(test=False) ``` 2. `ia upload` each partial to the 4 respective items: * `covers_0008` -> `covers_0008_00.index` and `covers_0008_00.tar` * `s_covers_0008` -> `s_covers_0008_00.index` and `s_covers_0008_00.tar` * `m_covers_0008` -> `m_covers_0008_00.index` and `m_covers_0008_00.tar` * `l_covers_0008` -> `l_covers_0008_00.index` and `l_covers_0008_00.tar` 3. Update the upper bound value in code.py ~L290 by +10k (on `ol-covers0` container 1 & 2 + restart) * `if (8100000 > int(value) >= 8000000):` (or whatever is the upper bound) ... 4. Restart the containers + test to make sure the service is resolving to archive.org for all sizes 5. Remove only the completed partial (e.g. 00 from each folder on /1/var/lib/openlibrary/coverstore/items/ * `rm /1/var/lib/openlibrary/coverstore/items/cover_0008/covers_0008_00.*` * `rm /1/var/lib/openlibrary/coverstore/items/s_cover_0008/s_covers_0008_00.*` * `rm /1/var/lib/openlibrary/coverstore/items/m_cover_0008/m_covers_0008_00.*` * `rm /1/var/lib/openlibrary/coverstore/items/l_cover_0008/l_covers_0008_00.*` --- ## File: openlibrary/plugins/README.md # Plugins The plugins directory defines and registers all the Routers responsible for responding to requests to the Open Library website. ## Understanding Routers In Open Library, when a patron navigates to an openlibrary.org url in their browser, the request is typically captured by a Router class which is defined by files within the `plugins/` directory. Any Router class will be responsible for doing 4 things: 1. Defining a `path` -- a regular expression pattern used to decide whether a patron's requests should be handled by this Router 2. Implementing necessary class methods (e.g. GET and POST) corresponding to the patron's HTTP request type 3. Making calls to the right data `models` 4. Prepare data to be returned (which could be e.g. a rendered template or json) -- this may include setting any headers or content-types necessary for this data The first step is to figure out where within the `plugins/` directory your new route should live and whether it best lives in an existing file or a new file. An overview of the different plugin types are discussed in [this technical overview video](https://archive.org/embed/openlibrary-tour-2020/technical_overview.mp4?start=1017). ## Understanding Directories The following describes what each plugin directory does, by convention, in decreasing order of presumed relevance: * openlibrary -- most public pages meant to be read/consumed live here * `code.py` defines the `/isbn` endpoint and what python functions our html templates have access to! * `js/` is where our javascript lives * `api.py` is many of our APIs are defined * `lists.py` defines many of the routes for the List feature * upstream -- handles incoming stateful data requests (wiki edits, uploads, adding a book, logins, etc) * responds to actions (e.g. borrowing) and account related requests: editing, mybooks pages, account login, settings * worksearch -- renders our `/search/*` related pages and all our search querying logic. * `code.py` contains most of the logic and routes for `/search`, the `/barcodescanner`, `/advancedsearch`, `/search/lists`, `/search/subjects`, etc, as well as their json returning counterparts * importapi -- defines our book import API endpoints * `code.py` again defines most of the core logic we care about * admin -- our admin pages and logic * books -- old-style book APIs (likely not relevant) ## Code.py Almost every `plugins/` directory defines a central `code.py` which is similar to `__init__.py` in that it contains the logic required to register its directory's plugins with the system. In addition to acting as an entry point for all the other plugins in the directory and performing setup(), it also may define its own Routers. If you're not sure where in a plugin to look for a Router, that's likely a good place to start. ## Tutorial: Implementing a new Route There are two ways to implement a new Open Library Route. The first way is to use your discretion to add a new Route to the correct pre-existing plugin file. The second way is to create a new directory and/or file which will require an additional step of registering your plugin with Open Library. We'll only walkthrough the first case here -- for those who need to register a new plugin, please see `openlibrary/plugins/openlibrary/code.py` for an [example](https://github.com/internetarchive/openlibrary/blob/208fe9de10f24d1f54b691f4a40d1dc4b0148745/openlibrary/plugins/openlibrary/code.py#L1105-L1143). Let's say I wanted to create a page on Open Library called `/search/many` for a search experience that lets patrons search for multiple books in bulk. The first thing I'd do is make a determination of where it should live -- likely in `worksearch/code.py`. We'd then define our Route as follows: ``` class search_many(delegate.page): path = '/search/many' def GET(self): i = web.input(q='') return render_template('search/many', q=i.q) ``` Every Router is a class that extends something like `delegate.page`. The Router will define a `GET`, `POST`, etc, as appropriate to deal with corresponding HTTP requests of that type from the patron. The function parameters defined by the GET and POST of the Router depend on and are linked to the regex pattern specified by the Router's `path`. The following example from [plugins/openlibrary/api.py](https://github.com/internetarchive/openlibrary/blob/master/openlibrary/plugins/openlibrary/api.py#L65-L71), The `trending_books_api` Router defines a `path` with a regex `"/trending(/?.*)"` that captures everything after `/trending/` and forwards it into the GET function as the parameter `period`. In this example, if a patron goes to `/trending/daily`, the request will match the Router's path regex and the Router will call the GET method with `period=daily` as an argument. ``` class trending_books_api(delegate.page): path = "/trending(/?.*)" # path = "/trending/(now|daily|weekly|monthly|yearly|forever)" encoding = "json" def GET(self, period="/daily"): ``` In our first example `i = web.input(q='')` is used to access the GET query parameters specified by the patron's request from the URL (e.g. if the patron queries for /search/many?q=test then `i.q` will be "test". The final line of the Router in our example makes a call to `render_template` which will fetch the corresponding template, by name, from the `templates/` directory (specified by the 1st argument -- in this case the file "templates/search/many.html) and then pass in any of the variables this template needs to render (in this case the value `q`). The values passed in to the template must also be defined in the header of the HTML template within the `$ def with(...)` line. For more information on rendering templates, refer to the [Front-end Guide](https://docs.openlibrary.org/developers/frontend/frontend-guide.html#routing-and-templates) in our docs. In order to get this example to work, you will first need to create a new file `templates/search/many.html` with the following body content: ``` $ def with (q="") Your query was $(q)! ``` You can now restart your server and test that your application is working by going to `/search/many?q=test&debug=true`, as described in our [debugging website errors video](https://archive.org/embed/openlibrary-tour-2020/openlibrary-debugging-webpage-errors.mp4) --- ## File: scripts/gh_scripts/README.md # GitHub Project Management Scripts This directory contains scripts that the Open Library team uses to interact with this GitHub repository. ## `new_pr_labeler.mjs` This script is used to automatically prioritize and assign a new PR to a lead. ### Usage: This script takes three positional arguments, followed by a vararg for the PR's body: **node new_pr_labeler.mjs** repository pr_author pr_number pr_body... `repository` the repository owner and name (ex. `internetarchive/openlibrary`) `pr_author` the GitHub username of the PR's author `pr_number` the PR's identification number `pr_body...` the body of the PR ### Details: The script does a case-insensitive search for a `closes #` statement in the newly created PR's body. If one is found, the corresponding issue is fetched. The first https://github.com/internetarchive/openlibrary/labels/Priority%3A%200, https://github.com/internetarchive/openlibrary/labels/Priority%3A%201, or https://github.com/internetarchive/openlibrary/labels/Priority%3A%202 label that is found on the issue is added to the PR. If a lead is assigned to the issue, and the lead is not also the PR's author, the lead will be assigned to the PR. Similar to the priority labels, only one lead can be assigned to a PR. ## `issue_comment_bot.py` This script fetches issues that have new comments from contributors within the past number of hours, then posts a message to the team in our Slack channel. ### Usage: ``` positional arguments: hours Fetch issues that have been updated since this many hours ago options: -h, --help show this help message and exit -c CONFIG, --config CONFIG Path to configuration file -s SLACK_CHANNEL, --slack-channel SLACK_CHANNEL Issues will be published to this Slack channel -t SLACK_TOKEN, --slack-token SLACK_TOKEN Slack auth token --no-labels Prevent the script from labeling the issues -v, --verbose Print detailed information about the issues that were found ``` #### Configuration A configuration file is required for this script to run properly. The file should contain a JSON string with the following fields: `leads` : Array of configurations for each lead. `leads.githubUsername` : The lead's GitHub username. `leads.leadLabel`: Text of the lead's `Lead: @` label. `leads.slackId`: The lead's Slack ID in their `mrkdwn` format, which is used to trigger Slack __Running the script locally:__ ``` docker compose exec -e PYTHONPATH=. web bash # Publish digest of new comments from the past day to #openlibrary-g: ./scripts/gh_scripts/issue_comment_bot.py 24 -s "#openlibrary-g" -t "replace-with-slack-token" -c "path/to/config/file" ``` __Note:__ When adding arguments, be sure to place any hyphenated values within double quotes. ## `weekly_status_report.mjs` This script prepares a digest of information helpful to leads, and publishes the digest to Slack. ### Usage **node weekly_status_report.mjs** config_filepath `config_filepath` the location of the script's configuration file A `SLACK_TOKEN` must be included as an environment variable in order to `POST` to Slack. Additionally, a `GITHUB_TOKEN` should also be added as an environment variable. `@octokit/action` adds this to the `octokit` object during instantiation, allowing `octokit` to make authenticated requests to GitHub. #### Configuration A configuration file is required for this script to run properly. The file should contain a JSON string with the following fields: `slackChannel` : The digest will be published here. `publishFullDigest` : Boolean that flags whether to publish a full or partial digest. If `false`, the digest will be published without several sections (see **Details**, below). `leads` : Array of configurations for each lead. `leads.githubUsername` : The lead's GitHub username. `leads.leadLabel`: Text of the lead's `Lead: @` label. `leads.slackId`: The lead's Slack ID in their `mrkdwn` format, which is used to trigger Slack notifications when the message is published. ### Details The script prepares a digest containing the following sections: *Recent comments* : A list of links to issues that need comments, broken down by lead. *Needs: Lead/Assignee* : Lists of pull requests that do not have an assignee, and issues that need a lead. Only present if `publishFullDigest` is `true`. *Untriaged issues* : List of issues which have the https://github.com/internetarchive/openlibrary/labels/Needs%3A%20Triage label. Only present if `publishFullDigest` is `true`. *Assigned PRs* : List of pull requests that have been assigned, broken down by lead. Links to higher priority PRs are also included here. *Staff PRs* : List of all open staff PRs. Only present if `publishFullDigest` is `true`. *Submitter Input for PRs* : List of PRs that are labeled https://github.com/internetarchive/openlibrary/labels/Needs%3A%20Submitter%20Input, broken down by leads. --- ## File: scripts/solr_builder/README.md ## Solr reindex from OL dump This goes through building a solr instance from a dump file. To build the reindex (only tested on the OJF environment): ### Steps 1. Start Jenkins (altering the details as necessary): ```bash docker run \ -u root \ -d \ -p 8080:8080 \ -p 50000:50000 \ -v jenkins-data:/var/jenkins_home \ -v /storage:/storage \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /var/lib/docker/volumes/jenkins-data:/var/lib/docker/volumes/jenkins-data \ --restart always \ --name jenkins \ jenkinsci/blueocean ``` 2. Follow the steps here to finish setting up Jenkins: https://jenkins.io/doc/book/installing/#setup-wizard 3. Follow these steps to create a new Pipeline Project using git: https://jenkins.io/doc/book/pipeline/getting-started/#defining-a-pipeline-in-scm - Be sure to set the script path to the `Jenkinsfile` in this directory (e.g. `scripts/solr_builder/Jenkinsfile`) 4. Run the pipeline! Notes: - Jenkins has a classic UI and a new, Blue Ocean UI. Each has its benefits. - Logs are stored as artifacts after the jobs are finished, but you can also access the logs by clicking on "workspaces" in the classic UI and navigating to the logs directory (you can usually ignore the paths containing `@`). - Although it _looks_ like we are starting a Jenkins docker image, which starts a building docker image, which starts the `solr_builder` docker images, because we are forwarding the docker socket, everything is actually happening with the host docker (this is best practice). This has some caveats: - You should not assume you have full control of what's running; if you're not careful you will stop other containers - You should not try to run multiple re-indexes at the same time. This might be possible, but the current pipeline makes assumptions which would cause this to error ### Possible Issues #### The SCM field is blank / has no options Per the "Defining A Pipeline in SCM" page above, the "Definition" section of the Pipeline should be "Pipeline script from SCM". The next step is to choose the source control system by clicking on the "SCM" field, but it might be blank. Solution: update and restart Jenkins, which can probably be done at the top of the page in the alerts section. #### Building the pipeline fails almost immediately (various errors) Check the console output and look for the errors listed below. ##### WorkflowScript: 3: Invalid agent type "dockerfile" specified ##### ``` org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: WorkflowScript: 3: Invalid agent type "dockerfile" specified. Must be one of [any, label, none] @ line 3, column 5. dockerfile { ^ ``` Cause: Missing `Docker` and `Docker Pipeline` plugins. Solution: Dashboard > Manage Jenkins > Manage Plugins. In the left-hand menu, click on "Available plugins" and search for "docker". Then install: - Docker (possibly displaying as "Docker plugin") - Docker Pipeline ##### java.lang.NoSuchMethodError: 'boolean org.kohsuke.groovy.sandbox.SandboxTransformer.mightBePositionalArgumentConstructor(org.codehaus.groovy.ast.expr.VariableExpression)' ##### ``` java.lang.NoSuchMethodError: 'boolean org.kohsuke.groovy.sandbox.SandboxTransformer.mightBePositionalArgumentConstructor(org.codehaus.groovy.ast.expr.VariableExpression)' at com.cloudbees.groovy.cps.SandboxCpsTransformer.visitAssignmentOrCast(SandboxCpsTransformer.java:93) ``` Cause: Out of date Groovy Pipeline plugin: Solution: Dashboard > Manage Jenkins > Manage Plugins. In the left-hand menu, click on "Installed plugins", search for the "Groovy Pipeline" plugin and update it. ### Editing the Jenkins Pipeline - If you want to modify the pipeline, you can set the Jenkin's Project to use a local path. Note that you will have to restart Jenkins so that it mounts the path (i.e. `-v "$HOME:/home"`). Alternatively, you can also change the Git url to that of your fork, and specify a branch (e.g. `*/solr-builder--jenkins`) and push to the branch whenever you want to test your code. Jenkins will pull from the branch when you run the project. - Use VS Code to write your Pipeline; it has a validation extension. See https://jenkins.io/doc/book/pipeline/development/ for other editors/tips - Read https://jenkins.io/doc/book/pipeline/development/ for general development tips ## Final Sync TODO. Something along the lines of: Add a solrupdater to compose.production.yaml that points to the new server, and set its offset to be the correct date. See [5493 Move production solr from solr1 to solr0](https://github.com/internetarchive/openlibrary/issues/5493) for hints. ## Deploy Now that the solr is ready, we can dump its database and import it into the solr on production. Here is the command to do that from the solrbuilder server: ```sh time docker run --rm \ --volumes-from solr_builder_solr-1 \ -v /tmp/solr:/backup \ ubuntu:xenial \ tar czf /backup/solrbuilder-$(date +%Y-%m-%d).tar.gz /var/solr/data ``` (Last run: 41min/14G with 2020-10 dump; OJF) Then on the production server (ol-solr0) run: ```sh cd /opt/openlibrary # Copy file from solrbuilder server (4min; 2020-11-05 ol-solr0) time scp YOU@SOLR_BUILDER_SERVER:/tmp/solr/solrbuilder-2020-03-02.tar.gz /tmp/solr/solrbuilder-2020-03-02.tar.gz # Restore backup file (7min; 2021-08-11 ol-solr0) # Note the name "openlibrary_solr-data" should match "{OL_DIR}_{SOLR_DATA_VOLUME}", where: # OL_DIR: is the name of the directory where the openlibrary repo is; likely openlibrary # SOLR_DATA_VOLUME: is the name of the volume the solr service uses; defined in compose.yaml time docker run -v openlibrary_solr-data:/var/solr/data -v /tmp/solr:/backup ubuntu:xenial \ tar xzf /backup/solrbuilder-2021-08-11.tar.gz # Start the services COMPOSE_FILE="compose.yaml:compose.production.yaml" HOSTNAME="$HOSTNAME"docker compose --profile=ol-solr0 up -d ``` ## Resetting In order to be able to re-run the job, you need to stop/remove any of the old containers you don't intend to reuse: ```sh # "new" solr containers/volumes docker rm -f -v solr_builder_solr-1 docker volume rm solr_builder_solr-data # DB containers/volumes docker rm -f -v solr_builder_db-1 solr_builder_adminer-1 docker volume rm solr_builder_postgres-data ```