openlibrary Documentation

File: Readme.md Copy Chapter

Open Library

Open Library is an open, editable library catalog, building towards a web page for every book ever published.

Getting Started Copy Chapter

Please read the Contributing guide thoroughly before contributing. It covers community norms, contribution standards, and how to work respectfully with other contributors.

Then follow the Quick Start to set up your environment and make your first contribution.

For full developer documentation, see docs.openlibrary.org.

To volunteer in non-development roles, visit openlibrary.org/volunteer.

License Copy Chapter

All source code published here is available under the terms of the GNU Affero General Public License, version 3.

File: conf/twa/README.md Copy Chapter

TWA

Trusted Web Activity is a new way to open your web-app content such as your Progressive Web App (PWA) from your Android app using a protocol based on Custom Tabs.

Bubblewrap CLI Copy Chapter

Bubblewrap is a Command Line Interface (CLI) that helps developers to create a Project for an Android application that launches an existing Progressive Web App (PWA) using a Trusted Web Activity (TWA).

Setting up the Environment Copy Chapter

When running Bubblewrap for the first time, it will offer to automatically download and install
external dependencies.

Quickstart Guide Copy Chapter

Installing Bubblewrap Copy Section

npm i --no-audit -g @bubblewrap/cli

Initializing an Android Project Copy Section

Generate an Android project from an existing Web Manifest:

bubblewrap init --manifest https://openlibrary.org/static/manifest.json

It will also ask you for the details needed to generate a signing key, used to sign the
app before uploading to the Play Store.

Building the Android Project Copy Section

bubblewrap build

File: docker/README.md Copy Chapter

Docker Guide

This guide covers everything Docker-related for local Open Library development. For the fastest path to your first contribution, start with the Quick Start guide.

Install Docker Copy Chapter

Linux users (including those using a VM) can install Docker Engine or Docker Desktop (see the Docker FAQ to determine which is right for you).

• Docker Engine: https://docs.docker.com/engine/install/#server

Windows and macOS users should use Docker Desktop.

• Docker Desktop: https://docs.docker.com/get-docker/

For macs using an M1 Chip, please make sure that Docker Desktop Preferences General / Use Docker Compose V2 is checked, enabling Docker to run without requiring Rosetta 2.

Test Docker Copy Section

Before continuing, ensure Docker is correctly configured by running the hello-world container.

docker run hello-world

The output should include a Hello from Docker! message. If Docker is unable to pull the hello-world:latest image from Docker Hub, try disabling your VPN if one is installed.

Linux users, note the lack of sudo before docker run hello-world. See the Linux post-installation instructions if you see a permission denied error.

Prepare your system Copy Chapter

[!IMPORTANT]
Windows users should see Fix line endings, symlinks, and git submodules. Skipping those steps will likely prevent the containers from coming up.

We recommend adjusting Docker's resource settings to use at least 4GB of RAM and 2GB of swap. The defaults of 2GB RAM and 1GB swap are likely to cause the error Killed when running build-assets to build JS dependencies.

Build the project Copy Chapter

To build the project, cd into the project root directory (where compose.yaml is located) and run:

docker compose build

The build process may take more than 15 minutes on older hardware or slower networks. If it times out and fails, simply re-run the command.

If you encounter any build errors, please check the Troubleshooting section below or browse our issues tagged with docker.

Run the app Copy Chapter

docker compose up    # Ctrl-C to stop
docker compose up -d # or, start in detached (silent) mode

You will know the environment is done loading when the text stops furiously scrolling by and you see the following repeating every few seconds on the console:

infobase_1      | 172.19.0.5:45716 - - [16/Feb/2023 16:54:10] "HTTP/1.1 GET /openlibrary.org/log/2023-02-16:0" - 200 OK
infobase_1      | 172.19.0.5:45730 - - [16/Feb/2023 16:54:15] "HTTP/1.1 GET /openlibrary.org/log/2023-02-16:0" - 200 OK
infobase_1      | 172.19.0.5:41790 - - [16/Feb/2023 16:54:20] "HTTP/1.1 GET /openlibrary.org/log/2023-02-16:0" - 200 OK

At this point, visit http://localhost:8080 and you should see the Open Library banner image with "development version" emblazoned upon it to signify that it's running from your local development server.

This exposes the following ports:

For example, to access Solr admin, go to http://localhost:8983/solr/admin/

Code Updates Copy Chapter

[!TIP]
When the Open Library code within your local docker image falls behind the code upstream, you can typically git pull the latest changes from GitHub without having to rebuild all your docker image(s).

These instructions are for contributors who have already built Open Library using Docker, but may have made or pulled changes to their codebase that conflict with what's baked into the image.

Building the Open Library project can take a while and most of its images don't change often, so we only want to rebuild them when necessary. When a docker image is created, its dependencies (pip, npm, apt-get) are frozen in time and will be inherited by any container based from it.

Occasionally, there will be cases when changes we pull or make to our codebase will require changes to our docker environment -- for instance, the addition of a python dependency, npm package, or changes to our js/vue/css that require assets to be rebuilt. In most of these cases, because the openlibrary repo is volume mounted, we can use the pattern docker compose run --rm home <command> to spin up a temporary container with instructions that run build commands to ensure our environment has all the necessary dependencies. For instance, if changes have been made to:

[!WARNING]
In the off-chance you find yourself needing to edit core dependencies like apt-get or networking, you will most likely need to do a full rebuild. If you are making this sort of change, you will know exactly what you are doing. See Developing the Dockerfile.

[!TIP]
In addition to using docker compose run --rm home <command> to spin up a temporary container to produce an artifact, you can also use docker compose exec -it home <command> to run the fix directly on the running container. This may also allow you to test one-off changes that might otherwise require a full rebuild. Use caution with this approach as you may end up breaking your container and causing a restart loop.

Teardown commands Copy Chapter

cd path/to/your/cloned/openlibrary
# stop the app (if started in detached mode)
docker compose down

# start specific service
docker compose up --no-deps -d solr

# remove all volumes (i.e. reset all databases); perform WHILE RUNNING
docker compose stop
docker compose rm -v

Fully Resetting Your Environment Copy Chapter

Been away for a while? Are you getting strange errors you weren't getting before? Sometimes changes are made to the docker configs which could cause your local environment to break.
To do a full reset, run the following script from the root of the repo:

scripts/fully_reset_environment.sh

bash scripts/fully_reset_environment.sh

Troubleshooting Copy Chapter

"openlibrary-web-1 | exec docker/ol-web-start.sh: no such file or directory" OR "/usr/bin/env: 'python\r': No such file or directory" Copy Section

These errors may appear in different containers or with different file names, but look for file-not-found type errors where \r is in the filename, or there's exec, a path that seems to exist, and a no such file or directory error, e.g. openlibrary-web-1 | exec docker/ol-web-start.sh: no such file or directory

The likely cause here is that text files were given CRLF line endings during cloning+checkout. See Fix line endings, symlinks, and git submodules.

Note: after fixing this error, be on the lookout for another error that will appear almost immediately after running docker compose up that mentions something about the role of openlibrary not existing. Read on for how to address that.

An error similar to: FATAL: role "openlibrary" does not exist Copy Section

Look for errors that appear very soon after running docker compose up that come from the openlibrary-db-1 container when it first starts mentioning something about the role of openlibrary not existing.

In this case, try the steps in Fully Resetting Your Environment.

Note: please update this README with the exact wording of the error if you run into it.

"OSError: [Errno 12] Cannot allocate memory: '/openlibrary/openlibrary/core'" Copy Section

OSError: [Errno 12] Cannot allocate memory: could occur in conjunction with /openlibrary/openlibrary/core or any number of files. Simply try increasing free RAM or increasing swap/page file/virtual memory for your operating system.

"No module named 'infogami'" Copy Section

The following should populate the target of the infogami symbolic link (i.e. vendor/infogami/):

cd path/to/your/cloned/openlibrary
git submodule init; git submodule sync; git submodule update

Windows users may need to see Fix line endings, symlinks, and git submodules.

"no configuration file provided: not found" when running `docker compose ` Copy Section

Ensure you're running docker compose commands from within the local-openlibrary-dev-directory.

ConnectionError: HTTPConnectionPool(host='solr', port=8983) Copy Section

The full error is something like (line breaks added):

/openlibrary/openlibrary/templates/home/index.html: error in processing
 template: ConnectionError: HTTPConnectionPool(host='solr', port=8983):
Max retries exceeded with url: /solr/openlibrary/select (Caused by
NameResolutionError(": Failed to resolve 'solr' ([Errno -2] Name or service
not known)")) (falling back to default template)

The following should get everything running again:

docker compose down
docker container ls -a
# If you see any openlibrary containers here, remove them with `docker rm -f NAME`
docker network ls
# If you see any open library networks here, remove them with `docker network rm NAME`
docker compose up  # or docker compose up -d

If you're curious and want to understand what happened, and why the above likely fixes it, first, verify the solr container is running (e.g. docker ps | grep solr, and then look for something like openlibrary-solr-1 that isn't solr-updater.) If the solr container isn't running, simply start it with docker compose up solr (or docker compose up -d solr) and that should fix it. If solr is running, verify too that you can also connect to solr at http://localhost:8983/solr/#/. If you can't, something else is likely wrong.

If the solr container is running and the error persists, one cause seems to be that the containers sometimes become disconnected from openlibrary_webnet (though this could happen with openlibrary_dbnet too). openlibrary-web-1/web should be connected to both openlibrary_webnet and openlibrary_dbnet, but when this problem occurs, instead only one is connected. E.g.:

docker container inspect --format '{{.NetworkSettings.Networks}}' openlibrary-web-1
# output: map[openlibrary_dbnet:0xc00037c1c0]

Because you've read this far, you can now directly fix the problem without removing the containers and networks. Simply reconnect the container to the network:

docker network connect openlibrary_webnet openlibrary-web-1  # or `openlibrary_dbnet` as the case may be.
docker container inspect --format '{{.NetworkSettings.Networks}}' openlibrary-web-1
# output: map[openlibrary_dbnet:0xc00016c460 openlibrary_webnet:0xc00016c540]

11874/hotfix/add-selinux-relabel-bind-mount-option-on-selinux-compose
No restart is required. If webnet no longer exists, recreating it should fix things: docker network create openlibrary_webnet.

To understand a bit more about what's going on here, there are docker networks configured in compose.yaml. The containers should be able to resolve one another based on the container names (e.g. web and solr), assuming compose.yaml has them on the same netork. For more, see Networking in Compose.

Permission denied errors on SELinux Copy Section

SELinux implements Mandatory Access Controls (MAC). Even if normal file permissions (user/group/other) would allow access, SELinux can still block resulting in permission denied errors. By adding the z or Z bind mount options for Docker/Podman this allows the container process to access the files, by changing the MAC label.

compose.selinux.yaml contains these label overrides to allow the containers to run with Docker or Podman within SELinux enabled systems.

podman-compose -f compose.yaml -f compose.override.yaml -f compose.selinux.yaml up -d

Technical notes Copy Chapter

master

No restart is required. If webnet no longer exists, recreating it should fix things: docker network create openlibrary_webnet.

To understand a bit more about what's going on here, there are docker networks configured in compose.yaml. The containers should be able to resolve one another based on the container names (e.g. web and solr), assuming compose.yaml has them on the same network. For more, see Networking in Compose.

Useful Runtime Commands Copy Chapter

See Docker's docs for more: https://docs.docker.com/compose/reference/overview

# Read a service's logs (replace `web` with service name)
docker compose logs web # Show all logs (onetime)
docker compose logs -f --tail=10 web # Show last 10 lines and follow

# Analyze a container
docker compose exec web bash # Launch terminal in `web` service

# Run tests
docker compose run --rm home make test

# Install Node.js modules (if you get an error running tests)
# Important: npm jobs need to be run inside the Docker environment.
docker compose run --rm home npm install --no-audit
# build JS/CSS assets:
docker compose run --rm home npm run build-assets

# Launch a temporary container and run tests
docker compose run --rm home make test

# Run Open Library using a local copy of Infogami for development

docker compose down && \
    docker compose -f compose.yaml -f compose.override.yaml -f compose.infogami-local.yaml up -d && \
    docker compose logs -f --tail=10 web

# In your browser, navigate to http://localhost:8080

# To test Open Library on another version of Python, modify Dockerfile.olbase and then
# rebuild olbase (see above) and oldev (`docker compose build`)

Developing the Dockerfile Copy Chapter

If you need to make changes to the dependencies in Dockerfile.olbase, rebuild it with:

docker build -t openlibrary/olbase:latest -f docker/Dockerfile.olbase . # 30+ min (Win10Home/Dec 2018)

This image is automatically rebuilt on deploy by ol-home0 and is pushed to at https://hub.docker.com/r/openlibrary/olbase.

If you're making changes you think might affect Docker Hub, you can create a branch starting with docker-test, e.g. docker-test-py2py3 (no weird chars), to trigger a build in docker hub at e.g. openlibrary/olbase:docker-test-py2py3.

Debugging and Profiling the Docker Image Copy Chapter

See Debugging and Performance Profiling for more information on how to attach a debugger when running in the Docker Container.

File: openlibrary/catalog/README.md Copy Chapter

Catalog

This contains code that was originally part of early standalone
import and catalog management tools which have now been integrated
into Open Library.

• add_book contains the main code used when books are imported into Open Library via /api/import.
• marc contains current and some legacy code used to parse binary and XML MARC records for import and display.
• utils contains an assortment of helper methods, many of which are legacy and unused. Current openlibrary/solr code still makes use of some parts.
• get_ia.py contains get_marc_record_from_ia() which is the main method used to read MARC records stored on archive.org.

File: openlibrary/components/dev/README.md Copy Chapter

This folder contains content exclusively for local development purposes.
It does not impact production builds.

File: openlibrary/components/README.md Copy Chapter

Files in the root of this directory are compiled and added to /static/build/components/. These components can be rendered from template files like so:

$:render_component('HelloWorld', attrs=dict(name="Jimmy"))

The building of these files happens on make components. Every top-level .vue
file is built at once into its own ol-{ComponentName}.js bundle.

To rebuild all components automatically on change during local development, run:

npm run watch:components

Inside Docker this becomes docker compose run --rm home npm run watch:components.

Live-reloading dev server Copy Chapter

The Vue components follow the following structure:

openlibrary/components/{MainComponent}.vue  -- The entrypoint to the component
openlibrary/components/{MainComponent}/...  -- Any sub components, utils, etc.

Outside the docker environment, run:

This way you can have a completely isolated component with hot reloading and easy to access
without clicking to the exact page on the localhost you want to use the component on.

npm install --no-audit
COMPONENT="HelloWorld" npm run serve

# Or
COMPONENT="LibraryExplorer" npm run serve
# Then open http://localhost:5173/?ol_base=openlibrary.org

Changing HelloWorld to be the name of the main component you want to work on.

For apps that are configured for it (like LibraryExplorer and MergeUI), when run
in this mode, the vue server will make requests to production openlibrary.org
for data like books, search results, covers, etc. You can configure where it fetches data
from by setting url parameters on the running app, eg ?ol_base=http://localhost:8080. See
openlibrary/components/configs.js for all the available url parameters.

Caveats Copy Chapter

• Vue is currently included with each component, so rendering multiple components per page results in very large load sizes
• JSON attributes currently don't work
• If Vue is embedded within a <form> input elements created by vue won't be picked up on form submission.
  This seems to be related to Vue's use of shadow dom. For a workaround, see the code in #5093.

Examples Copy Chapter

For an example of using Vue on existing pages see openlibrary/components/IdentifiersInput.vue

File: openlibrary/coverstore/README.md Copy Chapter

Coverstore README

Where are covers archived? Copy Chapter

• Covers 0 - 7,139,999 are stored in zip files within items https://archive.org/download/olcovers1 - https://archive.org/download/olcovers713 in the https://archive.org/details/ol_exports collection
• Covers 8,000,000 - 8,819,999 live in tar files within the https://archive.org/details/covers_0008 item
• Covers 8,820,000 - 8,829,999 live in a zip file also in the https://archive.org/details/covers_0008 item

Warnings Copy Chapter

As of 2022-11 there were 5,692,598 unarchived covers on ol-covers0 and archival hadn't occurred since 2014-11-29. This 5.7M number is sufficiently large that running /openlibrary/openlibrary/coverstore/archive.py archive() is still hanging after 5 minutes when trying to query for all unarchived covers.

As a result, it is recommended to adjust the cover query for unarchived items within archive.py to batch using some limit e.g. 1000. Also note that an initial id is specified (which is the last known successfully archived ID in 2014-11-29):

covers = _db.select('cover', where='archived=$f and id>6708293', order='id', vars={'f': False}, limit=1000)

How to run Covers Archival

First, ssh -A ol-covers0 and run docker exec -it openlibrary_covers_1 bash. Next, launch a python terminal and run:

from openlibrary.coverstore import config
from openlibrary.coverstore.server import load_config
from openlibrary.coverstore import archive
load_config("/olsystem/etc/coverstore.yml")
archive.archive(test=False)

How it works

As of 2022-11, the way coverstore works is that new covers that are uploaded to Open Library go into /1/var/lib/openlibrary/coverstore/localdisk/ within a directory named /YYYY/MM/DD/. A record for each cover (and its size variants) is recorded within the cover table of the coverstore psql db located on ol-db1.

At some (presumably advantageous if) regular interval, as the localdisk fills, the files can undergo archival, a process whereby covers are compressed and bundled into tar archives which are moved into the /1/var/lib/openlibrary/coverstore/items/ directory within folders called "staging items" (e.g. covers_0007). The database reference to these covers' filename paths are updated accordingly by the archive.py script.

Mek speculates that when coverstore attempts to look up a cover, its entry is looked up in the DB and if the filename is a tar, coverstore first looks on disk for a "staging item" folder within the staging directory /1/var/lib/openlibrary/coverstore/items/ and if no such "staging item" exists, the staging item is assumed to have been uploaded as an archive.org item having the same name (and thus redirects/resolves its request via archive.org).

State of Cover Archival

As of 2022-11 there are 5,692,598 unarchived covers on ol-covers0 and we're starting to run short on space. Specifically, cover archives haven't been happening since ~2014-11-29, as we can see from the following brutally slow query:

coverstore=# select id, olid, filename, last_modified from cover where archived=true order by id desc limit 1;

   id    |    olid     |               filename               |       last_modified  
---------+-------------+--------------------------------------+----------------------------
 7315539 | OL25645665M | covers_0007_31.tar:1849729536:247493 | 2014-11-29 22:34:37.329315

In the previous query, we see that the last cover (id #7,315,539) was archived on 2014-11-29 and resides within a tar covers_0007_31.tar. Coverstore assumes this tar resolves to an item folder called covers_0007, either staged on disk within /1/var/lib/openlibrary/coverstore/items/ or on archive.org/details/covers_0007. In this case, at the time of writing, this item was still staged on disk. As far as Mek can tell, staged items presumably get manually uploaded to archive.org under an item having the same name.

The item name itself (e.g. coverd_0007) is a combination of the prefix covers and the code web.numify("%010d.jpg" % cover.id)[:4] where, in this case, cover.id is 7315539. The "%010d" format parameter pads the cover.id with leading 0's until it is 10 digits long and then the [:4] takes the first 4 digits of this padded number. Anything lower than cover.id 1,000,000 will thus be in covers_0000 and from there the next 1M will be in covers_0002 and so on. In total, this scheme allows for just under 10B covers before it breaks, which is a sufficiently unlikely number to hit!

2022-12-03: Anand says: "The cover id is considered to be 10 digits, 4 digits go to items, 2 digits go to tar file and the remaining 4 go to the filename."

NB: We identified unarchived covers (denoted with archived=false within the covers table) prior to 2014-11-29 but early tests suggest the archive process may not have been ironed out and standardized before this date, and so we decided to use the latest successful archival date to resume our archival efforts.

Archival Process Copy Chapter

Recipe for moving one batch of 10k covers at a time into tars on archive.org.

1. On ol-covers0 docker container, run archive.py on ~10k items to create a new partial of unarchived covers, starting at stable ID 8M (e.g. covers_0008_00)
   text Copy

   from openlibrary.coverstore import config
   from openlibrary.coverstore.server import load_config
   from openlibrary.coverstore import archive
   load_config("/olsystem/etc/coverstore.yml")
   archive.archive(test=False)

2. ia upload each partial to the 4 respective items:
   • covers_0008 -> covers_0008_00.index and covers_0008_00.tar
   • s_covers_0008 -> s_covers_0008_00.index and s_covers_0008_00.tar
   • m_covers_0008 -> m_covers_0008_00.index and m_covers_0008_00.tar
   • l_covers_0008 -> l_covers_0008_00.index and l_covers_0008_00.tar
3. Update the upper bound value in code.py ~L290 by +10k (on ol-covers0 container 1 & 2 + restart)

• if (8100000 > int(value) >= 8000000): (or whatever is the upper bound) ...

4. Restart the containers + test to make sure the service is resolving to archive.org for all sizes
5. Remove only the completed partial (e.g. 00 from each folder on /1/var/lib/openlibrary/coverstore/items/

• rm /1/var/lib/openlibrary/coverstore/items/cover_0008/covers_0008_00.*
• rm /1/var/lib/openlibrary/coverstore/items/s_cover_0008/s_covers_0008_00.*
• rm /1/var/lib/openlibrary/coverstore/items/m_cover_0008/m_covers_0008_00.*
• rm /1/var/lib/openlibrary/coverstore/items/l_cover_0008/l_covers_0008_00.*

File: openlibrary/plugins/README.md Copy Chapter

Plugins

The plugins directory defines and registers all the Routers responsible for responding to requests to the Open Library website.

Understanding Routers Copy Chapter

In Open Library, when a patron navigates to an openlibrary.org url in their browser, the request is typically captured by a Router class which is defined by files within the plugins/ directory. Any Router class will be responsible for doing 4 things:

1. Defining a path -- a regular expression pattern used to decide whether a patron's requests should be handled by this Router
2. Implementing necessary class methods (e.g. GET and POST) corresponding to the patron's HTTP request type
3. Making calls to the right data models
4. Prepare data to be returned (which could be e.g. a rendered template or json) -- this may include setting any headers or content-types necessary for this data

The first step is to figure out where within the plugins/ directory your new route should live and whether it best lives in an existing file or a new file. An overview of the different plugin types are discussed in this technical overview video.

Understanding Directories Copy Chapter

The following describes what each plugin directory does, by convention, in decreasing order of presumed relevance:

• openlibrary -- most public pages meant to be read/consumed live here
  • code.py defines the /isbn endpoint and what python functions our html templates have access to!
  • js/ is where our javascript lives
  • api.py is many of our APIs are defined
  • lists.py defines many of the routes for the List feature
• upstream -- handles incoming stateful data requests (wiki edits, uploads, adding a book, logins, etc)
  • responds to actions (e.g. borrowing) and account related requests: editing, mybooks pages, account login, settings
• worksearch -- renders our /search/* related pages and all our search querying logic.
  • code.py contains most of the logic and routes for /search, the /barcodescanner, /advancedsearch, /search/lists, /search/subjects, etc, as well as their json returning counterparts
• importapi -- defines our book import API endpoints
  • code.py again defines most of the core logic we care about
• admin -- our admin pages and logic
• books -- old-style book APIs (likely not relevant)

Code.py Copy Chapter

Almost every plugins/ directory defines a central code.py which is similar to __init__.py in that it contains the logic required to register its directory's plugins with the system. In addition to acting as an entry point for all the other plugins in the directory and performing setup(), it also may define its own Routers. If you're not sure where in a plugin to look for a Router, that's likely a good place to start.

Tutorial: Implementing a new Route Copy Chapter

There are two ways to implement a new Open Library Route. The first way is to use your discretion to add a new Route to the correct pre-existing plugin file. The second way is to create a new directory and/or file which will require an additional step of registering your plugin with Open Library. We'll only walkthrough the first case here -- for those who need to register a new plugin, please see openlibrary/plugins/openlibrary/code.py for an example.

Let's say I wanted to create a page on Open Library called /search/many for a search experience that lets patrons search for multiple books in bulk. The first thing I'd do is make a determination of where it should live -- likely in worksearch/code.py. We'd then define our Route as follows:

class search_many(delegate.page):
    path = '/search/many'

    def GET(self):
        i = web.input(q='')
        return render_template('search/many', q=i.q)

Every Router is a class that extends something like delegate.page. The Router will define a GET, POST, etc, as appropriate to deal with corresponding HTTP requests of that type from the patron. The function parameters defined by the GET and POST of the Router depend on and are linked to the regex pattern specified by the Router's path.

The following example from plugins/openlibrary/api.py, The trending_books_api Router defines a path with a regex "/trending(/?.*)" that captures everything after /trending/ and forwards it into the GET function as the parameter period. In this example, if a patron goes to /trending/daily, the request will match the Router's path regex and the Router will call the GET method with period=daily as an argument.

class trending_books_api(delegate.page):
    path = "/trending(/?.*)"
    # path = "/trending/(now|daily|weekly|monthly|yearly|forever)"
    encoding = "json"

    def GET(self, period="/daily"):

In our first example i = web.input(q='') is used to access the GET query parameters specified by the patron's request from the URL (e.g. if the patron queries for /search/many?q=test then i.q will be "test".

The final line of the Router in our example makes a call to render_template which will fetch the corresponding template, by name, from the templates/ directory (specified by the 1st argument -- in this case the file "templates/search/many.html) and then pass in any of the variables this template needs to render (in this case the value q). The values passed in to the template must also be defined in the header of the HTML template within the $ def with(...) line. For more information on rendering templates, refer to the Front-end Guide in our docs.

In order to get this example to work, you will first need to create a new file templates/search/many.html with the following body content:

$ def with (q="")

Your query was $(q)!

You can now restart your server and test that your application is working by going to /search/many?q=test&debug=true, as described in our debugging website errors video

File: scripts/gh_scripts/README.md Copy Chapter

GitHub Project Management Scripts

This directory contains scripts that the Open Library team uses to interact with this GitHub repository.

`new_pr_labeler.mjs` Copy Chapter

This script is used to automatically prioritize and assign a new PR to a lead.

Usage: Copy Section

This script takes three positional arguments, followed by a vararg for the PR's body:

node new_pr_labeler.mjs repository pr_author pr_number pr_body...
repository the repository owner and name (ex. internetarchive/openlibrary)
pr_author the GitHub username of the PR's author
pr_number the PR's identification number
pr_body... the body of the PR

Details: Copy Section

The script does a case-insensitive search for a closes # statement in the newly created PR's body. If one is found, the corresponding issue is fetched.
The first https://github.com/internetarchive/openlibrary/labels/Priority%3A%200, https://github.com/internetarchive/openlibrary/labels/Priority%3A%201, or https://github.com/internetarchive/openlibrary/labels/Priority%3A%202 label that is found on the issue is added to the PR.
If a lead is assigned to the issue, and the lead is not also the PR's author, the lead will be assigned to the PR. Similar to the priority labels, only one lead can be assigned to a PR.

`issue_comment_bot.py` Copy Chapter

This script fetches issues that have new comments from contributors within the past number of hours, then posts a message to the team in our Slack channel.

Usage: Copy Section

positional arguments:
  hours                 Fetch issues that have been updated since this many hours ago

options:
  -h, --help                                       show this help message and exit
  -c CONFIG, --config CONFIG                       Path to configuration file
  -s SLACK_CHANNEL, --slack-channel SLACK_CHANNEL  Issues will be published to this Slack channel
  -t SLACK_TOKEN, --slack-token SLACK_TOKEN        Slack auth token
  --no-labels                                      Prevent the script from labeling the issues
  -v, --verbose                                    Print detailed information about the issues that were found

Configuration

A configuration file is required for this script to run properly. The file should contain a JSON string with the following fields:

leads : Array of configurations for each lead.

leads.githubUsername : The lead's GitHub username.

leads.leadLabel: Text of the lead's Lead: @ label.

leads.slackId: The lead's Slack ID in their mrkdwn format, which is used to trigger Slack

Running the script locally:

docker compose exec -e PYTHONPATH=. web bash

# Publish digest of new comments from the past day to #openlibrary-g:
./scripts/gh_scripts/issue_comment_bot.py 24 -s "#openlibrary-g" -t "replace-with-slack-token" -c "path/to/config/file"

Note: When adding arguments, be sure to place any hyphenated values within double quotes.

`weekly_status_report.mjs` Copy Chapter

This script prepares a digest of information helpful to leads, and publishes the digest to Slack.

Usage Copy Section

node weekly_status_report.mjs config_filepath
config_filepath the location of the script's configuration file

A SLACK_TOKEN must be included as an environment variable in order to POST to Slack.

Additionally, a GITHUB_TOKEN should also be added as an environment variable. @octokit/action adds
this to the octokit object during instantiation, allowing octokit to make authenticated requests
to GitHub.

Configuration

A configuration file is required for this script to run properly. The file should contain a JSON string with the following fields:

slackChannel : The digest will be published here.

publishFullDigest : Boolean that flags whether to publish a full or partial digest. If false, the digest will be published without several sections (see Details, below).

leads : Array of configurations for each lead.

leads.githubUsername : The lead's GitHub username.

leads.leadLabel: Text of the lead's Lead: @ label.

leads.slackId: The lead's Slack ID in their mrkdwn format, which is used to trigger Slack notifications when the message is published.

Details Copy Section

The script prepares a digest containing the following sections:

Recent comments : A list of links to issues that need comments, broken down by lead.

Needs: Lead/Assignee : Lists of pull requests that do not have an assignee, and issues that need a lead. Only present if publishFullDigest is true.

Untriaged issues : List of issues which have the https://github.com/internetarchive/openlibrary/labels/Needs%3A%20Triage label. Only present if publishFullDigest is true.

Assigned PRs : List of pull requests that have been assigned, broken down by lead. Links to higher priority PRs are also included here.

Staff PRs : List of all open staff PRs. Only present if publishFullDigest is true.

Submitter Input for PRs : List of PRs that are labeled https://github.com/internetarchive/openlibrary/labels/Needs%3A%20Submitter%20Input, broken down by leads.

File: scripts/solr_builder/README.md Copy Chapter

Solr reindex from OL dump Copy Chapter

This goes through building a solr instance from a dump file. To build the reindex (only tested on the OJF environment):

Steps Copy Section

1. Start Jenkins (altering the details as necessary):

docker run \
  -u root \
  -d \
  -p 8080:8080 \
  -p 50000:50000 \
  -v jenkins-data:/var/jenkins_home \
  -v /storage:/storage \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /var/lib/docker/volumes/jenkins-data:/var/lib/docker/volumes/jenkins-data \
  --restart always \
  --name jenkins \
  jenkinsci/blueocean

2. Follow the steps here to finish setting up Jenkins: https://jenkins.io/doc/book/installing/#setup-wizard
3. Follow these steps to create a new Pipeline Project using git: https://jenkins.io/doc/book/pipeline/getting-started/#defining-a-pipeline-in-scm
   • Be sure to set the script path to the Jenkinsfile in this directory (e.g. scripts/solr_builder/Jenkinsfile)
4. Run the pipeline!

Notes:

• Jenkins has a classic UI and a new, Blue Ocean UI. Each has its benefits.
• Logs are stored as artifacts after the jobs are finished, but you can also access the logs by clicking on "workspaces" in the classic UI and navigating to the logs directory (you can usually ignore the paths containing @).
• Although it looks like we are starting a Jenkins docker image, which starts a building docker image, which starts the solr_builder docker images, because we are forwarding the docker socket, everything is actually happening with the host docker (this is best practice). This has some caveats:
  • You should not assume you have full control of what's running; if you're not careful you will stop other containers
  • You should not try to run multiple re-indexes at the same time. This might be possible, but the current pipeline makes assumptions which would cause this to error

Possible Issues Copy Section

The SCM field is blank / has no options

Per the "Defining A Pipeline in SCM" page above, the "Definition" section of the Pipeline should be "Pipeline script from SCM". The next step is to choose the source control system by clicking on the "SCM" field, but it might be blank.

Solution: update and restart Jenkins, which can probably be done at the top of the page in the alerts section.

Building the pipeline fails almost immediately (various errors)

Check the console output and look for the errors listed below.

WorkflowScript: 3: Invalid agent type "dockerfile" specified

org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 3: Invalid agent type "dockerfile" specified. Must be one of [any, label, none] @ line 3, column 5.
       dockerfile {
       ^

Cause: Missing Docker and Docker Pipeline plugins.

Solution: Dashboard > Manage Jenkins > Manage Plugins. In the left-hand menu, click on "Available plugins" and search for "docker". Then install:

• Docker (possibly displaying as "Docker plugin")
• Docker Pipeline

java.lang.NoSuchMethodError: 'boolean org.kohsuke.groovy.sandbox.SandboxTransformer.mightBePositionalArgumentConstructor(org.codehaus.groovy.ast.expr.VariableExpression)'

java.lang.NoSuchMethodError: 'boolean org.kohsuke.groovy.sandbox.SandboxTransformer.mightBePositionalArgumentConstructor(org.codehaus.groovy.ast.expr.VariableExpression)'
	at com.cloudbees.groovy.cps.SandboxCpsTransformer.visitAssignmentOrCast(SandboxCpsTransformer.java:93)

Cause: Out of date Groovy Pipeline plugin:

Solution: Dashboard > Manage Jenkins > Manage Plugins. In the left-hand menu, click on "Installed plugins", search for the "Groovy Pipeline" plugin and update it.

Editing the Jenkins Pipeline Copy Section

• If you want to modify the pipeline, you can set the Jenkin's Project to use a local path. Note that you will have to restart Jenkins so that it mounts the path (i.e. -v "$HOME:/home"). Alternatively, you can also change the Git url to that of your fork, and specify a branch (e.g. */solr-builder--jenkins) and push to the branch whenever you want to test your code. Jenkins will pull from the branch when you run the project.
• Use VS Code to write your Pipeline; it has a validation extension. See https://jenkins.io/doc/book/pipeline/development/ for other editors/tips
• Read https://jenkins.io/doc/book/pipeline/development/ for general development tips

Final Sync Copy Chapter

TODO. Something along the lines of: Add a solrupdater to compose.production.yaml that points to the new server, and set its offset to be the correct date. See 5493 Move production solr from solr1 to solr0 for hints.

Deploy Copy Chapter

Now that the solr is ready, we can dump its database and import it into the solr on production. Here is the command to do that from the solrbuilder server:

time docker run --rm \
    --volumes-from solr_builder_solr-1 \
    -v /tmp/solr:/backup \
    ubuntu:xenial \
    tar czf /backup/solrbuilder-$(date +%Y-%m-%d).tar.gz /var/solr/data

(Last run: 41min/14G with 2020-10 dump; OJF)

Then on the production server (ol-solr0) run:

cd /opt/openlibrary

# Copy file from solrbuilder server (4min; 2020-11-05 ol-solr0)
time scp YOU@SOLR_BUILDER_SERVER:/tmp/solr/solrbuilder-2020-03-02.tar.gz /tmp/solr/solrbuilder-2020-03-02.tar.gz

# Restore backup file (7min; 2021-08-11 ol-solr0)
# Note the name "openlibrary_solr-data" should match "{OL_DIR}_{SOLR_DATA_VOLUME}", where:
#    OL_DIR: is the name of the directory where the openlibrary repo is; likely openlibrary
#    SOLR_DATA_VOLUME: is the name of the volume the solr service uses; defined in compose.yaml
time docker run -v openlibrary_solr-data:/var/solr/data -v /tmp/solr:/backup ubuntu:xenial \
    tar xzf /backup/solrbuilder-2021-08-11.tar.gz

# Start the services
COMPOSE_FILE="compose.yaml:compose.production.yaml" HOSTNAME="$HOSTNAME"docker compose --profile=ol-solr0 up -d

Resetting Copy Chapter

In order to be able to re-run the job, you need to stop/remove any of the old containers you don't intend to reuse:

# "new" solr containers/volumes
docker rm -f -v solr_builder_solr-1
docker volume rm solr_builder_solr-data

# DB containers/volumes
docker rm -f -v solr_builder_db-1 solr_builder_adminer-1
docker volume rm solr_builder_postgres-data