CONTRIBUTING Copy Section

Contributing to Tube Archivist

Welcome, and thanks for showing interest in improving Tube Archivist!

Table of Content Copy Chapter

Beta Testing Copy Chapter

There is always something that can get missed during development. Look at the commit messages tagged with #build - these are the unstable builds and give a quick overview of what has changed.

- Test the features mentioned, play around, try to break it.
- Test the update path by installing the :latest release first, then upgrade to :unstable to check for any errors.
- Test the unstable build on a fresh install.

Then provide feedback - even if you don't encounter any issues! You can do this in the #beta-testing channel on the Discord Discord server.

This helps ensure a smooth update for the stable release. Plus you get to test things out early!

How to open an issue Copy Chapter

Do:
- Do provide details and context, this matters a lot and makes it easier for people to help.
- Do familiarize yourself with the project first, some questions answer themselves when using the project for some time. Familiarize yourself with the Readme and the documentation, this covers a lot of the common questions, particularly the FAQ.
- Do respond to questions within a day or two so issues can progress. If the issue doesn't move forward due to a lack of response, we'll assume it's solved and we'll close it after some time to keep the list fresh.

Don't:
- Don't open duplicates, that includes open and closed issues. Also don't post the same issue on multiple platforms, that makes it unnecessarily hard for maintainers to keep up.
- Don't open an issue for something that's already on the roadmap, this needs your help to implement it, not another issue.
- Don't open an issue for something that's a known limitation. These are known by definition and don't need another reminder. Some limitations may be solved in the future, maybe by you?
- Don't overwrite the issue template, they are there for a reason. Overwriting that shows that you don't really care about this project. It shows that you have a misunderstanding how open source collaboration works and just want to push your ideas through. Overwriting the template may result in a ban.
- Don't redirect people trying to help to other platforms. E.g. in this context, imgur or other image hosting platforms are not needed, you can add an image directly in the issue as an attachments. Same goes for log files.

Bug Report Copy Section

Please keep in mind:
- Don't report bugs from yt-dlp here. There is a dedicated repo for that. Make sure to check for duplicates before opening a new issue there.
- Docker logs are the easiest way to understand what's happening when something goes wrong, always provide the logs upfront.
- Set the environment variable DJANGO_DEBUG=True to Tube Archivist and reproduce the bug for a better log output. Don't forget to remove that variable again after.
- A bug that can't be reproduced, is difficult or sometimes even impossible to fix. Provide very clear steps how to reproduce.

Feature Request Copy Section

Installation Help Copy Section

IMPORTANT: When receiving help, contribute back to the community by improving the installation instructions with your newly gained knowledge.

---

How to make a Pull Request Copy Chapter

Focus for the foreseeable future is on improving and building on existing functionality, not on adding and expanding the application.

This is a quick checklist to help streamline the process:

- If you are a new contributor, first welcome. Start off with a single PR first and wait for review. Don't open a bunch of PRs at once.
- Make your PR against the develop branch. That's where all active development happens. This simplifies the later merging into master, minimizes any conflicts and usually allows for easy and convenient fast-forward merging.
- Show off your progress, even if not yet complete, by creating a draft PR first and switch it as ready when you are ready.
- Make sure all your code is linted and formatted correctly, see below.

LLM and coding agents policy Copy Section

There is a AGENTS.md file committed on this repo. Make sure you and your coding agent are reading and following all instructions there.

In short for you as a human:

Coding agents are a great tool to get an understanding of the code base. They sometimes can be helpful in reviewing your changes.

- Use the LLMs for the intelligence part, as in understanding the code base, the patterns, narrowing down a bug you are trying to fix or for quick navigation through a large code base.
- Don't use the LLMs for making code changes. Don't instruct your coding agent to open PRs, respond to messages, etc. That is reserved for humans only as only humans will be responding too.
- Don't use LLMs to create PR descriptions. They are unnecessarily wordy and often confusing. A human will take the time to read it, you as a human take the time to describe the what and why of your PR.
- When in doubt, quality will be the decision making guide, but only when in doubt.

Documentation Changes Copy Section

All documentation is intended to represent the state of the latest release.

- If your PR with code changes also requires changes to documentation *.md files here in this repo, create a separate PR for that, so it can be merged separately at release.
- If your PR requires changes on the tubearchivist/docs, make the PR over there.
- Prepare your documentation updates at the same time as the code changes, so people testing your PR can consult the prepared docs if needed.

Code formatting and linting Copy Section

This project uses the excellent pre-commit library. The pre-commit-config.yml file is part of this repo.

Quick Start
- Run pre-commit install from the root of the repo.
- Next time you commit to your local git repo, the defined hooks will run.
- On first run, this will download and install the needed environments to your local machine, that can take some time. But that will be reused on sunsequent commits.

That is also running as a Git Hub action.

---

Contributions beyond the scope Copy Chapter

As you have read the FAQ and the known limitations and have gotten an idea what this project tries to do, there will be some obvious shortcomings that stand out, that have been explicitly excluded from the scope of this project, at least for the time being.

Extending the scope of this project will only be feasible with more regular contributors that are willing to help improve this project in the long run. Contributors that have an overall improvement of the project in mind and not just about implementing this one thing.

Small minor additions, or making a PR for a documented feature request or bug, even if that was and will be your only contribution to this project, are always welcome and is not what this is about.

Beyond that, general rules to consider:

- Maintainability is key: It's not just about implementing something and being done with it, it's about maintaining it, fixing bugs as they occur, improving on it and supporting it in the long run.
- Others can do it better: Some problems have been solved by very talented developers. These things don't need to be reinvented again here in this project.
- Develop for the 80%: New features and additions should be beneficial for 80% of the users. If you are trying to solve your own problem that only apply to you, maybe that would be better to do in your own fork or if possible by a standalone implementation using the API.
- If all of that sounds too strict for you, as stated above, start becoming a regular contributor to this project.

---

User Scripts Copy Chapter

---

Improving the Documentation Copy Chapter

The documentation is available at docs.tubearchivist.com, and is built from a separate repo: tubearchivist/docs. The Readme there has additional instructions on how to make changes.

---

Development Environment Copy Chapter

This codebase is set up to be developed natively outside of docker as well as in a docker container. Developing outside of a docker container can be convenient, as IDE and hot reload usually works out of the box. But testing inside of a container is still essential, as there are subtle differences, especially when working with the filesystem and networking between containers.

Note:
- This project doesn't look for contributions to this to cover additional dev setup environments from new contributors. If you are a regular contributor and you see ways to improve this, please reach out on Discord first.
- Subtitles currently fail to load with DJANGO_DEBUG=True, that is due to incorrect Content-Type error set by Django's static file implementation. That's only if you run the Django dev server, Nginx sets the correct headers in the container.

Native Instruction Copy Section

For convenience, it's recommended to still run Redis and ES in a docker container. Make sure both containers can be reachable over the network.

Set up your virtual environment and install the requirements defined in requirements-dev.txt.

There are options built in to load environment variables from a file using load_dotenv. Example .env file to place in the same folder as manage.py:

TA_HOST="localhost"
TA_USERNAME=tubearchivist
TA_PASSWORD=verysecret
TA_MEDIA_DIR="static/volume/media"
TA_CACHE_DIR="static"
TA_APP_DIR="."
REDIS_CON=redis://localhost:6379
ES_URL="http://localhost:9200"
ELASTIC_PASSWORD=verysecret
TZ=America/New_York
DJANGO_DEBUG=True

Then look at the container startup script run.sh, make sure all needed migrations and startup checks ran. To start the dev backend server from the same folder as manage.py run:

python manage.py runserver

The backend will be available on localhost:8000/api/.

You'll probably also want to have a Celery worker instance running, refer to run.sh for that. The Beat Scheduler might not be needed.

Then from the frontend folder, install the dependencies with:

npm install

Then to start the frontend development server:

npm run dev

And the frontend should be available at localhost:3000.

Docker Instructions Copy Section

Edit the docker-compose.yml file and replace the image: bbilly1/tubearchivist line with build: .. Also make any other changes to the environment variables and so on necessary to run the application, just like you're launching the application as normal.

Run docker compose up --build. This will bring up the application. Kill it with ctrl-c or by running docker compose down from a new terminal window in the same directory.

Make your changes locally and re-run docker compose up --build. The Dockerfile is structured in a way that the actual application code is in the last layer so rebuilding the image with only code changes utilizes the build cache for everything else and will just take a few seconds.

Develop environment inside a VM Copy Section

You may find it nice to run everything inside of a VM for complete environment snapshots and encapsulation, though this is not strictly necessary. There's a deploy.sh script which has some helpers for this use case:

- This assumes a standard Ubuntu Server VM with docker and docker compose already installed.
- Configure your local DNS to resolve tubearchivist.local to the IP of the VM.
- To deploy the latest changes and rebuild the application to the testing VM run:

./deploy.sh test

Working with Elasticsearch Copy Section

Quick start
Generate your access token in Elasitcsearch:

bin/elasticsearch-service-tokens create elastic/kibana kibana

Example docker compose, use same version as for Elasticsearch:

services:
  kibana:
    image: docker.elastic.co/kibana/kibana:0.0.0
    container_name: kibana
    environment:
    - "ELASTICSEARCH_HOSTS=http://archivist-es:9200"
    - "ELASTICSEARCH_SERVICEACCOUNTTOKEN=<your-token-here>"
    ports:
    - "5601:5601"

If you want to run queries on the Elasticsearch container directly from your host with for example curl or something like postman, you might want to publish the port 9200 instead of just exposing it.

Persist Token
The token will get stored in ES in the config folder, and not in the data folder. To persist the token between ES container rebuilds, you'll need to persist the config folder as an additional volume:

1. Create the token as described above
2. While the container is running, copy the current config folder out of the container, e.g.:

docker cp archivist-es:/usr/share/elasticsearch/config/ volume/es_config

- ./volume/es_config:/usr/share/elasticsearch/config

Now your token will persist between ES container rebuilds.

---

README Copy Section

more screenshots and video

<div align="center">
<a href="https://hub.docker.com/r/bbilly1/tubearchivist" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-docker.png" alt="tubearchivist-docker" title="Tube Archivist Docker Pulls" height="50" width="190"/></a>
<a href="https://github.com/tubearchivist/tubearchivist" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-github-star.png" alt="tubearchivist-github-star" title="Tube Archivist GitHub Stars" height="50" width="190"/></a>
<a href="https://github.com/tubearchivist/tubearchivist/forks" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-github-forks.png" alt="tubearchivist-github-forks" title="Tube Archivist GitHub Forks" height="50" width="190"/></a>
<a href="https://www.tubearchivist.com/discord" target="_blank"><img src="https://tiles.tilefy.me/t/tubearchivist-discord.png" alt="tubearchivist-discord" title="TA Discord Server Members" height="50" width="190"/></a>
</div>

Table of contents Copy Chapter

* Docs with FAQ, and API documentation
* Core functionality
* Resources
* Installing
* Getting Started
* Known limitations
* Port Collisions
* Common Errors
* Roadmap
* Donate

------------------------

Core functionality Copy Chapter

Once your YouTube video collection grows, it becomes hard to search and find a specific video. That's where Tube Archivist comes in: By indexing your video collection with metadata from YouTube, you can organize, search and enjoy your archived YouTube videos without hassle offline through a convenient web interface. This includes:

* Subscribe to your favorite YouTube channels
* Download Videos using yt-dlp
* Index and make videos searchable
* Play videos
* Keep track of viewed and unviewed videos

Resources Copy Chapter

* Discord: Connect with us on our Discord server.
* r/TubeArchivist: Join our Subreddit.
* Browser Extension Tube Archivist Companion, for Firefox and Chrome
* Jellyfin Plugin: Add your videos to Jellyfin
* Plex Plugin: Add your videos to Plex

Installing Copy Chapter

For minimal system requirements, the Tube Archivist stack needs around 2GB of available memory for a small testing setup and around 4GB of available memory for a mid to large sized installation. Minimal with dual core with 4 threads, better quad core plus.
This project requires docker. Ensure it is installed and running on your system.

The documentation has additional user provided instructions for Unraid, Synology and Podman.

The instructions here should get you up and running quickly, for Docker beginners and full explanation about each environment variable, see the docs.

Take a look at the example docker-compose.yml and configure the required environment variables.

All environment variables are explained in detail in the docs here.

Both TA_PASSWORD and ELASTIC_PASSWORD can be suffixed with _FILE to allow passing in passwords as secrets. _FILE is a convention used by some images including ElasticSearch

TubeArchivist Copy Section

| Environment Var | Value | Required |
| ----------------------------- | ----- | -------- |
| TA_HOST | Server IP or hostname http://tubearchivist.local:8000 | Required |
| TA_USERNAME | Initial username when logging into TA | Required |
| TA_PASSWORD | Initial password when logging into TA | Required |
| ELASTIC_PASSWORD | Password for ElasticSearch | Required |
| REDIS_CON | Connection string to Redis | Required |
| TZ | Set your timezone for the scheduler | Required |
| TA_PORT | Overwrite Nginx port | Optional |
| TA_BACKEND_PORT | Overwrite container internal backend server port | Optional |
| TA_ENABLE_AUTH_PROXY | Enables support for forwarding auth in reverse proxies | Read more |
| TA_AUTH_PROXY_USERNAME_HEADER | Header containing username to log in | Optional |
| TA_AUTH_PROXY_LOGOUT_URL | Logout URL for forwarded auth | Optional |
| ES_URL | URL That ElasticSearch runs on | Optional |
| ES_DISABLE_VERIFY_SSL | Disable ElasticSearch SSL certificate verification | Optional |
| ES_SNAPSHOT_DIR | Custom path where elastic search stores snapshots for master/data nodes | Optional |
| HOST_GID | Allow TA to own the video files instead of container user | Optional |
| HOST_UID | Allow TA to own the video files instead of container user | Optional |
| ELASTIC_USER | Change the default ElasticSearch user | Optional |
| TA_LDAP | Configure TA to use LDAP Authentication | Read more |
| DISABLE_STATIC_AUTH | Remove authentication from media files, (Google Cast...) | Read more |
| TA_AUTO_UPDATE_YTDLP | Configure TA to automatically install the latest yt-dlp on container start | Optional |
| DJANGO_DEBUG | Return additional error messages, for debug only | Optional |
| TA_LOGIN_AUTH_MODE | Configure the order of login authentication backends (Default: single) | Optional |

| TA_LOGIN_AUTH_MODE value | Description |
| ------------------------ | ----------- |
| single | Only use a single backend (default, or LDAP, or Forward auth, selected by TA_LDAP or TA_ENABLE_AUTH_PROXY) |
| local | Use local password database only |
| ldap | Use LDAP backend only |
| forwardauth | Use reverse proxy headers only |
| ldap_local | Use LDAP backend in addition to the local password database |

ElasticSearch Copy Section

| Environment Var | Value | Required |
| ---------------- | ----- | -------- |
| ELASTIC_PASSWORD | Matching password ELASTIC_PASSWORD from TubeArchivist | Required |
| http.port | Change the port ElasticSearch runs on | Optional |

Update Copy Chapter

Always use the latest (the default) or a named semantic version tag for the docker images. The unstable tags see CONTRIBUTING.md#beta-testing.

You will see the current version number of Tube Archivist in the footer of the interface. There is a daily version check task querying tubearchivist.com, notifying you of any new releases in the footer. After updating, check the footer to verify you are running the expected version.

* This project is tested for updates between one or two releases maximum. Further updates back may or may not be supported. Ideally apply new updates at least once per month.
There can be breaking changes between updates, particularly as the application grows, new environment variables or settings might be required for you to set in the your docker-compose file. Always* check the release notes: Any breaking changes will be marked there.
All testing and development is done with the Elasticsearch version number as mentioned in the provided docker-compose.yml* file. This will be updated from time to time. Running an older version of Elasticsearch is most likely not going to result in any issues, but it's still recommended to run the same version as mentioned. Use bbilly1/tubearchivist-es to automatically get the recommended version.

Getting Started Copy Chapter

1. Go through the settings page and look at the available options. Particularly set Download Format to your desired video quality before downloading. Tube Archivist downloads the best available quality by default. To support iOS or MacOS and some other browsers a compatible format must be specified. For example:

bestvideo[vcodec=avc1]+bestaudio[acodec=mp4a]/mp4

2. Subscribe to some of your favorite YouTube channels on the channels page.
3. On the downloads page, click on Rescan subscriptions to add videos from the subscribed channels to your Download queue or click on Add to download queue to manually add Video IDs, links, channels or playlists.
4. Click on Start download and let Tube Archivist to it's thing.
5. Enjoy your archived collection!

Port Collisions Copy Section

If you have a collision on port 8000, best solution is to use dockers HOST_PORT and CONTAINER_PORT distinction: To for example change the interface to port 9000 use 9000:8000 in your docker-compose file.

For more information on port collisions, check the docs.

Common Errors Copy Chapter

Here is a list of common errors and their solutions.

vm.max_map_count vm.max_map_count')" title="Copy section prompt for LLMs"> Copy Section

Elastic Search in Docker requires the kernel setting of the host machine vm.max_map_count to be set to at least 262144.

To temporary set the value run:

sudo sysctl -w vm.max_map_count=262144

To apply the change permanently depends on your host operating system:

* For example on Ubuntu Server add vm.max_map_count = 262144 to the file /etc/sysctl.conf.
* On Arch based systems create a file /etc/sysctl.d/max_map_count.conf with the content vm.max_map_count = 262144.
* On any other platform look up in the documentation on how to pass kernel parameters.

Permissions for elasticsearch Copy Section

If you see a message similar to Unable to access 'path.repo' (/usr/share/elasticsearch/data/snapshot) or failed to obtain node locks, tried [/usr/share/elasticsearch/data] and maybe these locations are not writable when initially starting elasticsearch, that probably means the container is not allowed to write files to the volume.
To fix that issue, shutdown the container and on your host machine run:

chown 1000:0 -R /path/to/mount/point

This will match the permissions with the UID and GID of elasticsearch process within the container and should fix the issue.

Disk usage Copy Section

The Elasticsearch index will turn to read only if the disk usage of the container goes above 95% until the usage drops below 90% again, you will see error messages like disk usage exceeded flood-stage watermark.

Similar to that, TubeArchivist will become all sorts of messed up when running out of disk space. There are some error messages in the logs when that happens, but it's best to make sure to have enough disk space before starting to download.

error setting rlimit error setting rlimit')" title="Copy chapter prompt for LLMs"> Copy Chapter

If you are seeing errors like failed to create shim: OCI runtime create failed and error during container init: error setting rlimits, this means docker can't set these limits, usually because they are set at another place or are incompatible because of other reasons. Solution is to remove the ulimits key from the ES container in your docker compose and start again.

This can happen if you have nested virtualizations, e.g. LXC running Docker in Proxmox.

Known limitations Copy Chapter

* Video files created by Tube Archivist need to be playable in your browser of choice. Not every codec is compatible with every browser and might require some testing with format selection.
* Every limitation of yt-dlp will also be present in Tube Archivist. If yt-dlp can't download or extract a video for any reason, Tube Archivist won't be able to either.
* There is no flexibility in naming of the media files.

Roadmap Copy Chapter

We have come far, nonetheless we are not short of ideas on how to improve and extend this project. Issues waiting for you to be tackled in no particular order:

- [ ] Audio download
- [ ] Podcast mode to serve channel as mp3
- [ ] Random and repeat controls (#108, #220)
- [ ] Auto play or play next link (#226)
- [ ] Multi language support
- [ ] Show total video downloaded vs total videos available in channel
- [ ] Download or Ignore videos by keyword (#163)
- [ ] Custom searchable notes to videos, channels, playlists (#144)
- [ ] Search comments
- [ ] Per user videos/channel/playlists

Implemented:
- [X] Search download queue [2025-07-31]
- [X] Configure shorts, streams and video sizes per channel [2024-07-15]
- [X] User created playlists [2024-04-10]
- [X] User roles, aka read only user [2023-11-10]
- [X] Add statistics of index [2023-09-03]
- [X] Implement Apprise for notifications [2023-08-05]
- [X] Download video comments [2022-11-30]
- [X] Show similar videos on video page [2022-11-30]
- [X] Implement complete offline media file import from json file [2022-08-20]
- [X] Filter and query in search form, search by url query [2022-07-23]
- [X] Make items in grid row configurable to use more of the screen [2022-06-04]
- [X] Add passing browser cookies to yt-dlp [2022-05-08]
- [X] Add SponsorBlock integration [2022-04-16]
- [X] Implement per channel settings [2022-03-26]
- [X] Subtitle download & indexing [2022-02-13]
- [X] Fancy advanced unified search interface [2022-01-08]
- [X] Auto rescan and auto download on a schedule [2021-12-17]
- [X] Optional automatic deletion of watched items after a specified time [2021-12-17]
- [X] Create playlists [2021-11-27]
- [X] Access control [2021-11-01]
- [X] Delete videos and channel [2021-10-16]
- [X] Add thumbnail embed option [2021-10-16]
- [X] Create a github wiki for user documentation [2021-10-03]
- [X] Grid and list view for both channel and video list pages [2021-10-03]
- [X] Un-ignore videos [2021-10-03]
- [X] Dynamic download queue [2021-09-26]
- [X] Backup and restore [2021-09-22]
- [X] Scan your file system to index already downloaded videos [2021-09-14]

User Scripts Copy Chapter

This is a list of useful user scripts, generously created from folks like you to extend this project and its functionality. Make sure to check the respective repository links for detailed license information.

This is your time to shine, read this then open a PR to add your script here.

* danieljue/ta_dl_page_script: Helper browser script to prioritize a channels' videos in download queue.
* dot-mike/ta-scripts: A collection of personal scripts for managing TubeArchivist.
* DarkFighterLuke/ta_base_url_nginx: Set base URL with Nginx when you can't use subdomains.
* lamusmaser/ta_migration_helper: Advanced helper script for migration issues to TubeArchivist v0.4.4 or later.
* lamusmaser/create_info_json: Script to generate .info.json files using ffmpeg collecting information from downloaded videos.
* lamusmaser/ta_fix_for_video_redirection: Script to fix videos that were incorrectly indexed by YouTube's "Video is Unavailable" response.
* RoninTech/ta-helper: Helper script to provide a symlink association to reference TubeArchivist videos with their original titles.
* tangyjoust/Tautulli-Notify-TubeArchivist-of-Plex-Watched-State Mark videos watched in Plex (through streaming not manually) through Tautulli back to TubeArchivist
* Dhs92/delete_shorts: A script to delete ALL YouTube Shorts from TubeArchivist
* arisenfromtheashes/TA_DVR: Scripts to assist in using Tube Archivist like a DVR
* WreckingBANG/Self.Tube: Client app for Android and Linux phones written in Flutter.

Donate Copy Chapter

The best donation to Tube Archivist is your time, take a look at the contribution page to get started.
Second best way to support the development is to provide for caffeinated beverages:

* GitHub Sponsor become a sponsor here on GitHub
* Paypal.me for a one time coffee
* Paypal Subscription for a monthly coffee
* ko-fi.com for an alternative platform

Notable mentions Copy Chapter

This is a selection of places where this project has been featured on reddit, in the news, blogs or any other online media, newest on top.

* xda-developers.com: 5 obscure self-hosted services worth checking out - Tube Archivist - To save your essential YouTube videos, [2024-10-13][link]
* selfhosted.show: why we're trying Tube Archivist, [2024-06-14][link]
* ycombinator: Tube Archivist on Hackernews front page, [2023-07-16][link]
* linux-community.de: Tube Archivist bringt Ordnung in die Youtube-Sammlung, [German][2023-05-01][link]
* noted.lol: Dev Debrief, An Interview With the Developer of Tube Archivist, [2023-03-30] [link]
* console.substack.com: Interview With Simon of Tube Archivist, [2023-01-29] [link]
* reddit.com: Tube Archivist v0.3.0 - Now Archiving Comments, [2022-12-02] [link]
* reddit.com: Tube Archivist v0.2 - Now with Full Text Search, [2022-07-24] [link]
* noted.lol: How I Control What Media My Kids Watch Using Tube Archivist, [2022-03-27] [link]
* thehomelab.wiki: Tube Archivist - A Youtube-DL Alternative on Steroids, [2022-01-27] [link]
* reddit.com: Celebrating TubeArchivist v0.1, [2022-01-09] [link]
* linuxunplugged.com: Pick: tubearchivist — Your self-hosted YouTube media server, [2021-09-11] [link] and [2021-10-05] [link]
* reddit.com: Introducing Tube Archivist, your self hosted Youtube media server, [2021-09-12] [link]

---