smallweb Documentation

Python repository by @kagisearch. Docker Compose & container deployment with 1.7k stars.

1,726 stars Python Python 4 files · ~2,576 tokens
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy smallweb on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name smallweb --restart always kagisearch/smallweb:latest

Technical Documentation: kagisearch/smallweb

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

Kagi Small Web

Kagi Small Web is an initiative by Kagi.

Kagi's mission is to humanize the web and this project is built to help surface recent results from the small web - people and stories that typically zip by in legacy search engines. Read more about it in the announcement blog post.

Few things to note:

⚠️ Guidelines for site inclusion to the list ⚠️

Add a new personal blog RSS/Atom feed to the list. Rules:

  • If submitting your own website, you must add at least 2 other sites that are not yours (and are not in list yet) in the same commit.
  • Locate and submit the RSS/Atom feed of the website. Place in the file so that it remains sorted.
  • Content must be in English (currently, other languages are not accepted).
  • No illegal or NSFW content.
  • No auto generated, LLM generated or spam content. (see below for the nuance)
  • Only personal blogs may be submitted (no multi-author blogs).
  • The blog must have a recent post, no older than 12 months, to meet the recency criteria for inclusion.
  • The site must not contain any forms of advertisements or undisclosed affiliate links
  • Site should not have popups (newsletter signup, cookie selection, etc)
  • A Youtube channel must not post more than twice a week.
  • A Youtube channel must have fewer than 100,000 subscribers.

For comics:

  • Must be independently created art (no AI generated content)
  • RSS feed must show the full comic in the feed
  • No commercial syndicated comics

Guidelines on LLM usage:

  • Intent matters - use of LLMs to deceive, overwhelm, or otherwise taint the rules of good discourse is not allowed.
  • No "content automation" - content must be heavily guided by and reviewed by humans and done at human scale for human audiences. Examples of "guided by" are: outlines or drafts of content covering the main idea and key points, and "reviewed by" is a human reading and checking to make sure it adheres to their intent.
  • Acknowledgment - Effort should be made to acknowledge heavier LLM usage (via sidebar, authorship byline, or other such marker.)
  • General assistance - Use of LLMs to assist in human authors communicating is otherwise fine, and viewed as having little difference from using a spell checker or grammar checker, as long as the above points are followed.

Add website RSS
feed

Hint: To extract the RSS link from a YouTube channel, you can use this tool.

Add YouTube channel RSS
feed

Add Comic RSS
feed

Remove a site or a channel

Remove a website if :

  • It does not adhere to the above guidelines
  • In the removal request, state which guideline does it break

Clicking "Remove website" will edit small web list in new tab, where you can locate and remove the website feed in question. Make sure to add in comments the reason for removal.

Remove website

Remove channel

What we show on the site

If the blog is included in small web feed list (which means it has content in English, it is informational/educational by nature and it is not trying to sell anything) we check for these two things to show it on the site:

  • Blog has recent posts (<7 days old)
  • The website can appear in an iframe

The pages will still show in Kagi search results even if they cannot show on the site due technical reasons (eg iframe not allowed).

The Small Web seal: badge of authentic creation

Small Web initiative members can display badges on their websites to identify themselves as part of a community committed to authentic, human-centered content.

Small web is beautiful

What is Small Web exactly? Recommend reading:

Info

smallweb.txt - Contains the feeds of indexed blogs

smallyt.txt - Contains the feeds of indexed YouTube channels

smallcomic.txt - Contains the feeds of indexed independent comics

yt_rejected.txt - Contains the list of YouTube channels that were reviewed (in an automated way) and rejected

app/ - App powering the Kagi Small Web website

Sources

Small web

The original list of small web blogs has been assembled from various
sources including:

YouTube channels

The seed list for YouTube channels has been assembled from these HN discussions.

Useful commands

Show duplicate domains:

text
awk -F/ '{print $3}' smallweb.txt | sort | uniq -d | while read domain; do echo "$domain"; grep "$domain" smallweb.txt; echo ""; done

Overview (`app/README.md`)

Run the App

Requirements

Python 3.X

bash
pip install -r app/requirements.txt

Running

To run anything that runs a Flask app, for example

bash
cd app
gunicorn --workers 1 --threads 4 sw:app

This will run the Small Web frontend app locally.

Open http://127.0.0.1:8000 in browser to access.

Overview (`extension/README.md`)

Kagi Small Web Browser Extension

Discover the indie web directly in your browser. Random posts from personal blogs, YouTube creators, GitHub projects, and webcomics.

Features

  • Discover: Random posts from curated indie content
  • Multiple Modes: Blogs, Liked, Videos, Code, Comics, Saved
  • Reader Mode: Clean reading experience with optional dyslexia-friendly font
  • Text-to-Speech: Listen to articles hands-free
  • Save for Later: Bookmark posts to your local collection
  • Like: Send likes to content creators
  • History: Per-mode browsing history
  • Keyboard Shortcuts:
    • Space - Discover next post
    • R - Toggle reader mode
    • S - Save/unsave post
    • T - Toggle text-to-speech
    • D - Toggle dyslexia font

Installation

Chrome

  1. Download smallweb-chrome.zip from dist/
  2. Unzip the file
  3. Open chrome://extensions/
  4. Enable "Developer mode" (toggle in top right)
  5. Click "Load unpacked"
  6. Select the unzipped folder

Firefox

  1. Download smallweb-firefox.zip from dist/
  2. Open about:debugging#/runtime/this-firefox
  3. Click "Load Temporary Add-on"
  4. Select the zip file or manifest.json inside

Building from Source

bash
cd extension
./build.sh

Outputs:

  • dist/chrome/ - Chrome extension
  • dist/firefox/ - Firefox extension
  • dist/smallweb-chrome.zip - Chrome distribution
  • dist/smallweb-firefox.zip - Firefox distribution

Usage

  1. Click the extension icon to open the side panel
  2. Click "Discover" to navigate to a random post
  3. Use mode tabs to switch content types (hover for item counts)
  4. Enable Reader Mode for distraction-free reading
  5. Save posts you want to revisit later

Privacy

  • All data stored locally in browser storage
  • No tracking or analytics
  • Referrer set to kagi.com/smallweb for proper attribution

Overview (`maintenance/README.md`)

Maintenance

The scripts in this directory are used for maintenance. Be sure to read and understand them before executing.

Usage

Execute [`crawl.sh`](crawl.sh)

This will crawl all entries from smallweb.txt via curl. Note that this can take a very long time! The results of this expensive operation will be written to crawl-result.csv. All further processing will use the results from this file and be much cheaper.

Execute [`process.sh`](process.sh)

This will run several cleanup steps. The script will pause between each step to give you time to assess the results, commit changes and make adjustments.

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →