## 1. Project Overview & Quickstart (soxoj/maigret)
## File: README.md
# Maigret
**English** · [简体中文](README.zh-CN.md)
**Maigret** collects a dossier on a person **by username only**, checking for accounts on a huge number of sites and gathering all the available information from web pages. No API keys required. **[AI profiling (demo)](#ai-analysis)**.
## Sponsors
[**IPcook**](https://www.ipcook.com/?ref=githubmaigret&utm_source=github&utm_medium=referral&utm_campaign=maigret) provides reliable residential proxies for online research, username discovery, and public data collection workflows. High success rates • 99.99% uptime • Response time under 0.5s • Monthly & Pay-as-you-go • Non-expiring traffic • Up to 10 free sub-accounts for team collaboration • Residential proxies from $0.3–$3.2/GB.
**Special Offer**: FREE 100MB trial available. Use code WELCOME20 for 20% off.
[**RapidProxy**](https://www.rapidproxy.io/?ref=soxoj) provides high-performance residential proxies for Twitter scraping, Selenium automation, and web data extraction. 90M+ IPs • Smart rotation • Anti-block • Non-expiring traffic.
**Special Offer**: Try it free — Plans from $0.65/GB. Use code **RAPID10** for 10% off.
## Contents
- [In one minute](#in-one-minute)
- [Main features](#main-features)
- [Demo](#demo)
- [Installation](#installation)
- [Usage](#usage)
- [Contributing](#contributing)
- [Commercial Use](#commercial-use)
- [About](#about)
## In one minute
Ensure you have Python 3.10 or higher.
```bash
pip install maigret
maigret YOUR_USERNAME
```
No install? Try the [community Telegram bot](https://sites.google.com/view/maigret-bot-link) or a [Cloud Shell](#cloud-shells).
Want a web UI? See [how to launch it](#web-interface).
See also: [Quick start](https://maigret.readthedocs.io/en/latest/quick-start.html).
## Main features
- Supports 3,000+ sites ([see full list](https://github.com/soxoj/maigret/blob/main/sites.md)). A default run checks the 500 highest-ranked sites by traffic; pass `-a` to scan everything, or `--tags` to narrow by category/country.
- Embeddable in Python projects — import `maigret` and run searches programmatically (see [library usage](https://maigret.readthedocs.io/en/latest/library-usage.html)).
- [Extracts](https://github.com/soxoj/socid_extractor) all available information about the account owner from profile pages and site APIs, including links to other accounts.
- Performs recursive search using discovered usernames and other IDs.
- Allows filtering by tags (site categories, countries).
- Detects and partially bypasses blocks, censorship, and CAPTCHA.
- Fetches an [auto-updated site database](https://maigret.readthedocs.io/en/latest/settings.html#database-auto-update) from GitHub each run (once per 24 hours), and falls back to the built-in database if offline.
- Works with Tor and I2P websites; able to check domains.
- Ships with a [web interface](#web-interface) for browsing results as a graph and downloading reports in every format from a single page.
- Optional [AI analysis mode](#ai-analysis) (`--ai`) that turns raw findings into a short investigation summary using an OpenAI-compatible API.
For the complete feature list, see the [features documentation](https://maigret.readthedocs.io/en/latest/features.html).
### Used by
Professional OSINT and social-media analysis tools built on Maigret:
[](https://github.com/SocialLinks-IO/sociallinks-api)
[](https://sociallinks.io/products/sl-crimewall)
[](https://usersearch.ai/)
## Demo
### Video
### Reports
[PDF report](https://raw.githubusercontent.com/soxoj/maigret/main/static/report_alexaimephotographycars.pdf), [HTML report](https://htmlpreview.github.io/?https://raw.githubusercontent.com/soxoj/maigret/main/static/report_alexaimephotographycars.html)
[Full console output](https://raw.githubusercontent.com/soxoj/maigret/main/static/recursive_search.md)
## Installation
Already ran the [In one minute](#one-minute) steps? You're set. Below are alternative methods.
Don't want to install anything? Use the [community Telegram bot](https://sites.google.com/view/maigret-bot-link).
### Windows
Download `maigret_standalone.exe` from [Releases](https://github.com/soxoj/maigret/releases). You can launch it two ways:
- **Double-click it** — Maigret will ask for a username, run a default search, and wait at the end so the report links stay visible.
- **Run it from a terminal** — open Command Prompt (press `Win+R`, type `cmd`, hit Enter) or PowerShell to pass extra options:
```cmd
cd %USERPROFILE%\Downloads
maigret_standalone.exe USERNAME
maigret_standalone.exe USERNAME --html :: also save an HTML report
maigret_standalone.exe --help :: list all options
```
Video guide: https://youtu.be/qIgwTZOmMmM.
### Cloud Shells
Run Maigret in the browser via cloud shells or Jupyter notebooks:
[](https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/soxoj/maigret&tutorial=cloudshell-tutorial.md)
[](https://repl.it/github/soxoj/maigret)
[](https://colab.research.google.com/gist/soxoj/879b51bc3b2f8b695abb054090645000/maigret-collab.ipynb)
[](https://mybinder.org/v2/gist/soxoj/9d65c2f4d3bec5dd25949197ea73cf3a/HEAD)
### Local installation (pip)
```bash
# install from pypi
pip3 install maigret
# usage
maigret username
```
### From source
```bash
# or clone and install manually
git clone https://github.com/soxoj/maigret && cd maigret
# build and install
pip3 install .
# usage
maigret username
```
### Docker
Two image variants are published:
- `soxoj/maigret:latest` — CLI mode (default)
- `soxoj/maigret:web` — auto-launches the [web interface](#web-interface)
```bash
# official image (CLI)
docker pull soxoj/maigret
# CLI usage
docker run -v /mydir:/app/reports soxoj/maigret:latest username --html
# Web UI (open http://localhost:5000)
docker run -p 5000:5000 soxoj/maigret:web
# Web UI on a custom port
docker run -e PORT=8080 -p 8080:8080 soxoj/maigret:web
# manual build
docker build -t maigret . # CLI image (default target)
docker build --target web -t maigret-web . # Web UI image
```
### Troubleshooting
Build errors? See the [troubleshooting guide](https://maigret.readthedocs.io/en/latest/installation.html#troubleshooting).
PDF reports (`--pdf`) are an optional extra — install with `pip install 'maigret[pdf]'`. They need system-level graphics libraries on Linux/macOS; see the [PDF reports section](https://maigret.readthedocs.io/en/latest/installation.html#optional-pdf-reports-maigret-pdf) for per-OS install steps.
## Usage
### Examples
```bash
# make HTML, PDF, and XMind reports
maigret user --html
maigret user --pdf
maigret user --xmind # legacy XML with a manifest for XMind 2022+ readers
# machine-readable exports
maigret user --json ndjson # newline-delimited JSON (also: --json simple)
maigret user --csv
maigret user --txt
maigret user --graph # interactive D3 graph (HTML)
maigret user --neo4j # Neo4j Cypher script (graph database)
# search on sites marked with tags photo & dating
maigret user --tags photo,dating
# search on sites marked with tag us
maigret user --tags us
# highlight sites whose page also mentions specific keywords
maigret user --keywords python rust
# keyword-matched sites are shown with "[++]" in bright green
# search for three usernames on all available sites
maigret user1 user2 user3 -a
# AI-assisted investigation summary (needs OPENAI_API_KEY)
maigret user --ai
```
`--neo4j` writes a `*_neo4j.cypher` script of the results graph; import it with `cypher-shell -u neo4j -p < report_user_neo4j.cypher` or paste it into the Neo4j Browser. Re-imports are idempotent. See the [Neo4j export docs](https://maigret.readthedocs.io/en/latest/command-line-options.html#neo4j-export).
Run `maigret --help` for all options. Docs: [CLI options](https://maigret.readthedocs.io/en/latest/command-line-options.html), [more examples](https://maigret.readthedocs.io/en/latest/usage-examples.html). Running into 403s or timeouts? See [TROUBLESHOOTING.md](TROUBLESHOOTING.md).
### Web interface
Maigret has a built-in web UI with a results graph and downloadable reports.
Don't want to run it yourself? Deploy the published `soxoj/maigret:web` Docker image as a hosted app in one click:
[](https://render.com/deploy?repo=https://github.com/soxoj/maigret&path=utils/render.yaml)
Runs on Render's free tier (spins down after 15 min idle, spins back up on the next request). No login is set up on the instance, so anyone with the URL can use it.
Web Interface Screenshots
```console
maigret --web 5000
```
Open http://127.0.0.1:5000, enter a username, and view results.
### Python library
**Maigret can be embedded in your own Python projects.** The CLI is a thin wrapper around an async function you can call directly — build custom pipelines, feed results into your own tooling, or run it inside a larger OSINT workflow.
See the full [library usage guide](https://maigret.readthedocs.io/en/latest/library-usage.html) for a working example, async patterns, and how to filter sites by tag.
### Useful CLI flags
- `--parse URL` — parse a profile page, extract IDs/usernames, and use them to kick off a recursive search.
- `--permute` — generate likely username variants from two or more inputs (e.g. `john doe` → `johndoe`, `j.doe`, …) and search for all of them.
- `--self-check [--auto-disable]` — verify `usernameClaimed` / `usernameUnclaimed` pairs against live sites for maintainers auditing the database.
- `--ai` / `--ai-model` — run the [AI analysis](#ai-analysis) over the search results and stream a short investigation summary to the terminal.
### AI analysis
[](https://asciinema.org/a/979404)
`--ai` collects the search results, builds an internal Markdown report, and sends it to an OpenAI-compatible chat completion endpoint to produce a short, neutral investigation summary (likely real name, location, occupation, interests, languages, confidence, follow-up leads). Per-site progress is suppressed and the model's output is streamed to stdout.
```bash
export OPENAI_API_KEY=sk-...
maigret user --ai
# pick a different model
maigret user --ai --ai-model gpt-4o-mini
```
The key can also be set as `openai_api_key` in `settings.json`. The endpoint defaults to `https://api.openai.com/v1`, but `openai_api_base_url` in `settings.json` can point to any OpenAI-compatible API (Azure OpenAI, OpenRouter, a local server, …). See the [settings docs](https://maigret.readthedocs.io/en/latest/settings.html) for the full list of options.
### Tor / I2P / proxies
Maigret can route checks through a proxy, Tor, or I2P — useful for `.onion` / `.i2p` sites and for bypassing WAFs that block datacenter IPs.
```bash
# any HTTP/SOCKS proxy
maigret user --proxy socks5://127.0.0.1:1080
# Tor (default gateway socks5://127.0.0.1:9050)
maigret user --tor-proxy socks5://127.0.0.1:9050
# I2P (default gateway http://127.0.0.1:4444)
maigret user --i2p-proxy http://127.0.0.1:4444
```
Start your Tor / I2P daemon before running the command — Maigret does not manage these gateways.
### Cloudflare bypass
> **Experimental.** The Cloudflare webgate is under active development; the configuration schema, CLI behaviour, and the set of routed sites may change without backwards-compatibility guarantees.
A subset of sites in the database require a real browser to solve a JavaScript challenge. Maigret can offload these checks to a local [FlareSolverr](https://github.com/FlareSolverr/FlareSolverr) instance:
```bash
docker run -d -p 8191:8191 --name flaresolverr ghcr.io/flaresolverr/flaresolverr:latest
maigret --cloudflare-bypass
```
The bypass is opt-in (`--cloudflare-bypass` or `cloudflare_bypass.enabled` in `settings.json`) and only fires for sites whose `protection` field matches. See the [feature docs](https://maigret.readthedocs.io/en/latest/features.html#cloudflare-bypass) for backend options and configuration.
## Contributing
Add or fix new sites surgically in `data.json` (no `json.load`/`json.dump`), then run `./utils/update_site_data.py` to regenerate `sites.md` and the database metadata, and open a pull request. For more details, see the [CONTRIBUTING guide](https://github.com/soxoj/maigret/blob/main/CONTRIBUTING.md) and [development docs](https://maigret.readthedocs.io/en/latest/development.html). Release history: [CHANGELOG.md](CHANGELOG.md).
## Commercial Use
The open-source Maigret is MIT-licensed and free for commercial use without restriction — but site checks break over time and need active maintenance.
For serious commercial use — with a **daily-updated site database** or a **username-check API** — reach out: 📧 [maigret@soxoj.com](mailto:maigret@soxoj.com)
- Private site database — 5 000+ sites, updated daily (separate from the public open-source database)
- Username check API — integrate Maigret into your product
## About
### Disclaimer
**For educational and lawful purposes only.** You are responsible for complying with all applicable laws (GDPR, CCPA, etc.) in your jurisdiction. The authors bear no responsibility for misuse.
### Feedback
[Open an issue](https://github.com/soxoj/maigret/issues) · [GitHub Discussions](https://github.com/soxoj/maigret/discussions) · [Telegram](https://t.me/soxoj)
### SOWEL classification
OSINT techniques used:
- [SOTL-2.2. Search For Accounts On Other Platforms](https://sowel.soxoj.com/other-platform-accounts)
- [SOTL-6.1. Check Logins Reuse To Find Another Account](https://sowel.soxoj.com/logins-reuse)
- [SOTL-6.2. Check Nicknames Reuse To Find Another Account](https://sowel.soxoj.com/nicknames-reuse)
### License
MIT © [Maigret](https://github.com/soxoj/maigret)
---
## File: docs/source/command-line-options.rst
.. _command-line-options:
Command line options
====================
Usernames
---------
``maigret username1 username2 ...``
You can specify several usernames separated by space. Usernames are
**not** mandatory as there are other operations modes (see below).
Parsing of account pages and online documents
---------------------------------------------
``maigret --parse URL``
Maigret will try to extract information about the document/account owner
(including username and other ids) and will make a search by the
extracted username and ids. See examples in the :ref:`extracting-information-from-pages` section.
Main options
------------
Options are also configurable through settings files, see
:doc:`settings section `.
``--tags`` - Filter sites for searching by tags: sites categories and
two-letter country codes (**not a language!**). E.g. photo, dating, sport; jp, us, global.
Multiple tags can be associated with one site. **Warning**: tags markup is
not stable now. Read more :doc:`in the separate section `.
``--exclude-tags`` - Exclude sites with specific tags from the search
(blacklist). E.g. ``--exclude-tags porn,dating`` will skip all sites
tagged with ``porn`` or ``dating``. Can be combined with ``--tags`` to
include certain categories while excluding others. Read more
:doc:`in the separate section `.
``-n``, ``--max-connections`` - Allowed number of concurrent connections
**(default: 100)**.
``-a``, ``--all-sites`` - Use all sites for scan **(default: top 500)**.
``--top-sites`` - Count of sites for scan ranked by Majestic Million
**(default: top 500)**.
**Mirrors:** After the top *N* sites by Majestic Million rank are chosen (respecting
``--tags``, ``--use-disabled-sites``, etc.), Maigret may add extra sites
whose database field ``source`` names a **parent platform** that itself falls
in the Majestic Million top *N* when ranking **including disabled** sites. For example,
if ``Twitter`` ranks in the first 500 by Majestic Million, a mirror such as ``memory.lol``
(with ``source: Twitter``) is included even though it has no rank and would
otherwise be cut off. The same applies to Instagram-related mirrors (e.g.
Picuki) when ``Instagram`` is in that parent top *N* by rank—even if the
official ``Instagram`` entry is disabled and not scanned by default, its
mirrors can still be pulled in. The final list is the ranked top *N* plus
these mirrors (no fixed upper bound on mirror count).
``--timeout`` - Time (in seconds) to wait for responses from sites
**(default: 30)**. A longer timeout will be more likely to get results
from slow sites. On the other hand, this may cause a long delay to
gather all results. The choice of the right timeout should be carried
out taking into account the bandwidth of the Internet connection.
Network and proxy options
~~~~~~~~~~~~~~~~~~~~~~~~~
``--proxy PROXY_URL`` / ``-p PROXY_URL`` - Route **every** check through
the given HTTP or SOCKS proxy. Example: ``socks5://127.0.0.1:1080``,
``http://user:pass@proxy.example:3128``. This is the flag to use for
routing the whole run through Tor (``--proxy socks5://127.0.0.1:9050``),
a residential proxy, or any corporate gateway. No default.
``socks5://`` and ``socks5h://`` are interchangeable: Maigret rewrites the
scheme to the spelling expected by the transport handling each site, so
either one resolves hostnames **at the proxy** for the whole database.
``--tor-proxy TOR_PROXY_URL`` - Gateway used **only** for ``.onion``
sites in the database **(default: socks5://127.0.0.1:9050)**. Clearweb
sites are unaffected — for them Maigret uses your direct connection or
``--proxy`` if you set one. Without this flag, ``.onion`` sites are
silently skipped.
``--i2p-proxy I2P_PROXY_URL`` - Gateway used **only** for ``.i2p``
sites in the database **(default: http://127.0.0.1:4444)**. Same
"only matching protocol" rule as ``--tor-proxy``.
Maigret does not start the Tor or I2P daemon for you — launch it first.
For a full walkthrough (Tor Browser vs system ``tor`` port numbers,
Tails OS recipe, timeout/retry tuning), see :doc:`tor-and-proxies`.
``--cookies-jar-file`` - File with custom cookies in Netscape format
(aka cookies.txt). You can install an extension to your browser to
download own cookies (`Chrome `_, `Firefox `_).
``--no-recursion`` - Disable parsing pages for other usernames and
recursive search by them.
``--use-disabled-sites`` - Use disabled sites to search (may cause many
false positives).
``--id-type`` - Specify identifier(s) type (default: username).
Supported types: gaia_id, steam_id, vk_id, yandex_public_id, ok_id,
wikimapia_uid, uidme_uguid, yelp_userid, orcid, qq_id, bilibili_id.
Sites whose type does not match are filtered out automatically. See
:ref:`supported-identifier-types` for details and an example.
``--ignore-ids`` - Do not make search by the specified username or other
ids. Useful for repeated scanning with found known irrelevant usernames.
``--db`` - Load Maigret database from a JSON file or an online, valid,
JSON file. See :ref:`custom-database` below.
``--no-autoupdate`` - Disable the automatic database update check that
runs at startup. The currently cached (or bundled) database is used
as-is.
``--force-update`` - Force a database update check at startup, ignoring
the usual check interval. Implies ``--no-autoupdate`` for the rest of
the run after the explicit update finishes.
``--retries RETRIES`` - Count of attempts to restart temporarily failed
requests.
``--with-domains`` *(experimental)* - Also resolve a small set of
``{username}.`` patterns through DNS (A-records) in parallel with
the normal HTTP checks. Currently 7 entries in the database use this
path (``.ddns.net``, ``.com``, ``.pro``, ``.me``, ``.biz``, ``.email``,
``.guru``). DNS-only hits can include parking domains and catch-all
wildcards, so treat results as a lead rather than confirmation.
See the :doc:`FAQ entry on DNS domain checks `.
``--cloudflare-bypass`` *(experimental)* - Route checks for sites tagged
``protection: ["cf_js_challenge"]`` / ``["cf_firewall"]`` / ``["webgate"]``
through a local Chrome-based solver (FlareSolverr by default). The bypass
is opt-in — without this flag (or
``settings.cloudflare_bypass.enabled = true``) those sites are checked
the usual way, which Cloudflare almost always blocks: you get an UNKNOWN
status with a JS-challenge / firewall error rather than a real result.
Configure the backend in ``settings.cloudflare_bypass.modules``.
See :ref:`cloudflare-bypass`. **Experimental** — the flag, schema and
routing rules may change without backwards-compatibility guarantees.
.. _custom-database:
Using a custom sites database
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``--db`` flag accepts three forms:
1. **HTTP(S) URL** — fetched as-is, e.g.
``--db https://example.com/my_db.json``.
2. **Local file path** — absolute (``--db /tmp/private.json``) or
relative to the current working directory
(``--db LLM/maigret_private_db.json``).
3. **Module-relative path** — kept for backwards compatibility, resolved
against the installed ``maigret/`` package directory (e.g. the
default ``resources/data.json``).
Resolution order for local paths: the path is first tried as given
(absolute or cwd-relative); if that file does not exist, Maigret falls
back to the legacy module-relative resolution. If neither location
contains the file, Maigret exits with an error rather than silently
loading the bundled database.
When ``--db`` points to a custom file, automatic database updates are
skipped — the file is used exactly as provided.
On every run Maigret prints the database it actually loaded, for
example::
[+] Using sites database: /path/to/maigret_private_db.json (6 sites)
If loading the requested database fails for any other reason (corrupt
JSON, missing required keys, …), Maigret prints a warning, falls back
to the bundled database, and reports the fallback explicitly::
[-] Falling back to bundled database: /…/maigret/resources/data.json
[+] Using sites database: /…/maigret/resources/data.json (3154 sites)
A typical invocation against a private database, with auto-update
disabled and all sites scanned, looks like::
python3 -m maigret username \
--db LLM/maigret_private_db.json \
--no-autoupdate -a
Reports
-------
``-P``, ``--pdf`` - Generate a PDF report (general report on all
usernames).
``-H``, ``--html`` - Generate an HTML report file (general report on all
usernames).
``-X``, ``--xmind`` - Generate a legacy XML XMind mindmap with a manifest for
modern readers (one report per username).
``-C``, ``--csv`` - Generate a CSV report (one report per username).
``-T``, ``--txt`` - Generate a TXT report (one report per username).
``-J``, ``--json`` - Generate a JSON report of specific type: simple,
ndjson (one report per username). E.g. ``--json ndjson``
``-M``, ``--md`` - Generate a Markdown report (general report on all
usernames). See :ref:`markdown-report` below.
``--neo4j`` - Generate a Neo4j Cypher report: a ``.cypher`` script that
recreates the maigret graph (the same one produced by ``--graph``) in a
Neo4j database, importable with ``cypher-shell`` or the Neo4j Browser
(general report on all usernames). See :ref:`neo4j-export` below.
``--ai`` - Run an AI-powered analysis of the search results using an
OpenAI-compatible chat completion API. The internal Markdown report is
sent to the model, which returns a short investigation summary that is
streamed to the terminal. See :ref:`ai-analysis` below.
``--ai-model`` - Model name to use with ``--ai``. Defaults to
``openai_model`` from settings (``gpt-4o`` out of the box).
``-fo``, ``--folderoutput`` - Results will be saved to this folder,
``results`` by default. Will be created if doesn’t exist.
``--web PORT`` - Start the built-in web interface on the given port and
serve results / downloadable reports from a single page. Example:
``maigret --web 5000`` → open ``http://127.0.0.1:5000``. Full
walkthrough with screenshots: :ref:`web-interface`.
Output options
--------------
``-v``, ``--verbose`` - Display extra information and metrics.
*(loglevel=WARNING)*
``-vv``, ``--info`` - Display service information. *(loglevel=INFO)*
``-vvv``, ``--debug``, ``-d`` - Display debugging information and site
responses. *(loglevel=DEBUG)*
``--print-not-found`` - Print sites where the username was not found.
``--print-errors`` - Print errors messages: connection, captcha, site
country ban, etc.
Other operations modes
----------------------
``--version`` - Display version information and dependencies.
``--self-check`` - Do self-checking for sites and database. Each site is
tested by looking up its known-claimed and known-unclaimed usernames and
verifying that the results match expectations. Individual site failures
(network errors, unexpected exceptions, etc.) are caught and logged
without stopping the overall process, so the check always runs to
completion. After checking, Maigret reports a summary of issues found.
If any sites were disabled (see ``--auto-disable``), Maigret asks if you
want to save updates; answering y/Y will rewrite the local database.
``--auto-disable`` - Used with ``--self-check``: automatically disable
sites that fail checks (incorrect detection of claimed/unclaimed
usernames, connection errors, or unexpected exceptions). Without this
flag, ``--self-check`` only **reports** issues without modifying the
database.
``--diagnose`` - Used with ``--self-check``: print detailed diagnosis
information for each failing site, including the check type, the list
of issues found, and recommendations (e.g. suggesting a different
``checkType``).
``--submit URL`` - Do an automatic analysis of the given account URL or
site main page URL to determine the site engine and methods to check
account presence. After checking Maigret asks if you want to add the
site, answering y/Y will rewrite the local database.
.. _markdown-report:
Markdown report (LLM-friendly)
------------------------------
The ``--md`` / ``-M`` flag generates a Markdown report designed for both human reading and analysis by AI assistants (ChatGPT, Claude, etc.).
.. code-block:: console
maigret username --md
The report includes:
- **Summary** with aggregated personal data (all fullnames, locations, bios found across accounts), country tags, website tags, first/last seen timestamps.
- **Per-account sections** with profile URL, site tags, and all extracted fields (username, bio, follower count, linked accounts, etc.).
- **Possible false positives** disclaimer explaining that accounts may belong to different people.
- **Ethical use** notice about applicable data protection laws.
**Using with AI tools:**
The Markdown format is optimized for LLM context windows. You can feed the report directly to an AI assistant for follow-up analysis:
.. code-block:: console
# Generate the report
maigret johndoe --md
# Feed it to an AI tool
cat reports/report_johndoe.md | llm "Analyze this OSINT report and summarize key findings"
The structured Markdown with per-site sections makes it easy for AI tools to extract relationships, cross-reference identities, and identify patterns across accounts.
For a built-in alternative that calls the model for you and prints the
summary directly, see :ref:`ai-analysis` below.
.. _ai-analysis:
AI analysis (built-in)
----------------------
The ``--ai`` flag turns the search results into a short investigation
summary by sending the internal Markdown report to an OpenAI-compatible
chat completion API and streaming the model's reply to the terminal.
.. code-block:: console
export OPENAI_API_KEY=sk-...
maigret username --ai
# use a smaller / cheaper model
maigret username --ai --ai-model gpt-4o-mini
While ``--ai`` is active, per-site progress lines and the short text
report at the end are suppressed so the streamed summary is the main
output. The Markdown report itself is built in memory and is **not**
written to disk by ``--ai`` alone — combine with ``--md`` if you also
want the file on disk.
The summary follows a fixed format with sections for the most likely
real name, location, occupation, interests, languages, main website,
username variants, number of platforms, active years, a confidence
rating, and a short list of follow-up leads. The model is instructed
to rely only on what is supported by the report and to avoid mixing
clearly unrelated profiles into the main identity.
**Configuration.** The API key is resolved from
``settings.openai_api_key`` first, then from the ``OPENAI_API_KEY``
environment variable. The endpoint defaults to
``https://api.openai.com/v1`` and can be redirected to any
OpenAI-compatible service (Azure OpenAI, OpenRouter, a local server,
…) by setting ``openai_api_base_url`` in ``settings.json``. See
:ref:`settings` for the full list of options.
.. note::
``--ai`` makes a network request to the configured chat completion
endpoint and sends the full Markdown report (which contains the
gathered profile data). Use it only with providers and accounts
you trust with that data.
.. _neo4j-export:
Neo4j export
------------
The ``--neo4j`` flag serializes the maigret graph — the same nodes and
relationships behind ``--graph`` — into a ``*_neo4j.cypher`` script you
can load into a `Neo4j `_ database for querying and
visual exploration of how identities, accounts, sites, and extracted
data points link together.
.. code-block:: console
maigret username --neo4j
This writes ``reports/report_username_neo4j.cypher``. No extra runtime
dependency is required (it reuses the ``networkx`` graph that ``--graph``
already builds).
**What the script contains.**
- A ``CREATE CONSTRAINT ... IF NOT EXISTS`` ensuring the ``name``
property of every ``:MaigretNode`` is unique.
- One ``MERGE`` per node. Each node carries three properties:
``name`` (the unique key, e.g. ``account: https://github.com/user``),
``type`` (``username``, ``account``, ``fullname``, ``uid``, …) and
``label`` (the human-readable value).
- One ``MERGE`` per edge as a ``[:LINKED_TO]`` relationship.
Because every node and edge is ``MERGE``-d on its unique key, importing
the same report twice is **idempotent** — it updates existing nodes
instead of creating duplicates.
**Importing.**
.. code-block:: console
# via cypher-shell (Neo4j must be running; default Bolt port 7687)
cypher-shell -u neo4j -p < reports/report_username_neo4j.cypher
Alternatively, open the Neo4j Browser at ``http://localhost:7474`` and
paste the script contents into the query editor. Drop the ``-p`` flag if
authentication is disabled, or let ``cypher-shell`` prompt for the
password interactively.
**Exploring the graph.** Once imported, inspect it with Cypher, e.g.:
.. code-block:: cypher
// show the whole graph
MATCH (n:MaigretNode)-[r:LINKED_TO]->(m) RETURN n, r, m;
// accounts linked to a given username
MATCH (u:MaigretNode {type: 'username'})-[:LINKED_TO]->(a:MaigretNode {type: 'account'})
RETURN u.label, a.label;
The ``CREATE CONSTRAINT ... IF NOT EXISTS FOR ... REQUIRE`` syntax
requires Neo4j 4.4+; the ``MERGE`` statements themselves work on any
version.
---
## File: docs/source/development.rst
.. _development:
Development
==============
Frequently Asked Questions
--------------------------
1. Where to find the list of supported sites?
The human-readable list of supported sites is available in the `sites.md `_ file in the repository.
It's been generated automatically from the main JSON file with the list of supported sites.
The machine-readable JSON file with the list of supported sites is available in the
`data.json `_ file in the directory `resources`.
2. Which methods to check the account presence are supported?
The supported methods (``checkType`` values in ``data.json``) are:
- ``message`` - the most reliable method, checks if any string from ``presenceStrs`` is present and none of the strings from ``absenceStrs`` are present in the HTML response
- ``status_code`` - checks that status code of the response is 2XX
- ``response_url`` - check if there is not redirect and the response is 2XX
.. note::
Maigret natively treats specific anti-bot HTTP status codes (like LinkedIn's ``HTTP 999``) as a standard "Not Found/Available" signal instead of throwing an infrastructure Server Error, gracefully preventing false positives.
See the details of check mechanisms in the `checking.py `_ file.
.. note::
Maigret now uses the **Majestic Million** dataset for site popularity sorting instead of the discontinued Alexa Rank API. For backward compatibility with existing configurations and parsers, the ranking field in `data.json` and internal site models remains named ``alexaRank`` and ``alexa_rank``.
**Mirrors and ``--top-sites``:** When you limit scans with ``--top-sites N``, Maigret also includes *mirror* sites (entries whose ``source`` field points at a parent platform such as Twitter or Instagram) if that parent would appear in the Majestic Million top *N* when disabled sites are considered for ranking. See the **Mirrors** paragraph under ``--top-sites`` in :doc:`command-line-options`.
Testing
-------
It is recommended use Python 3.10 for testing.
Install test requirements:
.. code-block:: console
poetry install --with dev
Use the following commands to check Maigret:
.. code-block:: console
# run linter and typing checks
# order of checks:
# - critical syntax errors or undefined names
# - flake checks
# - mypy checks
make lint
# run black formatter
make format
# run testing with coverage html report
# current test coverage is 58%
make test
# open html report
open htmlcov/index.html
Running the tests offline
^^^^^^^^^^^^^^^^^^^^^^^^^
Some tests reach real sites, which is why CI runs pytest with
``--reruns 3 --reruns-delay 5``. They all carry the ``slow`` marker, so
deselecting it leaves a suite that passes with no network at all:
.. code-block:: console
pytest tests -m "not slow"
This is the invocation to use when building a distribution package, since
distribution builders run without network access. Measured on 0.6.4: the full
suite fails 9 tests offline, ``-m "not slow"`` passes 403 and deselects 24.
If you add a test that talks to the network, mark it ``slow`` — otherwise you
silently break offline builds for every downstream packager.
# get flamechart of imports to estimate startup time
make speed
Site naming conventions
-----------------------------------------------
Site names are the keys in ``data.json`` and appear in user-facing reports. Follow these rules:
- **Title Case** by default: ``Product Hunt``, ``Hacker News``.
- **Lowercase** only if the brand itself is written that way: ``kofi``, ``note``, ``hi5``.
- **No domain suffix** (``calendly.com`` → ``Calendly``), unless the domain is part of the recognized brand name: ``last.fm``, ``VC.ru``, ``Archive.org``.
- **No full UPPERCASE** unless the brand is an acronym: ``VK``, ``CNET``, ``ICQ``, ``IFTTT``.
- **No** ``www.`` **or** ``https://`` **prefix** in the name.
- **Spaces** are allowed when the brand uses them: ``Star Citizen``, ``Google Maps``.
- **{username} templates** in names are acceptable: ``{username}.tilda.ws``.
When in doubt, check how the service refers to itself on its homepage.
How to fix false-positives
-----------------------------------------------
If you want to work with sites database, don't forget to activate statistics update git hook, command for it would look like this: ``git config --local core.hooksPath .githooks/``.
You should make your git commits from your maigret git repo folder, or else the hook wouldn't find the statistics update script.
1. Determine the problematic site.
If you already know which site has a false-positive and want to fix it specifically, go to the next step.
Otherwise, simply run a search with a random username (e.g. `laiuhi3h4gi3u4hgt`) and check the results.
Alternatively, you can use the `community Telegram bot `_.
2. Open the account link in your browser and check:
- If the site is completely gone, remove it from the list
- If the site still works but looks different, update in data.json how we check it
- If the site requires login to view profiles, disable checking it
3. Find the site in the `data.json `_ file.
If the ``checkType`` method is not ``message`` and you are going to fix check, update it:
- put ``message`` in ``checkType``
- put in ``absenceStrs`` a keyword that is present in the HTML response for an non-existing account
- put in ``presenceStrs`` a keyword that is present in the HTML response for an existing account
If you have trouble determining the right keywords, you can use automatic detection by passing the account URL with the ``--submit`` option:
.. code-block:: console
maigret --submit https://my.mail.ru/bk/alex
To disable checking, set ``disabled`` to ``true`` or simply run:
.. code-block:: console
maigret --self-check --site My.Mail.ru@bk.ru
To debug the check method using the response HTML, you can run:
.. code-block:: console
maigret soxoj --site My.Mail.ru@bk.ru -d 2> response.txt
There are few options for sites data.json helpful in various cases:
- ``engine`` - a predefined check for the sites of certain type (e.g. forums), see the ``engines`` section in the JSON file
- ``headers`` - a dictionary of additional headers to be sent to the site
- ``requestHeadOnly`` - set to ``true`` if it's enough to make a HEAD request to the site
- ``regexCheck`` - a regex to check if the username is valid, in case of frequent false-positives
- ``requestMethod`` - set the HTTP method to use (e.g., ``POST``). By default, Maigret natively defaults to GET or HEAD.
- ``requestPayload`` - a dictionary with the JSON payload to send for POST requests (e.g., ``{"username": "{username}"}``), extremely useful for parsing GraphQL or modern JSON APIs.
- ``protection`` - a list of protection types detected on the site (see below).
``protection`` (site protection tracking)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``protection`` field records what kind of anti-bot protection a site uses. Maigret reads this field and automatically applies the appropriate bypass mechanism where one exists.
Two categories of tag:
- **Load-bearing.** Maigret changes its HTTP client or headers based on the tag. Currently only ``tls_fingerprint`` (switches to ``curl_cffi`` with Chrome-class TLS).
- **Documentation-only.** Maigret does **not** change behavior based on the tag; it records *why* the site is hard so a future solver can target the right set of sites without re-auditing.
Within the documentation-only tags, there is a further split that dictates whether the site is ``disabled: true``:
- ``ip_reputation`` is the **only** doc-tag that **keeps the site enabled**. It means "works for most users, fails from datacenter/cloud IPs." Disabling would silently hide a working site from anyone with a clean IP. The fix is **external** to Maigret (residential IP or ``--proxy``).
- ``cf_js_challenge``, ``cf_firewall``, ``aws_waf_js_challenge``, ``ddos_guard_challenge``, ``custom_bot_protection``, ``js_challenge`` all pair with ``disabled: true``. They mean "does not work for anyone right now"; the tag identifies the provider so that when a bypass ships, every site with that tag can be re-enabled in one pass.
Supported values:
- ``tls_fingerprint`` *(load-bearing; site stays enabled)* — the site fingerprints the TLS handshake (JA3/JA4) and blocks non-browser clients. Maigret automatically uses ``curl_cffi`` with Chrome browser emulation to bypass this. Requires the ``curl_cffi`` package (included as a dependency). Examples: Instagram, NPM, Codepen, Kickstarter, Letterboxd.
- ``ip_reputation`` *(documentation-only; site stays enabled)* — the site blocks requests from datacenter/cloud IPs regardless of headers or TLS. Cannot be bypassed automatically; run Maigret from a regular internet connection (not a datacenter) or use a proxy (``--proxy``). The site is **not** marked ``disabled`` because it continues to work for users on residential IPs. Examples: Reddit, Patreon, Figma, OnlyFans.
- ``cf_js_challenge`` *(documentation-only; pair with ``disabled: true``)* — Cloudflare Managed Challenge / Turnstile JS challenge. Symptom: HTTP 403 with ``cf-mitigated: challenge`` header; body contains ``challenges.cloudflare.com``, ``_cf_chl_opt``, ``window._cf_chl``, or "Just a moment". Not bypassable via ``curl_cffi`` TLS impersonation (verified across Chrome 123/124/131, Safari 17/18, Firefox 133/135, Edge 101 — all return the same 403 challenge page); a real browser executing the challenge JS is required to obtain the clearance cookie. Sites stay ``disabled: true`` until a CF-challenge solver is integrated. Examples: DMOJ, Elakiri, Fanlore, Bdoutdoors, TheStudentRoom, forum.hr.
- ``cf_firewall`` *(documentation-only; pair with ``disabled: true``)* — Cloudflare firewall rule / bot score block (WAF action=block, **not** action=challenge). Symptom: HTTP 403 served by Cloudflare (``server: cloudflare``, ``cf-ray`` header) **without** JS-challenge markers — body typically shows "Access denied", "Attention Required", or just a bare 1015/1016/1020 error page. Unlike ``ip_reputation``, residential IPs are **not** sufficient to bypass — Cloudflare decides based on a composite of bot score, TLS fingerprint, UA, ASN, and custom site-owner rules, so ``curl_cffi`` Chrome impersonation from a residential line still returns 403. Sites stay ``disabled: true`` until a per-site bypass (cookies, real browser, or residential+clean session) is found. Examples: Fark, Fodors, Huntingnet, Hunttalk.
- ``aws_waf_js_challenge`` *(documentation-only; pair with ``disabled: true``)* — the site is protected by AWS WAF with a JavaScript challenge. Symptom: HTTP 202 with empty body and ``x-amzn-waf-action: challenge`` header (a token-granting challenge that requires executing the CAPTCHA/challenge JS bundle). Neither ``curl_cffi`` TLS impersonation nor User-Agent changes bypass this — a real browser or the official AWS WAF challenge-solver SDK is required. Sites stay ``disabled: true`` until a solver is integrated. Example: Dreamwidth.
- ``ddos_guard_challenge`` *(documentation-only; pair with ``disabled: true``)* — DDoS-Guard (ddos-guard.net) anti-bot page. Symptom: HTTP 403 with ``server: ddos-guard`` header; body contains "DDoS-Guard". DDoS-Guard fingerprints different UAs per source IP, so a single User-Agent override does not work across environments; a JS-capable bypass or DDoS-Guard-aware solver is required. Sites stay ``disabled: true`` until a solver is integrated. Example: ForumHouse.
- ``js_challenge`` *(documentation-only; pair with ``disabled: true``)* — **fallback** for JavaScript-challenge systems whose provider cannot be identified (custom in-house challenge pages that are not Cloudflare, AWS WAF, or any other recognized vendor). Prefer a provider-specific tag whenever the provider can be pinned down from response headers or body signatures.
- ``custom_bot_protection`` *(documentation-only; pair with ``disabled: true``)* — **fallback** for non-JS-challenge bot protection served by a custom/in-house system (not Cloudflare, not AWS WAF, not DDoS-Guard). Typical symptom: HTTP 403 from the site's own origin server (``server: nginx``, AWS ELB, etc.) with a branded block page, returned regardless of TLS fingerprint or residential IP. Not generically bypassable; investigate per site (cookies, session, proxy geography). Examples: Hackerearth ("HackerEarth Guardian"), FreelanceJob (nginx-level block).
**Rule: prefer provider-specific protection tags.** When a site is blocked by an identifiable anti-bot vendor, always record the vendor in the tag (``cf_js_challenge``, ``cf_firewall``, ``aws_waf_js_challenge``, ``ddos_guard_challenge``, and future additions such as ``sucuri_challenge``, ``incapsula_challenge``). The generic ``js_challenge`` and ``custom_bot_protection`` tags are reserved for custom/unknown systems. Rationale: bypass solvers are inherently provider-specific (a Cloudflare Turnstile solver does not help with AWS WAF); recording the provider in advance lets us fan out fixes the moment a per-provider solver is added, without re-auditing every disabled site. The same principle applies to other protection categories when the provider is identifiable.
Example:
.. code-block:: json
"Instagram": {
"url": "https://www.instagram.com/{username}/",
"checkType": "message",
"presenseStrs": ["\"routePath\":\"\\/"],
"absenceStrs": ["\"routePath\":null"],
"protection": ["tls_fingerprint"]
}
``urlProbe`` (optional profile probe URL)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
By default Maigret performs the HTTP request to the same URL as ``url`` (the public profile link pattern).
If you set ``urlProbe`` in ``data.json``, Maigret **fetches** that URL for the presence check (API, GraphQL, JSON endpoint, etc.), while **reports and ``url_user``** still use ``url`` — the human-readable profile page users should open.
Placeholders: ``{username}``, ``{urlMain}``, ``{urlSubpath}`` (same as for ``url``). Example: GitHub uses ``url`` ``https://github.com/{username}`` and ``urlProbe`` ``https://api.github.com/users/{username}``; Picsart uses the web profile ``https://picsart.com/u/{username}`` and probes ``https://api.picsart.com/users/show/{username}.json``.
.. warning::
``url`` must **always** stay a human-openable profile page — it is shown to the
user and printed as the clickable result link. Never put an API / JSON / GraphQL
endpoint in ``url``; that belongs in ``urlProbe``. If the check needs an API
endpoint, keep the browsable profile in ``url`` and add the API URL as
``urlProbe`` (e.g. Weibo: ``url`` ``https://weibo.com/n/{username}``, ``urlProbe``
``https://weibo.com/ajax/profile/info?screen_name={username}``).
Implementation: ``make_site_result`` in `checking.py `__.
Site check fixes using LLM
--------------------------
.. note::
The ``LLM/`` directory at the root of the repository contains detailed instructions for editing site checks (in Markdown format): checklist, full guide to ``checkType`` / ``data.json`` / ``urlProbe``, handling false positives, searching for public JSON APIs, and the proposal log for `socid_extractor `_.
Main files:
- `site-checks-playbook.md `_ — short checklist
- `site-checks-guide.md `_ — detailed guide
- `socid_extractor_improvements.log `_ — template and entries for identity extractor improvements; the upstream how-to is `Adding a scheme `_
These files should be kept up-to-date whenever changes are made to the check logic in the code or in ``data.json``.
.. _activation-mechanism:
Activation mechanism
--------------------
The activation mechanism helps make requests to sites requiring additional authentication like cookies, JWT tokens, or custom headers.
Let's study the Vimeo site check record from the Maigret database:
.. code-block:: json
"Vimeo": {
"tags": [
"us",
"video"
],
"headers": {
"Authorization": "jwt eyJ0..."
},
"activation": {
"url": "https://vimeo.com/_rv/viewer",
"marks": [
"Something strange occurred. Please get in touch with the app's creator."
],
"method": "vimeo"
},
"urlProbe": "https://api.vimeo.com/users/{username}?fields=name...",
"checkType": "status_code",
"alexaRank": 148,
"urlMain": "https://vimeo.com/",
"url": "https://vimeo.com/{username}",
"usernameClaimed": "blue",
"usernameUnclaimed": "noonewouldeverusethis7"
},
The activation method is:
.. code-block:: python
def vimeo(site, logger, cookies={}):
headers = dict(site.headers)
if "Authorization" in headers:
del headers["Authorization"]
import requests
r = requests.get(site.activation["url"], headers=headers)
jwt_token = r.json()["jwt"]
site.headers["Authorization"] = "jwt " + jwt_token
Here's how the activation process works when a JWT token becomes invalid:
1. The site check makes an HTTP request to ``urlProbe`` with the invalid token
2. The response contains an error message specified in the ``activation``/``marks`` field
3. When this error is detected, the ``vimeo`` activation function is triggered
4. The activation function obtains a new JWT token and updates it in the site check record
5. On the next site check (either through retry or a new Maigret run), the valid token is used and the check succeeds
Step 5 works across runs because minted tokens are written to a per-user cache rather than back into the site database — see :ref:`activation-token-cache`.
Examples of activation mechanism implementation are available in `activation.py `_ file.
How to publish new version of Maigret
-------------------------------------
**Collaborats rights are requires, write Soxoj to get them**.
For new version publishing you must create a new branch in repository
with a bumped version number and actual changelog first. After it you
must create a release, and GitHub action automatically create a new
PyPi package.
- New branch example: https://github.com/soxoj/maigret/commit/e520418f6a25d7edacde2d73b41a8ae7c80ddf39
- Release example: https://github.com/soxoj/maigret/releases/tag/v0.4.1
1. Make a new branch locally with a new version name. Check the current version number here: https://pypi.org/project/maigret/.
**Increase only patch version (third number)** if there are no breaking changes.
.. code-block:: console
git checkout -b 0.4.0
2. Update Maigret version in four files manually. **All four must be in
sync** — the previous bump missed ``docs/source/conf.py`` and
``snapcraft.yaml`` and they fell behind by a release.
- ``pyproject.toml`` — single line ``version = "X.Y.Z"`` under
``[tool.poetry]``.
- ``maigret/__version__.py`` — single line ``__version__ = 'X.Y.Z'``.
- ``docs/source/conf.py`` — **two** Sphinx fields. ``release`` is the
full version (``'X.Y.Z'``); ``version`` is the short ``major.minor``
(``'X.Y'``, **without** the patch number). Update **both**.
- ``snapcraft.yaml`` — single line ``version: X.Y.Z`` (no quotes, no
``v`` prefix).
After editing, sanity-check with ``grep -rE '0\.5\.|0\.6\.|'`` to
catch any straggler reference.
3. Create a new empty text section in the beginning of the file `CHANGELOG.md` with a current date:
.. code-block:: console
## [0.4.0] - 2022-01-03
4. Get auto-generate release notes:
- Open https://github.com/soxoj/maigret/releases/new
- Click `Choose a tag`, enter `v0.4.0` (your version)
- Click `Create new tag`
- Press `+ Auto-generate release notes`
- Copy all the text from description text field below
- Paste it to empty text section in `CHANGELOG.md`
- *Close the new release page*
Keep the ``## What's Changed`` heading, the ``## New Contributors``
section and the ``**Full Changelog**`` line — every existing section in
``CHANGELOG.md`` has them.
The same notes can be generated without opening the browser, which is
handy when you want them in a file:
.. code-block:: console
gh api repos/soxoj/maigret/releases/generate-notes \
-f tag_name=v0.4.0 -f previous_tag_name=v0.3.9 -f target_commitish=main \
--jq '.body' > notes.md
This is the same generator the `+ Auto-generate release notes` button
calls, so the contributor handles are the real GitHub logins. Do not
assemble the list from ``git log`` instead: authors who commit without a
``@users.noreply.github.com`` address have no derivable handle there, and
guessing from the display name produces entries like ``@Danilo Salve``
that are not valid GitHub users.
5. Commit all the changes, push, make pull request
.. code-block:: console
git add -p
git commit -m 'Bump to YOUR VERSION'
git push origin head
6. Merge pull request
7. Create new release
- Open https://github.com/soxoj/maigret/releases/new again
- Click `Choose a tag`
- Enter actual version in format `v0.4.0`
- Click `Create new tag`
- **Set** `Target` **to your version branch** (``0.4.0``), not ``main``
- Also enter actual version in the field `Release title`
- Press `+ Auto-generate release notes`
- **Press "Publish release" button**
.. warning::
``Target`` defaults to ``main``, and that default is wrong here. The
tag must point at a commit that already contains the bumped version —
otherwise the tag lands on the tip of ``main``, where the version is
still the previous one, and ``python-publish.yml`` builds and tries to
upload a package with the **old** version number. PyPI rejects it as a
duplicate and the release job fails.
This is why the release tags of ``v0.6.1``, ``v0.6.2`` and ``v0.6.3``
all point at commits on their version branches rather than at ``main``.
Merging the pull request from step 6 first is *not* required — pointing
the tag at the version branch is what matters.
To check a published tag: ``git show v0.4.0:maigret/__version__.py``
must print the new version.
The whole step can be done from the command line instead. ``--target``
takes a branch name or a commit SHA:
.. code-block:: console
gh release create v0.4.0 --target 0.4.0 --title v0.4.0 --notes-file notes.md
If a release was published against the wrong target, delete it together
with its tag and recreate it — the ``release: published`` event fires
again and PyPI upload is retried:
.. code-block:: console
gh release delete v0.4.0 --cleanup-tag --yes
gh release create v0.4.0 --target 0.4.0 --title v0.4.0 --notes-file notes.md
8. That's all, now you can simply wait push to PyPi. You can monitor it in Action page: https://github.com/soxoj/maigret/actions/workflows/python-publish.yml
Documentation updates
---------------------
Documentations is auto-generated and auto-deployed from the ``docs`` directory.
To manually update documentation:
1. Change something in the ``.rst`` files in the ``docs/source`` directory.
2. Install ``python -m pip install -e .`` in the docs directory.
3. Run ``make singlehtml`` in the terminal in the docs directory.
4. Open ``build/singlehtml/index.html`` in your browser to see the result.
5. If you edited any English ``.rst`` text (not just code blocks), refresh the
per-language translation catalogs — see *Translations* below. Skipping this
step lets the non-English builds silently fall back to English on the
changed strings.
6. If everything is ok, commit and push your changes to GitHub.
Translations
------------
The docs are translated via Sphinx's standard gettext workflow. English ``.rst`` files
are the source of truth; translations live as ``.po`` catalogs under
``docs/source/locale//LC_MESSAGES/`` (currently only ``zh_CN``).
After editing any English ``.rst`` file, refresh the catalogs so existing
translations stay aligned with the new strings:
.. code-block:: bash
cd docs
make intl-update LANG=zh_CN
This regenerates the ``.pot`` files via ``sphinx-build -b gettext`` and runs
``sphinx-intl update`` to merge them into the per-language ``.po`` files. New
English strings appear with an empty ``msgstr ""``; changed strings get a
``#, fuzzy`` marker that translators should review and re-translate.
Preview a translated build locally:
.. code-block:: bash
make html-zh_CN
open build/html_zh_CN/index.html
CJK escape-spaces gotcha
^^^^^^^^^^^^^^^^^^^^^^^^
reStructuredText inline markup (bold, inline code, hyperlinks) requires
whitespace or punctuation on both sides to close. In English this is free: a
space or full stop always follows. In Chinese / Japanese / Korean translations
the next character is often a CJK letter with no separator, and docutils then
emits warnings like::
:1: WARNING: Inline strong start-string without end-string.
:1: WARNING: Inline interpreted text or phrase reference start-string without end-string.
The fix is an explicit RST escape-space — a backslash followed by a space —
between the closing marker and the next CJK character. In the rendered ``.rst``
this is written as ``\``; inside a ``.po`` ``msgstr`` it must be
written as ``\\`` because the ``.po`` parser eats one backslash level.
.. code-block:: po
# WRONG — warning, markup leaks past the bold
msgstr "让**所有**检查请求通过指定代理"
# RIGHT — regular space breaks markup cleanly
msgstr "让 **所有** 检查请求通过指定代理"
# RIGHT — escape-space when no visual space is wanted
msgstr "让\\ **所有**\\ 检查请求通过指定代理"
The same rule applies after inline code before a CJK character, and after a
hyperlink before a CJK opening bracket — always insert ``\\``. After
editing any ``.po`` file, run ``make html-zh_CN``; these warnings only surface
at build time.
To add a new language, run ``make intl-update LANG=`` (e.g. ``ja``,
``de``, ``pt_BR``) — this scaffolds an empty catalog. Read the Docs publishes
each language as a separate project linked under the parent (see the
`Localization guide `_);
a maintainer needs to create the translation project once in the RTD admin UI,
set its language, and mark it as a translation of the main ``maigret`` project
to enable the language switcher.
Roadmap
-------
.. warning::
This roadmap requires updating to reflect the current project status and future plans.
.. figure:: https://i.imgur.com/kk8cFdR.png
:target: https://i.imgur.com/kk8cFdR.png
:align: center
---
## File: docs/source/features.rst
.. _features:
Features
========
This is the list of Maigret features.
.. _web-interface:
Web Interface
-------------
You can run Maigret with a web interface, where you can view the graph with results and download reports of all formats on a single page.
.. image:: https://raw.githubusercontent.com/soxoj/maigret/main/static/web_interface_screenshot_start.png
:alt: Web interface: how to start
.. image:: https://raw.githubusercontent.com/soxoj/maigret/main/static/web_interface_screenshot.png
:alt: Web interface: results
Instructions:
1. Run Maigret with the ``--web`` flag and specify the port number.
.. code-block:: console
maigret --web 5000
2. Open http://127.0.0.1:5000 in your browser and enter one or more usernames to make a search.
3. Wait a bit for the search to complete and view the graph with results, the table with all accounts found, and download reports of all formats.
.. _telegram-bot:
Telegram bot
------------
A community-maintained Telegram bot lets you run Maigret without
installing anything locally.
- Working instance: `sites.google.com/view/maigret-bot-link
`_ (redirect — the
hosted bot may move between providers).
- Source code: `github.com/soxoj/maigret-tg-bot
`_.
Personal info gathering
-----------------------
Maigret does the `parsing of accounts webpages and extraction `_ of personal info, links to other profiles, etc.
Extracted info displayed as an additional result in CLI output and as tables in HTML and PDF reports.
Also, Maigret use found ids and usernames from links to start a recursive search.
Enabled by default, can be disabled with ``--no extracting``.
.. code-block:: text
$ python3 -m maigret soxoj --timeout 5
[-] Starting a search on top 500 sites from the Maigret database...
[!] You can run search by full list of sites with flag `-a`
[*] Checking username soxoj on:
...
[+] GitHub: https://github.com/soxoj
├─uid: 31013580
├─image: https://avatars.githubusercontent.com/u/31013580?v=4
├─created_at: 2017-08-14T17:03:07Z
├─location: Amsterdam, Netherlands
├─follower_count: 1304
├─following_count: 54
├─fullname: Soxoj
├─public_gists_count: 3
├─public_repos_count: 88
├─twitter_username: sox0j
├─bio: Head of OSINT Center of Excellence in @SocialLinks-IO
├─is_company: Social Links
└─blog_url: soxoj.com
...
Recursive search
----------------
Maigret has the ability to scan account pages for :ref:`common identifiers ` and usernames found in links.
When people include links to their other social media accounts, Maigret can automatically detect and initiate new searches for those profiles.
Any information discovered through this process will be shown in both the command-line interface output and generated reports.
Enabled by default, can be disabled with ``--no-recursion``.
.. code-block:: text
$ python3 -m maigret soxoj --timeout 5
[-] Starting a search on top 500 sites from the Maigret database...
[!] You can run search by full list of sites with flag `-a`
[*] Checking username soxoj on:
...
[+] GitHub: https://github.com/soxoj
├─uid: 31013580
├─image: https://avatars.githubusercontent.com/u/31013580?v=4
├─created_at: 2017-08-14T17:03:07Z
├─location: Amsterdam, Netherlands
├─follower_count: 1304
├─following_count: 54
├─fullname: Soxoj
├─public_gists_count: 3
├─public_repos_count: 88
├─twitter_username: sox0j <===== another username found here
├─bio: Head of OSINT Center of Excellence in @SocialLinks-IO
├─is_company: Social Links
└─blog_url: soxoj.com
...
Searching |████████████████████████████████████████| 500/500 [100%] in 9.1s (54.85/s)
[-] You can see detailed site check errors with a flag `--print-errors`
[*] Checking username sox0j on:
[+] Telegram: https://t.me/sox0j
├─fullname: @Sox0j
...
Username permutations
---------------------
Maigret can generate permutations of usernames. Just pass a few usernames in the CLI and use ``--permute`` flag.
Thanks to `@balestek `_ for the idea and implementation.
.. code-block:: text
$ python3 -m maigret --permute hope dream --timeout 5
[-] 12 permutations from hope dream to check...
├─ hopedream
├─ _hopedream
├─ hopedream_
├─ hope_dream
├─ hope-dream
├─ hope.dream
├─ dreamhope
├─ _dreamhope
├─ dreamhope_
├─ dream_hope
├─ dream-hope
└─ dream.hope
[-] Starting a search on top 500 sites from the Maigret database...
[!] You can run search by full list of sites with flag `-a`
[*] Checking username hopedream on:
...
Reports
-------
Maigret currently supports HTML, PDF, TXT, CSV, XMind 8 mindmap, JSON, and
Markdown reports, plus graph exports: an interactive HTML graph (``--graph``)
and a Neo4j Cypher script (``--neo4j``) for loading the results into a graph
database.
HTML/PDF reports contain:
- profile photo
- all the gathered personal info
- additional information about supposed personal data (full name, gender, location), resulting from statistics of all found accounts
Also, there is a short text report in the CLI output after the end of a searching phase.
The ``--neo4j`` flag serialises the same graph that ``--graph`` builds into an
idempotent ``.cypher`` script, importable with ``cypher-shell`` or the Neo4j
Browser. See :ref:`neo4j-export` for the schema and import instructions.
.. warning::
XMind 8 mindmaps are incompatible with XMind 2022!
AI analysis
-----------
Maigret can produce a short, human-readable investigation summary on top
of the raw search results using the ``--ai`` flag. It builds the
internal Markdown report, sends it to an OpenAI-compatible chat
completion endpoint, and streams the model's reply directly to the
terminal.
.. code-block:: console
export OPENAI_API_KEY=sk-...
maigret username --ai
The summary uses a fixed format with the most likely real name,
location, occupation, interests, languages, main website, username
variants, number of platforms, active years, a confidence rating, and a
short list of follow-up leads. While ``--ai`` is active, per-site
progress and the short text report are suppressed so the streamed
summary is the main output.
The endpoint, model, and API key are configured via ``settings.json``
(``openai_api_key``, ``openai_model``, ``openai_api_base_url``) or the
``OPENAI_API_KEY`` environment variable. Any OpenAI-compatible API can
be used (Azure OpenAI, OpenRouter, a local server, …). See
:ref:`ai-analysis` and :ref:`settings` for details.
Tags
----
The Maigret sites database very big (and will be bigger), and it is maybe an overhead to run a search for all the sites.
Also, it is often hard to understand, what sites more interesting for us in the case of a certain person.
Tags markup allows selecting a subset of sites by interests (photo, messaging, finance, etc.) or by country. Tags of found accounts grouped and displayed in the reports.
See full description :doc:`in the Tags Wiki page `.
Censorship and captcha detection
--------------------------------
Maigret can detect common errors such as censorship stub pages, CloudFlare captcha pages, and others.
If you get more them 3% errors of a certain type in a session, you've got a warning message in the CLI output with recommendations to improve performance and avoid problems.
Retries
-------
Maigret will do retries of the requests with temporary errors got (connection failures, proxy errors, etc.).
One attempt by default, can be changed with option ``--retries N``.
Database self-check
-------------------
Maigret includes a self-check mode (``--self-check``) that validates every site
in the database by looking up its known-claimed and known-unclaimed usernames
and verifying that the detection results match expectations.
The self-check is **error-resilient**: if an individual site check raises an
unexpected exception (e.g. a network error or a parsing failure), the error is
caught, logged, and recorded as an issue — the remaining sites continue to be
checked without interruption. This means the process always runs to completion,
even when checking hundreds of sites with ``-a --self-check``.
Use ``--auto-disable`` together with ``--self-check`` to automatically disable
sites that fail checks. Without it, issues are only reported. Use ``--diagnose``
to print detailed per-site diagnosis including the check type, specific issues,
and recommendations.
.. code-block:: console
# Report-only mode (no changes to the database)
maigret --self-check
# Automatically disable failing sites and save updates
maigret -a --self-check --auto-disable
# Show detailed diagnosis for each failing site
maigret -a --self-check --diagnose
Archives and mirrors checking
-----------------------------
The Maigret database contains not only the original websites, but also mirrors, archives, and aggregators. For example:
- `Picuki `_, Instagram mirror
- (no longer available) `Reddit BigData search `_
- (no longer available) `Twitter shadowban `_ checker
It allows getting additional info about the person and checking the existence of the account even if the main site is unavailable (bot protection, captcha, etc.)
.. _cloudflare-bypass:
Cloudflare webgate bypass
-------------------------
.. warning::
**Experimental feature.** The Cloudflare webgate is under active
development. The configuration schema, CLI flag behaviour, and the set
of sites that route through it may change without backwards-compatibility
guarantees. Expect rough edges (CF rate limits, occasional solver
failures) and report issues so they can be ironed out.
Some sites sit behind a full Cloudflare JavaScript challenge or a CF firewall
hard block — these are tagged ``protection: ["cf_js_challenge"]`` or
``protection: ["cf_firewall"]`` in the database and are normally kept disabled
because neither aiohttp nor curl_cffi can solve the JS challenge on their own.
Maigret can offload these checks to a local Chrome-based solver. Two backends
are supported, configured in ``settings.json`` under
``cloudflare_bypass.modules`` (the first reachable module wins; subsequent
ones are tried as a fallback chain):
* **FlareSolverr** (recommended). Runs a real Chrome instance and exposes a
JSON API. The upstream HTTP status, headers and final URL are preserved, so
``checkType: status_code`` and ``checkType: response_url`` keep working
through the bypass.
.. code-block:: console
docker run -d -p 8191:8191 --name flaresolverr ghcr.io/flaresolverr/flaresolverr:latest
* **CloudflareBypassForScraping** (legacy fallback). Returns rendered HTML
only, so the upstream status code is lost — ``checkType: message`` keeps
working but ``status_code`` checks misfire (treated as 200 on success).
Activate the bypass either with the CLI flag::
maigret --cloudflare-bypass
or by setting ``cloudflare_bypass.enabled`` to ``true`` in ``settings.json``.
The web UI (``python -m maigret.web.app``) reads the same setting — there is
no separate toggle in the form, so flipping ``enabled`` is what activates
the bypass for browser-driven runs.
The bypass only fires for sites whose ``protection`` field intersects
``cloudflare_bypass.trigger_protection`` (default
``["cf_js_challenge", "cf_firewall", "webgate"]``); all other sites use the
normal aiohttp / curl_cffi path.
If all configured modules are unreachable, affected sites get an UNKNOWN
status with an actionable error pointing at the first module's URL — the
fix is almost always to start the FlareSolverr container.
FlareSolverr session reuse is automatic: Maigret pins a single
``session: -`` per run, so cf_clearance cookies are
shared between checks of the same domain (5–10× faster on subsequent
requests to that host).
Activation
----------
The activation mechanism helps make requests to sites requiring additional authentication like cookies, JWT tokens, or custom headers.
It works by implementing a custom function that:
1. Makes a specialized HTTP request to a specific website endpoint
2. Processes the response
3. Updates the headers/cookies for that site in the local Maigret database
Since activation only triggers after encountering specific errors, a retry (or another Maigret run) is needed to obtain a valid response with the updated authentication.
The activation mechanism is enabled by default, and cannot be disabled at the moment.
See for more details in Development section :ref:`activation-mechanism`.
.. _extracting-information-from-pages:
Extraction of information from account pages
--------------------------------------------
Maigret can parse URLs and content of web pages by URLs to extract info about account owner and other meta information.
You must specify the URL with the option ``--parse``, it's can be a link to an account or an online document. List of supported sites `see here `_.
After the end of the parsing phase, Maigret will start the search phase by :doc:`supported identifiers ` found (usernames, ids, etc.).
.. code-block:: console
$ maigret --parse https://docs.google.com/spreadsheets/d/1HtZKMLRXNsZ0HjtBmo0Gi03nUPiJIA4CC4jTYbCAnXw/edit\#gid\=0
Scanning webpage by URL https://docs.google.com/spreadsheets/d/1HtZKMLRXNsZ0HjtBmo0Gi03nUPiJIA4CC4jTYbCAnXw/edit#gid=0...
┣╸org_name: Gooten
┗╸mime_type: application/vnd.google-apps.ritz
Scanning webpage by URL https://clients6.google.com/drive/v2beta/files/1HtZKMLRXNsZ0HjtBmo0Gi03nUPiJIA4CC4jTYbCAnXw?fields=alternateLink%2CcopyRequiresWriterPermission%2CcreatedDate%2Cdescription%2CdriveId%2CfileSize%2CiconLink%2Cid%2Clabels(starred%2C%20trashed)%2ClastViewedByMeDate%2CmodifiedDate%2Cshared%2CteamDriveId%2CuserPermission(id%2Cname%2CemailAddress%2Cdomain%2Crole%2CadditionalRoles%2CphotoLink%2Ctype%2CwithLink)%2Cpermissions(id%2Cname%2CemailAddress%2Cdomain%2Crole%2CadditionalRoles%2CphotoLink%2Ctype%2CwithLink)%2Cparents(id)%2Ccapabilities(canMoveItemWithinDrive%2CcanMoveItemOutOfDrive%2CcanMoveItemOutOfTeamDrive%2CcanAddChildren%2CcanEdit%2CcanDownload%2CcanComment%2CcanMoveChildrenWithinDrive%2CcanRename%2CcanRemoveChildren%2CcanMoveItemIntoTeamDrive)%2Ckind&supportsTeamDrives=true&enforceSingleParent=true&key=AIzaSyC1eQ1xj69IdTMeii5r7brs3R90eck-m7k...
┣╸created_at: 2016-02-16T18:51:52.021Z
┣╸updated_at: 2019-10-23T17:15:47.157Z
┣╸gaia_id: 15696155517366416778
┣╸fullname: Nadia Burgess
┣╸email: nadia@gooten.com
┣╸image: https://lh3.googleusercontent.com/a-/AOh14GheZe1CyNa3NeJInWAl70qkip4oJ7qLsD8vDy6X=s64
┗╸email_username: nadia
.. code-block:: console
$ maigret.py --parse https://steamcommunity.com/profiles/76561199113454789
Scanning webpage by URL https://steamcommunity.com/profiles/76561199113454789...
┣╸steam_id: 76561199113454789
┣╸nickname: Pok
┗╸username: Machine42
Simple API
----------
Maigret can be easily integrated with the use of Python package `maigret `_.
Example: the community `Telegram bot `_
---
## File: docs/source/index.rst
.. _index:
Welcome to the Maigret docs!
============================
**Maigret** is an easy-to-use and powerful OSINT tool for collecting a dossier on a person by a username (alias) only.
This is achieved by checking for accounts on a huge number of sites and gathering all the available information from web pages.
The project's main goal — give to OSINT researchers and pentesters a **universal tool** to get maximum information
about a person of interest by a username and integrate it with other tools in automatization pipelines.
.. warning::
**This tool is intended for educational and lawful purposes only.**
The developers do not endorse or encourage any illegal activities or misuse of this tool.
Regulations regarding the collection and use of personal data vary by country and region,
including but not limited to GDPR in the EU, CCPA in the USA, and similar laws worldwide.
It is your sole responsibility to ensure that your use of this tool complies with all applicable laws
and regulations in your jurisdiction. Any illegal use of this tool is strictly prohibited,
and you are fully accountable for your actions.
The authors and developers of this tool bear no responsibility for any misuse
or unlawful activities conducted by its users.
You may be interested in:
-------------------------
- :doc:`Quick start `
- :doc:`Usage examples `
- :doc:`FAQ `
- :doc:`Command line options `
- :doc:`Features list `
- :doc:`Library usage `
- :doc:`Tor, I2P, and proxies `
.. toctree::
:hidden:
:caption: Sections
quick-start
installation
usage-examples
faq
command-line-options
features
philosophy
supported-identifier-types
tags
development
.. toctree::
:hidden:
:caption: Advanced usage
library-usage
settings
tor-and-proxies
.. toctree::
:hidden:
:caption: Use cases
use-cases/crypto
use-cases/scientists
---
## File: docs/source/installation.rst
.. _installation:
Installation
============
Maigret can be installed using pip, Docker, or simply can be launched from the cloned repo.
Also, it is available online via the `community Telegram bot `_,
source code of a bot is `available on GitHub `_.
Windows Standalone EXE-binaries
-------------------------------
A standalone ``maigret_standalone.exe`` for Windows is published in the
`Releases section `_ of the GitHub
repository. A fresh build is produced automatically after each commit to the
**main** and **dev** branches.
There are two ways to launch the EXE:
* **Double-click it from Explorer.** Maigret will prompt you for a username,
run a default search, and pause at the end so the printed report links
remain on screen until you press Enter.
* **Run it from a terminal** for full control over options:
1. Press ``Win+R``, type ``cmd``, and hit Enter (or use PowerShell).
2. Change to the folder where you saved the file, e.g.
``cd %USERPROFILE%\Downloads``.
3. Run it with at least one username:
.. code-block:: bat
maigret_standalone.exe USERNAME
maigret_standalone.exe USERNAME --html :: also save an HTML report
maigret_standalone.exe USERNAME --pdf :: also save a PDF report
maigret_standalone.exe --help :: list all options
Reports are written next to the EXE in a ``reports\`` subfolder.
Video guide on how to run it: https://youtu.be/qIgwTZOmMmM.
Cloud Shells and Jupyter notebooks
----------------------------------
In case you don't want to install Maigret locally, you can use cloud shells and Jupyter notebooks.
Press one of the buttons below and follow the instructions to launch it in your browser.
.. only:: html
.. image:: https://user-images.githubusercontent.com/27065646/92304704-8d146d80-ef80-11ea-8c29-0deaabb1c702.png
:target: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/soxoj/maigret&tutorial=README.md
:alt: Open in Cloud Shell
.. image:: https://replit.com/badge/github/soxoj/maigret
:target: https://repl.it/github/soxoj/maigret
:alt: Run on Replit
:height: 50
.. image:: https://colab.research.google.com/assets/colab-badge.svg
:target: https://colab.research.google.com/gist/soxoj/879b51bc3b2f8b695abb054090645000/maigret-collab.ipynb
:alt: Open In Colab
:height: 45
.. image:: https://mybinder.org/badge_logo.svg
:target: https://mybinder.org/v2/gist/soxoj/9d65c2f4d3bec5dd25949197ea73cf3a/HEAD
:alt: Open In Binder
:height: 45
.. only:: latex
Cloud Shell: https://console.cloud.google.com/cloudshell/open?git_repo=https://github.com/soxoj/maigret&tutorial=README.md
Replit: https://repl.it/github/soxoj/maigret
Google Colab: https://colab.research.google.com/gist/soxoj/879b51bc3b2f8b695abb054090645000/maigret-collab.ipynb
Binder: https://mybinder.org/v2/gist/soxoj/9d65c2f4d3bec5dd25949197ea73cf3a/HEAD
Local installation from PyPi
----------------------------
Maigret ships with a bundled site database. After installation from PyPI (or any other method), it can **automatically fetch a newer compatible database from GitHub** when you run it—see :ref:`database-auto-update` in :doc:`settings`.
.. note::
Python 3.10 or higher and pip is required, **Python 3.11 is recommended.**
.. code-block:: bash
# install from pypi
pip3 install maigret
# usage
maigret username
PDF report support is shipped as an **optional extra** because it relies on
system-level graphics libraries that pip cannot install for you. If you plan to
use ``--pdf``, install Maigret with the ``pdf`` extra:
.. code-block:: bash
pip3 install 'maigret[pdf]'
See :ref:`pdf-extra` below for the full background on why PDF support is
optional and how to fix the most common build errors.
Development version (GitHub)
----------------------------
.. code-block:: bash
git clone https://github.com/soxoj/maigret && cd maigret
pip3 install .
# OR
pip3 install git+https://github.com/soxoj/maigret.git
# usage
maigret username
# OR use poetry in case you plan to develop Maigret
pip3 install poetry
poetry run maigret
Docker
------
.. code-block:: bash
# official image of the development version, updated from the github repo
docker pull soxoj/maigret
# usage
docker run -v /mydir:/app/reports soxoj/maigret:latest username --html
# manual build
docker build -t maigret .
Troubleshooting
---------------
If you encounter build errors during installation such as ``cannot find ft2build.h``
or errors related to ``reportlab`` / ``_renderPM``, you need to install system-level
dependencies required to compile native extensions.
**Debian/Ubuntu/Kali:**
.. code-block:: bash
sudo apt install -y libfreetype6-dev libjpeg-dev libffi-dev
**Fedora/RHEL/CentOS:**
.. code-block:: bash
sudo dnf install -y freetype-devel libjpeg-devel libffi-devel
**Arch Linux:**
.. code-block:: bash
sudo pacman -S freetype2 libjpeg-turbo libffi
**macOS (Homebrew):**
.. code-block:: bash
brew install freetype
After installing the system dependencies, retry the maigret installation.
If you continue to have issues, consider using Docker instead, which includes all
necessary dependencies.
.. _pdf-extra:
Optional: PDF reports (``maigret[pdf]``)
----------------------------------------
The ``--pdf`` report format is shipped as an optional extra. To enable it:
.. code-block:: bash
pip3 install 'maigret[pdf]'
If PDF support is not installed and you pass ``--pdf``, Maigret prints a
warning and continues without crashing — every other output format
(``--html``, ``--json``, ``--csv``, ``--txt``, ``--xmind``, ``--graph``)
keeps working.
Why is PDF optional?
~~~~~~~~~~~~~~~~~~~~
Maigret renders PDFs by converting an HTML template, and that conversion
pipeline ultimately depends on the ``cairo`` graphics library through a
chain of Python packages roughly shaped like::
maigret[pdf] → xhtml2pdf → svglib → rlPyCairo → pycairo → libcairo2 (system)
The bottom of that chain is a C library — ``libcairo2`` — that has to exist
on the host *before* pip can build the Python bindings. The Python binding
package (``pycairo``) currently ships **only Windows wheels** on PyPI; on
Linux and macOS pip falls back to building from source, and the build fails
the moment ``pkg-config`` cannot find ``cairo``. The error looks like::
../cairo/meson.build:31:12: ERROR: Dependency "cairo" not found (tried pkg-config)
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
Pulling this whole chain for every Maigret install just so the much smaller
group of users who actually want PDFs can have them is a poor trade — so
``xhtml2pdf`` is gated behind the ``pdf`` extra.
Two more packages — ``arabic-reshaper`` and ``python-bidi`` — are bundled
into the same extra. Maigret core never imports them; they are only used
by ``xhtml2pdf`` to shape Arabic glyphs and lay out right-to-left text in
PDFs. ``python-bidi`` v0.5+ is also a Rust binding, so on niche platforms
without a published wheel it would otherwise pull in a Cargo build for
users who never asked for PDF support.
Installing the system prerequisites
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Install the cairo headers, ``pkg-config``, and a working C toolchain
*before* running ``pip install 'maigret[pdf]'``.
**Debian / Ubuntu / Linux Mint / Kali:**
.. code-block:: bash
sudo apt update
sudo apt install -y libcairo2-dev pkg-config python3-dev build-essential
pip3 install --upgrade pip setuptools wheel
pip3 install 'maigret[pdf]'
**Fedora / RHEL / CentOS:**
.. code-block:: bash
sudo dnf install -y cairo-devel pkgconfig python3-devel gcc
pip3 install 'maigret[pdf]'
**Arch Linux:**
.. code-block:: bash
sudo pacman -S cairo pkgconf base-devel
pip3 install 'maigret[pdf]'
**Alpine Linux:**
.. code-block:: bash
sudo apk add cairo-dev pkgconf python3-dev build-base
pip3 install 'maigret[pdf]'
**macOS (Homebrew):**
.. code-block:: bash
brew install cairo pkg-config
pip3 install --upgrade pip setuptools wheel
pip3 install 'maigret[pdf]'
**Windows:**
No system packages are needed — ``pycairo`` ships prebuilt wheels for
Windows. Just run:
.. code-block:: bash
pip install 'maigret[pdf]'
**Google Cloud Shell / Colab / Replit / generic CI:**
These environments behave like Debian/Ubuntu — install the same
``libcairo2-dev pkg-config python3-dev build-essential`` triple before
``pip install 'maigret[pdf]'``. If you do not control the base image and
cannot ``apt install``, skip the extra and use ``--html`` reports instead;
HTML reports contain the same data and open in any browser.
``maigret: command not found`` after install
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If pip prints warnings like::
WARNING: The scripts maigret and update_sitesmd are installed in
'/home//.local/bin' which is not on PATH.
…and ``maigret --version`` then fails with ``command not found``, your
``--user`` install put the entry-point script in a directory the shell does
not search. Add it to ``PATH``:
.. code-block:: bash
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc
Or install into a virtual environment, where the entry point lands in the
venv's ``bin/`` automatically:
.. code-block:: bash
python3 -m venv ~/.venvs/maigret
source ~/.venvs/maigret/bin/activate
pip install 'maigret[pdf]' # or just `pip install maigret`
Optional: Cloudflare bypass solver
----------------------------------
.. warning::
**Experimental.** The Cloudflare webgate is under active development;
the configuration schema and CLI behaviour may change without
backwards-compatibility guarantees.
Sites tagged ``cf_js_challenge`` / ``cf_firewall`` need a real browser to pass
their JavaScript challenge. To check those sites you can run a local
`FlareSolverr `_ instance —
Maigret will route protected checks to it when ``--cloudflare-bypass`` is set:
.. code-block:: bash
docker run -d -p 8191:8191 --name flaresolverr ghcr.io/flaresolverr/flaresolverr:latest
This is **optional** — Maigret runs without it; only sites whose
``protection`` field intersects ``settings.cloudflare_bypass.trigger_protection``
require the solver. See :ref:`cloudflare-bypass` for details.
---
## File: docs/source/philosophy.rst
.. _philosophy:
Philosophy
==========
*The Commissioner Jules Maigret is a fictional French police detective, created by Georges Simenon.
His investigation method is based on understanding the personality of different people and their
interactions.*
TL;DR: Username => Dossier
Maigret is designed to gather all the available information about person by his username.
What kind of information is this? First, links to person accounts. Secondly, all the machine-extractable
pieces of info, such as: other usernames, full name, URLs to people's images, birthday, location (country,
city, etc.), gender.
All this information forms some dossier, but it also useful for other tools and analytical purposes.
Each collected piece of data has a label of a certain format (for example, ``follower_count`` for the number
of subscribers or ``created_at`` for account creation time) so that it can be parsed and analyzed by various
systems and stored in databases.
Origins
-------
Maigret started from studying what OSINT investigators actually use in practice — and from
the realization that many popular tools do not deliver real investigative value. The original
research behind this observation is summarized in the article
`What's wrong with namecheckers `_.
For a broader landscape of username-checking tools, see the curated
`OSINT namecheckers list `_.
Two ideas grew out of that research:
- `socid-extractor `_
(`documentation `_) — a library focused on pulling
structured identity data (user IDs, full names, linked accounts, bios, timestamps, etc.) out of
account pages and public API responses, so that finding an account is not the end of the pipeline.
- **Maigret** itself — which started as a fork of
`Sherlock `_ but has long since outgrown the
original project in coverage, extraction depth, and check reliability. Today Maigret is used
as a component by major OSINT vendors in their commercial products.
---
## File: docs/source/quick-start.rst
.. _quick-start:
Quick start
===========
After :doc:`installing Maigret `, you can begin searching by providing one or more usernames to look up:
``maigret username1 username2 ...``
Maigret will search for accounts with the specified usernames across a vast number of websites. It will provide you with a list
of URLs to any discovered accounts, along with relevant information extracted from those profiles.
.. image:: maigret_screenshot.png
:alt: Maigret search results screenshot
:align: center
---
## File: docs/source/settings.rst
.. _settings:
Settings
==============
.. warning::
The settings system is under development and may be subject to change.
Options are also configurable through settings files. See
`settings JSON file `_
for the list of currently supported options.
After start Maigret tries to load configuration from the following sources in exactly the same order:
.. code-block:: console
# relative path, based on installed package path
resources/settings.json
# absolute path, system-wide policy set by an administrator
/etc/maigret/settings.json
# absolute path, configuration file in home directory
~/.maigret/settings.json
# relative path, based on current working directory
settings.json
Missing any of these files is not an error.
If the next settings file contains already known option,
this option will be rewritten. So it is possible to make
custom configuration for different users and directories.
.. _database-auto-update:
Database auto-update
--------------------
Maigret ships with a bundled site database, but it gets outdated between releases. To keep the database current, Maigret automatically checks for updates on startup.
**How it works:**
1. On startup, Maigret checks if more than 24 hours have passed since the last update check.
2. If so, it fetches a lightweight metadata file (~200 bytes) from GitHub to see if a newer database is available.
3. If a newer, compatible database exists, Maigret downloads it to ``~/.maigret/data.json`` and uses it instead of the bundled copy.
4. If the download fails or the new database is incompatible with your Maigret version, the bundled database is used as a fallback.
The downloaded database has **higher priority** than the bundled one — it replaces, not overlays.
**Status messages** are printed only when an action occurs:
.. code-block:: text
[*] DB auto-update: checking for updates...
[+] DB auto-update: database updated successfully (3180 sites)
[*] DB auto-update: database is up to date (3157 sites)
[!] DB auto-update: latest database requires maigret >= 0.6.0, you have 0.5.0
**Forcing an update:**
Use the ``--force-update`` flag to check for updates immediately, ignoring the check interval:
.. code-block:: console
maigret username --force-update
The update happens at startup, then the search continues normally with the freshly downloaded database.
**Disabling auto-update:**
Use the ``--no-autoupdate`` flag to skip the update check entirely:
.. code-block:: console
maigret username --no-autoupdate
Or set it permanently in ``~/.maigret/settings.json``:
.. code-block:: json
{
"no_autoupdate": true
}
This is recommended for **Docker containers**, **CI pipelines**, and **air-gapped environments**.
**Configuration options** (in ``settings.json``):
.. list-table::
:header-rows: 1
:widths: 35 15 50
* - Setting
- Default
- Description
* - ``no_autoupdate``
- ``false``
- Disable auto-update entirely
* - ``autoupdate_check_interval_hours``
- ``24``
- How often to check for updates (in hours)
* - ``db_update_meta_url``
- GitHub raw URL
- URL of the metadata file (for custom mirrors)
**Using a custom database** with ``--db`` always skips auto-update — you are explicitly choosing your data source.
.. _activation-token-cache:
Activation token cache
----------------------
A handful of sites reject anonymous requests until a short-lived token is obtained: Twitter needs a guest token, Vimeo a JWT, Weibo and Wikimapia a session cookie, OnlyFans a signed header pair, ProtonMail a bearer token. Maigret mints these during a run, the first time such a site answers with a challenge instead of a profile page.
Minted values are cached in:
.. code-block:: console
~/.maigret/activation.json
so the next run starts with a warm token and skips the extra round trip. The file is created with ``0600`` permissions because these tokens are session credentials — treat it as one, and do not attach it to a bug report.
**What is stored:** only the header values that differ from what the site database shipped. Storing the whole header set would pin the shipped values, so a later database update could never change them again.
**Safe to delete:** yes. The next run mints whatever it needs again; the only cost is one extra request per affected site.
**Why it is not kept in the site database:** the database is package data. It is versioned, checksum-verified and replaced wholesale by :ref:`database-auto-update`, and on a system-wide install (distribution package, snap, ``/nix/store``) its directory is not writable at all. Tokens are per-user state with a different lifetime, so they live in a separate file.
Cloudflare webgate
------------------
.. warning::
**Experimental.** The ``cloudflare_bypass`` block is under active
development; field names, defaults, and the trigger-protection routing
rules may change without backwards-compatibility guarantees.
The ``cloudflare_bypass`` block in ``settings.json`` configures the optional
bypass described in :ref:`cloudflare-bypass`. The same block is honoured by
the CLI, the Python API, and the web UI (``python -m maigret.web.app``); set
``enabled: true`` once and every entry point routes cf-protected sites
through the solver.
**Minimal FlareSolverr setup.** Start the solver in Docker, then drop the
following snippet into ``~/.maigret/settings.json`` (or any path listed in
:ref:`settings`):
.. code-block:: bash
docker run -d -p 8191:8191 --name flaresolverr \
ghcr.io/flaresolverr/flaresolverr:latest
.. code-block:: json
{
"cloudflare_bypass": {
"enabled": true,
"modules": [
{
"name": "flaresolverr",
"method": "json_api",
"url": "http://localhost:8191/v1",
"max_timeout_ms": 60000
}
]
}
}
That is enough — ``session_prefix`` and ``trigger_protection`` fall back to
sensible defaults (``"maigret"`` and
``["cf_js_challenge", "cf_firewall", "webgate"]`` respectively). On the next
run, Maigret logs ``Cloudflare webgate active: ...`` and routes matching
sites through the solver.
Default value (full schema with every supported field):
.. code-block:: json
{
"cloudflare_bypass": {
"enabled": false,
"session_prefix": "maigret",
"trigger_protection": ["cf_js_challenge", "cf_firewall", "webgate"],
"modules": [
{
"name": "flaresolverr",
"method": "json_api",
"url": "http://localhost:8191/v1",
"max_timeout_ms": 60000
},
{
"name": "chrome_webgate",
"method": "url_rewrite",
"url": "http://localhost:8000/html?url={url}&retries=1"
}
]
}
}
**Fields.**
.. list-table::
:header-rows: 1
:widths: 30 70
* - Field
- Description
* - ``enabled``
- When ``true``, the bypass is active for every run; when ``false``
(the default), it activates only on ``--cloudflare-bypass``.
* - ``trigger_protection``
- List of ``site.protection`` values that route a check through the
webgate. Sites whose protection is empty or doesn't intersect this
list use the default (aiohttp / curl_cffi) checker.
* - ``session_prefix``
- Prefix for the FlareSolverr ``session`` field. Maigret appends the
process PID so concurrent runs don't collide. Reusing a session
caches cf_clearance between checks of the same domain.
* - ``modules``
- Ordered list of backend modules. The first reachable module
handles the check; later ones serve as a fallback chain.
**Module methods.**
* ``json_api`` — FlareSolverr-compatible POST endpoint at ``url``.
Preserves real upstream HTTP status, headers and final URL.
Optional ``max_timeout_ms`` (default ``60000``) is the per-request
budget the solver is allowed to spend on the JS challenge.
* ``url_rewrite`` — legacy CloudflareBypassForScraping endpoint. The
``url`` must contain a ``{url}`` placeholder; the original probe URL
is URL-encoded and substituted in. Returns rendered HTML only —
``checkType: status_code`` and ``response_url`` checks misfire under
this method (treated as a synthetic HTTP 200 on success).
**Optional ``proxy`` field (``json_api`` only).**
A module may carry a ``proxy`` entry that the solver routes the upstream
request through. Useful when a site enforces ``ip_reputation`` rules
that block the solver host. Two forms are accepted:
.. code-block:: json
{ "proxy": "socks5://localhost:1080" }
.. code-block:: json
{ "proxy": { "url": "http://gw.example:3128",
"username": "u",
"password": "p" } }
Only ``url``/``username``/``password`` are forwarded; other keys are
dropped. Cloudflare ``Error 1015 / 1020`` responses indicate the IP is
rate-limited or banned — switch the proxy rather than retrying.
.. _ai-analysis-settings:
AI analysis
-----------
The ``--ai`` flag (see :ref:`ai-analysis`) talks to an OpenAI-compatible
chat completion API. Three settings control how that request is made:
.. list-table::
:header-rows: 1
:widths: 35 25 40
* - Setting
- Default
- Description
* - ``openai_api_key``
- ``""`` (empty)
- API key. If empty, Maigret falls back to the ``OPENAI_API_KEY``
environment variable.
* - ``openai_model``
- ``gpt-4o``
- Default model name. Overridable per-run with ``--ai-model``.
* - ``openai_api_base_url``
- ``https://api.openai.com/v1``
- Base URL of the chat completion API. Point this at any
OpenAI-compatible service (Azure OpenAI, OpenRouter, a local
server, …) to use it instead of OpenAI directly.
Example ``~/.maigret/settings.json`` snippet using a non-OpenAI
endpoint:
.. code-block:: json
{
"openai_api_key": "sk-...",
"openai_model": "gpt-4o-mini",
"openai_api_base_url": "https://openrouter.ai/api/v1"
}
The key resolution order is ``settings.openai_api_key`` → ``OPENAI_API_KEY``
environment variable; the first non-empty value wins.
.. note::
``--ai`` sends the full internal Markdown report (which contains the
gathered profile data) to the configured endpoint. Only use providers
and accounts you trust with that data.
---
## File: docs/source/supported-identifier-types.rst
.. _supported-identifier-types:
Supported identifier types
==========================
Maigret can search against not only ordinary usernames, but also through certain common identifiers. There is a list of all currently supported identifiers.
- **gaia_id** - Google inner numeric user identifier, in former times was placed in a Google Plus account URL.
- **steam_id** - Steam inner numeric user identifier.
- **wikimapia_uid** - Wikimapia.org inner numeric user identifier.
- **uidme_uguid** - uID.me inner numeric user identifier.
- **yandex_public_id** - Yandex sites inner letter user identifier. See also: `YaSeeker `_.
- **vk_id** - VK.com inner numeric user identifier.
- **ok_id** - OK.ru inner numeric user identifier.
- **yelp_userid** - Yelp inner user identifier.
- **qq_id** - QQ (Tencent) numeric account number, also used as the Qzone profile id.
- **bilibili_id** - Bilibili numeric user id (the ``mid`` in a ``space.bilibili.com/{mid}`` profile URL).
Example
-------
Pass the identifier as the target and select its type with ``--id-type``. For example, to look up the QQ number ``10001``:
.. code-block:: console
$ maigret 10001 --id-type qq_id
Maigret resolves the account via the Qzone portrait API and, when found, extracts the nickname and avatar. Any site whose ``type`` does not match the given ``--id-type`` is skipped, so a plain ``maigret 10001`` (default ``username``) will not query QQ.
Or, to look up the Bilibili user id ``2`` (the ``mid`` from ``space.bilibili.com/2``):
.. code-block:: console
$ maigret 2 --id-type bilibili_id
Maigret resolves the account via the Bilibili user-card API and, when found, extracts the nickname, avatar, signature and follower count.
## 2. Official Technical Reference & Guides (soxoj/soxoj.github.io)
# al-folio
## User community
The vibrant community of **al-folio** users is growing!
Academics around the world use this theme for their homepages, blogs, lab pages, as well as webpages for courses, workshops, conferences, meetups, and more.
Check out the community webpages below.
Feel free to add your own page(s) by sending a PR.
| Academics | [★](https://martinbulla.github.io)
[★](https://maruan.alshedivat.com)
[★](https://www.cs.columbia.edu/~chen1ru/)
[★](https://maithraraghu.com)
[★](https://platanios.org)
[★](https://otiliastr.github.io)
[★](https://www.maths.dur.ac.uk/~sxwc62/)
[★](https://jessachandler.com/)
[★](https://mayankm96.github.io/)
[★](https://markdean.info/)
[★](https://kakodkar.github.io/)
[★](https://sahirbhatnagar.com/)
[★](https://spd.gr/)
[★](https://jay-sarkar.github.io/)
[★](https://aborowska.github.io/)
[★](https://aditisgh.github.io/)
[★](https://alexhaydock.co.uk/)
[★](https://alixkeener.net/)
[★](https://andreea7b.github.io/)
[★](https://rishabhjoshi.github.io/)
[★](https://sheelabhadra.github.io/)
[★](https://giograno.me/)
[★](https://immsrini.github.io/)
[★](https://apooladian.github.io/)
[★](https://chinmoy-dutta.github.io/)
[★](https://liamcli.com/)
[★](https://yoonholee.com/)
[★](https://zrqiao.github.io/)
[★](https://abstractgeek.github.io/)
[★](https://www.compphys.de/)
[★](https://julianstreyczek.github.io)
[★](https://sdaza.com)
[★](https://niweera.gq)
[★](https://www.alihkw.com)
[★](https://amirpourmand.ir)
[★](https://scottleechua.github.io)
[★](https://sk1y101.github.io)
[★](https://yyang768osu.github.io)
[★](https://veedata.github.io)
[★](https://K-Wu.github.io)
[★](https://amalawilson.com)
[★](https://tirtharajdash.github.io)
[★](https://carolinacarreira.github.io)
[★](https://manandey.github.io)
[★](https://johanneshoerner.github.io)
[★](https://ioannismavromatis.com)
[★](https://taidnguyen.github.io)
[★](https://lbugnon.github.io)
[★](https://joahannes.github.io)
[★](https://dominikstrb.github.io)
[★](https://tylerbarna.com)
[★](https://daviddmc.github.io/)
[★](https://andreaskuster.ch/)
[★](https://ellisbrown.github.io/)
[★](https://noman-bashir.github.io/)
[★](https://djherron.github.io/)
[★](https://rodosingh.github.io/)
[★](https://vdivakar.github.io/)
[★](https://george-gca.github.io/)
[★](https://bashirkazimi.github.io/)
[★](https://dohaison.github.io/)
[★](https://raphaaal.github.io/)
[★](https://varuniyer.info/)
[★](https://yukimasano.github.io/)
[★](https://hashe037.github.io/)
[★](https://wang-boyu.github.io/)
[★](https://qingqingchen.info)
[★](https://bajinsheng.github.io/)
[★](https://www.silviofanzon.com/)
[★](https://kaikaiyao.github.io/)
[★](https://alchemz.github.io/)
[★](https://samadamday.com/)
[★](https://fanpu.io/)
[★](https://abigalekim.github.io/)
[★](https://lucasresck.github.io/)
[★](https://users.wpi.edu/~lfichera/)
[★](https://anmspro.github.io/)
[★](https://berlyne.net/)
[★](https://filippomazzoli.github.io/)
[★](https://www.escontrela.me/)
[★](https://raffaem.github.io/)
[★](https://cbueth.de/)
[★](https://kyleaoman.github.io/)
[★](https://decwest.github.io/)
[★](https://www.jedburkat.com)
[★](https://hrzhang.me)
[★](https://kudhru.github.io/)
[★](https://mbarbetti.github.io/)
[★](https://www.zhivotenko.com/)
[★](https://giordanodaloisio.github.io/)
[★](https://aadityaura.github.io/)
[★](https://abhinav-mehta.github.io/)
[★](https://shubhashisroydipta.com/)
[★](https://astanziola.github.io)
[★](https://tinkerer.in)
[★](https://sam-bieberich.github.io/)
[★](https://afraniomelo.github.io/en/)
[★](https://jonaruthardt.github.io)
[★](https://www.zla.app/)
[★](https://stavros.github.io)
[★](https://ericslyman.com)
[★](https://ztjona.github.io/)
[★](https://chrischoi314.github.io)
[★](https://riccobelli.faculty.polimi.it)
[★](https://kishanved.tech/)
[★](https://abhilesh.github.io/)
[★](https://jackjburnett.github.io/)
[★](https://physics-morris.github.io/)
[★](https://sraf.ir)
[★](https://acad.garywei.dev/)
[★](https://tonideleo.github.io/)
[★](https://alonkellner.com/)
[★](https://berylbir.github.io/)
[★](https://thefermi0n.github.io/)
[★](https://mingsun-kaust.github.io/)
[★](https://hdocmsu.github.io/)
[★](https://trandangtrungduc.github.io/)
[★](https://kinghowler.github.io/)
[★](https://anurye.github.io/)
[★](https://charlie-xiao.github.io/)
[★](https://giuseppeperelli.github.io/)
[★](https://shlee-lab.github.io/)
[★](https://devos50.github.io/)
[★](https://kocikowski.com/)
[★](https://vmooers.github.io/)
[★](https://jpfonseca.github.io/)
[★](https://dmitryryumin.github.io/) |
| --- | --- |
| Labs | [★](https://www.haylab.caltech.edu/)
[★](https://sjkimlab.github.io/)
[★](https://systemconsultantgroup.github.io/scg-folio/)
[★](https://decisionlab.ucsf.edu/)
[★](https://programming-group.com/)
[★](https://sailing-lab.github.io/)
[★](https://inbt.jhu.edu/epidiagnostics/)
[★](https://www.nuesl.org/)
[★](https://big-culture.github.io/) |
| Courses | CMU PGM ([S-19](https://sailinglab.github.io/pgm-spring-2019/))
CMU DeepRL ([S-21](https://cmudeeprl.github.io/403_website/), [F-21](https://cmudeeprl.github.io/703website_f21/), [S-22](https://cmudeeprl.github.io/403website_s22/), [F-22](https://cmudeeprl.github.io/703website_f22/), [S-23](https://cmudeeprl.github.io/403website_s23/), [F-23](https://cmudeeprl.github.io/703website_f23/))
CMU MMML ([F-20](https://cmu-multicomp-lab.github.io/mmml-course/fall2020/), [F-22](https://cmu-multicomp-lab.github.io/mmml-course/fall2022/))
CMU AMMML ([S-22](https://cmu-multicomp-lab.github.io/adv-mmml-course/spring2022/), [S-23](https://cmu-multicomp-lab.github.io/adv-mmml-course/spring2023/))
CMU ASI ([S-23](https://cmu-multicomp-lab.github.io/asi-course/spring2023/))
CMU Distributed Systems ([S-24](https://andrew.cmu.edu/course/15-440/)) |
| Conferences & workshops | ICLR Blog Post Track ([2023](https://iclr-blogposts.github.io/2023/), [2024](https://iclr-blogposts.github.io/2024/about))
ML Retrospectives (NeurIPS: [2019](https://ml-retrospectives.github.io/neurips2019/), [2020](https://ml-retrospectives.github.io/neurips2020/); ICML: [2020](https://ml-retrospectives.github.io/icml2020/))
HAMLETS (NeurIPS: [2020](https://hamlets-workshop.github.io/))
ICBINB (NeurIPS: [2020](https://i-cant-believe-its-not-better.github.io/), [2021](https://i-cant-believe-its-not-better.github.io/neurips2021/))
Neural Compression (ICLR: [2021](https://neuralcompression.github.io/))
Score Based Methods (NeurIPS: [2022](https://score-based-methods-workshop.github.io/))
Images2Symbols (CogSci: [ 2022](https://images2symbols.github.io/))
Medical Robotics Junior Faculty Forum (ISMR: [ 2023](https://junior-forum-ismr.github.io/))
Beyond Vision: Physics meets AI (ICIAP: [2023](https://physicsmeetsai.github.io/beyond-vision/))
Workshop on Diffusion Models (NeurIPS: [2023](https://diffusionworkshop.github.io/))
Workshop on Structured Probabilistic Inference & Generative Modeling (ICML: [2023](https://spigmworkshop.github.io/), [2024](https://spigmworkshop2024.github.io/)) |
## Lighthouse PageSpeed Insights
### Desktop
[](https://htmlpreview.github.io/?https://github.com/alshedivat/al-folio/blob/main/lighthouse_results/desktop/alshedivat_github_io_al_folio_.html)
Run the test yourself: [Google Lighthouse PageSpeed Insights](https://pagespeed.web.dev/report?url=https%3A%2F%2Falshedivat.github.io%2Fal-folio%2F&form_factor=desktop)
### Mobile
[](https://htmlpreview.github.io/?https://github.com/alshedivat/al-folio/blob/main/lighthouse_results/mobile/alshedivat_github_io_al_folio_.html)
Run the test yourself: [Google Lighthouse PageSpeed Insights](https://pagespeed.web.dev/report?url=https%3A%2F%2Falshedivat.github.io%2Fal-folio%2F&form_factor=mobile)
## Table Of Contents
- [al-folio](#al-folio)
- [User community](#user-community)
- [Lighthouse PageSpeed Insights](#lighthouse-pagespeed-insights)
- [Desktop](#desktop)
- [Mobile](#mobile)
- [Table Of Contents](#table-of-contents)
- [Getting started](#getting-started)
- [Installing and Deploying](#installing-and-deploying)
- [Customizing](#customizing)
- [Features](#features)
- [Light/Dark Mode](#lightdark-mode)
- [CV](#cv)
- [People](#people)
- [Publications](#publications)
- [Collections](#collections)
- [Layouts](#layouts)
- [The iconic style of Distill](#the-iconic-style-of-distill)
- [Full support for math \& code](#full-support-for-math--code)
- [Photos, Audio, Video and more](#photos-audio-video-and-more)
- [Other features](#other-features)
- [GitHub's repositories and user stats](#githubs-repositories-and-user-stats)
- [Theming](#theming)
- [Social media previews](#social-media-previews)
- [Atom (RSS-like) Feed](#atom-rss-like-feed)
- [Related posts](#related-posts)
- [Code quality checks](#code-quality-checks)
- [FAQ](#faq)
- [Contributing](#contributing)
- [Maintainers](#maintainers)
- [All Contributors](#all-contributors)
- [Star History](#star-history)
- [License](#license)
## Getting started
Want to learn more about Jekyll? Check out [this tutorial](https://www.taniarascia.com/make-a-static-website-with-jekyll/). Why Jekyll? Read [Andrej Karpathy's blog post](https://karpathy.github.io/2014/07/01/switching-to-jekyll/)! Why write a blog? Read [Rachel Thomas blog post](https://medium.com/@racheltho/why-you-yes-you-should-blog-7d2544ac1045).
## Installing and Deploying
For installation and deployment details please refer to [INSTALL.md](INSTALL.md).
## Customizing
For customization details please refer to [CUSTOMIZE.md](CUSTOMIZE.md).
## Features
### Light/Dark Mode
This template has a built-in light/dark mode. It detects the user preferred color scheme and automatically switches to it. You can also manually switch between light and dark mode by clicking on the sun/moon icon in the top right corner of the page.
---
### CV
There are currently 2 different ways of generating the CV page content. The first one is by using a json file located in [assets/json/resume.json](assets/json/resume.json). It is a [known standard](https://jsonresume.org/) for creating a CV programmatically. The second one, currently used as a fallback when the json file is not found, is by using a yml file located in [\_data/cv.yml](_data/cv.yml). This was the original way of creating the CV page content and since it is more human readable than a json file we decided to keep it as an option.
What this means is, if there is no resume data defined in [\_config.yml](_config.yml) and loaded via a json file, it will load the contents of [\_data/cv.yml](_data/cv.yml) as fallback.
[](https://alshedivat.github.io/al-folio/cv/)
---
### People
You can create a people page if you want to feature more than one person. Each person can have its own short bio, profile picture, and you can also set if every person will appear at the same or opposite sides.
[](https://alshedivat.github.io/al-folio/people/)
---
### Publications
Your publications' page is generated automatically from your BibTex bibliography. Simply edit [\_bibliography/papers.bib](_bibliography/papers.bib). You can also add new `*.bib` files and customize the look of your publications however you like by editing [\_pages/publications.md](_pages/publications.md). By default, the publications will be sorted by year and the most recent will be displayed first. You can change this behavior and more in the `Jekyll Scholar` section in [\_config.yml](_config.yml) file.
You can add extra information to a publication, like a PDF file in the [assets/pdf/](assets/pdf/) directory and add the path to the PDF file in the BibTeX entry with the `pdf` field. Some of the supported fields are: `abstract`, `altmetric`, `arxiv`, `bibtex_show`, `blog`, `code`, `dimensions`, `doi`, `eprint`, `html`, `isbn`, `pdf`, `pmid`, `poster`, `slides`, `supp`, `video`, and `website`.
[](https://alshedivat.github.io/al-folio/publications/)
---
### Collections
This Jekyll theme implements `collections` to let you break up your work into categories. The theme comes with two default collections: `news` and `projects`. Items from the `news` collection are automatically displayed on the home page. Items from the `projects` collection are displayed on a responsive grid on projects page.
[](https://alshedivat.github.io/al-folio/projects/)
You can easily create your own collections, apps, short stories, courses, or whatever your creative work is. To do this, edit the collections in the [\_config.yml](_config.yml) file, create a corresponding folder, and create a landing page for your collection, similar to `_pages/projects.md`.
---
### Layouts
**al-folio** comes with stylish layouts for pages and blog posts.
#### The iconic style of Distill
The theme allows you to create blog posts in the [distill.pub](https://distill.pub/) style:
[](https://alshedivat.github.io/al-folio/blog/2021/distill/)
For more details on how to create distill-styled posts using `` tags, please refer to [the example](https://alshedivat.github.io/al-folio/blog/2021/distill/).
#### Full support for math & code
**al-folio** supports fast math typesetting through [MathJax](https://www.mathjax.org/) and code syntax highlighting using [GitHub style](https://github.com/jwarby/jekyll-pygments-themes). Also supports [chartjs charts](https://www.chartjs.org/), [mermaid diagrams](https://mermaid-js.github.io/mermaid/#/), and [TikZ figures](https://tikzjax.com/).
#### Photos, Audio, Video and more
Photo formatting is made simple using [Bootstrap's grid system](https://getbootstrap.com/docs/4.4/layout/grid/). Easily create beautiful grids within your blog posts and project pages, also with support for [video](https://alshedivat.github.io/al-folio/blog/2023/videos/) and [audio](https://alshedivat.github.io/al-folio/blog/2023/audios/) embeds:
---
### Other features
#### GitHub's repositories and user stats
**al-folio** uses [github-readme-stats](https://github.com/anuraghazra/github-readme-stats) and [github-profile-trophy](https://github.com/ryo-ma/github-profile-trophy) to display GitHub repositories and user stats on the `/repositories/` page.
[](https://alshedivat.github.io/al-folio/repositories/)
Edit the `_data/repositories.yml` and change the `github_users` and `github_repos` lists to include your own GitHub profile and repositories to the `/repositories/` page.
You may also use the following codes for displaying this in any other pages.
```html
{% if site.data.repositories.github_users %}
{% for user in site.data.repositories.github_users %} {% include repository/repo_user.liquid username=user %} {% endfor %}
{% endif %}
{% if site.repo_trophies.enabled %} {% for user in site.data.repositories.github_users %} {% if site.data.repositories.github_users.size > 1 %}
{{ user }}
{% endif %}
{% include repository/repo_trophies.liquid username=user %}
{% endfor %} {% endif %}
{% if site.data.repositories.github_repos %}
{% for repo in site.data.repositories.github_repos %} {% include repository/repo.liquid repository=repo %} {% endfor %}
{% endif %}
```
---
#### Theming
A variety of beautiful theme colors have been selected for you to choose from. The default is purple, but you can quickly change it by editing the `--global-theme-color` variable in the `_sass/_themes.scss` file. Other color variables are listed there as well. The stock theme color options available can be found at [\_sass/\_variables.scss](_sass/_variables.scss). You can also add your own colors to this file assigning each a name for ease of use across the template.
---
#### Social media previews
**al-folio** supports preview images on social media. To enable this functionality you will need to set `serve_og_meta` to `true` in your [\_config.yml](_config.yml). Once you have done so, all your site's pages will include Open Graph data in the HTML head element.
You will then need to configure what image to display in your site's social media previews. This can be configured on a per-page basis, by setting the `og_image` page variable. If for an individual page this variable is not set, then the theme will fall back to a site-wide `og_image` variable, configurable in your [\_config.yml](_config.yml). In both the page-specific and site-wide cases, the `og_image` variable needs to hold the URL for the image you wish to display in social media previews.
---
#### Atom (RSS-like) Feed
It generates an Atom (RSS-like) feed of your posts, useful for Atom and RSS readers. The feed is reachable simply by typing after your homepage `/feed.xml`. E.g. assuming your website mountpoint is the main folder, you can type `yourusername.github.io/feed.xml`
---
#### Related posts
By default, there will be a related posts section on the bottom of the blog posts. These are generated by selecting the `max_related` most recent posts that share at least `min_common_tags` tags with the current post. If you do not want to display related posts on a specific post, simply add `related_posts: false` to the front matter of the post. If you want to disable it for all posts, simply set `enabled` to false in the `related_blog_posts` section in [\_config.yml](_config.yml).
---
#### Code quality checks
Currently, we run some checks to ensure that the code quality and generated site are good. The checks are done using GitHub Actions and the following tools:
- [Prettier](https://prettier.io/) - check if the formatting of the code follows the style guide
- [lychee](https://lychee.cli.rs/) - check for broken links
- [Axe](https://github.com/dequelabs/axe-core) (need to run manually) - do some accessibility testing
We decided to keep `Axe` runs manual because fixing the issues are not straightforward and might be hard for people without web development knowledge.
## FAQ
For frequently asked questions, please refer to [FAQ.md](FAQ.md).
## Contributing
Contributions to al-folio are very welcome! Before you get started, please take a look at [the guidelines](CONTRIBUTING.md).
If you would like to improve documentation or fix a minor inconsistency or bug, please feel free to send a PR directly to `main`. For more complex issues/bugs or feature requests, please open an issue using the appropriate template.
### Maintainers
Our most active contributors are welcome to join the maintainers team. If you are interested, please reach out!
| [**Maruan**](http://maruan.alshedivat.com) | [**Rohan Deb Sarkar**](http://rohandebsarkar.github.io) | [**Amir Pourmand**](https://amirpourmand.ir) | [**George**](https://george-gca.github.io/) |
| --- | --- | --- | --- |
### All Contributors
## Star History
## License
The theme is available as open source under the terms of the [MIT License](https://github.com/alshedivat/al-folio/blob/main/LICENSE).
Originally, **al-folio** was based on the [\*folio theme](https://github.com/bogoli/-folio) (published by [Lia Bogoev](https://liabogoev.com) and under the MIT license). Since then, it got a full re-write of the styles and many additional cool features.