## 1. Project Overview & Quickstart (projectdiscovery/katana)

## File: README.md

<h1 align="center">
  
  
</h1>

<h4 align="center">A next-generation crawling and spidering framework</h4>






# Features



 - Fast And fully configurable web crawling
 - **Standard** and **Headless** mode
 - **JavaScript** parsing / crawling
 - Customizable **automatic form filling**
 - **Scope control** - Preconfigured field / Regex 
 - **Knowledge base** - ML page-type / form classification (auto-downloaded model)
 - **Customizable output** - Preconfigured fields
 - INPUT - **STDIN**, **URL** and **LIST**
 - OUTPUT - **STDOUT**, **FILE** and **JSON**


## Installation

katana requires Go 1.26+ to install successfully. If you encounter any installation issues, we recommend trying with the latest available version of Go, as the minimum required version may have changed. Run the command below or download a pre-compiled binary from the [release page](https://github.com/projectdiscovery/katana/releases).

```console
CGO_ENABLED=1 go install github.com/projectdiscovery/katana/cmd/katana@latest
```

**More options to install / run katana-**


  Docker

> To install / update docker to latest tag -

```sh
docker pull projectdiscovery/katana:latest
```

> To run katana in standard mode using docker -


```sh
docker run projectdiscovery/katana:latest -u https://tesla.com
```

> To run katana in headless mode using docker -

```sh
docker run projectdiscovery/katana:latest -u https://tesla.com -system-chrome -headless
```




  Ubuntu

> It's recommended to install the following prerequisites -

```sh
sudo apt update
sudo apt install zip curl wget git snapd
sudo snap refresh
sudo snap install golang --classic

sudo install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://dl.google.com/linux/linux_signing_key.pub \
  | sudo gpg --dearmor -o /etc/apt/keyrings/google-chrome.gpg

echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/google-chrome.gpg] \
  http://dl.google.com/linux/chrome/deb/ stable main" \
  | sudo tee /etc/apt/sources.list.d/google-chrome.list > /dev/null

sudo apt update
sudo apt install google-chrome-stable
```

> install katana -


```sh
go install github.com/projectdiscovery/katana/cmd/katana@latest
```



## Usage

```console
katana -h
```

This will display help for the tool. Here are all the switches it supports.

```console
Katana is a fast crawler focused on execution in automation
pipelines offering both headless and non-headless crawling.

Usage:
  ./katana [flags]

Flags:
INPUT:
   -u, -list string[]     target url / list to crawl
   -resume string         resume scan using resume.cfg
   -e, -exclude string[]  exclude host matching specified filter ('cdn', 'private-ips', cidr, ip, regex)

CONFIGURATION:
   -r, -resolvers string[]       list of custom resolver (file or comma separated)
   -d, -depth int                maximum depth to crawl (default 3)
   -jc, -js-crawl                enable endpoint parsing / crawling in javascript file
   -jsl, -jsluice                enable jsluice parsing in javascript file (memory intensive)
   -ct, -crawl-duration value    maximum duration to crawl the target for (s, m, h, d) (default s)
   -kf, -known-files string      enable crawling of known files (all,robotstxt,sitemapxml), a minimum depth of 3 is required to ensure all known files are properly crawled.
   -mrs, -max-response-size int  maximum response size to read (default 4194304)
   -timeout int                  time to wait for request in seconds (default 10)
   -aff, -automatic-form-fill    enable automatic form filling (experimental)
   -fx, -form-extraction         extract form, input, textarea & select elements in jsonl output
   -retry int                    number of times to retry the request (default 1)
   -proxy string                 http/socks5 proxy to use
   -td, -tech-detect             enable technology detection
   -H, -headers string[]         custom header/cookie to include in all http request in header:value format (file)
   -config string                path to the katana configuration file
   -fc, -form-config string      path to custom form configuration file
   -flc, -field-config string    path to custom field configuration file
   -s, -strategy string          Visit strategy (depth-first, breadth-first) (default "depth-first")
   -iqp, -ignore-query-params    Ignore crawling same path with different query-param values
   -fsu, -filter-similar         filter crawling of similar looking URLs (e.g., /users/123 and /users/456)
   -fst, -filter-similar-threshold int  number of distinct values before a path position is treated as parameter (default 10)
   -tlsi, -tls-impersonate       enable experimental client hello (ja3) tls randomization
   -dr, -disable-redirects       disable following redirects (default false)
   -pcs, -page-content-similar   enable page content similarity filtering (simhash|tfidf|bm25)
   -pcsm, -page-content-similar-mode string  similarity mode: simhash, tfidf, or bm25 (default simhash)
   -pcsd, -page-content-similar-distance int  simhash max hamming distance (default 3)
   -pcst, -page-content-similar-threshold float  tfidf/bm25 min score 0-1 (default 0.85)
   -pcsn, -page-content-similar-budget int  pages to fully process per similarity cluster (default 1)
   -sdd, -similarity-deduplication  alias for -pcs
   -kb, -knowledge-base          enable knowledge base classification
   -kb-secrets                   enable secrets extractor in the knowledge base
   -kb-validate-secrets          validate detected secrets against their provider (sends live API calls)
   -kb-endpoints                 enable endpoints extractor (classifies REST/GraphQL/SOAP/XHR requests)
   -mdp, -max-domain-pages int   maximum number of pages to crawl per domain (default unlimited)

DEBUG:
   -health-check, -hc        run diagnostic check up
   -elog, -error-log string  file to write sent requests error log
   -pprof-server             enable pprof server

HEADLESS:
   -hl, -headless                    enable headless hybrid crawling (experimental)
   -sc, -system-chrome               use local installed chrome browser instead of katana installed
   -sb, -show-browser                show the browser on the screen with headless mode
   -ho, -headless-options string[]   start headless chrome with additional options
   -nos, -no-sandbox                 start headless chrome in --no-sandbox mode
   -cdd, -chrome-data-dir string     path to store chrome browser data
   -scp, -system-chrome-path string  use specified chrome browser for headless crawling
   -noi, -no-incognito               start headless chrome without incognito mode
   -cwu, -chrome-ws-url string       use chrome browser instance launched elsewhere with the debugger listening at this URL
   -xhr, -xhr-extraction             extract xhr request url,method in jsonl output
   -pls, -page-load-strategy string  page load strategy (heuristic, load, domcontentloaded, networkidle, none) (default "heuristic")
   -dwt, -dom-wait-time int          time in seconds to wait after page load when using domcontentloaded strategy (default 5)
   -csp, -captcha-solver-provider string  captcha solver provider (e.g. capsolver)
   -csk, -captcha-solver-key string       captcha solver provider api key

SCOPE:
   -cs, -crawl-scope string[]       in scope url regex to be followed by crawler
   -cos, -crawl-out-scope string[]  out of scope url regex to be excluded by crawler
   -fs, -field-scope string         pre-defined scope field (dn,rdn,fqdn) or custom regex (e.g., '(company-staging.io|company.com)') (default "rdn")
   -ns, -no-scope                   disables host based default scope
   -do, -display-out-scope          display external endpoint from scoped crawling

FILTER:
   -mr, -match-regex string[]             regex or list of regex to match on output url (cli, file)
   -fr, -filter-regex string[]            regex or list of regex to filter on output url (cli, file)
   -f, -field string                      field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir) (Deprecated: use -output-template instead)
   -sf, -store-field string               field to store in per-host output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir)
   -em, -extension-match string[]         match output for given extension (eg, -em php,html,js,none)
   -ef, -extension-filter string[]        filter output for given extension (eg, -ef png,css)
   -ndef, -no-default-ext-filter bool     remove default extensions from the filter list
   -mdc, -match-condition string          match response with dsl based condition
   -fdc, -filter-condition string         filter response with dsl based condition
   -duf, -disable-unique-filter           disable duplicate content filtering
   -filter-page-type string[]      filter response with page type (e.g. error,captcha,parked)

RATE-LIMIT:
   -c, -concurrency int          number of concurrent fetchers to use (default 10)
   -p, -parallelism int          number of concurrent inputs to process (default 10)
   -rd, -delay int               request delay between each request in seconds
   -rl, -rate-limit int          maximum requests to send per second (default 150)
   -rlm, -rate-limit-minute int  maximum number of requests to send per minute
   -hrl, -host-rate-limit int    maximum requests to send per second per host
   -hrlm, -host-rate-limit-minute int  maximum number of requests to send per minute per host

UPDATE:
   -up, -update                 update katana to latest version
   -duc, -disable-update-check  disable automatic katana update check

OUTPUT:
   -o, -output string                file to write output to
   -output-template string      custom output template
   -sr, -store-response              store http requests/responses
   -srd, -store-response-dir string  store http requests/responses to custom directory
   -ncb, -no-clobber                 do not overwrite output file
   -sfd, -store-field-dir string     store per-host field to custom directory
   -or, -omit-raw                    omit raw requests/responses from jsonl output
   -ob, -omit-body                   omit response body from jsonl output
   -lof, -list-output-fields         list available fields for jsonl output format
   -eof, -exclude-output-fields      exclude fields from jsonl output
   -j, -jsonl                        write output in jsonl format
   -nc, -no-color                    disable output content coloring (ANSI escape codes)
   -silent                           display output only
   -v, -verbose                      display verbose output
   -debug                            display debug output
   -version                          display project version
```

## Running Katana

### Input for katana

**katana** requires **url** or **endpoint** to crawl and accepts single or multiple inputs.

Input URL can be provided using `-u` option, and multiple values can be provided using comma-separated input, similarly **file** input is supported using `-list` option and additionally piped input (stdin) is also supported.

#### URL Input

```sh
katana -u https://tesla.com
```

#### Multiple URL Input (comma-separated)

```sh
katana -u https://tesla.com,https://google.com
```

#### List Input
```bash
$ cat url_list.txt

https://tesla.com
https://google.com
```

```
katana -list url_list.txt
```

#### STDIN (piped) Input

```sh
echo https://tesla.com | katana
```

```sh
cat domains | httpx | katana
```

### Page Content Similarity

Optional Layer-2 filtering after exact MD5 content dedup. Shared HTML normalization strips chrome (`nav`/`header`/`footer`/scripts), then one of three modes decides whether a page is similar enough to skip further parse/enqueue (cluster budget). URL-shape dedup (`-fsu`) remains independent.

| Mode | Flag | Best for |
|------|------|----------|
| `simhash` (default) | `-pcsm simhash` | Near-duplicate / template clones (Hamming via `-pcsd`) |
| `tfidf` | `-pcsm tfidf` | Topical cosine similarity (`-pcst`) |
| `bm25` | `-pcsm bm25` | Topical similarity with length normalization (`-pcst`) |

```sh
# Near-duplicate detection (default mode)
katana -u https://example.com -pcs

# TF-IDF topical filtering with higher budget
katana -u https://shop.example.com -pcs -pcsm tfidf -pcst 0.85 -pcsn 2

# BM25 mode
katana -u https://example.com -pcs -pcsm bm25

# Alias from older flag name
katana -u https://example.com -sdd -pcsm simhash
```

Example completion stats:
```console
[INF] Content similarity (simhash): 103 processed, 30 accepted, 73 filtered - 70.9% filter rate
[INF] Crawl completed in 14s. 21 endpoints found.
```

Example running katana -

```console
katana -u https://youtube.com

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.1                     

      projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions.
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
https://www.youtube.com/
https://www.youtube.com/about/
https://www.youtube.com/about/press/
https://www.youtube.com/about/copyright/
https://www.youtube.com/t/contact_us/
https://www.youtube.com/creators/
https://www.youtube.com/ads/
https://www.youtube.com/t/terms
https://www.youtube.com/t/privacy
https://www.youtube.com/about/policies/
https://www.youtube.com/howyoutubeworks?utm_campaign=ytgen&utm_source=ythp&utm_medium=LeftNav&utm_content=txt&u=https%3A%2F%2Fwww.youtube.com%2Fhowyoutubeworks%3Futm_source%3Dythp%26utm_medium%3DLeftNav%26utm_campaign%3Dytgen
https://www.youtube.com/new
https://m.youtube.com/
https://www.youtube.com/s/desktop/4965577f/jsbin/desktop_polymer.vflset/desktop_polymer.js
https://www.youtube.com/s/desktop/4965577f/cssbin/www-main-desktop-home-page-skeleton.css
https://www.youtube.com/s/desktop/4965577f/cssbin/www-onepick.css
https://www.youtube.com/s/_/ytmainappweb/_/ss/k=ytmainappweb.kevlar_base.0Zo5FUcPkCg.L.B1.O/am=gAE/d=0/rs=AGKMywG5nh5Qp-BGPbOaI1evhF5BVGRZGA
https://www.youtube.com/opensearch?locale=en_GB
https://www.youtube.com/manifest.webmanifest
https://www.youtube.com/s/desktop/4965577f/cssbin/www-main-desktop-watch-page-skeleton.css
https://www.youtube.com/s/desktop/4965577f/jsbin/web-animations-next-lite.min.vflset/web-animations-next-lite.min.js
https://www.youtube.com/s/desktop/4965577f/jsbin/custom-elements-es5-adapter.vflset/custom-elements-es5-adapter.js
https://www.youtube.com/s/desktop/4965577f/jsbin/webcomponents-sd.vflset/webcomponents-sd.js
https://www.youtube.com/s/desktop/4965577f/jsbin/intersection-observer.min.vflset/intersection-observer.min.js
https://www.youtube.com/s/desktop/4965577f/jsbin/scheduler.vflset/scheduler.js
https://www.youtube.com/s/desktop/4965577f/jsbin/www-i18n-constants-en_GB.vflset/www-i18n-constants.js
https://www.youtube.com/s/desktop/4965577f/jsbin/www-tampering.vflset/www-tampering.js
https://www.youtube.com/s/desktop/4965577f/jsbin/spf.vflset/spf.js
https://www.youtube.com/s/desktop/4965577f/jsbin/network.vflset/network.js
https://www.youtube.com/howyoutubeworks/
https://www.youtube.com/trends/
https://www.youtube.com/jobs/
https://www.youtube.com/kids/
```


## Crawling Mode

### Standard Mode

Standard crawling modality uses the standard go http library under the hood to handle HTTP requests/responses. This modality is much faster as it doesn't have the browser overhead. Still, it analyzes HTTP responses body as is, without any javascript or DOM rendering, potentially missing post-dom-rendered endpoints or asynchronous endpoint calls that might happen in complex web applications depending, for example, on browser-specific events.

### Headless Mode

Headless mode hooks internal headless calls to handle HTTP requests/responses directly within the browser context. This offers two advantages:
- The HTTP fingerprint (TLS and user agent) fully identify the client as a legitimate browser
- Better coverage since the endpoints are discovered analyzing the standard raw response, as in the previous modality, and also the browser-rendered one with javascript enabled.

Headless crawling is optional and can be enabled using `-headless` option.

Here are other headless CLI options -

```console
katana -h headless

Flags:
HEADLESS:
   -hl, -headless                    enable headless hybrid crawling (experimental)
   -sc, -system-chrome               use local installed chrome browser instead of katana installed
   -sb, -show-browser                show the browser on the screen with headless mode
   -ho, -headless-options string[]   start headless chrome with additional options
   -nos, -no-sandbox                 start headless chrome in --no-sandbox mode
   -cdd, -chrome-data-dir string     path to store chrome browser data
   -scp, -system-chrome-path string  use specified chrome browser for headless crawling
   -noi, -no-incognito               start headless chrome without incognito mode
   -cwu, -chrome-ws-url string       use chrome browser instance launched elsewhere with the debugger listening at this URL
   -xhr, -xhr-extraction             extract xhr requests
   -pls, -page-load-strategy string  page load strategy (heuristic, load, domcontentloaded, networkidle, none) (default "heuristic")
   -dwt, -dom-wait-time int          time in seconds to wait after page load when using domcontentloaded strategy (default 5)
   -csp, -captcha-solver-provider string  captcha solver provider (e.g. capsolver)
   -csk, -captcha-solver-key string       captcha solver provider api key
   -al, -auto-login string           automatic login with username:password (headless only)
   -rf, -recorded-flow string        chrome DevTools Recorder JSON (or explicit steps) to replay before crawl (headless only)
```

*`-no-sandbox`*
----

Runs headless chrome browser with **no-sandbox** option, useful when running as root user.

```console
katana -u https://tesla.com -headless -no-sandbox
```

*`-no-incognito`*
----

Runs headless chrome browser without incognito mode, useful when using the local browser.

```console
katana -u https://tesla.com -headless -no-incognito
```

To preserve cookies and other browser session data across runs, combine `-no-incognito` with `-chrome-data-dir` so Katana reuses your chosen Chrome profile directory instead of an isolated temporary one.

```console
katana -u https://tesla.com -headless -no-incognito -chrome-data-dir /tmp/katana-profile
```

*`-headless-options`*
----

When crawling in headless mode, additional chrome options can be specified using `-headless-options`, for example -


```console
katana -u https://tesla.com -headless -system-chrome -headless-options --disable-gpu,proxy-server=http://127.0.0.1:8080
```

*`-page-load-strategy`*
----

Controls how katana waits for pages to load in headless mode. Different strategies are useful for different types of web applications:

| Strategy | Description |
|----------|-------------|
| `heuristic` | (default) Smart waiting that adapts to page behavior - waits for load event, network idle, and DOM stability |
| `load` | Waits only for the browser's load event |
| `domcontentloaded` | Waits for DOMContentLoaded event plus additional time (configurable via `-dwt`) for JavaScript rendering |
| `networkidle` | Waits for network activity to stop |
| `none` | No waiting - returns immediately after navigation starts |

```console
katana -u https://tesla.com -headless -pls domcontentloaded
```

The `domcontentloaded` strategy is particularly useful for Single Page Applications (SPAs) that never fully complete loading due to continuous background requests (websockets, polling, etc.).

*`-dom-wait-time`*
----

When using the `domcontentloaded` page load strategy, this option specifies how many seconds to wait after the DOMContentLoaded event fires. This allows time for JavaScript to render interactive elements. Default is 5 seconds.

```console
katana -u https://tesla.com -headless -pls domcontentloaded -dwt 10
```


### Captcha Solving

Katana supports automatic captcha detection and solving during headless crawling. When a captcha page is encountered, katana identifies the captcha provider, solves it via an external service, and continues crawling.

Supported captcha types: **reCAPTCHA v2**, **reCAPTCHA v3**, **reCAPTCHA Enterprise**, **Cloudflare Turnstile**, **hCaptcha**

*`-captcha-solver-provider`*
----

Option to specify the captcha solver provider. Currently supported: `capsolver`.

*`-captcha-solver-key`*
----

API key for the captcha solver provider.

```console
katana -u https://example.com -headless -csp capsolver -csk YOUR_API_KEY
```

The provider and key can also be set via environment variables:

```console
export CAPTCHA_SOLVER_PROVIDER=capsolver
export CAPTCHA_SOLVER_KEY=YOUR_API_KEY
katana -u https://example.com -headless
```

## Scope Control

Crawling can be endless if not scoped, as such katana comes with multiple support to define the crawl scope.

*`-field-scope`*
----
Most handy option to define scope with predefined field name, `rdn` being default option for field scope.

   - `rdn` - crawling scoped to root domain name and all subdomains (e.g. `*example.com`) (default)
   - `fqdn` - crawling scoped to given sub(domain) (e.g. `www.example.com` or `api.example.com`)
   - `dn` - crawling scoped to domain name keyword (e.g. `example`)

```console
katana -u https://tesla.com -fs dn
```


*`-crawl-scope`*
------

For advanced scope control, `-cs` option can be used that comes with **regex** support.

```console
katana -u https://tesla.com -cs login
```

For multiple in scope rules, file input with multiline string / regex can be passed.

```bash
$ cat in_scope.txt

login/
admin/
app/
wordpress/
```

```console
katana -u https://tesla.com -cs in_scope.txt
```


*`-crawl-out-scope`*
-----

For defining what not to crawl, `-cos` option can be used and also support **regex** input.

```console
katana -u https://tesla.com -cos logout
```

For multiple out of scope rules, file input with multiline string / regex can be passed.

```bash
$ cat out_of_scope.txt

/logout
/log_out
```

```console
katana -u https://tesla.com -cos out_of_scope.txt
```

*`-no-scope`*
----

Katana is default to scope `*.domain`, to disable this `-ns` option can be used and also to crawl the internet.

```console
katana -u https://tesla.com -ns
```

*`-display-out-scope`*
----

As default, when scope option is used, it also applies for the links to display as output, as such **external URLs are default to exclude** and to overwrite this behavior, `-do` option can be used to display all the external URLs that exist in targets scoped URL / Endpoint.

```
katana -u https://tesla.com -do
```

Here is all the CLI options for the scope control -


```console
katana -h scope

Flags:
SCOPE:
   -cs, -crawl-scope string[]       in scope url regex to be followed by crawler
   -cos, -crawl-out-scope string[]  out of scope url regex to be excluded by crawler
   -fs, -field-scope string         pre-defined scope field (dn,rdn,fqdn) (default "rdn")
   -ns, -no-scope                   disables host based default scope
   -do, -display-out-scope          display external endpoint from scoped crawling
```


## Crawler Configuration

Katana comes with multiple options to configure and control the crawl as the way we want.

*`-depth`*
----

Option to define the `depth` to follow the urls for crawling, the more depth the more number of endpoint being crawled + time for crawl.

```
katana -u https://tesla.com -d 5
```

*`-js-crawl`*
----

Option to enable JavaScript file parsing + crawling the endpoints discovered in JavaScript files, disabled as default.

```
katana -u https://tesla.com -jc
```

*`-crawl-duration`*
----

Option to predefined crawl duration, disabled as default.

```
katana -u https://tesla.com -ct 2
```

*`-known-files`*
----
Option to enable crawling `robots.txt` and `sitemap.xml` file, disabled as default.

```
katana -u https://tesla.com -kf robotstxt,sitemapxml
```

*`-automatic-form-fill`*
----

Option to enable automatic form filling for known / unknown fields, known field values can be customized as needed by updating form config file at `$HOME/.config/katana/form-config.yaml`.

Automatic form filling is experimental feature.

```
katana -u https://tesla.com -aff
```

Form config values support DSL helper functions for dynamic data generation. All `rand_*` functions from the [projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) library are available:

```yaml
# $HOME/.config/katana/form-config.yaml
email: "rand_email()"
phone: "rand_phone()"
placeholder: "rand_first_name()"
password: 'rand_base(16, "")'
color: "#e66465"
```

*`-filter-similar`*
----

Option to filter crawling of similar looking URLs by normalizing variable path segments. This detects IDs, UUIDs, hashes, dates, and other dynamic values, and also learns repeating patterns at runtime. For example, `/users/123` and `/users/456` are treated as the same endpoint.

```
katana -u https://tesla.com -fsu
```

The promotion threshold (how many distinct values at a path position before it's treated as a parameter) can be tuned with `-fst`. Lower values are more aggressive (fewer URLs crawled), higher values are more permissive. Default is `10`.

```
katana -u https://tesla.com -fsu -fst 5
```

*`-max-domain-pages`*
----

Option to limit the number of pages crawled per domain. Prevents any single domain from consuming the entire crawl budget, useful for large sites or crawler trap protection.

```
katana -u https://tesla.com -mdp 100
```

## Knowledge Base Classification

Katana can enrich crawl results with a **knowledge base** — machine-learning classification of each crawled page powered by [dit](https://github.com/HappyHackingSpace/dit). When enabled, every response is classified by **page type** (e.g. `login`, `error`, `captcha`, `parked`) and any forms on the page are identified, with the result attached to the `knowledgebase` field of the JSONL output. This works across **all engines** (standard and headless).

> **Note**: The classification model is **downloaded automatically** on first use to `~/.dit/model.json` (from [Hugging Face](https://huggingface.co/datasets/happyhackingspace/dit)). This is a one-time, per-machine cost — subsequent runs reuse the cached model. No manual installation of `dit` is required.

*`-knowledge-base`*
----

Enable knowledge base classification. Page-type and form classification is added to the `knowledgebase` field of each result.

```console
katana -u https://example.com -kb -jsonl
```

```json
{
  "timestamp": "...",
  "request": { "...": "..." },
  "response": {
    "...": "...",
    "knowledgebase": {
      "PageType": "login",
      "Forms": [{ "type": "login", "fields": { "username": "username or email", "password": "password" } }]
    }
  }
}
```

*`-filter-page-type`*
----

Filter results to only the given page type(s). Enabling this implies `-kb` (the classifier is initialized automatically).

```console
katana -u https://example.com -fpt login,error
```

*`-kb-secrets`*
----

Enable the secrets extractor in the knowledge base, surfacing detected secrets (API keys, tokens, etc.) under the `secrets` key. Add `-kb-validate-secrets` to validate detected secrets against their provider — note this **sends live API calls**.

```console
katana -u https://example.com -kb-secrets
```

*`-kb-endpoints`*
----

Enable the endpoints extractor, which classifies requests as REST, GraphQL, SOAP, or XHR under the `endpoints` key.

```console
katana -u https://example.com -kb-endpoints
```

## Authenticated Crawling

Katana supports authenticated crawling in several ways: static headers/cookies, attaching to an already-logged-in Chrome session, automatic form login (`-auto-login`), and **recorded browser flows** (`-recorded-flow`) for multi-step / SPA / SSO-style logins.

### Headers / cookies

Authenticated crawling can include custom headers or cookies in HTTP requests to access protected resources. You can specify headers directly on the command line or provide them as a file.

> **Note**: For the header/cookie approach, perform authentication manually once and export the session cookie / header to use with katana.

*`-headers`*
----

Option to add a custom header or cookie to the request. 
> Syntax of [headers](https://datatracker.ietf.org/doc/html/rfc7230#section-3.2) in the HTTP specification

Here is an example of adding a cookie to the request:
```
katana -u https://tesla.com -H 'Cookie: usrsess=AmljNrESo'
```

It is also possible to supply headers or cookies as a file. For example:

```
$ cat cookie.txt

Cookie: PHPSESSIONID=XXXXXXXXX
X-API-KEY: XXXXX
TOKEN=XX
```

```
katana -u https://tesla.com -H cookie.txt
```

### Automatic login (`-auto-login`)

In pure headless mode, katana can detect a login form and submit credentials automatically:

```console
katana -hl -u https://app.example.com -al 'user:password'
```

This is best for simple username/password forms. For multi-step, SPA, or SSO logins prefer a recorded flow.

### Recorded flow (`-recorded-flow`)

Recorded flows let you export a real browser login from **Chrome DevTools → Recorder** and replay it once **before the crawl starts**. Session cookies and same-origin web storage stay in the headless browser context for the rest of the crawl.

**1. Record the login in Chrome**

1. Open Chrome DevTools → **Recorder**
2. Start a new recording and perform the login (navigate, type, click Next, type password, submit, …)
3. Export the recording as JSON

**2. Replay it with katana**

```console
katana -hl -u https://app.example.com/dashboard \
  -rf login.json \
  -al 'user:password'
```

Credential literals in the recording are replaced with `{{username}}` / `{{password}}` placeholders and filled from `-al`. If the recording (or an explicit step list) still needs those placeholders, `-al` is required.

Katana also accepts a hand-authored explicit step file:

```json
{
  "steps": [
    {"action": "navigate", "value": "https://app.example.com/login"},
    {"action": "fill", "selector": "#email", "value": "{{username}}"},
    {"action": "fill", "selector": "#password", "value": "{{password}}"},
    {"action": "click", "selector": "#submit"}
  ]
}
```

Supported step actions: `navigate`, `fill`, `click`, `doubleclick`, `waitvisible`, `wait`, `press`, `submit`.

**How it runs**

1. Katana launches headless Chrome
2. Captures the anonymous browser state, then replays the recorded flow once
3. Verifies that a cookie or same-origin web-storage value changed; if not, tries `-auto-login` as a fallback
4. Starts crawling `-u` with the established session
5. Skips logout URLs while authenticated

For applications that keep authentication entirely server-side without changing
cookies or web storage, end an explicit flow with `waitvisible` targeting an
element that only exists after login. A successful terminal visibility check is
treated as the flow's explicit authentication assertion.

> Recorded flows require **pure headless** (`-hl`). Hybrid (`-hh`) is disabled automatically when `-rf` is set.

There are more options to configure when needed, here is all the config related CLI options - 

```console
katana -h config

Flags:
CONFIGURATION:
   -r, -resolvers string[]       list of custom resolver (file or comma separated)
   -d, -depth int                maximum depth to crawl (default 3)
   -jc, -js-crawl                enable endpoint parsing / crawling in javascript file
   -ct, -crawl-duration int      maximum duration to crawl the target for
   -kf, -known-files string      enable crawling of known files (all,robotstxt,sitemapxml)
   -mrs, -max-response-size int  maximum response size to read (default 9223372036854775807)
   -timeout int                  time to wait for request in seconds (default 10)
   -aff, -automatic-form-fill    enable automatic form filling (experimental)
   -fx, -form-extraction         enable extraction of form, input, textarea & select elements
   -retry int                    number of times to retry the request (default 1)
   -proxy string                 http/socks5 proxy to use
   -H, -headers string[]         custom header/cookie to include in request
   -config string                path to the katana configuration file
   -fc, -form-config string      path to custom form configuration file
   -flc, -field-config string    path to custom field configuration file
   -s, -strategy string          Visit strategy (depth-first, breadth-first) (default "depth-first")
   -iqp, -ignore-query-params    Ignore crawling same path with different query-param values
   -fsu, -filter-similar         filter crawling of similar looking URLs (e.g., /users/123 and /users/456)
   -fst, -filter-similar-threshold int  number of distinct values before a path position is treated as parameter (default 10)
   -mdp, -max-domain-pages int   maximum number of pages to crawl per domain (default unlimited)
```

### Connecting to Active Browser Session

Katana can also connect to active browser session where user is already logged in and authenticated. and use it for crawling. The only requirement for this is to start browser with remote debugging enabled.

Here is an example of starting chrome browser with remote debugging enabled and using it with katana -

**step 1) First Locate path of chrome executable**

| Operating System | Chromium Executable Location | Google Chrome Executable Location |
|------------------|------------------------------|-----------------------------------|
| Windows (64-bit) | `C:\Program Files (x86)\Google\Chromium\Application\chrome.exe` | `C:\Program Files (x86)\Google\Chrome\Application\chrome.exe` |
| Windows (32-bit) | `C:\Program Files\Google\Chromium\Application\chrome.exe` | `C:\Program Files\Google\Chrome\Application\chrome.exe` |
| macOS | `/Applications/Chromium.app/Contents/MacOS/Chromium` | `/Applications/Google Chrome.app/Contents/MacOS/Google Chrome` |
| Linux | `/usr/bin/chromium` | `/usr/bin/google-chrome` |

**step 2) Start chrome with remote debugging enabled and it will return websocker url. For example, on MacOS, you can start chrome with remote debugging enabled using following command** -

```console
$ /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222


DevTools listening on ws://127.0.0.1:9222/devtools/browser/c5316c9c-19d6-42dc-847a-41d1aeebf7d6
```

> Now login to the website you want to crawl and keep the browser open.

**step 3) Now use the websocket url with katana to connect to the active browser session and crawl the website**

```console
katana -headless -u https://tesla.com -cwu ws://127.0.0.1:9222/devtools/browser/c5316c9c-19d6-42dc-847a-41d1aeebf7d6 -no-incognito
```

> **Note**: you can use `-cdd` option to specify custom chrome data directory to store browser data and cookies but that does not save session data if cookie is set to `Session` only or expires after certain time.


## Filters

*`-field`*
----

> [!WARNING]
> Deprecated: use [**`-output-template`**](#-output-template) instead. The field flag is still supported for backward compatibility.

Katana comes with built in fields that can be used to filter the output for the desired information, `-f` option can be used to specify any of the available fields.

```
   -f, -field string  field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,key,value,kv,dir,udir)
```

Here is a table with examples of each field and expected output when used - 


| FIELD   | DESCRIPTION                 | EXAMPLE                                                      |
| ------- | --------------------------- | ------------------------------------------------------------ |
| `url`   | URL Endpoint                | `https://admin.projectdiscovery.io/admin/login?user=admin&password=admin` |
| `qurl`  | URL including query param   | `https://admin.projectdiscovery.io/admin/login.php?user=admin&password=admin` |
| `qpath` | Path including query param  | `/login?user=admin&password=admin`                           |
| `path`  | URL Path                    | `https://admin.projectdiscovery.io/admin/login`              |
| `fqdn`  | Fully Qualified Domain name | `admin.projectdiscovery.io`                                  |
| `rdn`   | Root Domain name            | `projectdiscovery.io`                                        |
| `rurl`  | Root URL                    | `https://admin.projectdiscovery.io`                          |
| `ufile` | URL with File               | `https://admin.projectdiscovery.io/login.js`                 |
| `file`  | Filename in URL             | `login.php`                                                  |
| `key`   | Parameter keys in URL       | `user,password`                                              |
| `value` | Parameter values in URL     | `admin,admin`                                                |
| `kv`    | Keys=Values in URL          | `user=admin&password=admin`                                  |
| `dir`   | URL Directory name          | `/admin/`                                                    |
| `udir`  | URL with Directory          | `https://admin.projectdiscovery.io/admin/`                   |

Here is an example of using field option to only display all the urls with query parameter in it -

```
katana -u https://tesla.com -f qurl -silent

https://shop.tesla.com/en_au?redirect=no
https://shop.tesla.com/en_nz?redirect=no
https://shop.tesla.com/product/men_s-raven-lightweight-zip-up-bomber-jacket?sku=1740250-00-A
https://shop.tesla.com/product/tesla-shop-gift-card?sku=1767247-00-A
https://shop.tesla.com/product/men_s-chill-crew-neck-sweatshirt?sku=1740176-00-A
https://www.tesla.com/about?redirect=no
https://www.tesla.com/about/legal?redirect=no
https://www.tesla.com/findus/list?redirect=no
```

### Custom Fields

You can create custom fields to extract and store specific information from page responses using regex rules. These custom fields are defined using a YAML config file and are loaded from the default location at `$HOME/.config/katana/field-config.yaml`. Alternatively, you can use the `-flc` option to load a custom field config file from a different location.
Here is example custom field.

```yaml
- name: email
  type: regex
  regex:
  - '([a-zA-Z0-9._-]+@[a-zA-Z0-9._-]+\.[a-zA-Z0-9_-]+)'
  - '([a-zA-Z0-9+._-]+@[a-zA-Z0-9._-]+\.[a-zA-Z0-9_-]+)'

- name: phone
  type: regex
  regex:
  - '\d{3}-\d{8}|\d{4}-\d{7}'
```

When defining custom fields, following attributes are supported:

- **name** (required)

> The value of **name** attribute is used as the `-field` cli option value.

- **type** (required)

> The type of custom attribute, currently supported option - `regex` 

- **part** (optional)

> The part of the response to extract the information from. The default value is `response`, which includes both the header and body. Other possible values are `header` and `body`.

- group (optional)

> You can use this attribute to select a specific matched group in regex, for example: `group: 1`

#### Running katana using custom field:

```console
katana -u https://tesla.com -f email,phone
```

*`-store-field`*
---

To compliment `field` option which is useful to filter output at run time, there is `-sf, -store-fields` option which works exactly like field option except instead of filtering, it stores all the information on the disk under `katana_field` directory sorted by target url. Use `-sfd` or `-store-field-dir` to store data in a different location.

```
katana -u https://tesla.com -sf key,fqdn,qurl -silent
```

```bash
$ ls katana_field/

https_www.tesla.com_fqdn.txt
https_www.tesla.com_key.txt
https_www.tesla.com_qurl.txt
```

The `-store-field` option can be useful for collecting information to build a targeted wordlist for various purposes, including but not limited to:

- Identifying the most commonly used parameters
- Discovering frequently used paths
- Finding commonly used files
- Identifying related or unknown subdomains

### Katana Filters

*`-extension-match`*
---

Crawl output can be easily matched for specific extension using `-em` option to ensure to display only output containing given extension.

```
katana -u https://tesla.com -silent -em js,jsp,json
```

Use the special value `none` to also include URLs without a file extension in the output:

```
katana -u https://tesla.com -silent -em js,jsp,json,none
```

*`-extension-filter`*
---

Crawl output can be easily filtered for specific extension using `-ef` option which ensure to remove all the urls containing given extension.

```
katana -u https://tesla.com -silent -ef css,txt,md

```
*`-no-default-ext-filter`*
---

Katana filters several extensions by default. This can be disabled with the `-ndef` option.

```
katana -u https://tesla.com -silent -ndef
```

*`-match-regex`*
---
The `-match-regex` or `-mr` flag allows you to filter output URLs using regular expressions. When using this flag, only URLs that match the specified regular expression will be printed in the output.

```
katana -u https://tesla.com -mr 'https://shop\.tesla\.com/*' -silent
```
*`-filter-regex`*
---
The `-filter-regex` or `-fr` flag allows you to filter output URLs using regular expressions. When using this flag, it will skip the URLs that are match the specified regular expression.

```
katana -u https://tesla.com -fr 'https://www\.tesla\.com/*' -silent
```

### Advance Filtering

Katana supports DSL-based expressions for advanced matching and filtering capabilities:

- To match endpoints with a 200 status code:
```shell
katana -u https://www.hackerone.com -mdc 'status_code == 200'
```
- To match endpoints that contain "default" and have a status code other than 403:
```shell
katana -u https://www.hackerone.com -mdc 'contains(endpoint, "default") && status_code != 403'
```
- To match endpoints with PHP technologies:
```shell
katana -u https://www.hackerone.com -mdc 'contains(to_lower(technologies), "php")'
```
- To filter out endpoints running on Cloudflare:
```shell
katana -u https://www.hackerone.com -fdc 'contains(to_lower(technologies), "cloudflare")'
```
DSL functions can be applied to any keys in the jsonl output. For more information on available DSL functions, please visit the [dsl project](https://github.com/projectdiscovery/dsl).

Here are additional filter options -

```console
katana -h filter

Flags:
FILTER:
   -mr, -match-regex string[]             regex or list of regex to match on output url (cli, file)
   -fr, -filter-regex string[]            regex or list of regex to filter on output url (cli, file)
   -f, -field string                      field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir)
   -sf, -store-field string               field to store in per-host output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir)
   -em, -extension-match string[]         match output for given extension (eg, -em php,html,js,none)
   -ef, -extension-filter string[]        filter output for given extension (eg, -ef png,css)
   -ndef, -no-default-ext-filter bool     remove default extensions from the filter list
   -mdc, -match-condition string          match response with dsl based condition
   -fdc, -filter-condition string         filter response with dsl based condition
   -duf, -disable-unique-filter           disable duplicate content filtering
```


## Rate Limit

It's easy to get blocked / banned while crawling if not following target websites limits, katana comes with multiple option to tune the crawl to go as fast / slow we want.

*`-delay`*
-----

option to introduce a delay in seconds between each new request katana makes while crawling, disabled as default.

```
katana -u https://tesla.com -delay 20
```

*`-concurrency`*
-----
option to control the number of urls per target to fetch at the same time.

```
katana -u https://tesla.com -c 20
```


*`-parallelism`*
-----
option to define number of target to process at same time from list input.

```
katana -u https://tesla.com -p 20
```

*`-rate-limit`*
-----
Maximum requests per second, applied globally across all hosts.

```
katana -u https://tesla.com -rl 100
```

*`-rate-limit-minute`*
-----
Maximum requests per minute, applied globally across all hosts.

```
katana -u https://tesla.com -rlm 500
```

*`-host-rate-limit`*
-----
Maximum requests per second per host. Each host gets its own rate limit bucket, so a slow host won't throttle fast ones. Replaces the global rate limit when set. Katana also backs off automatically with exponential delay and jitter when a host returns 429 or 503.

```console
katana -u https://tesla.com -hrl 50
```

*`-host-rate-limit-minute`*
-----
Maximum requests per minute per host.

```console
katana -u https://tesla.com -hrlm 200
```

Here is all long / short CLI options for rate limit control -

```console
katana -h rate-limit

Flags:
RATE-LIMIT:
   -c, -concurrency int          number of concurrent fetchers to use (default 10)
   -p, -parallelism int          number of concurrent inputs to process (default 10)
   -rd, -delay int               request delay between each request in seconds
   -rl, -rate-limit int          maximum requests to send per second (default 150)
   -rlm, -rate-limit-minute int  maximum number of requests to send per minute
   -hrl, -host-rate-limit int    maximum requests to send per second per host
   -hrlm, -host-rate-limit-minute int  maximum number of requests to send per minute per host
```

## Output

Katana support both file output in plain text format as well as JSON which includes additional information like, `source`, `tag`, and `attribute` name to co-related the discovered endpoint.

*`-output`*
---

By default, katana outputs the crawled endpoints in plain text format. The results can be written to a file by using the -output option.


```console
katana -u https://example.com -no-scope -output example_endpoints.txt
```

*`-output-template`*
---

The `-output-template` option allows you to customize the output format using template, providing flexibility in defining the output structure. This option replaces the deprecated `-field` flag for filtering output. Instead of relying on predefined fields, you can specify a custom template directly in the command line to control how the extracted data is presented.

Example of using the `-output-template` option:

```sh
katana -u https://example.com -output-template '{{email}} - {{url}}'
```

In this example, `email` represents a [custom field](#custom-fields) that extracts and displays email addresses found within the source `url`.

> [!NOTE]
> If a specified field does not exist or does not contain a value, it will simply be omitted from the output.

This option can effectively structure the output in a way that best suits your use case, making data extraction more intuitive and customizable.

*`-jsonl`*
---

```console
katana -u https://example.com -jsonl | jq .
```

```json
{
  "timestamp": "2023-03-20T16:23:58.027559+05:30",
  "request": {
    "method": "GET",
    "endpoint": "https://example.com",
    "raw": "GET / HTTP/1.1\r\nHost: example.com\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36\r\nAccept-Encoding: gzip\r\n\r\n"
  },
  "response": {
    "status_code": 200,
    "headers": {
      "accept_ranges": "bytes",
      "expires": "Mon, 27 Mar 2023 10:53:58 GMT",
      "last_modified": "Thu, 17 Oct 2019 07:18:26 GMT",
      "content_type": "text/html; charset=UTF-8",
      "server": "ECS (dcb/7EA3)",
      "vary": "Accept-Encoding",
      "etag": "\"3147526947\"",
      "cache_control": "max-age=604800",
      "x_cache": "HIT",
      "date": "Mon, 20 Mar 2023 10:53:58 GMT",
      "age": "331239"
    },
    "body": "<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8\" />\n    <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n    <style type=\"text/css\">\n    body {\n        background-color: #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n        \n    }\n    div {\n        width: 600px;\n        margin: 5em auto;\n        padding: 2em;\n        background-color: #fdfdff;\n        border-radius: 0.5em;\n        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n    }\n    a:link, a:visited {\n        color: #38488f;\n        text-decoration: none;\n    }\n    @media (max-width: 700px) {\n        div {\n            margin: 0 auto;\n            width: auto;\n        }\n    }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission.</p>\n    <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n",
    "technologies": [
      "Azure",
      "Amazon ECS",
      "Amazon Web Services",
      "Docker",
      "Azure CDN"
    ],
    "raw": "HTTP/1.1 200 OK\r\nContent-Length: 1256\r\nAccept-Ranges: bytes\r\nAge: 331239\r\nCache-Control: max-age=604800\r\nContent-Type: text/html; charset=UTF-8\r\nDate: Mon, 20 Mar 2023 10:53:58 GMT\r\nEtag: \"3147526947\"\r\nExpires: Mon, 27 Mar 2023 10:53:58 GMT\r\nLast-Modified: Thu, 17 Oct 2019 07:18:26 GMT\r\nServer: ECS (dcb/7EA3)\r\nVary: Accept-Encoding\r\nX-Cache: HIT\r\n\r\n<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset=\"utf-8\" />\n    <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n    <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n    <style type=\"text/css\">\n    body {\n        background-color: #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n        \n    }\n    div {\n        width: 600px;\n        margin: 5em auto;\n        padding: 2em;\n        background-color: #fdfdff;\n        border-radius: 0.5em;\n        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n    }\n    a:link, a:visited {\n        color: #38488f;\n        text-decoration: none;\n    }\n    @media (max-width: 700px) {\n        div {\n            margin: 0 auto;\n            width: auto;\n        }\n    }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission.</p>\n    <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n"
  }
}
```

*`-store-response`*
----

The `-store-response` option allows for writing all crawled endpoint requests and responses to a text file. When this option is used, text files including the request and response will be written to the **katana_response** directory. If you would like to specify a custom directory, you can use the `-store-response-dir` option.

```console
katana -u https://example.com -no-scope -store-response
```

```bash
$ cat katana_response/index.txt

katana_response/example.com/327c3fda87ce286848a574982ddd0b7c7487f816.txt https://example.com (200 OK)
katana_response/www.iana.org/bfc096e6dd93b993ca8918bf4c08fdc707a70723.txt http://www.iana.org/domains/reserved (200 OK)
```

**Note:**

*`-store-response` option is not supported in `-headless` mode.*

*`-list-output-fields`*
----

The `-list-output-fields` or `-lof` flag displays all available fields that can be used in JSONL output format. This is useful for understanding what data is available when using custom output templates or when excluding specific fields.

```console
katana -lof
```

*`-exclude-output-fields`*
----

The `-exclude-output-fields` or `-eof` flag allows you to exclude specific fields from the JSONL output. This is useful for reducing output size or focusing on specific data by removing unwanted fields.

```console
katana -u https://example.com -jsonl -eof raw,body
```

Here are additional CLI options related to output -

```console
katana -h output

OUTPUT:
   -o, -output string                file to write output to
   -sr, -store-response              store http requests/responses
   -srd, -store-response-dir string  store http requests/responses to custom directory
   -lof, -list-output-fields         list available fields for jsonl output format
   -eof, -exclude-output-fields      exclude fields from jsonl output
   -j, -json                         write output in JSON Lines format
   -nc, -no-color                    disable output content coloring (ANSI escape codes)
   -silent                           display output only
   -v, -verbose                      display verbose output
   -version                          display project version
```

## Katana as a library
`katana` can be used as a library by creating an instance of the `Option` struct and populating it with the same options that would be specified via CLI. Using the options you can create `crawlerOptions` and so standard or hybrid `crawler`.
`crawler.Crawl` method should be called to crawl the input.

```go
package main

import (
	"math"

	"github.com/projectdiscovery/gologger"
	"github.com/projectdiscovery/katana/pkg/engine/standard"
	"github.com/projectdiscovery/katana/pkg/output"
	"github.com/projectdiscovery/katana/pkg/types"
)

func main() {
	options := &types.Options{
		MaxDepth:     3,             // Maximum depth to crawl
		FieldScope:   "rdn",         // Crawling Scope Field
		BodyReadSize: math.MaxInt,   // Maximum response size to read
		Timeout:      10,            // Timeout is the time to wait for request in seconds
		Concurrency:  10,            // Concurrency is the number of concurrent crawling goroutines
		Parallelism:  10,            // Parallelism is the number of urls processing goroutines
		Delay:        0,             // Delay is the delay between each crawl requests in seconds
		RateLimit:    150,           // Maximum requests to send per second
		Strategy:     "depth-first", // Visit strategy (depth-first, breadth-first)
		OnResult: func(result output.Result) { // Callback function to execute for result
			gologger.Info().Msg(result.Request.URL)
		},
	}
	crawlerOptions, err := types.NewCrawlerOptions(options)
	if err != nil {
		gologger.Fatal().Msg(err.Error())
	}
	defer crawlerOptions.Close()
	crawler, err := standard.New(crawlerOptions)
	if err != nil {
		gologger.Fatal().Msg(err.Error())
	}
	defer crawler.Close()
	var input = "https://www.hackerone.com"
	err = crawler.Crawl(input)
	if err != nil {
		gologger.Warning().Msgf("Could not crawl %s: %s", input, err.Error())
	}
}
```

## Reporting Issues & Feature Requests

To maintain issue tracking and improve triage efficiency:

**All reports start as [GitHub Discussions](https://github.com/projectdiscovery/katana/discussions)**

- **Bug Reports** → [Start a Q&A Discussion](https://github.com/projectdiscovery/katana/discussions/new?category=q-a)
- **Feature Requests** → [Start an Ideas Discussion](https://github.com/projectdiscovery/katana/discussions/new?category=ideas)  
- **Questions** → [Start a Q&A Discussion](https://github.com/projectdiscovery/katana/discussions/new?category=q-a)

**Why Discussions First?**
- **Community can help** with quick questions and troubleshooting
- **Better triage** - confirmed bugs/features become tracked issues  
- **Cleaner issue tracker** - focus on actionable items only

Maintainers will convert discussions to issues when appropriate after proper review.

--------

---

## File: LICENSE.md

MIT License

Copyright (c) 2022 ProjectDiscovery

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

---

## File: SECURITY.md

# Security Policy

## Reporting a Vulnerability
DO NOT CREATE AN ISSUE to report a security problem. Instead, please send an email to security@projectdiscovery.io, and we will acknowledge it within 3 working days.

---

## File: .goreleaser.yml

version: 2

env:
  - GO111MODULE=on
  - CGO_ENABLED=0

before:
  hooks:
    - go mod tidy

project_name: katana

builds:
  - id: katana
    ldflags:
      - -s -w
    binary: katana
    main: ./cmd/katana/main.go
    goos:
      - linux
      - windows
      - darwin
    goarch:
      - amd64
      - 386
      - arm64

archives:
  - formats: zip
    name_template: '{{ .ProjectName }}_{{ .Version }}_{{ if eq .Os "darwin" }}macOS{{ else }}{{ .Os }}{{ end }}_{{ .Arch }}'

checksum:
  name_template: "{{ .ProjectName }}-{{ .Version }}-checksums.txt"

dockers_v2:
  - images:
      - projectdiscovery/katana
    tags:
      - "{{ .Tag }}"
      - "v{{ .Major }}.{{ .Minor }}"
      - "v{{ .Major }}"
      - latest
    labels:
      org.opencontainers.image.created: "{{ .Date }}"
      org.opencontainers.image.ref.name: "{{ .Tag }}"
      org.opencontainers.image.revision: "{{ .FullCommit }}"
      org.opencontainers.image.version: "{{ .Version }}"
    platforms:
      - linux/amd64
      - linux/arm64
    # NOTE: CI snapshot runners may lack QEMU; only build images in CI on real
    # releases.
    disable: '{{ and .IsSnapshot (isEnvSet "CI") }}'

announce:
  slack:
    enabled: true
    channel: '#release'
    username: GoReleaser
    message_template: 'New Release: {{ .ProjectName }} {{.Tag}} is published! Check it out at {{ .ReleaseURL }}'
  discord:
    enabled: true
    message_template: '**New Release: {{ .ProjectName }} {{.Tag}}** is published! Check it out at {{ .ReleaseURL }}'

---

## File: .github/dependabot.yml

# To get started with Dependabot version updates, you'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for all configuration options:
# https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates

version: 2
updates:

  # Maintain dependencies for go modules
  - package-ecosystem: "gomod"
    directory: "/"
    schedule:
      interval: "weekly"
    target-branch: "dev"
    commit-message:
      prefix: "chore"
      include: "scope"
    labels:
      - "Type: Maintenance"
    allow:
      - dependency-name: "github.com/projectdiscovery/*"

  # Maintain dependencies for GitHub Actions
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"
    target-branch: "dev"
    commit-message:
      prefix: "chore"
      include: "scope"
    labels:
      - "Type: Maintenance"

  # Maintain dependencies for docker
  - package-ecosystem: "docker"
    directory: "/"
    schedule:
      interval: "weekly"
    target-branch: "dev"
    commit-message:
      prefix: "chore"
      include: "scope"
    labels:
      - "Type: Maintenance"

---

## File: .github/MAINTAINER_GUIDE.md

# Maintainer Guide: Discussion-First Issue Management

## Overview

Katana now uses a **discussion-first approach** for issue management to improve triage quality and reduce noise from questions being filed as bugs.

## How It Works

### 1. **All Reports Start as Discussions**
- Users cannot create issues directly
- All bug reports → **Q&A Discussions**  
- All feature requests → **Ideas Discussions**
- All questions → **Q&A Discussions**

### 2. **Automated Triage Helper**
- Auto-responds to discussions with helpful guidance
- Auto-flags potential bugs with keywords detection
- Provides checklists for proper bug reporting

### 3. **Maintainer Conversion Process**
- Review discussions for completeness
- Convert valid issues using GitHub's built-in feature
- Apply appropriate labels during conversion

## Conversion Guidelines

### 🐛 **Bug Reports** → Convert to Issue When:

**Well-Defined Problems:**
- Clear reproduction steps provided
- Katana version specified
- Expected vs actual behavior described  
- Environment details included
- Error messages/logs included

**Confirmed Bugs:**
- Issue reproduced by maintainer or community
- Not a configuration/usage question
- Not working as designed

**Keep as Discussion:**
- Incomplete information
- Usage questions ("How do I...?")
- Configuration problems
- Working as intended

### 💡 **Feature Requests** → Convert to Issue When:

**Solid Proposals:**
- Clear use case defined
- Benefits to community explained
- Implementation approach considered
- Not easily achievable with existing features

**Community Support:**
- Multiple users expressing interest
- Maintainer approval for implementation
- Fits project roadmap

**Keep as Discussion:**
- Vague ideas needing refinement
- Better suited as external tools/plugins
- Conflicts with project goals
- Needs more community input

## Conversion Process

### Using GitHub's Convert Feature:

1. **Open the discussion**
2. **Click the "⋯" menu** (top right)
3. **Select "Convert to issue"**
4. **Choose repository** (same repo)
5. **Review title/body** - edit if needed
6. **Add labels:**
   - `Type: Bug` for confirmed bugs
   - `Type: Enhancement` for approved features  
   - `Priority: High/Medium/Low` as appropriate
   - `Component: Engine/Parser/Output` etc.

### Template for Converted Issues:

When converting, consider adding this note:

```markdown
**Converted from Discussion:** #[discussion_number]

<!-- Original discussion provided community input and initial triage -->

[Original discussion content here]

---

**Maintainer Notes:**
- [ ] Issue confirmed through discussion
- [ ] Reproduction steps verified  
- [ ] Ready for implementation/investigation
```

## Workflow Benefits

### **For Project Health:**
- **Cleaner issue tracker** - Only actionable items
- **Better metrics** - Issues vs discussions clearly separated
- **Faster resolution** - Less time sorting questions from bugs

### **For Community:**
- **Inclusive discussions** - Everyone can participate in triage
- **Better help** - Community can answer questions quickly  
- **Learning opportunity** - Users see resolution process

### **For Maintainers:**
- **Pre-filtered issues** - Only valid bugs/features reach issue tracker
- **Rich context** - Discussion history provides background
- **Community input** - Others help validate before conversion

## Examples

### **Good Bug Discussion → Issue Conversion**

**Discussion Title:** "Katana crashes when using -hl with custom headers"

**Discussion Body:** 
- Katana version: v1.2.1
- Command: `katana -u example.com -hl -H "Custom: value"`
- Error: panic in hybrid engine
- Platform: macOS 14.1
- Reproduction: consistent crash

**→ Convert to Issue:** Clear bug with reproduction steps

### **Question → Keep as Discussion**

**Discussion Title:** "How to crawl only PDF files?"

**Discussion Body:**
- New user question
- Asking for usage help
- Not a bug or feature request

**→ Keep as Discussion:** Usage question, answer in discussion

### **Needs More Info → Keep as Discussion**

**Discussion Title:** "Katana doesn't work"

**Discussion Body:**
- Vague description
- No version, command, or error details
- No reproduction steps

**→ Keep as Discussion:** Request more information first

## Quick Reference

| Type | Action | Labels for Conversion |
|------|---------|-------------------|
| **Confirmed Bug** | Convert → Issue | `Type: Bug`, `Priority: [level]` |
| **Approved Feature** | Convert → Issue | `Type: Enhancement`, `Priority: [level]` |  
| **Usage Question** | Keep → Discussion | N/A |
| **Needs Info** | Keep → Discussion | N/A |
| **Security Issue** | Email → security@projectdiscovery.io | N/A |

This workflow ensures high-quality issues while maintaining an inclusive, helpful community environment!

---

## File: .github/PULL_REQUEST_TEMPLATE.md

## Proposed changes



### Proof



## Checklist



- [ ] Pull request is created against the [dev](https://github.com/projectdiscovery/katana/tree/dev) branch
- [ ] All checks passed (lint, unit/integration/regression tests etc.) with my changes
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] I have added necessary documentation (if appropriate)

---

## File: .github/release.yml

changelog:
  exclude:
    authors:
      - app/dependabot
      - dependabot
  categories:
    - title: 🎉 New Features
      labels:
        - "Type: Enhancement"
    - title: 🐞 Bug Fixes
      labels:
        - "Type: Bug" 
    - title: 🔨 Maintenance
      labels:
        - "Type: Maintenance" 
    - title: Other Changes
      labels:
        - "*"

---

## File: .github/ISSUE_TEMPLATE/README.md

# Issue Template Management

## New Issue Workflow (Discussion-First)

To improve issue triage and reduce noise from questions being filed as bugs, all issue creation now goes through GitHub Discussions first.

### For Users:

**❌ Direct issue creation is disabled**
**✅ All reports must start as discussions**

Users will be redirected to:
- 🐛 **Bug Reports** → [Q&A Discussions](https://github.com/projectdiscovery/katana/discussions/new?category=q-a)
- 💡 **Feature Requests** → [Ideas Discussions](https://github.com/projectdiscovery/katana/discussions/new?category=ideas)  
- ❓ **Questions** → [Q&A Discussions](https://github.com/projectdiscovery/katana/discussions/new?category=q-a)

### For Maintainers:

#### Converting Discussions to Issues:

1. **Review the discussion** thoroughly
2. **Determine if it's a valid bug/feature** (not just a question)
3. **Convert to issue** using GitHub's "Convert to Issue" feature:
   - Go to the discussion
   - Click "⋯" menu → "Convert to issue"
   - Add appropriate labels and assignees
   
#### Triage Guidelines:

**Convert to Issue:**
- ✅ Confirmed bugs with reproduction steps
- ✅ Well-defined feature requests with clear use cases
- ✅ Security vulnerabilities (after initial assessment)

**Keep as Discussion:**
- ❌ Usage questions ("How do I...?")
- ❌ Configuration help
- ❌ Unclear or incomplete bug reports
- ❌ Feature ideas that need more discussion/refinement

### Benefits:

- 📊 **Better issue quality** - Only confirmed bugs/features become issues
- 🎯 **Easier triage** - Questions don't clutter the issue tracker
- 💬 **Community involvement** - Discussion encourages collaboration before formal issues
- 🧹 **Cleaner issue tracker** - Focus on actionable items only

## Re-enabling Templates (If Needed)

If you need to temporarily re-enable direct issue creation:

```bash
# Re-enable templates
mv issue-report.md.disabled issue-report.md
mv feature_request.md.disabled feature_request.md

# Update config.yml to add them back
```

## Template Files:

- `config.yml` - Main configuration (redirects all to discussions)
- `issue-report.md.disabled` - Bug report template (disabled)
- `feature_request.md.disabled` - Feature request template (disabled)

---

## File: .github/ISSUE_TEMPLATE/config.yml

blank_issues_enabled: false

contact_links:
  - name: Report a Bug (Discussion First)
    url: https://github.com/projectdiscovery/katana/discussions/new?category=q-a
    about: Report bugs or issues via discussion for proper triage before issue creation

  - name: Feature Request (Discussion First)
    url: https://github.com/projectdiscovery/katana/discussions/new?category=ideas
    about: Share feature ideas via discussion for evaluation before implementation

  - name: Ask a Question
    url: https://github.com/projectdiscovery/katana/discussions/new?category=q-a
    about: Ask questions about usage, configuration, or troubleshooting

  - name: General Discussion
    url: https://github.com/projectdiscovery/katana/discussions/new?category=general
    about: General discussion about the project

  - name: Security Issues
    url: mailto:security@projectdiscovery.io
    about: Report security vulnerabilities privately via email

  - name: Discord Community
    url: https://discord.gg/projectdiscovery
    about: Connect with PD Team and community for real-time discussion

---

## File: .github/workflows/build-test.yml

name: 🔨 Build Test

on:
  workflow_dispatch:
  pull_request:
    branches:
      - dev
    paths:
      - '**.go'
      - '**.mod'
jobs:
  lint:
    name: "Lint"
    if: "${{ !endsWith(github.actor, '[bot]') }}"
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: projectdiscovery/actions/setup/go@v1
      - uses: projectdiscovery/actions/golangci-lint/v2@v1

  build:
    name: Test Builds
    needs: [lint]
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macOS-latest]
    steps:
      - name: Check out code
        uses: actions/checkout@v7

      - name: Set up Go
        uses: projectdiscovery/actions/setup/go@v1

      - name: Test
        run: go test ./...
        working-directory: .

      - name: Build
        run: go build .
        working-directory: cmd/katana/

      - name: Integration Tests
        env:
          GH_ACTION: true
        run: bash run.sh
        working-directory: integration_tests/

      - name: Install
        run: go install
        working-directory: cmd/katana/

      - name: Race Condition Tests
        run: go build -race .
        working-directory: cmd/katana/

---

## File: .github/workflows/codeql-analysis.yml

name: 🚨 CodeQL Analysis

on:
  workflow_dispatch:
  pull_request:
    branches:
      - dev
    paths:
      - '**.go'
      - '**.mod'

jobs:
  analyze:
    name: Analyze
    runs-on: ubuntu-latest
    permissions:
      actions: read
      contents: read
      security-events: write

    strategy:
      fail-fast: false
      matrix:
        language: [ 'go' ]
        # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python' ]

    steps:
    - name: Checkout repository
      uses: actions/checkout@v7

    # Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
      uses: github/codeql-action/init@v4.39.0
      with:
        languages: ${{ matrix.language }}

    - name: Autobuild
      uses: github/codeql-action/autobuild@v4.39.0

    - name: Perform CodeQL Analysis
      uses: github/codeql-action/analyze@v4.39.0

---

## File: .github/workflows/compat-checks.yaml

name: ♾️ Compatibility Checks

on:
  pull_request:
    types: [opened, synchronize]
    branches:
      - dev

jobs:
  check:
    if: github.actor == 'dependabot[bot]'
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@v7
      - uses: projectdiscovery/actions/setup/go/compat-checks@v1

---

## File: .github/workflows/dep-auto-merge.yml

name: 🤖 Auto Merge

on:
  pull_request_review:
    types: [submitted]
  workflow_run:
    workflows: ["♾️ Compatibility Check"]
    types:
      - completed

permissions:
  pull-requests: write
  issues: write
  repository-projects: write

jobs:
  auto-merge:
    runs-on: ubuntu-latest
    if: github.actor == 'dependabot[bot]'
    steps:
      - uses: actions/checkout@v7
        with:
          token: ${{ secrets.DEPENDABOT_PAT }}

      - uses: ahmadnassri/action-dependabot-auto-merge@v2
        with:
          github-token: ${{ secrets.DEPENDABOT_PAT }}
          target: all

---

## File: .github/workflows/discussion-triage.yml

name: Discussion Triage Helper

on:
  discussion:
    types: [created]

permissions:
  discussions: write

jobs:
  triage:
    runs-on: ubuntu-latest
    steps:
      - name: Add initial response to discussions
        uses: actions/github-script@v9
        with:
          script: |
            const { discussion } = context.payload;
            
            // Add helpful response based on discussion category
            let response = "";
            
            if (discussion.category.slug === "q-a") {
              if (discussion.title.toLowerCase().includes("bug") || 
                  discussion.title.toLowerCase().includes("error") || 
                  discussion.title.toLowerCase().includes("crash") ||
                  discussion.title.toLowerCase().includes("fail")) {
                
                response = "Thanks for the discussion! This looks like it might be a bug report.\n\n" +
                          "**For Maintainers:** If this is confirmed to be a bug with proper reproduction steps, consider converting to an issue using \"Convert to issue\" in the discussion menu.\n\n" +
                          "**For the Reporter:** To help us triage this effectively, please provide:\n" +
                          "- [ ] Katana version (`katana -version`)\n" +
                          "- [ ] Complete command that reproduces the issue\n" +
                          "- [ ] Expected vs actual behavior\n" +
                          "- [ ] Operating system and environment details\n" +
                          "- [ ] Any error messages or logs\n\n" +
                          "This helps us determine if this should become a tracked issue. Thank you!";
              
              } else {
                response = "Thanks for your question!\n\n" +
                          "The community and maintainers will help answer this. If this turns out to be a bug or feature request after discussion, a maintainer can convert it to a tracked issue.\n\n" +
                          "**Tip:** For faster responses, you can also join our [Discord server](https://discord.gg/projectdiscovery) for real-time help!";
              }
            }
            
            else if (discussion.category.slug === "ideas") {
              response = "Thanks for sharing this idea!\n\n" +
                        "**For Maintainers:** If this feature request is well-defined and approved for implementation, consider converting to an issue using \"Convert to issue\" in the discussion menu.\n\n" +
                        "**For the Reporter:** To help evaluate this feature:\n" +
                        "- [ ] Describe the specific use case this would solve\n" +
                        "- [ ] Explain how this would benefit the katana user community\n" +  
                        "- [ ] Consider if this could be implemented as a plugin or external tool\n" +
                        "- [ ] Check if similar functionality already exists\n\n" +
                        "Well-defined features with clear benefits may be converted to tracked issues for implementation. Thanks for contributing!";
            }
            
            if (response) {
              await github.rest.discussions.createComment({
                owner: context.repo.owner,
                repo: context.repo.repo,
                discussion_number: discussion.number,
                body: response
              });
            }

      - name: Auto-label potential bug reports
        uses: actions/github-script@v9
        with:
          script: |
            const { discussion } = context.payload;
            const title = discussion.title.toLowerCase();
            const body = discussion.body.toLowerCase();
            
            // Check for keywords that suggest this might be a bug
            const bugKeywords = [
              "bug", "error", "crash", "fail", "broken", "issue", "problem", 
              "exception", "panic", "segfault", "hang", "freeze", "incorrect"
            ];
            
            const hasBugKeywords = bugKeywords.some(keyword => 
              title.includes(keyword) || body.includes(keyword)
            );
            
            if (hasBugKeywords && discussion.category.slug === "q-a") {
              // Add a comment suggesting this might need to be converted to an issue
              const detectedKeywords = bugKeywords.filter(k => title.includes(k) || body.includes(k));
              await github.rest.discussions.createComment({
                owner: context.repo.owner,
                repo: context.repo.repo,  
                discussion_number: discussion.number,
                body: "**Maintainer Note:** This discussion contains potential bug keywords and may need to be converted to a tracked issue after confirmation.\n\n" +
                      "**Keywords detected:** " + detectedKeywords.join(', ')
              });
            }

---

## File: .github/workflows/functional-test.yml

name: 🧪 Functional Test

on:
  pull_request:
    paths:
      - '**.go'
      - '**.mod'
  workflow_dispatch:

jobs:  
  functional:
    name: Functional Test
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macOS-latest]
    steps:
      - name: Check out code
        uses: actions/checkout@v7

      - name: Set up Go
        uses: projectdiscovery/actions/setup/go@v1

      - name: Functional Tests
        run: |
          chmod +x run.sh
          bash run.sh ${{ matrix.os }}
        working-directory: cmd/functional-test

---

## File: .github/workflows/release-test.yml

name: 🔨 Release Test

on:
  pull_request:
    paths:
      - "**.yml"
      - "**.go"
      - "**.mod"
  workflow_dispatch:

jobs:
  release-test:
    runs-on: ubuntu-latest
    steps:
      - name: "Check out code"
        uses: actions/checkout@v7
        with:
          fetch-depth: 0

      - name: Set up Go
        uses: projectdiscovery/actions/setup/go@v1

      - name: release test
        uses: projectdiscovery/actions/goreleaser@v1
        with:
          args: --snapshot --clean
          workdir: .
        env:
          GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"

---

## File: .github/workflows/release.yml

name: 🎉 Release

on:
  push:
    tags:
      - v*
  workflow_dispatch:

jobs:
  release:
    runs-on: ubuntu-latest-16-cores
    steps:
      - name: "Check out code"
        uses: actions/checkout@v7
        with:
          fetch-depth: 0

      - name: "Set up Go"
        uses: projectdiscovery/actions/setup/go@v1

      - name: "Login to DockerHub"
        uses: docker/login-action@v4
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_TOKEN }}

      - name: "Create release on GitHub"
        uses: projectdiscovery/actions/goreleaser@v1
        with:
          release: true
        env:
          GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
          SLACK_WEBHOOK: "${{ secrets.RELEASE_SLACK_WEBHOOK }}"
          DISCORD_WEBHOOK_ID: "${{ secrets.DISCORD_WEBHOOK_ID }}"
          DISCORD_WEBHOOK_TOKEN: "${{ secrets.DISCORD_WEBHOOK_TOKEN }}"

---

## File: .github/workflows/security-crawl-maze-score.yaml

name: 🙏🏻 Security Crawl Maze Score

on:
  workflow_dispatch:

jobs:
  build:
    name: Run Scoring
    runs-on: ubuntu-latest-16-cores
    steps:
      - name: Check out code
        uses: actions/checkout@v7

      - name: Set up Go
        uses: projectdiscovery/actions/setup/go@v1

      - name: Build
        run: go build .
        working-directory: cmd/katana/

      - name: Run Katana Standard
        run: ./katana -u https://security-crawl-maze.app/ -kf all -jc -jsluice -d 10 -o output_standard.txt -cos node_modules
        working-directory: cmd/katana
        
      - name: Run Katana Headless
        run: ./katana -u https://security-crawl-maze.app/ -kf all -jc -jsluice -d 10 -headless -o output_headless.txt -cos node_modules
        working-directory: cmd/katana

      - name: Run Score
        run: go build .; ./crawl-maze-score ../../katana/output_standard.txt ../../katana/output_headless.txt
        working-directory: cmd/tools/crawl-maze-score

---

## File: .github/workflows/stale.yml

name: 💤 Stale

on:
  schedule:
    - cron: '0 0 * * 0' # Weekly

jobs:
  stale:
    runs-on: ubuntu-latest
    permissions:
      actions: write
      contents: write # only for delete-branch option
      issues: write
      pull-requests: write
    steps:
      - uses: actions/stale@v11
        with:
          days-before-stale: 90
          days-before-close: 7
          stale-issue-label: "Status: Stale"
          stale-pr-label: "Status: Stale"
          stale-issue-message: >
            This issue has been automatically marked as stale because it has not
            had recent activity. It will be closed in 7 days if no further
            activity occurs. Thank you for your contributions!
          stale-pr-message: >
            This pull request has been automatically marked as stale due to
            inactivity. It will be closed in 7 days if no further activity
            occurs. Please update if you wish to keep it open.
          close-issue-message: >
            This issue has been automatically closed due to inactivity. If you
            think this is a mistake or would like to continue the discussion,
            please comment or feel free to reopen it.
          close-pr-message: >
            This pull request has been automatically closed due to inactivity.
            If you think this is a mistake or would like to continue working on
            it, please comment or feel free to reopen it.
          close-issue-label: "Status: Abandoned"
          close-pr-label: "Status: Abandoned"
          exempt-issue-labels: "Status: Abandoned"
          exempt-pr-labels: "Status: Abandoned"

---

## File: .github/workflows/workflow-monitor.yml

name: Workflow Monitor

on:
  schedule:
    # Run weekly on Mondays at 9 AM UTC
    - cron: '0 9 * * 1'
  workflow_dispatch: # Allow manual triggering

permissions:
  issues: read
  discussions: read

jobs:
  monitor-workflow:
    runs-on: ubuntu-latest
    steps:
      - name: Check Discussion vs Issue Activity
        uses: actions/github-script@v9
        with:
          script: |
            const oneWeekAgo = new Date();
            oneWeekAgo.setDate(oneWeekAgo.getDate() - 7);
            const since = oneWeekAgo.toISOString();
            
            // Get discussions created in the last week
            const discussions = await github.graphql(`
              query($owner: String!, $repo: String!, $since: DateTime!) {
                repository(owner: $owner, name: $repo) {
                  discussions(first: 100, orderBy: {field: CREATED_AT, direction: DESC}) {
                    nodes {
                      title
                      createdAt
                      category {
                        name
                        slug
                      }
                      comments(first: 1) {
                        totalCount
                      }
                    }
                  }
                }
              }
            `, {
              owner: context.repo.owner,
              repo: context.repo.repo,
              since: since
            });
            
            // Get issues created in the last week
            const issues = await github.rest.issues.listForRepo({
              owner: context.repo.owner,
              repo: context.repo.repo,
              since: since,
              state: 'all'
            });
            
            // Filter discussions created in the last week
            const recentDiscussions = discussions.repository.discussions.nodes.filter(
              d => new Date(d.createdAt) >= oneWeekAgo
            );
            
            // Generate and log simple report
            console.log("=== Discussion-First Workflow Report ===");
            console.log(`Period: ${oneWeekAgo.toDateString()} - ${new Date().toDateString()}`);
            console.log(`Discussions Created: ${recentDiscussions.length}`);
            console.log(`Issues Created: ${issues.data.length}`);
            console.log(`Conversion Ratio: ${issues.data.length}/${recentDiscussions.length}`);
            
            if (recentDiscussions.length > 0) {
              console.log("\nDiscussion Categories:");
              recentDiscussions.forEach(d => {
                console.log(`- ${d.category.name}: ${d.title} (${d.comments.totalCount} comments)`);
              });
            } else {
              console.log("No discussions this week");
            }
            
            if (issues.data.length > 0) {
              console.log("\nDirect Issues Created:");
              issues.data.forEach(i => {
                console.log(`- ${i.title} (${new Date(i.created_at).toDateString()})`);
              });
              console.log("\nWARNING: Direct issues were created - check workflow configuration");
            } else {
              console.log("\nNo direct issues created - workflow working correctly");
            }
            
            const responseRate = recentDiscussions.length > 0 ? 
              (recentDiscussions.filter(d => d.comments.totalCount > 0).length / recentDiscussions.length * 100).toFixed(1) : 0;
            console.log(`Community Response Rate: ${responseRate}%`);
            
            // You could also create an issue or discussion with this report
            // For now, just log it to Actions output

## 2. Official Technical Reference & Guides (projectdiscovery/docs)

## File: bin/README-auto-grouped-api.md

# Auto-Grouped API Documentation Generator

This script **automatically generates and maintains** API documentation with logical grouping based on OpenAPI tags. It combines the best of both worlds: auto-generation (no missing endpoints) and logical organization.

## Usage

```bash
# Use default source file (Downloads directory)
./bin/generate-auto-grouped-api.sh

# Use custom source file  
./bin/generate-auto-grouped-api.sh /path/to/your/openapi.yaml
```

## What it does

### 🤖 **Auto-Generation**
- Reads OpenAPI specification
- Filters out admin and internal APIs
- Generates MDX files for all public endpoints  
- **Never misses endpoints** - everything is included automatically

### 📁 **Logical Grouping** 
The script organizes endpoints into logical groups based on OpenAPI tags:

- **Assets** → `assets` + `enumerations` tags
- **Templates** → `templates` + `template/v2` tags  
- **Scans** → `scans` tag
- **Results** → `results` + `retests` tags
- **Configurations** → `configurations` tag
- **Users** → `users` + `user` tags
- **Logs** → `elog` + `scan_log` tags
- **Export** → `export` tag
- **Automations** → `automations` tag
- **Integration** → `oauth` tag  
- **Utilities** → `chaos`, `vulns`, `leaks`, `billing`, `usage` tags

### 🔄 **Auto-Updates Navigation**
- Automatically updates `mint.json` with new structure
- Maintains logical grouping
- Sorts endpoints alphabetically within groups
- Handles new endpoints automatically

## Benefits

✅ **No Manual Maintenance** - Add new endpoints to OpenAPI spec and they appear automatically  
✅ **Never Miss Endpoints** - All public endpoints are included  
✅ **Logical Organization** - Grouped by functionality, not alphabetically  
✅ **Consistent Naming** - Uses OpenAPI summaries and operation IDs  
✅ **Future-Proof** - Works with any OpenAPI specification changes

## Output Structure

After running, your `mint.json` will have clean navigation like:

```json
{
  "group": "API Reference",
  "pages": [
    "api-reference/introduction",
    {
      "group": "Assets", 
      "pages": [
        "api-reference/assets/get-asset-list",
        "api-reference/assets/upload-asset",
        "api-reference/enumerations/create-enumeration",
        "..."
      ]
    },
    {
      "group": "Templates",
      "pages": [
        "api-reference/templates/create-template",
        "api-reference/templates/get-template-list",
        "..."
      ]
    }
  ]
}
```

## Comparison with Manual Approach

| Feature | Manual (`generate-filtered-api.sh`) | Auto-Grouped (`generate-auto-grouped-api.sh`) |
|---------|--------------------------------------|-----------------------------------------------|
| **Missing Endpoints** | ❌ Requires manual addition to `mint.json` | ✅ Never - all endpoints auto-included |
| **Logical Grouping** | ✅ Full manual control | ✅ Smart tag-based grouping |
| **Maintenance** | ❌ High - update `mint.json` for new endpoints | ✅ Zero - fully automated |
| **Consistency** | ❌ Depends on manual updates | ✅ Always consistent |
| **New API Versions** | ❌ Manual review and updates needed | ✅ Automatically handles changes |

## When to Use

**Use Auto-Grouped when:**
- You want zero maintenance overhead
- You frequently add/remove API endpoints  
- You have a team that might forget to update navigation
- You want consistent organization based on API design

**Use Manual when:**
- You need very specific custom grouping that doesn't match tags
- You want to exclude certain public endpoints from docs
- You have complex navigation requirements

## Next Steps

After running this script:

1. **Preview:** `mintlify dev`
2. **Deploy:** Your CI/CD can run this script automatically when OpenAPI changes
3. **Customize:** Modify the `logicalGroups` mapping in the script if needed

---

## File: bin/README-filtered-api.md

# Filtered API Documentation Generator

This script generates API documentation from an OpenAPI specification while excluding internal and admin endpoints.

## Usage

```bash
# Use default source file (Downloads directory)
./bin/generate-filtered-api.sh

# Use custom source file
./bin/generate-filtered-api.sh /path/to/your/openapi.yaml
```

**Note:** The script will create a filtered `openapi.yaml` file in the current directory for Mintlify to use.

## What gets filtered out

The script automatically excludes:

1. **Admin APIs**: All paths starting with `/v1/admin/`
   - Examples: `/v1/admin/user/search`, `/v1/admin/team/change_owner`

2. **Internal APIs**: Endpoints tagged with `internal`
   - Any endpoint with `"internal"` in its tags array

## Output

- Creates filtered API documentation in the `api-reference/` directory
- Removes existing `internal/` and `admin/` folders before regenerating
- Uses Mintlify scraping to generate MDX files for each endpoint
- Provides detailed console output showing what was filtered

## Requirements

- Node.js and npm/npx
- Internet connection (to download @mintlify/scraping package)
- Valid OpenAPI 3.x specification file

## Integration with existing workflow

You can replace the existing `./bin/generate-api.sh` call in your build process with:

```bash
# Old way (includes internal/admin APIs)
./bin/generate-api.sh

# New way (excludes internal/admin APIs)  
./bin/generate-filtered-api.sh
```

## Troubleshooting

If you encounter errors:

1. **"OpenAPI source file not found"**: Ensure the file path is correct
2. **"Node.js is required"**: Install Node.js from https://nodejs.org
3. **Network errors**: Check internet connection for npm package downloads
4. **Permission errors**: Ensure the script is executable (`chmod +x bin/generate-filtered-api.sh`)