## 1. Project Overview & Quickstart (projectdiscovery/httpx)

<h1 align="center">
  
  
</h1>








`httpx` is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the [retryablehttp](https://github.com/projectdiscovery/retryablehttp-go) library. It is designed to maintain result reliability with an increased number of threads.

# Features

<h1 align="center">
  
  
</h1>

 - Simple and modular code base making it easy to contribute.
 - Fast And fully configurable flags to probe multiple elements.
 - Supports multiple HTTP based probings.
 - Smart auto fallback from https to http by default. 
 - Supports hosts, URLs and CIDR as input.
 - Handles edge cases doing retries, backoffs etc for handling WAFs.

### Supported probes

| Probes          | Default check | Probes         | Default check |
|-----------------|---------------|----------------|---------------|
| URL             | true          | IP             | true          |
| Title           | true          | CNAME          | true          |
| Status Code     | true          | Raw HTTP       | false         |
| Content Length  | true          | HTTP2          | false         |
| TLS Certificate | true          | HTTP Pipeline  | false         |
| CSP Header      | true          | Virtual host   | false         |
| Line Count      | true          | Word Count     | true          |
| Location Header | true          | CDN            | false         |
| Web Server      | true          | Paths          | false         |
| Web Socket      | true          | Ports          | false         |
| Response Time   | true          | Request Method | true          |
| Favicon Hash    | false         | Probe  Status  | false         |
| Body Hash       | true          | Header  Hash   | true          |
| Redirect chain  | false         | URL Scheme     | true          |
| JARM Hash       | false         | ASN            | false         |

# Installation Instructions

`httpx` requires **go >=1.25.0** to install successfully. Run the following command to get the repo:

```sh
go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest
```

To learn more about installing httpx, see https://docs.projectdiscovery.io/tools/httpx/install.

| :exclamation:  **Disclaimer**  |
|---------------------------------|
| **This project is in active development**. Expect breaking changes with releases. Review the changelog before updating. |
| This project was primarily built to be used as a standalone CLI tool. **Running it as a service may pose security risks.** It's recommended to use with caution and additional security measures. |

# Usage

```sh
httpx -h
```

This will display help for the tool. Here are all the switches it supports.


```
/* Detailed source-code truncated for AI context efficiency. */
```

# Running httpx

For details about running httpx, see https://docs.projectdiscovery.io/tools/httpx/running.

### Using `httpx` as a library
`httpx` can be used as a library by creating an instance of the `Option` struct and populating it with the same options that would be specified via CLI. Once validated, the struct should be passed to a runner instance (to be closed at the end of the program) and the `RunEnumeration` method should be called. A minimal example of how to do it is in the [examples](examples/) folder.

## Common Recipes

Below are practical one-liners for common use cases leveraging httpx's composable primitives. These recipes are validated in `runner/wellknown_recipes_test.go`.

Use `-mdc` with DSL helpers such as `contains(content_type, ...)` and `contains(body, ...)` to match response metadata and body content. (`-mr`/`-ms` match the full raw response; use `-mdc` for structured field matching.)

### Well-known files

**security.txt**  
Probe for a valid [RFC 9116](https://www.rfc-editor.org/rfc/rfc9116.html) security.txt file at the standard paths (`/.well-known/security.txt`, `/security.txt`):

```bash
echo target.com | httpx -path '/.well-known/security.txt,/security.txt' -mc 200 -mdc 'contains(content_type, "text/plain") && contains(body, "Contact:") && contains_any(body, "mailto:", "https://")'
```

- `-path` tests custom path(s)
- `-mc 200` matches HTTP 200
- `-mdc` matches using DSL expressions on response fields such as `content_type` and `body`

**robots.txt**

```bash
echo target.com | httpx -path '/robots.txt' -mc 200 -mdc 'contains(content_type, "text/plain")'
```

**sitemap.xml**

```bash
echo target.com | httpx -path '/sitemap.xml' -mc 200 -mdc 'contains_any(content_type, "application/xml", "text/xml") && contains(body, "<urlset")'
```

**humans.txt**

```bash
echo target.com | httpx -path '/humans.txt' -mc 200 -mdc 'contains(content_type, "text/plain")'
```

**ads.txt**

```bash
echo target.com | httpx -path '/ads.txt' -mc 200 -mdc 'contains(content_type, "text/plain") && contains(body, "google.com")'
```

### Well-known URIs

**OpenID Connect discovery** ([spec](https://openid.net/specs/openid-connect-discovery-1_0.html))

```bash
echo target.com | httpx -path '/.well-known/openid-configuration' -mc 200 -mdc 'contains(content_type, "application/json") && contains(body, "\"issuer\"")'
```

**Apple Universal Links**

```bash
echo target.com | httpx -path '/.well-known/apple-app-site-association,/.well-known/apple-app-site-association.json' -mc 200 -mdc 'contains(content_type, "application/json") && contains(body, "\"applinks\"")'
```

**Android App Links**

```bash
echo target.com | httpx -path '/.well-known/assetlinks.json' -mc 200 -mdc 'contains(content_type, "application/json") && contains(body, "\"android_app\"")'
```

**crossdomain.xml** (legacy Flash policy file)

```bash
echo target.com | httpx -path '/crossdomain.xml' -mc 200 -mdc 'contains_any(content_type, "application/xml", "text/xml") && contains(body, "cross-domain-policy")'
```

**Other well-known URIs** ([IANA registry](https://www.iana.org/assignments/well-known-uris/well-known-uris.xhtml)):

```bash
echo target.com | httpx -path '/.well-known/security.txt,/.well-known/change-password,/.well-known/openid-configuration' -mc 200
```

# Notes

- As default, `httpx` probe with **HTTPS** scheme and fall-back to **HTTP** only if **HTTPS** is not reachable.
- Burp Suite XML exports can be used as input with `-l burp-export.xml -im burp`
- The `-no-fallback` flag can be used to probe and display both **HTTP** and **HTTPS** result.
- Custom scheme for ports can be defined, for example `-ports http:443,http:80,https:8443`
- Custom resolver supports multiple protocol (**doh|tcp|udp**) in form of `protocol:resolver:port` (e.g. `udp:127.0.0.1:53`)
- Secret files can be used for domain-based authentication via `-sf secrets.yaml`. Supported auth types: `BasicAuth`, `BearerToken`, `Header`, `Cookie`, `Query`. Example:
  ```yaml
  id: example-auth
  info:
    name: Example Auth Config
  static:
    - type: Header
      domains:
        - api.example.com
      headers:
        - key: X-API-Key
          value: secret-key-here
    - type: BasicAuth
      domains-regex:
        - ".*\\.internal\\.com$"
      username: admin
      password: secret
  ```
- The following flags should be used for specific use cases instead of running them as default with other probes:
   - `-ports`
   - `-path`
   - `-vhost`
   - `-screenshot`
   - `-csp-probe`
   - `-tls-probe`
   - `-favicon`
   - `-http2`
   - `-pipeline`
   - `-tls-impersonate`


# Acknowledgement

Probing feature is inspired by [@tomnomnom/httprobe](https://github.com/tomnomnom/httprobe) work ❤️


--------

## 2. Official Technical Reference & Guides (projectdiscovery/docs)

## File: README.md

# ProjectDiscovery Documentation

<h4 align="center">
    This is the source code for the ProjectDiscovery documentation located at https://docs.projectdiscovery.io
</h4>






---

## Development

1. Checkout this repository
1. Install mintlify with `npm i mintlify@latest`
1. Run `mintlify dev`

## Deploying 

To build the final product, we have a couple of additional steps:

1. Build the JS Protocol Docs

- `npm install jsdoc-to-markdown`
- `./bin/jsdocs.sh`

2. Build the PDCP API reference documentation

- Either download the latest `openapi.yaml` manually or run `./bin/download-api.sh`
- Run `./bin/generate-api.sh` to generate any new API files
- **OR** run `./bin/generate-filtered-api.sh` to generate API docs without internal/admin endpoints

3. Deployment

After those, Mintlify handles the deployment automatically.

---

## File: bin/README-auto-grouped-api.md

# Auto-Grouped API Documentation Generator

This script **automatically generates and maintains** API documentation with logical grouping based on OpenAPI tags. It combines the best of both worlds: auto-generation (no missing endpoints) and logical organization.

## Usage

```bash
# Use default source file (Downloads directory)
./bin/generate-auto-grouped-api.sh

# Use custom source file  
./bin/generate-auto-grouped-api.sh /path/to/your/openapi.yaml
```

## What it does

### 🤖 **Auto-Generation**
- Reads OpenAPI specification
- Filters out admin and internal APIs
- Generates MDX files for all public endpoints  
- **Never misses endpoints** - everything is included automatically

### 📁 **Logical Grouping** 
The script organizes endpoints into logical groups based on OpenAPI tags:

- **Assets** → `assets` + `enumerations` tags
- **Templates** → `templates` + `template/v2` tags  
- **Scans** → `scans` tag
- **Results** → `results` + `retests` tags
- **Configurations** → `configurations` tag
- **Users** → `users` + `user` tags
- **Logs** → `elog` + `scan_log` tags
- **Export** → `export` tag
- **Automations** → `automations` tag
- **Integration** → `oauth` tag  
- **Utilities** → `chaos`, `vulns`, `leaks`, `billing`, `usage` tags

### 🔄 **Auto-Updates Navigation**
- Automatically updates `mint.json` with new structure
- Maintains logical grouping
- Sorts endpoints alphabetically within groups
- Handles new endpoints automatically

## Benefits

✅ **No Manual Maintenance** - Add new endpoints to OpenAPI spec and they appear automatically  
✅ **Never Miss Endpoints** - All public endpoints are included  
✅ **Logical Organization** - Grouped by functionality, not alphabetically  
✅ **Consistent Naming** - Uses OpenAPI summaries and operation IDs  
✅ **Future-Proof** - Works with any OpenAPI specification changes

## Output Structure

After running, your `mint.json` will have clean navigation like:

```json
{
  "group": "API Reference",
  "pages": [
    "api-reference/introduction",
    {
      "group": "Assets", 
      "pages": [
        "api-reference/assets/get-asset-list",
        "api-reference/assets/upload-asset",
        "api-reference/enumerations/create-enumeration",
        "..."
      ]
    },
    {
      "group": "Templates",
      "pages": [
        "api-reference/templates/create-template",
        "api-reference/templates/get-template-list",
        "..."
      ]
    }
  ]
}
```

## Comparison with Manual Approach

| Feature | Manual (`generate-filtered-api.sh`) | Auto-Grouped (`generate-auto-grouped-api.sh`) |
|---------|--------------------------------------|-----------------------------------------------|
| **Missing Endpoints** | ❌ Requires manual addition to `mint.json` | ✅ Never - all endpoints auto-included |
| **Logical Grouping** | ✅ Full manual control | ✅ Smart tag-based grouping |
| **Maintenance** | ❌ High - update `mint.json` for new endpoints | ✅ Zero - fully automated |
| **Consistency** | ❌ Depends on manual updates | ✅ Always consistent |
| **New API Versions** | ❌ Manual review and updates needed | ✅ Automatically handles changes |

## When to Use

**Use Auto-Grouped when:**
- You want zero maintenance overhead
- You frequently add/remove API endpoints  
- You have a team that might forget to update navigation
- You want consistent organization based on API design

**Use Manual when:**
- You need very specific custom grouping that doesn't match tags
- You want to exclude certain public endpoints from docs
- You have complex navigation requirements

## Next Steps

After running this script:

1. **Preview:** `mintlify dev`
2. **Deploy:** Your CI/CD can run this script automatically when OpenAPI changes
3. **Customize:** Modify the `logicalGroups` mapping in the script if needed

---

## File: bin/README-filtered-api.md

# Filtered API Documentation Generator

This script generates API documentation from an OpenAPI specification while excluding internal and admin endpoints.

## Usage

```bash
# Use default source file (Downloads directory)
./bin/generate-filtered-api.sh

# Use custom source file
./bin/generate-filtered-api.sh /path/to/your/openapi.yaml
```

**Note:** The script will create a filtered `openapi.yaml` file in the current directory for Mintlify to use.

## What gets filtered out

The script automatically excludes:

1. **Admin APIs**: All paths starting with `/v1/admin/`
   - Examples: `/v1/admin/user/search`, `/v1/admin/team/change_owner`

2. **Internal APIs**: Endpoints tagged with `internal`
   - Any endpoint with `"internal"` in its tags array

## Output

- Creates filtered API documentation in the `api-reference/` directory
- Removes existing `internal/` and `admin/` folders before regenerating
- Uses Mintlify scraping to generate MDX files for each endpoint
- Provides detailed console output showing what was filtered

## Requirements

- Node.js and npm/npx
- Internet connection (to download @mintlify/scraping package)
- Valid OpenAPI 3.x specification file

## Integration with existing workflow

You can replace the existing `./bin/generate-api.sh` call in your build process with:

```bash
# Old way (includes internal/admin APIs)
./bin/generate-api.sh

# New way (excludes internal/admin APIs)  
./bin/generate-filtered-api.sh
```

## Troubleshooting

If you encounter errors:

1. **"OpenAPI source file not found"**: Ensure the file path is correct
2. **"Node.js is required"**: Install Node.js from https://nodejs.org
3. **Network errors**: Check internet connection for npm package downloads
4. **Permission errors**: Ensure the script is executable (`chmod +x bin/generate-filtered-api.sh`)