1. Project Overview & Quickstart (projectdiscovery/httpx) Copy Chapter

httpx is a fast and multi-purpose HTTP toolkit that allows running multiple probes using the retryablehttp library. It is designed to maintain result reliability with an increased number of threads.

Features

• Simple and modular code base making it easy to contribute.
• Fast And fully configurable flags to probe multiple elements.
• Supports multiple HTTP based probings.
• Smart auto fallback from https to http by default.
• Supports hosts, URLs and CIDR as input.
• Handles edge cases doing retries, backoffs etc for handling WAFs.

Supported probes Copy Section

Installation Instructions

httpx requires go >=1.25.0 to install successfully. Run the following command to get the repo:

go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest

To learn more about installing httpx, see https://docs.projectdiscovery.io/tools/httpx/install.

Usage

httpx -h

This will display help for the tool. Here are all the switches it supports.

/* Detailed source-code truncated for AI context efficiency. */

Running httpx

For details about running httpx, see https://docs.projectdiscovery.io/tools/httpx/running.

Using `httpx` as a library Copy Section

httpx can be used as a library by creating an instance of the Option struct and populating it with the same options that would be specified via CLI. Once validated, the struct should be passed to a runner instance (to be closed at the end of the program) and the RunEnumeration method should be called. A minimal example of how to do it is in the examples folder.

Common Recipes Copy Chapter

Below are practical one-liners for common use cases leveraging httpx's composable primitives. These recipes are validated in runner/wellknown_recipes_test.go.

Use -mdc with DSL helpers such as contains(content_type, ...) and contains(body, ...) to match response metadata and body content. (-mr/-ms match the full raw response; use -mdc for structured field matching.)

Well-known files Copy Section

security.txt
Probe for a valid RFC 9116 security.txt file at the standard paths (/.well-known/security.txt, /security.txt):

echo target.com | httpx -path '/.well-known/security.txt,/security.txt' -mc 200 -mdc 'contains(content_type, "text/plain") && contains(body, "Contact:") && contains_any(body, "mailto:", "https://")'

• -path tests custom path(s)
• -mc 200 matches HTTP 200
• -mdc matches using DSL expressions on response fields such as content_type and body

robots.txt

echo target.com | httpx -path '/robots.txt' -mc 200 -mdc 'contains(content_type, "text/plain")'

sitemap.xml

echo target.com | httpx -path '/sitemap.xml' -mc 200 -mdc 'contains_any(content_type, "application/xml", "text/xml") && contains(body, "<urlset")'

humans.txt

echo target.com | httpx -path '/humans.txt' -mc 200 -mdc 'contains(content_type, "text/plain")'

ads.txt

echo target.com | httpx -path '/ads.txt' -mc 200 -mdc 'contains(content_type, "text/plain") && contains(body, "google.com")'

Well-known URIs Copy Section

OpenID Connect discovery (spec)

echo target.com | httpx -path '/.well-known/openid-configuration' -mc 200 -mdc 'contains(content_type, "application/json") && contains(body, "\"issuer\"")'

Apple Universal Links

echo target.com | httpx -path '/.well-known/apple-app-site-association,/.well-known/apple-app-site-association.json' -mc 200 -mdc 'contains(content_type, "application/json") && contains(body, "\"applinks\"")'

Android App Links

echo target.com | httpx -path '/.well-known/assetlinks.json' -mc 200 -mdc 'contains(content_type, "application/json") && contains(body, "\"android_app\"")'

crossdomain.xml (legacy Flash policy file)

echo target.com | httpx -path '/crossdomain.xml' -mc 200 -mdc 'contains_any(content_type, "application/xml", "text/xml") && contains(body, "cross-domain-policy")'

Other well-known URIs (IANA registry):

echo target.com | httpx -path '/.well-known/security.txt,/.well-known/change-password,/.well-known/openid-configuration' -mc 200

Notes

• As default, httpx probe with HTTPS scheme and fall-back to HTTP only if HTTPS is not reachable.
• Burp Suite XML exports can be used as input with -l burp-export.xml -im burp
• The -no-fallback flag can be used to probe and display both HTTP and HTTPS result.
• Custom scheme for ports can be defined, for example -ports http:443,http:80,https:8443
• Custom resolver supports multiple protocol (doh|tcp|udp) in form of protocol:resolver:port (e.g. udp:127.0.0.1:53)
• Secret files can be used for domain-based authentication via -sf secrets.yaml. Supported auth types: BasicAuth, BearerToken, Header, Cookie, Query. Example:
  yaml Copy

  id: example-auth
  info:
    name: Example Auth Config
  static:
    - type: Header
      domains:
        - api.example.com
      headers:
        - key: X-API-Key
          value: secret-key-here
    - type: BasicAuth
      domains-regex:
        - ".*\\.internal\\.com$"
      username: admin
      password: secret

• The following flags should be used for specific use cases instead of running them as default with other probes:
  • -ports
  • -path
  • -vhost
  • -screenshot
  • -csp-probe
  • -tls-probe
  • -favicon
  • -http2
  • -pipeline
  • -tls-impersonate

Acknowledgement

Probing feature is inspired by @tomnomnom/httprobe work ❤️

2. Official Technical Reference & Guides (projectdiscovery/docs) Copy Chapter

File: README.md Copy Chapter

ProjectDiscovery Documentation

This is the source code for the ProjectDiscovery documentation located at https://docs.projectdiscovery.io

Development Copy Chapter

1. Checkout this repository
2. Install mintlify with npm i mintlify@latest
3. Run mintlify dev

Deploying Copy Chapter

To build the final product, we have a couple of additional steps:

1. Build the JS Protocol Docs

• npm install jsdoc-to-markdown
• ./bin/jsdocs.sh

2. Build the PDCP API reference documentation

• Either download the latest openapi.yaml manually or run ./bin/download-api.sh
• Run ./bin/generate-api.sh to generate any new API files
• OR run ./bin/generate-filtered-api.sh to generate API docs without internal/admin endpoints

3. Deployment

After those, Mintlify handles the deployment automatically.

File: bin/README-auto-grouped-api.md Copy Chapter

Auto-Grouped API Documentation Generator

This script automatically generates and maintains API documentation with logical grouping based on OpenAPI tags. It combines the best of both worlds: auto-generation (no missing endpoints) and logical organization.

Usage Copy Chapter

# Use default source file (Downloads directory)
./bin/generate-auto-grouped-api.sh

# Use custom source file  
./bin/generate-auto-grouped-api.sh /path/to/your/openapi.yaml

What it does Copy Chapter

🤖 **Auto-Generation** Copy Section

• Reads OpenAPI specification
• Filters out admin and internal APIs
• Generates MDX files for all public endpoints
• Never misses endpoints - everything is included automatically

📁 **Logical Grouping** Copy Section

The script organizes endpoints into logical groups based on OpenAPI tags:

• Assets → assets + enumerations tags
• Templates → templates + template/v2 tags
• Scans → scans tag
• Results → results + retests tags
• Configurations → configurations tag
• Users → users + user tags
• Logs → elog + scan_log tags
• Export → export tag
• Automations → automations tag
• Integration → oauth tag
• Utilities → chaos, vulns, leaks, billing, usage tags

🔄 **Auto-Updates Navigation** Copy Section

• Automatically updates mint.json with new structure
• Maintains logical grouping
• Sorts endpoints alphabetically within groups
• Handles new endpoints automatically

Benefits Copy Chapter

✅ No Manual Maintenance - Add new endpoints to OpenAPI spec and they appear automatically
✅ Never Miss Endpoints - All public endpoints are included
✅ Logical Organization - Grouped by functionality, not alphabetically
✅ Consistent Naming - Uses OpenAPI summaries and operation IDs
✅ Future-Proof - Works with any OpenAPI specification changes

Output Structure Copy Chapter

After running, your mint.json will have clean navigation like:

{
  "group": "API Reference",
  "pages": [
    "api-reference/introduction",
    {
      "group": "Assets", 
      "pages": [
        "api-reference/assets/get-asset-list",
        "api-reference/assets/upload-asset",
        "api-reference/enumerations/create-enumeration",
        "..."
      ]
    },
    {
      "group": "Templates",
      "pages": [
        "api-reference/templates/create-template",
        "api-reference/templates/get-template-list",
        "..."
      ]
    }
  ]
}

Comparison with Manual Approach Copy Chapter

When to Use Copy Chapter

Use Auto-Grouped when:

• You want zero maintenance overhead
• You frequently add/remove API endpoints
• You have a team that might forget to update navigation
• You want consistent organization based on API design

Use Manual when:

• You need very specific custom grouping that doesn't match tags
• You want to exclude certain public endpoints from docs
• You have complex navigation requirements

Next Steps Copy Chapter

After running this script:

1. Preview: mintlify dev
2. Deploy: Your CI/CD can run this script automatically when OpenAPI changes
3. Customize: Modify the logicalGroups mapping in the script if needed

File: bin/README-filtered-api.md Copy Chapter

Filtered API Documentation Generator

This script generates API documentation from an OpenAPI specification while excluding internal and admin endpoints.

Usage Copy Chapter

# Use default source file (Downloads directory)
./bin/generate-filtered-api.sh

# Use custom source file
./bin/generate-filtered-api.sh /path/to/your/openapi.yaml

Note: The script will create a filtered openapi.yaml file in the current directory for Mintlify to use.

What gets filtered out Copy Chapter

The script automatically excludes:

1. Admin APIs: All paths starting with /v1/admin/

   • Examples: /v1/admin/user/search, /v1/admin/team/change_owner

2. Internal APIs: Endpoints tagged with internal

   • Any endpoint with "internal" in its tags array

Output Copy Chapter

• Creates filtered API documentation in the api-reference/ directory
• Removes existing internal/ and admin/ folders before regenerating
• Uses Mintlify scraping to generate MDX files for each endpoint
• Provides detailed console output showing what was filtered

Requirements Copy Chapter

• Node.js and npm/npx
• Internet connection (to download @mintlify/scraping package)
• Valid OpenAPI 3.x specification file

Integration with existing workflow Copy Chapter

You can replace the existing ./bin/generate-api.sh call in your build process with:

# Old way (includes internal/admin APIs)
./bin/generate-api.sh

# New way (excludes internal/admin APIs)  
./bin/generate-filtered-api.sh

Troubleshooting Copy Chapter

If you encounter errors:

1. "OpenAPI source file not found": Ensure the file path is correct
2. "Node.js is required": Install Node.js from https://nodejs.org
3. Network errors: Check internet connection for npm package downloads
4. Permission errors: Ensure the script is executable (chmod +x bin/generate-filtered-api.sh)