## 1. Project Overview & Quickstart (owasp-amass/LICENSE)

# LICENSE

Open-source repository owasp-amass/LICENSE

### Repository Details
- **Repository:** [owasp-amass/LICENSE](https://github.com/owasp-amass/LICENSE)
- **Primary Language:** Code

*Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.*

## 2. Official Technical Reference & Guides (owasp-amass/docs)

## File: docs/data_sources/data_sources.md

# :simple-owasp: Data Sources

Data sources are plugins that query external services
to discover assets. They include API integrations, web
scrapers, and WHOIS/RDAP lookups. Each data source
registers one or more Handlers that are triggered when
specific asset types are processed by the engine.

All data sources that require credentials are configured
through a `datasources.yaml` file.

## :material-key: Configuring Credentials

Point Amass to your data sources file from
`config.yaml`:

```yaml
options:
  datasources: "./datasources.yaml"
```

Each source in `datasources.yaml` follows this format:

```yaml
datasources:
  - name: Shodan
    ttl: 10080       # optional: cache TTL (minutes)
    creds:
      account:
        apikey: YOUR_API_KEY
```

For sources requiring a username and API key:

```yaml
  - name: PassiveTotal
    creds:
      account:
        username: your@email.com
        apikey: YOUR_API_KEY
```

For sources requiring a username and password:

```yaml
  - name: CIRCL
    creds:
      account:
        username: YOUR_USERNAME
        password: YOUR_PASSWORD
```

Multiple credential accounts are supported per
source — useful for rate limit distribution:

```yaml
  - name: C99
    creds:
      account1:
        apikey: FIRST_KEY
      account2:
        apikey: SECOND_KEY
```

## :material-tune: Global Options

Set a minimum TTL floor that applies to all sources:

```yaml
global_options:
  minimum_ttl: 1440  # 1 day (minutes) — default
```

Sources without an explicit `ttl` value in their entry
use this global floor.

## :material-table: Available Data Sources

All 38 data sources require credentials. Default TTL
is 1440 minutes (1 day) unless listed otherwise. The
full template for all sources is in
[`resources/datasources.yaml`][ds-yaml].

[ds-yaml]:
  https://github.com/owasp-amass/amass/blob/main/resources/datasources.yaml

| Source | Cred Type | TTL |
| :--- | :--- | :---: |
| 360PassiveDNS | API Key | 3600 |
| ASNLookup | API Key | 1440 |
| Ahrefs | API Key | 4320 |
| AlienVault | API Key | 1440 |
| BeVigil | API Key | 1440 |
| BigDataCloud | API Key | 1440 |
| BinaryEdge | API Key | 10080 |
| BufferOver | API Key | 1440 |
| BuiltWith | API Key | 10080 |
| C99 | API Key (multi-acct) | 4320 |
| CertCentral | User + API Key | 1440 |
| Chaos | API Key | 4320 |
| CIRCL | User + Password | 1440 |
| DNSDB | API Key | 4320 |
| DNSRepo | API Key | 1440 |
| Detectify | API Key | 1440 |
| FOFA | User + API Key | 10080 |
| FullHunt | API Key | 1440 |
| GitHub | API Key | 4320 |
| GitLab | API Key | 4320 |
| HackerTarget | API Key | 1440 |
| Hunter | API Key | 1440 |
| IntelX | API Key | 1440 |
| IPdata | API Key | 1440 |
| IPinfo | API Key | 1440 |
| LeakIX | API Key | 1440 |
| Netlas | API Key | 1440 |
| PassiveTotal | User + API Key | 10080 |
| PentestTools | API Key | 10080 |
| PublicWWW | API Key | 10080 |
| SecurityTrails | API Key | 1440 |
| Shodan | API Key | 10080 |
| URLScan | API Key | 1440 |
| VirusTotal | API Key | 10080 |
| WhoisXMLAPI | API Key | 1440 |
| Yandex | User + API Key | 1440 |
| ZETAlytics | API Key | 1440 |
| ZoomEye | User + Password | 1440 |

---

*© 2025 Jeff Foley — Licensed under Apache 2.0.*

---

## File: docs/contributing/contributing.md

# :simple-owasp: Contributing to OWASP Amass

Welcome to the OWASP Amass Project! This guide covers
everything you need to contribute — whether you're filing
a bug, writing documentation, developing a data source
plugin, or working on the engine internals.

## :material-forum: Getting Involved

Start by joining the community:

- **Discord:** <https://discord.gg/ANTyEDUXt5> — the
  primary place for discussion and questions
- **GitHub Issues:**
  <https://github.com/owasp-amass/amass/issues> —
  check what needs help
- **Repos:** All under <https://github.com/owasp-amass>:
  `amass`, `asset-db`, `open-asset-model`, `docs`

## :material-source-branch: Git Workflow

### Forking

Go requires code to live at its original import path.
Use `git remote` to set up your fork against the
original repository:

```bash
# 1. Fork on GitHub, then from your local clone:
git remote rename origin upstream
git remote add origin \
  git@github.com:YOUR_USERNAME/amass.git

# 2. Fetch updates from upstream
git fetch upstream
git rebase upstream/develop
```

### Branch and PR Rules

- Branch from the tip of `develop`, **not** `main`
- Do not `--force` push onto `develop` (only allowed
  when reverting a broken commit)
- Rebase your branch on top of `develop` before
  opening a PR: `git rebase upstream/develop`
- All PRs target `develop` — do not open PRs
  against `main`

## :material-code-tags: Code Standards

```bash
# Required before every commit
gofmt -w .

# Lint (same flags as CI)
golangci-lint run --timeout=60m ./...

# Build (CGO must be disabled)
CGO_ENABLED=0 go install -v ./...

# Test
CGO_ENABLED=0 go test -v ./...
CGO_ENABLED=0 go test -v \
  -run TestName ./path/to/package/...
```

Go version: **1.26.0**. `CGO_ENABLED=0` is required
for all builds and tests.

## :material-engine-outline: Engine Architecture

Understanding how the engine works is essential for
plugin development and engine contributions.

```mermaid
flowchart TD
    A["REST API\n:4000\nPOST /sessions/{id}/assets"] --> B

    B["Session Backlog\nSQLite\nclaim/ack/lease"]

    B -->|"ClaimNext(assetType)"| C

    C["Dispatcher\nrunEvents loop\nauto-scales / 5 s"]

    C -->|"consistent hash ring"| D

    D["PipelinePool\n1 pool per asset type\nFQDN/IP: 4-32\nothers: 1-4"]

    D --> E["Pipeline Stages\nPos 1-50\nFIFO/DynPool\nor Parallel"]

    E --> F["Handler Callback\nfunc(*Event) error"]

    F -->|"DispatchEvent\nnew assets"| B
```

### Key Components

**SessionManager** — each enumeration runs in an
isolated `Session`. Each session has its own database
connection, `Backlog`, `Scope`, structured logger,
and temp directory.

**Registry** — plugins register `Handler` structs at
startup, keyed by `EventType` (asset type) and
`Position` (priority 1–50). The registry builds
pipelines from these registrations.

**Dispatcher** — claims events from session backlogs,
routes them to asset-type-specific `PipelinePool`
instances via a consistent hash ring, and auto-scales
pool size every 5 seconds based on backlog depth.

**PipelinePool** — per-asset-type pool. A consistent
hash ring with 50 slots routes events to specific
instances (ensuring locality). FQDN and IPAddress
pools scale between 4 and 32 instances; all other
types scale between 1 and 4. Hard cap is
`maxInstances × 2`.

**Backlog** — durable SQLite-backed work queue with
claim/ack/lease semantics. Items progress through:
`queued → leased (in-flight) → done`. Expired leases
automatically return items to `queued`, enabling
recovery from failures.

### Handler Instance Constants

Defined in `engine/plugins/support/support.go`:

| Constant | Value | Typical Use |
| :--- | :---: | :--- |
| `MinHandlerInstances` | 4 | Low-volume handlers |
| `MidHandlerInstances` | 16 | Standard API sources |
| `HighHandlerInstances` | 32 | High-throughput |
| `MaxHandlerInstances` | 64 | DNS apex, bulk res. |

## :material-puzzle-outline: Writing a Plugin

Plugins live in `engine/plugins/` organized by
category (`api/`, `dns/`, `scrape/`, `whois/`,
`enrich/`, `brute/`, `horizontals/`,
`service_discovery/`).

### Step 1 — Implement the Plugin Interface

```go
// engine/types/registry.go
type Plugin interface {
    Name() string
    // Called once at startup; register handlers here
    Start(r Registry) error
    // Clean up goroutines and channels
    Stop()
}
```

### Step 2 — Register Handlers in Start

```go
func (p *myPlugin) Start(r et.Registry) error {
    p.log = r.Log().WithGroup("plugin").
        With("name", p.name)

    if err := r.RegisterHandler(&et.Handler{
        Plugin:   p,
        Name:     "MyPlugin-Handler",
        // Priority 1-50; lower runs first
        Position: 21,
        // Max concurrency (4/16/32/64)
        MaxInstances: support.MidHandlerInstances,
        // Asset type that triggers this handler
        EventType: oam.FQDN,
        // Asset types this handler may produce
        Transforms: []string{
            string(oam.FQDN),
        },
        Callback: p.check,
    }); err != nil {
        return err
    }

    p.log.Info("Plugin started")
    return nil
}
```

Handler field reference:

| Field | Description |
| :--- | :--- |
| `Position` | Priority 1–50; lower runs first |
| `Exclusive` | If `true`, sole handler at this pos |
| `MaxInstances` | Enables `DynamicPool` when `> 1` |
| `Transforms` | Asset types you will dispatch |
| `EventType` | OAM asset type triggering handler |
| `Callback` | `func(*Event) error` — the logic |

### Step 3 — Implement the Callback

The pattern below is drawn from
`engine/plugins/api/chaos.go`, the clearest
minimal example:

```go
func (p *myPlugin) check(e *et.Event) error {
    // 1. Type-assert the triggering asset
    fqdn, ok := e.Entity.Asset.(*oamdns.FQDN)
    if !ok {
        return errors.New(
            "failed to extract the FQDN asset")
    }

    // 2. Confirm the SLD is in scope
    if !support.HasSLDInScope(e) {
        return nil
    }

    // 3. Confirm credentials are present
    ds := e.Session.Config().
        GetDataSourceConfig(p.name)
    if ds == nil || len(ds.Creds) == 0 {
        return nil
    }

    // 4. Check TTL — skip if processed recently
    since, err := support.TTLStartTime(
        e.Session.Config(),
        string(oam.FQDN),
        string(oam.FQDN),
        p.name,
    )
    if err != nil {
        return err
    }
    if support.AssetMonitoredWithinTTL(
        e.Session, e.Entity, p.source, since) {
        return nil
    }

    // 5. Query, store, dispatch
    names := p.query(e, fqdn.Name)
    support.MarkAssetMonitored(
        e.Session, e.Entity, p.source)

    if len(names) > 0 {
        entities := support.StoreFQDNsWithSource(
            e.Session, names, p.source,
            p.name, p.name+"-Handler")
        support.ProcessFQDNsWithSource(
            e, entities, p.source)
    }
    return nil
}
```

Common support utilities
(`engine/plugins/support/support.go`):

| Function | Purpose |
| :--- | :--- |
| `HasSLDInScope(e)` | True if SLD is in scope |
| `TTLStartTime(...)` | Cache window cutoff time |
| `AssetMonitoredWithinTTL(...)` | Already done? |
| `MarkAssetMonitored(...)` | Record processing |
| `StoreFQDNsWithSource(...)` | Bulk-create FQDNs |
| `ProcessFQDNsWithSource(...)` | Dispatch events |

### Step 4 — Register in load.go

Add your constructor to the `pluginNewFuncs` slice
in `engine/plugins/load.go`:

```go
var pluginNewFuncs = []func() et.Plugin{
    // ...existing plugins...
    mypackage.NewMyPlugin,
}
```

### Step 5 — Add Credentials Template

If your plugin requires credentials, add a
commented-out entry to `resources/datasources.yaml`:

```yaml
#  - name: MyDataSource
#    ttl: 1440
#    creds:
#      account:
#        apikey: null
```

## :material-source-pull: Contributing to Other Repos

The same git workflow applies to all repos under the
`owasp-amass` organization:

- **`asset-db`** — database schema, repository
  interface, and migrations
- **`open-asset-model`** — asset, relation, and
  property type definitions in Go
- **`docs`** — this documentation site;
  `mkdocs build --strict` must pass before
  opening a PR

---

*© 2025 Jeff Foley — Licensed under Apache 2.0.*

---

## File: docs/configuration/data_sources.md

# Amass Data Sources Configuration

This document explains how to configure external data sources for Amass using the `datasources.yaml` file. Data sources provide additional information for enumeration and require credentials for access.

> **Note:** Some data sources require a paid subscription or API key to access advanced features. Using paid or premium data sources can unlock extra capabilities and may provide a better experience for the Amass engine, including more comprehensive results and faster queries.

---

## Data Sources Configuration: `datasources.yaml`

This file defines which external data sources Amass will use and their credentials.

### `global_options`

Set global defaults for all data sources.

- `minimum_ttl`: Default TTL for data source results (in minutes).

**Format:**

```yaml
# these are the global options related to data sources.
global_options:
# minimum_ttl is the default used when datasources don't specify a larger ttl value.
  minimum_ttl: 1440 #one day
```

### `datasources`

The `datasources` section is a list of external data sources Amass will use. Each entry can specify credentials, TTL, and multiple accounts. All fields are optional except for `name`.

**Supported Fields:**

- `name` (string, required): The name of the data source (e.g., `Shodan`, `CIRCL`).
- `ttl` (int, optional): Time-to-live for results from this data source, in minutes. If omitted, the global minimum TTL is used.
- `creds` (object, optional): A map of account names to credential sets. You can name the account key anything (e.g., `account`, `personal`, `work`, `default`), which helps organize multiple credential sets for the same data source. Amass will accept any key name here.
  - `<account>` (object): The account name (arbitrary string) for this credential set.
    - `apikey` (string, optional): API key for the data source.
    - `username` (string, optional): Username for authentication.
    - `password` (string, optional): Password for authentication.
    - `secret` (string, optional): Secret value for authentication (used by some APIs).

**Format:**

```yaml
datasources:
  - name: <DataSourceName>
    ttl: <minutes> # optional
    creds:
      <account>: # You can name this key anything (e.g., 'account', 'personal', 'work')
        apikey: <your_api_key>
        username: <your_username>
        password: <your_password>
        secret: <your_secret>
```

**Example:**

```yaml
datasources:
  - name: Shodan
    ttl: 10080
    creds:
      account:
        apikey: YOUR_SHODAN_API_KEY
  - name: CIRCL
    creds:
      account:
        username: YOUR_USERNAME
        password: YOUR_PASSWORD
```

---

## Tips

- Comment or uncomment sections in YAML to enable/disable features.
- Use relative paths for wordlists and data source files for portability.
- Always keep your API keys and credentials secure.

For more details, see the [example configuration files](https://github.com/owasp-amass/amass/tree/main/resources) and the [Amass documentation](https://github.com/owasp-amass/amass/wiki).

---

## Advanced Configuration Details

- **Custom Data Source Credentials:** In the data source config, [credentials are stored as a map](https://github.com/owasp-amass/amass/blob/f46cda51b3e9a701a0498eafa0e9f82a0f26e864/config/datasrcs.go#L16-L29) (`map[string]*Credentials`). This allows you to define multiple credential sets per data source in YAML, and Amass will deserialize them into Go structs. You can add new credential types as needed for custom integrations.

**Tip:** This flexibility is powerful for advanced users and developers, but be careful—unsupported or misspelled options may be silently ignored unless handled in code.

---

## File: docs/configuration/index.md

# Amass Configuration Guide

This document explains the structure and options available in Amass configuration files. Amass uses YAML files to define its scan scope, data sources, and operational options. Understanding these files will help you customize Amass for your use case.

---

## Main Configuration: `config.yaml`

The main configuration file, typically named `config.yaml`, controls the scope of your scan, scanning options, and transformation rules. Below is a breakdown of its main sections:

### `scope`

The `scope` section defines the targets and boundaries for enumeration and scanning. Each field is parsed and normalized by Amass, supporting a variety of input formats.

**Supported Fields:**

- `domains` (array of strings): Registered domain names in scope (e.g., `owasp.org`).
- `ips` (array of strings): IP addresses in scope. Supports single IPs (e.g., `192.0.2.1`), dash-ranges (e.g., `192.168.0.3-8`), and full ranges (e.g., `192.168.0.10-192.168.0.20`).
- `asns` (array of strings): Autonomous System Numbers (ASNs) in scope (e.g., `AS15169`).
- `cidrs` (array of strings): CIDR ranges in scope (e.g., `192.0.2.0/24`).
- `ports` (array of ints): Ports to be used when actively scanning services (e.g., `80`, `443`, `8080`).
- `blacklist` (array of strings): Subdomains to be excluded from results.

**Example:**

```yaml
scope:
  domains:
    - owasp.org
  ips:
    - 192.0.2.1
    - 192.168.0.10-192.168.0.20
  cidrs:
    - 192.0.2.0/24
  ports:
    - 80
    - 443
  blacklist:
    - test.owasp.org
```

### `seed`

The `seed` section is optional and allows you to specify an initial set of domains, IPs, CIDRs, ASNs, ports, and blacklist entries. If provided, Amass uses `seed` to initialize the scan scope. If the `scope` section is not specified, the `seed` will also be used as the scope. Otherwise, `seed` and `scope` are treated as separate entities, and both can be specified independently.

**Supported Fields:**

- `domains` (array of strings): Registered domain names to seed the scan (e.g., `owasp.org`).
- `ips` (array of strings): IP addresses to seed the scan. Supports single IPs, dash-ranges, and full ranges.
- `asns` (array of strings): Autonomous System Numbers to seed the scan (e.g., `AS15169`).
- `cidrs` (array of strings): CIDR ranges to seed the scan (e.g., `192.0.2.0/24`).
- `ports` (array of ints): Ports to be used when actively scanning services.
- `blacklist` (array of strings): Subdomains to be excluded from results.

**Example:**

```yaml
seed:
  domains:
    - owasp.org
  ips:
    - 192.0.2.1
    - 192.168.0.10-192.168.0.20
  cidrs:
    - 192.0.2.0/24
  ports:
    - 80
    - 443
  blacklist:
    - test.owasp.org
```

### `options`

The `options` section controls data sources, brute force, alterations, engine/database connections, and default values for transforms. Each option is handled by dedicated loader functions in Amass, and must match the expected type.

**Supported Options:**

- `datasources` (string): Path to the data sources YAML file. Required for external data source integration.
  - For information about the format and fields in this file, see the [Data Sources Configuration](datasources.md) section.
- `engine` (string): URL for the Amass engine API (e.g., `http://127.0.0.1:4000/graphql`).
- `database` (string): Database connection string (e.g., `bolt://neo4j:amass4OWASP@neo4j:7687/neo4j` or `postgres://amass:amass4OWASP@assetdb:5432/assetdb`).
- `bruteforce` (object):
  - `enabled` (bool): Enable or disable brute force subdomain enumeration.
  - `wordlists` (array of strings): List of wordlist files for brute forcing. Each file can be plain text or gzipped.
- `alterations` (object):
  - `enabled` (bool): Enable or disable name alteration techniques.
  - `wordlists` (array of strings): List of wordlist files for alterations. Each file can be plain text or gzipped.
- `default_transform_values` (object):
  - `ttl` (int): Default time-to-live for transform results (in minutes).
  - `confidence` (int): Default confidence value (percentage).
  - `priority` (int): Default priority value.
- `active` (bool): Enable active enumeration (e.g., zone transfers, certificate checks).
- `rigid_boundaries` (bool): Enforce strict scope boundaries for enumeration.

> **Notice:** Relative paths are relative to the config file location.

**Example:**

```yaml
options:
  datasources: "./datasources.yaml"
  engine: "http://127.0.0.1:4000/graphql"
  database: "bolt://neo4j:amass4OWASP@neo4j:7687/neo4j"
  bruteforce:
    enabled: false
    wordlists:
      - "./namelist.txt"
  alterations:
    enabled: false
    wordlists:
      - "./alterations.txt"
  default_transform_values:
    ttl: 1440
    confidence: 50
    priority: 5
  active: true
  rigid_boundaries: false
```

#### Environment Variables for Engine and Database

Amass supports configuring the Engine API and Database connection using environment variables as an alternative to specifying the `engine` or `database` URI in the `options` section. **If you specify a URI in the config file for either `engine` or `database`, Amass will NOT use the corresponding environment variables for that object.** There is no merging of environment variables and config values per object—it's either one or the other for each.

**Engine API Environment Variables:**

- `AMASS_ENGINE_USER`: Username for Engine API authentication
- `AMASS_ENGINE_PASSWORD`: Password for Engine API authentication
- `AMASS_ENGINE_HOST`: Host for the Engine API (default: `localhost`)
- `AMASS_ENGINE_PORT`: Port for the Engine API (default: `4000`)
- `AMASS_ENGINE_PATH`: Path for the Engine API (default: `graphql`)
- `AMASS_ENGINE_SCHEME`: Scheme for the Engine API (`http` or `https`, default: `http`)

**Database Environment Variables:**

- `AMASS_DB_USER`: Username for the database
- `AMASS_DB_PASSWORD`: Password for the database
- `AMASS_DB_HOST`: Host for the database (default: `localhost`)
- `AMASS_DB_PORT`: Port for the database (default: `5432`)
- `AMASS_DB_NAME`: Database name (default: `assetdb`)

**Important:**

- If you set the `engine` or `database` URI in the config file, Amass will use only that value and ignore the related environment variables for that object.
- You cannot mix or merge environment variable values with a URI specified in the config for either the Engine API or the Database. For now, it is one or the other, per object.

---

### `transformations`

The `transformations` section controls how assets should be proccessed, how long results from specific transforms are considered valid (TTL), and the minimum confidence level. This can be customized per entity and transform type. 

- Each transformation key is parsed as `<from>-><to>`. Amass validates these against the Open Asset Model (OAM) and ensures there are no conflicts (e.g., you can't have both `FQDN->ALL` and `FQDN->none`). You can set `ttl`, `confidence`, `priority`, and `exclude` for each transformation.
- If a transformation omits `ttl`, `confidence`, or `priority`, Amass uses the values from `default_transform_values` in `options`.
- You can add or modify transformations in Go by editing the `Config.Transformations` map and calling validation methods.

**Supported Format:**
  
- `<from>-><to>` (object):
  - `ttl` (int, optional): Time-to-live for this transformation result, in minutes.
  - `confidence` (int, optional): Confidence value for this transformation (percentage).
  - `priority` (int, optional): Priority value for this transformation.
  - `exclude` (array of strings, optional): List of transforms to exclude.
In this example, the TTL and confidence settings will only apply when the transformation is handled by the plugin named `myplugin`.

> **Tip:** Use `ALL` as the transform to apply the rule to all transforms for a given entity.

**Example:**

```yaml
transformations:
  FQDN->DNS:
    ttl: 1440
  FQDN->DomainRecord:
    ttl: 43200
  Account->ALL:
    ttl: 1080
```

**Granular Example (per plugin):**
[Amass handles plugin level granularity](https://github.com/owasp-amass/amass/blob/main/engine/plugins/support/support.go#L91C1-L105C1). You can specify transformation rules as granularly as you want—even down to a single plugin. If you use the plugin's name as the `<to>` in your `<from>-><to>` transformation key, Amass will apply that rule only for that plugin. This enables highly targeted configuration and fine-grained control over how results are handled for each plugin.

```yaml
transformations:
  FQDN->myplugin:
    ttl: 60
    confidence: 90
```

---

For more details, see the [example configuration files](https://github.com/owasp-amass/amass/tree/main/resources) and the [Amass documentation](https://github.com/owasp-amass/amass/wiki).

---

## How Amass Loads Configuration

Amass supports flexible configuration loading. allowing you to control settings via YAML files, environment variables, and programmatic overrides.

### Configuration Loading Order

1. **Explicit file**: If you specify a config file path (e.g., with `-config`), it is loaded first.
2. **Environment variable**: If the `AMASS_CONFIG` environment variable is set, its path is used.
3. **Default locations**: Amass looks for `config.yaml` in the output directory (usually `~/.config/amass/`) or `/etc/amass/config.yaml` on Linux.

### Programmatic Configuration

If you use Amass as a Go library, you can construct a `Config` object directly in code, set fields, and call methods like `LoadSettings()` or `AcquireConfig()` to merge YAML and programmatic settings.

---

## Advanced Configuration Details

Amass uses Go interfaces and flexible types in its configuration structs to allow for advanced and custom settings:

- **Flexible Options:** [The `options` section in YAML is loaded into a `map[string]interface{}`](https://github.com/owasp-amass/amass/blob/f46cda51b3e9a701a0498eafa0e9f82a0f26e864/config/config.go#L67). This means you can add arbitrary keys and values—even ones not officially documented. If Amass has a loader for your option (like `active`, `rigid_boundaries`, or future features), it will be parsed and used (e.g., check out the code for [loading the `database` option](https://github.com/owasp-amass/amass/blob/f46cda51b3e9a701a0498eafa0e9f82a0f26e864/config/graphdb.go#L40-L57)). Otherwise, you can access these values programmatically if you use Amass as a library.

- **Scope Parsing:** The `Scope` struct uses fields like `PortsRaw []interface{}` so you can specify ports as single values, lists, or ranges in YAML. Amass normalizes these internally, so you have flexibility in how you define scope.

- **Loader Functions:** After deserialization, [Amass runs a series of loader functions to process, validate, and normalize settings](https://github.com/owasp-amass/amass/blob/f46cda51b3e9a701a0498eafa0e9f82a0f26e864/config/config.go#L292-L302). This means you can experiment with new options or override behaviors by adding new keys to your YAML, as long as you handle them in your own fork or via the Go API.

- **Programmatic Access:** If you use Amass as a Go library, you can read, modify, or extend the config structs directly, including any custom or experimental options you add to your YAML.

> **Tip:** This flexibility is powerful for advanced users and developers, but be careful—unsupported or misspelled options may be silently ignored unless handled in code.

---

## File: docs/configuration/transformations.md

# :material-auto-fix: Asset Transformations Configuration

The `transformations` section of the OWASP Amass `config.yaml` file is one of the most powerful parts of the collection engine. It controls **how data flows through the system**, defines **which types of assets can be transformed into others**, and sets **constraints like freshness (TTL), trustworthiness (confidence), and urgency (priority)** on those transformations.

This section empowers users to **customize and optimize their data collection workflows** based on their goals, risk tolerance, and update requirements.

## :material-help-circle-outline: What Are Transformations?

In Amass, the data collection process is modeled as a pipeline of **asset transformations**. Each asset observed (like an IP address, domain, ASN, etc.) can trigger **handlers**, which attempt to enrich, correlate, or expand on that asset type by transforming it into new assets.

For example:
- A discovered `FQDN` might trigger a handler to look up its DNS records (`FQDN -> DNS`)
- A known `AutonomousSystem` could be transformed into its RDAP metadata (`AutonomousSystem -> RDAP`)
- A `Product` seen on a web service might lead to discovery of its `ProductRelease` metadata

The `transformations` section defines:
- **Which transformations are allowed**
- **How frequently** each transformation should be retried (`ttl`)
- **How much confidence** the system should have in the results (`confidence`)
- **How important** the transformation is (`priority`)

## :material-cog-outline: Configuration Overview

Here's the structure of a typical configuration:

```yaml
options:
  default_transform_values:
    ttl: 1440         # in minutes (default is 1 day)
    confidence: 50    # default confidence threshold (0–100%)
    priority: 5       # default priority level (1=low, 10=high)

transformations:
  FQDN->DNS:
    ttl: 1440
  AutonomousSystem->RDAP:
    ttl: 43200        # 30 days
  Identifier->GLEIF:
    ttl: 43200
  Product->ALL:
    ttl: 10080        # 7 days
```

### :gear: `default_transform_values`

This section defines fallback values used when no custom values are given for a transformation.

| Key          | Type    | Description                                                                                                                 |
| ------------ | ------- | --------------------------------------------------------------------------------------------------------------------------- |
| `ttl`        | integer | Time-to-live (in minutes) for data freshness. If expired, the data will be fetched again from the source instead of reused. |
| `confidence` | integer | Minimum confidence (0–100%) required to accept the result of a transformation.                                              |
| `priority`   | integer | Priority score (1=lowest, 10=highest) that may influence queue ordering in some future extensions.                          |

## :hammer_and_wrench: Defining Transformations

Each transformation follows this format:

```yaml
-><TargetAssetType|Plugin Name>:
  ttl:            # Optional override of default
  confidence:     # Optional override
  priority:       # Optional override
```

Use `->ALL` as a wildcard to enable **all available transformations from a given source asset type**.

Example:

```yaml
FQDN->ALL:
```

This enables all known FQDN transformations (e.g., `FQDN->IPAddress`, `FQDN->DomainRecord`, etc.).

## :page_facing_up: Example Config Breakdown

```yaml
transformations:
  FQDN->DNS:
    ttl: 1440  # 1 day
  FQDN->DomainRecord:
    ttl: 43200  # 30 days
  IPAddress->ALL:
  TLSCertificate->ALL:
    ttl: 10080  # 7 days
```

### Explanation:

* **FQDN->DNS**: Amass will try to resolve DNS for fully qualified domain names once per day.
* **FQDN->DomainRecord**: Domain ownership records are more stable, so these are refreshed every 30 days.
* **IPAddress->ALL**: All available transformations for IPs are enabled (e.g., geolocation, RDAP, reverse DNS).
* **TLSCertificate->ALL**: Certificates fetched from services are checked weekly.

## :material-refresh: What is `ttl`?

`ttl` (Time To Live) controls **how often a transformation can be retried** from the original source:

* If the data is **still fresh** (within TTL), Amass will use the previously stored result from the database.
* If the TTL has **expired**, the handler will attempt to **re-run the transformation**, such as querying the data source again.

This ensures the system avoids unnecessary queries and controls bandwidth/load.

## :material-shield-check: What is `confidence`?

`confidence` helps Amass **filter out noisy or speculative results**. Some plugins or handlers may return results with associated confidence scores.

* If a handler returns a transformation with `confidence: 40`, and your threshold is `50`, **it will be ignored**.
* Use this to reduce false positives or to tune behavior in environments where high data quality is crucial.

## :material-star: What is `priority`?

The `priority` value is a **relative score (1–10)** that can help inform which transformations are more important. While not strictly enforced in the engine today, this allows **future prioritization of more urgent or valuable tasks**—like scanning attack surfaces or refreshing high-risk domains.

## :material-clipboard-list-outline: Tips and Best Practices

* ✅ Use `->ALL` to simplify enabling all known transformations for an asset type.
* ✅ Use higher TTLs (e.g., 30+ days) for records that rarely change (e.g., `RDAP`, `GLEIF`, `DomainRecord`).
* ⚠️ Keep `ttl` low (e.g., 60–1440 min) for time-sensitive records like DNS, services, or IP geolocation.
* ✅ Set `confidence` thresholds higher (e.g., 70–90) in production pipelines where trust is critical.
* ✅ Consider adjusting `priority` for critical infrastructure or high-value assets.

## :material-rocket-launch: Summary

The `transformations` section of the Amass configuration lets users **shape the intelligence collection process**, optimize for **freshness vs. efficiency**, and **control data quality** through TTLs and confidence scoring.

It is a key part of how Amass turns passive and active discoveries into structured asset graphs that can drive attack surface monitoring, red teaming, or asset attribution.

---

For a full list of supported asset types, refer to the [Open Asset Model documentation](../open_asset_model/index.md).

*© 2025 Jeff Foley — Licensed under Apache 2.0.*

---

## File: docs/asset_db/index.md

# :simple-owasp: Asset Database

The **Asset Database** is the persistent store for all
entities and relations discovered during an Amass
enumeration. It is backed by the
[`owasp-amass/asset-db`][asset-db] library, which
provides a unified repository interface over multiple
storage backends.

[asset-db]: https://github.com/owasp-amass/asset-db

Every Amass **Session** receives its own database
connection. Assets are stored as typed entities
following the [Open Asset Model][oam]; the connections
between them are stored as directed edges (relations)
with optional properties attached to both entities
and edges.

[oam]: ../open_asset_model/index.md

## :material-database: Supported Backends

| Backend | Connection String | Best For |
| :--- | :--- | :--- |
| **SQLite** | Auto-created | Dev, quick scans |
| **PostgreSQL** | `postgres://u:p@host:5432/db` | Production |
| **Neo4j** | `neo4j://u:p@host:7687/db` | Graph queries |

If no database is configured, Amass automatically
creates a local SQLite database in its configuration
directory.

## :material-cog: Configuring the Database

Set the connection string in `config.yaml`:

```yaml
options:
  database: "postgres://amass:amass4OWASP@host:5432/db"
```

You can also configure the database via environment
variables, which take precedence over the config file:

```text
AMASS_DB_USER      database username
AMASS_DB_PASSWORD  database password
AMASS_DB_HOST      database host
AMASS_DB_PORT      database port
AMASS_DB_NAME      database name
```

## :material-graph-outline: How It Fits Together

The asset database sits at the center of the data
model. The Amass engine writes every discovered asset
into the database during enumeration. Assets are
stored once and referenced many times — the same IP
address discovered by multiple data sources is a
single entity with multiple `SourceProperty`
annotations.

Relations between assets (e.g., an FQDN resolving to
an IP address via a `dns_record` relation) are stored
as directed edges. This graph structure is what makes
the [Triples Query Language](triples.md) possible.

## :material-page-next: Next Steps

- [PostgreSQL Setup](postgres.md) — production
  PostgreSQL setup guide
- [Triples Query Language](triples.md) — traverse
  the asset graph with subject-predicate-object queries

---

*© 2025 Jeff Foley — Licensed under Apache 2.0.*

---

## File: docs/asset_db/postgres.md

# Setting Up a PostgreSQL Database for OWASP Amass

The OWASP Amass framework can store collected data in a PostgreSQL database. This page walks you through the recommended setup process, including environment variables, database initialization, and configuration in your `config.yaml` file.

> **Note:** These instructions assume PostgreSQL is already installed and running on your system (e.g., `localhost:5432`). You’ll need access to a user with sufficient privileges (typically `postgres`).

## 1. Define Environment Variables

Before running the setup commands, export the following environment variables to define your database, user, and passwords. These values will be used in the setup process and your Amass configuration.

```bash
export POSTGRES_USER=postgres
export POSTGRES_PASSWORD=postgres
export AMASS_DB=assetdb
export AMASS_USER=amass
export AMASS_PASSWORD=amass4OWASP
```

??? info "Secrets Management"
    Consider storing these in a `.env` file and loading them with `source .env` to avoid retyping. Never commit secrets to version control.

## 2. Create the Amass Database and User

Run the following commands in your shell to initialize the database and create a dedicated user for Amass. This uses the `psql` CLI with inline SQL for automation.

```bash
# Add single quotes around the password to handle special characters
export TEMPPASS="'$AMASS_PASSWORD'"

# Create the database and user
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" <<-EOSQL
    \getenv assetdb AMASS_DB
    \getenv username AMASS_USER
    \getenv password TEMPPASS

    CREATE DATABASE :assetdb;
    ALTER DATABASE :assetdb SET timezone TO 'UTC';
    CREATE USER :username WITH PASSWORD :password;
EOSQL
```

This will:

* Create the `assetdb` database
* Set its default timezone to UTC (recommended for consistency)
* Create a new user (`amass`) with the specified password

## 3. Enable Extensions and Grant Privileges

Next, connect to the new database and enable the required PostgreSQL extension and assign privileges to the Amass user.

```bash
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$AMASS_DB" <<-EOSQL
    \getenv username AMASS_USER

    CREATE EXTENSION pg_trgm SCHEMA public;

    GRANT USAGE ON SCHEMA public TO :username;
    GRANT CREATE ON SCHEMA public TO :username;
    GRANT ALL ON ALL TABLES IN SCHEMA public TO :username;
EOSQL
```

This will:

* Enable the `pg_trgm` extension (used by Amass for efficient fuzzy string matching)
* Grant the necessary privileges for Amass to create and manage data within the `public` schema

## 4. Update Your Amass Configuration

Once your database is set up, update your Amass `config.yaml` file with the connection string:

```yaml
options:
  # Be sure to replace the credentials with values matching your environment
  database: "postgres://amass:amass4OWASP@127.0.0.1:5432/assetdb"
```

??? info "Security Reminder"
    Avoid committing passwords to source control. Where possible, consider injecting the connection string using an environment variable (e.g., `${AMASS_DB_URI}`).

## 5. Test the Connection

You can test whether the Amass framework is successfully connecting to your PostgreSQL database by running a standard enumeration command:

```bash
amass enum -config config.yaml
```

If the configuration is correct, the collected data will be stored in the PostgreSQL backend you configured.

## ✅ You're Done!

Amass is now ready to store data in your PostgreSQL database. This enables you to persist, analyze, and query discovered assets using SQL or integrate with other tooling and dashboards.

## Troubleshooting Tips

* **Connection Refused?** Ensure PostgreSQL is listening on `127.0.0.1:5432` and that the database server is running.
* **Authentication Failed?** Double-check your environment variable values, especially the user and password.
* **Extension Errors?** Make sure the `pg_trgm` extension is available and installed. You can check with `\dx` in `psql`.

## See Also

* [Amass Configuration](../configuration/index.md)
* [PostgreSQL Documentation](https://www.postgresql.org/docs/current/index.html)
* [PostgreSQL `pg_trgm` Extension Docs](https://www.postgresql.org/docs/current/pgtrgm.html)
* [Managing Environment Variables Securely](https://direnv.net/)

---

## File: docs/asset_db/triples.md

# Triples Query Language

The **Triples Query Language** allows users of the OWASP Amass framework to request data from the Asset Database using the **OWASP Open Asset Model**. This query language enables traversals across the graph, where each triple describes a directed edge between two nodes.

A **triple** is a traversal path that describes a step in a graph walk, or a "hop" in your graph between two nodes. Each triple consists of a **subject**, a **predicate**, and an **object**. The subject is the node being queried, the predicate describes the relationship, and the object is the target node. Results from the previous triple can serve as subjects for subsequent triples, enabling complex queries across the graph.

## Syntax Overview

A single **triple** follows the format for outgoing relations from the subject:

```
<subject> -  -> <object>
```

Or, the arrow pointing in the other direction for incoming relations to the subject:

```
<subject> <-  - <object>
```

- **Subject**: The starting node of the traversal.
- **Predicate**: Describes the relationship.
- **Object**: The ending node of the traversal.

### Components of a Triple

Each node and predicate in the triple has the following format:

```
<type:label,since:DATE,prop:[type:label,atrribute:value]>


```

Here is an example of 

- **type**: The type of the node (e.g., `fqdn` or `ipaddress`).
- **label**: The specific value associated with the node (e.g. `dns_record`).
- **since**: An optional filter specifying a date to limit results after a certain point.
- **prop**: Optional properties for the node, such as additional attributes or metadata.
- **attributes**: Optional fields from the data type used for filtering (e.g. `header.rr_type:#/1|28/#`).

### Supported Query Elements

- **Constant Values**: Specific, static values for filtering (e.g., `fqdn:owasp.org`).
- **Wildcard ('*')**: A wildcard character that can match any value (e.g., `ipaddress:*`).
- **Regular Expressions ('#//#')**: A regular expression for more specific filtering (e.g. `#/.*google.*/#`).

## Example Queries

### 1. From Root Domain Name to IP Addresses

This query retrieves all IP addresses associated with the root domain name `owasp.org`, starting from the domain and considering DNS records since July 1st, 2025.

```
<fqdn:owasp.org> - <*:dns_record,since:2025-07-01> -> 
```

- **Subject**: `<fqdn:owasp.org>`
- **Predicate**: `<*:dns_record,since:2025-07-01>`
- **Object**: ``

### 2. Root Domain Name to Subdomains

This query retrieves all subdomains of the root domain `owasp.org`, starting from the domain and considering the relationship to nodes since July 1st, 2025.

```
<fqdn:owasp.org> - <*:node,since:2025-07-01> -> <*>
```

- **Subject**: `<fqdn:owasp.org>`
- **Predicate**: `<*:node,since:2025-07-01>`
- **Object**: `<*>`

The object can simply specify the wildcard character, since the 'node' relation outcoming from an FQDN must connect with another FQDN.

### 3. Subdomain to IP Address

This query retrieves IP addresses for all subdomain names (e.g. `subdomain.owasp.org`), acquired from a previous triple, and their DNS records since July 1st, 2025.

```
<fqdn:#/.*owasp.org/#> - <*:dns_record,since:2025-07-01> -> 
```

- **Subject**: `<fqdn:#/.*owasp.org/#>`
- **Predicate**: `<*:dns_record,since:2025-07-01>`
- **Object**: ``

## Filtering with Regular Expressions

You can use regular expressions to filter the query results. For example, if you want to query IP addresses associated with domain names that match a specific pattern, you can use:

```
<fqdn:google.com> - <*:dns_record> -> 
```

- **Subject**: `<fqdn:google.com>`
- **Predicate**: `<*:dns_record>`
- **Object**: ``

This query retrieves IP addresses for `google.com` where the IP address matches the `192.168.*` range.

## Traversing Multiple Steps

Triples allow for multiple traversals in a query. You can chain multiple triples to traverse from one node to another through various relationships. For example:

```
amass assoc -t1 '<fqdn:owasp.org> - <*:node> -> <*>' -t2 '<fqdn:*> - <*:dns_record> -> '
```

Here, the first triple retrieves all nodes related to the root domain `owasp.org`, and the second triple retrieves IP addresses associated with those nodes. The entire walk that traverses all triples, and all related properties, is provided in the JSON output.

## Conclusion

The **Triples Query Language** is a powerful way to interact with the OWASP Asset Database and extract relevant data from the Open Asset Model. By using this language, users can perform flexible, precise queries to navigate the complex relationships between assets, making it a valuable tool for asset discovery and attack surface management.

## See Also

* [Asset Database](./index.md)
* [Setting Up PostgreSQL](./postgres.md)