# Technical Documentation: anchore/grype
> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/anchore/grype)
### Overview (`README.md`)
# Grype
**A vulnerability scanner for container images and filesystems.**
## Features
- Scan **container images**, **filesystems**, and **SBOMs** for known vulnerabilities (see the docs for a full list of [supported scan targets](https://oss.anchore.com/docs/guides/vulnerability/scan-targets/))
- Supports major OS package ecosystems (Alpine, Debian, Ubuntu, RHEL, Oracle Linux, Amazon Linux, and [more](https://oss.anchore.com/docs/capabilities/all-os/))
- Supports language-specific packages (Ruby, Java, JavaScript, Python, .NET, Go, PHP, Rust, and [more](https://oss.anchore.com/docs/capabilities/all-packages/))
- Supports Docker, OCI, and [Singularity](https://github.com/sylabs/singularity) image formats
- Threat & risk prioritization with **EPSS**, **KEV**, and **risk scoring** (see [interpreting the results docs](https://oss.anchore.com/docs/guides/vulnerability/interpreting-results/))
- [OpenVEX](https://github.com/openvex) support for filtering and augmenting scan results
> [!TIP]
> New to Grype? Check out the [Getting Started guide](https://oss.anchore.com/docs/guides/vulnerability/getting-started/) for a walkthrough!
## Installation
The quickest way to get up and going:
```bash
curl -sSfL https://get.anchore.io/grype | sudo sh -s -- -b /usr/local/bin
```
> [!TIP]
> See [Installation docs](https://oss.anchore.com/docs/installation/grype/) for more ways to get Grype, including Homebrew, Docker, Chocolatey, MacPorts, and more!
## The basics
Scan a container image or directory for vulnerabilities:
```bash
# container image
grype alpine:latest
# directory
grype ./my-project
```
### Local containers-storage images
On Linux, grype can scan images from a local containers-storage store (e.g. built with Buildah or Podman) when asked explicitly; plain image references never look there:
```bash
grype --from containers-storage localhost/myimage:latest
# rootless stores must be read from inside the builder's user namespace
podman unshare grype --from containers-storage localhost/myimage:latest
```
This is included in the Linux release binaries. When building from source, add `-tags containers_image_openpgp,exclude_graphdriver_btrfs`.
Scan an SBOM for even faster vulnerability detection:
```bash
# scan a Syft SBOM
grype sbom:./sbom.json
# pipe an SBOM into Grype
cat ./sbom.json | grype
```
> [!TIP]
> Check out the [Getting Started guide](https://oss.anchore.com/docs/guides/vulnerability/getting-started/) to explore all of the capabilities and features.
>
> Want to know all of the ins-and-outs of Grype? Check out the [CLI docs](https://oss.anchore.com/docs/reference/grype/cli/) and [configuration docs](https://oss.anchore.com/docs/reference/grype/configuration/).
## Contributing
We encourage users to help make these tools better by [submitting issues](https://github.com/anchore/grype/issues) when you find a bug or want a new feature.
Check out our [contributing overview](https://oss.anchore.com/docs/contributing/) and [developer-specific documentation](https://oss.anchore.com/docs/contributing/grype/) if you are interested in providing code contributions.
Grype development is sponsored by Anchore, and is released under the Apache-2.0 License.
The Grype logo by Anchore is licensed under CC BY 4.0
For commercial support options with Syft or Grype, please [contact Anchore](https://get.anchore.com/contact/).
## Come talk to us!
The Grype Team holds regular community meetings online. All are welcome to join to bring topics for discussion.
- Check the [calendar](https://calendar.google.com/calendar/u/0/r?cid=Y182OTM4dGt0MjRtajI0NnNzOThiaGtnM29qNEBncm91cC5jYWxlbmRhci5nb29nbGUuY29t) for the next meeting date.
- Add items to the [agenda](https://docs.google.com/document/d/1ZtSAa6fj2a6KRWviTn3WoJm09edvrNUp4Iz_dOjjyY8/edit?usp=sharing) (join [this group](https://groups.google.com/g/anchore-oss-community) for write access to the [agenda](https://docs.google.com/document/d/1ZtSAa6fj2a6KRWviTn3WoJm09edvrNUp4Iz_dOjjyY8/edit?usp=sharing))
- See you there!
---
### Overview (`templates/README.md`)
# Grype Templates
This folder contains a set of "helper" go templates you can use for your own reports.
Please feel free to extend and/or update the templates for your needs, be sure to contribute back into this folder any new templates!
Current templates:
.
├── README.md
├── html.tmpl
├── junit.tmpl
├── csv.tmpl
└── table.tmpl
## Table
This template mimics the "default" table output of Grype, there are some drawbacks to using the template vs the native output such as:
- unsorted
- duplicate rows
- no (wont-fix) logic
As you can see from the above list, it's not perfect but it's a start.
## HTML
Produces a nice html template with a dynamic table using datatables.js.
You can also modify the templating filter to limit the output to a subset.
Default includes all
```
{{- if or (eq $vuln.Vulnerability.Severity "Critical") (eq $vuln.Vulnerability.Severity "High") (eq $vuln.Vulnerability.Severity "Medium") (eq $vuln.Vulnerability.Severity "Low") (eq $vuln.Vulnerability.Severity "Unknown") }}
```
We can limit it to only Critical, High, and Medium by editing the filter as follows
```
{{- if or (eq $vuln.Vulnerability.Severity "Critical") (eq $vuln.Vulnerability.Severity "High") (eq $vuln.Vulnerability.Severity "Medium") }}
```
---
### Overview (`internal/dbtest/README.md`)
# dbtest - Database Testing Utilities
This package provides utilities for building and testing grype vulnerability databases from fixture data, along with fluent assertion helpers that decouple test code from internal API shapes.
## Quick Start
### Using Fixtures in Tests
```go
func TestVulnerabilityMatching(t *testing.T) {
// use a fixture from your package's testdata directory
dbtest.DBs(t, "my-fixture").Run(func(t *testing.T, db *dbtest.DB) {
// db implements vulnerability.Provider
vulns, err := db.FindVulnerabilities(...)
})
}
func TestWithSharedFixture(t *testing.T) {
// use a fixture from internal/dbtest/testdata/shared/
dbtest.SharedDBs(t, "all").Run(func(t *testing.T, db *dbtest.DB) {
// db.Match returns a FindingsAssertion for fluent assertions
db.Match(t, matcher, pkg).
SelectMatch("CVE-2024-1234").
SelectDetailByType(match.ExactDirectMatch).
AsDistroSearch()
})
}
```
### Filtering Fixtures at Test Time
```go
// only include specific CVEs from a larger fixture
dbtest.SharedDBs(t, "all").
SelectOnly("CVE-2024-1234", "CVE-2024-5678").
Run(func(t *testing.T, db *dbtest.DB) {
// db only contains the selected CVEs
})
```
## Motivation
Testing grype's vulnerability matching requires realistic database fixtures. However, real vunnel caches can contain millions of records, making them impractical for test fixtures. This package solves several problems:
1. **Focused fixtures**: Extract only the specific CVEs or namespaces needed for a test scenario
2. **Deterministic tests**: Create reproducible fixtures from real vunnel data
3. **Easy maintenance**: Append new records to existing fixtures as test cases evolve
4. **Cross-package sharing**: Share fixtures between test packages via `SharedDBs()`
5. **Fixture provenance**: Track how fixtures were created and regenerate them when needed
6. **API-agnostic assertions**: Decouple test assertions from internal data structures to survive refactors
## Go API
### Building Test Databases
The `Builder` type provides a fluent API for building test databases from fixtures:
```go
// DBs looks for testdata/ relative to the calling test file
func DBs(t *testing.T, fixtureName string) *Builder
// SharedDBs looks for fixtures in internal/dbtest/testdata/shared/
func SharedDBs(t *testing.T, fixtureName string) *Builder
// SelectOnly filters which records are included (patterns combined with OR)
func (b *Builder) SelectOnly(patterns ...string) *Builder
// Run executes a test function for each schema version
func (b *Builder) Run(fn func(t *testing.T, db *DB))
// Build returns databases without running them through t.Run
func (b *Builder) Build(schemas ...int) []*DB
```
### The DB Type
`DB` wraps a `vulnerability.Provider` with test metadata:
```go
type DB struct {
Name string // e.g., "v6"
SchemaVersion int
Path string
}
// implements vulnerability.Provider
func (db *DB) FindVulnerabilities(criteria ...vulnerability.Criteria) ([]vulnerability.Vulnerability, error)
func (db *DB) VulnerabilityMetadata(ref vulnerability.Reference) (*vulnerability.Metadata, error)
func (db *DB) PackageSearchNames(p grypePkg.Package) []string
// runs matcher and returns fluent assertion chain
func (db *DB) Match(t *testing.T, matcher Matcher, p grypePkg.Package) *FindingsAssertion
```
### Fluent Assertions
The assertion helpers provide a **string-based, API-agnostic** way to validate match results. This design has several benefits:
- **Survives refactors**: Tests don't break when internal struct shapes change (e.g., grype v1 API changes)
- **Removes boilerplate**: No need to manually iterate matches/details or check types
- **Completeness checking**: By default, tests fail if any matches or details are not asserted
- **Readable tests**: Fluent chains clearly express what's being validated
This approach is similar to syft's `pkgtest` helpers - abstracting test assertions from implementation details.
#### Basic Usage
```go
// match and assert in one fluent chain
db.Match(t, &matcher, pkg).
SelectMatch("CVE-2024-1234").
SelectDetailByType(match.ExactDirectMatch).
AsDistroSearch("< 1.0.0") // validates constraint
// or use AssertFindings directly with pre-existing matches
dbtest.AssertFindings(t, matches, pkg).
HasCount(2).
OnlyHasVulnerabilities("CVE-2024-1234", "CVE-2024-5678")
```
#### Completeness Checking
By default, assertions require that **all matches and details are asserted**. This catches cases where a matcher returns unexpected results:
```go
// this will fail if there are matches other than CVE-2024-1234
db.Match(t, &matcher, pkg).
SelectMatch("CVE-2024-1234").
SelectDetailByType().
AsDistroSearch()
```
`SkipCompleteness()` inverts the contract: it asserts the chain is **intentionally partial**, and fails if every match and detail ended up being asserted on anyway. This keeps `SkipCompleteness` calls from rotting in tests that have grown into being exhaustive — drop the call once your chain covers everything:
```go
// passes: only CVE-2024-1234 was asserted, others left alone
db.Match(t, &matcher, pkg).SkipCompleteness().
ContainsVulnerabilities("CVE-2024-1234")
// fails: SkipCompleteness was called but the chain ended up exhaustive,
// so the SkipCompleteness call is dead weight and should be removed
db.Match(t, &matcher, pkg).SkipCompleteness().
SelectMatch("CVE-2024-1234").
SelectDetailByType().
AsDistroSearch()
```
The same inversion applies to `Ignores().SkipCompleteness()`.
#### Detail Assertions
After selecting a match, drill into details by type or search parameters:
```go
// select by match type
findings.SelectMatch("CVE-2024-1234").
SelectDetailByType(match.ExactDirectMatch).
AsDistroSearch()
// select by distro (when multiple distro details exist)
findings.SelectMatch("CVE-2024-1234").
SelectDetailByDistro("debian", "11", "< 1.0.0")
// select by CPE
findings.SelectMatch("CVE-2024-1234").
SelectDetailByCPE("cpe:2.3:a:vendor:product:1.0:*:*:*:*:*:*:*").
FoundCPEs("cpe:2.3:a:vendor:product:*:*:*:*:*:*:*:*")
// select by ecosystem/language
findings.SelectMatch("CVE-2024-1234").
SelectDetailByEcosystem("python", "< 2.0.0")
```
#### Available Assertions
| Method | Description |
|--------|-------------|
| `IsEmpty()` | Assert the matcher produced nothing — no matches AND no ignore filters |
| `ContainsVulnerabilities(ids...)` | Assert these CVEs are present (others may exist) |
| `OnlyHasVulnerabilities(ids...)` | Assert exactly these CVEs and no others |
| `DoesNotHaveAnyVulnerabilities(ids...)` | Assert these CVEs are not present |
| `SelectMatch(id)` | Select a specific match for detail assertions; fails if multiple matches share the ID |
| `SelectMatches(id)` | Select the subset of matches with a vulnerability ID; chain `WithDetailType` to disambiguate |
| `HasMatchType(type)` | Assert at least one detail has this match type |
| `HasOnlyMatchTypes(types...)` | Assert all details have one of these types |
| `SkipCompleteness()` | Assert this chain is intentionally partial; fails if the chain is actually exhaustive |
`IsEmpty()` covers the **whole result** — both matches and ignore filters. For tests that expect zero matches but ≥1 ignore filter, just skip `IsEmpty()` and assert on the ignores via `Ignores()`; matches-side completeness checking will still enforce that no matches were produced.
For ignore filters, `Ignores().SelectRelatedPackageIgnore(reason, vulnID)` selects one ignore for fine-grained assertions; `Ignores().SelectRelatedPackageIgnores(reason, vulnIDs...)` selects a batch sharing a reason and fans `ForPackage`/`WithRelationshipType` over all of them — useful for AlmaLinux alias unwinding where one ALSA emits ignores for itself plus each aliased CVE.
**Ignore-completeness is on by default** — every matcher-emitted ignore must be asserted on, even by tests that never call `Ignores()` explicitly. Tests that produce zero ignores don't need `Ignores().IsEmpty()` boilerplate; the completeness check fails if any unasserted ignore is emitted.
#### Extending the API (no escape hatches)
The assertion API is a **façade** over `match.Match` and `vulnerability.Vulnerability`. The whole point is to let grype refactor those internal types without touching tests. Two consequences:
- **There is no `Match()` accessor on `SingleFindingAssertion` and there should never be one.** `FindingsAssertion.Matches()` exists as a deprecated escape hatch and is not used anywhere — leave it that way.
- **When a test needs a new assertion, add a focused helper.** That is the documented extension story. `HasFix`, `HasAdvisories`, `InNamespace`, `SelectMatches.WithDetailType`, `SelectDetailBy{Distro,CPE,Ecosystem}`, `SelectRelatedPackageIgnores` were all added this way. Hypothetical future helpers: `HasSeverity`, `HasCVSSScore`, `HasRelatedVulnerabilities`, `HasFixAvailableDate`.
If you find yourself reaching for raw struct fields, stop and add the helper. A 5-line addition to `assertions.go` is cheaper than a 50-test sweep next time the underlying struct moves.
#### Universe-level vs. element-level assertions
When the chain ends in one or more `SelectMatch` / `SelectMatches` calls that fully cover the universe, universe-level assertions (`ContainsVulnerabilities`, `OnlyHasVulnerabilities`) are dead weight — completeness checking already proves them. Drop them:
```go
// dead weight - ContainsVulnerabilities is subsumed by SelectMatch + completeness
findings := db.Match(t, &matcher, p).
ContainsVulnerabilities("CVE-2024-0340")
findings.SelectMatch("CVE-2024-0340")...
// preferred
findings := db.Match(t, &matcher, p)
findings.SelectMatch("CVE-2024-0340")...
```
The same logic applies to `Ignores()`: with ignore-completeness on by default, any explicit ignore-side count assertion is subsumed by the exhaustive `SelectRelatedPackageIgnore`/`SelectRelatedPackageIgnores` chain. There is intentionally no `HasCount` method on any of these assertion types — if the count matters, prove it by asserting on each entry.
### Extracting Fixtures from Vunnel Cache
Use `FixtureExtractor` to create fixtures from a vunnel data directory:
```go
extractor := dbtest.NewFixtureExtractor("/path/to/vunnel/data")
// extract to new fixture
err := extractor.
From("debian").
Select("CVE-2024-1234", "debian:10").
WriteTo("internal/dbtest/testdata/shared/my-fixture")
// append to existing fixture
err := extractor.
From("rhel").
Select("RHSA-2024:").
AppendTo("internal/dbtest/testdata/shared/my-fixture")
// multi-provider extraction
err := extractor.
FromMultiple().
Provider("debian", "CVE-2024-1234").
Provider("nvd", "CVE-2024-1234").
WriteTo("internal/dbtest/testdata/shared/multi-provider")
```
## CLI Tool
The manager CLI provides commands for creating and maintaining fixtures.
### Extract Command
Create fixtures from vunnel SQLite caches:
```bash
# extract specific CVEs
go run ./internal/dbtest/cmd/manager extract \
--vunnel-data ~/vunnel/data \
--provider debian \
--select "CVE-2024-1234" \
--output internal/dbtest/testdata/shared/my-fixture
# extract by namespace (all CVEs in debian:11)
go run ./internal/dbtest/cmd/manager extract \
--vunnel-data ~/vunnel/data \
--provider debian \
--select "debian:11" \
--output internal/dbtest/testdata/shared/debian11-vulns
# extract with multiple patterns (OR logic)
go run ./internal/dbtest/cmd/manager extract \
--vunnel-data ~/vunnel/data \
--provider debian \
--select "CVE-2024-1234" --select "CVE-2024-5678" \
--output internal/dbtest/testdata/shared/specific-cves
# append to existing fixture
go run ./internal/dbtest/cmd/manager extract \
--vunnel-data ~/vunnel/data \
--provider rhel \
--select "RHSA-2024:" \
--append internal/dbtest/testdata/shared/my-fixture
# extract from multiple providers
go run ./internal/dbtest/cmd/manager extract \
--vunnel-data ~/vunnel/data \
--provider debian --provider nvd \
--select "CVE-2024-1234" \
--output internal/dbtest/testdata/shared/multi-provider
```
| Flag | Description |
|------|-------------|
| `--vunnel-data` | Path to vunnel data directory (required) |
| `--provider` | Provider name to extract from (repeatable) |
| `--select` | Pattern for record selection (repeatable) |
| `--output` | Path for new fixture directory |
| `--append` | Path to existing fixture to append to |
### Status Command
Check the status of all fixtures:
```bash
# show status of all fixtures
go run ./internal/dbtest/cmd/manager status
# show status of a specific fixture
go run ./internal/dbtest/cmd/manager status --fixture internal/dbtest/testdata/shared/my-fixture
```
Example output:
```
internal/dbtest/testdata/shared/all OK (automatic, synced)
grype/matcher/dpkg/testdata/eol-debian8 OK (automatic, synced)
internal/dbtest/testdata/shared/manual-vuln OK (manual)
internal/dbtest/testdata/shared/modified-one CONTENT DRIFT (lock: a1b2c3d4, actual: e5f6g7h8)
```
### Regenerate Command
Regenerate fixtures from their `db.yaml` configs:
```bash
# dry run to see what would be regenerated
go run ./internal/dbtest/cmd/manager regenerate --vunnel-data ~/vunnel/data --dry-run
# regenerate all fixtures
go run ./internal/dbtest/cmd/manager regenerate --vunnel-data ~/vunnel/data
# regenerate a specific fixture
go run ./internal/dbtest/cmd/manager regenerate --vunnel-data ~/vunnel/data --fixture path/to/fixture
# force regeneration even if fixture has been modified
go run ./internal/dbtest/cmd/manager regenerate --vunnel-data ~/vunnel/data --force
```
| Flag | Description |
|------|-------------|
| `--vunnel-data` | Path to vunnel data directory (required) |
| `--fixture` | Path to a specific fixture to regenerate |
| `--search-root` | Root directory to search for fixtures (repeatable) |
| `--force` | Regenerate even if fixture has been modified |
| `--dry-run` | Show what would be regenerated without making changes |
## Pattern Matching
Patterns use SQL `LIKE` matching with automatic wildcards. All patterns are wrapped with `%` for partial matching.
| Pattern | Matches | Description |
|---------|---------|-------------|
| `CVE-2024-1234` | `debian:10/CVE-2024-1234`, `ubuntu:20.04/CVE-2024-1234` | Matches CVE in any namespace |
| `debian:10` | `debian:10/CVE-2024-1234`, `debian:10/CVE-2024-5678` | Matches all CVEs in namespace |
| `debian` | `debian:10/...`, `debian:11/...` | Matches all debian namespaces |
| `RHSA-2024:` | `rhel:8/RHSA-2024:0001`, `rhel:9/RHSA-2024:0002` | Matches all 2024 RHSAs |
| `debian:11/CVE-2024-1234` | `debian:11/CVE-2024-1234` | Exact match |
Multiple `--select` patterns are combined with OR logic.
## Fixture Structure
Extracted fixtures follow the vunnel workspace format:
```
fixture/
├── db.yaml # extraction config (auto-generate flag, patterns)
├── db-lock.json # state tracking (content hash, timestamps)
├── provider-name/
│ ├── metadata.json # provider state (store: "flat-file")
│ └── results/
│ ├── debian@11_CVE-2024-1234.json
│ ├── debian@11_CVE-2024-5678.json
│ └── listing.xxh64
└── another-provider/
├── metadata.json
└── results/
└── ...
```
Note: `:` in identifiers is replaced with `@` and `/` with `_` in filenames.
## Tracking and Regenerating Fixtures
When you extract a fixture, the manager creates two metadata files:
- `db.yaml` - **Intent**: What extractions to perform (human-editable)
- `db-lock.json` - **State**: What was done (machine-generated)
### db.yaml
```yaml
auto-generate: true
extractions:
debian:
- CVE-2024-1234
- debian:10
nvd:
- CVE-2024-1234
```
| Field | Description |
|-------|-------------|
| `auto-generate` | If `true`, fixture can be regenerated from vunnel cache |
| `extractions` | Map of provider name to list of patterns |
### db-lock.json
```json
{
"content_hash": "a1b2c3d4e5f6g7h8",
"created_at": "2024-01-15T12:00:00Z",
"regenerated_at": "2024-03-12T10:30:00Z",
"providers": {
"debian": {
"vunnel_version": "vunnel@0.55.2",
"timestamp": "2024-03-11T16:25:19Z"
}
}
}
```
| Field | Description |
|-------|-------------|
| `content_hash` | xxh64 hash of fixture content (excludes db.yaml and db-lock.json) |
| `created_at` | When fixture was first created |
| `regenerated_at` | When last regenerated (omitted if never regenerated) |
| `providers..vunnel_version` | Vunnel version from provider's metadata |
| `providers..timestamp` | Provider's data timestamp |
### Fixture Status
| Status | Description |
|--------|-------------|
| `OK` (automatic) | `auto-generate: true` and content hash matches lock |
| `OK` (manual) | `auto-generate: false` - manually created, never auto-regenerated |
| `content_drift` | `auto-generate: true` but files have changed since last generation |
| `config_ahead` | `auto-generate: true` but config has providers not in lock |
| `no_lock` | `auto-generate: true` but db-lock.json is missing |
| `no_config` | No db.yaml file present |
## Workflows
### Creating a New Fixture
```bash
# 1. extract (db.yaml and db-lock.json created automatically)
go run ./internal/dbtest/cmd/manager extract \
--vunnel-data ~/vunnel/data \
--provider debian --select "CVE-2024-1234" \
--output internal/dbtest/testdata/shared/my-fixture
# 2. optionally append more providers/CVEs
go run ./internal/dbtest/cmd/manager extract \
--vunnel-data ~/vunnel/data \
--provider nvd --select "CVE-2024-1234" \
--append internal/dbtest/testdata/shared/my-fixture
# 3. commit
git add internal/dbtest/testdata/shared/my-fixture/
git commit -m "add my-fixture test fixture"
```
### Creating a Manual Fixture
Manual fixtures are never automatically regenerated, useful when you need custom modifications:
```bash
# 1. extract base
go run ./internal/dbtest/cmd/manager extract \
--vunnel-data ~/vunnel/data \
--provider debian --select "CVE-2024-1234" \
--output internal/dbtest/testdata/shared/manual-fixture
# 2. manually modify files as needed
vim internal/dbtest/testdata/shared/manual-fixture/debian/results/...
# 3. edit db.yaml to set auto-generate: false
# (prevents automatic regeneration)
# 4. commit
git add internal/dbtest/testdata/shared/manual-fixture/
git commit -m "add manual-fixture with custom modifications"
```
### Regenerating Fixtures
When vunnel data is updated, regenerate fixtures to get fresh vulnerability data:
```bash
# 1. check current status
go run ./internal/dbtest/cmd/manager status
# 2. dry run to see what would happen
go run ./internal/dbtest/cmd/manager regenerate --vunnel-data ~/vunnel/data --dry-run
# 3. regenerate (skips manual and modified fixtures)
go run ./internal/dbtest/cmd/manager regenerate --vunnel-data ~/vunnel/data
# 4. run tests to verify
go test ./...
```
---
### Overview (`test/quality/README.md`)
# Match quality testing
This form of testing compares the results from various releases of grype using a
static set of reference container images. The kinds of comparisons made are:
1) "relative": find the vulnerability matching differences between both tools
for a given image. This helps identify when a change has occurred in matching
behavior and where the changes are.
2) "against labels": pair each tool results for an image with ground truth. This
helps identify how well the matching behavior is performing (did it get
better or worse).
## Getting started
For information about required setup see: [Required setup](#required-setup).
To capture raw tool output and store into the local `.yardstick` directory for
further analysis:
```
make capture
```
To analyze the tool output and evaluate a pass/fail result:
```
make validate
```
A pass/fail result is shown in the output with reasons for the failure being
listed explicitly.
## What is the quality gate criteria
The label comparison results are used to determine a pass/fail result,
specifically with the following criteria:
- fail when current grype F1 score drops below last grype release F1 score (or
F1 score is indeterminate)
- fail when the indeterminate matches % > 10% in the current grype results
- fail when there is a rise in FNs relative to the results from the last grype
release
- otherwise, pass
F1 score is the primary way that tool matching performance is characterized. F1
score combines the TP, FP, and FN counts into a single metric between 0 and 1.
Ideally the F1 score for an image-tool pair should be 1. F1 score is a good way
to summarize the matching performance but does not explain why the matching
performance is what it is.
Indeterminate matches are matches from results that could not be paired with a
label (TP or FP). This could also mean that multiple conflicting labels were
found for a single match. The more indeterminate matches there are the less
confident you can be about the F1 score. Ideally there should be 0 indeterminate
matches, but this is difficult to achieve since vulnerability data is constantly
changing.
False negatives represent matches that should have been made by the tool but
were missed. We should always make certain that this value does not increase
between releases of grype.
## Assumptions
1. **Comparing vulnerability results taken at different times is invalid**.
We leverage the yardstick result-set feature to capture all vulnerability
results at one time for a specific image and tool set. Why? If we use grype
at version `a` on monday and grype at version `b` on tuesday and attempt to
compare the results, if differences are found it will not be immediately
clear why the results are different. That is, it is entirely possible that
the vulnerability databases from the run of `b` simply had more up to date
information, and if `grype@a` were run at the same time (on tuesday) this
reason can be almost entirely eliminated.
2. **Comparing vulnerability results across images with different digests is invalid**.
It may be very tempting to compare vulnerability results for
`alpine:3.2` from monday and `alpine:3.2` from tuesday to see if there are
any changes. However, this is potentially inaccurate as the image references
are for the same tag, but the publisher may have pushed a new image with
differing content. Any change could lead to different vulnerability matching
results but we are only interested in vulnerability match differences that
are due to actionable reasons (grype matcher logic problems or [SBOM] input
data into matchers).
## Approach
Vulnerability matching has essentially two inputs:
- the packages that were found in the scanned artifact
- the vulnerability data from upstream providers (e.g. NVD, GHSA, etc.)
These are both moving targets!
We may implement more catalogers in syft that raise up more packages discovered
over time (for the same artifact scanned). Also the world is continually finding
and reporting new vulnerabilities. The more moving parts there are in this form
of testing the harder it is to come to a conclusion about the actual quality of
the output over time.
To reduce the eroding value over time we've decided to change as many moving
targets into fixed targets as possible:
- Vulnerability results beyond a particular year are ignored (the current config
allows for <= 2020). Though there are still retroactive CVEs created, this
helps a lot in terms of keeping vulnerability results relatively stable.
- SBOMs are used as input into grype instead of the raw container images. This
allows the artifacts under test to remain truly fixed and saves a lot of time
when capturing grype results (as the container image is no longer needed
during analysis).
- For the captured SBOMs, container images referenced must be with a digest, not
just a tag. In case we update a tool version (say syft) we want to make
certain that we are scanning the exact same artifact later when we re-run the
analysis.
- Versions of tools used are fixed to a specific `major.minor.patch` release used.
This allows us to account for capability differences between tool runs.
To reduce maintenance effort of this comparison over time there are a few things
to keep in mind:
- Once an image is labeled (at a specific digest) the image digest should be
considered immutable (never updated). Why? It takes a lot of effort to label
images and there are no "clearly safe" assumptions that can be made when it
comes to migrating labels from one image to another no matter how "similar"
the images may be. There is also no value in updating the image; these images
are not being executed and their only purpose is to survey the matching
performance of grype. In the philosophy of "maximizing fixed points" it
doesn't make sense to change these assets. Over time it may be that we remove
assets that are no longer useful for comparison, but this should rarely be
done.
- Consider not changing the CVE year max-ceiling (currently set to 2020).
Pushing this ceiling will likely raise the number of unlabled matches
significantly for all images. Only bump this ceiling if all possible matches
are labeled.
## Workflow
One way of working is to simply run `yardstick` and `gate.py` in the `test/quality` directory.
You will need to make sure the `vulnerability-match-labels` submodule has been initialized. This happens automatically
for some `make` commands, but you can ensure this by `git submodule update --init`. After the submodule has been
initialized, the match data from `vulnerability-match-labels` will be available locally.
**TIP**: when dealing with submodules, it may be convenient to set the git config option `submodule.recurse` to `true`
so `git checkout` will automatically update submodules to the correct commit:
```shell
git config submodule.recurse true
```
To do this we need some results to begin with. As noted above, start with (this does ensure the submodule is initialized):
```shell
make capture
```
This will download prebuilt SBOMs for the configured images and generate match results for configured tools (here:
the previous Grype version as well as the local version).
After `make capture` has finished, we should have results and can now start inspecting and
modifying the comparison labels.
To get started, let's assume we see some quality gate failure in like this (something found in CI
or after running `./gate.py`):
```
Running comparison against labels...
Results used:
├── f4fb4e6e-c911-41b6-9a10-f90b3954a41a : grype@v0.53.1-19-g8900767 against docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
└── fcebdd0b-d80a-4fe2-b81a-802c7b98d83b : grype@v0.53.1 against docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
Match differences between tooling (with labels):
TOOL PARTITION PACKAGE VULNERABILITY LABEL COMMENTARY
grype@v0.53.1 ONLY node@14.18.2 CVE-2021-44531 TruePositive (this is a new FN 😱)
grype@v0.53.1 ONLY node@14.18.2 CVE-2021-44532 TruePositive (this is a new FN 😱)
grype@v0.53.1 ONLY node@14.18.2 CVE-2021-44533 TruePositive (this is a new FN 😱)
Failed quality gate
- current F1 score is lower than the latest release F1 score: current=0.80 latest=0.80 image=docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
- current indeterminate matches % is greater than 10%: current=13.60% image=docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
- current false negatives is greater than the latest release false negatives: current=6 latest=3 image=docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
```
This tells us some important information: which package, version, and vulnerability had a difference;
how it was previously labeled, and most importantly: the image we need to focus on (`docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9`).
Using the SHA above, we can run `yardstick` to see which results are available:
```shell
$ yardstick result list --result-set pr_vs_latest_via_sbom | grep 808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
5bf0611b-183f-4525-a1ab-f268f62f48b6 docker.io/anchore/test_images:appstreams-centos-stream-8-1a287dd@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9 grype@v0.53.1 2022-12-09 20:49:56+00:00
43a9650a-d5de-4687-b3ba-459105e32cb8 docker.io/anchore/test_images:appstreams-centos-stream-8-1a287dd@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9 grype@v0.53.1-15-gf29a32b 2022-12-09 20:49:53+00:00
67913f57-690f-4f35-a2d9-ffccd2a0b2a1 docker.io/anchore/test_images:appstreams-centos-stream-8-1a287dd@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9 syft@v0.60.1 2022-11-01 20:30:52+00:00
```
We'll need to use the UUIDs to explore the labels, so copy the first UUID, which we can see was run against the last Grype release (`grype@v0.53.1`). Use the UUID to explore and edit the results with
`yardstick label explore`:
```shell
yardstick label explore 5bf0611b-183f-4525-a1ab-f268f62f48b6
```
At this point we can use the TUI to explore and modify the match data, by deleting things or labeling as
true positives, false positives, etc.. **After making changes make sure to save the results** (`Ctrl-S`)!
At this point you can run the quality gate using updated label data. The quality gate can run against
just one image, for example the image we first found in the failure, so run the quality gate and see
how changes to the label data have affected the result:
```shell
./gate.py --image docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
```
After iterating on all the changes we need using `yardstick label explore`, we're now ready to commit the changes. Since
we're using `git submodules`, we need to complete two steps:
1. get the changes merged to the `vulnerability-match-labels` repository `main` branch
2. update the submodule in this repository
To create a pull request for the `vulnerability-match-labels`, make sure you are in the `vulnerability-match-labels`
subdirectory and create a branch -- something like:
```shell
git checkout --no-track -b my-branch-name
```
Commit the changes to this branch, push, create a pull request like normal. NOTE: you may need to add a fork
(`git remote add ...`) and push to the fork if you don't have push permissions against the main
`vulnerability-match-labels` repo. After the PR is approved and merged to `vulnerability-match-labels` repo's `main`
branch, update the submodule locally using:
```shell
git submodule update --remote
```
Next, _commit the submodule change_ as part of any other changes
to the Grype pull request and push as part of the in-progress PR
against Grype. The PR will now use the updated match labels when running
the quality check.
## Required setup
In order to manage Python versions, [pyenv](https://github.com/pyenv/pyenv) can be used. (e.g. `brew install pyenv`)
Both this project and `yardstick` require Python 3.10.
Using `pyenv`, see which python versions are available, for example:
```shell
$ pyenv install --list|grep 3.10
3.10.0
...
3.10.7
...
```
In this case, we see `3.10.7` is the latest version, so we'll use that for the rest of the setup:
Install this version using `pyenv`:
```shell
pyenv install 3.10.7
```
NOTE: to view the specific Python versions installed use `pyenv versions`:
```shell
$ pyenv versions
system
* 3.8.13 (set by /Users/usr/.pyenv/version)
3.10.7
```
To select the `3.10` version use the exact version number:
```shell
pyenv shell 3.10.7
```
(or maybe just: `pyenv shell $(pyenv versions | grep 3.10 | tail -1)`)
Verify this has worked properly by running:
```shell
python --version
```
**Important:** it is also required to have `oras` installed (e.g. `brew install oras`)
**After** setting the working Python version to 3.10, in the `test/quality` directory,
you need to set up a virtual environment using:
```shell
make venv
```
**After** creating the virtual environment, you can now activate it to set up a
working shell using:
```shell
. venv/bin/activate
```
You should now have a shell running in the correct virtual environment, it might look something
like this:
```shell
(venv) user@HOST quality %
```
Now you should be able to run both `yardstick` and `./gate.py`.
## Troubleshooting
As noted above, yardstick requires Python 3.10. If you try to run with an older version, such as
the default macOS 3.8 version, you will likely see an error similar to:
```
Traceback (most recent call last):
File "./vulnerability-match-labels/sboms.py", line 12, in
import yardstick
File "/grype/test/quality/vulnerability-match-labels/venv/lib/python3.8/site-packages/yardstick/__init__.py", line 4, in
from . import arrange, artifact, capture, cli, comparison, label, store, tool, utils
File "/grype/test/quality/vulnerability-match-labels/venv/lib/python3.8/site-packages/yardstick/arrange.py", line 4, in
from yardstick import artifact
File "/grype/test/quality/vulnerability-match-labels/venv/lib/python3.8/site-packages/yardstick/artifact.py", line 482, in
class ResultSet:
File "/grype/test/quality/vulnerability-match-labels/venv/lib/python3.8/site-packages/yardstick/artifact.py", line 484, in ResultSet
state: list[ResultState] = field(default_factory=list)
TypeError: 'type' object is not subscriptable
```
---
### Overview (`schema/grype/db/README.md`)
# Grype v6 Database Schemas
This directory contains the schemas for the Grype v6 vulnerability database. These schemas are automatically generated from the Go code definitions and are used to track schema evolution over time.
## What Are These Schemas?
The Grype database has two types of schemas that are tracked:
### 1. SQL Schema (`sql/`)
The SQLite table definitions (CREATE TABLE and CREATE INDEX statements) generated from GORM models in `grype/db/v6/models.go`. This captures:
- Database tables and columns
- Foreign key relationships
- Indexes for query performance
- All structural aspects of the database
**Example tables:** `vulnerability_handles`, `packages`, `operating_systems`, `cpes`, `blobs`, etc.
### 2. Blob JSON Schema (`blob/json/`)
A unified JSON schema for all blob types stored in the `blobs` table. The blobs are JSON data referenced by various handle tables and include:
- `VulnerabilityBlob`: Core vulnerability advisory data
- `PackageBlob`: Package version ranges and fix information
- `KnownExploitedVulnerabilityBlob`: CISA KEV catalog data
## Schema Files
Each schema type has two files per version:
- **`schema-X.Y.Z.{sql|json}`**: The versioned schema file that should never be modified after creation
- **`schema-latest.{sql|json}`**: A copy of the most recent version to show diffs in PR reviews
The `-latest` files exist to make PR reviews easier. When you increment the schema version and regenerate, Git shows the new `schema-X.Y.Z` file as entirely new (just additions), which makes it hard to see what actually changed. The `-latest` file, however, is already tracked by Git, so it shows as a **diff** - making it easy to review exactly what changed in the schema.
## How to Regenerate Schemas
When you make changes to:
- GORM models in `grype/db/v6/models.go`
- Blob types in `grype/db/v6/blobs.go`
You need to regenerate the schemas:
```bash
task generate-db-schema
```
This will:
1. Create an in-memory SQLite database with all GORM models
2. Extract the SQL schema from the database
3. Generate a unified JSON schema for all blob types
4. Write the schemas to versioned files
## What to Do When Schema Changes
### If You're Adding Compatible Changes (Addition)
Examples: Adding a new optional field to a blob, adding a new index
1. Increment `Addition` in `grype/db/v6/db.go`:
```go
Addition = 2 // was 1
```
2. Regenerate schemas:
```bash
task generate-db-schema
```
3. Commit the new schema files along with your code changes
### If You're Making Potentially Breaking Changes (Revision)
Examples: Changing field types, removing optional fields, altering table structure
1. Increment `Revision` in `grype/db/v6/db.go` and reset `Addition`:
```go
Revision = 2 // was 1
Addition = 0 // reset
```
2. Regenerate and commit as above
### If You're Making Definitely Breaking Changes (Model)
Please meet with the team about this - it requires careful planning and
should be rare.
## Versioning Rules (SchemaVer)
This project uses [SchemaVer](https://docs.snowplowanalytics.com/docs/pipeline-components-and-applications/iglu/common-architecture/schemaver/) for schema versioning: `MODEL.REVISION.ADDITION`
- **MODEL**: Increment for breaking changes that prevent interaction with ALL historical data
- **REVISION**: Increment for changes that may prevent interaction with SOME historical data
- **ADDITION**: Increment for changes that are compatible with ALL historical data
**Important:** Never delete or modify existing versioned schema files! Only add new versions.
## CI Drift Detection
The static analysis CI check runs:
```bash
task check-db-schema-drift
```
This:
1. Checks that working directory is clean
2. Runs `task generate-db-schema`
3. Checks if any schema files changed
If schemas changed but weren't committed, the check fails. This ensures:
- Schema changes are always tracked
- Version numbers are incremented appropriately
- Code changes and schema changes stay in sync
This catches cases where you modified models or blob types but forgot to regenerate and commit the schemas.
## Common Errors
### "Cowardly refusing to overwrite existing schema"
This means:
- The schema has changed (code differs from committed schema)
- But the version number hasn't been incremented
**Solution:** Increment the appropriate version constant in `grype/db/v6/db.go`
### "Database blob schemas have uncommitted changes"
This means:
- You made schema changes
- Regenerated the schemas
- But haven't committed the new schema files
**Solution:** Add and commit the schema files in `schema/grype/db/`
## More Information
- **Generator Code**: `grype/db/v6/schema/main.go` - The Go program that generates these schemas
- **Version Constants**: `grype/db/v6/db.go` - Where `ModelVersion`, `Revision`, and `Addition` are defined
- **GORM Models**: `grype/db/v6/models.go` - The source for SQL schema generation
- **Blob Types**: `grype/db/v6/blobs.go` - The source for blob JSON schema generation
---
### Overview (`schema/grype/db-diff/json/README.md`)
# `db-diff` JSON Schema
This is the JSON schema for output from the `grype db diff` command. The required inputs for defining the JSON schema are as follows:
- the value of `grype/db/v6/diff.SchemaVersion` that governs the schema version
- the `Result` type definition within `github.com/anchore/grype/grype/db/v6/diff/result.go` that governs the overall document shape
## Versioning
Versioning the JSON schema must be done manually by changing the `SchemaVersion` constant within `grype/db/v6/diff/schema.go`.
Follow semver guidelines for incrementing the version.
## Generating a New Schema
Create the new schema by running `make generate-json-schema` from the root of the repo:
- If there is **not** an existing schema for the given version, then the new schema file will be written to `schema/grype/db-diff/json/schema-$VERSION.json`
- If there is an existing schema for the given version and the new schema matches the existing schema, no action is taken
- If there is an existing schema for the given version and the new schema **does not** match the existing schema, an error is shown indicating to increment the version appropriately (see the "Versioning" section)
***Note: never delete a JSON schema and never change an existing JSON schema once it has been published in a release!*** Only add new schemas with a newly incremented version.
---
### Overview (`schema/grype/db-search-vuln/json/README.md`)
# `db-search vuln` JSON Schema
This is the JSON schema for output from the `grype db search vuln` command. The required inputs for defining the JSON schema are as follows:
- the value of `cmd/grype/cli/commands/internal/dbsearch.VulnerabilitiesSchemaVersion` that governs the schema version
- the `Vulnerabilities` type definition within `github.com/anchore/grype/cmd/grype/cli/commands/internal/dbsearch/vulnerabilities.go` that governs the overall document shape
## Versioning
Versioning the JSON schema must be done manually by changing the `VulnerabilitiesSchemaVersion` constant within `cmd/grype/cli/commands/internal/dbsearch/versions.go`.
This schema is being versioned based off of the "SchemaVer" guidelines, which slightly diverges from Semantic Versioning to tailor for the purposes of data models.
Given a version number format `MODEL.REVISION.ADDITION`:
- `MODEL`: increment when you make a breaking schema change which will prevent interaction with any historical data
- `REVISION`: increment when you make a schema change which may prevent interaction with some historical data
- `ADDITION`: increment when you make a schema change that is compatible with all historical data
## Generating a New Schema
Create the new schema by running `make generate-json-schema` from the root of the repo:
- If there is **not** an existing schema for the given version, then the new schema file will be written to `schema/grype/db-search-vuln/json/schema-$VERSION.json`
- If there is an existing schema for the given version and the new schema matches the existing schema, no action is taken
- If there is an existing schema for the given version and the new schema **does not** match the existing schema, an error is shown indicating to increment the version appropriately (see the "Versioning" section)
***Note: never delete a JSON schema and never change an existing JSON schema once it has been published in a release!*** Only add new schemas with a newly incremented version.
---
### Overview (`schema/grype/db-search/json/README.md`)
# `db-search` JSON Schema
This is the JSON schema for output from the `grype db search` command. The required inputs for defining the JSON schema are as follows:
- the value of `cmd/grype/cli/commands/internal/dbsearch.MatchesSchemaVersion` that governs the schema version
- the `Matches` type definition within `github.com/anchore/grype/cmd/grype/cli/commands/internal/dbsearch/matches.go` that governs the overall document shape
## Versioning
Versioning the JSON schema must be done manually by changing the `MatchesSchemaVersion` constant within `cmd/grype/cli/commands/internal/dbsearch/versions.go`.
This schema is being versioned based off of the "SchemaVer" guidelines, which slightly diverges from Semantic Versioning to tailor for the purposes of data models.
Given a version number format `MODEL.REVISION.ADDITION`:
- `MODEL`: increment when you make a breaking schema change which will prevent interaction with any historical data
- `REVISION`: increment when you make a schema change which may prevent interaction with some historical data
- `ADDITION`: increment when you make a schema change that is compatible with all historical data
## Generating a New Schema
Create the new schema by running `make generate-json-schema` from the root of the repo:
- If there is **not** an existing schema for the given version, then the new schema file will be written to `schema/grype/db-search/json/schema-$VERSION.json`
- If there is an existing schema for the given version and the new schema matches the existing schema, no action is taken
- If there is an existing schema for the given version and the new schema **does not** match the existing schema, an error is shown indicating to increment the version appropriately (see the "Versioning" section)
***Note: never delete a JSON schema and never change an existing JSON schema once it has been published in a release!*** Only add new schemas with a newly incremented version.
---
### Overview (`grype/db/v5/distribution/testdata/tls/README.md`)
# TLS test utils
Note: Makefile, server.crt, and server.key are used in automated testing, the remaining files are for convenience in manual verification.
You will require Python 3 to run these utils.
To stand up a test server:
```
make serve
```
To test grype against this server:
```
# without the custom cert configured (thus will fail)
make grype-test-fail
# with the custom cert configured
make grype-test-pass
```
To remove all temp files:
```
make clean
```
---
--- METRICS ---
- Files Extracted: 9
- Estimated Token Budget: ~13275 tokens
- Canonical Reference: https://codewiki.google/github.com/anchore/grype
