1. Project Overview & Quickstart (hashicorp/vault)
File: README.md
Vault [](https://github.com/hashicorp/vault/actions/workflows/build.yml) [](https://github.com/hashicorp/vault/actions/workflows/ci.yml) [](https://www.hashicorp.com/products/vault/?utm_source=github&utm_medium=banner&utm_campaign=github-vault-enterprise)
Please note: We take Vault's security and our users' trust very seriously. If you believe you have found a security issue in Vault, please responsibly disclose by contacting us at [email protected].
- Website: developer.hashicorp.com/vault
- Announcement list: Google Groups
- Discussion forum: Discuss
- Documentation: https://developer.hashicorp.com/vault/docs
- Tutorials: https://developer.hashicorp.com/vault/tutorials
- Certification exam: https://developer.hashicorp.com/certifications/security-automation
- Documentation source: https://github.com/hashicorp/web-unified-docs
Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, and more. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log.
A modern system requires access to a multitude of secrets: database credentials, API keys for external services, credentials for service-oriented architecture communication, etc. Understanding who is accessing what secrets is already very difficult and platform-specific. Adding on key rolling, secure storage, and detailed audit logs is almost impossible without a custom solution. This is where Vault steps in.
The key features of Vault are:
Secure Secret Storage: Vault can store arbitrary key/value pairs. Vault encrypts data before writing it to persistent
storage, so gaining access to the raw storage isn't enough to access
your secrets. Vault can write to disk, Consul,
and more.Dynamic Secrets: Vault can generate secrets on-demand for some
systems, such as AWS or SQL databases. For example, when an application
needs to access an S3 bucket, it asks Vault for credentials, and Vault
will generate an AWS keypair with valid permissions on demand. After
creating these dynamic secrets, Vault will also automatically revoke them
after the lease is up.Data Encryption: Vault can encrypt and decrypt data without storing
it. This allows security teams to define encryption parameters and
developers to store encrypted data in a location such as a SQL database without
having to design their own encryption methods.Leasing and Renewal: Vault associates a lease with each secret.
At the end of the lease, Vault automatically revokes the
secret. Clients are able to renew leases via built-in renew APIs.Revocation: Vault has built-in support for secret revocation. Vault
can revoke not only single secrets, but a tree of secrets, for example,
all secrets read by a specific user, or all secrets of a particular type.
Revocation assists in key rolling as well as locking down systems in the
case of an intrusion.
Documentation, Getting Started, and Certification Exams
Documentation is available on the Vault website.
If you're new to Vault and want to get started with security automation, please
check out our Getting Started guides
on HashiCorp's learning platform. There are also additional guides
to continue your learning.
For examples of how to interact with Vault from inside your application in different programming languages, see the vault-examples repo. An out-of-the-box sample application is also available.
Show off your Vault knowledge by passing a certification exam. Visit the
certification page
for information about exams and find study materials
on HashiCorp's learning platform.
Developing Vault
If you wish to work on Vault itself or any of its built-in systems, you'll
first need Go installed on your machine.
For local dev first make sure Go is properly installed, including setting up a
GOPATH, then setting the
GOBIN variable to $GOPATH/bin.
Ensure that $GOPATH/bin is in your path as some distributions bundle the old version
of build tools.
Next, clone this repository. Vault uses Go Modules,
so it is recommended that you clone the repository outside of the GOPATH.
You can then download any required build tools by bootstrapping your environment:
$ make bootstrap
...To compile a development version of Vault, run make or make dev. This will
put the Vault binary in the bin and $GOPATH/bin folders:
$ make dev
...
$ bin/vault
...To compile a development version of Vault with the UI, run make static-dist dev-ui. This will
put the Vault binary in the bin and $GOPATH/bin folders:
$ make static-dist dev-ui
...
$ bin/vault
...To run tests, type make test. Note: this requires Docker to be installed. If
this exits with exit status 0, then everything is working!
$ make test
...If you're developing a specific package, you can run tests for just that
package by specifying the TEST variable. For example below, onlyvault package tests will be run.
$ make test TEST=./vault
...Troubleshooting
If you encounter an error like could not read Username for 'https://github.com' you may need to adjust your git config like so:
$ git config --global --add url."[email protected]:".insteadOf "https://github.com/"Importing Vault
This repository publishes two libraries that may be imported by other projects:github.com/hashicorp/vault/api and github.com/hashicorp/vault/sdk.
Note that this repository also contains Vault (the product), and as with most Go
projects, Vault uses Go modules to manage its dependencies. The mechanism to do
that is the go.mod file. As it happens, the presence of that file
also makes it theoretically possible to import Vault as a dependency into other
projects. Some other projects have made a practice of doing so in order to take
advantage of testing tooling that was developed for testing Vault itself. This
is not, and has never been, a supported way to use the Vault project. We aren't
likely to fix bugs relating to failure to import github.com/hashicorp/vault
into your project.
See also the section "Docker-based tests" below.
Acceptance Tests
Vault has comprehensive acceptance tests
covering most of the features of the secret and auth methods.
If you're working on a feature of a secret or auth method and want to
verify it is functioning (and also hasn't broken anything else), we recommend
running the acceptance tests.
Warning: The acceptance tests create/destroy/modify real resources, which
may incur real costs in some cases. In the presence of a bug, it is technically
possible that broken backends could leave dangling data behind. Therefore,
please run the acceptance tests at your own risk. At the very least,
we recommend running them in their own private account for whatever backend
you're testing.
To run the acceptance tests, invoke make testacc:
$ make testacc TEST=./builtin/logical/consul
...The TEST variable is required, and you should specify the folder where the
backend is. The TESTARGS variable is recommended to filter down to a specific
resource to test, since testing all of them at once can sometimes take a very
long time.
Acceptance tests typically require other environment variables to be set for
things such as access keys. The test itself should error early and tell
you what to set, so it is not documented here.
For more information on Vault Enterprise features, visit the Vault Enterprise site.
Docker-based Tests
We have created an experimental new testing mechanism inspired by NewTestCluster.
An example of how to use it:
import (
"testing"
"github.com/hashicorp/vault/sdk/helper/testcluster/docker"
)
func Test_Something_With_Docker(t *testing.T) {
opts := &docker.DockerClusterOptions{
ImageRepo: "hashicorp/vault", // or "hashicorp/vault-enterprise"
ImageTag: "latest",
}
cluster := docker.NewTestDockerCluster(t, opts)
client := cluster.Nodes()[0].APIClient()
_, err := client.Logical().Read("sys/storage/raft/configuration")
if err != nil {
t.Fatal(err)
}
}Or for Enterprise:
import (
"testing"
"github.com/hashicorp/vault/sdk/helper/testcluster/docker"
)
func Test_Something_With_Docker(t *testing.T) {
opts := &docker.DockerClusterOptions{
ImageRepo: "hashicorp/vault-enterprise",
ImageTag: "latest",
VaultLicense: licenseString, // not a path, the actual license bytes
}
cluster := docker.NewTestDockerCluster(t, opts)
}Here is a more realistic example of how we use it in practice. DefaultOptions useshashicorp/vault:latest as the repo and tag, but it also looks at the environment
variable VAULT_BINARY. If populated, it will copy the local file referenced by
VAULT_BINARY into the container. This is useful when testing local changes.
Instead of setting the VaultLicense option, you can set the VAULT_LICENSE_CI environment
variable, which is better than committing a license to version control.
Optionally you can set COMMIT_SHA, which will be appended to the image name we
build as a debugging convenience.
func Test_Custom_Build_With_Docker(t *testing.T) {
opts := docker.DefaultOptions(t)
cluster := docker.NewTestDockerCluster(t, opts)
}There are a variety of helpers in the github.com/hashicorp/vault/sdk/helper/testcluster
package, e.g. these tests below will create a pair of 3-node clusters and link them using
PR or DR replication respectively, and fail if the replication state doesn't become healthy
before the passed context expires.
Again, as written, these depend on having a Vault Enterprise binary locally and the env
var VAULT_BINARY set to point to it, as well as having VAULT_LICENSE_CI set.
func TestStandardPerfReplication_Docker(t *testing.T) {
opts := docker.DefaultOptions(t)
r, err := docker.NewReplicationSetDocker(t, opts)
if err != nil {
t.Fatal(err)
}
defer r.Cleanup()
ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
defer cancel()
err = r.StandardPerfReplication(ctx)
if err != nil {
t.Fatal(err)
}
}
func TestStandardDRReplication_Docker(t *testing.T) {
opts := docker.DefaultOptions(t)
r, err := docker.NewReplicationSetDocker(t, opts)
if err != nil {
t.Fatal(err)
}
defer r.Cleanup()
ctx, cancel := context.WithTimeout(context.Background(), time.Minute)
defer cancel()
err = r.StandardDRReplication(ctx)
if err != nil {
t.Fatal(err)
}
}Finally, here's an example of running an existing OSS docker test with a custom binary:
$ GOOS=linux make dev
$ VAULT_BINARY=$(pwd)/bin/vault go test -run 'TestRaft_Configuration_Docker' ./vault/external_tests/raft/raft_binary
ok github.com/hashicorp/vault/vault/external_tests/raft/raft_binary 20.960sFile: api/README.md
Vault API
This provides the github.com/hashicorp/vault/api package which contains code useful for interacting with a Vault server.
For examples of how to use this module, see the vault-examples repo.
For a step-by-step walkthrough on using these client libraries, see the developer quickstart.
File: builtin/logical/pkiext/README.md
What is `pkiext`?
pkiext exists to split the Docker tests into a separate package from the
main PKI tests. Because the Docker tests execute in a smaller runner with
fewer resources, and we were hitting timeouts waiting for the entire PKI
test suite to run, we need to split the larger non-Docker PKI tests from
the smaller Docker tests, to ensure the former can execute.
This package should lack any non-test related targets.
File: changelog/README.md
changelog
This folder holds changelog updates from commit 3bc7d15 onwards.
Release notes are text files with three lines:
An opening code block with the
release-note:<MODE>type annotation.For example:
text```
/* Detailed source-code truncated for AI context efficiency. */
**Feature Name**: Description of feature - for example "Custom password policies are now supported for all database engines."
```
---
## File: command/agent/README.md
# Vault Agent
Vault Agent is a client daemon that provides Auth-Auth, Caching, and Template
features.
Vault Agent provides a number of different helper features, specifically
addressing the following challenges:
- Automatic authentication
- Secure delivery/storage of tokens
- Lifecycle management of these tokens (renewal & re-authentication)
See the usage documentation on the Vault website here:
- https://developer.hashicorp.com/vault/docs/agent-and-proxy/agent
---
## File: enos/modules/benchmark/README.md
# Benchmark modules
These benchmark modules are designed to benchmark a Vault cluster using either
raft storage or an external Consul cluster for storage. It supports creating
an external telemetry collector using a combination of Prometheus and Grafana
and supports collecting Vault, Consul, and node metrics.
When using this module in your scenario there are a few things to consider:
- There are currently assumptions that Ubuntu 22.04 is the target runner for
the k6 runner instance.
- The Vault and Consul clusters are assumed to be three node clusters.
---
## File: enos/README.md
# Enos
Enos is an quality testing framework that allows composing and executing quality
requirement scenarios as code. For Vault, it is currently used to perform
infrastructure integration testing using the artifacts that are created as part
of the `build` workflow. While intended to be executed via Github Actions using
the results of the `build` workflow, scenarios are also executable from a developer
machine that has the requisite dependencies and configuration.
Refer to the [Enos documentation](https://github.com/hashicorp/Enos-Docs)
for further information regarding installation, execution or composing Enos scenarios.
## When to use Enos
Determining whether to use `vault.NewTestCluster()` or Enos for testing a feature
or scenario is ultimately up to the author. Sometimes one, the other, or both
might be appropriate depending on the requirements. Generally, `vault.NewTestCluster()`
is going to give you faster feedback and execution time, whereas Enos is going
to give you a real-world execution and validation of the requirement. Consider
the following cases as examples of when one might opt for an Enos scenario:
- The feature require third-party integrations. Whether that be networked
dependencies like a real Consul backend, a real KMS key to test awskms
auto-unseal, auto-join discovery using AWS tags, or Cloud hardware KMS's.
- The feature might behave differently under multiple configuration variants
and therefore should be tested with both combinations, e.g. auto-unseal and
manual shamir unseal or replication in HA mode with integrated storage or
Consul storage.
- The scenario requires coordination between multiple targets. For example,
consider the complex lifecycle event of migrating the seal type or storage,
or manually triggering a raft disaster scenario by partitioning the network
between the leader and follower nodes. Or perhaps an auto-pilot upgrade between
a stable version of Vault and our candidate version.
- The scenario has specific deployment strategy requirements. For example,
if we want to add a regression test for an issue that only arises when the
software is deployed in a certain manner.
- The scenario needs to use actual build artifacts that will be promoted
through the pipeline.
## Requirements
- AWS access. HashiCorp Vault developers should use Doormat.
- Terraform >= 1.7
- Enos >= v0.0.34. You can [download a release](https://github.com/hashicorp/enos/releases/) or
install it with Homebrew:
```shell
brew tap hashicorp/tap && brew update && brew install hashicorp/tap/enos- An SSH keypair in the AWS region you wish to run the scenario. You can use
Doormat to log in to the AWS console to create or upload an existing keypair. - A Vault artifact is downloaded from the GHA artifacts when using the
artifact_source:crtvariants, from Artifactory when usingartifact_source:artifactory, and is built locally from the current branch when usingartifact_source:localvariant.
Scenario Variables
In CI, each scenario is executed via Github Actions and has been configured using
environment variable inputs that follow the ENOS_VAR_varname pattern.
For local execution you can specify all the required variables using environment
variables, or you can update enos.vars.hcl with values and uncomment the lines.
Variables that are required:
aws_ssh_keypair_nameaws_ssh_private_key_pathvault_bundle_pathvault_license_path(only required for non-OSS editions)
See enos.vars.hcl or enos-variables.hcl
for further descriptions of the variables.
Additional variable information can also be found in the Scenario Outlines
Scenario Outlines
Enos is capable of producing an outline of each scenario that is defined in a given directory. These
scenarios often include a description of what behavior the scenario performs, which variants are
available, and which variables are required. They also provide a step by step breakdown including
which quality requirments are verifiend by a given step.
You can generate outlines of all scenarios or specify one via it's name.
From the enos directory:
enos scenario outline smokeThere are also HTML versions available for an improved reading experience:
enos scenario outline --format html > index.html
open index.htmlExecuting Scenarios
From the enos directory:
# List all available scenarios
enos scenario list
# Run the smoke or upgrade scenario with an artifact that is built locally. Make sure
# the local machine has been configured as detailed in the requirements
# section. This will execute the scenario and clean up any resources if successful.
enos scenario run smoke artifact_source:local
enos scenario run upgrade artifact_source:local
# To run the same scenario variants that are run in CI, refer to the scenarios listed
# in json files under .github/enos-run-matrices directory,
# adding `artifact_source:local` to run locally.
enos scenario run smoke backend:consul consul_version:1.12.3 distro:ubuntu seal:awskms artifact_source:local arch:amd64 edition:oss
# Launch an individual scenario but leave infrastructure up after execution
enos scenario launch smoke artifact_source:local
# Check an individual scenario for validity. This is useful during scenario
# authoring and debugging.
enos scenario validate smoke artifact_source:local
# If you've run the tests and desire to see the outputs, such as the URL or
# credentials, you can run the output command to see them. Please note that
# after "run" or destroy there will be no "outputs" as the infrastructure
# will have been destroyed and state cleared.
enos scenario output smoke artifact_source:local
# Explicitly destroy all existing infrastructure
enos scenario destroy smoke artifact_source:localRefer to the Enos documentation
for further information regarding installation, execution or composing scenarios.
UI Tests
The ui scenario creates a Vault cluster (deployed to AWS) using a version
built from the current checkout of the project. Once the cluster is available the UI acceptance tests
are run in a headless browser.
Variables
In addition to the required variables that must be set, as described in the [Scenario Variables](#Scenario Variables),
the ui scenario has two optional variables:
ui_test_filter - An optional test filter to limit the tests that are run, i.e. '!enterprise'.
To set a filter export the variable as follows:
> export ENOS_VAR_ui_test_filter="some filter"ui_run_tests - An optional boolean variable to run or not run the tests. The default value is true.
Setting this value to false is useful in the case where you want to create a cluster, but run the tests
manually. The section [Running the Tests](#Running the Tests) describes the different ways to run the
'UI' acceptance tests.
Running the Tests
The UI tests can be run fully automated or manually.
Fully Automated
The following will deploy the cluster, run the tests, and subsequently tear down the cluster:
> export ENOS_VAR_ui_test_filter="some filter" # <-- optional
> cd enos
> enos scenario ui run edition:ossManually
The UI tests can be run manually as follows:
> export ENOS_VAR_ui_test_filter="some filter" # <-- optional
> export ENOS_VAR_ui_run_tests=false
> cd enos
> enos scenario ui launch edition:oss
# once complete the scenario will output a set of environment variables that must be exported. The
# output will look as follows:
export TEST_FILTER='some filter>' \
export VAULT_ADDR='http://<some ip address>:8200' \
export VAULT_TOKEN='<some token>' \
export VAULT_UNSEAL_KEYS='["<some key>","<some key>","<some key>"]'
# copy and paste the above into the terminal to export the values
> cd ../ui
> yarn test:enos # run headless
# or
> yarn test:enos -s # run manually in a web browser
# once testing is complete
> cd ../enos
> enos scenario ui destroy edition:ossVariants
Both scenarios support a matrix of variants. In order to achieve broad coverage while
keeping test run time reasonable, the variants executed by the enos-run Github
Actions are tailored to maximize variant distribution per scenario.
`artifact_source:crt`
This variant is designed for use in Github Actions. The enos-run.yml workflow
downloads the artifact built by the build.yml workflow, unzips it, and sets thevault_bundle_path to the zip file and the vault_local_binary_path to the binary.
`artifact_source:local`
This variant is for running the Enos scenario locally. It builds the Vault bundle
from the current branch, placing the bundle at the vault_bundle_path and the
unzipped Vault binary at the vault_local_binary_path.
`artifact_source:artifactory`
This variant is for running the Enos scenario to test an artifact from Artifactory. It requires following Enos variables to be set:
artifactory_tokenaws_ssh_keypair_nameaws_ssh_private_key_pathvault_product_versionvault_revision
CI Bootstrap
In order to execute any of the scenarios in this repository, it is first necessary to bootstrap the
CI AWS account with the required permissions, service quotas and supporting AWS resources. There are
two Terraform modules which are used for this purpose, service-user-iam for
the account permissions, and service quotas and bootstrap for the supporting resources.
Supported Regions - enos scenarios are supported in the following regions:"us-east-1", "us-east-2", "us-west-1", "us-west-2"
Bootstrap Process
These steps should be followed to bootstrap this repo for enos scenario execution:
Set up CI service user IAM role and Service Quotas
The service user that is used when executing enos scenarios from any GitHub Action workflow must have
a properly configured IAM role granting the access required to create resources in AWS. Additionally,
service quotas need to be adjusted to ensure that normal use of the ci account does not cause any
service quotas to be exceeded. The service-user-iam module contains the IAM
Policy and Role for that grants this access as well as the service quota increase requests to adjust
the service quotas. This module should be updated whenever a new AWS resource type is required for a
scenario or a service quota limit needs to be increased. Since this is persistent and cannot be created
and destroyed each time a scenario is run, the Terraform state will be managed by Terraform Cloud.
Here are the steps to configure the GitHub Actions service user:
Pre-requisites
- Full access to the CI AWS account is required.
Notes:
- For help with access to Terraform Cloud and the CI Account, contact the Vault Automation team on Slack (#team-vault-automation)
for an invite. After receiving an invite to Terraform Cloud, a personal access token can be created
by clickingUser Settings-->Tokens-->Create an API token. - Access to the AWS account can be done via Doormat, at: https://doormat.hashicorp.services/.
- For the vault repo the account is:
vault_ciand for the vault-enterprise repo, the account is:vault-enterprise_ci. - Access can be requested by clicking:
Cloud Access-->AWS-->Request Account Access.
- For the vault repo the account is:
Create the Terraform Cloud Workspace - The name of the workspace to be created depends on the
repository for which it is being created, but the pattern is:<repository>-ci-enos-service-user-iam,
e.g.vault-ci-enos-service-user-iam. It is important that the execution mode for the workspace be set
tolocal. For help on setting up the workspace, contact the Vault Automation team on Slack (#team-vault-automation)Execute the Terraform module
> cd ./enos/ci/service-user-iam
> export TF_WORKSPACE=<repo name>-ci-enos-service-user-iam
> export TF_TOKEN_app_terraform_io=<Terraform Cloud Token>
> export TF_VAR_repository=<repository name>
> terraform init
> terraform plan
> terraform apply -auto-approveBootstrap the CI resources
Bootstrapping of the resources in the CI account is accomplished via the GitHub Actions workflow:
enos-bootstrap-ci. Before this workflow can be run a
workspace must be created as follows:
- Create the Terraform Cloud Workspace - The name workspace to be created depends on the repository
for which it is being created, but the pattern is:<repository>-ci-bootstrap, e.g.vault-ci-bootstrap. It is important that the execution mode for the workspace be set tolocal. For help on setting up the workspace, contact the Vault Automation team on Slack (#team-vault-automation).
Once the workspace has been created, changes to the bootstrap module will automatically be applied via
the GitHub PR workflow. Each time a PR is created for changes to files within that module the module
will be planned via the workflow described above. If the plan is ok and the PR is merged, the module
will automatically be applied via the same workflow.
File: helper/pkcs7/README.md
PKCS7
This code is used to verify PKCS7 signatures for the EC2 auth method. The code
was forked from mozilla-services/pkcs7
and modified for Vault.
File: internalshared/README.md
HashiCorp-internal libs
Do not use these unless you know what you're doing.
These libraries are used by other HashiCorp software to reduce code duplication
and increase consistency. They are not libraries needed by Vault plugins --
those are in the sdk/ module.
There are no compatibility guarantees. Things in here may change or move or
disappear at any time.
If you are a Vault plugin author and think you need a library in here in your
plugin, please open an issue for discussion.
File: physical/foundationdb/README.md
FoundationDB storage backend
Extra steps are required to produce a Vault build containing the FoundationDB
backend; attempts to use the backend on a build produced without following
this procedure will fail with a descriptive error message at runtime.
Installing the Go bindings
Picking a version
The version of the Go bindings and the FoundationDB client library used to
build them must match.
This version will determine the minimum API version that can be used, hence
it should be no higher than the version of FoundationDB used in your cluster,
and must also satisfy the requirements of the backend code.
The minimum required API version for the FoundationDB backend is 520.
Installation
Make sure you have Mono installed (core is enough), then install the
Go bindings using the fdb-go-install.sh script:
$ physical/foundationdb/fdb-go-install.sh install --fdbver x.y.zBy default, if --fdbver x.y.z is not specified, version 5.2.4 will be used.
Building Vault
To build Vault the FoundationDB backend, add FDB_ENABLED=1 when invokingmake, e.g.
$ make dev FDB_ENABLED=1Running tests
Similarly, add FDB_ENABLED=1 to your make invocation when running tests,
e.g.
$ make test TEST=./physical/foundationdb FDB_ENABLED=12. Official Technical Reference & Guides (hashicorp/web-unified-docs)
Web Unified Docs
[!IMPORTANT]
This README is for developers working on the documentation website. If you
want to contribute docs content, refer to the Contribute to HashiCorp
documentation guide.
The project in this repository, hashicorp/web-unified-docs, aims to implement DEVDOT-023: Unified Product Documentation Repository. The RFC for this project was intentionally light on implementation details, in order to foster consensus on the broad direction.
- PR previews: Show broken links in comments for awareness (informational only, don't block PRs)
- Production monitoring: Weekly scans create GitHub issues and send critical alerts to Datadog when users are affected
The weekly broken-link-check-full workflow generates comprehensive broken link reports with prioritization guidance. When contributors create PRs that modify content, the link checker shows any broken links in PR comments with actionable guidance without blocking development.
Quick tips for contributors:
- Fix internal HashiCorp links (high priority)
- Check external docs/API links (medium priority)
- Consider removing unreliable external links (low priority)
For detailed information about the monitoring system, see Broken Link
Monitoring Documentation. The EDU-053: Unified Education Style Guide
for this project was intentionally light on implementation details, in order
to foster consensus on the broad direction.
The existing API (content.hashicorp.com) has endpoints that serve documentation content. You can find the source code in hashicorp/mktg-content-workflows.
The goal of the unified docs API is to host all of HashiCorp's product documentation. The unified docs API will eventually replace the existing content API.
Local development
Requirements
- Docker and Docker Compose (for managing containers)
Quick start
To get a migration preview running, run make from the root of this repo. The make command starts the unified-docs Docker profile that spins up a local instance of unified-devdot-api and dev-portal.
Once this command completes, you can access the following endpoints:
http://localhost:3000 - An instance of the
dev-portalcontainer configured to pull from the experimental docs API (this repo). This image depends on the unified docs API (unified-devdot-api).http://localhost:8080 - An instance of the unified docs API container (this repo -
unified-devdot-api) that serves content from thecontentdirectory. On startup, this container processes the content and assets in/contentintopublic/assetsandpublic/content. In addition, the container also generatesapp/api/docsPaths.jsonandapp/api/versionMetadata.jsonfrom the contents within/content.Use the following example to test this endpoint: http://localhost:8080/api/content/terraform-plugin-framework/doc/latest/plugin/framework
[!NOTE]
The unified docs API container takes time to process the content and assets. You must wait for both theunified-devdot-apianddev-portalcontainers to complete before you can successfully test content in thedev-portalpreview environment (localhost:3000). Visit http://localhost:8080/api/all-docs-paths to verify theunified-devdot-apicontainer is complete.
To spin this down gracefully, run make clean in a separate terminal.
If you wish to remove the local Docker images as well, run make clean CLEAN_OPTION=full.
More commands
The makefile serves as a convenience tool start the local preview. If you need more granular control, the package.json file contains a full list of available commands.
To use these, you will need to intentionally run npm install and npm run prebuild before anything else.
Use npm run coverage to run coverage tests.
Preview environments for unified-docs and dev-portal
Unified docs API serves as one of the content APIs for dev-portal (frontend application for DevDot). As a result, when implementing new features, you may need to modify both the backend (this repo) and the frontend (dev-portal).
If you are working on a ticket that requires changes to both the unified docs API and dev-portal, please set custom environment variables for your branch in Vercel to simplify testing instructions.
For example, in Vercel, for your dev-portal branch, you can set the following environment variables:
| Environment variable | Value |
|---|---|
HASHI_ENV |
unified-docs-sandbox |
UNIFIED_DOCS_API |
`` |
Vercel will use these values to create deploy previews.
API development
Reach out to team #team-web-presence if you need to do local API development
Background
Project Rationale
- Storing documentation in one branch of one repo dramatically simplifies the workflow for contributing documentation.
- Publishing changes to multiple versions can be done in a single PR, as opposed to multiple PRs which is required by the current setup.
- Finding and making the same change across multiple versions is as simple as doing a find-and-replace since all the versioned docs are on the filesystem at the same time.
- Adding a new product is as easy as making a new folder, as opposed to the current process which requires code-changes on the API side and the installation of a GitHub App to monitor for events.
- Sourcing from one branch in one repo eliminates the situation where a missed GitHub event can result in out-of-date documentation. If something goes wrong in the publishing process, simply run it again instead of relying on incoming commit/release events from the GitHub API.
- Since we can make edits to all docs for all products and versions from a single PR, making platform-level changes is dramatically simplified (such as updating to MDX v2, or rewriting URLs).
- Adding new features like content conformance (basically linting for docs) can be done for the entire codebase at once.
- Removes the ability for docs to break the release workflow in product repos.
- Enables us to support fully versioned deployment previews, whereas current previews are limited to the branch being modified.
[Architectural Decision Records](https://github.com/hashicorp/web-unified-docs/tree/main/docs/decisions)
Update product repo documentation
This script helps with product documentation migration to the web-unified-docs repository. When migrating documentation:
- The
web-unified-docsrepository becomes the source of truth - Original documentation may remain temporarily as a fallback
- Users should be directed to make future changes in
web-unified-docsonly
This script automatically adds a prominent notice to all MDX files in the original location, informing contributors where to make future changes.
./scripts/update-mdx-files.sh ~/Desktop/hashicorp/terraform-plugin-framework/website/docsExample output:
Progress:
Files processed: 135
Files updated: 135
Files with no frontmatter: 0
Files with errors: 0
Completed! All MDX files have been processed.Run broken link checker locally
The repository uses a focused broken link monitoring system:
- PR previews: Show broken links in comments for awareness (informational only, don't block PRs)
- Production monitoring: Weekly scans create GitHub issues and Datadog alerts for user-facing problems
The weekly broken-link-check-full workflow generates comprehensive broken link reports. When contributors create PRs that modify content, the link checker shows any broken links in PR comments without blocking development.
For detailed information about the monitoring system, see Broken Link Monitoring Documentation.
UDR Migration Link Check
For teams migrating products to UDR (Unified Docs Renderer), use the dedicated migration workflow:
- Go to Actions → UDR Product Link Check
- Click "Run workflow" and select your product
- Review migration-specific broken link analysis in the generated GitHub issue
This workflow provides targeted link checking with migration-focused reporting and prioritization.
Local Testing
You can also run the broken link checker locally. The following commands launch a lychee Docker container to check the content directories you specify.
Run the broken link checker on all content.
npm run broken-linkCheck a specific directory within content.
npm run broken-link terraform-plugin-frameworkCheck multiple directories.
npm run broken-link terraform-plugin-framework-log terraform-plugin-muxArchitecture
The following diagram illustrates the relationships between the unified docs API (this repo), dev-portal, and the existing content API:
graph LR
subgraph "Content sources (non-migrated)"
BDY[boundary]
CSL[consul]
HCP[hcp-docs]
NMD[nomad]
PKR[packer]
SNT[sentinel]
TF[terraform]
TFC[terraform-cdk]
TFA[terraform-docs-agents]
TFD[terraform-docs-common]
VGT[vagrant]
VLT[vault]
WPT[waypoint]
CURALL["/content or /website"]
BDY & CSL & HCP & NMD & PKR & SNT & TF & TFC & TFA & TFD & VGT & VLT & WPT --> CURALL
end
subgraph "Migrated content repo"
TPF[terraform-plugin-framework]
TPL[terraform-plugin-log]
TPM[terraform-plugin-mux]
TPS[terraform-plugin-sdk]
TPT[terraform-plugin-testing]
TFE[terraform-enterprise]
MIGALL["/content"]
TPF & TPL & TPM & TPS & TPT & TFE --> MIGALL
end
subgraph "APIs"
CP[Content APIcontent.hashicorp.com]
UDR[Unified Docs Repositoryweb-unified-docs]
end
subgraph "Frontend"
DP[Dev Portaldev-portal]
end
%% BDY & CSL & HCP & NMD & PKR & PTF & SNT & TF & TFC & TFA & TFD & VGT & VLT & WPT --> CP
%% TPF & TPL & TPM & TPS & TPT --> UDR
CURALL -->|Current content flow| CP
MIGALL -->|Migrated content| UDR
CP -->|Serves most content| DP
UDR -->|Serves unified/content content| DP
class TPF,TPL,TPM,TPS,TPT,BDY,CSL,HCP,NMD,PKR,PTF,SNT,TF,TFC,TFA,TFD,VGT,VLT,WPT productRepoThe diagram shows:
- The content API — the existing system that sources product documentation content from product repositories
- The unified docs API — the new system that sources product documentation from this repo's
/contentdirectory. The migrated repos will use a directory approach to versioning (rather than the historic branch and tag strategy) - The Dev Portal — the frontend that serves the main DevDot interface. Dev Portal sources its content from both the existing content API and unified docs API.