## 1. Project Overview & Quickstart (hashicorp/terraform) ## File: README.md # Terraform - Website: https://developer.hashicorp.com/terraform - Forums: [HashiCorp Discuss](https://discuss.hashicorp.com/c/terraform-core) - Documentation: [https://developer.hashicorp.com/terraform/docs](https://developer.hashicorp.com/terraform/docs) - Tutorials: [HashiCorp's Learn Platform](https://developer.hashicorp.com/terraform/tutorials) - Certification Exam: [HashiCorp Certified: Terraform Associate](https://www.hashicorp.com/certification/#hashicorp-certified-terraform-associate) Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. Terraform can manage existing and popular service providers as well as custom in-house solutions. The key features of Terraform are: - **Infrastructure as Code**: Infrastructure is described using a high-level configuration syntax. This allows a blueprint of your datacenter to be versioned and treated as you would any other code. Additionally, infrastructure can be shared and re-used. - **Execution Plans**: Terraform has a "planning" step where it generates an execution plan. The execution plan shows what Terraform will do when you call apply. This lets you avoid any surprises when Terraform manipulates infrastructure. - **Resource Graph**: Terraform builds a graph of all your resources, and parallelizes the creation and modification of any non-dependent resources. Because of this, Terraform builds infrastructure as efficiently as possible, and operators get insight into dependencies in their infrastructure. - **Change Automation**: Complex changesets can be applied to your infrastructure with minimal human interaction. With the previously mentioned execution plan and resource graph, you know exactly what Terraform will change and in what order, avoiding many possible human errors. For more information, refer to the [What is Terraform?](https://www.terraform.io/intro) page on the Terraform website. ## Getting Started & Documentation Documentation is available on the [Terraform website](https://developer.hashicorp.com/terraform): - [Introduction](https://developer.hashicorp.com/terraform/intro) - [Documentation](https://developer.hashicorp.com/terraform/docs) If you're new to Terraform and want to get started creating infrastructure, please check out our [Getting Started guides](https://learn.hashicorp.com/terraform#getting-started) on HashiCorp's learning platform. There are also [additional guides](https://learn.hashicorp.com/terraform#operations-and-development) to continue your learning. Show off your Terraform knowledge by passing a certification exam. Visit the [certification page](https://www.hashicorp.com/certification/) for information about exams and find [study materials](https://learn.hashicorp.com/terraform/certification/terraform-associate) on HashiCorp's learning platform. ## Developing Terraform This repository contains only Terraform core, which includes the command line interface and the main graph engine. Providers are implemented as plugins, and Terraform can automatically download providers that are published on [the Terraform Registry](https://registry.terraform.io). HashiCorp develops some providers, and others are developed by other organizations. For more information, refer to [Plugin development](https://developer.hashicorp.com/terraform/plugin). - To learn more about compiling Terraform and contributing suggested changes, refer to [the contributing guide](.github/CONTRIBUTING.md). - To learn more about how we handle bug reports, refer to the [bug triage guide](./BUGPROCESS.md). - To learn how to contribute to the Terraform documentation, refer to the [Web Unified Docs repository](https://github.com/hashicorp/web-unified-docs). ## License [Business Source License 1.1](https://github.com/hashicorp/terraform/blob/main/LICENSE) --- ## File: docs/plugin-protocol/object-wire-format.md # Wire Format for Terraform Objects and Associated Values The provider wire protocol (as of major version 5) includes a protobuf message type `DynamicValue` which Terraform uses to represent values from the Terraform Language type system, which result from evaluating the content of `resource`, `data`, and `provider` blocks, based on a schema defined by the corresponding provider. Because the structure of these values is determined at runtime, `DynamicValue` uses one of two possible dynamic serialization formats for the values themselves: MessagePack or JSON. Terraform most commonly uses MessagePack, because it offers a compact binary representation of a value. However, a server implementation of the provider protocol should fall back to JSON if the MessagePack field is not populated, in order to support both formats. The remainder of this document describes how Terraform translates from its own type system into the type system of the two supported serialization formats. A server implementation of the Terraform provider protocol can use this information to decode `DynamicValue` values from incoming messages into whatever representation is convenient for the provider implementation. A server implementation must also be able to _produce_ `DynamicValue` messages as part of various response messages. When doing so, servers should always use MessagePack encoding, because Terraform does not consistently support JSON responses across all request types and all Terraform versions. Both the MessagePack and JSON serializations are driven by information the provider previously returned in a `Schema` message. Terraform will encode each value depending on the type constraint given for it in the corresponding schema, using the closest possible MessagePack or JSON type to the Terraform language type. Therefore a server implementation can decode a serialized value using a standard MessagePack or JSON library and assume it will conform to the serialization rules described below. ## MessagePack Serialization Rules The MessagePack types referenced in this section are those defined in [The MessagePack type system specification](https://github.com/msgpack/msgpack/blob/master/spec.md#type-system). Note that MessagePack defines several possible serialization formats for each type, and Terraform may choose any of the formats of a specified type. The exact serialization chosen for a given value may vary between Terraform versions, but the types given here are contractual. Conversely, server implementations that are _producing_ MessagePack-encoded values are free to use any of the valid serialization formats for a particular type. However, we recommend choosing the most compact format that can represent the value without a loss of range. ### `Schema.Block` Mapping Rules for MessagePack To represent the content of a block as MessagePack, Terraform constructs a MessagePack map that contains one key-value pair per attribute and one key-value pair per distinct nested block described in the `Schema.Block` message. The key-value pairs representing attributes have values based on [the `Schema.Attribute` mapping rules](#Schema.Attribute-mapping-rules-for-messagepack). The key-value pairs representing nested block types have values based on [the `Schema.NestedBlock` mapping rules](#Schema.NestedBlock-mapping-rules-for-messagepack). ### `Schema.Attribute` Mapping Rules for MessagePack The MessagePack serialization of an attribute value depends on the value of the `type` field of the corresponding `Schema.Attribute` message. The `type` field is a compact JSON serialization of a [Terraform type constraint](https://developer.hashicorp.com/terraform/language/expressions/type-constraints), which consists either of a single string value (for primitive types) or a two-element array giving a type kind and a type argument. The following table describes the type-specific mapping rules. Along with those type-specific rules there are two special rules that override the mappings in the table below, regardless of type: * A null value is represented as a MessagePack nil value. * An unknown value (that is, a placeholder for a value that will be decided only during the apply operation) is represented as a [MessagePack extension](https://github.com/msgpack/msgpack/blob/master/spec.md#extension-types) value, described in more detail below. | `type` Pattern | MessagePack Representation | |---|---| | `"string"` | A MessagePack string containing the Unicode characters from the string value serialized as normalized UTF-8. | | `"number"` | Either MessagePack integer, MessagePack float, or MessagePack string representing the number. If a number is represented as a string then the string contains a decimal representation of the number which may have a larger mantissa than can be represented by a 64-bit float. | | `"bool"` | A MessagePack boolean value corresponding to the value. | | `["list",T]` | A MessagePack array with the same number of elements as the list value, each of which is represented by the result of applying these same mapping rules to the nested type `T`. | | `["set",T]` | Identical in representation to `["list",T]`, but the order of elements is undefined because Terraform sets are unordered. | | `["map",T]` | A MessagePack map with one key-value pair per element of the map value, where the element key is serialized as the map key (always a MessagePack string) and the element value is represented by a value constructed by applying these same mapping rules to the nested type `T`. | | `["object",ATTRS]` | A MessagePack map with one key-value pair per attribute defined in the `ATTRS` object. The attribute name is serialized as the map key (always a MessagePack string) and the attribute value is represented by a value constructed by applying these same mapping rules to each attribute's own type. | | `["tuple",TYPES]` | A MessagePack array with one element per element described by the `TYPES` array. The element values are constructed by applying these same mapping rules to the corresponding element of `TYPES`. | | `"dynamic"` | A MessagePack array with exactly two elements. The first element is a MessagePack binary value containing a JSON-serialized type constraint in the same format described in this table. The second element is the result of applying these same mapping rules to the value with the type given in the first element. This special type constraint represents values whose types will be decided only at runtime. | Unknown values have two possible representations, both using [MessagePack extension](https://github.com/msgpack/msgpack/blob/master/spec.md#extension-types) values. The older encoding is for unrefined unknown values and uses an extension code of zero, with the extension value payload completely ignored. Newer Terraform versions can produce "refined" unknown values which carry some additional information that constrains the possible range of the final value/ Refined unknown values have extension code 12 and then the extension object's payload is a MessagePack-encoded map using integer keys to represent different kinds of refinement: * `1` represents "nullness", and the value of that key will be a boolean value that is true if the value is definitely null or false if it is definitely not null. If this key isn't present at all then the value may or may not be null. It's not actually useful to encode that an unknown value is null; use a known null value instead in that case, because there is only one null value of each type. * `2` represents string prefix, and the value is a string that the final value is known to begin with. This is valid only for unknown values of string type. * `3` and `4` represent the lower and upper bounds respectively of a number value, and the value of both is a two-element msgpack array whose first element is a valid encoding of a number (as in the table above) and whose second element is a boolean value that is true for an inclusive bound and false for an exclusive bound. This is valid only for unknown values of number type. * `5` and `6` represent the lower and upper bounds respectively of the length of a collection value. The value of both is an integer representing an inclusive bound. This is valid only for unknown values of the three kinds of collection types: list, set, and map. Unknown value refinements are an optional way to reduce the range of possible values for situations where that makes it possible to produce a known result for unknown inputs or where it allows detecting an error during the planning phase that would otherwise be detected only during the apply phase. It's always safe to ignore refinements and just treat an unknown value as wholly unknown, but considering refinements may allow a more precise answer. A provider that produces refined values in its planned new state (from `PlanResourceChange`) _must_ honor those refinements in the final state (from `ApplyResourceChange`). Unmarshalling code should ignore refinement map keys that they don't know about, because future versions of the protocol might define additional refinements. When encoding an unknown value without any refinements, always use the older format with extension code zero instead of using extension code 12 with an empty refinement map. Any refined unknown value _must_ have at least one refinement map entry. This rule ensures backward compatibility with older implementations that predate the value refinements concept. A server implementation of the protocol should treat _any_ MessagePack extension code as representing an unknown value, but should ignore the payload of that extension value entirely unless the extension code is 12 to indicate that the body represents refinements. Future versions of this protocol may define specific formats for other extension codes, but they will always represent unknown values. ### `Schema.NestedBlock` Mapping Rules for MessagePack The MessagePack serialization of a collection of blocks of a particular type depends on the `nesting` field of the corresponding `Schema.NestedBlock` message. The `nesting` field is a value from the `Schema.NestingBlock.NestingMode` enumeration. All `nesting` values cause the individual blocks of a type to be represented by applying [the `Schema.Block` mapping rules](#Schema.Block-mapping-rules-for-messagepack) to the block's contents based on the `block` field, producing what we'll call a _block value_ in the table below. The `nesting` value then in turn defines how Terraform will collect all of the individual block values together to produce a single property value representing the nested block type. For all `nesting` values other than `MAP`, blocks may not have any labels. For the `nesting` value `MAP`, blocks must have exactly one label, which is a string we'll call a _block label_ in the table below. | `nesting` Value | MessagePack Representation | |---|---| | `SINGLE` | The block value of the single block of this type, or nil if there is no block of that type. | | `LIST` | A MessagePack array of all of the block values, preserving the order of definition of the blocks in the configuration. | | `SET` | A MessagePack array of all of the block values in no particular order. | | `MAP` | A MessagePack map with one key-value pair per block value, where the key is the block label and the value is the block value. | | `GROUP` | The same as with `SINGLE`, except that if there is no block of that type Terraform will synthesize a block value by pretending that all of the declared attributes are null and that there are zero blocks of each declared block type. | For the `LIST` and `SET` nesting modes, Terraform guarantees that the MessagePack array will have a number of elements between the `min_items` and `max_items` values given in the schema, _unless_ any of the block values contain nested unknown values. When unknown values are present, Terraform considers the value to be potentially incomplete and so Terraform defers validation of the number of blocks. For example, if the configuration includes a `dynamic` block whose `for_each` argument is unknown then the final number of blocks is not predictable until the apply phase. ## JSON Serialization Rules The JSON serialization is a secondary representation for `DynamicValue`, with MessagePack preferred due to its ability to represent unknown values via an extension. The JSON encoding described in this section is also used for the `json` field of the `RawValue` message that forms part of an `UpgradeResourceState` request. However, in that case the data is serialized per the schema of the provider version that created it, which won't necessarily match the schema of the _current_ version of that provider. ### `Schema.Block` Mapping Rules for JSON To represent the content of a block as JSON, Terraform constructs a JSON object that contains one property per attribute and one property per distinct nested block described in the `Schema.Block` message. The properties representing attributes have property values based on [the `Schema.Attribute` mapping rules](#Schema.Attribute-mapping-rules-for-json). The properties representing nested block types have property values based on [the `Schema.NestedBlock` mapping rules](#Schema.NestedBlock-mapping-rules-for-json). ### `Schema.Attribute` Mapping Rules for JSON The JSON serialization of an attribute value depends on the value of the `type` field of the corresponding `Schema.Attribute` message. The `type` field is a compact JSON serialization of a [Terraform type constraint](https://developer.hashicorp.com/terraform/language/expressions/type-constraints), which consists either of a single string value (for primitive types) or a two-element array giving a type kind and a type argument. The following table describes the type-specific mapping rules. Along with those type-specific rules there is one special rule that overrides the rules in the table regardless of type: * A null value is always represented as JSON `null`. | `type` Pattern | JSON Representation | |---|---| | `"string"` | A JSON string containing the Unicode characters from the string value. | | `"number"` | A JSON number representing the number value. Terraform numbers are arbitrary-precision floating point, so the value may have a larger mantissa than can be represented by a 64-bit float. | | `"bool"` | Either JSON `true` or JSON `false`, depending on the boolean value. | | `["list",T]` | A JSON array with the same number of elements as the list value, each of which is represented by the result of applying these same mapping rules to the nested type `T`. | | `["set",T]` | Identical in representation to `["list",T]`, but the order of elements is undefined because Terraform sets are unordered. | | `["map",T]` | A JSON object with one property per element of the map value, where the element key is serialized as the property name string and the element value is represented by a property value constructed by applying these same mapping rules to the nested type `T`. | | `["object",ATTRS]` | A JSON object with one property per attribute defined in the `ATTRS` object. The attribute name is serialized as the property name string and the attribute value is represented by a property value constructed by applying these same mapping rules to each attribute's own type. | | `["tuple",TYPES]` | A JSON array with one element per element described by the `TYPES` array. The element values are constructed by applying these same mapping rules to the corresponding element of `TYPES`. | | `"dynamic"` | A JSON object with two properties: `"type"` specifying one of the `type` patterns described in this table in-band, giving the exact runtime type of the value, and `"value"` specifying the result of applying these same mapping rules to the table for the specified runtime type. This special type constraint represents values whose types will be decided only at runtime. | ### `Schema.NestedBlock` Mapping Rules for JSON The JSON serialization of a collection of blocks of a particular type depends on the `nesting` field of the corresponding `Schema.NestedBlock` message. The `nesting` field is a value from the `Schema.NestingBlock.NestingMode` enumeration. All `nesting` values cause the individual blocks of a type to be represented by applying [the `Schema.Block` mapping rules](#Schema.Block-mapping-rules-for-json) to the block's contents based on the `block` field, producing what we'll call a _block value_ in the table below. The `nesting` value then in turn defines how Terraform will collect all of the individual block values together to produce a single property value representing the nested block type. For all `nesting` values other than `MAP`, blocks may not have any labels. For the `nesting` value `MAP`, blocks must have exactly one label, which is a string we'll call a _block label_ in the table below. | `nesting` Value | JSON Representation | |---|---| | `SINGLE` | The block value of the single block of this type, or `null` if there is no block of that type. | | `LIST` | A JSON array of all of the block values, preserving the order of definition of the blocks in the configuration. | | `SET` | A JSON array of all of the block values in no particular order. | | `MAP` | A JSON object with one property per block value, where the property name is the block label and the value is the block value. | | `GROUP` | The same as with `SINGLE`, except that if there is no block of that type Terraform will synthesize a block value by pretending that all of the declared attributes are null and that there are zero blocks of each declared block type. | For the `LIST` and `SET` nesting modes, Terraform guarantees that the JSON array will have a number of elements between the `min_items` and `max_items` values given in the schema. --- ## File: docs/plugin-protocol/README.md # Terraform Plugin Protocol This directory contains documentation about the physical wire protocol that Terraform Core uses to communicate with provider plugins. Most providers are not written directly against this protocol. Instead, prefer to use an SDK that implements this protocol and write the provider against the SDK's API. ---- **If you want to write a plugin for Terraform, please refer to [Plugin development](https://developer.hashicorp.com/terraform/plugin) instead.** This documentation is for those who are developing _Terraform SDKs_, rather than those implementing plugins. ---- From Terraform v0.12.0 onwards, Terraform's plugin protocol is built on [gRPC](https://grpc.io/). This directory contains `.proto` definitions of different versions of Terraform's protocol. Only `.proto` files published as part of Terraform release tags are actually official protocol versions. If you are reading this directory on the `main` branch or any other development branch then it may contain protocol definitions that are not yet finalized and that may change before final release. ## RPC Plugin Model Terraform plugins are normal executable programs that, when launched, expose gRPC services on a server accessed via the loopback interface. Terraform Core discovers and launches plugins, waits for a handshake to be printed on the plugin's `stdout`, and then connects to the indicated port number as a gRPC client. For this reason, we commonly refer to Terraform Core itself as the plugin "client" and the plugin program itself as the plugin "server". Both of these processes run locally, with the server process appearing as a child process of the client. Terraform Core controls the lifecycle of these server processes and will terminate them when they are no longer required. The startup and handshake protocol is not currently documented. We hope to document it here or to link to external documentation on it in future. ## Versioning Strategy The Plugin Protocol uses a versioning strategy that aims to allow gradual enhancements to the protocol while retaining compatibility, but also to allow more significant breaking changes from time to time while allowing old and new plugins to be used together for some period. The versioning strategy described below was introduced with protocol version 5.0 in Terraform v0.12. Prior versions of Terraform and prior protocol versions do not follow this strategy. The authoritative definition for each protocol version is in this directory as a Protocol Buffers (protobuf) service definition. The files follow the naming pattern `tfpluginX.proto`, where X is the major version. ### Major and minor versioning The minor version increases for each change introducing optional new functionality that can be ignored by implementations of prior versions. For example, if a new field were added to an response message, it could be a minor release as long as Terraform Core can provide some default behavior when that field is not populated. The major version increases for any significant change to the protocol where compatibility is broken. However, Terraform Core and an SDK may both choose to support multiple major versions at once: the plugin handshake includes a negotiation step where client and server can work together to select a mutually-supported major version. The major version number is encoded into the protobuf package name: major version 5 uses the package name `tfplugin5`, and one day major version 6 will switch to `tfplugin6`. This change of name allows a plugin server to implement multiple major versions at once, by exporting multiple gRPC services. Minor version differences rely instead on feature-detection mechanisms, so they are not represented directly on the wire and exist primarily as a human communication tool to help us easily talk about which software supports which features. ## Version compatibility for Core, SDK, and Providers A particular version of Terraform Core has both a minimum minor version it requires and a maximum major version that it supports. A particular version of Terraform Core may also be able to optionally use a newer minor version when available, but fall back on older behavior when that functionality is not available. Likewise, each provider plugin release is compatible with a set of versions. The compatible versions for a provider are a list of major and minor version pairs, such as "4.0", "5.2", which indicates that the provider supports the baseline features of major version 4 and supports major version 5 including the enhancements from both minor versions 1 and 2. This provider would therefore be compatible with a Terraform Core release that supports only protocol version 5.0, since major version 5 is supported and the optional 5.1 and 5.2 enhancements will be ignored. If Terraform Core and the plugin do not have at least one mutually-supported major version, Terraform Core will return an error from `terraform init` during plugin installation: ``` Provider "aws" v1.0.0 is not compatible with Terraform v0.12.0. Provider version v2.0.0 is the earliest compatible version. Select it with the following version constraint: version = "~> 2.0.0" ``` ``` Provider "aws" v3.0.0 is not compatible with Terraform v0.12.0. Provider version v2.34.0 is the latest compatible version. Select it with the following constraint: version = "~> 2.34.0" Alternatively, upgrade to the latest version of Terraform for compatibility with newer provider releases. ``` The above messages are for plugins installed via `terraform init` from a Terraform registry, where the registry API allows Terraform Core to recognize the protocol compatibility for each provider release. For plugins that are installed manually to a local plugin directory, Terraform Core has no way to suggest specific versions to upgrade or downgrade to, and so the error message is more generic: ``` The installed version of provider "example" is not compatible with Terraform v0.12.0. This provider was loaded from: /usr/local/bin/terraform-provider-example_v0.1.0 ``` ## Adding/removing major version support in SDK and Providers The set of supported major versions is decided by the SDK used by the plugin. Over time, SDKs will add support for new major versions and phase out support for older major versions. In doing so, the SDK developer passes those capabilities and constraints on to any provider using their SDK, and that will in turn affect the compatibility of the plugin in ways that affect its semver-based version numbering: - If an SDK upgrade adds support for a new provider protocol, that will usually be considered a new feature and thus warrant a new minor version. - If an SDK upgrade removes support for an old provider protocol, that is always a breaking change and thus requires a major release of the provider. For this reason, SDK developers must be clear in their release notes about the addition and removal of support for major versions. Terraform Core also makes an assumption about major version support when it produces actionable error messages for users about incompatibilities: a particular protocol major version is supported for a single consecutive range of provider releases, with no "gaps". ## Using the protobuf specifications in an SDK If you wish to build an SDK for Terraform plugins, an early step will be to copy one or more `.proto` files from this directory into your own repository (depending on which protocol versions you intend to support) and use the `protoc` protocol buffers compiler (with gRPC extensions) to generate suitable RPC stubs and types for your target language. For example, if you happen to be targeting Python, you might generate the stubs using a command like this: ``` protoc --python_out=. --grpc_python_out=. tfplugin5.1.proto ``` You can find out more about the tool usage for each target language in [the gRPC Quick Start guides](https://grpc.io/docs/quickstart/). The protobuf specification for a version is immutable after it has been included in at least one Terraform release. Any changes will be documented in a new `.proto` file establishing a new protocol version. The protocol buffer compiler will produce some sort of library object appropriate for the target language, which depending on the language might be called a module, or a package, or something else. We recommend to include the protocol major version in your module or package name so that you can potentially support multiple versions concurrently in future. For example, if you are targeting major version 5 you might call your package or module `tfplugin5`. To upgrade to a newer minor protocol version, copy the new `.proto` file from this directory into the same location as your previous version, delete the previous version, and then run the protocol buffers compiler again against the new `.proto` file. Because minor releases are backward-compatible, you can simply update your previous stubs in-place rather than creating a new set alongside. To support a new _major_ protocol version, create a new package or module and copy the relevant `.proto` file into it, creating a separate set of stubs that can in principle allow your SDK to support both major versions at the same time. We recommend supporting both the previous and current major versions together for a while across a major version upgrade so that users can avoid having to upgrade both Terraform Core and all of their providers at the same time, but you can delete the previous major version stubs once you remove support for that version. **Note:** Some of the `.proto` files contain statements about being updated in-place for minor versions. This reflects an earlier version management strategy which is no longer followed. The current process is to create a new file in this directory for each new minor version and consider all previously-tagged definitions as immutable. The outdated comments in those files are retained in order to keep the promise of immutability, even though it is now incorrect. ## Updating the plugin protocol in Terraform core > Note: This section's audience is contributors to Terraform, not developers creating a new SDK. New features added to Terraform core often require an update to the plugin protocol. These changes are reflected in a new minor version of the plugin protocol. A new minor release of Terraform should only introduce one new minor release of the plugin-protocol. The minor release numbers of the plugin protocol and Terraform do not match, but they should have a one-to-one relationship. ### Add updates to a new plugin protocol minor version 1) Edit the `.proto` files for Protocol 5 and 6 in `docs/plugin-protocol/`. * If this is the first bigger change after a Terraform release, you may want to increase the minor protocol version in the file header 1) Commit your changes. 1) Run `make protobuf`. * This will use the symlinks in the `internal/tfplugin*` directories to access the latest minor versions' `.proto` files. * You should see diffs in `internal/tfplugin5/tfplugin5.pb.go` and `internal/tfplugin6/tfplugin6.pb.go`. 1) Run `make generate`. * You should see diffs in `internal/plugin/mock_proto/mock.go` and `internal/plugin6/mock_proto/mock.go`. --- ## File: docs/plugin-protocol/releasing-new-version.md # Releasing a New Version of the Protocol Terraform's plugin protocol is the contract between Terraform's plugins and Terraform, and as such releasing a new version requires some coordination between those pieces. This document is intended to be a checklist to consult when adding a new major version of the protocol (X in X.Y) to ensure that everything that needs to be is aware of it. ## New Protobuf File The protocol is defined in protobuf files that live in the hashicorp/terraform repository. Adding a new version of the protocol involves creating a new `.proto` file in that directory. It is recommended that you copy the latest protocol file, and modify it accordingly. ## New terraform-plugin-go Package The [hashicorp/terraform-plugin-go](https://github.com/hashicorp/terraform-plugin-go) repository serves as the foundation for Terraform's plugin ecosystem. It needs to know about the new major protocol version. Either open an issue in that repo to have the Plugin SDK team add the new package, or if you would like to contribute it yourself, open a PR. It is recommended that you copy the package for the latest protocol version and modify it accordingly. ## Update the Registry's List of Allowed Versions The Terraform Registry validates the protocol versions a provider advertises support for when ingesting providers. Providers will not be able to advertise support for the new protocol version until it is added to that list. ## Update Terraform's Version Constraints Terraform only downloads providers that speak protocol versions it is compatible with from the Registry during `terraform init`. When adding support for a new protocol, you need to tell Terraform it knows that protocol version. Modify the `SupportedPluginProtocols` variable in hashicorp/terraform's `internal/getproviders/registry_client.go` file to include the new protocol. ## Test Running a Provider With the Test Framework Use the provider test framework to test a provider written with the new protocol. This end-to-end test ensures that providers written with the new protocol work correctly with the test framework, especially in communicating the protocol version between the test framework and Terraform. ## Test Retrieving and Running a Provider From the Registry Publish a provider, either to the public registry or to the staging registry, and test running `terraform init` and `terraform apply`, along with exercising any of the new functionality the protocol version introduces. This end-to-end test ensures that all the pieces needing to be updated before practitioners can use providers built with the new protocol have been updated. --- ## File: docs/architecture.md # Terraform Core Architecture Summary This document is a summary of the main components of Terraform Core and how data and requests flow between these components. It's intended as a primer to help navigate the codebase to dig into more details. We assume some familiarity with user-facing Terraform concepts like configuration, state, CLI workflow, etc. The Terraform website has documentation on these ideas. **Links to Go documentation assume you are running [`pkgsite`](https://pkg.go.dev/golang.org/x/pkgsite/cmd/pkgsite) locally, with the default URL of localhost:8080.** ## Terraform Request Flow The following diagram shows an approximation of how a user command is executed in Terraform: Each of the different subsystems (solid boxes) in this diagram is described in more detail in a corresponding section below. ## CLI (`command` package) Each time a user runs the `terraform` program, aside from some initial bootstrapping in the root package (not shown in the diagram) execution transfers immediately into one of the "command" implementations in [the `command` package](http://localhost:8080/github.com/hashicorp/terraform/internal/command). The mapping between the user-facing command names and their corresponding `command` package types can be found in the `commands.go` file in the root of the repository. The full flow illustrated above does not actually apply to _all_ commands, but it applies to the main Terraform workflow commands `terraform plan` and `terraform apply`, along with a few others. For these commands, the role of the command implementation is to read and parse any command line arguments, command line options, and environment variables that are needed for the given command and use them to produce a [`backendrun.Operation`](http://localhost:8080/github.com/hashicorp/terraform/internal/backend/backendrun#Operation) object that describes an action to be taken. An _operation_ consists of: * The action to be taken (e.g. "plan", "apply"). * The name of the [workspace](https://developer.hashicorp.com/terraform/language/state/workspaces) where the action will be taken. * Root module input variables to use for the action. * For the "plan" operation, a path to the directory containing the configuration's root module. * For the "apply" operation, the plan to apply. * Various other less-common options/settings such as `-target` addresses, the "force" flag, etc. The operation is then passed to the currently-selected [backend](https://developer.hashicorp.com/terraform/language/backend). Each backend name corresponds to an implementation of [`backend.Backend`](http://localhost:8080/github.com/hashicorp/terraform/internal/backend#Backend), using a mapping table in [the `backend/init` package](http://localhost:8080/github.com/hashicorp/terraform/internal/backend/init). Backends that are able to execute operations additionally implement [`backendrun.OperationsBackend`](http://localhost:8080/github.com/hashicorp/terraform/internal/backend/backendrun#OperationsBackend); the command-handling code calls `Operation` with the operation it has constructed, and then the backend is responsible for executing that action. Backends that execute operations, however, do so as an architectural implementation detail and not a general feature of backends. That is, the term 'backend' as a Terraform feature is used to refer to a plugin that determines where Terraform stores its state snapshots - only the default `local` backend and HCP Terraform's backends (`remote`, `cloud`) perform operations. Thus, most backends do _not_ implement this interface, and so the `command` package wraps these backends in an instance of [`local.Local`](http://localhost:8080/github.com/hashicorp/terraform/internal/backend/local#Local), causing the operation to be executed locally within the `terraform` process itself. ## Backends A _backend_ determines where Terraform should store its state snapshots. As described above, the `local` backend also executes operations on behalf of most other backends. It uses a _state manager_ (either [`statemgr.Filesystem`](http://localhost:8080/github.com/hashicorp/terraform/internal/states/statemgr#Filesystem) if the local backend is being used directly, or an implementation provided by whatever backend is being wrapped) to retrieve the current state for the workspace specified in the operation, then uses the _config loader_ to load and do initial processing/validation of the configuration specified in the operation. It then uses these, along with the other settings given in the operation, to construct a [`terraform.Context`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#Context), which is the main object that actually performs Terraform operations. The `local` backend finally calls an appropriate method on that context to begin execution of the relevant command, such as [`Plan`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#Context.Plan) or [`Apply`](), which in turn constructs a graph using a _graph builder_, described in a later section. ## Configuration Loader The top-level configuration structure is represented by model types in [package `configs`](http://localhost:8080/github.com/hashicorp/terraform/internal/configs). A whole configuration (the root module plus all of its descendant modules) is represented by [`configs.Config`](http://localhost:8080/github.com/hashicorp/terraform/internal/configs#Config). The `configs` package contains some low-level functionality for constructing configuration objects, but the main entry point is in the sub-package [`configload`](http://localhost:8080/github.com/hashicorp/terraform/internal/configs/configload), via [`configload.Loader`](http://localhost:8080/github.com/hashicorp/terraform/internal/configs/configload#Loader). A loader deals with all of the details of installing child modules (during `terraform init`) and then locating those modules again when a configuration is loaded by a backend. It takes the path to a root module and recursively loads all of the child modules to produce a single [`configs.Config`](http://localhost:8080/github.com/hashicorp/terraform/internal/configs#Config) representing the entire configuration. Terraform expects configuration files written in the Terraform language, which is a DSL built on top of [HCL](https://github.com/hashicorp/hcl). Some parts of the configuration cannot be interpreted until we build and walk the graph, since they depend on the outcome of other parts of the configuration, and so these parts of the configuration remain represented as the low-level HCL types [`hcl.Body`](http://localhost:8080/github.com/hashicorp/hcl/v2/#Body) and [`hcl.Expression`](http://localhost:8080/github.com/hashicorp/hcl/v2/#Expression), allowing Terraform to interpret them at a more appropriate time. ## State Manager A _state manager_ is responsible for storing and retrieving snapshots of the [Terraform state](https://developer.hashicorp.com/terraform/language/state) for a particular workspace. Each manager is an implementation of some combination of interfaces in [the `statemgr` package](http://localhost:8080/github.com/hashicorp/terraform/internal/states/statemgr), with most practical managers implementing the full set of operations described by [`statemgr.Full`](http://localhost:8080/github.com/hashicorp/terraform/internal/states/statemgr#Full) provided by a _backend_. The smaller interfaces exist primarily for use in other function signatures to be explicit about what actions the function might take on the state manager; there is little reason to write a state manager that does not implement all of `statemgr.Full`. The implementation [`statemgr.Filesystem`](http://localhost:8080/github.com/hashicorp/terraform/internal/states/statemgr#Filesystem) is used by default (by the `local` backend) and is responsible for the familiar `terraform.tfstate` local file that most Terraform users start with, before they switch to [remote state](https://developer.hashicorp.com/terraform/language/state/remote). Other implementations of `statemgr.Full` are used to implement remote state. Each of these saves and retrieves state via a remote network service appropriate to the backend that creates it. A state manager accepts and returns a state snapshot as a [`states.State`](http://localhost:8080/github.com/hashicorp/terraform/internal/states#State) object. The state manager is responsible for exactly how that object is serialized and stored, but all state managers at the time of writing use the same JSON serialization format, storing the resulting JSON bytes in some kind of arbitrary blob store. ## Graph Builder A _graph builder_ is called by a [`terraform.Context`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#Context) method (e.g. `Plan` or `Apply`) to produce the graph that will be used to represent the necessary steps for that operation and the dependency relationships between them. In most cases, the [vertices](https://en.wikipedia.org/wiki/Vertex_(graph_theory)) of Terraform's graphs each represent a specific object in the configuration, or something derived from those configuration objects. For example, each `resource` block in the configuration has one corresponding [`GraphNodeConfigResource`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#GraphNodeConfigResource) vertex representing it in the "plan" graph. (Terraform Core uses terminology inconsistently, describing graph _vertices_ also as graph _nodes_ in various places. These both describe the same concept.) The [edges](https://en.wikipedia.org/wiki/Glossary_of_graph_theory_terms#edge) in the graph represent "must happen after" relationships. These define the order in which the vertices are evaluated, ensuring that e.g. one resource is created before another resource that depends on it. Each operation has its own graph builder, because the graph building process is different for each. For example, a "plan" operation needs a graph built directly from the configuration, but an "apply" operation instead builds its graph from the set of changes described in the plan that is being applied. The graph builders all work in terms of a sequence of _transforms_, which are implementations of [`terraform.GraphTransformer`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#GraphTransformer). Implementations of this interface just take a graph and mutate it in any way needed, and so the set of available transforms is quite varied. Some important examples include: * [`ConfigTransformer`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#ConfigTransformer), which creates a graph vertex for each `resource` block in the configuration. * [`StateTransformer`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#StateTransformer), which creates a graph vertex for each resource instance currently tracked in the state. * [`ReferenceTransformer`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#ReferenceTransformer), which analyses the configuration to find dependencies between resources and other objects and creates any necessary "happens after" edges for these. * [`ProviderTransformer`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#ProviderTransformer), which associates each resource or resource instance with exactly one provider configuration (implementing [the inheritance rules](https://developer.hashicorp.com/terraform/language/modules/develop/providers)) and then creates "happens after" edges to ensure that the providers are initialized before taking any actions with the resources that belong to them. There are many more different graph transforms, which can be discovered by reading the source code for the different graph builders. Each graph builder uses a different subset of these depending on the needs of the operation that is being performed. The result of graph building is a [`terraform.Graph`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#Graph), which can then be processed using a _graph walker_. ## Graph Walk The process of walking the graph visits each vertex of that graph in a way which respects the "happens after" edges in the graph. The walk algorithm itself is implemented in [the low-level `dag` package](http://localhost:8080/github.com/hashicorp/terraform/internal/dag#AcyclicGraph.Walk) (where "DAG" is short for [_Directed Acyclic Graph_](https://en.wikipedia.org/wiki/Directed_acyclic_graph)), in [`AcyclicGraph.Walk`](http://localhost:8080/github.com/hashicorp/terraform/internal/dag#AcyclicGraph.Walk). However, the "interesting" Terraform walk functionality is implemented in [`terraform.ContextGraphWalker`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#ContextGraphWalker), which implements a small set of higher-level operations that are performed during the graph walk: * `EnterPath` is called once for each module in the configuration, taking a module address and returning a [`terraform.EvalContext`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#EvalContext) that tracks objects within that module. `terraform.Context` is the _global_ context for the entire operation, while `terraform.EvalContext` is a context for processing within a single module, and is the primary means by which the namespaces in each module are kept separate. Each vertex in the graph is evaluated, in an order that guarantees that the "happens after" edges will be respected. If possible, the graph walk algorithm will evaluate multiple vertices concurrently. Vertex evaluation code must therefore make careful use of concurrency primitives such as mutexes in order to coordinate access to shared objects such as the `states.State` object. In most cases, we use the helper wrapper [`states.SyncState`](http://localhost:8080/github.com/hashicorp/terraform/internal/states#SyncState) to safely implement concurrent reads and writes from the shared state. ## Vertex Evaluation The action taken for each vertex during the graph walk is called _execution_. Execution runs a sequence of arbitrary actions that make sense for a particular vertex type. For example, evaluation of a vertex representing a resource instance during a plan operation would include the following high-level steps: * Retrieve the resource's associated provider from the `EvalContext`. This should already be initialized earlier by the provider's own graph vertex, due to the "happens after" edge between the resource node and the provider node. * Retrieve from the state the portion relevant to the specific resource instance being evaluated. * Evaluate the attribute expressions given for the resource in configuration. This often involves retrieving the state of _other_ resource instances so that their values can be copied or transformed into the current instance's attributes, which is coordinated by the `EvalContext`. * Pass the current instance state and the resource configuration to the provider, asking the provider to produce an _instance diff_ representing the differences between the state and the configuration. * Save the instance diff as part of the plan that is being constructed by this operation. Each execution step for a vertex is an implementation of [`terraform.GraphNodeExecutable.Execute`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#GraphNodeExecutable.Execute). As with graph transforms, the behavior of these implementations varies widely: whereas graph transforms can take any action against the graph, an `Execute` implementation can take any action against the `EvalContext`. The implementation of `terraform.EvalContext` used in real processing (as opposed to testing) is [`terraform.BuiltinEvalContext`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#BuiltinEvalContext). It provides coordinated access to plugins, the current state, and the current plan via the `EvalContext` interface methods. In order to be executed, a vertex must implement [`terraform.GraphNodeExecutable`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#GraphNodeExecutable), which has a single `Execute` method that handles. There are numerous `Execute` implementations with different behaviors, but some prominent examples are: * [`NodePlannableResource.Execute`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#NodePlannableResourceInstance.Execute), which handles the `plan` operation. * [`NodeApplyableResourceInstance.Execute`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#NodeApplyableResourceInstance.Execute), which handles the main `apply` operation. * [`NodeDestroyResourceInstance.Execute`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#EvalWriteState), which handles the main `destroy` operation. A vertex must complete successfully before the graph walk will begin evaluation for other vertices that have "happens after" edges. Evaluation can fail with one or more errors, in which case the graph walk is halted and the errors are returned to the user. ### Expression Evaluation An important part of vertex evaluation for most vertex types is evaluating any expressions in the configuration block associated with the vertex. This completes the processing of the portions of the configuration that were not processed by the configuration loader. The high-level process for expression evaluation is: 1. Analyze the configuration expressions to see which other objects they refer to. For example, the expression `aws_instance.example[1]` refers to one of the instances created by a `resource "aws_instance" "example"` block in configuration. This analysis is performed by [`lang.References`](http://localhost:8080/github.com/hashicorp/terraform/internal/lang#References), or more often one of the helper wrappers around it: [`lang.ReferencesInBlock`](http://localhost:8080/github.com/hashicorp/terraform/internal/lang#ReferencesInBlock) or [`lang.ReferencesInExpr`](http://localhost:8080/github.com/hashicorp/terraform/internal/lang#ReferencesInExpr) 1. Retrieve from the state the data for the objects that are referred to and create a lookup table of the values from these objects that the HCL evaluation code can refer to. 1. Prepare the table of built-in functions so that HCL evaluation can refer to them. 1. Ask HCL to evaluate each attribute's expression (a [`hcl.Expression`](http://localhost:8080/github.com/hashicorp/hcl/v2/#Expression) object) against the data and function lookup tables. In practice, steps 2 through 4 are usually run all together using one of the methods on [`lang.Scope`](http://localhost:8080/github.com/hashicorp/terraform/internal/lang#Scope); most commonly, [`lang.EvalBlock`](http://localhost:8080/github.com/hashicorp/terraform/internal/lang#Scope.EvalBlock) or [`lang.EvalExpr`](http://localhost:8080/github.com/hashicorp/terraform/internal/lang#Scope.EvalExpr). Expression evaluation produces a dynamic value represented as a [`cty.Value`](http://localhost:8080/github.com/zclconf/go-cty/cty#Value). This Go type represents values from the Terraform language and such values are eventually passed to provider plugins. ### Sub-graphs Some vertices have a special additional behavior that happens after their evaluation steps are complete, where the vertex implementation is given the opportunity to build another separate graph which will be walked as part of the evaluation of the vertex. The main example of this is when a `resource` block has the `count` argument set. In that case, the plan graph initially contains one vertex for each `resource` block, but that graph then _dynamically expands_ to have a sub-graph containing one vertex for each instance requested by the count. That is, the sub-graph of `aws_instance.example` might contain vertices for `aws_instance.example[0]`, `aws_instance.example[1]`, etc. This is necessary because the `count` argument may refer to other objects whose values are not known when the main graph is constructed, but become known while evaluating other vertices in the main graph. This special behavior applies to vertex objects that implement [`terraform.GraphNodeDynamicExpandable`](http://localhost:8080/github.com/hashicorp/terraform/internal/terraform#GraphNodeDynamicExpandable). Such vertices have their own nested _graph builder_, _graph walk_, and _vertex evaluation_ steps, with the same behaviors as described in these sections for the main graph. The difference is in which graph transforms are used to construct the graph and in which evaluation steps apply to the nodes in that sub-graph. --- ## File: docs/debugging.md # How to Debug Terraform Contents: - [Debugging automated tests](#debugging-automated-tests) - [Debugging automated tests in VSCode](#debugging-automated-tests-in-vscode) - [Debugging Terraform operations that use real Terraform configurations](#debugging-terraform-operations-that-use-real-terraform-configurations) - [Launch Terraform with the `dlv` CLI tool](#launch-terraform-with-the-dlv-cli-tool) - [Launch Terraform with VS Code's debugging tool](#launch-terraform-with-vs-codes-debugging-tool) As Terraform is written in Go you may use [Delve](https://github.com/go-delve/delve) to debug it. GoLand includes [debugging features](https://www.jetbrains.com/help/go/debugging-code.html), and the [Go extension for VS Code](https://code.visualstudio.com/docs/languages/go#_debugging) makes it easy to use Delve when debugging Go codebases in VS Code. ## Debugging automated tests Debugging an automated test is often the most straightforward workflow for debugging a section of the codebase. For example, the [Go extension for VS Code](https://code.visualstudio.com/docs/languages/go#_debugging) adds `run test | debug test` options above all tests in a `*_test.go` file. These allow debugging without any prior configuration. ### Debugging automated tests in VSCode As described above, debugging tests in VS Code is easily achieved through the Go extension. If you need more control over how tests are run while debugging, e.g. environment variable values, look at the [example debugger launch configuration 'Run selected test'](./debugging-configs/vscode/debug-automated-tests/launch.json). You can adapt this example to create your own [launch configuration file](https://code.visualstudio.com/docs/editor/debugging#_launch-configurations). When using this launch configuration you must highlight a test's name before starting the debugger: ## Debugging Terraform operations that use real Terraform configurations ### Launch Terraform with the `dlv` CLI tool In this workflow you: * Build Terraform using compiler flags. * Start a debug server with a command containing the terraform command you want to debug. * This command is run in the working directory that contains your Terraform configuration. * Connect to the debug server to monitor progress through breakpoints. #### 1. Compile & Start Debug Server One way to do it is to compile a binary with the [appropriate compiler flags](https://pkg.go.dev/cmd/compile#hdr-Command_Line): ```sh go install -gcflags="all=-N -l" ``` This enables you to then execute the compiled binary via Delve, pass any arguments and spin up a debug server which you can then connect to: ```sh # Update the path to the terraform binary if your install directory is influenced by $GOPATH or $GOBIN dlv exec $HOME/go/bin/terraform --headless --listen :2345 --log -- apply ``` #### 2a. Connect via CLI You may connect to the headless debug server via Delve CLI ```sh dlv connect :2345 ``` #### 2b. Connect from VS Code The repository provides [an example 'Connect to dlv server' launch configuration](./debugging-configs/vscode/launch-via-cli/launch.json), making it possible to use VS Code's native debugging integration via the [Go extension for VS Code](https://code.visualstudio.com/docs/languages/go#_debugging): ### Launch Terraform with VS Code's debugging tool In this workflow you: * Update the debugger's launch configuration to point at the directory containing your Terraform configuration. * Start the debugger through VS Code and monitor progress through breakpoints. #### 1. Update the debugger's launch configuration Look at the [example debugger launch configuration 'Run Terraform in debug mode'](./debugging-configs/vscode/launch-from-vscode-debugger/launch.json). You can adapt this example to create your own [launch configuration file](https://code.visualstudio.com/docs/editor/debugging#_launch-configurations). To use this launch configuration: * Prepare a local Terraform project. * Get the absolute path to that directory. * Update the launch configuration to use that path, either in the `-chdir` argument or as a `cwd` attribute in the launch configuration. * Make sure the `args` array's element reflect the command you'd like to debug. * Provide any required environment variables through the `env` or `envFile` attributes. #### 2. Run the launch configuration in VS Code Navigate to the Run and Debug view in the left-side Activity Bar. After selecting the `Run Terraform in debug mode` configuration in the Run and Debug view from the left-side, press the green arrow. This is equivalent to running a Terraform CLI command in the local Terraform project's directory. For example, if you run and debug a plan command that saves a plan file, that plan file will be created. This workflow is useful if you need to set up a complicated prior state to replicate a bug or if you want to debug code behaviour given a specific configuration. --- ## File: docs/dependency-upgrades.md # Upgrading Terraform's Library Dependencies This codebase depends on a variety of external Go modules. At the time of writing, a Terraform CLI build includes the union of all dependencies required by Terraform CLI and Core itself and each of the remote state backends. Because the remote state backends have quite a different set of needs than Terraform itself -- special client libraries, in particular -- we've declared them as being separate Go Modules with their own `go.mod` and `go.sum` files, even though we don't intend to ever publish them separately. The goal is just to help track which dependencies are used by each of these components, and to more easily determine which of the components are affected by a particular dependency upgrade so that we can make sure to do the appropriate testing. ## Dependency Synchronization Because all of these components ultimately link into the same executable, there can be only one version of each distinct module and thus all of the modules must agree on which version to use. The makefile target `syncdeps` runs a script which synchronizes all of the modules to declare compatible dependencies, selecting the newest version of each external module selected across all of the internal modules: ```shell make syncdeps ``` After running this, use `git status` to see what's been changed. If you've changed the dependencies of any of the modules then that should typically cause an update to the root module, because that one imports all of the others. ## Upgrading a Dependency To select a newer version of one of the dependencies, use `go get` in the root to specify which version to use: ```shell go get example.com/foo/bar@v1.0.0 ``` Or, if you just want to move to the latest stable release, you can use the `latest` pseudo-version: ```shell go get example.com/foo/bar@latest ``` Then run `make syncdeps` to update any of the child modules that also use this dependency. The remote state backends use only a subset of the packages in Terraform CLI/Core, so not all dependency updates will affect the remote state backends, and an update might affect only a subset of the backends. When you open the pull request for your change, our code owners rules will automatically request review from the team that maintains any affected remote state backend. The affected teams can judge whether the update seems likely to affect their backend and run their acceptance tests if so, before approving the pull request. As usual, these PRs should also be reviewed by at least one member of the Terraform Core team since they are ultimately responsible for the complete set of dependencies used in Terraform CLI releases. **Note:** Currently our code owners rules are simplistic and will request review for _any_ change under a remote state backend module directory, but in practice an update that only changes a backend's `go.sum` cannot affect the runtime behavior of the backend, and so those review requests are not strictly required. You should therefore remove the review requests for any backend whose only diff is the `go.sum` file once you've opened the pull request. ## Dependabot Updates When Dependabot automatically opens a pull request to upgrade a dependency, unfortunately it isn't smart enough to automatically synchronize the change across the modules and so the code consistency checks for the change will typically fail. To apply the proposed change, you'll need to check out the branch that Dependabot created on your development system, run `make syncdeps`, add all of the files that get modified, and then amend Dependabot's commit using `git commit --amend`. After you've done this, use `git push --force` to replace Dependabot's original commit with your new commit, and then wait for GitHub to re-run the PR checks. The code consistency checks should now pass. We've configured Dependabot to monitor only the root `go.mod` file for potential upgrades, because that one aggregates the dependencies for all other child modules. Therefore there should never be a Dependabot upgrade targeting a module in a subdirectory. If one _does_ get created somehow, you should close it and perform the same upgrade at the root of the repository instead, using the instructions in [Upgrading a Dependency](#upgrading-a-dependency) above. ## Dependencies with Special Requirements Most of our dependencies can be treated generically, but a few have some special constraints due to how Terraform uses them: * HCL, cty, and their indirect dependencies `golang.org/x/text` and `github.com/apparentlymart/go-textseg` all include logic based on Unicode specifications, and so should be updated with care to make sure that Terraform's support for Unicode follows a consistent Unicode version throughout. Additionally, each time we adopt a new minor release of Go, we may need to upgrade some or all of these dependencies to match the Unicode version used by the Go standard library. For more information, refer to [How Terraform Uses Unicode](unicode.md). (This concern does not apply if the new version we're upgrading to is built for the same version of Unicode that Terraform was already using.) * `github.com/hashicorp/go-getter` represents a significant part of Terraform CLI's remote module installer, and is the final interpreter of Terraform's module source address syntax. Because the module source address syntax is protected by the Terraform v1.x Compatibility Promises, for each upgrade we must make sure that: - The upgrade doesn't expand the source address syntax in a way that is undesirable from a Terraform product standpoint or in a way that we would not feel comfortable supporting indefinitely under the compatibility promises. - The upgrade doesn't break any already-supported source address forms that would therefore cause the next Terraform version to break the v1.x compatibility promises. Terraform's use of `go-getter` is all encapsulated in `internal/getmodules` and is set up to try to minimize the possibility that a go-getter upgrade would immediately introduce new functionality, but that encapsulation cannot prevent adoption of changes made to pre-existing functionality that Terraform already exposes. * `github.com/hashicorp/go-tfe` -- the client library for the HCP Terraform API -- includes various types corresponding to HCP Terraform API requests and responses. The internal package `internal/cloud` contains mock implementations of some of those types, which may need to be updated when the client library is upgraded. These upgrades should typically be done only in conjunction with a project that will somehow use the new features through the Cloud integration, so that the team working on that project can perform any needed updates to the mocks as part of their work. * `go.opentelemetry.io/otel` and the other supporting OpenTelemetry modules should typically be upgraded together in lockstep, because some of the modules define interfaces that other modules implement, and strange behavior can emerge if one is upgraded without the other. The main modules affected by this rule are the ones under the `go.opentelemetry.io/otel` prefix. The "contrib" packages can be trickier to upgrade because they tend to have dependencies that overlap with ours and so might affect non-telemetry-related behavior, and so it's acceptable for those to lag slightly behind to reduce risk in routine upgrades. --- ## File: docs/destroying.md # Terraform Core Resource Destruction Notes This document intends to describe some of the details and complications involved in the destruction of resources. It covers the ordering defined for related create and destroy operations, as well as changes to the lifecycle ordering imposed by `create_before_destroy`. It is not intended to enumerate all possible combinations of dependency ordering, only to outline the basics and document some of the more complicated aspects of resource destruction. The graph diagrams here will continue to use the inverted graph structure used internally by Terraform, where edges represent dependencies rather than order of operations. ## Simple Resource Creation In order to describe resource destruction, we first need to create the resources and define their order. The order of creation is that which fulfills the dependencies for each resource. In this example, `A` has no dependencies, `B` depends on `A`, and `C` depends on `B`, and transitively depends on `A`. Order of operations: 1. `A` is created 1. `B` is created 1. `C` is created ## Resource Updates An existing resource may be updated with references to a newly created resource. The ordering here is exactly the same as one would expect for creation. Order of operations: 1. `A` is created 1. `B` is updated 1. `C` is updated ## Simple Resource Destruction The order for destroying resource is exactly the inverse used to create them. This example shows the graph for the destruction of the same nodes defined above. While destroy nodes will not contain attribute references, we will continue to use the inverted edges showing dependencies for destroy, so the operational ordering is still opposite the flow of the arrows. Order of operations: 1. `C` is destroyed 1. `B` is destroyed 1. `A` is destroyed ## Resource Replacement Resource replacement is the logical combination of the above scenarios. Here we will show the replacement steps involved when `B` depends on `A`. In this first example, we simultaneously replace both `A` and `B`. Here `B` is destroyed before `A`, then `A` is recreated before `B`. Order of operations: 1. `B` is destroyed 1. `A` is destroyed 1. `A` is created 1. `B` is created This second example replaces only `A`, while updating `B`. Resource `B` is only updated once `A` has been destroyed and recreated. Order of operations: 1. `A` is destroyed 1. `A` is created 1. `B` is updated While the dependency edge from `B update` to `A destroy` isn't necessary in these examples, it is shown here as an implementation detail which will be mentioned later on. A final example based on the replacement graph; starting with the above configuration where `B` depends on `A`. The graph is reduced to an update of `A` while only destroying `B`. The interesting feature here is the remaining dependency of `A update` on `B destroy`. We can derive this ordering of operations from the full replacement example above, by replacing `A create` with `A update` and removing the unused nodes. ## Create Before Destroy Currently, the only user-controllable method for changing the ordering of create and destroy operations is with the `create_before_destroy` resource `lifecycle` attribute. This has the obvious effect of causing a resource to be created before it is destroyed when replacement is required, but has a couple of other effects we will detail here. Taking the previous replacement examples, we can change the behavior of `A` to be that of `create_before_destroy`. Order of operations: 1. `B` is destroyed 2. `A` is created 1. `B` is created 1. `A` is destroyed Note that in this first example, the creation of `B` is inserted in between the creation of `A` and the destruction of `A`. This becomes more important in the update example below. Order of operations: 1. `A` is created 1. `B` is updated 1. `A` is destroyed Here we can see clearly how `B` is updated after the creation of `A` and before the destruction of the _deposed_ resource `A`. (The prior resource `A` is sometimes referred to as "deposed" before it is destroyed, to disambiguate it from the newly created `A`.) This ordering is important for resource that "register" other resources, and require updating before the dependent resource can be destroyed. The transformation used to create these graphs is also where we use the extra edges mentioned above connecting `B` to `A destroy`. The algorithm to change a resource from the default ordering to `create_before_destroy` simply inverts any incoming edges from other resources, which automatically creates the necessary dependency ordering for dependent updates. This also ensures that reduced versions of this example still adhere to the same ordering rules, such as when the dependency is only being removed: Order of operations: 1. `B` is updated 1. `A` is destroyed ### Forced Create Before Destroy In the previous examples, only resource `A` was being used as is it were `create_before_destroy`. The minimal graphs used show that it works in isolation, but that is only when the `create_before_destroy` resource has no dependencies of it own. When a `create_before_resource` depends on another resource, that dependency is "infected" by the `create_before_destroy` lifecycle attribute. This example demonstrates why forcing `create_before_destroy` is necessary. `B` has `create_before_destroy` while `A` does not. If we only invert the ordering for `B`, we can see that results in a cycle. In order to resolve these cycles, all resources that precede a resource with `create_before_destroy` must in turn be handled in the same manner. Reversing the incoming edges to `A destroy` resolves the problem: Order of operations: 1. `A` is created 1. `B` is created 1. `B` is destroyed 1. `A` is destroyed This also demonstrates why `create_before_destroy` cannot be overridden when it is inherited; changing the behavior here isn't possible without removing the initial reason for `create_before_destroy`; otherwise cycles are always introduced into the graph. --- ## File: docs/planning-behaviors.md # Planning Behaviors A key design tenet for Terraform is that any actions with externally-visible side-effects should be carried out via the standard process of creating a plan and then applying it. Any new features should typically fit within this model. There are also some historical exceptions to this rule, which we hope to supplement with plan-and-apply-based equivalents over time. This document describes the default planning behavior of Terraform in the absence of any special instructions, and also describes the three main design approaches we can choose from when modelling non-default behaviors that require additional information from outside of Terraform Core. This document focuses primarily on actions relating to _resource instances_, because that is Terraform's main concern. However, these design principles can potentially generalize to other externally-visible objects, if we can describe their behaviors in a way comparable to the resource instance behaviors. This is developer-oriented documentation rather than user-oriented documentation. See [the main Terraform documentation](https://developer.hashicorp.com/terraform/docs) for information on existing planning behaviors and other behaviors as viewed from an end-user perspective. ## Default Planning Behavior When given no explicit information to the contrary, Terraform Core will automatically propose taking the following actions in the appropriate situations: - **Create**, if either of the following are true: - There is a `resource` block in the configuration that has no corresponding managed resource in the prior state. - There is a `resource` block in the configuration that is recorded in the prior state but whose `count` or `for_each` argument (or lack thereof) describes an instance key that is not tracked in the prior state. - **Delete**, if either of the following are true: - There is a managed resource tracked in the prior state which has no corresponding `resource` block in the configuration. - There is a managed resource tracked in the prior state which has a corresponding `resource` block in the configuration _but_ its `count` or `for_each` argument (or lack thereof) lacks an instance key that is tracked in the prior state. - **Update**, if there is a corresponding resource instance both declared in the configuration (in a `resource` block) and recorded in the prior state (unless it's marked as "tainted") but there are differences between the prior state and the configuration which the corresponding provider doesn't explicitly classify as just being normalization. - **Replace**, if there is a corresponding resource instance both declared in the configuration (in a `resource` block) and recorded in the prior state _marked as "tainted"_. The special "tainted" status means that the process of creating the object failed partway through and so the existing object does not necessarily match the configuration, so Terraform plans to replace it in order to ensure that the resulting object is complete. - **Read**, if there is a `data` block in the configuration. - If possible, Terraform will eagerly perform this action during the planning phase, rather than waiting until the apply phase. - If the configuration contains at least one unknown value, or if the data resource directly depends on a managed resource that has any change proposed elsewhere in the plan, Terraform will instead delay this action to the apply phase so that it can react to the completion of modification actions on other objects. - **No-op**, to explicitly represent that Terraform considered a particular resource instance but concluded that no action was required. The **Replace** action described above is really a sort of "meta-action", which Terraform expands into separate **Create** and **Delete** operations. There are two possible orderings, and the first one is the default planning behavior unless overridden by a special planning behavior as described later. The two possible lowerings of **Replace** are: 1. **Delete** then **Create**: first delete the existing object bound to an instance, and then create a new object at the same address based on the current configuration. 2. **Create** then **Delete**: mark the existing object bound to an instance as "deposed" (still exists but not current), create a new current object at the same address based on the current configuration, and then delete the deposed object. ## Special Planning Behaviors For the sake of this document, a "special" planning behavior is one where Terraform Core will select a different action than the defaults above, based on explicit instructions given either by a module author, an operator, or a provider. There are broadly three different design patterns for special planning behaviors, and so each "special" use-case will typically be met by one or more of the following depending on which stakeholder is activating the behavior: - [Configuration-driven Behaviors](#configuration-driven-behaviors) are activated by additional annotations given in the source code of a module. This design pattern is good for situations where the behavior relates to a particular module and so should be activated for anyone using that module. These behaviors are therefore specified by the module author, such that any caller of the module will automatically benefit with no additional work. - [Provider-driven Behaviors](#provider-driven-behaviors) are activated by optional fields in a provider's response when asked to help plan one of the default actions given above. This design pattern is good for situations where the behavior relates to the behavior of the remote system that a provider is wrapping, and so from the perspective of a user of the provider the behavior should appear "automatic". Because these special behaviors are activated by values in the provider's response to the planning request from Terraform Core, behaviors of this sort will typically represent "tweaks" to or variants of the default planning behaviors, rather than entirely different behaviors. - [Single-run Behaviors](#single-run-behaviors) are activated by explicitly setting additional "plan options" when calling Terraform Core's plan operation. This design pattern is good for situations where the direct operator of Terraform needs to do something exceptional or one-off, such as when the configuration is correct but the real system has become degraded or damaged in a way that Terraform cannot automatically understand. However, this design pattern has the disadvantage that each new single-run behavior type requires custom work in every wrapping UI or automaton around Terraform Core, in order provide the user of that wrapper some way to directly activate the special option, or to offer an "escape hatch" to use Terraform CLI directly and bypass the wrapping automation for a particular change. We've also encountered use-cases that seem to call for a hybrid between these different patterns. For example, a configuration construct might cause Terraform Core to _invite_ a provider to activate a special behavior, but let the provider make the final call about whether to do it. Or conversely, a provider might advertise the possibility of a special behavior but require the user to specify something in the configuration to activate it. The above are just broad categories to help us think through potential designs; some problems will require more creative combinations of these patterns than others. ### Configuration-driven Behaviors Within the space of configuration-driven behaviors, we've encountered two main sub-categories: - Resource-specific behaviors, whose effect is scoped to a particular resource. The configuration for these often lives inside the `resource` or `data` block that declares the resource. - Global behaviors, whose effect can span across more than one resource and sometimes between resources in different modules. The configuration for these often lives in a separate location in a module, such as a separate top-level block which refers to other resources using the typical address syntax. The following is a non-exhaustive list of existing examples of configuration-driven behaviors, selected to illustrate some different variations that might be useful inspiration for new designs: - The `ignore_changes` argument inside `resource` block `lifecycle` blocks tells Terraform that if there is an existing object bound to a particular resource instance address then Terraform should ignore the configured value for a particular argument and use the corresponding value from the prior state instead. This can therefore potentially cause what would've been an **Update** to be a **No-op** instead. - The `replace_triggered_by` argument inside `resource` block `lifecycle` blocks can use a proposed change elsewhere in a module to force Terraform to propose one of the two **Replace** variants for a particular resource. - The `create_before_destroy` argument inside `resource` block `lifecycle` blocks only takes effect if a particular resource instance has a proposed **Replace** action. If not set or set to `false`, Terraform will decompose it to **Destroy** then **Create**, but if set to `true` Terraform will use the inverted ordering. Because Terraform Core will never select a **Replace** action automatically by itself, this is an example of a hybrid design where the config-driven `create_before_destroy` combines with any other behavior (config-driven or otherwise) that might cause **Replace** to customize exactly what that **Replace** will mean. - Top-level `moved` blocks in a module activate a special behavior during the planning phase, where Terraform will first try to change the bindings of existing objects in the prior state to attach to new addresses before running the normal planning process. This therefore allows a module author to document certain kinds of refactoring so that Terraform can update the state automatically once users upgrade to a new version of the module. This special behavior is interesting because it doesn't _directly_ change what actions Terraform will propose, but instead it adds an extra preparation step before the typical planning process which changes the addresses that the planning process will consider. It can therefore _indirectly_ cause different proposed actions for affected resource instances, such as transforming what by default might've been a **Delete** of one instance and a **Create** of another into just a **No-op** or **Update** of the second instance. This one is an example of a "global behavior", because at minimum it affects two resource instance addresses and, if working with whole resource or whole module addresses, can potentially affect a large number of resource instances all at once. ### Provider-driven Behaviors Providers get an opportunity to activate some special behaviors for a particular resource instance when they respond to the `PlanResourceChange` function of the provider plugin protocol. When Terraform Core executes this RPC, it has already selected between **Create**, **Delete**, or **Update** actions for the particular resource instance, and so the special behaviors a provider may activate will typically serve as modifiers or tweaks to that base action, and will not allow the provider to select another base action altogether. The provider wire protocol does not talk about the action types explicitly, and instead only implies them via other content of the request and response, with Terraform Core making the final decision about how to react to that information. The following is a non-exhaustive list of existing examples of provider-driven behaviors, selected to illustrate some different variations that might be useful inspiration for new designs: - When the base action is **Update**, a provider may optionally return one or more paths to attributes which have changes that the provider cannot implement as an in-place update due to limitations of the remote system. In that case, Terraform Core will replace the **Update** action with one of the two **Replace** variants, which means that from the provider's perspective the apply phase will really be two separate calls for the decomposed **Create** and **Delete** actions (in either order), rather than **Update** directly. - When the base action is **Update**, a provider may optionally return a proposed new object where one or more of the arguments has its value set to what was in the prior state rather than what was set in the configuration. This represents any situation where a remote system supports multiple different serializations of the same value that are all equivalent, and so changing from one to another doesn't represent a real change in the remote system. If all of those taken together causes the new object to match the prior state, Terraform Core will treat the update as a **No-op** instead. Of the three genres of special behaviors, provider-driven behaviors is the one we've made the least use of historically but one that seems to have a lot of opportunities for future exploration. Provider-driven behaviors can often be ideal because their effects appear as if they are built in to Terraform so that "it just works", with Terraform automatically deciding and explaining what needs to happen and why, without any special effort on the user's part. ### Single-run Behaviors Terraform Core's "plan" operation takes a set of arguments that we collectively call "plan options", that can modify Terraform's planning behavior on a per-run basis without any configuration changes or special provider behaviors. As noted above, this particular genre of designs is the most burdensome to implement because any wrapping software that can ask Terraform Core to create a plan must ideally offer some way to set all of the available planning options, or else some part of Terraform's functionality won't be available to anyone using that wrapper. However, we've seen various situations where single-run behaviors really are the most appropriate way to handle a particular use-case, because the need for the behavior originates in some process happening outside of the scope of any particular Terraform module or provider. The following is a non-exhaustive list of existing examples of single-run behaviors, selected to illustrate some different variations that might be useful inspiration for new designs: - The "replace" planning option specifies zero or more resource instance addresses. For any resource instance specified, Terraform Core will transform any **Update** or **No-op** action for that instance into one of the **Replace** actions, thereby allowing an operator to respond to something having become degraded in a way that Terraform and providers cannot automatically detect and force Terraform to replace that object with a new one that will hopefully function correctly. - The "refresh only" planning mode ("planning mode" is a single planning option that selects between a few mutually-exclusive behaviors) forces Terraform to treat every resource instance as **No-op**, regardless of what is bound to that address in state or present in the configuration. ## Legacy Operations Some of the legacy operations Terraform CLI offers that _aren't_ integrated with the plan and apply flow could be thought of as various degenerate kinds of single-run behaviors. Most don't offer any opportunity to preview an effect before applying it, but do meet a similar set of use-cases where an operator needs to take some action to respond to changes to the context Terraform is in rather than to the Terraform configuration itself. Most of these legacy operations could therefore most readily be translated to single-run behaviors, but before doing so it's worth researching whether people are using them as a workaround for missing configuration-driven and/or provider-driven behaviors. A particular legacy operation might be better replaced with a different sort of special behavior, or potentially by multiple different special behaviors of different genres if it's currently serving as a workaround for many different unmet needs. --- ## File: docs/README.md # Terraform Core Codebase Documentation This directory contains some documentation about the Terraform Core codebase, aimed at readers who are interested in making code contributions. If you're looking for information on _using_ Terraform, please instead refer to [the main Terraform CLI documentation](https://developer.hashicorp.com/terraform/cli). ## Terraform Core Architecture Documents * [Modules Runtime Architecture Summary](./architecture.md): an overview of the main components of Terraform Core related to planning and applying modules. This is the best starting point if you are diving in to this codebase for the first time. * [Stacks Runtime Architecture Summary](../internal/stacks/README.md): an overview of the main components of Terraform Core related to planning and applying stack configurations. * [Resource Instance Change Lifecycle](./resource-instance-change-lifecycle.md): a description of the steps in validating, planning, and applying a change to a resource instance, from the perspective of the provider plugin RPC operations. This may be useful for understanding the various expectations Terraform enforces about provider behavior, either if you intend to make changes to those behaviors or if you are implementing a new Terraform plugin SDK and so wish to conform to them. (If you are planning to write a new provider using the _official_ SDK then please refer to [the Plugin development documentation](https://developer.hashicorp.com/terraform/plugin) instead; it presents similar information from the perspective of the SDK API, rather than the plugin wire protocol.) * [Plugin Protocol](./plugin-protocol/): gRPC/protobuf definitions for the plugin wire protocol and information about its versioning strategy. This documentation is for SDK developers, and is not necessary reading for those implementing a provider using the official SDK. * [Terraform Core RPC API](../internal/rpcapi/README.md): an integration point for external software that needs to integrate Terraform Core functionality. * [Upgrading Terraform's Dependencies](./dependency-upgrades.md): guidance on some special details that arise when we upgrade Go Module dependencies, due to this codebase containing Terraform CLI, Terraform Core, and the various remote state backends which all have some overlapping dependencies. * [How Terraform Uses Unicode](./unicode.md): an overview of the various features of Terraform that rely on Unicode and how to change those features to adopt new versions of Unicode. * [Deadlock-free Promises](../internal/promising/README.md): a utility package used by the Terraform Stacks runtime to coordinate its concurrent work. ## Contribution Guides * [Contributing to Terraform](../.github/CONTRIBUTING.md): a complete guideline for those who want to contribute to this project. ## 2. Official Technical Reference & Guides (hashicorp/web-unified-docs) # Web Unified Docs > [!IMPORTANT] > This README is for developers working on the documentation website. If you > want to contribute docs content, refer to the [Contribute to HashiCorp > documentation](CONTRIBUTING.md) guide. --- The project in this repository, `hashicorp/web-unified-docs`, aims to implement [DEVDOT-023: Unified Product Documentation Repository](https://hermes-sharepoint.hashicorp.services/document/01XOO7K4NGQTFCAWZCHJBY5ZPCZLO6Y2ZB). The RFC for this project was intentionally light on implementation details, in order to foster consensus on the broad direction. - **PR previews**: Show broken links in comments for awareness (informational only, don't block PRs) - **Production monitoring**: Weekly scans create GitHub issues and send critical alerts to Datadog when users are affected The weekly [`broken-link-check-full`](https://github.com/hashicorp/web-unified-docs/blob/main/.github/workflows/broken-link-check-full.yml) workflow generates comprehensive broken link reports with prioritization guidance. When contributors create PRs that modify content, the link checker shows any broken links in PR comments with actionable guidance without blocking development. **Quick tips for contributors**: - **Fix internal HashiCorp links** (high priority) - **Check external docs/API links** (medium priority) - **Consider removing unreliable external links** (low priority) For detailed information about the monitoring system, see [Broken Link Monitoring Documentation](./.github/BROKEN_LINK_MONITORING.md). The [EDU-053: Unified Education Style Guide](https://hermes-sharepoint.hashicorp.services/document/01XOO7K4KLLV22I7VTKBALQC55YEB6TXZN) for this project was intentionally light on implementation details, in order to foster consensus on the broad direction. The existing API (`content.hashicorp.com`) has endpoints that serve documentation content. You can find the source code in [hashicorp/mktg-content-workflows](https://github.com/hashicorp/mktg-content-workflows/blob/main/api/content.ts). The goal of the unified docs API is to host all of HashiCorp's product documentation. The unified docs API will eventually replace the existing content API. ## Local development ### Requirements - [Docker](https://www.docker.com/) and [Docker Compose](https://docs.docker.com/compose/) (for managing containers) ### Quick start To get a migration preview running, run `make` from the root of this repo. The `make` command starts the `unified-docs` Docker profile that spins up a local instance of `unified-devdot-api` and `dev-portal`. Once this command completes, you can access the following endpoints: - http://localhost:3000 - An instance of the `dev-portal` container configured to pull from the experimental docs API (this repo). This image depends on the unified docs API (`unified-devdot-api`). - http://localhost:8080 - An instance of the unified docs API container (this repo - `unified-devdot-api`) that serves content from the `content` directory. On startup, this container processes the content and assets in `/content` into `public/assets` and `public/content`. In addition, the container also generates `app/api/docsPaths.json` and `app/api/versionMetadata.json` from the contents within `/content`. Use the following example to test this endpoint: http://localhost:8080/api/content/terraform-plugin-framework/doc/latest/plugin/framework > [!NOTE] > The unified docs API container takes time to process the content and assets. You must wait for both the `unified-devdot-api` and `dev-portal` containers to complete before you can successfully test content in the `dev-portal` preview environment (`localhost:3000`). Visit http://localhost:8080/api/all-docs-paths to verify the `unified-devdot-api` container is complete. To spin this down gracefully, run `make clean` in a separate terminal. If you wish to remove the local Docker images as well, run `make clean CLEAN_OPTION=full`. ### More commands The `makefile` serves as a convenience tool start the local preview. If you need more granular control, the `package.json` file contains a full list of available commands. To use these, you will need to intentionally run `npm install` and `npm run prebuild` before anything else. Use `npm run coverage` to run coverage tests. ### Preview environments for unified-docs and dev-portal Unified docs API serves as one of the content APIs for `dev-portal` (frontend application for DevDot). As a result, when implementing new features, you may need to modify both the backend (this repo) and the frontend (`dev-portal`). If you are working on a ticket that requires changes to both the unified docs API and `dev-portal`, please set [custom environment variables for your branch](https://vercel.com/hashicorp/dev-portal/settings/environment-variables) in Vercel to simplify testing instructions. For example, in Vercel, for your `dev-portal` branch, you can set the following environment variables: | Environment variable | Value | | -------------------- | ---------------------- | | `HASHI_ENV` | `unified-docs-sandbox` | | `UNIFIED_DOCS_API` | `` | Vercel will use these values to create deploy previews. ### API development Reach out to team #team-web-presence if you need to do local API development ## Background ### Project Rationale - Storing documentation in one branch of one repo dramatically simplifies the workflow for contributing documentation. - Publishing changes to multiple versions can be done in a single PR, as opposed to multiple PRs which is required by the current setup. - Finding and making the same change across multiple versions is as simple as doing a find-and-replace since all the versioned docs are on the filesystem at the same time. - Adding a new product is as easy as making a new folder, as opposed to the current process which requires code-changes on the API side and the installation of a GitHub App to monitor for events. - Sourcing from one branch in one repo eliminates the situation where a missed GitHub event can result in out-of-date documentation. If something goes wrong in the publishing process, simply run it again instead of relying on incoming commit/release events from the GitHub API. - Since we can make edits to all docs for all products and versions from a single PR, making platform-level changes is dramatically simplified (such as updating to MDX v2, or rewriting URLs). - Adding new features like content conformance (basically linting for docs) can be done for the entire codebase at once. - Removes the ability for docs to break the release workflow in product repos. - Enables us to support fully versioned deployment previews, whereas current previews are limited to the branch being modified. ### [Architectural Decision Records](https://github.com/hashicorp/web-unified-docs/tree/main/docs/decisions) ## Update product repo documentation This script helps with product documentation migration to the web-unified-docs repository. When migrating documentation: 1. The `web-unified-docs` repository becomes the source of truth 2. Original documentation may remain temporarily as a fallback 3. Users should be directed to make future changes in `web-unified-docs` only This script automatically adds a prominent notice to all MDX files in the original location, informing contributors where to make future changes. ``` ./scripts/update-mdx-files.sh ~/Desktop/hashicorp/terraform-plugin-framework/website/docs ``` Example output: ``` Progress: Files processed: 135 Files updated: 135 Files with no frontmatter: 0 Files with errors: 0 Completed! All MDX files have been processed. ``` ## Run broken link checker locally The repository uses a focused broken link monitoring system: - **PR previews**: Show broken links in comments for awareness (informational only, don't block PRs) - **Production monitoring**: Weekly scans create GitHub issues and Datadog alerts for user-facing problems The weekly [`broken-link-check-full`](https://github.com/hashicorp/web-unified-docs/blob/main/.github/workflows/broken-link-check-full.yml) workflow generates comprehensive broken link reports. When contributors create PRs that modify content, the link checker shows any broken links in PR comments without blocking development. For detailed information about the monitoring system, see [Broken Link Monitoring Documentation](./.github/BROKEN_LINK_MONITORING.md). ### UDR Migration Link Check For teams migrating products to UDR (Unified Docs Renderer), use the dedicated migration workflow: 1. Go to [Actions → UDR Product Link Check](https://github.com/hashicorp/web-unified-docs/actions/workflows/udr-product-link-check.yml) 2. Click "Run workflow" and select your product 3. Review migration-specific broken link analysis in the generated GitHub issue This workflow provides targeted link checking with migration-focused reporting and prioritization. ### Local Testing You can also run the broken link checker locally. The following commands launch a lychee Docker container to check the content directories you specify. Run the broken link checker on all content. ``` npm run broken-link ``` Check a specific directory within content. ``` npm run broken-link terraform-plugin-framework ``` Check multiple directories. ``` npm run broken-link terraform-plugin-framework-log terraform-plugin-mux ``` ## Architecture The following diagram illustrates the relationships between the unified docs API (this repo), `dev-portal`, and the existing content API: ```mermaid graph LR subgraph "Content sources (non-migrated)" BDY[boundary] CSL[consul] HCP[hcp-docs] NMD[nomad] PKR[packer] SNT[sentinel] TF[terraform] TFC[terraform-cdk] TFA[terraform-docs-agents] TFD[terraform-docs-common] VGT[vagrant] VLT[vault] WPT[waypoint] CURALL["/content or /website"] BDY & CSL & HCP & NMD & PKR & SNT & TF & TFC & TFA & TFD & VGT & VLT & WPT --> CURALL end subgraph "Migrated content repo" TPF[terraform-plugin-framework] TPL[terraform-plugin-log] TPM[terraform-plugin-mux] TPS[terraform-plugin-sdk] TPT[terraform-plugin-testing] TFE[terraform-enterprise] MIGALL["/content"] TPF & TPL & TPM & TPS & TPT & TFE --> MIGALL end subgraph "APIs" CP[Content APIcontent.hashicorp.com] UDR[Unified Docs Repositoryweb-unified-docs] end subgraph "Frontend" DP[Dev Portaldev-portal] end %% BDY & CSL & HCP & NMD & PKR & PTF & SNT & TF & TFC & TFA & TFD & VGT & VLT & WPT --> CP %% TPF & TPL & TPM & TPS & TPT --> UDR CURALL -->|Current content flow| CP MIGALL -->|Migrated content| UDR CP -->|Serves most content| DP UDR -->|Serves unified/content content| DP class TPF,TPL,TPM,TPS,TPT,BDY,CSL,HCP,NMD,PKR,PTF,SNT,TF,TFC,TFA,TFD,VGT,VLT,WPT productRepo ``` The diagram shows: - The content API — the existing system that sources product documentation content from product repositories - The unified docs API — the new system that sources product documentation from this repo's `/content` directory. The migrated repos will use a directory approach to versioning (rather than the historic branch and tag strategy) - The Dev Portal — the frontend that serves the main DevDot interface. Dev Portal sources its content from both the existing content API and unified docs API.