hcl Documentation

Technical Documentation: hashicorp/hcl

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

HCL

HCL is a toolkit for creating structured configuration languages that are
both human- and machine-friendly, for use with command-line tools.
Although intended to be generally useful, it is primarily targeted
towards devops tools, servers, etc.

NOTE: This is major version 2 of HCL, whose Go API is incompatible with
major version 1. Both versions are available for selection in Go Modules
projects. HCL 2 cannot be imported from Go projects that are not using Go Modules. For more information, see
our version selection guide.

HCL has both a native syntax, intended to be pleasant to read and write for
humans, and a JSON-based variant that is easier for machines to generate
and parse.

The HCL native syntax is inspired by libucl,
nginx configuration,
and others.

It includes an expression syntax that allows basic inline computation and,
with support from the calling application, use of variables and functions
for more dynamic configuration languages.

HCL provides a set of constructs that can be used by a calling application to
construct a configuration language. The application defines which attribute
names and nested block types are expected, and HCL parses the configuration
file, verifies that it conforms to the expected structure, and returns
high-level objects that the application can use for further processing.

package main

import (
	"log"

	"github.com/hashicorp/hcl/v2/hclsimple"
)

type Config struct {
	IOMode  string        `hcl:"io_mode"`
	Service ServiceConfig `hcl:"service,block"`
}

type ServiceConfig struct {
	Protocol   string          `hcl:"protocol,label"`
	Type       string          `hcl:"type,label"`
	ListenAddr string          `hcl:"listen_addr"`
	Processes  []ProcessConfig `hcl:"process,block"`
}

type ProcessConfig struct {
	Type    string   `hcl:"type,label"`
	Command []string `hcl:"command"`
}

func main() {
	var config Config
	err := hclsimple.DecodeFile("config.hcl", nil, &config)
	if err != nil {
		log.Fatalf("Failed to load configuration: %s", err)
	}
	log.Printf("Configuration is %#v", config)
}

A lower-level API is available for applications that need more control over
the parsing, decoding, and evaluation of configuration. For more information,
see the package documentation.

Why? Copy Chapter

Newcomers to HCL often ask: why not JSON, YAML, etc?

Whereas JSON and YAML are formats for serializing data structures, HCL is
a syntax and API specifically designed for building structured configuration
formats.

HCL attempts to strike a compromise between generic serialization formats
such as JSON and configuration formats built around full programming languages
such as Ruby. HCL syntax is designed to be easily read and written by humans,
and allows declarative logic to permit its use in more complex applications.

HCL is intended as a base syntax for configuration formats built
around key-value pairs and hierarchical blocks whose structure is well-defined
by the calling application, and this definition of the configuration structure
allows for better error messages and more convenient definition within the
calling application.

It can't be denied that JSON is very convenient as a lingua franca
for interoperability between different pieces of software. Because of this,
HCL defines a common configuration model that can be parsed from either its
native syntax or from a well-defined equivalent JSON structure. This allows
configuration to be provided as a mixture of human-authored configuration
files in the native syntax and machine-generated files in JSON.

Information Model and Syntax Copy Chapter

HCL is built around two primary concepts: attributes and blocks. In
native syntax, a configuration file for a hypothetical application might look
something like this:

io_mode = "async"

service "http" "web_proxy" {
  listen_addr = "127.0.0.1:8080"
  
  process "main" {
    command = ["/usr/local/bin/awesome-app", "server"]
  }

  process "mgmt" {
    command = ["/usr/local/bin/awesome-app", "mgmt"]
  }
}

The JSON equivalent of this configuration is the following:

{
  "io_mode": "async",
  "service": {
    "http": {
      "web_proxy": {
        "listen_addr": "127.0.0.1:8080",
        "process": {
          "main": {
            "command": ["/usr/local/bin/awesome-app", "server"]
          },
          "mgmt": {
            "command": ["/usr/local/bin/awesome-app", "mgmt"]
          },
        }
      }
    }
  }
}

Regardless of which syntax is used, the API within the calling application
is the same. It can either work directly with the low-level attributes and
blocks, for more advanced use-cases, or it can use one of the decoder
packages to declaratively extract into either Go structs or dynamic value
structures.

Attribute values can be expressions as well as just literal values:

# Arithmetic with literals and application-provided variables
sum = 1 + addend

# String interpolation and templates
message = "Hello, ${name}!"

# Application-provided functions
shouty_message = upper(message)

Although JSON syntax doesn't permit direct use of expressions, the interpolation
syntax allows use of arbitrary expressions within JSON strings:

{
  "sum": "${1 + addend}",
  "message": "Hello, ${name}!",
  "shouty_message": "${upper(message)}"
}

For more information, see the detailed specifications:

• Syntax-agnostic Information Model
• HCL Native Syntax
• JSON Representation

Changes in 2.0 Copy Chapter

Version 2.0 of HCL combines the features of HCL 1.0 with those of the
interpolation language HIL to produce a single configuration language that
supports arbitrary expressions.

This new version has a completely new parser and Go API, with no direct
migration path. Although the syntax is similar, the implementation takes some
very different approaches to improve on some "rough edges" that existed with
the original implementation and to allow for more robust error handling.

It's possible to import both HCL 1 and HCL 2 into the same program using Go's
semantic import versioning mechanism:

import (
    hcl1 "github.com/hashicorp/hcl"
    hcl2 "github.com/hashicorp/hcl/v2"
)

Acknowledgements Copy Chapter

HCL was heavily inspired by libucl,
by Vsevolod Stakhov.

HCL and HIL originate in HashiCorp Terraform,
with the original parsers for each written by
Mitchell Hashimoto.

The original HCL parser was ported to pure Go (from yacc) by
Fatih Arslan. The structure-related portions of
the new native syntax parser build on that work.

The original HIL parser was ported to pure Go (from yacc) by
Martin Atkins. The expression-related
portions of the new native syntax parser build on that work.

HCL 2, which merged the original HCL and HIL languages into this single new
language, builds on design and prototyping work by
Martin Atkins in
zcl.

Overview (`ext/README.md`) Copy Section

HCL Extensions

This directory contains some packages implementing some extensions to HCL
that add features by building on the core API in the main hcl package.

These serve as optional language extensions for use-cases that are limited only
to specific callers. Generally these make the language more expressive at
the expense of increased dynamic behavior that may be undesirable for
applications that need to impose more rigid structure on configuration.

Overview (`specsuite/README.md`) Copy Section

HCL Language Test Suite

This directory contains an implementation-agnostic test suite that can be used
to verify the correct behavior not only of the HCL implementation in this
repository but also of possible other implementations.

The harness for running this suite -- a Go program in this directory -- uses
the hcldec program as a level of abstraction to avoid depending directly on
the Go implementation. As a result, other HCL implementations must also
include a version of hcldec in order to run this spec.

The tests defined in this suite each correspond to a detail of
the HCL spec. This suite is separate from and not a
substitute for direct unit tests of a given implementation that would presumably
also exercise that implementation's own programmatic API.

To run the suite, first build the harness using Go:

go install github.com/hashicorp/hcl/v2/cmd/hclspecsuite

Then run it, passing it the directory containing the test definitions (the
"tests" subdirectory of this directory) and the path to the hcldec executable
to use.

For example, if working in the root of this repository and using the hcldec
implementation from here:

go install ./cmd/hcldec
hclspecsuite ./specsuite/tests $GOPATH/bin/hcldec

For developers working on the Go implementation of HCL from this repository,
please note that this spec suite is run as part of a normal go test ./...
execution for this whole repository and so does not need to be run separately.

Overview (`cmd/hcldec/README.md`) Copy Section

hcldec

hcldec is a command line tool that transforms HCL input into JSON output
using a decoding specification given by the user.

This tool is intended as a "glue" tool, with use-cases like the following:

• Define a HCL-based configuration format for a third-party tool that takes
  JSON as input, and then translate the HCL configuration into JSON before
  running the tool. (See the npm-package example.)

• Use HCL from languages where a HCL parser/decoder is not yet available.
  At the time of writing, that's any language other than Go.

• In particular, define a HCL-based configuration format for a shell script
  and then use jq to load the result into environment variables for
  further processing. (See the sh-config-file example.)

Installation Copy Chapter

If you have a working Go development environment, you can install this tool
with go get in the usual way:

$ go get -u github.com/hashicorp/hcl/v2/cmd/hcldec

This will install hcldec in $GOPATH/bin, which usually places it into
your shell PATH so you can then run it as hcldec.

Usage Copy Chapter

usage: hcldec --spec=<spec-file> [options] [hcl-file ...]
  -o, --out string          write to the given file, instead of stdout
  -s, --spec string         path to spec file (required)
  -V, --vars json-or-file   provide variables to the given configuration file(s)
  -v, --version             show the version number and immediately exit

The most important step in using hcldec is to write the specification that
defines how to interpret the given configuration files and translate them
into JSON. The following is a simple specification that creates a JSON
object from two top-level attributes in the input configuration:

object {
  attr "name" {
    type     = string
    required = true
  }
  attr "is_member" {
    type = bool
  }
}

Specification files are conventionally kept in files with a .hcldec
extension. We'll call this one example.hcldec.

With the above specification, the following input file example.conf is
valid:

name = "Raul"

The spec and the input file can then be provided to hcldec to extract a
JSON representation:

$ hcldec --spec=example.hcldec example.conf
{"name": "Raul"}

The specification defines both how to map the input into a JSON data structure
and what input is valid. The required = true specified for the name
allows hcldec to detect and raise an error when an attribute of that name
is not provided:

$ hcldec --spec=example.hcldec typo.conf
Error: Unsupported attribute

  on example.conf line 1:
   1: namme = "Juan"

An attribute named "namme" is not expected here. Did you mean "name"?

Error: Missing required attribute

  on example.conf line 2:

The attribute "name" is required, but no definition was found.

Further Reading Copy Chapter

For more details on the .hcldec specification file format, see
the spec file documentation.

Overview (`cmd/hclspecsuite/README.md`) Copy Section

`hclspecsuite`

hclspecsuite is the test harness for
the HCL specification test suite.

Overview (`ext/customdecode/README.md`) Copy Section

HCL Custom Static Decoding Extension

This HCL extension provides a mechanism for defining arguments in an HCL-based
language whose values are derived using custom decoding rules against the
HCL expression syntax, overriding the usual behavior of normal expression
evaluation.

"Arguments", for the purpose of this extension, currently includes the
following two contexts:

• For applications using hcldec for dynamic decoding, a hcldec.AttrSpec
  or hcldec.BlockAttrsSpec can be given a special type constraint that
  opts in to custom decoding behavior for the attribute(s) that are selected
  by that specification.

• When working with the HCL native expression syntax, a function given in
  the hcl.EvalContext during evaluation can have parameters with special
  type constraints that opt in to custom decoding behavior for the argument
  expression associated with that parameter in any call.

The above use-cases are rather abstract, so we'll consider a motivating
real-world example: sometimes we (language designers) need to allow users
to specify type constraints directly in the language itself, such as in
Terraform's Input Variables.
Terraform's variable blocks include an argument called type which takes
a type constraint given using HCL expression building-blocks as defined by
the HCL typeexpr extension.

A "type constraint expression" of that sort is not an expression intended to
be evaluated in the usual way. Instead, the physical expression is
deconstructed using the static analysis operations
to produce a cty.Type as the result, rather than a cty.Value.

The purpose of this Custom Static Decoding Extension, then, is to provide a
bridge to allow that sort of custom decoding to be used via mechanisms that
normally deal in cty.Value, such as hcldec and native syntax function
calls as listed above.

(Note: gohcl has
its own mechanism to support this use case, exploiting the fact that it is
working directly with "normal" Go types. Decoding into a struct field of
type hcl.Expression obtains the expression directly without evaluating it
first. The Custom Static Decoding Extension is not necessary for that gohcl
technique. You can also implement custom decoding by working directly with
the lowest-level HCL API, which separates extraction of and evaluation of
expressions into two steps.)

Custom Decoding Types Copy Chapter

This extension relies on a convention implemented in terms of
Capsule Types in the underlying cty type system. cty allows a capsule type to carry arbitrary
extension metadata values as an aid to creating higher-level abstractions like
this extension.

A custom argument decoding mode, then, is implemented by creating a new cty
capsule type that implements the ExtensionData custom operation to return
a decoding function when requested. For example:

var keywordType cty.Type
keywordType = cty.CapsuleWithOps("keyword", reflect.TypeOf(""), &cty.CapsuleOps{
    ExtensionData: func(key interface{}) interface{} {
        switch key {
        case customdecode.CustomExpressionDecoder:
            return customdecode.CustomExpressionDecoderFunc(
                func(expr hcl.Expression, ctx *hcl.EvalContext) (cty.Value, hcl.Diagnostics) {
                    var diags hcl.Diagnostics
                    kw := hcl.ExprAsKeyword(expr)
                    if kw == "" {
                        diags = append(diags, &hcl.Diagnostic{
                            Severity: hcl.DiagError,
                            Summary:  "Invalid keyword",
                            Detail:   "A keyword is required",
                            Subject:  expr.Range().Ptr(),
                        })
                        return cty.UnkownVal(keywordType), diags
                    }
                    return cty.CapsuleVal(keywordType, &kw)
                },
            )
        default:
            return nil
        }
    },
})

The boilerplate here is a bit fussy, but the important part for our purposes
is the case customdecode.CustomExpressionDecoder: clause, which uses
a custom extension key type defined in this package to recognize when a
component implementing this extension is checking to see if a target type
has a custom decode implementation.

In the above case we've defined a type that decodes expressions as static
keywords, so a keyword like foo would decode as an encapsulated "foo"
string, while any other sort of expression like "baz" or 1 + 1 would
return an error.

We could then use keywordType as a type constraint either for a function
parameter or a hcldec attribute specification, which would require the
argument for that function parameter or the expression for the matching
attributes to be a static keyword, rather than an arbitrary expression.
For example, in a hcldec.AttrSpec:

keywordSpec := &hcldec.AttrSpec{
    Name: "keyword",
    Type: keywordType,
}

The above would accept input like the following and would set its result to
a cty.Value of keywordType, after decoding:

keyword = foo

The Expression and Expression Closure `cty` types Copy Chapter

Building on the above, this package also includes two capsule types that use
the above mechanism to allow calling applications to capture expressions
directly and thus defer analysis to a later step, after initial decoding.

The customdecode.ExpressionType type encapsulates an hcl.Expression alone,
for situations like our type constraint expression example above where it's
the static structure of the expression we want to inspect, and thus any
variables and functions defined in the evaluation context are irrelevant.

The customdecode.ExpressionClosureType type encapsulates a
*customdecode.ExpressionClosure value, which binds the given expression to
the hcl.EvalContext it was asked to evaluate against and thus allows the
receiver of that result to later perform normal evaluation of the expression
with all the same variables and functions that would've been available to it
naturally.

Both of these types can be used as type constraints either for hcldec
attribute specifications or for function arguments. Here's an example of
ExpressionClosureType to implement a function that can evaluate
an expression with some additional variables defined locally, which we'll
call the with(...) function:

/* Detailed source-code truncated for AI context efficiency. */

If the above function were placed into an hcl.EvalContext as with, it
could be used in a native syntax call to that function as follows:

foo = with({name = "Cory"}, "${greeting}, ${name}!")

The above assumes a variable in the main context called greeting, to which
the with function adds name before evaluating the expression given in
its second argument. This makes that second argument context-sensitive -- it
would behave differently if the user wrote the same thing somewhere else -- so
this capability should be used with care to make sure it doesn't cause confusion
for the end-users of your language.

There are some other examples of this capability to evaluate expressions in
unusual ways in the tryfunc directory that is a sibling of this one.

Overview (`ext/dynblock/README.md`) Copy Section

HCL Dynamic Blocks Extension

This HCL extension implements a special block type named "dynamic" that can
be used to dynamically generate blocks of other types by iterating over
collection values.

Normally the block structure in an HCL configuration file is rigid, even
though dynamic expressions can be used within attribute values. This is
convenient for most applications since it allows the overall structure of
the document to be decoded easily, but in some applications it is desirable
to allow dynamic block generation within certain portions of the configuration.

Dynamic block generation is performed using the dynamic block type:

toplevel {
  nested {
    foo = "static block 1"
  }

  dynamic "nested" {
    for_each = ["a", "b", "c"]
    iterator = nested
    content {
      foo = "dynamic block ${nested.value}"
    }
  }

  nested {
    foo = "static block 2"
  }
}

The above is interpreted as if it were written as follows:

toplevel {
  nested {
    foo = "static block 1"
  }

  nested {
    foo = "dynamic block a"
  }

  nested {
    foo = "dynamic block b"
  }

  nested {
    foo = "dynamic block c"
  }

  nested {
    foo = "static block 2"
  }
}

Since HCL block syntax is not normally exposed to the possibility of unknown
values, this extension must make some compromises when asked to iterate over
an unknown collection. If the length of the collection cannot be statically
recognized (because it is an unknown value of list, map, or set type) then
the dynamic construct will generate a single dynamic block whose iterator
key and value are both unknown values of the dynamic pseudo-type, thus causing
any attribute values derived from iteration to appear as unknown values. There
is no explicit representation of the fact that the length of the collection may
eventually be different than one.

Usage Copy Chapter

Pass a body to function Expand to obtain a new body that will, on access
to its content, evaluate and expand any nested dynamic blocks.
Dynamic block processing is also automatically propagated into any nested
blocks that are returned, allowing users to nest dynamic blocks inside
one another and to nest dynamic blocks inside other static blocks.

HCL structural decoding does not normally have access to an EvalContext, so
any variables and functions that should be available to the for_each
and labels expressions must be passed in when calling Expand. Expressions
within the content block are evaluated separately and so can be passed a
separate EvalContext if desired, during normal attribute expression
evaluation.

Detecting Variables Copy Chapter

Some applications dynamically generate an EvalContext by analyzing which
variables are referenced by an expression before evaluating it.

This unfortunately requires some extra effort when this analysis is required
for the context passed to Expand: the HCL API requires a schema to be
provided in order to do any analysis of the blocks in a body, but the low-level
schema model provides a description of only one level of nested blocks at
a time, and thus a new schema must be provided for each additional level of
nesting.

To make this arduous process as convenient as possible, this package provides
a helper function WalkForEachVariables, which returns a WalkVariablesNode
instance that can be used to find variables directly in a given body and also
determine which nested blocks require recursive calls. Using this mechanism
requires that the caller be able to look up a schema given a nested block type.
For simple formats where a specific block type name always has the same schema
regardless of context, a walk can be implemented as follows:

func walkVariables(node dynblock.WalkVariablesNode, schema *hcl.BodySchema) []hcl.Traversal {
	vars, children := node.Visit(schema)

	for _, child := range children {
		var childSchema *hcl.BodySchema
		switch child.BlockTypeName {
		case "a":
			childSchema = &hcl.BodySchema{
				Blocks: []hcl.BlockHeaderSchema{
					{
						Type:       "b",
						LabelNames: []string{"key"},
					},
				},
			}
		case "b":
			childSchema = &hcl.BodySchema{
				Attributes: []hcl.AttributeSchema{
					{
						Name:     "val",
						Required: true,
					},
				},
			}
		default:
			// Should never happen, because the above cases should be exhaustive
			// for the application's configuration format.
			panic(fmt.Errorf("can't find schema for unknown block type %q", child.BlockTypeName))
		}

		vars = append(vars, walkVariables(child.Node, childSchema)...)
	}
}

Detecting Variables with `hcldec` Specifications Copy Section

For applications that use the higher-level hcldec package to decode nested
configuration structures into cty values, the same specification can be used
to automatically drive the recursive variable-detection walk described above.

The helper function ForEachVariablesHCLDec allows an entire recursive
configuration structure to be analyzed in a single call given a hcldec.Spec
that describes the nested block structure. This means a hcldec-based
application can support dynamic blocks with only a little additional effort:

func decodeBody(body hcl.Body, spec hcldec.Spec) (cty.Value, hcl.Diagnostics) {
	// Determine which variables are needed to expand dynamic blocks
	neededForDynamic := dynblock.ForEachVariablesHCLDec(body, spec)

	// Build a suitable EvalContext and expand dynamic blocks
	dynCtx := buildEvalContext(neededForDynamic)
	dynBody := dynblock.Expand(body, dynCtx)

	// Determine which variables are needed to fully decode the expanded body
	// This will analyze expressions that came both from static blocks in the
	// original body and from blocks that were dynamically added by Expand.
	neededForDecode := hcldec.Variables(dynBody, spec)

	// Build a suitable EvalContext and then fully decode the body as per the
	// hcldec specification.
	decCtx := buildEvalContext(neededForDecode)
	return hcldec.Decode(dynBody, spec, decCtx)
}

func buildEvalContext(needed []hcl.Traversal) *hcl.EvalContext {
	// (to be implemented by your application)
}

Performance

This extension is going quite harshly against the grain of the HCL API, and
so it uses lots of wrapping objects and temporary data structures to get its
work done. HCL in general is not suitable for use in high-performance situations
or situations sensitive to memory pressure, but that is especially true for
this extension.

Overview (`ext/tryfunc/README.md`) Copy Section

"Try" and "can" functions

This Go package contains two cty functions intended for use in an
hcl.EvalContext when evaluating HCL native syntax expressions.

The first function try attempts to evaluate each of its argument expressions
in order until one produces a result without any errors.

try(non_existent_variable, 2) # returns 2

If none of the expressions succeed, the function call fails with all of the
errors it encountered.

The second function can is similar except that it ignores the result of
the given expression altogether and simply returns true if the expression
produced a successful result or false if it produced errors.

Both of these are primarily intended for working with deep data structures
which might not have a dependable shape. For example, we can use try to
attempt to fetch a value from deep inside a data structure but produce a
default value if any step of the traversal fails:

result = try(foo.deep[0].lots.of["traversals"], null)

The final result to try should generally be some sort of constant value that
will always evaluate successfully.

Using these functions Copy Chapter

Languages built on HCL can make try and can available to user code by
exporting them in the hcl.EvalContext used for expression evaluation:

ctx := &hcl.EvalContext{
    Functions: map[string]function.Function{
        "try": tryfunc.TryFunc,
        "can": tryfunc.CanFunc,
    },
}

Overview (`ext/typeexpr/README.md`) Copy Section

HCL Type Expressions Extension

This HCL extension defines a convention for describing HCL types using function
call and variable reference syntax, allowing configuration formats to include
type information provided by users.

The type syntax is processed statically from a hcl.Expression, so it cannot
use any of the usual language operators. This is similar to type expressions
in statically-typed programming languages.

variable "example" {
  type = list(string)
}

The extension is built using the hcl.ExprAsKeyword and hcl.ExprCall
functions, and so it relies on the underlying syntax to define how "keyword"
and "call" are interpreted. The above shows how they are interpreted in
the HCL native syntax, while the following shows the same information
expressed in JSON:

{
  "variable": {
    "example": {
      "type": "list(string)"
    }
  }
}

Notice that since we have additional contextual information that we intend
to allow only calls and keywords the JSON syntax is able to parse the given
string directly as an expression, rather than as a template as would be
the case for normal expression evaluation.

For more information, see the godoc reference.

Type Expression Syntax Copy Chapter

When expressed in the native syntax, the following expressions are permitted
in a type expression:

• string - string
• bool - boolean
• number - number
• any - cty.DynamicPseudoType (in function TypeConstraint only)
• list(<type_expr>) - list of the type given as an argument
• set(<type_expr>) - set of the type given as an argument
• map(<type_expr>) - map of the type given as an argument
• tuple([<type_exprs...>]) - tuple with the element types given in the single list argument
• object({<attr_name>=<type_expr>, ...} - object with the attributes and corresponding types given in the single map argument

For example:

• list(string)
• object({name=string,age=number})
• map(object({name=string,age=number}))

Note that the object constructor syntax is not fully-general for all possible
object types because it requires the attribute names to be valid identifiers.
In practice it is expected that any time an object type is being fixed for
type checking it will be one that has identifiers as its attributes; object
types with weird attributes generally show up only from arbitrary object
constructors in configuration files, which are usually treated either as maps
or as the dynamic pseudo-type.

Optional Object Attributes Copy Section

As part of object expressions attributes can be marked as optional. Missing
object attributes would typically result in an error when type constraints are
validated or used. Optional missing attributes, however, would not result in an
error. The cty "convert" function will populate
missing optional attributes with null values.

For example:

• object({name=string,age=optional(number)})

Optional attributes can also be specified with default values. The
TypeConstraintWithDefaults function will return a Defaults object that can
be used to populate missing optional attributes with defaults in a given
cty.Value.

For example:

• object({name=string,age=optional(number, 0)})

Type Constraints as Values Copy Chapter

Along with defining a convention for writing down types using HCL expression
constructs, this package also includes a mechanism for representing types as
values that can be used as data within an HCL-based language.

typeexpr.TypeConstraintType is a
cty capsule type
that encapsulates cty.Type values. You can construct such a value directly
using the TypeConstraintVal function:

tyVal := typeexpr.TypeConstraintVal(cty.String)

// We can unpack the type from a value using TypeConstraintFromVal
ty := typeExpr.TypeConstraintFromVal(tyVal)

However, the primary purpose of typeexpr.TypeConstraintType is to be
specified as the type constraint for an argument, in which case it serves
as a signal for HCL to treat the argument expression as a type constraint
expression as defined above, rather than as a normal value expression.

"An argument" in the above in practice means the following two locations:

• As the type constraint for a parameter of a cty function that will be
  used in an hcl.EvalContext. In that case, function calls in the HCL
  native expression syntax will require the argument to be valid type constraint
  expression syntax and the function implementation will receive a
  TypeConstraintType value as the argument value for that parameter.

• As the type constraint for a hcldec.AttrSpec or hcldec.BlockAttrsSpec
  when decoding an HCL body using hcldec. In that case, the attributes
  with that type constraint will be required to be valid type constraint
  expression syntax and the result will be a TypeConstraintType value.

Note that the special handling of these arguments means that an argument
marked in this way must use the type constraint syntax directly. It is not
valid to pass in a value of TypeConstraintType that has been obtained
dynamically via some other expression result.

TypeConstraintType is provided with the intent of using it internally within
application code when incorporating type constraint expression syntax into
an HCL-based language, not to be used for dynamic "programming with types". A
calling application could support programming with types by defining its own
capsule type, but that is not the purpose of TypeConstraintType.

The "convert" `cty` Function Copy Chapter

Building on the TypeConstraintType described in the previous section, this
package also provides typeexpr.ConvertFunc which is a cty function that
can be placed into a cty.EvalContext (conventionally named "convert") in
order to provide a general type conversion function in an HCL-based language:

foo = convert("true", bool)

The second parameter uses the mechanism described in the previous section to
require its argument to be a type constraint expression rather than a value
expression. In doing so, it allows converting with any type constraint that
can be expressed in this package's type constraint syntax. In the above example,
the foo argument would receive a boolean true, or cty.True in cty terms.

The target type constraint must always be provided statically using inline
type constraint syntax. There is no way to dynamically select a type
constraint using this function.

Overview (`ext/userfunc/README.md`) Copy Section

HCL User Functions Extension

This HCL extension allows a calling application to support user-defined
functions.

Functions are defined via a specific block type, like this:

function "add" {
  params = [a, b]
  result = a + b
}

function "list" {
  params         = []
  variadic_param = items
  result         = items
}

The extension is implemented as a pre-processor for cty.Body objects. Given
a body that may contain functions, the DecodeUserFunctions function searches
for blocks that define functions and returns a functions map suitable for
inclusion in a hcl.EvalContext. It also returns a new cty.Body that
contains the remainder of the content from the given body, allowing for
further processing of remaining content.

For more information, see the godoc reference.

Overview (`hclsyntax/fuzz/README.md`) Copy Section

hclsyntax fuzzing utilities

This directory contains helper functions and corpora that can be used to
fuzz-test the hclsyntax parsers using Go's native fuzz testing capabilities.

Please see https://go.dev/doc/fuzz/ for more information on fuzzing.

Prerequisites Copy Chapter

• Go 1.18

Running the fuzzer Copy Chapter

Each exported function in the hclsyntax package has a corresponding fuzz test.
These can be run one at a time via go test:

$ cd fuzz
$ go test -fuzz FuzzParseTemplate
$ go test -fuzz FuzzParseTraversalAbs
$ go test -fuzz FuzzParseExpression
$ go test -fuzz FuzzParseConfig

This command will exit only when a crasher is found (see "Understanding the
result" below.)

Seed corpus Copy Chapter

The seed corpus for each fuzz test function is stored in the corresponding
directory under hclsyntax/fuzz/testdata/fuzz/. For example:

$ ls hclsyntax/fuzz/testdata/fuzz/FuzzParseTemplate
empty.tmpl
escape-dollar.tmpl
escape-newline-tmpl
...

Additional seed inputs can be added to this corpus. Each file must be in the Go 1.18 corpus file format. Files can be converted to this format using the file2fuzz tool. To install it:

$ go install golang.org/x/tools/cmd/file2fuzz@latest
$ file2fuzz -help

Understanding the result Copy Chapter

A small number of subdirectories will be created in the work directory.

If you let go-fuzz run for a few minutes (the more minutes the better) it
may detect "crashers", which are inputs that caused the parser to panic.
These are written to hclsyntax/fuzz/testdata/fuzz/<fuzz test name>/:

$ ls hclsyntax/fuzz/testdata/fuzz/FuzzParseTemplate
582528ddfad69eb57775199a43e0f9fd5c94bba343ce7bb6724d4ebafe311ed4

A good first step to fixing a detected crasher is to copy the failing input
into one of the unit tests in the hclsyntax package and see it crash there
too. After that, it's easy to re-run the test as you try to fix it.

Overview (`hclwrite/fuzz/README.md`) Copy Section

hclwrite fuzzing utilities

This directory contains helper functions and corpora that can be used to
fuzz-test the hclwrite parsers using Go's native fuzz testing capabilities.

Please see https://go.dev/doc/fuzz/ for more information on fuzzing.

Prerequisites Copy Chapter

• Go 1.18

Running the fuzzer Copy Chapter

Each exported function in the hclwrite package has a corresponding fuzz test.
These can be run one at a time via go test:

$ cd fuzz
$ go test -fuzz FuzzParseConfig

This command will exit only when a crasher is found (see "Understanding the
result" below.)

Seed corpus Copy Chapter

The seed corpus for each fuzz test function is stored in the corresponding
directory under hclwrite/fuzz/testdata/fuzz/FuzzTest. For example:

$ ls hclwrite/fuzz/testdata/fuzz/FuzzParseConfig
attr-expr.hcl
attr.hcl
attr-literal.hcl
...

Additional seed inputs can be added to this corpus. Each file must be in the Go 1.18 corpus file format. Files can be converted to this format using the file2fuzz tool. To install it:

$ go install golang.org/x/tools/cmd/file2fuzz@latest
$ file2fuzz -help

Understanding the result Copy Chapter

A small number of subdirectories will be created in the work directory.

If you let go-fuzz run for a few minutes (the more minutes the better) it
may detect "crashers", which are inputs that caused the parser to panic.
These are written to hclwrite/fuzz/testdata/fuzz/<fuzz test name>/:

$ ls hclwrite/fuzz/testdata/fuzz/FuzzParseConfig
582528ddfad69eb57775199a43e0f9fd5c94bba343ce7bb6724d4ebafe311ed4

A good first step to fixing a detected crasher is to copy the failing input
into one of the unit tests in the hclwrite package and see it crash there
too. After that, it's easy to re-run the test as you try to fix it.

Overview (`json/fuzz/README.md`) Copy Section

JSON syntax fuzzing utilities

This directory contains helper functions and corpora that can be used to
fuzz-test the HCL JSON parser using Go's native fuzz testing capabilities.

Please see https://go.dev/doc/fuzz/ for more information on fuzzing.

Prerequisites Copy Chapter

• Go 1.18

Running the fuzzer Copy Chapter

Each exported function in the json package has a corresponding fuzz test.
These can be run one at a time via go test:

$ cd fuzz
$ go test -fuzz FuzzParse

This command will exit only when a crasher is found (see "Understanding the
result" below).

Seed corpus Copy Chapter

The seed corpus for each fuzz test function is stored in the corresponding
directory under json/fuzz/testdata/fuzz. For example:

$ ls json/fuzz/testdata/fuzz/FuzzParse
attr-expr.hcl.json
attr-literal.hcl.json
block-attrs.hcl.json
...

Additional seed inputs can be added to this corpus. Each file must be in the Go 1.18 corpus file format. Files can be converted to this format using the file2fuzz tool. To install it:

$ go install golang.org/x/tools/cmd/file2fuzz@latest
$ file2fuzz -help

Understanding the result Copy Chapter

A small number of subdirectories will be created in the work directory.

If you let go-fuzz run for a few minutes (the more minutes the better) it
may detect "crashers", which are inputs that caused the parser to panic.
These are written to json/fuzz/testdata/fuzz/<fuzz test name>/:

$ ls json/fuzz/testdata/fuzz/FuzzParseTemplate
582528ddfad69eb57775199a43e0f9fd5c94bba343ce7bb6724d4ebafe311ed4

A good first step to fixing a detected crasher is to copy the failing input
into one of the unit tests in the json package and see it crash there
too. After that, it's easy to re-run the test as you try to fix it.

Guide Go (`guide/go.rst`) Copy Section

Using HCL in a Go application

HCL is itself written in Go_ and currently it is primarily intended for use as
a library within other Go programs.

This section describes a number of different ways HCL can be used to define
and process a configuration language within a Go program. For simple situations,
HCL can decode directly into Go struct values in a similar way as encoding
packages such as encoding/json and encoding/xml.

The HCL Go API also offers some alternative approaches however, for processing
languages that may be more complex or that include portions whose expected
structure cannot be determined until runtime.

The following sections give an overview of different ways HCL can be used in
a Go program.

.. toctree::
:maxdepth: 1
:caption: Sub-sections:

go_parsing
go_diagnostics
go_decoding_gohcl
go_decoding_hcldec
go_expression_eval
go_decoding_lowlevel
go_patterns

.. _Go: https://golang.org/

Guide Go Decoding Gohcl (`guide/go_decoding_gohcl.rst`) Copy Section

.. go:package:: gohcl

.. _go-decoding-gohcl:

Decoding Into Native Go Values

The most straightforward way to access the content of an HCL file is to
decode into native Go values using reflect, similar to the technique used
by packages like encoding/json and encoding/xml.

Package gohcl provides functions for this sort of decoding. Function
DecodeBody attempts to extract values from an HCL body and write them
into a Go value given as a pointer:

.. code-block:: go

type ServiceConfig struct {
Type string hcl:"type,label"
Name string hcl:"name,label"
ListenAddr string hcl:"listen_addr"
}
type Config struct {
IOMode string hcl:"io_mode"
Services []ServiceConfig hcl:"service,block"
}

var c Config
moreDiags := gohcl.DecodeBody(f.Body, nil, &c)
diags = append(diags, moreDiags...)

The above example decodes the root body of a file f, presumably loaded
previously using a parser, into the variable c. The field labels within
the struct types imply the schema of the expected language, which is a cut-down
version of the hypothetical language we showed in :ref:intro.

The struct field labels consist of two comma-separated values. The first is
the name of the corresponding argument or block type as it will appear in
the input file, and the second is the type of element being named. If the
second value is omitted, it defaults to attr, requesting an attribute.

Nested blocks are represented by a struct or a slice of that struct, and the
special element type label within that struct declares that each instance
of that block type must be followed by one or more block labels. In the above
example, the service block type is defined to require two labels, named
type and name. For label fields in particular, the given name is used
only to refer to the particular label in error messages when the wrong number
of labels is used.

By default, all declared attributes and blocks are considered to be required.
An optional value is indicated by making its field have a pointer type, in
which case nil is written to indicate the absense of the argument.

The sections below discuss some additional decoding use-cases. For full details
on the gohcl package, see
the godoc reference <https://godoc.org/github.com/hashicorp/hcl/v2/gohcl>_.

.. _go-decoding-gohcl-evalcontext:

Variables and Functions Copy Chapter

By default, arguments given in the configuration may use only literal values
and the built in expression language operators, such as arithmetic.

The second argument to gohcl.DecodeBody, shown as nil in the previous
example, allows the calling application to additionally offer variables and
functions for use in expressions. Its value is a pointer to an
hcl.EvalContext, which will be covered in more detail in the later section
:ref:go-expression-eval. For now, a simple example of making the id of the
current process available as a single variable called pid:

.. code-block:: go

type Context struct {
Pid string
}
ctx := gohcl.EvalContext(&Context{
Pid: os.Getpid()
})
var c Config
moreDiags := gohcl.DecodeBody(f.Body, ctx, &c)
diags = append(diags, moreDiags...)

gohcl.EvalContext constructs an expression evaluation context from a Go
struct value, making the fields available as variables and the methods
available as functions, after transforming the field and method names such
that each word (starting with an uppercase letter) is all lowercase and
separated by underscores.

.. code-block:: hcl

name = "example-program (${pid})"

Partial Decoding Copy Chapter

In the examples so far, we've extracted the content from the entire input file
in a single call to DecodeBody. This is sufficient for many simple
situations, but sometimes different parts of the file must be evaluated
separately. For example:

• If different parts of the file must be evaluated with different variables
  or functions available.

• If the result of evaluating one part of the file is used to set variables
  or functions in another part of the file.

There are several ways to perform partial decoding with gohcl, all of
which involve decoding into HCL's own types, such as hcl.Body.

The most general approach is to declare an additional struct field of type
hcl.Body, with the special field tag type remain:

.. code-block:: go

type ServiceConfig struct {
Type string hcl:"type,label"
Name string hcl:"name,label"
ListenAddr string hcl:"listen_addr"
Remain hcl.Body hcl:",remain"
}

When a remain field is present, any element of the input body that is
not matched is retained in a body saved into that field, which can then be
decoded in a later call, potentially with a different evaluation context.

Another option is to decode an attribute into a value of type hcl.Expression,
which can then be evaluated separately as described in
:ref:expression-eval.

Guide Go Decoding Hcldec (`guide/go_decoding_hcldec.rst`) Copy Section

.. go:package:: hcldec

.. _go-decoding-hcldec:

Decoding With Dynamic Schema

In section :ref:go-decoding-gohcl, we saw the most straightforward way to
access the content from an HCL file, decoding directly into a Go value whose
type is known at application compile time.

For some applications, it is not possible to know the schema of the entire
configuration when the application is built. For example, HashiCorp Terraform_
uses HCL as the foundation of its configuration language, but parts of the
configuration are handled by plugins loaded dynamically at runtime, and so
the schemas for these portions cannot be encoded directly in the Terraform
source code.

HCL's hcldec package offers a different approach to decoding that allows
schemas to be created at runtime, and the result to be decoded into
dynamically-typed data structures.

The sections below are an overview of the main parts of package hcldec.
For full details, see
the package godoc <https://godoc.org/github.com/hashicorp/hcl/v2/hcldec>_.

.. _HashiCorp Terraform: https://www.terraform.io/

Decoder Specification Copy Chapter

Whereas :go:pkg:gohcl infers the expected schema by using reflection against
the given value, hcldec obtains schema through a decoding specification,
which is a set of instructions for mapping HCL constructs onto a dynamic
data structure.

The hcldec package contains a number of different specifications, each
implementing :go:type:hcldec.Spec and having a Spec suffix on its name.
Each spec has two distinct functions:

• Adding zero or more validation constraints on the input configuration file.

• Producing a result value based on some elements from the input file.

The most common pattern is for the top-level spec to be a
:go:type:hcldec.ObjectSpec with nested specifications defining either blocks
or attributes, depending on whether the configuration file will be
block-structured or flat.

.. code-block:: go

spec := hcldec.ObjectSpec{
"io_mode": &hcldec.AttrSpec{
Name: "io_mode",
Type: cty.String,
},
"services": &hcldec.BlockMapSpec{
TypeName: "service",
LabelNames: []string{"type", "name"},
Nested: hcldec.ObjectSpec{
"listen_addr": &hcldec.AttrSpec{
Name: "listen_addr",
Type: cty.String,
Required: true,
},
"processes": &hcldec.BlockMapSpec{
TypeName: "service",
LabelNames: []string{"name"},
Nested: hcldec.ObjectSpec{
"command": &hcldec.AttrSpec{
Name: "command",
Type: cty.List(cty.String),
Required: true,
},
},
},
},
},
}
val, moreDiags := hcldec.Decode(f.Body, spec, nil)
diags = append(diags, moreDiags...)

The above specification expects a configuration shaped like our example in
:ref:intro, and calls for it to be decoded into a dynamic data structure
that would have the following shape if serialized to JSON:

.. code-block:: JSON

{
"io_mode": "async",
"services": {
"http": {
"web_proxy": {
"listen_addr": "127.0.0.1:8080",
"processes": {
"main": {
"command": ["/usr/local/bin/awesome-app", "server"]
},
"mgmt": {
"command": ["/usr/local/bin/awesome-app", "mgmt"]
}
}
}
}
}
}

.. go:package:: cty

Types and Values With ``cty`` Copy Chapter

HCL's expression interpreter is implemented in terms of another library called
:go:pkg:cty, which provides a type system which HCL builds on and a robust
representation of dynamic values in that type system. You could think of
:go:pkg:cty as being a bit like Go's own :go:pkg:reflect, but for the
results of HCL expressions rather than Go programs.

The full details of this system can be found in
its own repository <https://github.com/zclconf/go-cty>_, but this section
will cover the most important highlights, because hcldec specifications
include :go:pkg:cty types (as seen in the above example) and its results are
:go:pkg:cty values.

hcldec works directly with :go:pkg:cty — as opposed to converting values
directly into Go native types — because the functionality of the :go:pkg:cty
packages then allows further processing of those values without any loss of
fidelity or range. For example, :go:pkg:cty defines a JSON encoding of its
values that can be decoded losslessly as long as both sides agree on the value
type that is expected, which is a useful capability in systems where some sort
of RPC barrier separates the main program from its plugins.

Types are instances of :go:type:cty.Type, and are constructed from functions
and variables in :go:pkg:cty as shown in the above example, where the string
attributes are typed as cty.String, which is a primitive type, and the list
attribute is typed as cty.List(cty.String), which constructs a new list
type with string elements.

Values are instances of :go:type:cty.Value, and can also be constructed from
functions in :go:pkg:cty, using the functions that include Val in their
names or using the operation methods available on :go:type:cty.Value.

In most cases you will eventually want to use the resulting data as native Go
types, to pass it to non-:go:pkg:cty-aware code. To do this, see the guides
on
Converting between types <https://github.com/zclconf/go-cty/blob/master/docs/convert.md>_
(staying within :go:pkg:cty) and
Converting to and from native Go values <https://github.com/zclconf/go-cty/blob/master/docs/gocty.md>_.

Partial Decoding Copy Chapter

Because the hcldec result is always a value, the input is always entirely
processed in a single call, unlike with :go:pkg:gohcl.

However, both :go:pkg:gohcl and :go:pkg:hcldec take :go:type:hcl.Body as
the representation of input, and so it is possible and common to mix them both
in the same program.

A common situation is that :go:pkg:gohcl is used in the main program to
decode the top level of configuration, which then allows the main program to
determine which plugins need to be loaded to process the leaf portions of
configuration. In this case, the portions that will be interpreted by plugins
are retained as opaque :go:type:hcl.Body until the plugins have been loaded,
and then each plugin provides its :go:type:hcldec.Spec to allow decoding the
plugin-specific configuration into a :go:type:cty.Value which be
transmitted to the plugin for further processing.

In our example from :ref:intro, perhaps each of the different service types
is managed by a plugin, and so the main program would decode the block headers
to learn which plugins are needed, but process the block bodies dynamically:

.. code-block:: go

type ServiceConfig struct {
Type string hcl:"type,label"
Name string hcl:"name,label"
PluginConfig hcl.Body hcl:",remain"
}
type Config struct {
IOMode string hcl:"io_mode"
Services []ServiceConfig hcl:"service,block"
}

var c Config
moreDiags := gohcl.DecodeBody(f.Body, nil, &c)
diags = append(diags, moreDiags...)
if moreDiags.HasErrors() {
// (show diags in the UI)
return
}

for _, sc := range c.Services {
pluginName := block.Type

// Totally-hypothetical plugin manager (not part of HCL)
   plugin, err := pluginMgr.GetPlugin(pluginName)
   if err != nil {
       diags = diags.Append(&hcl.Diagnostic{ /* ... */ })
       continue
   }
   spec := plugin.ConfigSpec() // returns hcldec.Spec

   // Decode the block body using the plugin's given specification
   configVal, moreDiags := hcldec.Decode(sc.PluginConfig, spec, nil)
   diags = append(diags, moreDiags...)
   if moreDiags.HasErrors() {
       continue
   }

   // Again, hypothetical API within your application itself, and not
   // part of HCL. Perhaps plugin system serializes configVal as JSON
   // and sends it over to the plugin.
   svc := plugin.NewService(configVal)
   serviceMgr.AddService(sc.Name, svc)

}

Variables and Functions Copy Chapter

The final argument to hcldec.Decode is an expression evaluation context,
just as with gohcl.DecodeBlock.

This object can be constructed using
:ref:the gohcl helper function <go-decoding-gohcl-evalcontext> as before if desired, but
you can also choose to work directly with :go:type:hcl.EvalContext as
discussed in :ref:go-expression-eval:

.. code-block:: go

ctx := &hcl.EvalContext{
Variables: map[string]cty.Value{
"pid": cty.NumberIntVal(int64(os.Getpid())),
},
}
val, moreDiags := hcldec.Decode(f.Body, spec, ctx)
diags = append(diags, moreDiags...)

As you can see, this lower-level API also uses :go:pkg:cty, so it can be
particularly convenient in situations where the result of dynamically decoding
one block must be available to expressions in another block.

Guide Go Decoding Lowlevel (`guide/go_decoding_lowlevel.rst`) Copy Section

.. _go-decoding-lowlevel:

Advanced Decoding With The Low-level API

In previous sections we've discussed :go:pkg:gohcl and :go:pkg:hcldec,
which both deal with decoding of HCL bodies and the expressions within them
using a high-level description of the expected configuration schema.
Both of these packages are implemented in terms of HCL's low-level decoding
interfaces, which we will explore in this section.

HCL decoding in the low-level API has two distinct phases:

• Structural decoding: analyzing the arguments and nested blocks present in a
  particular body.

• Expression evaluation: obtaining final values for each argument expression
  found during structural decoding.

The low-level API gives the calling application full control over when each
body is decoded and when each expression is evaluated, allowing for more
complex configuration formats where e.g. different variables are available in
different contexts, or perhaps expressions within one block can refer to
values defined in another block.

The low-level API also gives more detailed access to source location
information for decoded elements, and so may be desirable for applications that
do a lot of additional validation of decoded data where more specific source
locations lead to better diagnostic messages.

Since all of the decoding mechanisms work with the same :go:type:hcl.Body
type, it is fine and expected to mix them within an application to get access
to the more detailed information where needed while using the higher-level APIs
for the more straightforward portions of a configuration language.

The following subsections will give an overview of the low-level API. For full
details, see the godoc reference <https://godoc.org/github.com/hashicorp/hcl/v2/hcl>_.

Structural Decoding Copy Chapter

As seen in prior sections, :go:type:hcl.Body is an opaque representation of
the arguments and child blocks at a particular nesting level. An HCL file has
a root body containing the top-level elements, and then each nested block has
its own body presenting its own content.

:go:type:hcl.Body is a Go interface whose methods serve as the structural
decoding API:

.. go:currentpackage:: hcl

.. go:type:: Body

Represents the structural elements at a particular nesting level.

.. go:function:: func (b Body) Content(schema *BodySchema) (*BodyContent, Diagnostics)

Decode the content from the receiving body using the given schema. The
  schema is considered exhaustive of all content within the body, and so
  any elements not covered by the schema will generate error diagnostics.

.. go:function:: func (b Body) PartialContent(schema *BodySchema) (*BodyContent, Body, Diagnostics)

Similar to `Content`, but allows for additional arguments and block types
  that are not described in the given schema. The additional body return
  value is a special body that contains only the *remaining* elements, after
  extraction of the ones covered by the schema. This returned body can be
  used to decode the remaining content elsewhere in the calling program.

.. go:function:: func (b Body) JustAttributes() (Attributes, Diagnostics)

Decode the content from the receving body in a special *attributes-only*
  mode, allowing the calling application to enumerate the arguments given
  inside the body without needing to predict them in schema.

  When this method is used, a body can be treated somewhat like a map
  expression, but it still has a rigid structure where the arguments must
  be given directly with no expression evaluation. This is an advantage for
  declarations that must themselves be resolved before expression
  evaluation is possible.

  If the body contains any blocks, error diagnostics are returned. JSON
  syntax relies on schema to distinguish arguments from nested blocks, and
  so a JSON body in attributes-only mode will treat all JSON object
  properties as arguments.

.. go:function:: func (b Body) MissingItemRange() Range

Returns a source range that points to where an absent required item in
  the body might be placed. This is a "best effort" sort of thing, required
  only to be somewhere inside the receving body, as a way to give source
  location information for a "missing required argument" sort of error.

The main content-decoding methods each require a :go:type:hcl.BodySchema
object describing the expected content. The fields of this type describe the
expected arguments and nested block types respectively:

.. code-block:: go

schema := &hcl.BodySchema{
Attributes: []hcl.AttributeSchema{
{
Name: "io_mode",
Required: false,
},
},
Blocks: []hcl.BlockHeaderSchema{
{
Type: "service",
LabelNames: []string{"type", "name"},
},
},
}
content, moreDiags := body.Content(schema)
diags = append(diags, moreDiags...)

:go:type:hcl.BodyContent is the result of both Content and
PartialContent, giving the actual attributes and nested blocks that were
found. Since arguments are uniquely named within a body and unordered, they
are returned as a map. Nested blocks are ordered and may have many instances
of a given type, so they are returned all together in a single slice for
further interpretation by the caller.

Unlike the two higher-level approaches, the low-level API always works only
with one nesting level at a time. Decoding a nested block returns the "header"
for that block, giving its type and label values, but its body remains an
:go:type:hcl.Body for later decoding.

Each returned attribute corresponds to one of the arguments in the body, and
it has an :go:type:hcl.Expression object that can be used to obtain a value
for the argument during expression evaluation, as described in the next
section.

Expression Evaluation Copy Chapter

Expression evaluation in general has its own section, imaginitively titled
:ref:go-expression-eval, so this section will focus only on how it is
achieved in the low-level API.

All expression evaluation in the low-level API starts with an
:go:type:hcl.Expression object. This is another interface type, with various
implementations depending on the expression type and the syntax it was parsed
from.

.. go:currentpackage:: hcl

.. go:type:: Expression

Represents a unevaluated single expression.

.. go:function:: func (e Expression) Value(ctx *EvalContext) (cty.Value, Diagnostics)

Evaluates the receiving expression in the given evaluation context. The
  result is a :go:type:`cty.Value` representing the result value, along
  with any diagnostics that were raised during evaluation.

  If the diagnostics contains errors, the value may be incomplete or
  invalid and should either be discarded altogether or used with care for
  analysis.

.. go:function:: func (e Expression) Variables() []Traversal

Returns information about any nested expressions that access variables
  from the *global* evaluation context. Does not include references to
  temporary local variables, such as those generated by a
  "``for`` expression".

.. go:function:: func (e Expression) Range() Range

Returns the source range for the entire expression. This can be useful
  when generating application-specific diagnostic messages, such as
  value validation errors.

.. go:function:: func (e Expression) StartRange() Range

Similar to ``Range``, but if the expression is complex, such as a tuple
  or object constructor, may indicate only the opening tokens for the
  construct to avoid creating an overwhelming source code snippet.

  This should be used in diagnostic messages only in situations where the
  error is clearly with the construct itself and not with the overall
  expression. For example, a type error indicating that a tuple was not
  expected might use ``StartRange`` to draw attention to the beginning
  of a tuple constructor, without highlighting the entire expression.

Method Value is the primary API for expressions, and takes the same kind
of evaluation context object described in :ref:go-expression-eval.

.. code-block:: go

ctx := &hcl.EvalContext{
Variables: map[string]cty.Value{
"name": cty.StringVal("Ermintrude"),
"age": cty.NumberIntVal(32),
},
}
val, moreDiags := expr.Value(ctx)
diags = append(diags, moreDiags...)

Guide Go Diagnostics (`guide/go_diagnostics.rst`) Copy Section

.. _go-diagnostics:

Diagnostic Messages

An important concern for any machine language intended for human authoring is
to produce good error messages when the input is somehow invalid, or has
other problems.

HCL uses diagnostics to describe problems in an end-user-oriented manner,
such that the calling application can render helpful error or warning messages.
The word "diagnostic" is a general term that covers both errors and warnings,
where errors are problems that prevent complete processing while warnings are
possible concerns that do not block processing.

HCL deviates from usual Go API practice by returning its own hcl.Diagnostics
type, instead of Go's own error type. This allows functions to return
warnings without accompanying errors while not violating the usual expectation
that the absense of errors is indicated by a nil error.

In order to easily accumulate and return multiple diagnostics at once, the
usual pattern for functions returning diagnostics is to gather them in a
local variable and then return it at the end of the function, or possibly
earlier if the function cannot continue due to the problems.

.. code-block:: go

func returningDiagnosticsExample() hcl.Diagnostics {
var diags hcl.Diagnostics

// ...

  // Call a function that may itself produce diagnostics.
  f, moreDiags := parser.LoadHCLFile("example.conf")
  // always append, in case warnings are present
  diags = append(diags, moreDiags...)
  if diags.HasErrors() {
    // If we can't safely continue in the presence of errors here, we
    // can optionally return early.
    return diags
  }

  // ...

  return diags

}

A common variant of the above pattern is calling another diagnostics-generating
function in a loop, using continue to begin the next iteration when errors
are detected, but still completing all iterations and returning the union of
all of the problems encountered along the way.

In :ref:go-parsing, we saw that the parser can generate diagnostics which
are related to syntax problems within the loaded file. Further steps to decode
content from the loaded file can also generate diagnostics related to semantic
problems within the file, such as invalid expressions or type mismatches, and
so a program using HCL will generally need to accumulate diagnostics across
these various steps and then render them in the application UI somehow.

Rendering Diagnostics in the UI Copy Chapter

The best way to render diagnostics to an end-user will depend a lot on the
type of application: they might be printed into a terminal, written into a
log for later review, or even shown in a GUI.

HCL leaves the responsibility for rendering diagnostics to the calling
application, but since rendering to a terminal is a common case for command-line
tools, the hcl package contains a default implementation of this in the
form of a "diagnostic text writer":

.. code-block:: go

wr := hcl.NewDiagnosticTextWriter(
os.Stdout, // writer to send messages to
parser.Files(), // the parser's file cache, for source snippets
78, // wrapping width
true, // generate colored/highlighted output
)
wr.WriteDiagnostics(diags)

This default implementation of diagnostic rendering includes relevant lines
of source code for context, like this:

::

Error: Unsupported block type

on example.tf line 4, in resource "aws_instance" "example":
 2: provisionr "local-exec" {

Blocks of type "provisionr" are not expected here. Did you mean "provisioner"?

If the "color" flag is enabled, the severity will be additionally indicated by
a text color and the relevant portion of the source code snippet will be
underlined to draw further attention.

Guide Go Expression Eval (`guide/go_expression_eval.rst`) Copy Section

.. _go-expression-eval:

Expression Evaluation

Each argument attribute in a configuration file is interpreted as an
expression. In the HCL native syntax, certain basic expression functionality
is always available, such as arithmetic and template strings, and the calling
application can extend this by making available specific variables and/or
functions via an evaluation context.

We saw in :ref:go-decoding-gohcl and :ref:go-decoding-hcldec some basic
examples of populating an evaluation context to make a variable available.
This section will look more closely at the hcl.EvalContext type and how
HCL expression evaluation behaves in different cases.

This section does not discuss in detail the expression syntax itself. For more
information on that, see the HCL Native Syntax specification.

.. go:currentpackage:: hcl

.. go:type:: EvalContext

hcl.EvalContext is the type used to describe the variables and functions
available during expression evaluation, if any. Its usage is described in
the following sections.

Defining Variables Copy Chapter

As we saw in :ref:go-decoding-hcldec, HCL represents values using an
underlying library called :go:pkg:cty. When defining variables, their values
must be given as :go:type:cty.Value values.

A full description of the types and value constructors in :go:pkg:cty is
in the reference documentation <https://github.com/zclconf/go-cty/blob/master/docs/types.md>_.
Variables in HCL are defined by assigning values into a map from string names
to :go:type:cty.Value:

.. code-block:: go

ctx := &hcl.EvalContext{
Variables: map[string]cty.Value{
"name": cty.StringVal("Ermintrude"),
"age": cty.NumberIntVal(32),
},
}

If this evaluation context were passed to one of the evaluation functions we
saw in previous sections, the user would be able to refer to these variable
names in any argument expression appearing in the evaluated portion of
configuration:

.. code-block:: hcl

message = "${name} is ${age} ${age == 1 ? "year" : "years"} old!"

If you place cty's object values in the evaluation context, then their
attributes can be referenced using the HCL attribute syntax, allowing for more
complex structures:

.. code-block:: go

ctx := &hcl.EvalContext{
Variables: map[string]cty.Value{
"path": cty.ObjectVal(map[string]cty.Value{
"root": cty.StringVal(rootDir),
"module": cty.StringVal(moduleDir),
"current": cty.StringVal(currentDir),
}),
},
}

.. code-block:: hcl

source_file = "${path.module}/foo.txt"

.. _go-expression-funcs:

Defining Functions Copy Chapter

Custom functions can be defined by you application to allow users of its
language to transform data in application-specific ways. The underlying
function mechanism is also provided by :go:pkg:cty, allowing you to define
the arguments a given function expects, what value type it will return for
given argument types, etc. The full functions model is described in the
cty documentation section
Functions System <https://github.com/zclconf/go-cty/blob/master/docs/functions.md>_.

There are a number of "standard library" functions <https://godoc.org/github.com/apparentlymart/go-cty/cty/function/stdlib>_
available in a stdlib package within the :go:pkg:cty repository, avoiding
the need for each application to re-implement basic functions for string
manipulation, list manipulation, etc. It also includes function-shaped versions
of several operations that are native operators in HCL, which should generally
not be exposed as functions in HCL-based configuration formats to avoid user
confusion.

You can define functions in the Functions field of :go:type:hcl.EvalContext:

.. code-block:: go

ctx := &hcl.EvalContext{
Variables: map[string]cty.Value{
"name": cty.StringVal("Ermintrude"),
},
Functions: map[string]function.Function{
"upper": stdlib.UpperFunc,
"lower": stdlib.LowerFunc,
"min": stdlib.MinFunc,
"max": stdlib.MaxFunc,
"strlen": stdlib.StrlenFunc,
"substr": stdlib.SubstrFunc,
},
}

If this evaluation context were passed to one of the evaluation functions we
saw in previous sections, the user would be able to call any of these functions
in any argument expression appearing in the evaluated portion of configuration:

.. code-block:: hcl

message = "HELLO, ${upper(name)}!"

Expression Evaluation Modes Copy Chapter

HCL uses a different expression evaluation mode depending on the evaluation
context provided. In HCL native syntax, evaluation modes are used to provide
more relevant error messages. In JSON syntax, which embeds the native
expression syntax in strings using "template" syntax, the evaluation mode
determines whether strings are evaluated as templates at all.

If the given :go:type:hcl.EvalContext is nil, native syntax expressions
will react to users attempting to refer to variables or functions by producing
errors indicating that these features are not available at all, rather than
by saying that the specific variable or function does not exist. JSON syntax
strings will not be evaluated as templates at all in this mode, making them
function as literal strings.

If the evaluation context is non-nil but either Variables or
Functions within it is nil, native syntax will similarly produce
"not supported" error messages. JSON syntax strings will parse templates
in this case, but can also generate "not supported" messages if e.g. the
user accesses a variable when the variables map is nil.

If neither map is nil, HCL assumes that both variables and functions are
supported and will instead produce error messages stating that the specific
variable or function accessed by the user is not defined.

Guide Go Parsing (`guide/go_parsing.rst`) Copy Section

.. _go-parsing:

Parsing HCL Input

The first step in processing HCL input provided by a user is to parse it.
Parsing turns the raw bytes from an input file into a higher-level
representation of the arguments and blocks, ready to be decoded into an
application-specific form.

The main entry point into HCL parsing is :go:pkg:hclparse, which provides
:go:type:hclparse.Parser:

.. code-block:: go

parser := hclparse.NewParser()
f, diags := parser.ParseHCLFile("server.conf")

Variable f is then a pointer to an :go:type:hcl.File, which is an
opaque abstract representation of the file, ready to be decoded.

Variable diags describes any errors or warnings that were encountered
during processing; HCL conventionally uses this in place of the usual error
return value in Go, to allow returning a mixture of multiple errors and
warnings together with enough information to present good error messages to the
user. We'll cover this in more detail in the next section,
:ref:go-diagnostics.

.. go:package:: hclparse

Package ``hclparse`` Copy Chapter

.. go:type:: Parser

.. go:function:: func NewParser() *Parser

Constructs a new parser object. Each parser contains a cache of files
  that have already been read, so repeated calls to load the same file
  will return the same object.

.. go:function:: func (*Parser) ParseHCL(src []byte, filename string) (*hcl.File, hcl.Diagnostics)

Parse the given source code as HCL native syntax, saving the result into
 the parser's file cache under the given filename.

.. go:function:: func (*Parser) ParseHCLFile(filename string) (*hcl.File, hcl.Diagnostics)

Parse the contents of the given file as HCL native syntax. This is a
 convenience wrapper around ParseHCL that first reads the file into memory.

.. go:function:: func (*Parser) ParseJSON(src []byte, filename string) (*hcl.File, hcl.Diagnostics)

Parse the given source code as JSON syntax, saving the result into
 the parser's file cache under the given filename.

.. go:function:: func (*Parser) ParseJSONFile(filename string) (*hcl.File, hcl.Diagnostics)

Parse the contents of the given file as JSON syntax. This is a
 convenience wrapper around ParseJSON that first reads the file into memory.

The above list just highlights the main functions in this package.
For full documentation, see
the hclparse godoc <https://godoc.org/github.com/hashicorp/hcl/v2/hclparse>_.

Guide Go Patterns (`guide/go_patterns.rst`) Copy Section

Design Patterns for Complex Systems

In previous sections we've seen an overview of some different ways an
application can decode a language its has defined in terms of the HCL grammar.
For many applications, those mechanisms are sufficient. However, there are
some more complex situations that can benefit from some additional techniques.
This section lists a few of these situations and ways to use the HCL API to
accommodate them.

.. _go-interdep-blocks:

Interdependent Blocks Copy Chapter

In some configuration languages, the variables available for use in one
configuration block depend on values defined in other blocks.

For example, in Terraform many of the top-level constructs are also implicitly
definitions of values that are available for use in expressions elsewhere:

.. code-block:: hcl

variable "network_numbers" {
type = list(number)
}

variable "base_network_addr" {
type = string
default = "10.0.0.0/8"
}

locals {
network_blocks = {
for x in var.number:
x => cidrsubnet(var.base_network_addr, 8, x)
}
}

resource "cloud_subnet" "example" {
for_each = local.network_blocks

cidr_block = each.value

}

output "subnet_ids" {
value = cloud_subnet.example[*].id
}

In this example, the variable "network_numbers" block makes
var.network_numbers available to expressions, the
resource "cloud_subnet" "example" block makes cloud_subnet.example
available, etc.

Terraform achieves this by decoding the top-level structure in isolation to
start. You can do this either using the low-level API or using :go:pkg:gohcl
with :go:type:hcl.Body fields tagged as "remain".

Once you have a separate body for each top-level block, you can inspect each
of the attribute expressions inside using the Variables method on
:go:type:hcl.Expression, or the Variables function from package
:go:pkg:hcldec if you will eventually use its higher-level API to decode as
Terraform does.

The detected variable references can then be used to construct a dependency
graph between the blocks, and then perform a
topological sort <https://en.wikipedia.org/wiki/Topological_sorting>_ to
determine the correct order to evaluate each block's contents so that values
will always be available before they are needed.

Since :go:pkg:cty values are immutable, it is not convenient to directly
change values in a :go:type:hcl.EvalContext during this gradual evaluation,
so instead construct a specialized data structure that has a separate value
per object and construct an evaluation context from that each time a new
value becomes available.

Using :go:pkg:hcldec to evaluate block bodies is particularly convenient in
this scenario because it produces :go:type:cty.Value results which can then
just be directly incorporated into the evaluation context.

Distributed Systems Copy Chapter

Distributed systems cause a number of extra challenges, and configuration
management is rarely the worst of these. However, there are some specific
considerations for using HCL-based configuration in distributed systems.

For the sake of this section, we are concerned with distributed systems where
at least two separate components both depend on the content of HCL-based
configuration files. Real-world examples include the following:

• HashiCorp Nomad loads configuration (job specifications) in its servers
  but also needs these results in its clients and in its various driver plugins.

• HashiCorp Terraform parses configuration in Terraform Core but can write
  a partially-evaluated execution plan to disk and continue evaluation in a
  separate process later. It must also pass configuration values into provider
  plugins.

Broadly speaking, there are two approaches to allowing configuration to be
accessed in multiple subsystems, which the following subsections will discuss
separately.

Ahead-of-time Evaluation
^^^^^^^^^^^^^^^^^^^^^^^^

Ahead-of-time evaluation is the simplest path, with the configuration files
being entirely evaluated on entry to the system, and then only the resulting
constant values being passed between subsystems.

This approach is relatively straightforward because the resulting
:go:type:cty.Value results can be losslessly serialized as either JSON or
msgpack as long as all system components agree on the expected value types.
Aside from passing these values around "on the wire", parsing and decoding of
configuration proceeds as normal.

Both Nomad and Terraform use this approach for interacting with plugins,
because the plugins themselves are written by various different teams that do
not coordinate closely, and so doing all expression evaluation in the core
subsystems ensures consistency between plugins and simplifies plugin development.

In both applications, the plugin is expected to describe (using an
application-specific protocol) the schema it expects for each element of
configuration it is responsible for, allowing the core subsystems to perform
decoding on the plugin's behalf and pass a value that is guaranteed to conform
to the schema.

Gradual Evaluation
^^^^^^^^^^^^^^^^^^

Although ahead-of-time evaluation is relatively straightforward, it has the
significant disadvantage that all data available for access via variables or
functions must be known by whichever subsystem performs that initial
evaluation.

For example, in Terraform, the "plan" subcommand is responsible for evaluating
the configuration and presenting to the user an execution plan for approval, but
certain values in that plan cannot be determined until the plan is already
being applied, since the specific values used depend on remote API decisions
such as the allocation of opaque id strings for objects.

In Terraform's case, both the creation of the plan and the eventual apply
of that plan both entail evaluating configuration, with the apply step
having a more complete set of input values and thus producing a more complete
result. However, this means that Terraform must somehow make the expressions
from the original input configuration available to the separate process that
applies the generated plan.

Good usability requires error and warning messages that are able to refer back
to specific sections of the input configuration as context for the reported
problem, and the best way to achieve this in a distributed system doing
gradual evaluation is to send the configuration source code between
subsystems. This is generally the most compact representation that retains
source location information, and will avoid any inconsistency caused by
introducing another intermediate serialization.

In Terraform's, for example, the serialized plan incorporates both the data
structure describing the partial evaluation results from the plan phase and
the original configuration files that produced those results, which can then
be re-evalauated during the apply step.

In a gradual evaluation scenario, the application should verify correctness of
the input configuration as completely as possible at each state. To help with
this, :go:pkg:cty has the concept of
unknown values <https://github.com/zclconf/go-cty/blob/master/docs/concepts.md#unknown-values-and-the-dynamic-pseudo-type>_,
which can stand in for values the application does not yet know while still
retaining correct type information. HCL expression evaluation reacts to unknown
values by performing type checking but then returning another unknown value,
causing the unknowns to propagate through expressions automatically.

.. code-block:: go

ctx := &hcl.EvalContext{
Variables: map[string]cty.Value{
"name": cty.UnknownVal(cty.String),
"age": cty.UnknownVal(cty.Number),
},
}
val, moreDiags := expr.Value(ctx)
diags = append(diags, moreDiags...)

Each time an expression is re-evaluated with additional information, fewer of
the input values will be unknown and thus more of the result will be known.
Eventually the application should evaluate the expressions with no unknown
values at all, which then guarantees that the result will also be wholly-known.

Static References, Calls, Lists, and Maps Copy Chapter

In most cases, we care more about the final result value of an expression than
how that value was obtained. A particular list argument, for example, might
be defined by the user via a tuple constructor, by a for expression, or by
assigning the value of a variable that has a suitable list type.

In some special cases, the structure of the expression is more important than
the result value, or an expression may not have a reasonable result value.
For example, in Terraform there are a few arguments that call for the user
to name another object by reference, rather than provide an object value:

.. code-block:: hcl

resource "cloud_network" "example" {
# ...
}

resource "cloud_subnet" "example" {
cidr_block = "10.1.2.0/24"

depends_on = [
   cloud_network.example,
 ]

}

The depends_on argument in the second resource block appears as an
expression that would construct a single-element tuple containing an object
representation of the first resource block. However, Terraform uses this
expression to construct its dependency graph, and so it needs to see
specifically that this expression refers to cloud_network.example, rather
than determine a result value for it.

HCL offers a number of "static analysis" functions to help with this sort of
situation. These all live in the :go:pkg:hcl package, and each one imposes
a particular requirement on the syntax tree of the expression it is given,
and returns a result derived from that if the expression conforms to that
requirement.

.. go:currentpackage:: hcl

.. go:function:: func ExprAsKeyword(expr Expression) string

This function attempts to interpret the given expression as a single keyword,
returning that keyword as a string if possible.

A "keyword" for the purposes of this function is an expression that can be
understood as a valid single identifier. For example, the simple variable
reference foo can be interpreted as a keyword, while foo.bar
cannot.

As a special case, the language-level keywords true, false, and
null are also considered to be valid keywords, allowing the calling
application to disregard their usual meaning.

If the given expression cannot be reduced to a single keyword, the result
is an empty string. Since an empty string is never a valid keyword, this
result unambiguously signals failure.

.. go:function:: func AbsTraversalForExpr(expr Expression) (Traversal, Diagnostics)

This is a generalization of ExprAsKeyword that will accept anything that
can be interpreted as a traversal, which is a variable name followed by
zero or more attribute access or index operators with constant operands.

For example, all of foo, foo.bar and foo[0] are valid
traversals, but foo[bar] is not, because the bar index is not
constant.

This is the function that Terraform uses to interpret the items within the
depends_on sequence in our example above.

As with ExprAsKeyword, this function has a special case that the
keywords true, false, and null will be accepted as if they were
variable names by this function, allowing null.foo to be interpreted
as a traversal even though it would be invalid if evaluated.

If error diagnostics are returned, the traversal result is invalid and
should not be used.

.. go:function:: func RelTraversalForExpr(expr Expression) (Traversal, Diagnostics)

This is very similar to AbsTraversalForExpr, but the result is a
relative traversal, which is one whose first name is considered to be
an attribute of some other (implied) object.

The processing rules are identical to AbsTraversalForExpr, with the
only exception being that the first element of the returned traversal is
marked as being an attribute, rather than as a root variable.

.. go:function:: func ExprList(expr Expression) ([]Expression, Diagnostics)

This function requires that the given expression be a tuple constructor,
and if so returns a slice of the element expressions in that constructor.
Applications can then perform further static analysis on these, or evaluate
them as normal.

If error diagnostics are returned, the result is invalid and should not be
used.

This is the fucntion that Terraform uses to interpret the expression
assigned to depends_on in our example above, then in turn using
AbsTraversalForExpr on each enclosed expression.

.. go:function:: func ExprMap(expr Expression) ([]KeyValuePair, Diagnostics)

This function requires that the given expression be an object constructor,
and if so returns a slice of the element key/value pairs in that constructor.
Applications can then perform further static analysis on these, or evaluate
them as normal.

If error diagnostics are returned, the result is invalid and should not be
used.

.. go:function:: func ExprCall(expr Expression) (*StaticCall, Diagnostics)

This function requires that the given expression be a function call, and
if so returns an object describing the name of the called function and
expression objects representing the call arguments.

If error diagnostics are returned, the result is invalid and should not be
used.

The Variables method on :go:type:hcl.Expression is also considered to be
a "static analysis" helper, but is built in as a fundamental feature because
analysis of referenced variables is often important for static validation and
for implementing interdependent blocks as we saw in the section above.

Guide Index (`guide/index.rst`) Copy Section

HCL Config Language Toolkit

HCL is a toolkit for creating structured configuration languages that are both
human- and machine-friendly, for use with command-line tools, servers, etc.

HCL has both a native syntax, intended to be pleasant to read and write for
humans, and a JSON-based variant that is easier for machines to generate and
parse. The native syntax is inspired by libucl_, nginx configuration_, and
others.

It includes an expression syntax that allows basic inline computation and, with
support from the calling application, use of variables and functions for more
dynamic configuration languages.

HCL provides a set of constructs that can be used by a calling application to
construct a configuration language. The application defines which argument
names and nested block types are expected, and HCL parses the configuration
file, verifies that it conforms to the expected structure, and returns
high-level objects that the application can use for further processing.

At present, HCL is primarily intended for use in applications written in Go_,
via its library API.

.. toctree::
:maxdepth: 1
:caption: Contents:

intro
go
language_design

.. _libucl: https://github.com/vstakhov/libucl
.. _nginx configuration: http://nginx.org/en/docs/beginners_guide.html#conf_structure
.. _Go: https://golang.org/

Guide Intro (`guide/intro.rst`) Copy Section

.. _intro:

Introduction to HCL

HCL-based configuration is built from two main constructs: arguments and
blocks. The following is an example of a configuration language for a
hypothetical application:

.. code-block:: hcl

io_mode = "async"

service "http" "web_proxy" {
listen_addr = "127.0.0.1:8080"

process "main" {
  command = ["/usr/local/bin/awesome-app", "server"]
}

process "mgmt" {
  command = ["/usr/local/bin/awesome-app", "mgmt"]
}

}

In the above example, io_mode is a top-level argument, while service
introduces a block. Within the body of a block, further arguments and nested
blocks are allowed. A block type may also expect a number of labels, which
are the quoted names following the service keyword in the above example.

The specific keywords io_mode, service, process, etc here are
application-defined. HCL provides the general block structure syntax, and
can validate and decode configuration based on the application's provided
schema.

HCL is a structured configuration language rather than a data structure
serialization language. This means that unlike languages such as JSON, YAML,
or TOML, HCL is always decoded using an application-defined schema.

However, HCL does have a JSON-based alternative syntax, which allows the same
structure above to be generated using a standard JSON serializer when users
wish to generate configuration programmatically rather than hand-write it:

.. code-block:: json

{
"io_mode": "async",
"service": {
"http": {
"web_proxy": {
"listen_addr": "127.0.0.1:8080",
"process": {
"main": {
"command": ["/usr/local/bin/awesome-app", "server"]
},
"mgmt": {
"command": ["/usr/local/bin/awesome-app", "mgmt"]
},
}
}
}
}
}

The calling application can choose which syntaxes to support. JSON syntax may
not be important or desirable for certain applications, but it is available for
applications that need it. The schema provided by the calling application
allows JSON input to be properly decoded even though JSON syntax is ambiguous
in various ways, such as whether a JSON object is representing a nested block
or an object expression.

The collection of arguments and blocks at a particular nesting level is called
a body. A file always has a root body containing the top-level elements,
and each block also has its own body representing the elements within it.

The term "attribute" can also be used to refer to what we've called an
"argument" so far. The term "attribute" is also used for the fields of an
object value in argument expressions, and so "argument" is used to refer
specifically to the type of attribute that appears directly within a body.

The above examples show the general "texture" of HCL-based configuration. The
full details of the syntax are covered in the language specifications.

.. todo:: Once the language specification documents have settled into a
final location, link them from above.

Argument Expressions Copy Chapter

The value of an argument can be a literal value shown above, or it may be an
expression to allow arithmetic, deriving one value from another, etc.

.. code-block:: hcl

listen_addr = env.LISTEN_ADDR

Built-in arithmetic and comparison operators are automatically available in all
HCL-based configuration languages. A calling application may optionally
provide variables that users can reference, like env in the above example,
and custom functions to transform values in application-specific ways.

Full details of the expression syntax are in the HCL native syntax
specification. Since JSON does not have an expression syntax, JSON-based
configuration files use the native syntax expression language embedded inside
JSON strings.

.. todo:: Once the language specification documents have settled into a
final location, link to the native syntax specification from above.

Guide Language Design (`guide/language_design.rst`) Copy Section

Configuration Language Design

In this section we will cover some conventions for HCL-based configuration
languages that can help make them feel consistent with other HCL-based
languages, and make the best use of HCL's building blocks.

HCL's native and JSON syntaxes both define a mapping from input bytes to a
higher-level information model. In designing a configuration language based on
HCL, your building blocks are the components in that information model:
blocks, arguments, and expressions.

Each calling application of HCL, then, effectively defines its own language.
Just as Atom and RSS are higher-level languages built on XML, HashiCorp
Terraform has a higher-level language built on HCL, while HashiCorp Nomad has
its own distinct language that is also built on HCL.

From an end-user perspective, these are distinct languages but have a common
underlying texture. Users of both are therefore likely to bring some
expectations from one to the other, and so this section is an attempt to
codify some of these shared expectations to reduce user surprise.

These are subjective guidelines however, and so applications may choose to
ignore them entirely or ignore them in certain specialized cases. An
application providing a configuration language for a pre-existing system, for
example, may choose to eschew the identifier naming conventions in this section
in order to exactly match the existing names in that underlying system.

Language Keywords and Identifiers Copy Chapter

Much of the work in defining an HCL-based language is in selecting good names
for arguments, block types, variables, and functions.

The standard for naming in HCL is to use all-lowercase identifiers with
underscores separating words, like service or io_mode. HCL identifiers
do allow uppercase letters and dashes, but this primarily for natural
interfacing with external systems that may have other identifier conventions,
and so these should generally be avoided for the identifiers native to your
own language.

The distinction between "keywords" and other identifiers is really just a
convention. In your own language documentation, you may use the word "keyword"
to refer to names that are presented as an intrinsic part of your language,
such as important top-level block type names.

Block type names are usually singular, since each block defines a single
object. Use a plural block name only if the block is serving only as a
namespacing container for a number of other objects. A block with a plural
type name will generally contain only nested blocks, and no arguments of its
own.

Argument names are also singular unless they expect a collection value, in
which case they should be plural. For example, name = "foo" but
subnet_ids = ["abc", "123"].

Function names will generally not use underscores and will instead just run
words together, as is common in the C standard library. This is a result of
the fact that several of the standard library functions offered in cty
(covered in a later section) have names that follow C library function names
like substr. This is not a strong rule, and applications that use longer
names may choose to use underscores for them to improve readability.

Blocks vs. Object Values Copy Chapter

HCL blocks and argument values of object type have quite a similar appearance
in the native syntax, and are identical in JSON syntax:

.. code-block:: hcl

block {
foo = bar
}

argument with object constructor expression

argument = {
foo = bar
}

In spite of this superficial similarity, there are some important differences
between these two forms.

The most significant difference is that a child block can contain nested blocks
of its own, while an object constructor expression can define only attributes
of the object it is creating.

The user-facing model for blocks is that they generally form the more "rigid"
structure of the language itself, while argument values can be more free-form.
An application will generally define in its schema and documentation all of
the arguments that are valid for a particular block type, while arguments
accepting object constructors are more appropriate for situations where the
arguments themselves are freely selected by the user, such as when the
expression will be converted by the application to a map type.

As a less contrived example, consider the resource block type in Terraform
and its use with a particular resource type aws_instance:

.. code-block:: hcl

resource "aws_instance" "example" {
ami = "ami-abc123"
instance_type = "t2.micro"

tags = {
   Name = "example instance"
 }

 ebs_block_device {
   device_name = "hda1"
   volume_size = 8
   volume_type = "standard"
 }

}

The top-level block type resource is fundamental to Terraform itself and
so an obvious candidate for block syntax: it maps directly onto an object in
Terraform's own domain model.

Within this block we see a mixture of arguments and nested blocks, all defined
as part of the schema of the aws_instance resource type. The tags
map here is specified as an argument because its keys are free-form, chosen
by the user and mapped directly onto a map in the underlying system.
ebs_block_device is specified as a nested block, because it is a separate
domain object within the remote system and has a rigid schema of its own.

As a special case, block syntax may sometimes be used with free-form keys if
those keys each serve as a separate declaration of some first-class object
in the language. For example, Terraform has a top-level block type locals
which behaves in this way:

.. code-block:: hcl

locals {
instance_type = "t2.micro"
instance_id = aws_instance.example.id
}

Although the argument names in this block are arbitrarily selected by the
user, each one defines a distinct top-level object. In other words, this
approach is used to create a more ergonomic syntax for defining these simple
single-expression objects, as a pragmatic alternative to more verbose and
redundant declarations using blocks:

.. code-block:: hcl

local "instance_type" {
value = "t2.micro"
}
local "instance_id" {
value = aws_instance.example.id
}

The distinction between domain objects, language constructs and user data will
always be subjective, so the final decision is up to you as the language
designer.

Standard Functions Copy Chapter

HCL itself does not define a common set of functions available in all HCL-based
languages; the built-in language operators give a baseline of functionality
that is always available, but applications are free to define functions as they
see fit.

With that said, there's a number of generally-useful functions that don't
belong to the domain of any one application: string manipulation, sequence
manipulation, date formatting, JSON serialization and parsing, etc.

Given the general need such functions serve, it's helpful if a similar set of
functions is available with compatible behavior across multiple HCL-based
languages, assuming the language is for an application where function calls
make sense at all.

The Go implementation of HCL is built on an underlying type and function system
:go:pkg:cty, whose usage was introduced in :ref:go-expression-funcs. That
library also has a package of "standard library" functions which we encourage
applications to offer with consistent names and compatible behavior, either by
using the standard implementations directly or offering compatible
implementations under the same name.

The "standard" functions that new configuration formats should consider
offering are:

• abs(number) - returns the absolute (positive) value of the given number.
• coalesce(vals...) - returns the value of the first argument that isn't null. Useful only in formats where null values may appear.
• compact(vals...) - returns a new tuple with the non-null values given as arguments, preserving order.
• concat(seqs...) - builds a tuple value by concatenating together all of the given sequence (list or tuple) arguments.
• format(fmt, args...) - performs simple string formatting similar to the C library function printf.
• hasindex(coll, idx) - returns true if the given collection has the given index. coll may be of list, tuple, map, or object type.
• int(number) - returns the integer component of the given number, rounding towards zero.
• jsondecode(str) - interprets the given string as JSON format and return the corresponding decoded value.
• jsonencode(val) - encodes the given value as a JSON string.
• length(coll) - returns the length of the given collection.
• lower(str) - converts the letters in the given string to lowercase, using Unicode case folding rules.
• max(numbers...) - returns the highest of the given number values.
• min(numbers...) - returns the lowest of the given number values.
• sethas(set, val) - returns true only if the given set has the given value as an element.
• setintersection(sets...) - returns the intersection of the given sets
• setsubtract(set1, set2) - returns a set with the elements from set1 that are not also in set2.
• setsymdiff(sets...) - returns the symmetric difference of the given sets.
• setunion(sets...) - returns the union of the given sets.
• strlen(str) - returns the length of the given string in Unicode grapheme clusters.
• substr(str, offset, length) - returns a substring from the given string by splitting it between Unicode grapheme clusters.
• timeadd(time, duration) - takes a timestamp in RFC3339 format and a possibly-negative duration given as a string like "1h" (for "one hour") and returns a new RFC3339 timestamp after adding the duration to the given timestamp.
• upper(str) - converts the letters in the given string to uppercase, using Unicode case folding rules.

Not all of these functions will make sense in all applications. For example, an
application that doesn't use set types at all would have no reason to provide
the set-manipulation functions here.

Some languages will not provide functions at all, since they are primarily for
assigning values to arguments and thus do not need nor want any custom
computations of those values.

Block Results as Expression Variables Copy Chapter

In some applications, top-level blocks serve also as declarations of variables
(or of attributes of object variables) available during expression evaluation,
as discussed in :ref:go-interdep-blocks.

In this case, it's most intuitive for the variables map in the evaluation
context to contain an value named after each valid top-level block
type and for these values to be object-typed or map-typed and reflect the
structure implied by block type labels.

For example, an application may have a top-level service block type
used like this:

.. code-block:: hcl

service "http" "web_proxy" {
listen_addr = "127.0.0.1:8080"

process "main" {
  command = ["/usr/local/bin/awesome-app", "server"]
}

process "mgmt" {
  command = ["/usr/local/bin/awesome-app", "mgmt"]
}

}

If the result of decoding this block were available for use in expressions
elsewhere in configuration, the above convention would call for it to be
available to expressions as an object at service.http.web_proxy.

If it the contents of the block itself that are offered to evaluation -- or
a superset object derived from the block contents -- then the block arguments
can map directly to object attributes, but it is up to the application to
decide which value type is most appropriate for each block type, since this
depends on how multiple blocks of the same type relate to one another, or if
multiple blocks of that type are even allowed.

In the above example, an application would probably expose the listen_addr
argument value as service.http.web_proxy.listen_addr, and may choose to
expose the process blocks as a map of objects using the labels as keys,
which would allow an expression like
service.http.web_proxy.service["main"].command.

If multiple blocks of a given type do not have a significant order relative to
one another, as seems to be the case with these process blocks,
representation as a map is often the most intuitive. If the ordering of the
blocks is significant then a list may be more appropriate, allowing the use
of HCL's "splat operators" for convenient access to child arguments. However,
there is no one-size-fits-all solution here and language designers must
instead consider the likely usage patterns of each value and select the
value representation that best accommodates those patterns.

Some applications may choose to offer variables with slightly different names
than the top-level blocks in order to allow for more concise references, such
as abbreviating service to svc in the above examples. This should be
done with care since it may make the relationship between the two less obvious,
but this may be a good tradeoff for names that are accessed frequently that
might otherwise hurt the readability of expressions they are embedded in.
Familiarity permits brevity.

Many applications will not make blocks results available for use in other
expressions at all, in which case they are free to select whichever variable
names make sense for what is being exposed. For example, a format may make
environment variable values available for use in expressions, and may do so
either as top-level variables (if no other variables are needed) or as an
object named env, which can be used as in env.HOME.

Text Editor and IDE Integrations Copy Chapter

Since HCL defines only low-level syntax, a text editor or IDE integration for
HCL itself can only really provide basic syntax highlighting.

For non-trivial HCL-based languages, a more specialized editor integration may
be warranted. For example, users writing configuration for HashiCorp Terraform
must recall the argument names for numerous different provider plugins, and so
auto-completion and documentation hovertips can be a great help, and
configurations are commonly spread over multiple files making "Go to Definition"
functionality useful. None of this functionality can be implemented generically
for all HCL-based languages since it relies on knowledge of the structure of
Terraform's own language.

Writing such text editor integrations is out of the scope of this guide. The
Go implementation of HCL does have some building blocks to help with this, but
it will always be an application-specific effort.

However, in order to enable such integrations, it is best to establish a
conventional file extension other than .hcl for each non-trivial HCL-based
language, thus allowing text editors to recognize it and enable the suitable
integration. For example, Terraform requires .tf and .tf.json filenames
for its main configuration, and the hcldec utility in the HCL repository
accepts spec files that should conventionally be named with an .hcldec
extension.

For simple languages that are unlikely to benefit from specific editor
integrations, using the .hcl extension is fine and may cause an editor to
enable basic syntax highlighting, absent any other deeper features. An editor
extension for a specific HCL-based language should not match generically the
.hcl extension, since this can cause confusing results for users
attempting to write configuration files targeting other applications.

--- METRICS ---

• Files Extracted: 24
• Estimated Token Budget: ~28306 tokens
• Canonical Reference: https://codewiki.google/github.com/hashicorp/hcl