Acme Divergences
Boulder divergences from ACME
While Boulder attempts to implement the ACME specification ([RFC 8555]) as strictly as possible there are places at which we will diverge from the letter of the specification for various reasons. This document describes the difference between [RFC 8555] and Boulder's implementation of ACME, informally called ACMEv2 and available at https://acme-v02.api.letsencrypt.org/directory. A listing of RFC conformant design decisions that may differ from other ACME servers is listed in implementation_details.
Presently, Boulder diverges from the [RFC 8555] ACME spec in the following ways:
Section 6.3
Boulder supports POST-as-GET but does not mandate it for requests
that simply fetch a resource (certificate, order, authorization, or challenge).
Section 7.1.2
Boulder does not supply the orders field on account objects. We intend to
support this non-essential feature in the future. Please follow Boulder Issue
#3335.
Section 7.4
Boulder does not accept the optional notBefore and notAfter fields of anewOrder request payload.
Section 7.4.1
Pre-authorization is an optional feature and we have no plans to implement it.
V2 clients should use order based issuance without pre-authorization.
Section 7.4.2
Boulder does not process Accept headers for Content-Type negotiation when retrieving certificates.
Section 8.2
Boulder does not implement the ability to retry challenges or the Retry-After header.
[RFC 8555]: https://tools.ietf.org/html/rfc8555
---
Acme Implementation Details
Boulder implementation details
The ACME specification ([RFC 8555]) clearly dictates what Clients and Servers
must do to properly implement the protocol.
The specification is intentionally silent, or vague, on certain points to give
developers freedom in making certain decisions or to follow guidance from other
RFCs. Due to this, two ACME Servers might fully conform to the RFC but behave
slightly differently. ACME Clients should not "over-fit" on Boulder or the
Let's Encrypt production service, and aim to be compatible with a wide range of
ACME Servers, including the Pebble
test server.
The following items are a partial listing of RFC-conformant design decisions
Boulder and/or LetsEncrypt have made. This listing is not complete, and is
based on known details which have caused issues for developers in the past. This
listing may not reflect the current status of Boulder or the configuration of
LetsEncrypt's production instance and is provided only as a reference for client
developers.
Please note: these design implementation decisions are fully conformant with the
RFC specification and are not
divergences.
Object Reuse
The ACME specification does not prohibit certain objects to be re-used.
Authorization
Boulder may recycle previously "valid" or "pending" Authorizations for a givenAccount when creating a new Order.
Order
Boulder may return a previously created Order when a given Account submits
a new Order that is identical to a previously submitted Order that is in
the "pending" or "ready" state.
Alternate Chains
The production Boulder instance for LetsEncrypt in enabled with support for
Alternate chains.
Certificate Request Domains
The RFC states the following:
The CSR MUST indicate the exact same
set of requested identifiers as the initial newOrder request.
Identifiers of type "dns" MUST appear either in the commonName
portion of the requested subject name or in an extensionRequest
attribute [RFC2985] requesting a subjectAltName extension, or both.
Boulder requires all domains to be specified in the subjectAltName
extension, and will reject a CSR if a domain specified in the commonName is
not present in the subjectAltName. Additionally, usage of the commonName
was previously deprecated by the CA/B Forum and in earlier RFCs.
For more information on this see Pebble Issue #304
and Pebble Issue #233.
RSA Key Size
The ACME specification is silent as to minimum key size.
The CA/Browser Forum sets the key size requirements
which LetsEncrypt adheres to.
Effective 2020-09-17, LetsEncrypt further requires all RSA keys for end-entity
(leaf) certificates have a modulus of length 2048, 3072, or 4096. Other CAs may
or may not have the same restricted set of supported RSA key sizes.
For more information
read the Official Announcement.
---
CODE OF CONDUCT
Code of Conduct
The code of conduct for everyone participating in this community in any capacity
is available for reference
on the community forum.
---
Config Validation
Configuration Validation
We use a fork of https://github.com/go-playground/validator which can be found
at https://github.com/letsencrypt/validator.
Usage
By default Boulder validates config files for all components with a registered
validator. Validating a config file for a given component is as simple as
running the component directly:
$ ./bin/boulder-observer -config test/config-next/observer.yml
Error validating config file "test/config-next/observer.yml": Key: 'ObsConf.MonConfs[1].Kind' Error:Field validation for 'Kind' failed on the 'oneof' tagor by running the boulder binary and passing the component name as a
subcommand:
$ ./bin/boulder boulder-observer -config test/config-next/observer.yml
Error validating config file "test/config-next/observer.yml": Key: 'ObsConf.MonConfs[1].Kind' Error:Field validation for 'Kind' failed on the 'oneof' tagStruct Tag Tips
You can find the full list of struct tags supported by the validator [here]
(https://pkg.go.dev/github.com/go-playground/validator/v10#section-documentation).
The following are some tips for struct tags that are commonly used in our
configuration files.
required
The required tag means that the field is not allowed to take its zero value, or
equivalently, is not allowed to be omitted. Note that this does not validate
that slices or maps have contents, it simply guarantees that they are not nil.
For fields of those types, you should use min=1 or similar to ensure they are
not empty.
There are also "conditional" required tags, such as required_with,required_with_all, required_without, required_without_all, andrequired_unless. These behave exactly like the basic required tag, but only if
their conditional (usually the presence or absence of one or more other named
fields) is met.
omitempty
The omitempty tag allows a field to be empty, or equivalently, to take its zero
value. If the field is omitted, none of the other validation tags on the field
will be enforced. This can be useful for tags like validate="omitempty,url", for
a field which is optional, but must be a URL if it is present.
The omitempty tag can be "overruled" by the various conditional required tags.
For example, a field with tag validate="omitempty,url,required_with=Foo" is
allowed to be empty when field Foo is not present, but if field Foo is present,
then this field must be present and must be a URL.
-
Normally, config validation descends into all struct-type fields, recursively
validating their fields all the way down. Sometimes this can pose a problem,
when a nested struct declares one of its fields as required, but a parent struct
wants to treat the whole nested struct as optional. The "-" tag tells the
validation not to recurse, marking the tagged field as optional, and therefore
making all of its sub-fields optional as well. We use this tag for many config
duration and password file struct valued fields which are optional in some
configs but required in others.
structonly
The structonly tag allows a struct valued field to be empty, or equivalently, to
take its zero value, if it's not "overruled" by various conditional tags. If the
field is omitted the recursive validation of the structs fields will be skipped.
This can be useful for tags like validate:"required_without=Foo,structonly"
for a struct valued field which is only required, and thus should only be
validated, if field Foo is not present.
min=1, gte=1
These validate that the value of integer valued field is greater than zero and
that the length of the slice or map is greater than zero.
For instance, the following would be valid config for a slice valued field
tagged with required.
{
"foo": [],
}But, only the following would be valid config for a slice valued field tagged
with min=1.
{
"foo": ["bar"],
}len
Same as eq (equal to) but can also be used to validate the length of the
strings.
hostname_port
The
docs
for this tag are scant with detail, but it validates that the value is a valid
RFC 1123 hostname and port. It is used to validate many of theListenAddress and DebugAddr fields of our components.
#### Future Work
This tag is compatible with IPv4 addresses, but not IPv6 addresses. We should
consider fixing this in our fork of the validator.
dive
This tag is used to validate the values of a slice or map. For instance, the
following would be valid config for a slice valued field ([]string) tagged
with min=1,dive,oneof=bar baz.
{
"foo": ["bar", "baz"],
}Note that the dive tag introduces an order-dependence in writing tags: tags
that come before dive apply to the current field, while tags that come afterdive apply to the current field's child values. In the example above: min=1
applies to the length of the slice ([]string), while oneof=bar baz applies
to the value of each string in the slice.
We can also use dive to validate the values of a map. For instance, the
following would be valid config for a map valued field (map[string]string)
tagged with min=1,dive,oneof=one two.
{
"foo": {
"bar": "one",
"baz": "two"
},
}dive can also be invoked multiple times to validate the values of nested
slices or maps. For instance, the following would be valid config for a slice of
slice valued field ([][]string) tagged with min=1,dive,min=2,dive,oneof=bar.
baz
{
"foo": [
["bar", "baz"],
["baz", "bar"],
],
}- min=1 will be applied to the outer slice ([]).
- min=2 will be applied to inner slice ([]string).
- oneof=bar baz will be applied to each string in the inner slice.
keys and endkeys
These tags are used to validate the keys of a map. For instance, the following
would be valid config for a map valued field (map[string]string) tagged withmin=1,dive,keys,eq=1|eq=2,endkeys,required.
{
"foo": {
"1": "bar",
"2": "baz",
},
}- min=1 will be applied to the map itself
- eq=1|eq=2 will be applied to the map keys
- required will be applied to map values
---
CONTRIBUTING
Thanks for helping us build Boulder! This page contains requirements and
guidelines for Boulder contributions.
Patch Requirements
* All new functionality and fixed bugs must be accompanied by tests.
* All patches must meet the deployability requirements listed below.
* We prefer pull requests from external forks be created with the "Allow edits
from
maintainers"
checkbox selected.
Review Requirements
* All pull requests must receive at least one approval by a CODEOWNER other than the author. This is enforced by GitHub itself.
* All pull requests should receive at least two approvals by Trusted Contributors.
This requirement may be waived when:
* the change only modifies documentation;
* the change only modifies tests;
* in exceptional circumstances, such as when no second reviewer is available at all.
This requirement should not be waived when:
* the change is not written by a Trusted Contributor, to ensure that at least two TCs have eyes on it.
* New commits pushed to a branch invalidate previous reviews. In other words, a
reviewer must give positive reviews of a branch after its most recent pushed
commit.
* If a branch contains commits from multiple authors, it needs a reviewer who
is not an author of commits on that branch.
* Review changes to or addition of tests just as rigorously as you review code
changes. Consider: Do tests actually test what they mean to test? Is this the
best way to test the functionality in question? Do the tests cover all the
functionality in the patch, including error cases?
* Are there new RPCs or config fields? Make sure the patch meets the
Deployability rules below.
* All review conversations must be resolved. This helps to ensure that review
feedback has been fully addressed. This is enforced by GitHub itself.
Merge Requirements
We have a bot that will comment on some PRs indicating there are:
1. configuration changes
2. SQL schema changes
3. feature flag changes
These may require either a CP/CPS review or filing of a ticket to make matching changes
in production. It is the responsibility of the person merging the PR to make sure
the required action has been performed before merging. Usually this will be confirmed
in a comment or in the PR description.
When creating a pull request that has external dependencies, like another pull
request being merged or deployed to production, or approval from SRE, the author
should leave a review comment on the pull request (i.e. create a "conversation"
by commenting on a specific file or line of a file) noting what the pull request
is blocked on. The comment should include instructions to only resolve that
conversation once the external conditions are met. Since our repository is
configured to require all conversations be resolved before merging, this acts as
a protection against early merges.
Patch Guidelines
* Please include helpful comments. No need to gratuitously comment clear code,
but make sure it's clear why things are being done. Include information in
your pull request about what you're trying to accomplish with your patch.
* Avoid named return values. See
#3017 for an example of a
subtle problem they can cause.
* Do not include XXXs or naked TODOs. Use
the formats:
// TODO(<email-address>): Hoverboard + Time-machine unsupported until upstream patch.
// TODO(#<num>): Pending hoverboard/time-machine interface.
// TODO(@githubusername): Enable hoverboard kickflips once interface is stable.Squash merging
Once a pull request is approved and the tests are passing, the author or any
other committer can merge it. We always use squash
merges via GitHub's web
interface. That means that during the course of your review you should
generally not squash or amend commits, or force push. Even if the changes in
each commit are small, keeping them separate makes it easier for us to review
incremental changes to a pull request. Rest assured that those tiny changes
will get squashed into a nice meaningful-size commit when we merge.
If the CI tests are failing on your branch, you should look at the logs
to figure out why. Sometimes (though rarely) they fail spuriously, in which
case you can post a comment requesting that a project owner kick the build.
Error handling
All errors must be addressed in some way: That may be simply by returning an
error up the stack, or by handling it in some intelligent way where it is
generated, or by explicitly ignoring it and assigning to _. We use theerrcheck tool in our integration tests to make sure all errors are
addressed. Note that ignoring errors, even in tests, should be rare, since
they may generate hard-to-debug problems.
When handling errors, always do the operation which creates the error (usually
a function call) and the error checking on separate lines:
err := someOperation(args)
if err != nil {
return nil, fmt.Errorf("some operation failed: %w", err)
}We avoid the
if err := someOperation(args); err != nil {...} style as we findit to be less readable and it can give rise to surprising scoping behavior.
We define two special types of error. BoulderError, defined in
errors/errors.go, is used specifically when an typed error needs to be passed
across an RPC boundary. For instance, if the SA returns "not found", callers
need to be able to distinguish that from a network error. Not every error that
may pass across an RPC boundary needs to be a BoulderError, only those errors
that need to be handled by type elsewhere. Handling by type may be as simple as
turning a BoulderError into a specific type of ProblemDetail.
The other special type of error is ProblemDetails. We try to treat these as a
presentation-layer detail, and use them only in parts of the system that are
responsible for rendering errors to end-users, i.e. WFE2. Note
one exception: The VA RPC layer defines its own ProblemDetails type, which is
returned to the RA and stored as part of a challenge (to eventually be rendered
to the user).
Within WFE2, ProblemDetails are sent to the client by callingsendError(), which also logs the error. For internal errors like timeout,
or any error type that we haven't specifically turned into a ProblemDetail, we
return a ServerInternal error. This avoids unnecessarily exposing internals.
It's possible to add additional errors to a logEvent using .AddError(), but
this should only be done when there is is internal-only information to log
that isn't redundant with the ProblemDetails sent to the user. Note that the
final argument to sendError(), ierr, will automatically get added to the
logEvent for ServerInternal errors, so when sending a ServerInternal error it's
not necessary to separately call .AddError.
Deployability
We want to ensure that a new Boulder revision can be deployed to the
currently running Boulder production instance without requiring config
changes first. We also want to ensure that during a deploy, services can be
restarted in any order. That means two things:
Good zero values for config fields
Any newly added config field must have a usable zero
value. That is to say, if a config field
is absent, Boulder shouldn't crash or misbehave. If that config file names a
file to be read, Boulder should be able to proceed without that file being
read.
Note that there are some config fields that we want to be a hard requirement.
To handle such a field, first add it as optional, then file an issue to make
it required after the next deploy is complete.
In general, we would like our deploy process to be: deploy new code + old
config; then immediately after deploy the same code + new config. This makes
deploys cheaper so we can do them more often, and allows us to more readily
separate deploy-triggered problems from config-triggered problems.
Flag-gating features
When adding significant new features or replacing existing RPCs theboulder/features package should be used to gate its usage. To add a flag, a
new field of the features.Config struct should be added. All flags default
to false.
In order to test if the flag is enabled elsewhere in the codebase you can usefeatures.Get().ExampleFeatureName which gets the bool value from a global
config.
Each service should include a map[string]bool named Features in its
configuration object at the top level and call features.Set with that map
immediately after parsing the configuration. For example to enableUseNewMetrics and disable AccountRevocation you would add this object:
{
...
"features": {
"UseNewMetrics": true,
"AccountRevocation": false,
}
}Feature flags are meant to be used temporarily and should not be used for
permanent boolean configuration options.
Deprecating a feature flag
Once a feature has been enabled in both staging and production, someone on the
team should deprecate it:
- Remove any instances of features.Get().ExampleFeatureName, adjusting code
as needed.
- Move the field to the top of the features.Config struct, under a comment
saying it's deprecated.
- Remove all references to the feature flag from test/config-next.
- Add the feature flag to test/config. This serves to check that we still
tolerate parsing the flag at startup, even though it is ineffective.
- File a ticket to remove the feature flag in staging and production.
- Once the feature flag is removed in staging and production, delete it from
test/config and features.Config.
Gating RPCs
When you add a new RPC to a Boulder service (e.g. SA.GetFoo()), all
components that call that RPC should gate those calls using a feature flag.
Since the feature's zero value is false, a deploy with the existing config
will not call SA.GetFoo(). Then, once the deploy is complete and we know
that all SA instances support the GetFoo() RPC, we do a followup config
deploy that sets the default value to true, and finally remove the flag
entirely once we are confident the functionality it gates behaves correctly.
Gating migrations
We use database migrations
to modify the existing schema. These migrations will be run on live data
while Boulder is still running, so we need Boulder code at any given commit
to be capable of running without depending on any changes in schemas that
have not yet been applied.
For instance, if we're adding a new column to an existing table, Boulder should
run correctly in three states:
1. Migration not yet applied.
2. Migration applied, flag not yet flipped.
3. Migration applied, flag flipped.
Specifically, that means that all of our SELECT statements should enumerate
columns to select, and not use *. Also, generally speaking, we will need a
separate model struct for serializing and deserializing data before and
after the migration. This is because the ORM package we use,borp, expects every field in a struct to
map to a column in the table. If we add a new field to a model struct and
Boulder attempts to write that struct to a table that doesn't yet have the
corresponding column (case 1), borp will fail with Insert failed table posts. There are examples of such models in sa/model.go,
has no column named Foo
along with code to turn a model into a struct used internally.
An example of a flag-gated migration, adding a new IsWizard field to Person
controlled by a AllowWizards feature flag:
features/features.go:
const (
unused FeatureFlag = iota // unused is used for testing
AllowWizards // Added!
)
...
var features = map[FeatureFlag]bool{
unused: false,
AllowWizards: false, // Added!
}
sa/sa.go:
struct Person {
HatSize int
IsWizard bool // Added!
}
struct personModelv1 {
HatSize int
}
// Added!
struct personModelv2 {
personModelv1
IsWizard bool
}
func (ssa *SQLStorageAuthority) GetPerson() (Person, error) {
if features.Enabled(features.AllowWizards) { // Added!
var model personModelv2
ssa.dbMap.SelectOne(&model, "SELECT hatSize, isWizard FROM people")
return Person{
HatSize: model.HatSize,
IsWizard: model.IsWizard,
}
} else {
var model personModelv1
ssa.dbMap.SelectOne(&model, "SELECT hatSize FROM people")
return Person{
HatSize: model.HatSize,
}
}
}
func (ssa *SQLStorageAuthority) AddPerson(p Person) (error) {
if features.Enabled(features.AllowWizards) { // Added!
return ssa.dbMap.Insert(context.Background(), personModelv2{
personModelv1: {
HatSize: p.HatSize,
},
IsWizard: p.IsWizard,
})
} else {
return ssa.dbMap.Insert(context.Background(), personModelv1{
HatSize: p.HatSize,
// p.IsWizard ignored
})
}
}
You will also need to update the initTables function from sa/database.go to
tell borp which table to use for your versioned model structs. Make sure to
consult the flag you defined so that only one of the table maps is added at
any given time, otherwise borp will error. Depending on your table you may also
need to add SetKeys and SetVersionCol entries for your versioned models.
Example:
func initTables(dbMap *borp.DbMap) {
// < unrelated lines snipped for brevity > if features.Enabled(features.AllowWizards) {
dbMap.AddTableWithName(personModelv2, "person")
} else {
dbMap.AddTableWithName(personModelv1, "person")
}
}
New migrations should be added at ./sa/db-next:
$ cd sa/db
$ sql-migrate new -env="boulder_sa" AddWizards
Created migration boulder_sa/20220906165519-AddWizards.sqlFinally, edit the resulting file
(sa/db-next/boulder_sa/20220906165519-AddWizards.sql) to define your migration:
-- +migrate Up
ALTER TABLE people ADD isWizard BOOLEAN SET DEFAULT false;-- +migrate Down
ALTER TABLE people DROP isWizard BOOLEAN SET DEFAULT false;
Expressing "optional" Timestamps
Timestamps in protocol buffers must always be expressed as
timestamppb.Timestamp.
Timestamps must never contain their zero value, in the sense of
timestamp.AsTime().IsZero(). When a timestamp field is optional, absence mustbe expressed through the absence of the field, rather than present with a zero
value. The
core.IsAnyNilOrZero function can check these cases.Senders must check that timestamps are non-zero before sending them. Receivers
must check that timestamps are non-zero before accepting them.
Rounding time in DB
All times that we send to the database are truncated to one second's worth of
precision. This reduces the size of indexes that include timestamps, and makes
querying them more efficient. The Storage Authority (SA) is responsible for this
truncation, and performs it for SELECT queries as well as INSERT and UPDATE.
Release Process
The current Boulder release process is described in
release.md. New
releases are tagged weekly, and artifacts are automatically produced for each
release by GitHub Actions.
Dependencies
We use go modules and vendor our dependencies.
To add a dependency, add the import statement to your .go file, then rungo build on it. This will automatically add the dependency to go.mod. Next,
run go mod vendor && git add vendor/ to save a copy in the vendor folder.
When vendorizing dependencies, it's important to make sure tests pass on the
version you are vendorizing. Currently we enforce this by requiring that pull
requests containing a dependency update to any version other than a tagged
release include a comment indicating that you ran the tests and that they
succeeded, preferably with the command line you run them with. Note that you
may have to get a separate checkout of the dependency (using go get outside
of the boulder repository) in order to run its tests, as some vendored
modules do not bring their tests with them.
Updating Dependencies
To upgrade a dependency, see the Go
docs.
Typically you want go get <dependency> rather than go get -u, which can introduce a lot of unexpected updates. After running
<dependency>go get, make sure to run go mod vendor && git add vendor/ to update the
vendor directory. If you forget, CI tests will catch this.
If you are updating a dependency to a version which is not a tagged release,
see the note above about how to run all of a dependency's tests and note that
you have done so in the PR.
Note that updating dependencies can introduce new, transitive dependencies. In
general we try to keep our dependencies as narrow as possible in order to
minimize the number of people and organizations whose code we need to trust.
As a rule of thumb: If an update introduces new packages or modules that are
inside a repository where we already depend on other packages or modules, it's
not a big deal. If it introduces a new dependency in a different repository,
please try to figure out where that dependency came from and why (for instance:
"package X, which we depend on, started supporting XML config files, so now we
depend on an XML parser") and include that in the PR description. When there are
a large number of new dependencies introduced, and we don't need the
functionality they provide, we should consider asking the relevant upstream
repository for a refactoring to reduce the number of transitive dependencies.
Go Version
The Boulder development
environment
does not use the Go version installed on the host machine, and instead uses a
Go environment baked into a "boulder-tools" Docker image. We build a separate
boulder-tools container for each supported Go version. Please see the
Boulder-tools
README
for more information on upgrading Go versions.
ACME Protocol Divergences
While Boulder attempts to implement the ACME specification as strictly as
possible there are places at which we will diverge from the letter of the
specification for various reasons. We detail these divergences (for both the
V1 and V2 API) in the ACME divergences
doc.
ACME Protocol Implementation Details
The ACME specification allows developers to make certain decisions as to how
various elements in the RFC are implemented. Some of these fully conformant
decisions are listed in ACME implementation details
doc.
Code of Conduct
The code of conduct for everyone participating in this community in any capacity
is available for reference
on the community forum.
Problems or questions?
The best place to ask dev related questions is on the Community
Forums.
---
CRLS
CRLs
For each issuer certificate, Boulder generates several sharded CRLs.
The responsibility is shared across these components:
- crl-updater
- sa
- ca
- crl-storer
The crl-updater starts the process: for each shard of each issuer,
it requests revoked certificate information from the SA. It sends
that information to the CA for signing, and receives back a signed
CRL. It sends the signed CRL to the crl-storer for upload to an
S3-compatible data store.
The crl-storer uploads the CRLs to the filename <issuerID>/<shard>.crl,
where issuerID is an integer that uniquely identifies the Subject of
the issuer certificate (based on hashing the Subject's encoded bytes).
There's one more component that's not in this repository: an HTTP server
to serve objects from the S3-compatible data store. For Let's Encrypt, this
role is served by a CDN. Note that the CA must be carefully configured so
that the CRLBaseURL for each issuer matches the publicly accessible URL
where that issuer's CRLs will be served.
Shard assignment
Certificates are assigned to shards explicitly at issuance time, with the
selected shard baked into the certificate as part of its CRLDistributionPoints
extension. The shard is selected based on taking the (random) low bytes of the
serial number modulo the number of shards produced by that certificate's issuer.
Storage
When a certificate is revoked, the new status is written to both thecertificateStatus table and the revokedCertificates table. The former
contains an entry for every certificate, explicitly recording that newly-issued
certificates are not revoked. The latter is less explicit but more scalable,
containing rows only for certificates which have been revoked.
The SA only exposes the latter of these two mechanisms via theGetRevokedCertsByShard method, which returns revoked certificates whoseshardIdx matches the requested shard. The certificateStatus table will be
removed in the near future.
---
Error Handling
Error Handling Guidance
Previously Boulder has used a mix of various error types to represent errors internally, mainly the core.XXXError types and probs.ProblemDetails, without any guidance on which should be used when or where.
We have switched away from this to using a single unified internal error type, boulder/errors.BoulderError which should be used anywhere we need to pass errors between components and need to be able to indicate and test the type of the error that was passed. probs.ProblemDetails should only be used in the WFE when creating a problem document to pass directly back to the user client.
A mapping exists in the WFE to map all of the available boulder/errors.ErrorTypes to the relevant probs.ProblemTypes. Internally errors should be wrapped when doing so provides some further context to the error that aides in debugging or will be passed back to the user client. An error may be unwrapped, or a simple stdlib error may be used, but doing so means the probs.ProblemType mapping will always be probs.ServerInternalProblem so should only be used for errors that do not need to be presented back to the user client.
boulder/errors.BoulderErrors have two components: an internal type, boulder/errors.ErrorType, and a detail string. The internal type should be used for a. allowing the receiver to determine what caused the error, e.g. by using boulder/errors.NotFound to indicate a DB operation couldn't find the requested resource, and b. allowing the WFE to convert the error to the relevant probs.ProblemType for display to the user. The detail string should provide a user readable explanation of the issue to be presented to the user; the only exception to this is when the internal type is boulder/errors.InternalServer in which case the detail of the error will be stripped by the WFE and the only message presented to the user will be provided by the caller in the WFE.
Error type testing should be done with boulder/errors.Is instead of locally doing a type cast test.
---
Health
gRPC Health checking
We automatically implement the [gRPC health service] automatically for all our
gRPC servers.
There are two ways a service implementation can offer health information:
- By implementing Health(context.Context) error, which will be called every
5s. If it returns nil, the service is set to healthy. If it returns
non-nil, the service is set to unhealthy. The health check interval can be
controlled with grpc.serverBuilder.WithCheckInterval at build time.
- By implementing OnHealthy(func()). This will be called by
grpc.serverBuilder.Build, passing in a closure that sets the service status
to healthy. This is useful for services that start unhealthy and then become
healthy exactly once. At a protocol layer, setting the service healthy [pushes]
out a message to clients immediately, so they don't need to wait on the next
health check poll.
[gRPC health service]: https://pkg.go.dev/google.golang.org/grpc/health
[pushes]: https://github.com/grpc/grpc/blob/5b6492ea90b2b867a6adad1b10a6edda28e860d1/src/proto/grpc/health/v1/health.proto#L47-L62
---
ISSUANCE CYCLE
The Issuance Cycle
What happens during an ACME finalize request?
At a high level:
1. Check that all authorizations are good.
2. Recheck CAA for hostnames that need it.
3. Allocate and store a serial number.
4. Generate and store linting precertificate.
5. Sign, log (and don't store) precertificate.
6. Submit precertificate to CT.
7. Generate linting final certificate. Not logged or stored.
8. Sign, log, and store final certificate.
9. Submit final certificate to CT.
Revocation can happen at any time after (5), whether or not step (6) was successful. We do things this way so that even in the event of a power failure or error storing data, we have a record of what we planned to sign (the tbsCertificate bytes of the linting certificate).
Note that to avoid needing a migration, we chose to store the linting certificate from (5) in the "precertificates" table, which is now a bit of a misnomer.
---
Logging
Logging
Boulder can log to stdout/stderr, syslog, or both. Boulder components
generally have a syslog portion of their JSON config that indicates the
maximum level of log that should be sent to a given destination. For instance,
in test/config/wfe2.json:
"syslog": {
"stdoutlevel": 4,
"sysloglevel": 6
},This indicates that logs of level 4 or below (error and warning) should be
emitted to stdout/stderr, and logs of level 6 or below (error, warning, notice, and
info) should be emitted to syslog, using the local Unix socket method. The
highest meaningful value is 7, which enables debug logging.
The stdout/stderr logger uses ANSI escape codes to color warnings as yellow
and errors as red, if stdout is detected to be a terminal.
The default value for these fields is 6 (INFO) for syslogLevel and 0 (no logs)
for stdoutLevel. To turn off syslog logging entirely, set syslogLevel to -1.
In Boulder's development environment, we enable stdout logging because that
makes it easier to see what's going on quickly. In production, we disable stdout
logging because it would duplicate the syslog logging. We preferred the syslog
logging because it provides things like severity level in a consistent way with
other components. But we may move to stdout/stderr logging to make it easier to
containerize Boulder.
Boulder has a number of adapters to take other packages' log APIs and send them
to syslog as expected. For instance, we provide a custom logger for mysql, grpc,
and prometheus that forwards to syslog. This is configured in StatsAndLogging in
cmd/shell.go.
There are some cases where we output to stdout regardless of the JSON config
settings:
- Panics are always emitted to stdout
- Packages that Boulder relies on may occasionally emit to stdout (though this
is generally not ideal and we try to get it changed).
Typically these output lines will be collected by systemd and forwarded to
syslog.
Verification
We attach a simple checksum to each log line. This is not a cryptographically
secure hash, but is intended to let us catch corruption in the log system. This
is a short chunk of base64 encoded data near the beginning of the log line. It
is consumed by cmd/log-validator.
---
Multi Va
Multi-VA implementation
Boulder supports a multi-perspective validation feature intended to increase
resilience against local network hijacks and BGP attacks. It is currently
deployed in a production
capacity
by Let's Encrypt.
If you follow the Development Instructions
to set up a Boulder environment in Docker and then change your docker-compose.yml'sBOULDER_CONFIG_DIR to test/config-next instead of test/config you'll have
a Boulder environment configured with two primary VA instances (validation
requests are load balanced across the two) and two remote VA instances (each
primary VA will ask both remote VAs to perform matching validations for each
primary validation). Of course this is a development environment so both the
primary and remote VAs are all running on one host.
The boulder-va service (here and remoteva service (here) are distinct pieces of software that utilize the same package (here).
The boulder-ra uses the same RPC interface
to ask for a primary validation as the primary VA uses to ask a remote VA for a
confirmation validation.
Primary VA instances contain a "remoteVAs" configuration element. If present
it specifies gRPC service addresses for remoteva instances to use as remote
VAs. There's also a handful of feature flags that control how the primary VAs
handle the remote VAs.
In the development environment with config-next the two primary VAs are va1.service.consul:9092 andva2.service.consul:9092 and usetest/config-next/va.json
as their configuration. This config file specifies two "remoteVA"s,rva1.service.consul:9097 and va2.service.consul:9098 and enforces
that a maximum of 1 of the 2 remote VAs disagree
with the primary VA for all validations. The remote VA instances usetest/config-next/remoteva-a.json
andtest/config-next/remoteva-b.json
as their config files.
We require that almost all remote validation requests succeed; the exact number
is controlled by the VA based on the thresholds required by MPIC. If the number of
failing remote VAs exceeds that threshold, validation is terminated. If the
number of successful remote VAs is high enough that it would be impossible for
the outstanding remote VAs to exceed that threshold, validation immediately
succeeds.
There are some integration tests that test this end to end. The most relevant is
probablytest_http_multiva_threshold_fail.
It tests that a HTTP-01 challenge made to a webserver that only gives the
correct key authorization to the primary VA and not the remotes will fail the
multi-perspective validation.
---
Profiling
Profiling
Boulder components expose profiling endpoints on the port specified
by their --debug-addr flag. An index of available endpoints can be
found at /debug/pprof/ on each service.
Additionally, if the environment variable $GOMEMLIMIT is set, Boulder
components will automatically write a heap and goroutine dump when
it's hit. Note that $GOMEMLIMIT also sets a soft memory limit for
the runtime. See https://pkg.go.dev/runtime#hdr-Environment_Variables.
The dump will happen at most once per hour.
---
Redis
Redis
We use Redis for storing rate limit data. The Boulder dev environment stands up
two nodes. We use the Ring client in the github.com/redis/go-redis package to
consistently hash our reads and writes across these two nodes.
Debugging
Our main tool for interacting with Redis is redis-cli. You probably don't
have redis-cli on your host, so we'll run it in a Docker container. We
also need to pass some specific arguments for TLS and authentication. There's a
script that handles all that for you: test/redis-cli.sh. First, make sure your
redis is running:
docker compose up boulderThen, in a different window, run the following to connect to bredis_1:
./test/redis-cli.sh -h 10.77.77.4Similarly, to connect to bredis_2:
./test/redis-cli.sh -h 10.77.77.5You can pass any IP address for the -h (host) parameter. The full list of IP
addresses for Redis nodes is in docker-compose.yml. You can also pass other
redis-cli commandline parameters. They'll get passed through.
You may want to go a level deeper and communicate with a Redis node using the
Redis protocol. Here's the command to do that (run from the Boulder root):
openssl s_client -connect 10.77.77.4:4218 \
-CAfile test/certs/ipki/minica.pem \
-cert test/certs/ipki/localhost/cert.pem \
-key test/certs/ipki/localhost/key.pemThen, first thing when you connect, run AUTH <user> <password>. You can get a
list of usernames and passwords from test/redis-ratelimits.config.
---
Release
Boulder Release Process
A description and demonstration of the full process for tagging a normal weekly
release and a hotfix release.
Once a release is tagged, it will be generally deployed to
staging and then to
production over the next few days.
Goals
1. All development, including reverts and hotfixes needed to patch a broken
release, happens on the main branch of this repository. Code is never
deployed without being reviewed and merged here first, and code is never
landed on a release branch that isn't landed on main first.
2. Doing a normal release requires approximately zero thought. It Just Works.
3. Doing a hotfix release differs as little as possible from the normal release
process.
Release Schedule
Boulder developers make a new release at the beginning of each week, typically
around 10am PST Monday. Operations deploys the new release to the staging
environment on Tuesday,
typically by 2pm PST. If there have been no issues discovered with the release
from its time in staging, then on Thursday the operations team deploys the
release to the production environment.
Holidays, unexpected bugs, and other resource constraints may affect the above
schedule and result in staging or production updates being skipped. It should be
considered a guideline for normal releases but not a strict contract.
Release Structure
As of 2025-06-30, releases are tagged with a tag of the form v0.YYYYMMDD.N, where
the YYYYMMDD is the date that the initial release is cut (usually the Monday
of the current week), and N is an integer indicating the hotfix number,
starting at 0. For example, a regular release might be v0.20250707.0, and
the first hotfix for that release would be v0.20250707.1.
Historically, releases were tagged with the form release-YYYY-MM-DD[x], where[x] was an optional lowercase letter suffix for hotfixes. For example, the
second hotfix release (i.e. third release overall) in the third week of
January 2022 was release-2022-01-18b.
All release tags are signed with a key associated with a Boulder developer. Tag
signatures are automatically verified by GitHub using the public keys that
developer has uploaded, and are additionally checked before being built and
deployed to our staging and production environments. Note that, due to how Git
works, in order for a tag to be signed it must also have a message; we set the
tag message to just be a slightly more readable version of the tag name.
Making a Release
Prerequisites
* You must have a GPG key with signing capability:
* Checking for existing GPG keys
* If you don't have a GPG key with signing capability, create one:
* Generating a new local GPG key
* Generating a new Yubikey GPG key
* The signing GPG key must be added to your GitHub account:
* Adding a new GPG key to your GitHub
account
git may* need to be configured to call the correct GPG binary:
* The default: git config --global gpg.program gpg is correct for most Linux platforms
* On macOS and some Linux platforms: git config --global gpg.program gpg2 is correct
* git must be configured to use the correct GPG key:
* Telling Git about your GPG key
* Understand the process for signing tags
Regular Releases
Simply create a signed tag. The tools/release/tag tool will automatically
determine the correct tag name based on the current date.
go run github.com/letsencrypt/boulder/tools/release/tag@mainThis will print the newly-created tag and instructions on how to push it after
you are satisfied that it is correct. Alternately you can run the command with
the -push flag to push the resulting tag automatically.
Hotfix Releases
Sometimes it is necessary to create a new release which looks like a prior
release but with one or more additional commits added. This is usually the case
when we discover a critical bug in the currently-deployed version that needs to
be fixed, but we don't want to include other changes that have already been
merged to main since the currently-deployed release was tagged.
In this situation, we create a new hotfix release branch starting at the point
of the previous release tag. We then use the normal GitHub PR and code-review
process to copy the necessary fix(es) from main (where they must already be
merged) to the release branch. Finally we create a new release tag at the tip
of the release branch instead of the tip of main.
To create the new release branch, substitute the name of the release tag which you want to use as the starting point into this command:
go run github.com/letsencrypt/boulder/tools/release/branch@main v0.YYYYMMDD.0This will create a release branch named release-branch-v0.YYYYMMDD. When all necessary PRs have been merged into that branch, create the new tag by substituting the branch name into this command:
go run github.com/letsencrypt/boulder/tools/release/tag@main release-branch-v0.YYYYMMDDDeploying Releases
When doing a release, SRE's tooling will check that:
1. GitHub shows that tests have passed for the commit at the planned release
tag.
2. The planned release tag is an ancestor of the current main on GitHub, or
the planned release tag is equal to the head of a branch named
release-branch-XXX, and all commits between main and the head of that
branch are cherry-picks of commits which landed on main following the
normal review process.
These checks ensure that all deployed code has been properly reviewed and tested before reaching production environments.
---
README
Boulder - An ACME CA
[](https://github.com/letsencrypt/boulder/actions/workflows/boulder-ci.yml?query=branch%3Amain)
This is an implementation of an ACME-based CA. The ACME
protocol allows the CA to automatically
verify that an applicant for a certificate actually controls an identifier, and
allows subscribers to issue and revoke certificates for the identifiers they
control. Boulder is the software that runs Let's
Encrypt.
Contents
* Overview
* Setting up Boulder
* Development
* Working with Certbot
* Working with another ACME Client
* Production
* Contributing
* License
Overview
Boulder is divided into the following main components:
1. Web Front Ends (one per API version)
2. Registration Authority
3. Validation Authority
4. Certificate Authority
5. Storage Authority
6. Publisher
7. CRL Updater
This component model lets us separate the function of the CA by security
context. The Web Front End, Validation Authority, CRL Storer, and
Publisher need access to the Internet, which puts them at greater risk of
compromise. The Registration Authority can live without Internet
connectivity, but still needs to talk to the Web Front End and Validation
Authority. The Certificate Authority need only receive instructions from the
Registration Authority. All components talk to the SA for storage, so most
lines indicating SA RPCs are not shown here.
CA ---------> Publisher
^
|
Subscriber -> WFE --> RA --> SA --> MariaDB
| ^
Subscriber server <- VA <----+ |
|
Browser -----> S3 <----- CRL Storer/UpdaterInternally, the logic of the system is based around five types of objects:
accounts, authorizations, challenges, orders and certificates, mapping directly
to the resources of the same name in ACME. Requests from ACME clients result in
new objects and changes to objects. The Storage Authority maintains persistent
copies of the current set of objects.
Boulder uses gRPC for inter-component communication. For components that you
want to be remote, it is necessary to instantiate a "client" and "server" for
that component. The client implements the component's Go interface, while the
server has the actual logic for the component. A high level overview for this
communication model can be found in the gRPC
documentation.
The full details of how the various ACME operations happen in Boulder are
laid out in the
Boulder implementation details.
Setting up Boulder
Development
Boulder has a Dockerfile and uses Docker Compose to make it easy to install
and set up all its dependencies. This is how the maintainers work on Boulder,
and is our main recommended way to run it for development/experimentation. It
is not suitable for use as a production environment.
While we aim to make Boulder easy to setup ACME client developers may find
Pebble, a miniature version of
Boulder, to be better suited for continuous integration and quick
experimentation.
We recommend setting git's fsckObjects
setting
before getting a copy of Boulder to have better integrity guarantees for
updates.
Clone the boulder repository:
git clone https://github.com/letsencrypt/boulder/
cd boulderAdditionally, make sure you have Docker Engine 1.13.0+ and Docker Compose
1.10.0+ installed. If you do not, you can follow Docker's installation
instructions.
We recommend having at least 2GB of RAM available on your Docker host. In
practice using less RAM may result in the MariaDB container failing in
non-obvious ways.
To run our standard battery of tests (lints, unit, integration):
./t.shTo run all unit tests:
./t.sh -uTo run specific unit tests (example is of the ./va directory):
./t.sh -u -p ./vaTo run all integration tests:
./t.sh -iTo run unit tests and integration tests with coverage:
./t.sh -ui -c --coverage-dir=./test/coverage/mytestrunTo run specific integration tests (example runs TestGenerateValidity and TestWFECORS):
./t.sh -i -f TestGenerateValidity/TestWFECORSTo do any of the above, but using the "config-next" configuration, which
represents a likely future state (e.g. including new feature flags):
./tn.sh -your -options -hereTo start Boulder in a Docker container, first run:
docker compose run bsetupthis will write the necessary certificates into
test/certs/[.softhsm-tokens,ipki,webpki];You only need to run this once to create the certificates. If you
need to remove all of the certificates and start over, you can remove
the directories
./test/certs/.softhsm-tokens, ./test/certs/ipki,and
./test/certs/webpki and re-run docker compose run bsetup.Then run:
docker compose upThe configuration in docker-compose.yml mounts your boulder checkout at
/boulder so you can edit code on your host and it will be immediately
reflected inside the Docker containers run with docker compose.
If you have problems with Docker, you may want to try removing all
containers and
volumes.
By default, Boulder uses a fake DNS resolver that resolves all hostnames to
127.0.0.1. This is suitable for running integration tests inside the Docker
container. If you want Boulder to be able to communicate with a client
running on your host instead, you should find your host's Docker IP with:
ifconfig docker0 | grep "inet addr:" | cut -d: -f2 | awk '{ print $1}'And edit docker-compose.yml to change the FAKE_DNS environment variable to
match. This will cause Boulder's stubbed-out DNS resolver (sd-test-srv) to
respond to all A queries with the address in FAKE_DNS.
If you use a host-based firewall (e.g. ufw or iptables) make sure you allow
connections from the Docker instance to your host on the required validation
ports to your ACME client.
Alternatively, you can override the docker-compose.yml default with an
environmental variable using -e (replace 172.17.0.1 with the host IPv4
address found in the command above)
docker compose run --use-aliases -e FAKE_DNS=172.17.0.1 --service-ports boulder ./start.pyRunning tests without the ./test.sh wrapper:
Run unit tests locally, without docker (only works for some directories):
go test ./issuance/...Run all unit tests:
docker compose run --use-aliases boulder go test -p 1 ./...Run unit tests for a specific directory:
docker compose run --use-aliases boulder go test <DIRECTORY>Run integration tests (omit --filter <REGEX> to run all):
docker compose run --use-aliases boulder python3 test/integration-test.py --chisel --gotest --filter <REGEX>Working with Certbot
Check out the Certbot client from https://github.com/certbot/certbot and
follow their setup instructions. Once you've got the client set up, you'll
probably want to run it against your local Boulder. There are a number of
command line flags that are necessary to run the client against a local
Boulder, and without root access. The simplest way to run the client locally
is to use a convenient alias for certbot (certbot_test) with a customSERVER environment variable:
SERVER=http://localhost:4001/directory certbot_test certonly --standalone -d test.example.comYour local Boulder instance uses a fake DNS resolver that returns 127.0.0.1
for any query, so you can use any value for the -d flag. To return an answer
other than 127.0.0.1 change the Boulder FAKE_DNS environment variable to
another IP address.
Working with another ACME Client
Once you have followed the Boulder development environment instructions and have
started the containers you will find the ACME endpoints exposed to your host at
the following URLs:
* ACME v2, HTTP: http://localhost:4001/directory
* ACME v2, HTTPS: https://localhost:4431/directory
To access the HTTPS versions of the endpoints you will need to configure your
ACME client software to use a CA truststore that contains thetest/certs/ipki/minica.pem CA certificate. Seetest/certs/README.md
for more information.
Your local Boulder instance uses a fake DNS resolver that returns 127.0.0.1
for any query, allowing you to issue certificates for any domain as if it
resolved to your localhost. To return an answer other than 127.0.0.1 change
the Boulder FAKE_DNS environment variable to another IP address.
Most often you will want to configure FAKE_DNS to point to your host
machine where you run an ACME client.
Production
Boulder is custom built for Let's Encrypt and is intended only to support the
Web PKI and the CA/Browser forum's baseline requirements. In our experience
often Boulder is not the right fit for organizations that are evaluating it for
production usage. In most cases a centrally managed PKI that doesn't require
domain-authorization with ACME is a better choice. For this environment we
recommend evaluating a project other than Boulder.
We offer a brief deployment and implementation
guide
that describes some of the required work and security considerations involved in
using Boulder in a production environment. As-is the docker based Boulder
development environment is not suitable for
production usage. It uses private key material that is publicly available,
exposes debug ports and is brittle to component failure.
While we are supportive of other organization's deploying Boulder in
a production setting we prioritize support and development work that favors
Let's Encrypt's mission. This means we may not be able to provide timely support
or accept pull-requests that deviate significantly from our first line goals. If
you've thoroughly evaluated the alternatives and Boulder is definitely the best
fit we're happy to answer questions to the best of our ability.
Contributing
Please take a look at
CONTRIBUTING.md
for our guidelines on submitting patches, code review process, code of conduct,
and various other tips related to working on the codebase.
Code of Conduct
The code of conduct for everyone participating in this community in any capacity
is available for reference
on the community forum.
License
This project is licensed under the Mozilla Public License 2.0, the full text
of which can be found in the
LICENSE.txt
file.
---