### Acme Divergences # Boulder divergences from ACME While Boulder attempts to implement the ACME specification ([RFC 8555]) as strictly as possible there are places at which we will diverge from the letter of the specification for various reasons. This document describes the difference between [RFC 8555] and Boulder's implementation of ACME, informally called ACMEv2 and available at https://acme-v02.api.letsencrypt.org/directory. A listing of RFC conformant design decisions that may differ from other ACME servers is listed in [implementation_details](https://github.com/letsencrypt/boulder/blob/main/docs/acme-implementation_details.md). Presently, Boulder diverges from the [RFC 8555] ACME spec in the following ways: ## [Section 6.3](https://tools.ietf.org/html/rfc8555#section-6.3) Boulder supports POST-as-GET but does not mandate it for requests that simply fetch a resource (certificate, order, authorization, or challenge). ## [Section 7.1.2](https://tools.ietf.org/html/rfc8555#section-7.1.2) Boulder does not supply the `orders` field on account objects. We intend to support this non-essential feature in the future. Please follow Boulder Issue [#3335](https://github.com/letsencrypt/boulder/issues/3335). ## [Section 7.4](https://tools.ietf.org/html/rfc8555#section-7.4) Boulder does not accept the optional `notBefore` and `notAfter` fields of a `newOrder` request payload. ## [Section 7.4.1](https://tools.ietf.org/html/rfc8555#section-7.4.1) Pre-authorization is an optional feature and we have no plans to implement it. V2 clients should use order based issuance without pre-authorization. ## [Section 7.4.2](https://tools.ietf.org/html/rfc8555#section-7.4.2) Boulder does not process `Accept` headers for `Content-Type` negotiation when retrieving certificates. ## [Section 8.2](https://tools.ietf.org/html/rfc8555#section-8.2) Boulder does not implement the ability to retry challenges or the `Retry-After` header. [RFC 8555]: https://tools.ietf.org/html/rfc8555 --- ### Acme Implementation Details # Boulder implementation details The ACME specification ([RFC 8555]) clearly dictates what Clients and Servers must do to properly implement the protocol. The specification is intentionally silent, or vague, on certain points to give developers freedom in making certain decisions or to follow guidance from other RFCs. Due to this, two ACME Servers might fully conform to the RFC but behave slightly differently. ACME Clients should not "over-fit" on Boulder or the Let's Encrypt production service, and aim to be compatible with a wide range of ACME Servers, including the [Pebble](https://github.com/letsencrypt/pebble) test server. The following items are a partial listing of RFC-conformant design decisions Boulder and/or LetsEncrypt have made. This listing is not complete, and is based on known details which have caused issues for developers in the past. This listing may not reflect the current status of Boulder or the configuration of LetsEncrypt's production instance and is provided only as a reference for client developers. Please note: these design implementation decisions are fully conformant with the RFC specification and are not [divergences](https://github.com/letsencrypt/boulder/blob/main/docs/acme-divergences.md). ## Object Reuse The ACME specification does not prohibit certain objects to be re-used. ### Authorization Boulder may recycle previously "valid" or "pending" `Authorizations` for a given `Account` when creating a new `Order`. ### Order Boulder may return a previously created `Order` when a given `Account` submits a new `Order` that is identical to a previously submitted `Order` that is in the "pending" or "ready" state. ## Alternate Chains The production Boulder instance for LetsEncrypt in enabled with support for Alternate chains. ## Certificate Request Domains The RFC states the following: The CSR MUST indicate the exact same set of requested identifiers as the initial newOrder request. Identifiers of type "dns" MUST appear either in the commonName portion of the requested subject name or in an extensionRequest attribute [RFC2985] requesting a subjectAltName extension, or both. Boulder requires all domains to be specified in the `subjectAltName` extension, and will reject a CSR if a domain specified in the `commonName` is not present in the `subjectAltName`. Additionally, usage of the `commonName` was previously deprecated by the CA/B Forum and in earlier RFCs. For more information on this see [Pebble Issue #304](https://github.com/letsencrypt/pebble/issues/304) and [Pebble Issue #233](https://github.com/letsencrypt/pebble/issues/233). ## RSA Key Size The ACME specification is silent as to minimum key size. The [CA/Browser Forum](https://cabforum.org/) sets the key size requirements which LetsEncrypt adheres to. Effective 2020-09-17, LetsEncrypt further requires all RSA keys for end-entity (leaf) certificates have a modulus of length 2048, 3072, or 4096. Other CAs may or may not have the same restricted set of supported RSA key sizes. For more information [read the Official Announcement](https://community.letsencrypt.org/t/issuing-for-common-rsa-key-sizes-only/133839). --- ### CODE OF CONDUCT # Code of Conduct The code of conduct for everyone participating in this community in any capacity is available for reference [on the community forum](https://community.letsencrypt.org/guidelines). --- ### Config Validation # Configuration Validation We use a fork of https://github.com/go-playground/validator which can be found at https://github.com/letsencrypt/validator. ## Usage By default Boulder validates config files for all components with a registered validator. Validating a config file for a given component is as simple as running the component directly: ```shell $ ./bin/boulder-observer -config test/config-next/observer.yml Error validating config file "test/config-next/observer.yml": Key: 'ObsConf.MonConfs[1].Kind' Error:Field validation for 'Kind' failed on the 'oneof' tag ``` or by running the `boulder` binary and passing the component name as a subcommand: ```shell $ ./bin/boulder boulder-observer -config test/config-next/observer.yml Error validating config file "test/config-next/observer.yml": Key: 'ObsConf.MonConfs[1].Kind' Error:Field validation for 'Kind' failed on the 'oneof' tag ``` ## Struct Tag Tips You can find the full list of struct tags supported by the validator [here] (https://pkg.go.dev/github.com/go-playground/validator/v10#section-documentation). The following are some tips for struct tags that are commonly used in our configuration files. ### `required` The required tag means that the field is not allowed to take its zero value, or equivalently, is not allowed to be omitted. Note that this does not validate that slices or maps have contents, it simply guarantees that they are not nil. For fields of those types, you should use min=1 or similar to ensure they are not empty. There are also "conditional" required tags, such as `required_with`, `required_with_all`, `required_without`, `required_without_all`, and `required_unless`. These behave exactly like the basic required tag, but only if their conditional (usually the presence or absence of one or more other named fields) is met. ### `omitempty` The omitempty tag allows a field to be empty, or equivalently, to take its zero value. If the field is omitted, none of the other validation tags on the field will be enforced. This can be useful for tags like validate="omitempty,url", for a field which is optional, but must be a URL if it is present. The omitempty tag can be "overruled" by the various conditional required tags. For example, a field with tag `validate="omitempty,url,required_with=Foo"` is allowed to be empty when field Foo is not present, but if field Foo is present, then this field must be present and must be a URL. ### `-` Normally, config validation descends into all struct-type fields, recursively validating their fields all the way down. Sometimes this can pose a problem, when a nested struct declares one of its fields as required, but a parent struct wants to treat the whole nested struct as optional. The "-" tag tells the validation not to recurse, marking the tagged field as optional, and therefore making all of its sub-fields optional as well. We use this tag for many config duration and password file struct valued fields which are optional in some configs but required in others. ### `structonly` The structonly tag allows a struct valued field to be empty, or equivalently, to take its zero value, if it's not "overruled" by various conditional tags. If the field is omitted the recursive validation of the structs fields will be skipped. This can be useful for tags like `validate:"required_without=Foo,structonly"` for a struct valued field which is only required, and thus should only be validated, if field `Foo` is not present. ### `min=1`, `gte=1` These validate that the value of integer valued field is greater than zero and that the length of the slice or map is greater than zero. For instance, the following would be valid config for a slice valued field tagged with `required`. ```json { "foo": [], } ``` But, only the following would be valid config for a slice valued field tagged with `min=1`. ```json { "foo": ["bar"], } ``` ### `len` Same as `eq` (equal to) but can also be used to validate the length of the strings. ### `hostname_port` The [docs](https://pkg.go.dev/github.com/go-playground/validator/v10#hdr-HostPort) for this tag are scant with detail, but it validates that the value is a valid RFC 1123 hostname and port. It is used to validate many of the `ListenAddress` and `DebugAddr` fields of our components. #### Future Work This tag is compatible with IPv4 addresses, but not IPv6 addresses. We should consider fixing this in our fork of the validator. ### `dive` This tag is used to validate the values of a slice or map. For instance, the following would be valid config for a slice valued field (`[]string`) tagged with `min=1,dive,oneof=bar baz`. ```json { "foo": ["bar", "baz"], } ``` Note that the `dive` tag introduces an order-dependence in writing tags: tags that come before `dive` apply to the current field, while tags that come after `dive` apply to the current field's child values. In the example above: `min=1` applies to the length of the slice (`[]string`), while `oneof=bar baz` applies to the value of each string in the slice. We can also use `dive` to validate the values of a map. For instance, the following would be valid config for a map valued field (`map[string]string`) tagged with `min=1,dive,oneof=one two`. ```json { "foo": { "bar": "one", "baz": "two" }, } ``` `dive` can also be invoked multiple times to validate the values of nested slices or maps. For instance, the following would be valid config for a slice of slice valued field (`[][]string`) tagged with `min=1,dive,min=2,dive,oneof=bar baz`. ```json { "foo": [ ["bar", "baz"], ["baz", "bar"], ], } ``` - `min=1` will be applied to the outer slice (`[]`). - `min=2` will be applied to inner slice (`[]string`). - `oneof=bar baz` will be applied to each string in the inner slice. ### `keys` and `endkeys` These tags are used to validate the keys of a map. For instance, the following would be valid config for a map valued field (`map[string]string`) tagged with `min=1,dive,keys,eq=1|eq=2,endkeys,required`. ```json { "foo": { "1": "bar", "2": "baz", }, } ``` - `min=1` will be applied to the map itself - `eq=1|eq=2` will be applied to the map keys - `required` will be applied to map values --- ### CONTRIBUTING Thanks for helping us build Boulder! This page contains requirements and guidelines for Boulder contributions. # Patch Requirements * All new functionality and fixed bugs must be accompanied by tests. * All patches must meet the deployability requirements listed below. * We prefer pull requests from external forks be created with the ["Allow edits from maintainers"](https://github.com/blog/2247-improving-collaboration-with-forks) checkbox selected. # Review Requirements * All pull requests must receive at least one approval by a [CODEOWNER](../CODEOWNERS) other than the author. This is enforced by GitHub itself. * All pull requests should receive at least two approvals by [Trusted Contributors](https://github.com/letsencrypt/cp-cps/blob/main/CP-CPS.md#161-definitions). This requirement may be waived when: * the change only modifies documentation; * the change only modifies tests; * in exceptional circumstances, such as when no second reviewer is available at all. This requirement should not be waived when: * the change is not written by a Trusted Contributor, to ensure that at least two TCs have eyes on it. * New commits pushed to a branch invalidate previous reviews. In other words, a reviewer must give positive reviews of a branch after its most recent pushed commit. * If a branch contains commits from multiple authors, it needs a reviewer who is not an author of commits on that branch. * Review changes to or addition of tests just as rigorously as you review code changes. Consider: Do tests actually test what they mean to test? Is this the best way to test the functionality in question? Do the tests cover all the functionality in the patch, including error cases? * Are there new RPCs or config fields? Make sure the patch meets the Deployability rules below. * All review conversations must be resolved. This helps to ensure that review feedback has been fully addressed. This is enforced by GitHub itself. # Merge Requirements We have a bot that will comment on some PRs indicating there are: 1. configuration changes 2. SQL schema changes 3. feature flag changes These may require either a CP/CPS review or filing of a ticket to make matching changes in production. It is the responsibility of the person merging the PR to make sure the required action has been performed before merging. Usually this will be confirmed in a comment or in the PR description. When creating a pull request that has external dependencies, like another pull request being merged or deployed to production, or approval from SRE, the author should leave a review comment on the pull request (i.e. create a "conversation" by commenting on a specific file or line of a file) noting what the pull request is blocked on. The comment should include instructions to only resolve that conversation once the external conditions are met. Since our repository is configured to require all conversations be resolved before merging, this acts as a protection against early merges. # Patch Guidelines * Please include helpful comments. No need to gratuitously comment clear code, but make sure it's clear why things are being done. Include information in your pull request about what you're trying to accomplish with your patch. * Avoid named return values. See [#3017](https://github.com/letsencrypt/boulder/pull/3017) for an example of a subtle problem they can cause. * Do not include `XXX`s or naked `TODO`s. Use the formats: ```go // TODO(): Hoverboard + Time-machine unsupported until upstream patch. // TODO(#): Pending hoverboard/time-machine interface. // TODO(@githubusername): Enable hoverboard kickflips once interface is stable. ``` # Squash merging Once a pull request is approved and the tests are passing, the author or any other committer can merge it. We always use [squash merges](https://github.com/blog/2141-squash-your-commits) via GitHub's web interface. That means that during the course of your review you should generally not squash or amend commits, or force push. Even if the changes in each commit are small, keeping them separate makes it easier for us to review incremental changes to a pull request. Rest assured that those tiny changes will get squashed into a nice meaningful-size commit when we merge. If the CI tests are failing on your branch, you should look at the logs to figure out why. Sometimes (though rarely) they fail spuriously, in which case you can post a comment requesting that a project owner kick the build. # Error handling All errors must be addressed in some way: That may be simply by returning an error up the stack, or by handling it in some intelligent way where it is generated, or by explicitly ignoring it and assigning to `_`. We use the `errcheck` tool in our integration tests to make sure all errors are addressed. Note that ignoring errors, even in tests, should be rare, since they may generate hard-to-debug problems. When handling errors, always do the operation which creates the error (usually a function call) and the error checking on separate lines: ``` err := someOperation(args) if err != nil { return nil, fmt.Errorf("some operation failed: %w", err) } ``` We avoid the `if err := someOperation(args); err != nil {...}` style as we find it to be less readable and it can give rise to surprising scoping behavior. We define two special types of error. `BoulderError`, defined in errors/errors.go, is used specifically when an typed error needs to be passed across an RPC boundary. For instance, if the SA returns "not found", callers need to be able to distinguish that from a network error. Not every error that may pass across an RPC boundary needs to be a BoulderError, only those errors that need to be handled by type elsewhere. Handling by type may be as simple as turning a BoulderError into a specific type of ProblemDetail. The other special type of error is `ProblemDetails`. We try to treat these as a presentation-layer detail, and use them only in parts of the system that are responsible for rendering errors to end-users, i.e. WFE2. Note one exception: The VA RPC layer defines its own `ProblemDetails` type, which is returned to the RA and stored as part of a challenge (to eventually be rendered to the user). Within WFE2, ProblemDetails are sent to the client by calling `sendError()`, which also logs the error. For internal errors like timeout, or any error type that we haven't specifically turned into a ProblemDetail, we return a ServerInternal error. This avoids unnecessarily exposing internals. It's possible to add additional errors to a logEvent using `.AddError()`, but this should only be done when there is is internal-only information to log that isn't redundant with the ProblemDetails sent to the user. Note that the final argument to `sendError()`, `ierr`, will automatically get added to the logEvent for ServerInternal errors, so when sending a ServerInternal error it's not necessary to separately call `.AddError`. # Deployability We want to ensure that a new Boulder revision can be deployed to the currently running Boulder production instance without requiring config changes first. We also want to ensure that during a deploy, services can be restarted in any order. That means two things: ## Good zero values for config fields Any newly added config field must have a usable [zero value](https://tour.golang.org/basics/12). That is to say, if a config field is absent, Boulder shouldn't crash or misbehave. If that config file names a file to be read, Boulder should be able to proceed without that file being read. Note that there are some config fields that we want to be a hard requirement. To handle such a field, first add it as optional, then file an issue to make it required after the next deploy is complete. In general, we would like our deploy process to be: deploy new code + old config; then immediately after deploy the same code + new config. This makes deploys cheaper so we can do them more often, and allows us to more readily separate deploy-triggered problems from config-triggered problems. ## Flag-gating features When adding significant new features or replacing existing RPCs the `boulder/features` package should be used to gate its usage. To add a flag, a new field of the `features.Config` struct should be added. All flags default to false. In order to test if the flag is enabled elsewhere in the codebase you can use `features.Get().ExampleFeatureName` which gets the `bool` value from a global config. Each service should include a `map[string]bool` named `Features` in its configuration object at the top level and call `features.Set` with that map immediately after parsing the configuration. For example to enable `UseNewMetrics` and disable `AccountRevocation` you would add this object: ```json { ... "features": { "UseNewMetrics": true, "AccountRevocation": false, } } ``` Feature flags are meant to be used temporarily and should not be used for permanent boolean configuration options. ### Deprecating a feature flag Once a feature has been enabled in both staging and production, someone on the team should deprecate it: - Remove any instances of `features.Get().ExampleFeatureName`, adjusting code as needed. - Move the field to the top of the `features.Config` struct, under a comment saying it's deprecated. - Remove all references to the feature flag from `test/config-next`. - Add the feature flag to `test/config`. This serves to check that we still tolerate parsing the flag at startup, even though it is ineffective. - File a ticket to remove the feature flag in staging and production. - Once the feature flag is removed in staging and production, delete it from `test/config` and `features.Config`. ### Gating RPCs When you add a new RPC to a Boulder service (e.g. `SA.GetFoo()`), all components that call that RPC should gate those calls using a feature flag. Since the feature's zero value is false, a deploy with the existing config will not call `SA.GetFoo()`. Then, once the deploy is complete and we know that all SA instances support the `GetFoo()` RPC, we do a followup config deploy that sets the default value to true, and finally remove the flag entirely once we are confident the functionality it gates behaves correctly. ### Gating migrations We use [database migrations](https://en.wikipedia.org/wiki/Schema_migration) to modify the existing schema. These migrations will be run on live data while Boulder is still running, so we need Boulder code at any given commit to be capable of running without depending on any changes in schemas that have not yet been applied. For instance, if we're adding a new column to an existing table, Boulder should run correctly in three states: 1. Migration not yet applied. 2. Migration applied, flag not yet flipped. 3. Migration applied, flag flipped. Specifically, that means that all of our `SELECT` statements should enumerate columns to select, and not use `*`. Also, generally speaking, we will need a separate model `struct` for serializing and deserializing data before and after the migration. This is because the ORM package we use, [`borp`](https://github.com/letsencrypt/borp), expects every field in a struct to map to a column in the table. If we add a new field to a model struct and Boulder attempts to write that struct to a table that doesn't yet have the corresponding column (case 1), borp will fail with `Insert failed table posts has no column named Foo`. There are examples of such models in sa/model.go, along with code to turn a model into a `struct` used internally. An example of a flag-gated migration, adding a new `IsWizard` field to Person controlled by a `AllowWizards` feature flag: ```go # features/features.go: const ( unused FeatureFlag = iota // unused is used for testing AllowWizards // Added! ) ... var features = map[FeatureFlag]bool{ unused: false, AllowWizards: false, // Added! } ``` ```go # sa/sa.go: struct Person { HatSize int IsWizard bool // Added! } struct personModelv1 { HatSize int } // Added! struct personModelv2 { personModelv1 IsWizard bool } func (ssa *SQLStorageAuthority) GetPerson() (Person, error) { if features.Enabled(features.AllowWizards) { // Added! var model personModelv2 ssa.dbMap.SelectOne(&model, "SELECT hatSize, isWizard FROM people") return Person{ HatSize: model.HatSize, IsWizard: model.IsWizard, } } else { var model personModelv1 ssa.dbMap.SelectOne(&model, "SELECT hatSize FROM people") return Person{ HatSize: model.HatSize, } } } func (ssa *SQLStorageAuthority) AddPerson(p Person) (error) { if features.Enabled(features.AllowWizards) { // Added! return ssa.dbMap.Insert(context.Background(), personModelv2{ personModelv1: { HatSize: p.HatSize, }, IsWizard: p.IsWizard, }) } else { return ssa.dbMap.Insert(context.Background(), personModelv1{ HatSize: p.HatSize, // p.IsWizard ignored }) } } ``` You will also need to update the `initTables` function from `sa/database.go` to tell borp which table to use for your versioned model structs. Make sure to consult the flag you defined so that only **one** of the table maps is added at any given time, otherwise borp will error. Depending on your table you may also need to add `SetKeys` and `SetVersionCol` entries for your versioned models. Example: ```go func initTables(dbMap *borp.DbMap) { // < unrelated lines snipped for brevity > if features.Enabled(features.AllowWizards) { dbMap.AddTableWithName(personModelv2, "person") } else { dbMap.AddTableWithName(personModelv1, "person") } } ``` New migrations should be added at `./sa/db-next`: ```shell $ cd sa/db $ sql-migrate new -env="boulder_sa" AddWizards Created migration boulder_sa/20220906165519-AddWizards.sql ``` Finally, edit the resulting file (`sa/db-next/boulder_sa/20220906165519-AddWizards.sql`) to define your migration: ```mysql -- +migrate Up ALTER TABLE people ADD isWizard BOOLEAN SET DEFAULT false; -- +migrate Down ALTER TABLE people DROP isWizard BOOLEAN SET DEFAULT false; ``` # Expressing "optional" Timestamps Timestamps in protocol buffers must always be expressed as [timestamppb.Timestamp](https://pkg.go.dev/google.golang.org/protobuf/types/known/timestamppb). Timestamps must never contain their zero value, in the sense of `timestamp.AsTime().IsZero()`. When a timestamp field is optional, absence must be expressed through the absence of the field, rather than present with a zero value. The `core.IsAnyNilOrZero` function can check these cases. Senders must check that timestamps are non-zero before sending them. Receivers must check that timestamps are non-zero before accepting them. # Rounding time in DB All times that we send to the database are truncated to one second's worth of precision. This reduces the size of indexes that include timestamps, and makes querying them more efficient. The Storage Authority (SA) is responsible for this truncation, and performs it for SELECT queries as well as INSERT and UPDATE. # Release Process The current Boulder release process is described in [release.md](https://github.com/letsencrypt/boulder/blob/main/docs/release.md). New releases are tagged weekly, and artifacts are automatically produced for each release by GitHub Actions. # Dependencies We use [go modules](https://github.com/golang/go/wiki/Modules) and vendor our dependencies. To add a dependency, add the import statement to your .go file, then run `go build` on it. This will automatically add the dependency to go.mod. Next, run `go mod vendor && git add vendor/` to save a copy in the vendor folder. When vendorizing dependencies, it's important to make sure tests pass on the version you are vendorizing. Currently we enforce this by requiring that pull requests containing a dependency update to any version other than a tagged release include a comment indicating that you ran the tests and that they succeeded, preferably with the command line you run them with. Note that you may have to get a separate checkout of the dependency (using `go get` outside of the boulder repository) in order to run its tests, as some vendored modules do not bring their tests with them. ## Updating Dependencies To upgrade a dependency, [see the Go docs](https://github.com/golang/go/wiki/Modules#how-to-upgrade-and-downgrade-dependencies). Typically you want `go get ` rather than `go get -u `, which can introduce a lot of unexpected updates. After running `go get`, make sure to run `go mod vendor && git add vendor/` to update the vendor directory. If you forget, CI tests will catch this. If you are updating a dependency to a version which is not a tagged release, see the note above about how to run all of a dependency's tests and note that you have done so in the PR. Note that updating dependencies can introduce new, transitive dependencies. In general we try to keep our dependencies as narrow as possible in order to minimize the number of people and organizations whose code we need to trust. As a rule of thumb: If an update introduces new packages or modules that are inside a repository where we already depend on other packages or modules, it's not a big deal. If it introduces a new dependency in a different repository, please try to figure out where that dependency came from and why (for instance: "package X, which we depend on, started supporting XML config files, so now we depend on an XML parser") and include that in the PR description. When there are a large number of new dependencies introduced, and we don't need the functionality they provide, we should consider asking the relevant upstream repository for a refactoring to reduce the number of transitive dependencies. # Go Version The [Boulder development environment](https://github.com/letsencrypt/boulder/blob/main/README.md#setting-up-boulder) does not use the Go version installed on the host machine, and instead uses a Go environment baked into a "boulder-tools" Docker image. We build a separate boulder-tools container for each supported Go version. Please see [the Boulder-tools README](https://github.com/letsencrypt/boulder/blob/main/test/boulder-tools/README.md) for more information on upgrading Go versions. # ACME Protocol Divergences While Boulder attempts to implement the ACME specification as strictly as possible there are places at which we will diverge from the letter of the specification for various reasons. We detail these divergences (for both the V1 and V2 API) in the [ACME divergences doc](https://github.com/letsencrypt/boulder/blob/main/docs/acme-divergences.md). # ACME Protocol Implementation Details The ACME specification allows developers to make certain decisions as to how various elements in the RFC are implemented. Some of these fully conformant decisions are listed in [ACME implementation details doc](https://github.com/letsencrypt/boulder/blob/main/docs/acme-implementation_details.md). ## Code of Conduct The code of conduct for everyone participating in this community in any capacity is available for reference [on the community forum](https://community.letsencrypt.org/guidelines). ## Problems or questions? The best place to ask dev related questions is on the [Community Forums](https://community.letsencrypt.org/). --- ### CRLS # CRLs For each issuer certificate, Boulder generates several sharded CRLs. The responsibility is shared across these components: - crl-updater - sa - ca - crl-storer The crl-updater starts the process: for each shard of each issuer, it requests revoked certificate information from the SA. It sends that information to the CA for signing, and receives back a signed CRL. It sends the signed CRL to the crl-storer for upload to an S3-compatible data store. The crl-storer uploads the CRLs to the filename `/.crl`, where `issuerID` is an integer that uniquely identifies the Subject of the issuer certificate (based on hashing the Subject's encoded bytes). There's one more component that's not in this repository: an HTTP server to serve objects from the S3-compatible data store. For Let's Encrypt, this role is served by a CDN. Note that the CA must be carefully configured so that the CRLBaseURL for each issuer matches the publicly accessible URL where that issuer's CRLs will be served. ## Shard assignment Certificates are assigned to shards explicitly at issuance time, with the selected shard baked into the certificate as part of its CRLDistributionPoints extension. The shard is selected based on taking the (random) low bytes of the serial number modulo the number of shards produced by that certificate's issuer. ## Storage When a certificate is revoked, the new status is written to both the `certificateStatus` table and the `revokedCertificates` table. The former contains an entry for every certificate, explicitly recording that newly-issued certificates are not revoked. The latter is less explicit but more scalable, containing rows only for certificates which have been revoked. The SA only exposes the latter of these two mechanisms via the `GetRevokedCertsByShard` method, which returns revoked certificates whose `shardIdx` matches the requested shard. The `certificateStatus` table will be removed in the near future. --- ### Error Handling # Error Handling Guidance Previously Boulder has used a mix of various error types to represent errors internally, mainly the `core.XXXError` types and `probs.ProblemDetails`, without any guidance on which should be used when or where. We have switched away from this to using a single unified internal error type, `boulder/errors.BoulderError` which should be used anywhere we need to pass errors between components and need to be able to indicate and test the type of the error that was passed. `probs.ProblemDetails` should only be used in the WFE when creating a problem document to pass directly back to the user client. A mapping exists in the WFE to map all of the available `boulder/errors.ErrorType`s to the relevant `probs.ProblemType`s. Internally errors should be wrapped when doing so provides some further context to the error that aides in debugging or will be passed back to the user client. An error may be unwrapped, or a simple stdlib `error` may be used, but doing so means the `probs.ProblemType` mapping will always be `probs.ServerInternalProblem` so should only be used for errors that do not need to be presented back to the user client. `boulder/errors.BoulderError`s have two components: an internal type, `boulder/errors.ErrorType`, and a detail string. The internal type should be used for a. allowing the receiver to determine what caused the error, e.g. by using `boulder/errors.NotFound` to indicate a DB operation couldn't find the requested resource, and b. allowing the WFE to convert the error to the relevant `probs.ProblemType` for display to the user. The detail string should provide a user readable explanation of the issue to be presented to the user; the only exception to this is when the internal type is `boulder/errors.InternalServer` in which case the detail of the error will be stripped by the WFE and the only message presented to the user will be provided by the caller in the WFE. Error type testing should be done with `boulder/errors.Is` instead of locally doing a type cast test. --- ### Health # gRPC Health checking We automatically implement the [gRPC health service] automatically for all our gRPC servers. There are two ways a service implementation can offer health information: - By implementing `Health(context.Context) error`, which will be called every 5s. If it returns `nil`, the service is set to healthy. If it returns non-`nil`, the service is set to unhealthy. The health check interval can be controlled with `grpc.serverBuilder.WithCheckInterval` at build time. - By implementing `OnHealthy(func())`. This will be called by `grpc.serverBuilder.Build`, passing in a closure that sets the service status to healthy. This is useful for services that start unhealthy and then become healthy exactly once. At a protocol layer, setting the service healthy [pushes] out a message to clients immediately, so they don't need to wait on the next health check poll. [gRPC health service]: https://pkg.go.dev/google.golang.org/grpc/health [pushes]: https://github.com/grpc/grpc/blob/5b6492ea90b2b867a6adad1b10a6edda28e860d1/src/proto/grpc/health/v1/health.proto#L47-L62 --- ### ISSUANCE CYCLE # The Issuance Cycle What happens during an ACME finalize request? At a high level: 1. Check that all authorizations are good. 2. Recheck CAA for hostnames that need it. 3. Allocate and store a serial number. 4. Generate and store linting precertificate. 5. Sign, log (and don't store) precertificate. 6. Submit precertificate to CT. 7. Generate linting final certificate. Not logged or stored. 8. Sign, log, and store final certificate. 9. Submit final certificate to CT. Revocation can happen at any time after (5), whether or not step (6) was successful. We do things this way so that even in the event of a power failure or error storing data, we have a record of what we planned to sign (the tbsCertificate bytes of the linting certificate). Note that to avoid needing a migration, we chose to store the linting certificate from (5) in the "precertificates" table, which is now a bit of a misnomer. --- ### Logging # Logging Boulder can log to stdout/stderr, syslog, or both. Boulder components generally have a `syslog` portion of their JSON config that indicates the maximum level of log that should be sent to a given destination. For instance, in `test/config/wfe2.json`: ``` "syslog": { "stdoutlevel": 4, "sysloglevel": 6 }, ``` This indicates that logs of level 4 or below (error and warning) should be emitted to stdout/stderr, and logs of level 6 or below (error, warning, notice, and info) should be emitted to syslog, using the local Unix socket method. The highest meaningful value is 7, which enables debug logging. The stdout/stderr logger uses ANSI escape codes to color warnings as yellow and errors as red, if stdout is detected to be a terminal. The default value for these fields is 6 (INFO) for syslogLevel and 0 (no logs) for stdoutLevel. To turn off syslog logging entirely, set syslogLevel to -1. In Boulder's development environment, we enable stdout logging because that makes it easier to see what's going on quickly. In production, we disable stdout logging because it would duplicate the syslog logging. We preferred the syslog logging because it provides things like severity level in a consistent way with other components. But we may move to stdout/stderr logging to make it easier to containerize Boulder. Boulder has a number of adapters to take other packages' log APIs and send them to syslog as expected. For instance, we provide a custom logger for mysql, grpc, and prometheus that forwards to syslog. This is configured in StatsAndLogging in cmd/shell.go. There are some cases where we output to stdout regardless of the JSON config settings: - Panics are always emitted to stdout - Packages that Boulder relies on may occasionally emit to stdout (though this is generally not ideal and we try to get it changed). Typically these output lines will be collected by systemd and forwarded to syslog. ## Verification We attach a simple checksum to each log line. This is not a cryptographically secure hash, but is intended to let us catch corruption in the log system. This is a short chunk of base64 encoded data near the beginning of the log line. It is consumed by cmd/log-validator. --- ### Multi Va # Multi-VA implementation Boulder supports a multi-perspective validation feature intended to increase resilience against local network hijacks and BGP attacks. It is currently [deployed in a production capacity](https://letsencrypt.org/2020/02/19/multi-perspective-validation.html) by Let's Encrypt. If you follow the [Development Instructions](https://github.com/letsencrypt/boulder#development) to set up a Boulder environment in Docker and then change your `docker-compose.yml`'s `BOULDER_CONFIG_DIR` to `test/config-next` instead of `test/config` you'll have a Boulder environment configured with two primary VA instances (validation requests are load balanced across the two) and two remote VA instances (each primary VA will ask both remote VAs to perform matching validations for each primary validation). Of course this is a development environment so both the primary and remote VAs are all running on one host. The `boulder-va` service ([here](https://github.com/letsencrypt/boulder/tree/main/cmd/boulder-va) and `remoteva` service ([here](https://github.com/letsencrypt/boulder/tree/main/cmd/remoteva)) are distinct pieces of software that utilize the same package ([here](https://github.com/letsencrypt/boulder/tree/main/va)). The boulder-ra uses [the same RPC interface](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/va/proto/va.proto#L8-L10) to ask for a primary validation as the primary VA uses to ask a remote VA for a confirmation validation. Primary VA instances contain a `"remoteVAs"` configuration element. If present it specifies gRPC service addresses for `remoteva` instances to use as remote VAs. There's also a handful of feature flags that control how the primary VAs handle the remote VAs. In the development environment with `config-next` the two primary VAs are `va1.service.consul:9092` and `va2.service.consul:9092` and use [`test/config-next/va.json`](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/test/config-next/va.json) as their configuration. This config file specifies two `"remoteVA"s`, `rva1.service.consul:9097` and `va2.service.consul:9098` and enforces [that a maximum of 1 of the 2 remote VAs disagree](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/test/config-next/va.json#L44) with the primary VA for all validations. The remote VA instances use [`test/config-next/remoteva-a.json`](https://github.com/letsencrypt/boulder/blob/5c27eadb1db0605f380e41c8bd444a7f4ffe3c08/test/config-next/remoteva-a.json) and [`test/config-next/remoteva-b.json`](https://github.com/letsencrypt/boulder/blob/5c27eadb1db0605f380e41c8bd444a7f4ffe3c08/test/config-next/remoteva-b.json) as their config files. We require that almost all remote validation requests succeed; the exact number is controlled by the VA based on the thresholds required by MPIC. If the number of failing remote VAs exceeds that threshold, validation is terminated. If the number of successful remote VAs is high enough that it would be impossible for the outstanding remote VAs to exceed that threshold, validation immediately succeeds. There are some integration tests that test this end to end. The most relevant is probably [`test_http_multiva_threshold_fail`](https://github.com/letsencrypt/boulder/blob/ea231adc36746cce97f860e818c2cdf92f060543/test/v2_integration.py#L876-L908). It tests that a HTTP-01 challenge made to a webserver that only gives the correct key authorization to the primary VA and not the remotes will fail the multi-perspective validation. --- ### Profiling # Profiling Boulder components expose profiling endpoints on the port specified by their --debug-addr flag. An index of available endpoints can be found at /debug/pprof/ on each service. Additionally, if the environment variable $GOMEMLIMIT is set, Boulder components will automatically write a heap and goroutine dump when it's hit. Note that $GOMEMLIMIT also sets a soft memory limit for the runtime. See https://pkg.go.dev/runtime#hdr-Environment_Variables. The dump will happen at most once per hour. --- ### Redis # Redis We use Redis for storing rate limit data. The Boulder dev environment stands up two nodes. We use the Ring client in the github.com/redis/go-redis package to consistently hash our reads and writes across these two nodes. ## Debugging Our main tool for interacting with Redis is `redis-cli`. You probably don't have redis-cli on your host, so we'll run it in a Docker container. We also need to pass some specific arguments for TLS and authentication. There's a script that handles all that for you: `test/redis-cli.sh`. First, make sure your redis is running: ```shell docker compose up boulder ``` Then, in a different window, run the following to connect to `bredis_1`: ```shell ./test/redis-cli.sh -h 10.77.77.4 ``` Similarly, to connect to `bredis_2`: ```shell ./test/redis-cli.sh -h 10.77.77.5 ``` You can pass any IP address for the -h (host) parameter. The full list of IP addresses for Redis nodes is in `docker-compose.yml`. You can also pass other redis-cli commandline parameters. They'll get passed through. You may want to go a level deeper and communicate with a Redis node using the Redis protocol. Here's the command to do that (run from the Boulder root): ```shell openssl s_client -connect 10.77.77.4:4218 \ -CAfile test/certs/ipki/minica.pem \ -cert test/certs/ipki/localhost/cert.pem \ -key test/certs/ipki/localhost/key.pem ``` Then, first thing when you connect, run `AUTH `. You can get a list of usernames and passwords from test/redis-ratelimits.config. --- ### Release # Boulder Release Process A description and demonstration of the full process for tagging a normal weekly release and a hotfix release. Once a release is tagged, it will be generally deployed to [staging](https://letsencrypt.org/docs/staging-environment/) and then to [production](https://acme-v02.api.letsencrypt.org/) over the next few days. ## Goals 1. All development, including reverts and hotfixes needed to patch a broken release, happens on the `main` branch of this repository. Code is never deployed without being reviewed and merged here first, and code is never landed on a release branch that isn't landed on `main` first. 2. Doing a normal release requires approximately zero thought. It Just Works. 3. Doing a hotfix release differs as little as possible from the normal release process. ## Release Schedule Boulder developers make a new release at the beginning of each week, typically around 10am PST **Monday**. Operations deploys the new release to the [staging environment](https://letsencrypt.org/docs/staging-environment/) on **Tuesday**, typically by 2pm PST. If there have been no issues discovered with the release from its time in staging, then on **Thursday** the operations team deploys the release to the production environment. Holidays, unexpected bugs, and other resource constraints may affect the above schedule and result in staging or production updates being skipped. It should be considered a guideline for normal releases but not a strict contract. ## Release Structure As of 2025-06-30, releases are tagged with a tag of the form `v0.YYYYMMDD.N`, where the `YYYYMMDD` is the date that the initial release is cut (usually the Monday of the current week), and `N` is an integer indicating the hotfix number, starting at `0`. For example, a regular release might be `v0.20250707.0`, and the first hotfix for that release would be `v0.20250707.1`. Historically, releases were tagged with the form `release-YYYY-MM-DD[x]`, where `[x]` was an optional lowercase letter suffix for hotfixes. For example, the second hotfix release (i.e. third release overall) in the third week of January 2022 was [`release-2022-01-18b`](https://github.com/letsencrypt/boulder/releases/tag/release-2022-01-18b). All release tags are signed with a key associated with a Boulder developer. Tag signatures are automatically verified by GitHub using the public keys that developer has uploaded, and are additionally checked before being built and deployed to our staging and production environments. Note that, due to how Git works, in order for a tag to be signed it must also have a message; we set the tag message to just be a slightly more readable version of the tag name. ## Making a Release ### Prerequisites * You must have a GPG key with signing capability: * [Checking for existing GPG keys](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/checking-for-existing-gpg-keys) * If you don't have a GPG key with signing capability, create one: * [Generating a new local GPG key](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/generating-a-new-gpg-key) * [Generating a new Yubikey GPG key](https://support.yubico.com/hc/en-us/articles/360013790259-Using-Your-YubiKey-with-OpenPGP) * The signing GPG key must be added to your GitHub account: * [Adding a new GPG key to your GitHub account](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/adding-a-new-gpg-key-to-your-github-account) * `git` *may* need to be configured to call the correct GPG binary: * The default: `git config --global gpg.program gpg` is correct for most Linux platforms * On macOS and some Linux platforms: `git config --global gpg.program gpg2` is correct * `git` must be configured to use the correct GPG key: * [Telling Git about your GPG key](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/telling-git-about-your-signing-key) * Understand the [process for signing tags](https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/signing-tags) ### Regular Releases Simply create a signed tag. The `tools/release/tag` tool will automatically determine the correct tag name based on the current date. ```sh go run github.com/letsencrypt/boulder/tools/release/tag@main ``` This will print the newly-created tag and instructions on how to push it after you are satisfied that it is correct. Alternately you can run the command with the `-push` flag to push the resulting tag automatically. ### Hotfix Releases Sometimes it is necessary to create a new release which looks like a prior release but with one or more additional commits added. This is usually the case when we discover a critical bug in the currently-deployed version that needs to be fixed, but we don't want to include other changes that have already been merged to `main` since the currently-deployed release was tagged. In this situation, we create a new hotfix release branch starting at the point of the previous release tag. We then use the normal GitHub PR and code-review process to copy the necessary fix(es) from `main` (where they must already be merged) to the release branch. Finally we create a new release tag at the tip of the release branch instead of the tip of main. To create the new release branch, substitute the name of the release tag which you want to use as the starting point into this command: ```sh go run github.com/letsencrypt/boulder/tools/release/branch@main v0.YYYYMMDD.0 ``` This will create a release branch named `release-branch-v0.YYYYMMDD`. When all necessary PRs have been merged into that branch, create the new tag by substituting the branch name into this command: ```sh go run github.com/letsencrypt/boulder/tools/release/tag@main release-branch-v0.YYYYMMDD ``` ## Deploying Releases When doing a release, SRE's tooling will check that: 1. GitHub shows that tests have passed for the commit at the planned release tag. 2. The planned release tag is an ancestor of the current `main` on GitHub, or the planned release tag is equal to the head of a branch named `release-branch-XXX`, and all commits between `main` and the head of that branch are cherry-picks of commits which landed on `main` following the normal review process. These checks ensure that all deployed code has been properly reviewed and tested before reaching production environments. --- ### README # Boulder - An ACME CA [](https://github.com/letsencrypt/boulder/actions/workflows/boulder-ci.yml?query=branch%3Amain) This is an implementation of an ACME-based CA. The [ACME protocol](https://github.com/ietf-wg-acme/acme/) allows the CA to automatically verify that an applicant for a certificate actually controls an identifier, and allows subscribers to issue and revoke certificates for the identifiers they control. Boulder is the software that runs [Let's Encrypt](https://letsencrypt.org). ## Contents * [Overview](#overview) * [Setting up Boulder](#setting-up-boulder) * [Development](#development) * [Working with Certbot](#working-with-certbot) * [Working with another ACME Client](#working-with-another-acme-client) * [Production](#production) * [Contributing](#contributing) * [License](#license) ## Overview Boulder is divided into the following main components: 1. Web Front Ends (one per API version) 2. Registration Authority 3. Validation Authority 4. Certificate Authority 5. Storage Authority 6. Publisher 7. CRL Updater This component model lets us separate the function of the CA by security context. The Web Front End, Validation Authority, CRL Storer, and Publisher need access to the Internet, which puts them at greater risk of compromise. The Registration Authority can live without Internet connectivity, but still needs to talk to the Web Front End and Validation Authority. The Certificate Authority need only receive instructions from the Registration Authority. All components talk to the SA for storage, so most lines indicating SA RPCs are not shown here. ```text CA ---------> Publisher ^ | Subscriber -> WFE --> RA --> SA --> MariaDB | ^ Subscriber server <- VA <----+ | | Browser -----> S3 <----- CRL Storer/Updater ``` Internally, the logic of the system is based around five types of objects: accounts, authorizations, challenges, orders and certificates, mapping directly to the resources of the same name in ACME. Requests from ACME clients result in new objects and changes to objects. The Storage Authority maintains persistent copies of the current set of objects. Boulder uses gRPC for inter-component communication. For components that you want to be remote, it is necessary to instantiate a "client" and "server" for that component. The client implements the component's Go interface, while the server has the actual logic for the component. A high level overview for this communication model can be found in the [gRPC documentation](https://www.grpc.io/docs/). The full details of how the various ACME operations happen in Boulder are laid out in the [Boulder implementation details](./docs/acme-implementation_details.md). ## Setting up Boulder ### Development Boulder has a Dockerfile and uses Docker Compose to make it easy to install and set up all its dependencies. This is how the maintainers work on Boulder, and is our main recommended way to run it for development/experimentation. It is not suitable for use as a production environment. While we aim to make Boulder easy to setup ACME client developers may find [Pebble](https://github.com/letsencrypt/pebble), a miniature version of Boulder, to be better suited for continuous integration and quick experimentation. We recommend setting git's [fsckObjects setting](https://groups.google.com/forum/#!topic/binary-transparency/f-BI4o8HZW0/discussion) before getting a copy of Boulder to have better integrity guarantees for updates. Clone the boulder repository: ```shell git clone https://github.com/letsencrypt/boulder/ cd boulder ``` Additionally, make sure you have Docker Engine 1.13.0+ and Docker Compose 1.10.0+ installed. If you do not, you can follow Docker's [installation instructions](https://docs.docker.com/compose/install/). We recommend having **at least 2GB of RAM** available on your Docker host. In practice using less RAM may result in the MariaDB container failing in non-obvious ways. To run our standard battery of tests (lints, unit, integration): ```shell ./t.sh ``` To run all unit tests: ```shell ./t.sh -u ``` To run specific unit tests (example is of the ./va directory): ```shell ./t.sh -u -p ./va ``` To run all integration tests: ```shell ./t.sh -i ``` To run unit tests and integration tests with coverage: ```shell ./t.sh -ui -c --coverage-dir=./test/coverage/mytestrun ``` To run specific integration tests (example runs TestGenerateValidity and TestWFECORS): ```shell ./t.sh -i -f TestGenerateValidity/TestWFECORS ``` To do any of the above, but using the "config-next" configuration, which represents a likely future state (e.g. including new feature flags): ```shell ./tn.sh -your -options -here ``` To start Boulder in a Docker container, first run: ```shell docker compose run bsetup ``` this will write the necessary certificates into `test/certs/[.softhsm-tokens,ipki,webpki]`; You only need to run this once to create the certificates. If you need to remove all of the certificates and start over, you can remove the directories `./test/certs/.softhsm-tokens`, `./test/certs/ipki`, and `./test/certs/webpki` and re-run `docker compose run bsetup`. Then run: ```shell docker compose up ``` The configuration in docker-compose.yml mounts your boulder checkout at /boulder so you can edit code on your host and it will be immediately reflected inside the Docker containers run with `docker compose`. If you have problems with Docker, you may want to try [removing all containers and volumes](https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes). By default, Boulder uses a fake DNS resolver that resolves all hostnames to 127.0.0.1. This is suitable for running integration tests inside the Docker container. If you want Boulder to be able to communicate with a client running on your host instead, you should find your host's Docker IP with: ```shell ifconfig docker0 | grep "inet addr:" | cut -d: -f2 | awk '{ print $1}' ``` And edit docker-compose.yml to change the `FAKE_DNS` environment variable to match. This will cause Boulder's stubbed-out DNS resolver (`sd-test-srv`) to respond to all A queries with the address in `FAKE_DNS`. If you use a host-based firewall (e.g. `ufw` or `iptables`) make sure you allow connections from the Docker instance to your host on the required validation ports to your ACME client. Alternatively, you can override the docker-compose.yml default with an environmental variable using -e (replace 172.17.0.1 with the host IPv4 address found in the command above) ```shell docker compose run --use-aliases -e FAKE_DNS=172.17.0.1 --service-ports boulder ./start.py ``` Running tests without the `./test.sh` wrapper: Run unit tests locally, without docker (only works for some directories): ```shell go test ./issuance/... ``` Run all unit tests: ```shell docker compose run --use-aliases boulder go test -p 1 ./... ``` Run unit tests for a specific directory: ```shell docker compose run --use-aliases boulder go test ``` Run integration tests (omit `--filter ` to run all): ```shell docker compose run --use-aliases boulder python3 test/integration-test.py --chisel --gotest --filter ``` ### Working with Certbot Check out the Certbot client from https://github.com/certbot/certbot and follow their setup instructions. Once you've got the client set up, you'll probably want to run it against your local Boulder. There are a number of command line flags that are necessary to run the client against a local Boulder, and without root access. The simplest way to run the client locally is to use a convenient alias for certbot (`certbot_test`) with a custom `SERVER` environment variable: ```shell SERVER=http://localhost:4001/directory certbot_test certonly --standalone -d test.example.com ``` Your local Boulder instance uses a fake DNS resolver that returns 127.0.0.1 for any query, so you can use any value for the -d flag. To return an answer other than `127.0.0.1` change the Boulder `FAKE_DNS` environment variable to another IP address. ### Working with another ACME Client Once you have followed the Boulder development environment instructions and have started the containers you will find the ACME endpoints exposed to your host at the following URLs: * ACME v2, HTTP: `http://localhost:4001/directory` * ACME v2, HTTPS: `https://localhost:4431/directory` To access the HTTPS versions of the endpoints you will need to configure your ACME client software to use a CA truststore that contains the `test/certs/ipki/minica.pem` CA certificate. See [`test/certs/README.md`](https://github.com/letsencrypt/boulder/blob/main/test/certs/README.md) for more information. Your local Boulder instance uses a fake DNS resolver that returns 127.0.0.1 for any query, allowing you to issue certificates for any domain as if it resolved to your localhost. To return an answer other than `127.0.0.1` change the Boulder `FAKE_DNS` environment variable to another IP address. Most often you will want to configure `FAKE_DNS` to point to your host machine where you run an ACME client. ### Production Boulder is custom built for Let's Encrypt and is intended only to support the Web PKI and the CA/Browser forum's baseline requirements. In our experience often Boulder is not the right fit for organizations that are evaluating it for production usage. In most cases a centrally managed PKI that doesn't require domain-authorization with ACME is a better choice. For this environment we recommend evaluating a project other than Boulder. We offer a brief [deployment and implementation guide](https://github.com/letsencrypt/boulder/wiki/Deployment-&-Implementation-Guide) that describes some of the required work and security considerations involved in using Boulder in a production environment. As-is the docker based Boulder development environment is **not suitable for production usage**. It uses private key material that is publicly available, exposes debug ports and is brittle to component failure. While we are supportive of other organization's deploying Boulder in a production setting we prioritize support and development work that favors Let's Encrypt's mission. This means we may not be able to provide timely support or accept pull-requests that deviate significantly from our first line goals. If you've thoroughly evaluated the alternatives and Boulder is definitely the best fit we're happy to answer questions to the best of our ability. ## Contributing Please take a look at [CONTRIBUTING.md](https://github.com/letsencrypt/boulder/blob/main/docs/CONTRIBUTING.md) for our guidelines on submitting patches, code review process, code of conduct, and various other tips related to working on the codebase. ## Code of Conduct The code of conduct for everyone participating in this community in any capacity is available for reference [on the community forum](https://community.letsencrypt.org/guidelines). ## License This project is licensed under the Mozilla Public License 2.0, the full text of which can be found in the [LICENSE.txt](https://github.com/letsencrypt/boulder/blob/main/LICENSE.txt) file. ---