## 1. Project Overview & Quickstart (emissary-ingress/emissary) ## File: README.md Emissary-ingress ================ [![Version][badge-version-img]][badge-version-link] [![Image Size][badge-ghcr-size]][badge-ghcr-link] [![Image Pulls][badge-ghcr-pulls]][badge-ghcr-link] [![Join Slack][badge-slack-img]][badge-slack-link] [![Core Infrastructure Initiative: Best Practices][badge-cii-img]][badge-cii-link] [badge-version-img]: https://ghcr-badge.egpl.dev/emissary-ingress/emissary/latest_tag?trim=major&label=version [badge-version-link]: https://github.com/emissary-ingress/emissary/releases [badge-ghcr-size]: https://ghcr-badge.egpl.dev/emissary-ingress/emissary/size [badge-ghcr-pulls]: https://img.shields.io/badge/dynamic/json?url=https://ghcr-badge.elias.eu.org/api/emissary-ingress/emissary/emissary&query=downloadCount&logo=docker&label=ghcr%20pulls&color=2496ed [badge-ghcr-link]: https://github.com/emissary-ingress/emissary/packages [badge-slack-img]: https://img.shields.io/badge/slack-join-orange.svg [badge-slack-link]: https://communityinviter.com/apps/cloud-native/cncf [badge-cii-img]: https://bestpractices.coreinfrastructure.org/projects/1852/badge [badge-cii-link]: https://bestpractices.coreinfrastructure.org/projects/1852 [Emissary-Ingress](https://www.getambassador.io/docs/open-source) is an open-source Kubernetes-native API Gateway + Layer 7 load balancer + Kubernetes Ingress built on [Envoy Proxy]. Emissary-ingress is a CNCF incubation project (and was formerly known as Ambassador API Gateway). Emissary-ingress enables its users to: * Manage ingress traffic with [load balancing], support for multiple protocols ([gRPC and HTTP/2], [TCP], and [web sockets]), and Kubernetes integration * Manage changes to routing with an easy to use declarative policy engine and [self-service configuration], via Kubernetes [CRDs] or annotations * Secure microservices with [authentication], [rate limiting], and [TLS] * Ensure high availability with [sticky sessions], [rate limiting], and [circuit breaking] * Leverage observability with integrations with [Grafana], [Prometheus], and [Datadog], and comprehensive [metrics] support * Enable progressive delivery with [canary releases] * Connect service meshes including [Consul], [Linkerd], and [Istio] * [Knative serverless integration] See the full list of [features](https://emissary-ingress.dev/docs/4.0/about/features-and-benefits/) here. Branches ======== (If you are looking at this list on a branch other than `main`, it is likely to be wrong.) - [`main`](https://github.com/emissary-ingress/emissary/tree/main): main Emissary-ingress development work (:heavy_check_mark: active development) - [`master`](https://github.com/emissary-ingress/emissary/tree/master): **frozen** branch for Emissary-ingress 3.10 (:heavy_check_mark: frozen) - [`release/v2.5`](https://github.com/emissary-ingress/emissary/tree/release/v2.5): **frozen** branch for Emissary-ingress 2.5 (:heavy_check_mark: frozen) Building Emissary 4 =================== `gmake images` should make all the images you need using [goreleaser](https://goreleaser.com/). **Note that `.goreleaser.yaml` is a generated file**; do not edit it directly. It's this way so that we can use the open-source version of `goreleaser`. If you want to update the Envoy version, edit the `ENVOY_IMAGE` value in the `Makefile`. Architecture ============ Emissary is configured via Kubernetes CRDs, or via annotations on Kubernetes `Service`s. Internally, it uses the [Envoy Proxy] to actually handle routing data; externally, it relies on Kubernetes for scaling and resiliency. Getting Started =============== The fastest way to get started with Emissary is to follow the [Quickstart]. For other common questions, check out the [Emissary FAQ] or the full [Emissary documentation]. Community ========= Emissary-ingress is a CNCF Incubating project and welcomes any and all contributors. **At this point, Emissary's original parent company has stepped back from direct involvement in the project**, so the community is now more important than ever. As we've said at the past few KubeCons, Emissary needs you. The best way to join the community is to join the [CNCF Slack] and hop into the `#emissary-ingress` channel. Checking out [open issues] or [contributing documentation] are both great ways to get started contributing. You can also check out the project's governance documentation: - [`CODE_OF_CONDUCT.md`](Community/CODE_OF_CONDUCT.md): the community code of conduct - [`GOVERNANCE.md`](Community/GOVERNANCE.md): the governance structure - [`MAINTAINERS.md`](Community/MAINTAINERS.md): the list of maintainers - [`MEETING_SCHEDULE.md`](Community/MEETING_SCHEDULE.md): the schedule of community meetings [authentication]: https://www.emissary-ingress.dev/docs/4.0/topics/running/services/auth-service/ [canary releases]: https://www.emissary-ingress.dev/docs/4.0/topics/using/canary/ [circuit breaking]: https://www.emissary-ingress.dev/docs/4.0/topics/using/circuit-breakers/ [CNCF Slack]: https://communityinviter.com/apps/cloud-native/cncf [Consul]: https://www.emissary-ingress.dev/docs/4.0/howtos/consul/ [CRDs]: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ [Datadog]: https://www.emissary-ingress.dev/docs/4.0/topics/running/statistics/#datadog [Emissary documentation]: https://www.emissary-ingress.dev/docs/4.0/ [Emissary FAQ]: https://www.emissary-ingress.dev/docs/4.0/about/faq/ [Envoy Proxy]: https://www.envoyproxy.io [Grafana]: https://www.emissary-ingress.dev/docs/4.0/topics/running/statistics/#grafana [gRPC and HTTP/2]: https://www.emissary-ingress.dev/docs/4.0/howtos/grpc/ [Istio]: https://www.emissary-ingress.dev/docs/4.0/howtos/istio/ [Knative serverless integration]: https://www.emissary-ingress.dev/docs/4.0/howtos/knative/ [Linkerd]: https://www.emissary-ingress.dev/docs/4.0/howtos/linkerd2/ [load balancing]: https://www.emissary-ingress.dev/docs/4.0/topics/running/load-balancer/ [metrics]: https://www.emissary-ingress.dev/docs/4.0/topics/running/statistics/ [Prometheus]: https://www.emissary-ingress.dev/docs/4.0/topics/running/statistics/#prometheus [Quickstart]: https://emissary-ingress.dev/docs/4.0/quick-start/ [rate limiting]: https://www.emissary-ingress.dev/docs/4.0/topics/running/services/rate-limit-service/ [self-service configuration]: https://www.emissary-ingress.dev/docs/4.0/topics/using/mappings/ [sticky sessions]: https://www.emissary-ingress.dev/docs/4.0/topics/running/load-balancer/#sticky-sessions--session-affinity [TCP]: https://www.emissary-ingress.dev/docs/4.0/topics/using/tcpmappings/ [TLS]: https://www.emissary-ingress.dev/docs/4.0/howtos/tls-termination/ [web sockets]: https://www.emissary-ingress.dev/docs/4.0/topics/using/tcpmappings/ --- ## File: build-aux/docs/conventions.md # Conventions for use in build-aux Each `.mk` snippet starts with a reference header-comment of the format: ```make # Copyright statement. # # A sentence or two introducing what this file does. # ## Section heading ## # - type: item # - type: item ## Section heading ## # - item # - item # # High-level prose documentation. ``` Always include (at a minimum) these 4 sections, even if their content is `(none)`: - `## Eager inputs ##` (mostly variables) Eager-evaluated inputs that need to be defined *before* loading the snippet - `## Lazy inputs ##` (mostly variables) Lazy-evaluated inputs can be defined before *or* after loading the snippet - `## Outputs ##` (targets, variables, et c.) - `## common.mk targets ##` If there are which targets from snippets other than `common.mk` that it hooks in to, a section for each of those snippets should exist too. The most common reason for an input to be eager is if it is listed as a dependency of a target defined in the `.mk` file. Bullet points under "Eager inputs", "Lazy inputs", or "Outputs" should be of the format "Type: thing [extra info]", optionally with the ":" aligned between rows. Types currently used are: - `Target(s)` (only for use as an output): Indicates that the snippet defines rules to create the specified file(s). Prefer to list each target separately; only use the plural `Targets` when listing an expression that govers a large range of actual files. - `.PHONY Target(s)` (only for use as an output): Like `Target`, but is declared as `.PHONY`; a file with that name is never actually created. - `File` (only for use as an input): Indicates that this file is parsed by the Makefile snippet. Note that it should not be a file described by a Target, since it needs to be there at Makefile-parse-time. - `Variable`: The meaning is obvious. * For inputs: If there is a default value, the listing should include `?= value` for a possibly pseudo-code or comment `value` documenting what the default value is. * For outputs: The listing should include `= value` or `?= value` for a possibly pseudo-code or comment `value` documenting what the variable is set to contain. The `=` should be `?=` as appropriate, to document whether it can be overridden by the caller/user. * If a `?=` variable is listed in "Outputs", consider also listing it in "inputs"; if an input gets a default value set, consider also listing it in "Outputs". - `NAME` in `go-mod.mk`. It isn't listed as an an input, because it isn't one "in spirit". - "Executables" (below) are set with `?=`, but don't list them as "inputs". - `Function`: (only for use as an output) This is a special case of a Variable that defines a function to be called with `$(call funcname,args)`. Should not be listed as an input; if you depend on a function, just go ahead and include the snippet that defines it. - `Executable` (only for use as an output): An "Executable" output is a special-case of both a "Target" output and a "Variable" input/output. An "Executable" output has the following attributes: * It is exposed as a variable that stores an absolute path to an executable file (this makes it safe to list as a dependency). * That variable can be overridden using an environment variable (that is: it is set with `?=`). * It is NOT guaranteed to already exist (you may need to depend on it), but a rule to create it is guaranteed be there it it doesn't already exist. * Overriding it using an environment variable will not affect the rule to create the standard version, or cleanup rules to remove it; those targets do not obey the environment variable. --- ## File: build-aux/docs/intro.md # Datawire build-aux Introduction build-aux is a collection of Makefile `.mk` snippets (and associated utilities). Each of the `.mk` snippets is written to be as independent from the others as possible, but integrate well by following common conventions. You may pick-and-choose which to include simply by writing `include build-aux/NAME.mk` in your `Makefile`. Each `.mk` snippet starts with a reference header-comment identifying: - any eager-evaluated inputs (mostly variables) - any lazy-evaluated inputs (mostly variables) - any outputs (targets, variables) - which targets from other snippets it hooks in to (mostly hooking in to `common.mk` targets) Eager inputs need to be defined *before* loading the snippet; lazy inputs can be defined later. See [`conventions.mk`][./conventions.md] for more information on the reference header-comment. For the most part, you don't need to worry about dependencies between `.mk` files; each file will automatically `include` the others it depends on. However, if you would like to use an output from one snippet as an eager input to another, then you do need to worry about include order; if you would like to use `kubernaut-ui.mk` to set `KUBECONFIG` for `teleproxy.mk`, then you will need to make sure you include `kubernaut-ui.mk` *before* you include `teleproxy.mk`. You don't need to worry about including a file twice; this is safe, as they all have C-header-style include guards. ## `common.mk` Most (but not all) of the snippets include `common.mk`, which 1. Configures Make to use sane settings (since the default settings are set for historical compatibility, not the "right thing"). 2. Defines several `$(call …)`-able helper functions, like `joinlist`, `path.trimprefix`, or `quote.shell`. See [`common.mk`](../common.mk) itself for the full list, and documentation on how to use them. 3. Declares several utility variables: - `GOOS`, `GOARCH`: Operating system name, and CPU architecture. `GO` is included in the variable names to indicate that they use the same names as `go env` (as opposed to using `uname -s`/`uname -m` names, or something else). - `NL`: a single newline character, since that's hard to type in Make - `SPACE`: a single space character, since that's hard to type in Make 4. Declares several high-level common targets, like `make build`, `make clean`, or `make check`. See [`common.mk`](../common.mk) itself for the full list. By defining a list of common user-facing targets, the rest of the snippets can hook in to that and provide a uniform interface. With the exception of `make check` (which is special, see below), your `Makefile` can extend any of these by adding dependencies to the target name. For example, to have `make build` build the `foo` file, you could write: build: foo or to have `make lint` run the `flake8` Python linter, you could write lint: flake8 flake8: flake8 mypackage/ .PHONY: flake8 or lint: flake8 mypackage/ The former has the advantage that it allows you to add multiple hooks on to `lint` (so it's the only method that build-aux snippets use), and also allows flake8 to run in parallel with other linters set up by build-aux snippets. `make check` is special in how it works. Instead of writing check: my-test # WRONG you write test-suite.tap: my-test.tap # CORRECT See [`testing.md`](./testing.md) for information about writing the recipe for `my-test.tap`. Because of `make check`'s use of `test-suite.tap`, `common.mk` also goes ahead and has `make clean` run `rm -f test-suite.tap`. With the exceptions of `make check` and `make clean`, `common.mk` only provides empty definitions; it is up to your `Makefile`, or other `.mk` snippets to make these rules actually do something. ## `help.mk` The snippet `help.mk` adds a `make help` target, that display information about your project (customizable by setting the `help.body` variable from your `Makefile`), and a table of all of the `make FOO` targets that Make knows about. Any targets you write in your `Makefile` that you think should be visible can be added to this listing by writing a magic comment: frob: ## Frobnicate the splorks See [`help.mk`](../help.mk) itself for full documentation. --- ## File: build-aux/docs/testing.md # Datawire build-aux Test Harness `common.mk` includes a built-in test harness that allows you to plug in your own tests, and have them aggregated and summarized, like: $ make check ... PASS: go-test 5 - github.com/datawire/apro/cmd/amb-sidecar.TestAppNoToken PASS: go-test 6 - github.com/datawire/apro/cmd/amb-sidecar.TestAppBadToken PASS: go-test 7 - github.com/datawire/apro/cmd/amb-sidecar.TestAppBadCookie PASS: go-test 8 - github.com/datawire/apro/cmd/amb-sidecar.TestAppCallback PASS: go-test 9 - github.com/datawire/apro/cmd/amb-sidecar.TestAppCallbackNoCode PASS: tests/local/apictl.tap.gen 1 - check_version ============================================================================ test-suite summary ============================================================================ # TOTAL: 10 # SKIP: 0 # PASS: 10 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ make[1]: Leaving directory '/home/lukeshu/src/apro' (If you were viewing that in a terminal, it would also be pretty and colorized.) Each test-case can have one of 6 results: - `pass`, `fail`, `skip`: You can guess what these mean. - `xfail` ("expected fail"): The test failed, but it was expected to fail. Perhaps you wrote a test for a feature for before implementing the feature. Perhaps you wrote a test that captures a bug report, but haven't written the fix yet. Makes Test-Driven-Development possible. - `xpass` ("unexpected pass"): The test passed, but it was expected to xfail. This either means you did a poor job implementing the test, and it doesn't really check what you think it checks, or it means you've implemented the fix, but forgot to replace `xfail` with `fail` in the test. - `error`: It isn't that the test decided that there's bug in the code-under-test, it's that the test itself encountered an error. You can use any test framework or language you like, as long as it can emit [TAP, the Test Anything Protocol][TAP] (or something that can be translated to TAP). Both TAP version 12 and version 13 are supported. `not ok # TODO` is "xfail", while , `ok # TODO` is "xpass" > Side-Note: pytest-tap emits `ok # TODO` for both xfail and xpass, > which is wrong, and I consider to be a bug in pytest-tap. ## Built-in test runners By default, `common.mk` knows how to run test cases of 2 types (but you can easily add more): - `*.test` GNU Automake-compatible standalone test cases. One file is one test case. `FOO.test` must be an executable file. It is run with no arguments, stdout and stderr are ignored (but logged to `FOO.log`); it is the exit code that determines the test result: * 0 => pass * 77 => skip * anything else => fail It is not possible to xfail or xpass with this type of test. - `*.tap.gen` TAP-emitting test cases. You may have multiple test cases per file. `FOO.tap.gen` must be an executable file. It is run with no arguments; stdout and stderr are merged, and are taken to be a TAP stream (both v12 and v13 are supported). The exit code is ignored. `common.mk` does *not* scan your source directory for tests (but `go-*.mk` will scan for `go test` tests). In your `Makefile` You must explicitly tell it about any `.test` or `.tap.gen` files that you would like it to include in `make check`. For example, if you would like it to include any `.test` or `.tap.gen` files in the `./tests/` directory, you could write: test-suite.tap: $(patsubst %.test,%.tap,$(wildcard tests/*.test)) test-suite.tap: $(patsubst %.tap.gen,%.tap,$(wildcard tests/*.tap.gen)) ## Adding your own test runners To add a new test runner, you just need a command that emits TAP: `tee` it to a `.tap` file, and pipe that to `$(tools/tap-driver) stream -n TEST_GROUP_NAME` to pretty-print the results as they happen: # Tell Make how to run the test command, and stream the results to # `$(tools/tap-driver) stream` to pretty-print the results as they happen. my-test.tap: my-test.input $(tools/tap-driver) FORCE @SOME_COMMAND_THAT_EMITS_TAP 2>&1 | tee $@ | $(tools/tap-driver) stream -n my-test # Tell Make to include 'my-test' in `make check` test-suite.tap: my-test.tap For example, to use [BATS (Bash Automated Testing System)][BATS], you would write: %.tap: %.bats $(tools/tap-driver) FORCE @bats --tap $< | tee $@ | $(tools/tap-driver) stream -n $< # Automatically include `./tests/*.bats` test-suite.tap: $(patsubst %.bats,%.tap,$(wildcard tests/*.bats)) If your test framework of choice doesn't support TAP output, you can pipe it to a helper program that can translate it. For example, `go test` doesn't support TAP output, but `go test -json` output is parsable, so we pipe that to [gotest2tap][gotest2tap], which translates it to TAP. If you set `SHELL = sh -o pipefail` in your `Makefile` (the pros and cons of which I won't comment on here), you should be sure that if your test-runner indicates success or failure with an exit code, that you ignore that exit code: %.tap: %.bats $(tools/tap-driver) FORCE @{ bats --tap $< || true; } | tee $@ | $(tools/tap-driver) stream -n $< ## Adding dependencies of tests It is assumed that *all* tests depend on `make build`. To add a dependency shared by all tests, to declare a dependency that all tests should depend on, declare it as a dependency of `check` itself. For example, `common.mk` says: check: lint build As another example, the `Makefile` for Ambassador Pro says: check: $(if $(HAVE_DOCKER),deploy proxy) To declare a dependency for an individual test is a little trickier, because you must keep in mind what type of test it is. For `.tap.gen` tests, you must declare it both on the `.tap` file and (if the dependency is not a `.tap`) on `check` itself: test-suite.tap: tests/cluster/oauth-e2e.tap check tests/cluster/oauth-e2e.tap: tests/cluster/oauth-e2e/node_modules If that were a `.test` test, you would need to declare it on the `.log` file instead of `.tap`: test-suite.tap: tests/cluster/oauth-e2e.tap check tests/cluster/oauth-e2e.log: tests/cluster/oauth-e2e/node_modules If you need one test to depend on another test, write the dependency using the `.tap` suffix (not `.log`). You do not need to write the depenency for `check`, since it will already depend on the `.tap` through `test-suite.tap`: test-suite.tap: foo.tap bar.tap foo.log: bar.tap [TAP]: https://testanything.org [BATS]: https://github.com/sstephenson/bats [gotest2tap]: ../bin-go/gotest2tap/ --- ## File: build-aux/README.md # Datawire build-aux This is a collection of Makefile snippets (and associated utilities) for use in Datawire projects. It has the following downsides: 1. It does not support out-of-tree builds. 2. It has no notion of nesting. You cannot `cd` to a sub-directory, run `make`, and have it just build the stuff in that directory. 3. It has no notion of nesting. If you want per-directory build descriptions, you'll have to build that functionality yourself, or offload it to a separate build-system, like `go` or `setup.py`. 4. Most of the `.mk` snippets have a hard dependency on the `go` program being in `PATH`, even if none of the sources are Go. If any of those are a bummer, but you still want the "Make snippets in a `./build-aux/` folder" concept, consider [Autothing](https://git.lukeshu.com/autothing/) (which has the downsides that it requires GNU Make 3.82 or above, and is slow). At Datawire, those are good trade-offs, since: - We're mostly a Python and Go shop (we don't care about #2 or #3, and only care about #4 for Python-only projects). - I've never heard anyone here even mention out-of-tree builds (#1). - We need to support macOS (which still ships GNU Make 3.81, as it's the last GPLv2 version) (so Autothing is ruled out). ## How to use Add `build-aux.git` as `build-aux/` in the git repository of the project that you want to use this from. I recommend that you do this using `git subtree`, but `git submodule` is fine too. Then, in your Makefile, write `include build-aux/FOO.mk` for each common bit of functionality that you want to make use of. ### Using `git-subtree` to manage `./build-aux/` - Start using build-aux: $ git subtree add --squash --prefix=build-aux git@github.com:datawire/build-aux.git master - Update to latest build-aux: $ ./build-aux/build-aux-pull - Push "vendored" changes upstream to build-aux.git: $ ./build-aux/build-aux-push ### Documentation - Each `.mk` snippet contains a reference-quality header comment identifying - any "eager" inputs (mostly variables); these must be defined *before* including the `.mk` file. - any "lazy" inputs (mostly variables); these may be defined before or after including the `.mk` file. - any outputs (targets, variables) - which targets from other snippets it hooks in to (mostly hooking in to `common.mk` targets) - [`docs/intro.md`](./docs/intro.md) is an introduction to big-picture ideas in `*.mk` snippets. - [`docs/golang.md`](./docs/golang.md) discusses support for building software written in Go. - [`docs/testing.md`](./docs/testing.md) discusses adding tests to `make check`. - [`HACKING.md`](./HACKING.md) has guidelines for contributing to `build-aux.git`. --- ## File: cmd/kubestatus/README.md The kubestatus program provides a command line interface for updating the status of a kubernetes resource. The kubestatus program operates on a set of resources. This set is defined by the combination of the passed on the command line, and the --field-selector and --label-selector arguments, e.g.: ``` kubestatus service -f metadata.name=foo ``` Note that by default kubestatus will just list the statuses you have selected. If you want to update the status, you need to supply it with a new status via the -u or --update flag: ``` kubestatus service -f metadata.name=foo -u updated-status.json ``` The status must be supplied in a valid json format in the referenced file. This status also needs to conform to whatever the kubernetes schema is for the given resource. You can use kubectl explain to figure out what that is, e.g.: ``` kubectl explain service.status ... kubectl explain service.status.loadBalancer ... ``` --- ## File: pkg/ambex/README.md Ambex: Ambassador Experimental^H^H^H^H^H^H^H^H^H^H^H^H ADS service ============================= Ambassador prior to v0.50.0 worked by writing out Envoy v1 JSON configuration files, then triggering a hot restart of Envoy. This works, but it has some unpleasant limitations: - Restarts take awhile, and as a result you can't change the configuration very quickly. - Restarts can drop connections (cf Envoy #2776; more on this later). - Envoy itself is deprecating the v1 configuration, and will only support v2 in a bit. To get around these limitations, and generally go for a better experience all 'round, we want to switch to the so-called `xDS` model, in which Envoy's configuration is supplied by its various "*D*iscovery *S*ervices": e.g. the `CDS` is the Cluster Discovery Service; the `EDS` is the Endpoint Discovery Service. For Ambassador, the Aggregated Discovery Service or `ADS` is the one we want to use -- basically, it brings the other services together under one aegis and lets you just tell Envoy "get everything dynamically." However, the whole `ADS` thing is a bit of a pain: - Envoy makes a bidirectional gRPC stream to the `ADS` server. - The `ADS` then makes gRPC calls _to the Envoy_ to feed the Envoy configuration elements, but: - The `ADS` has to carefully order things such that the configuration elements match what Envoy expects for consistency. Rather than do all that logic by hand, we'll use the Envoy `go-control-plane`[^1] for the heavy lifting. This is also something of a pain, given that it's not well documented, but here's the deal: - The root of the world is a `SnapshotCache`: - `import github.com/datawire/ambassador/pkg/envoy-control-plane/cache`, then refer to `cache.SnapshotCache`. - A collection of internally consistent configuration objects is a `Snapshot` (`cache.Snapshot`). - `Snapshot`s are collected in the `SnapshotCache`. - A given `SnapshotCache` can hold configurations for multiple Envoys, identified by the Envoy `nodeID`, which must be configured for the Envoy. - The `SnapshotCache` can only hold `go-control-plane` configuration objects, so you have to build these up to hand to the `SnapshotCache`. - The gRPC stuff is handled by a `Server`: - `import github.com/datawire/ambassador/pkg/envoy-control-plane/server`, then refer to `server.Server`. - Our `runManagementServer` function (largely ripped off from the `go-control-plane` tests) gets this running. It takes a `SnapshotCache` (cleverly called `config` for no reason I (Flynn) understand) and a standard Go `gRPCServer` as arguments. - _ALL_ the gRPC madness is handled by the `Server`, with the assistance of the methods in its `callback` object. - Once the `Server` is running, Envoy can open a gRPC stream to it. - On connection, Envoy will get handed the most recent `Snapshot` that the `Server`'s `SnapshotCache` knows about. - Whenever a newer `Snapshot` is added to the `SnapshotCache`, that `Snapshot` will get sent to the Envoy. - We manage the `SnapshotCache` by loading Envoy configuration files on disk: - it ignores files that start with a `.` (hidden files) - it interprets `*.json` files as [JSON-encoded protobuf](https://developers.google.com/protocol-buffers/docs/proto3#json) - it interprets `*.pb` files as [text-encoded protobuf](https://pkg.go.dev/google.golang.org/protobuf/encoding/prototext) - all other files are ignored As for when it loads those files: - By default when we get a SIGHUP we reload the configuration. - When passed the `--watch` argument we reload whenever any file in the directory changes. Be careful about updating files atomically if you use this! [^1]: The Envoy `go-control-plane` usually refers to `github.com/envoyproxy/go-control-plane`, but we've "forked" it as `github.com/datawire/ambassador/pkg/envoy-control-plane` in order to build it against the protobufs for our patched Envoy. Running Ambex ============= You'll need the Go toolchain, and will want to have a functioning `envoy`. Then, you can run the ambex CLI using `busyambassador`: ```shell go run github.com/datawire/ambassador/cmd/busyambassador ambex ARGs... ``` If you're on a platform other than GNU/Linux, in order to have a functioning `envoy`, you may want to run all of this in the builder shell: `make shell`. Try it out ---------- You'll want to run both `ambex` and an instance of `envoy` with a boostrap config pointing at that `ambex. 1. First, start the `ambex`: ```shell go run github.com/datawire/ambassador/cmd/busyambassador ambex ./example/ambex/ ``` or ```shell go run github.com/datawire/ambassador/cmd/busyambassador ambex --watch ./example/ambex/ ``` 2. Second, in another shell, start the `envoy`: ```shell envoy -l debug -c ./example/envoy/bootstrap-ads.yaml ``` You should now be able to run some curls in (yet) another shell: ```console $ curl localhost:8080/hello Hello ADS!!! $ curl localhost:8080/get { "args": {}, "headers": { "Accept": "*/*", "Connection": "close", "Host": "httpbin.org", "User-Agent": "curl/7.54.0", "X-Envoy-Expected-Rq-Timeout-Ms": "15000" }, "origin": "72.74.69.156", "url": "http://httpbin.org/get" } ``` Edit and/or add more files to the `./example/ambex/` directory in order to play with more configurations and see them reload _instantaneously_ (if you used the `--watch` flag), or when-triggered (if you didn't use the `--watch` flag; trigger a relead by signaling the process with `killall -HUP ambex`). Clean up -------- Kill Ambex and Envoy with `Ctrl-C`. --- ## File: python/kat/README.md # kat Kubernetes Acceptance Testing tool This is a testing tool that was originally developed for testing of ambassador, but could potentially be useful for any sort of application built on top of kubernetes. This tool streamlines the following process: 1. apply a bunch of yaml 2. wait for services to spin up 3. run a bunch of queries against the spun up APIs 4. perform a bunch of assertions on those query results We originally built these sort of tests in a unit-testing style framework where each test would proceed through this sequence on its own. This turns out to be very slow since steps two and three are highly synchronous/chatty. This test framework runs these sorts of tests several orders of magnitude faster by a) applying the yaml for all tests in a single big batch up front, and b) performing all the queries in step 3 asynchronously using a high performance client. Please see the [tutorial](docs/tutorial.rst) to get started. --- ## File: python/README.md Ambassador Developer's Guide ============================ Concepts -------- Ambassador sits between users and an Envoy. The primary job that an Ambassador does is to take an _Ambassador configuration_ and, from that, generate an _Envoy configuration_. This generation happens using an _intermediate representation (IR)_ to manage all the internal logic that Ambassador needs: ```asciiart Ambassador config => IR => Envoy config ``` ### Ambassador Components and Ports Ambassador comprises several different components: | Component | Type | Function. | | :------------------- | :----- | :-------------------------------------------------------------------------------------------------------- | | `diagd` | Python | Increasingly-misnamed core of the system; manages changing configurations and provides the diagnostics UI | | `entrypoint/ambex` | Go | Envoy `go-control-plane` implementation; supplies Envoy with current configuration | | `entrypoint/watcher` | Go | Service/secret discovery; interface to Kubernetes and Consul | | `envoy` | C++ | The actual proxy process | `entrypoint`, `diagd`, and `envoy` are all long-running daemons. If any of them exit, the pod as a whole will exit. Ambassador uses several TCP ports while running. All but one of them are in the range 8000-8499, and any future assignments for Ambassador ports should come from this range. | Port | Process | Function | | :--- | :------------------------------------ | :----------------------------------------------------------------------------------------------------------------- | | 8001 | `envoy` | Internal stats, logging, etc.; not exposed outside pod | | 8002 | `entrypoint/watcher` | Internal `watt` snapshot access; not exposed outside pod | | 8003 | `entrypoint/ambex` | Internal `ambex` snapshot access; not exposed outside pod | | 8004 | `diagd` | Internal `diagd` access when `AMBASSADOR_FAST_RECONFIGURE` is set; not exposed outside pod | | 8005 | `entrypoint/external_snapshot_server` | Exposes configuration snapshots for integration with other tools | | 8006 | `envoy` | Default ready listener port | | 8080 | `envoy` | Default HTTP service port | | 8443 | `envoy` | Default HTTPS service port | | 8877 | `diagd` | Direct access to diagnostics UI; provided by `busyambassador entrypoint` when `AMBASSADOR_FAST_RECONFIGURE` is set | ### The Ambassador Configuration An Ambassador configuration is a collection of _Ambassador configuration resources_, which are represented by subclasses of `ambassador.config.ACResource`. The configuration as a whole is represented by an `ambassador.Config` object. `ambassador.Config` does not know how to parse YAML, interact with Kubernetes, or look at the filesystem. Instead, its consumer must construct a consistent list of fully-instantiated `ACResource` objects and tell `ambassador.Config` to load these resources: ```python aconf: Config = Config() aconf.load_all(resources: List[ACResource]) ``` `load_all` is only meant to be called once, with a complete set of all the resources comprising the Ambassador configuration. To change the configuration, instantiate a new `ambassador.Config`. The `ambassador.Config` class is relatively unsophisticated: for the most part, all it does is to save the resources handed to it in a way that preserves the type of the resources, and permits consumers to query the `Config` for various kinds of resources: * `get_config(self, key: str) -> Any` Fetches all the configuration information the `Config` has for the type tagged as `key`, e.g. `aconf.get_config("mappings")` will fetch all the `Mapping`s that the `Config` has stored. Current keys include: - `auth_configs`: information from `AuthService` definitions - `mappings`: information from `Mapping`s - `modules`: information from `Module`s, including the `Ambassador` and `TLS` `Module`s - `ratelimit_configs`: information about `RateLimitService`s - `tracing_configs`: information about `TracingService`s * `get_module(self, module_name: str) -> Optional[ACResource]` Fetches the `Module` with a given name (e.g. `aconf.get_module("ambassador")` will fetch information about the `Ambassador` `Module`). Returns `None` if no `Module` with the given name is found. * `module_lookup(self, module_name: str, key: str, default: Any=None) -> Any` Looks up a specific `key` in a specific `Module`. e.g. ```aconf.module_lookup('ambassador', 'service_port', 8080)``` will look up the `service_port` from the `Ambassador` `Module`; if no `service_port` is defined, the default value will be 8080. * `dump(self, output=sys.stdout) -> None` Dumps the entire `Config` object to the given `output`, for debugging. Once a `Config` has loaded resources, it can be used to create an `IR`. ### The IR The Intermediate Representation is where most of the logic of Ambassador lives. The IR as a whole is represented by an `ambassador.IR` object, which contains many objects descended from `ambassador.ir.IRResource`. The logic needed to synthesize the IR from an `ambassador.Config` is mostly contained in these `IRResource` subclasses; relatively little is in the `ambassador.IR` class itself. An `ambassador.IR` can only be instantiated from an `ambassador.Config`: ``` ir = IR(aconf: Config) ``` The general rule in the `IR` is that everything interesting is stored in objects descended from `IRResource`. There are two cases: #### Only One `IRResource` Only one of some `IRResource` objects can exist within a given IR, no matter how many distinct `ACResource`s they collect information from. A good example here is the `IRAmbassador` object, which contains global configuration information pooled from the `Ambassador` `Module` as well any `TLS` `Module`: only one of these will ever exist. For these single-instance objects: 1. First the resource is instantiated with the `ambassador.IR` and `ambassador.Config` objects as parameters. Whatever information is needed to initialize the `IRResource`, its `__init__` method should pull directly from the `Config` and/or `IR` objects. 2. Next, the resource's `setup` method is called, again with the `IR` and `Config` as parameters. `setup` can perform further initialization, consistency checks, etc., and must return a boolean: `True` will allow the resource to become active, `False` will mean the resource isn't needed after all. (For example, the `IRAuth` object will return `False` if it doesn't find any configured authentication services.) The default `setup` method simply returns `True`, so any `IRResource` subclass that doesn't override it will always be active. 3. If the resource's `setup` returns `True`, the `IR` will save it as an active resource. 4. After all resources have been probed, the `IR` will walk the list of active resources and call the `add_mappings` method for each active resource. `add_mappings` should add any mappings or clusters needed by the resource. (For example, the `IRAmbassador` object adds mappings (and thus clusters) to handle probes and diagnostics.) The default `add_mappings` method does nothing, so no mappings or clusters will be added for `IRResource` subclasses that don't override `add_mappings`. For single-instance objects, the distinction between `__init__` and `setup` is rarely relevant. It's OK to do everything in `__init__` and allow the default `setup` to always return `True`; likewise it's OK to just save incoming data in `__init__` and have all logic in `setup`. #### Multiple `IRResource`s There can be multiple instances of some `IRResource` objects: for example, a single `Config` can contain many `Mapping`s and multiple `Listener`s. For these resources, we use a `Factory` class, which must implement the `load_all` classmethod and may implement the `finalize` classmethod, both of which receive the `IR` and `Config` as parameters. 1. `load_all` is called early in the instantiation process, and is responsible for creating as many individual resources as needed and saving them (usually in the `IR` itself). 2. `finalize` is called only after all the factories have had `load_all` called and all other single-instance resources have had `add_mappings` called, and is responsible for any normalization or other initialization that depends on global knowledge (for example, `MappingFactory.finalize` does the normalization of weights across `Mapping` groups). #### The Full Sequence The full IR instantiation sequence can be found in `ambassador.ir.IR.__init__`: - TLS defaults - `IRAmbassadorTLS` (TLS contexts, etc.) - `IRAmbassador` (global Ambassador config) - `IRAuth` (extauth services) - `IRRateLimit` (rate limiting) - `ListenerFactory` (listeners -- creates `IRListener` objects) - `MappingFactory` (mappings -- creates `IRMappingGroup`, `IRMapping`, and `IRCluster` objects) - Cluster naming normalization #### Helpers - `IR.add_mapping` adds or looks up an `IRMappingGroup` with associated `IRMapping`s and `IRCluster`s. - `IR.add_cluster` adds only an `IRCluster`. - `IR.has_cluster` and `IR.get_cluster` do `IRCluster` lookups. - `IR.dump` dumps the entire IR for debugging. ### The Envoy V1 Configuration Finally, an Envoy V1 configuration is represented by `ambassador.envoy.V1Config`. A V1 config is built from an IR, again with most of the logic to do so contained in classes that mirror the structure of the V1 config. As we support later Envoy configuration versions, they will have their own classes. The root of the Envoy V1 configuration is `ambassador.envoy.v1.V1Config`. Overall Life Cycle ------------------ 0. Construct a collection of `ACResource` objects (from disk, from K8s, whatever). This will mostly involve `ACResource.from_dict` or `ACResource.from_yaml`. 1. Instantiate an `ambassador.Config`. Use its `load_all()` method to load up the collection of `ACResource` objects. 2. Instantiate an `ambassador.IR` from the `ambassador.Config`. 3. Instantiate an `ambassador.envoy.V1Config` from the `ambassador.IR`. Developing in Ambassador ------------------------ In all cases, understanding the class hierarchy and the lifecycle around the IR will be important. Both of these are discussed below. ### Adding Features Adding a feature will start with the Ambassador configuration resources: - The simple case will involve modifying a schema file and possibly modifying an `ACResource` class. - The less simple case will involve adding a new schema and a new `ACResource` subclass. - Unless the new class needs complex logic (it shouldn't), you can just let the existing `Config` code save your new resource. - If it does need complex logic, you'll need to add a handler method to `Config`. Once the Ambassador config is dealt with, you'll add or modify the IR to cope. Most of what you do here should involve the `IRResource` subclasses, _not_ the `IR` class itself (although if you're adding something completely new, you'll need to add code to `IR.__init__` to call your new class). Once the IR is dealt with, you'll need to add or modify the `V1Config` to cope with the IR changes. ### Handling Bugs The trick with bugs will be figuring out what you need to change. In general, work from V1Config to IR to Ambassador config -- the closer to Envoy that the fix can go, the simpler it will probably be. - `ambassador.Config` and `ambassador.IR` both have `dump` methods that are invaluable for studying their contents. - You can also attach debuggers and look at objects. Most of the things you're working with are subclasses of `dict`, so many introspection tools are simple. Class Hierarchy ---------------- * `IR` * `IRResource` * `IRAdmin` * `IRAmbassador` * `IRCluster` * `IRFilter` * `IRAuth` * `IRListener` * `IRMapping` * `IRMappingGroup` * `IRRateLimit` * `IRAmbassadorTLS` * `IREnvoyTLS` * `envoy` * `V1Config` * `V1Admin` * `V1Cluster` * `V1ClusterManager` * `V1Listener` (This diagram is mostly about the way the classes are used, rather than strictly reflecting implementation. For example, the `Config` class is actually `ambassador.config.config.Config` but is imported into `ambassador` to make usage easier.) The `Resource` Class -------------------- `IRResource` and `ACResource` are subclasses of `ambassador.Resource`, although they are shown in the packages where they are logically used. A `Resource` is a kind of `dict` that can keep track of where it came from, what makes use of it, and any errors associated with it. To initialize a `Resource` requires: * `rkey`: a short identifier that is used as the primary key for _all_ the Ambassador classes to identify this single specific resource. It should be something like "ambassador-default.1" or the like that is very specific, though it doesn't have to be fun for humans. * `location`: a more human-readable string describing where the human should go to find the source of this resource. "Service ambassador, namespace default, object 1". This is primarly used for diagnostics. * `kind`: the kind of resource this is -- "Mapping", "TLS", "AuthService", whatever. * `serialization`: the _original input serialization_, if we have it, of the object. If we don't have it, this should be `None` -- don't just serialize the object to no purpose. All of these should be passed as keyword arguments (although it is possible to pass `rkey` and `kind` as positional arguments, it is discouraged). A `Resource` can also accept any other arbitrary keyword arguments, which will be saved in the `Resource` as they would be in a `dict`. `Resource` defines multiple common methods and mechanisms: * Dot notation and brace notation are equivalent for `Resource`: `rsrc.foo` and `rsrc["foo"]` are, by definition, equivalent. * `rsrc.post_error(status: RichStatus)` posts an error notification that will be tracked throughout the life of the object. It will be the case that anything trying to use the `Resource` will inherit its errors; this is not fully implemented yet. * `rsrc.referenced_by(other: Resource)` marks this `Resource` as being referenced by another `Resource`. For example, an Ambassador `Mapping` will cause Envoy clusters to be created. The `IRCluster` object created to track the cluster will reference the `ACMapping` that defined the mapping that caused the cluster to be created. This gives the diagnostics service a way to track from the cluster back to the annotation that caused it to exist. * `rsrc.references(other: Resource)` is the other direction: it marks the other `Resource` as being referenced by us. * `rsrc.is_referenced_by(rkey: str) -> Optional[Resource]` returns `None` if the given `rkey` is not associated with a `Resource` that references this `Resource`, or the referencing `Resource` if it is. * `rsrc.as_dict() -> dict` returns a raw-dictionary form of just the data fields of the `Resource`. Things like the location and the references table are removed. * Class method `Resource.from_dict(rkey: str, location: str, serialization: Optional[str], attrs: dict)` creates a new `Resource` or subclass from a dictionary. The `kind` passed in the dictionary determines the actual class of the object returned (see below for more). * Class method `Resource.from_yaml(rkey: str, location: str, serialization: str)` deserializes the YAML `serialization` and passes that to `Resource.from_dict`. * Class method `Resource.from_resource(...)` clones a `Resource`, allowing optionally overriding any field using keyword arguments. The `ACResource` class ---------------------- `ACResource` is a subclass of `Resource` which specifically refers to Ambassador configuration resources. It adds no new behaviors, but two additional keyword arguments are present when initializing: * `name` is the name given to this Ambassador resource. Required for every type except `Pragma`. * `apiVersion` is the API version to use when interpreting this resource. If not given, it defaults to "ambassador/v0". Also, `ACResource.from_dict` will look first for `ACResource` subclasses when interpreting types. The `IRResource` class ---------------------- `IRResource` is a subclass of `Resource` which specifically refers to IR resources. `IRResource` doesn't add any new fields (and, in fact, many `IRResource` subclasses default some fields) but several new behaviors are added. --- ## File: tools/src/py-mkopensource/README.md # py-mkopensource `py-mkopensource` is a program for generating reports of the libraries used by a Python application, in order to be in compliance with the attribution requirements of various opensource licenses. `py-mkopensource` takes as input the dependencies generated by `pip show`, and produces a report in either JSON or markdown output. ## Running `py-mkopensource` requires `pip` and `pip-tools`. To generate the list of dependencies from `requirements.txt`, run the following commands: ```shell pip3 --disable-pip-version-check install -r requirements.txt pip3 --disable-pip-version-check freeze --exclude-editable | cut -d= -f1 | xargs pip show | sed 's/^---$//' >dependenncies.txt echo '' >>dependenncies.txt cat dependenncies.txt | ./py-mkopensource ``` ## Command line options ### Output type Parameter `--output-type` controls the output format. #### `--output-type=markdown` Program outputs dependency information in Markdown format. This is the default value. #### `--output-type=json` Program outputs dependency information in JSON format ## 2. Official Technical Reference & Guides (emissary-ingress/emissary-ingress.github.io) # emissary-ingress.dev The [emissary-ingress.dev](https://emissary-ingress.dev) website, built using [Hugo][] and hosted on [Netlify][]. ## Local build To build and serve the site, you'll need the latest [LTS release][] of **Node**. Like Netlify, we use **[nvm][]**, the Node Version Manager, to install and manage Node versions: ```console $ nvm install --lts ``` You must also install [hugo](https://gohugo.io/getting-started/installing/) to build the site. ### Setup 1. Clone this repo. 2. From a terminal window, change to the cloned repo directory. 3. Get NPM packages and git submodules, including the the [Docsy][] theme: ```console $ npm install ``` ### Build or serve the site To locally serve the site at [localhost:1313][], run the following command: ```console $ make run ``` [Docsy]: https://www.docsy.dev [Hugo]: https://gohugo.io [localhost:1313]: http://localhost:1313 [LTS release]: https://nodejs.org/en/download/ [Netlify]: https://netlify.com [nvm]: https://github.com/nvm-sh/nvm/blob/master/README.md#installing-and-updating