## 1. Project Overview & Quickstart (emissary-ingress/testing)

# testing

Open-source repository emissary-ingress/testing

### Repository Details
- **Repository:** [emissary-ingress/testing](https://github.com/emissary-ingress/testing)
- **Primary Language:** Code

*Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.*

## 2. Official Technical Reference & Guides (emissary-ingress/emissary-ingress.github.io)

## File: README.md

# emissary-ingress.dev

The [emissary-ingress.dev](https://emissary-ingress.dev) website, built using [Hugo][] and hosted on [Netlify][].

## Local build

To build and serve the site, you'll need the latest [LTS release][] of **Node**.
Like Netlify, we use **[nvm][]**, the Node Version Manager, to install and
manage Node versions:

```console
$ nvm install --lts
```

You must also install [hugo](https://gohugo.io/getting-started/installing/) to build the site.

### Setup

 1. Clone this repo.
 2. From a terminal window, change to the cloned repo directory.
 3. Get NPM packages and git submodules, including the the [Docsy][] theme:
    ```console
    $ npm install
    ```

### Build or serve the site

To locally serve the site at [localhost:1313][], run the following command:

```console
$ make run
```

[Docsy]: https://www.docsy.dev
[Hugo]: https://gohugo.io
[localhost:1313]: http://localhost:1313
[LTS release]: https://nodejs.org/en/download/
[Netlify]: https://netlify.com
[nvm]: https://github.com/nvm-sh/nvm/blob/master/README.md#installing-and-updating

---

## File: content/en/docs/1.14/about/alternatives.md

---
title: "Alternatives"
---

Alternatives to the Emissary fall into three basic categories:

* Hosted API gateways, such as the [Amazon API gateway](https://aws.amazon.com/api-gateway/).
* Traditional API gateways, such as [Kong](https://konghq.org/).
* L7 proxies, such as [Traefik](https://traefik.io/), [NGINX](http://nginx.org/), [HAProxy](http://www.haproxy.org/), or [Envoy](https://www.envoyproxy.io), or Ingress controllers built on these proxies.

Both hosted API gateways and traditional API gateways are:

* Not self-service. The management interfaces on traditional API gateways are not designed for developer self-service, and provide limited safety and usability for developers.
* Not Kubernetes-native. They're typically configured using REST APIs, making it challenging to adopt cloud-native patterns such as GitOps and declarative configuration.
* [Designed for API management, versus microservices](../../topics/concepts/microservices-api-gateways).

A Layer 7 proxy can be used as an API gateway, but typically requires additional bespoke development to support microservices use cases. In fact, many API gateways package the additional features needed for an API gateway on top of an L7 proxy. The Emissary uses Envoy, while Kong uses NGINX. If you're interested in deploying Envoy directly, we've written an [introductory tutorial](https://www.datawire.io/guide/traffic/getting-started-lyft-envoy-microservices-resilience/).

## Istio

[Istio](https://istio.io) is an open-source service mesh, built on Envoy. A service mesh is designed to manage East/West traffic (traffic between servers and your data center), while an API gateway manages North/South traffic (in and out of your data center). Documentation on how to deploy the Emissary with Istio is [here](../../howtos/istio). In general, we've found that North/South traffic is quite different from East/West traffic (i.e., you don't control the client in the North/South use case).

---

## File: content/en/docs/1.14/about/faq.md

---
title: Frequently Asked Questions
---

## General

### Why Emissary?

Kubernetes shifts application architecture for microservices, as well as the
development workflow for a full-cycle development. Emissary is designed for
the Kubernetes world with:

* Sophisticated traffic management capabilities (thanks to its use of [Envoy Proxy](https://www.envoyproxy.io)), such as load balancing, circuit breakers, rate limits, and automatic retries.
* API management capabilities such as a developer portal and OpenID Connect integration for Single Sign-On.
* A declarative, self-service management model built on Kubernetes Custom Resource Definitions, enabling GitOps-style continuous delivery workflows.

We've written about [the history of Emissary](https://blog.getambassador.io/building-ambassador-an-open-source-api-gateway-on-kubernetes-and-envoy-ed01ed520844), [Why Emissary In Depth](../why-ambassador), [Features and Benefits](../features-and-benefits) and about the [evolution of API Gateways](../../topics/concepts/microservices-api-gateways/).

### What's the difference between the Emissary and the Ambassador Edge Stack?

The Emissary was the name of the original open-source project. As the project evolved, we realized that the functionality we were building had extended far beyond an API Gateway. In particular, the Ambassador Edge Stack is intended to provide all the functionality you need at the edge -- hence, an "edge stack." This includes an API Gateway, ingress controller, load balancer, developer portal, and more.

### How is Ambassador Edge Stack licensed?

The core Emissary is open source under the Apache Software License 2.0. The GitHub repository for the core is [https://github.com/datawire/ambassador](https://github.com/datawire/ambassador). Some additional features of the Ambassador Edge Stack (e.g., Single Sign-On) are not open source and available under a proprietary license.

### Can I use the add-on features for Ambassador Edge Stack for free?

Yes! The core functionality of the Ambassador Edge Stack is free and has no limits whatsoever. If you wish to use one of our additional, proprietary features such as Single Sign-On, you can get a free community license for up to 5 requests per second. Please contact [sales](/contact-us/) if you need more than 5 RPS.

For more details on core unlimited features and premium features, see the [editions page](/editions).

### How does Emissary use Envoy Proxy?

Emissary uses [Envoy Proxy](https://www.envoyproxy.io) as its core proxy. Envoy is an open-source, high-performance proxy originally written by Lyft. Envoy is now part of the Cloud Native Computing Foundation.

### Is Emissary production ready?

Yes. Thousands of organizations, large and small, run Emissary in production.
Public users include Chick-Fil-A, ADP, Microsoft, NVidia, and AppDirect, among others.

### What is the performance of Emissary?

There are many dimensions to performance. We published a benchmark of [Emissary performance on Kubernetes](/resources/envoyproxy-performance-on-k8s/). Our internal performance regressions cover many other scenarios; we expect to publish more data in the future.

### What's the difference between a service mesh (such as Istio) and Emissary?

Service meshes focus on routing internal traffic from service to service
("east-west"). Emissary focuses on traffic into your cluster ("north-south").
While both a service mesh and Emissary can route L7 traffic, the reality is that
these use cases are quite different. Many users will integrate Emissary with a
service mesh. Production customers of Emissary have integrated with Consul,
Istio, and Linkerd2.

## Common Configurations

### How do I disable the 404 landing page?

Established users will want to better control 404 behavior both for usability and
security.  You can leverage the Mapping resource to implement this functionality to
your cluster.  Emissary users can use a 'catch-all' mapping using the '/'
prefix in a mapping configuration.  The simplest mapping, described below, returns only 404 text.
To use a custom 404 landing page, simply insert your service and remove the rewrite value.

```yaml
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: "404-fallback"
spec:
  prefix: "/"
  rewrite: "/404/" # This must not map to any existing prefix!
  service: localhost:8500
```

For more information on the Mapping resource, see [Advanced Mapping Configuration](../../topics/using/mappings).

### How do I disable the default Admin mappings?

In a production environment, public access to the console and admin endpoints is not an
ideal situation.  To solve this, we will be using an Ambassador Module to remove the default
mappings and create a new, host-based mapping to expose the Admin endpoint more securely.  The
Ambassador module applies system-wide configuration settings for Emissary to follow.

```yaml
apiVersion: getambassador.io/v2
kind: Module
metadata:
  name: ambassador
spec:
  config:
    diagnostics:
      enabled: false
```

After applying this module, the admin endpoint is no longer accessible from the outside world.
We cannot, however, exclude actual administrators from this endpoint, so to create a more managed
endpoint for them to use, create a mapping to expose the endpoint.

```yaml
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: admin-mapping
spec:
  host: admin.example.com
  prefix: /edge_stack/
  rewrite: /edge_stack_ui/edge_stack/
  service: localhost:8500
```

Now, administrators can connect to the admin console via hostname.  Additional [Mapping](../../topics/using/intro-mappings) and
[Filter](/docs/edge-stack/latest/topics/using/filters/) settings can be appropriately configured to better control access to admin services.  To
learn more about Ambassador Module configurations, see [Ambassador Module](../../topics/running/ambassador)

## Troubleshooting

### How do I get help for Emissary?

We have an online [Slack community](http://a8r.io/slack) with thousands of
users. We try to help out as often as possible, although we can't promise a
particular response time. If you need a guaranteed SLA, we also have commercial
contracts. [Contact sales](/contact-us/) for more information.

### What do I do when I get the error `no healthy upstream`?

This error means that Emissary could not connect to your backend service.
Start by verifying that your backend service is actually available and
responding by sending an HTTP response directly to the pod. Then, verify that
Emissary is routing by deploying a test service and seeing if the mapping
works. Then, verify that your load balancer is properly routing requests to
Emissary. In general, verifying each network hop between your client and
backend service is critical to finding the source of the problem.

---

## File: content/en/docs/1.14/about/features-and-benefits.md

---
title: Features and Benefits
---

In cloud-native organizations, developers frequently take on responsibility for the full development lifecycle of a service, from development to QA to operations. Emissary was specifically designed for these organizations where developers have operational responsibility for their service(s).

As such, the Emissary is designed to be used by both developers and operators.

## Self-Service via Kubernetes Annotations

Emissary is built from the start to support _self-service_ deployments -- a developer working on a new service doesn't have to go to Operations to get their service added to the mesh, they can do it themselves in a matter of seconds. Likewise, a developer can remove their service from the mesh, or merge services, or separate services, as needed, at their convenience. All of these operations are performed via Kubernetes annotations, so it can easily integrate with your existing development workflow.

## Flexible canary deployments

Canary deployments are an essential component of cloud-native development workflows. In a canary deployment, a small percentage of production traffic is routed to a new version of a service to test it under real-world conditions. Emissary allows developers to easily control and manage the amount of traffic routed to a given service through annotations. [This tutorial](https://www.datawire.io/faster/canary-workflow/) covers a complete canary workflow using the Emissary.

## Kubernetes-native architecture

Emissary relies entirely on Kubernetes for reliability, availability, and scalability. For example, Emissary persists all state in Kubernetes, instead of requiring a separate database. Scaling the Emissary is as simple as changing the replicas in your deployment, or using a [horizontal pod autoscaler](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).

Emissary uses [Envoy](https://www.envoyproxy.io) for all traffic routing and proxying. Envoy is a modern L7 proxy that is used in production at companies including Lyft, Apple, Google, and Stripe.

## gRPC and HTTP/2 support

Emissary fully supports gRPC and HTTP/2 routing, thanks to Envoy's extensive capabilities in this area. See [gRPC and the Emissary](../../howtos/grpc) for more information.

## Istio Integration

Emissary integrates with the [Istio](https://istio.io) service mesh as the edge proxy. In this configuration, Emissary routes external traffic to the internal Istio service mesh. See [Istio and the Emissary](../../howtos/istio) for details.

## Authentication

Emissary supports authenticating incoming requests with a [custom authentication service](../../howtos/basic-auth/), and Ambassador Edge Stack natively supports OAuth/OpenID Connect, or JWT. When configured, the Ambassador Edge Stack will check with a third party authentication service prior to routing an incoming request. For more information, see the [authentication guide](/docs/edge-stack/latest/topics/using/filters/).

## Rate limiting

Emissary supports rate limiting incoming requests. When configured, the Emissary will check with a third party rate limit service prior to routing an incoming request. For more information, see the [rate limiting guide](../../topics/using/rate-limits/).

## Integrated UI

Emissary includes a diagnostics service so that you can quickly debug issues associated with configuring the Emissary. For more information, see [running Emissary in Production](../../topics/running).

---

## File: content/en/docs/1.14/about/support.md

---
title: Support
---

If you need help deploying Emissary at your organization, there are several different options available to you.

## Support tiers

### Emissary community support

If you are running the Emissary or the Ambassador Edge Stack with free, community licenses, [join our Slack channel](http://a8r.io/slack) to talk with other users in the community and get your questions answered.

If you can’t find an answer there, [contact us](/contact-us) to learn more about the support options available with Ambassador Edge Stack Enterprise.

### Ambassador Edge Stack Enterprise

With Ambassador Edge Stack Enterprise, you have access to deployment and production support. To learn more, [contact sales](/contact-us).

**Deployment and Update Support**: Ambassador Edge Stack can accelerate your migration to Kubernetes, or your upgrade between versions of Ambassador Edge Stack. Deployment support helps you with the Ambassador Edge Stack and Kubernetes migration, before you move to production.

**Production Support**: We offer two types of production support contracts for users deploying the Ambassador Edge Stack in production. We offer both business hour (8am - 5pm EST, M-F) and 24x7 Sev 1 support for the Ambassador Edge Stack. 24x7 Sev 1 support includes custom hotfix support for production outages if necessary.

## File a Github Issue

If you see a bug you want to fix, see room for documentation improvements, or have something else you want to change, you can [file an issue on github](https://github.com/datawire/ambassador/issues/new).

## Pricing

[Contact us](/contact-us) to learn how we can help, and for detailed pricing information.

---

## File: content/en/docs/1.14/about/why-ambassador.md

---
title: Why Emissary?
---

Emissary gives platform engineers a comprehensive, self-service edge stack for managing the boundary between end-users and Kubernetes. Built on the [Envoy Proxy](https://www.envoyproxy.io) and fully Kubernetes-native, Emissary is made to support multiple, independent teams that need to rapidly publish, monitor, and update services for end-users. A true edge stack, Emissary can also be used to handle the functions of an API Gateway, a Kubernetes ingress controller and a layer 7 load balancer (for more, see [this blog post](https://blog.getambassador.io/kubernetes-ingress-nodeport-load-balancers-and-ingress-controllers-6e29f1c44f2d)).

## How Does Emissary work?

Emissary is an open-source, Kubernetes-native [microservices API gateway](../../topics/concepts/microservices-api-gateways) built on the [Envoy Proxy](https://www.envoyproxy.io). Emissary is built from the ground up to support multiple, independent teams that need to rapidly publish, monitor, and update services for end-users. Emissary can also be used to handle the functions of a Kubernetes ingress controller and load balancer (for more, see [this blog post](https://blog.getambassador.io/kubernetes-ingress-nodeport-load-balancers-and-ingress-controllers-6e29f1c44f2d)).

## Cloud-native applications today

Traditional cloud applications were built using a monolithic approach. These applications were designed, coded, and deployed as a single unit. Today's cloud-native applications, by contrast, consist of many individual (micro)services. This results in an architecture that is:

* __Heterogeneous__: Services are implemented using multiple (polyglot) languages, they are designed using multiple architecture styles, and they communicate with each other over multiple protocols.
* __Dynamic__: Services are frequently updated and released (often without coordination), which results in a constantly-changing application.
* __Decentralized__: Services are managed by independent product-focused teams, with different development workflows and release cadences.

### Heterogeneous services

Emissary is commonly used to route traffic to a wide variety of services. It supports:

* configuration on a *per-service* basis, enabling fine-grained control of timeouts, rate limiting, authentication policies, and more.
* a wide range of L7 protocols natively, including HTTP, HTTP/2, gRPC, gRPC-Web, and WebSockets.
* Can route raw TCP for services that use protocols not directly supported by Emissary.

### Dynamic services

Service updates result in a constantly changing application. The dynamic nature of cloud-native applications introduces new challenges around configuration updates, release, and testing. Emissary:

* Enables [progressive delivery](../../topics/concepts/progressive-delivery), with support for canary routing and traffic shadowing.
* Exposes high-resolution observability metrics, providing insight into service behavior.
* Uses a zero downtime configuration architecture, so configuration changes have no end-user impact.

### Decentralized workflows

Independent teams can create their own workflows for developing and releasing functionality that are optimized for their specific service(s). With Emissary, teams can:

* Leverage a [declarative configuration model](../../topics/concepts/gitops-continuous-delivery), making it easy to understand the canonical configuration and implement GitOps-style best practices.
* Independently configure different aspects of Emissary, eliminating the need to request configuration changes through a centralized operations team.

## Emissary is engineered for Kubernetes

Emissary takes full advantage of Kubernetes and Envoy Proxy.

* All of the state required for Emissary is stored directly in Kubernetes, eliminating the need for an additional database.
* The Emissary team has added extensive engineering efforts and integration testing to ensure optimal performance and scale of Envoy and Kubernetes.

## For more information

[Deploy Emissary today](../../tutorials/getting-started) and join the community [Slack Channel](http://a8r.io/slack).

Interested in learning more?

* [Why did we start building Emissary?](https://blog.getambassador.io/building-ambassador-an-open-source-api-gateway-on-kubernetes-and-envoy-ed01ed520844)
* [Emissary Architecture overview](../../topics/concepts/architecture)

---

## File: content/en/docs/1.14/howtos/_index.md

---
title: "How-to guides"
weight: 20
---

These guides are designed to help users quickly accomplish common tasks. The guides assume a certain level of understanding of Emissary. Many of these guides are contributed by third parties; we welcome contributions via Pull Request at https://github.com/emissary-ingress/emissary.

* Integrating with Service Mesh. Emissary natively integrates with many service meshes.
  * [HashiCorp Consul](consul)
  * [Istio](istio)
  * [Linkerd](linkerd2)
* Distributed tracing. Emissary natively supports a number of distributed tracing systems to enable developers to visualize request flow in microservice and service-oriented architectures.
  * [Datadog](tracing-datadog)
  * [Zipkin](tracing-zipkin)
* Identity providers. Ambassador Edge Stack integrates with a number of OAuth Identity Providers via OpenID Connect.
* Monitoring. Emissary integrates with a number of different monitoring/metrics providers.
  * [Prometheus](prometheus)
* Frameworks and Protocols. Emissary supports a wide range of protocols and cloud-native frameworks.
  * [gRPC](grpc)
  * [Knative Serverless Framework](knative)
  * [WebSockets](websockets)
* Security. Emissary supports a number of strategies for securing Kubernetes services.
  * [HTTPS and TLS termination](tls-termination)
  * [Certificate Manager](cert-manager) can be used to automatically obtain and renew TLS certificates; Ambassador Edge Stack natively integrates this functionality.
  * [Client Certificate Validation](client-cert-validation)
  * [Basic Authentication](basic-auth) is a tutorial on how to use the external authentication API to code your own authentication service.
  * [Basic Rate Limiting](rate-limiting-tutorial)
  * [Advanced Rate Limiting](advanced-rate-limiting)

---

## File: content/en/docs/1.14/howtos/advanced-rate-limiting.md

---
title: Advanced rate limiting
---

Ambassador Edge Stack integrates a flexible, high-performance Rate Limit Service (RLS). Similar to Emissary, the RLS features a decentralized configuration model so that individual teams can manage their own rate limits. For example:

* A service owner may want to manage load shedding characteristics and ensuring specific types of requests take precedence over other types of requests
* An operations engineer may want to ensure service availability overall when request volume is high and limit the total number of requests being passed to upstream services
* A security engineer may want to protect against denial-of-service attacks from a bad actor

Like Ambassador Edge Stack, the Ambassador Edge Stack RLS is designed so that many different teams, with different requirements, can independently manage and control rate limiting as necessary.

## Request labels and domains

In Ambassador Edge Stack, each request can have multiple *labels*. Labels are arbitrary key/value pairs that contain any metadata about a given request, e.g., its IP, a hard-coded value, the path of the request, and so forth. The Rate Limit Service processes these labels and enforces any limits that are set on a label. Labels can be assigned to *domains*, which are separate namespaces. Typically, different teams would be responsible for different domains.

## Configuring rate limiting: The 50,000 foot view

Logically, configuring rate limiting is straightforward.

1. Configure a specific mapping to include one or more request labels.
2. Configure a limit for a given request label with the `RateLimit` resource.

In the examples below, we'll use the backend service of the quote sample application.

## Example 1: Global rate limiting for availability

Imagine the backend service is a Rust-y application that can only handle 3 requests per minute before crashing. While the engineering team really wants to rewrite the backend service in Golang (because Rust isn't fast enough), they haven't had a chance to do so. We want to rate limit all requests for this service to 3 requests per minute. (ProTip: Using requests per minute simplifies testing.)

We update the mapping for the `quote` service to add a request label `backend` to the route as part of a `request_label_group`:

```yaml
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: quote-backend
spec:
  prefix: /backend/
  service: quote
  labels:
    ambassador:
      - request_label_group:
        - backend
```

*Note* If you're modifying an existing mapping, make sure you to update the apiVersion to `v2` as above from `v1`.

We then need to configure the rate limit for the backend service. Create a new YAML file, `backend-ratelimit.yaml`, and put the following configuration into the file.

```yaml
apiVersion: getambassador.io/v2
kind: RateLimit
metadata:
  name: backend-rate-limit
spec:
  domain: ambassador
  limits:
   - pattern: [{generic_key: backend}]
     rate: 3
     unit: minute
```

`generic_key` in the example above is a special, hard-coded value that is used when a single string label is added to a request.

Deploy the rate limit with `kubectl apply -f backend-ratelimit.yaml`. (Make sure you `kubectly apply` the modified `quote-backend` mapping as well.)

## Example 2: Per user rate limiting

Suppose you've rewritten the quote backend service in Golang, and it's humming along nicely. You then discover that some users are taking advantage of this speed to sometimes cause a big spike in requests. You want to make sure that your API doesn't get overwhelmed by any single user. We use the `remote_address` special value in our mapping, which will automatically label all requests with the calling IP address:

```yaml
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: quote-backend
spec:
  prefix: /backend/
  service: quote
  labels:
    ambassador:
      - request_label_group:
        - remote_address
```

We then update our rate limits to limit on `remote_address`:

```yaml
apiVersion: getambassador.io/v2
kind: RateLimit
metadata:
  name: backend-rate-limit
spec:
  domain: ambassador
  limits:
   - pattern: [{remote_address: "*"}]
     rate: 3
     unit: minute
```

Note for this to work, you need to make sure you've properly configured Ambassador Edge Stack to [propagate your original client IP address](../../topics/running/ambassador#trust-downstream-client-ip).

## Example 3: Load shedding GET requests

You've dramatically improved availability of the quote backend service, thanks to the per-user rate limiting. However, you've realized that on occasion the queries (e.g., the 'GET' requests) cause so much volume that updates to the backend (e.g., the 'POST' requests) don't get processed. So we're going to add a more sophisticated load shedding strategy:

* We're going to rate limit per user.
* We're going to implement a global rate limit on `GET` requests, but not `POST` requests.

```yaml
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: quote-backend
spec:
  prefix: /backend/
  service: quote
  labels:
    ambassador:
      - request_label_group:
        - remote_address
        - backend_http_method:
            header: ":method"
            omit_if_not_present: true
```

When we add multiple criteria to a pattern, the entire pattern matches when ANY of the rules match (i.e., a logical OR). A pattern match then triggers a rate limit event. Our rate limiting configuration becomes:

```yaml
apiVersion: getambassador.io/v2
kind: RateLimit
metadata:
  name: backend-rate-limit
spec:
  domain: ambassador
  limits:
   - pattern: [{remote_address: "*"}, {backend_http_method: GET}]
     rate: 3
     unit: minute
```

## Example 4: Global rate limiting

Suppose, like [Example 2](#example-2-per-user-rate-limiting), you want to ensure a single user cannot overload your server with too many requests to any service. You need to add a request label to every request so you can rate limit off every request a calling IP makes. This can be configured with a [global rate limit](../../topics/using/rate-limits) that add the `remote_address` special value to every request:

```yaml
---
apiVersion: getambassador.io/v2
kind: Module
metadata:
  name: ambassador
spec:
  config:
    use_remote_address: true
    default_label_domain: ambassador
    default_labels:
      ambassador:
        defaults:
        - remote_address
```

We can then configure a global `RateLimit` object that limits on `remote_address`:

```yaml
apiVersion: getambassador.io/v2
kind: RateLimit
metadata:
  name: global-rate-limit
spec:
  domain: ambassador
  limits:
   - pattern: [{remote_address: "*"}]
     rate: 10
     unit: minute
```

### Bypassing a global rate limit

Sometimes, you may have an API that cannot handle as much load as others in your cluster. In this case, a global rate limit may not be enough to ensure this API is not overloaded with requests from a user. To protect this API, you will need to create a label that tells Ambassador Edge Stack to apply a stricter limit on requests. With the above global rate limit configuration rate limiting based on `remote_address`, you will need to add a request label to the services `Mapping`.

```yaml
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  name: quote-backend
spec:
  prefix: /backend/
  service: quote
  labels:
    ambassador:
      - request_label_group:
        - backend
```

Now, the `request_label_group`, contains both the `generic_key: backend` *and* the `remote_address` key applied from the global rate limit. This allows us to create a separate `RateLimit` object for this route:

```yaml
apiVersion: getambassador.io/v2
kind: RateLimit
metadata:
  name: backend-rate-limit
spec:
  domain: ambassador
  limits:
   - pattern: [{remote_address: "*"}, {generic_key: backend}]
     rate: 3
     unit: minute
```

Now, requests to `/backend/` will be rate limited after only 3 requests.

## Rate limiting matching rules

The following rules apply to the rate limit patterns:

* Patterns are order-sensitive, and must respect the order in which a request is labeled.
* Every label in a label group must exist in the pattern in order for matching to occur.
* By default, any type of failure will let the request pass through (fail open).
* Ambassador Edge Stack sets a hard timeout of 20ms on the rate limiting service. If the rate limit service does not respond within the timeout period, the request will pass through.
* If a pattern does not match, the request will pass through.

## Troubleshooting rate limiting

The most common source of failure of the rate limiting service will occur when the labels generated by Ambassador Edge Stack do not match the rate limiting pattern. By default, the rate limiting service will log all incoming labels from Ambassador Edge Stack. Use a tool such as [Stern](https://github.com/stern/stern) to watch the rate limiting logs from Ambassador Edge Stack, and ensure the labels match your descriptor.

## More

For more on rate limiting, see the [rate limit guide](../../topics/using/rate-limits/).

---

## File: content/en/docs/1.14/howtos/auth-kubectl-keycloak.md

---
title: Kubernetes SSO with OIDC and Keycloak
---

Developers use `kubectl` to access Kubernetes clusters. By default `kubectl` uses a certificate to authenticate to the Kubernetes API. This means that when multiple developers need to access a cluster, the certificate needs to be shared. Sharing the credentials to access a Kubernetes cluster presents a significant security problem. Compromise of the certificate is very easy and the consequences can be catastrophic.

In this tutorial, we walk through how to set up your Kubernetes cluster to add Single Sign-On support for `kubectl` using OpenID Connect (OIDC) and Keycloak. Instead of using a shared certificate, users will be able to use their own personal credentials to use `kubectl` with `kubelogin`.

## Prerequisites

This tutorial relies on Ambassador Edge Stack to manage access to your Kubernetes cluster, and uses Keycloak as your identity provider. To get started:

*Note* This guide was designed and validated using an Azure AKS Cluster.  It's possible that this procedure will work with other cloud providers, but there is a lot of variance in the Authentication mechanisms for the Kubernetes API.  See the troubleshooting note at the bottom for more info.

* Azure AKS Cluster [here](https://docs.microsoft.com/en-us/azure/aks/tutorial-kubernetes-deploy-cluster)
* Install Ambassador Edge Stack [here](../../topics/install/)
* Deploy Keycloak on Kubernetes [here](https://www.keycloak.org/getting-started/getting-started-kube)

## Cluster Setup

In this section, we'll configure your Kubernetes cluster for single-sign on.

### 1. Authenticate Ambassador Edge Stack with Kubernetes API

1. Delete the openapi mapping from the Ambassador namespace `kubectl delete -n ambassador ambassador-devportal-api`. (this mapping can conflict with `kubectl` commands)

2. Create a new private key using `openssl genrsa -out aes-key.pem 4096`.

3. Create a file `aes-csr.cnf` and paste the following config.

    ```cnf
    [ req ]
    default_bits = 2048
    prompt = no
    default_md = sha256
    distinguished_name = dn

    [ dn ]
    CN = ambassador-kubeapi # Required

    [ v3_ext ]
    authorityKeyIdentifier=keyid,issuer:always
    basicConstraints=CA:FALSE
    keyUsage=keyEncipherment,dataEncipherment
    extendedKeyUsage=serverAuth,clientAuth
    ```

4. Create a certificate signing request with the config file we just created.  `openssl req -config ./aes-csr.cnf -new -key aes-key.pem -nodes -out aes-csr.csr`.

5. Create and apply the following YAML for a CertificateSigningRequest.  Replace {{BASE64_CSR}} with the value from `cat aes-csr.csr | base64`.  Note that this is `aes-csr.csr`, and not `aes-csr.cnf`.

    ```yaml
    apiVersion: certificates.k8s.io/v1beta1
    kind: CertificateSigningRequest
    metadata:
      name: aes-csr
    spec:
      groups:
      - system:authenticated
      request: {{BASE64_CSR}} # Base64 encoded aes-csr.csr
      usages:
      - digital signature
      - key encipherment
      - server auth
      - client auth
    ```

6. Check csr was created: `kubectl get csr` (it will be in pending state).  After confirmation, run `kubectl certificate approve aes-csr`.  You can check `kubectl get csr` again to see that it's in the `Approved, Issued` state.

7. Get the resulting certificate and put it into a pem file.  `kubectl get csr aes-csr -o jsonpath='{.status.certificate}' | base64 -d > aes-cert.pem`.

8. Create a TLS `Secret` using our private key and public certificate.  `kubectl create secret tls -n ambassador aes-kubeapi --cert ./aes-cert.pem --key ./aes-key.pem`

9. Create a `Mapping` and `TLSContext` for the Kube API.

    ```yaml
    ---
    apiVersion: getambassador.io/v2
    kind: TLSContext
    metadata:
      name: aes-kubeapi-context
      namespace: ambassador
    spec:
      hosts:
      - "*"
      secret: aes-kubeapi
    ---
    apiVersion: getambassador.io/v2
    kind: Mapping
    metadata:
      name: aes-kubeapi-mapping
      namespace: ambassador
    spec:
      prefix: /
      allow_upgrade:
      - spdy/3.1
      service: https://kubernetes.default.svc
      timeout_ms: 0
      tls: aes-kubeapi-context
    ```

10. Create RBAC for the "aes-kubeapi" user by applying the following YAML.

    ```yaml
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: aes-impersonator-role
    rules:
    - apiGroups: [""]
      resources: ["users", "groups", "serviceaccounts"]
      verbs: ["impersonate"]
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: aes-impersonator-rolebinding
    subjects:
    - apiGroup: rbac.authorization.k8s.io
      kind: User
      name: aes-kubeapi
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: aes-impersonator-role
    ```

As a quick check, you should be able to `curl https://<ambassador-domain>/api` and get a response similar to the following:

  ```json
  {
    "kind": "APIVersions",
    "versions": [
      "v1"
    ],
    "serverAddressByClientCIDRs": [
      {
        "clientCIDR": "0.0.0.0/0",
        "serverAddress": "\"<some-kubernetes-service-address>\":443"
      }
    ]
  }%
  ```

### 2. Set up Keycloak config

1. Create a new Realm and Client (e.g. ambassador, ambassador)
2. Make sure that `http://localhost:8000` and `http://localhost:18000` are valid Redirect URIs
3. Set access type to confidential and Save
4. Go to the Credentials tab and note down the secret
5. Go to the user tab and create a user with the first name "john"

### 3. Create a ClusterRole and ClusterRoleBinding for the OIDC user "john"

1. Add the following RBAC to create a user "john" that only allowed to perform `kubectl get services` in the cluster.

    ```yaml
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: john-binding
    subjects:
    - kind: User
      name: john
      apiGroup: rbac.authorization.k8s.io
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: john-role
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: john-role
    rules:
      - apiGroups: [""]
        resources: ["services"]
        verbs: ["get", "list"]
    ```

2. Test the API again with the following 2 `curls`: `curl https://<ambassador-domain>/api/v1/namespaces/default/services?limit=500 -H "Impersonate-User: "john"` and `curl https://<ambassador-domain>/api/v1/namespaces/default/pods?limit=500 -H "Impersonate-User: "john"`.  You will find that the first curl should succeeds and the second curl should fail with the following response.

```json
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "pods is forbidden: User \"john\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}
```

### 4. Create a JWT filter to authenticate the user

1. Create the following JWT `Filter` and `FilterPolicy` based on this template:

    ```yaml
    ---
    apiVersion: getambassador.io/v2
    kind: Filter
    metadata:
      name: "kubeapi-jwt-filter"
      namespace: "ambassador"
    spec:
      JWT:
        jwksURI: https://<keycloak-domain>/auth/realms/<my-realm>/protocol/openid-connect/certs # If the keycloak instance is internal, you may want to use the internal k8s endpoint (e.g. http://keycloak.keycloak) instead of figuring out how to exclude JWKS requests from the FilterPolicy
        injectRequestHeaders:
        - name: "Impersonate-User" # Impersonate-User is mandatory, you can also add an Impersonate-Groups if you want to do group-based RBAC
          value: "{{ .token.Claims.given_name }}" # This uses the first name we specified in the Keycloak user account
    ---
    apiVersion: getambassador.io/v2
    kind: FilterPolicy
    metadata:
      name: "kubeapi-filter-policy"
      namespace: "ambassador"
    spec:
      rules:
      - host: "*"
        path: "*"
        filters:
        - name: kubeapi-jwt-filter
    ```

## Client set up

Now, we need to set up the client. Each user who needs to access the Kubernetes cluster will need to follow these steps.

### 1. Install kubelogin

1. Install [kubelogin](https://github.com/int128/kubelogin#getting-started). Kubelogin is a `kubectl` plugin that enables OpenID Connect login with `kubectl`.

2. Edit your local Kubernetes config file (either `~/.kube/config`, or your `$KUBECONFIG` file) to include the following, making sure to replace the templated values.

    ```yaml
    apiVersion: v1
    kind: Config
    clusters:
    - name: azure-ambassador
      cluster:
        server: https://<ambassador-domain>
    contexts:
    - name: azure-ambassador-kube-api
      context:
        cluster: azure-ambassador
        user: azure-ambassador
    users:
    - name: azure-ambassador
      user:
        exec:
          apiVersion: client.authentication.k8s.io/v1beta1
          command: kubectl
          args:
          - oidc-login
          - get-token
          - --oidc-issuer-url=https://<keycloak-domain>/auth/realms/<my-realm>
          - --oidc-client-id=<client-id>
          - --oidc-client-secret=<client-secret>
    ```

3. Switch to the context set above (in the example it's `azure-ambassador-kube-api`).

4. Run `kubectl get svc`.  This should open a browser page to the Keycloak login.  Type in the credentials for "john" and, on success, return to the terminal to see the kubectl response. Congratulations, you've set up Single Sign-On with Kubernetes!

5. Now try running `kubectl get pods`, and notice we get an `Error from server (Forbidden): pods is forbidden: User "john" cannot list resource "pods" in API group "" in the namespace "default"`.  This is expected because we explicitly set up "john" to only have access to view `Service` resources, and not `Pods`.

### 7. Logging Out

1. Delete the token cache with `rm -r ~/.kube/cache/oidc-login`
2. You may also have to remove session cookies in your browser or do a remote logout in the keycloak admin page.

### Troubleshooting

1. Why isn't this process working in my `` cluster?
  Authentication to the Kubernetes API is highly cluster specific.  Many use x509 certificates, but as a notable exception, Amazon's Elastic Kubernetes Service, for example, uses an Authenticating Webhook that connects to their IAM solution for Authentication, and so is not compatible specifically with this guide.
2. What if I want to use RBAC Groups?
  User impersonation allows you to specify a Group using the `Impersonate-Group` header.  As such, if you wanted to use any kind of custom claims for the ID token, they can be mapped to the `Impersonate-Group` header.  Note that you always have to use an `Impersonate-Name` header, even if you're relying solely on the Group for Authorization.
3. I keep getting a 401 `Failure`, `Unauthorized` message, even for `https://<ambassador-domain>/api`.
  This likely means that there is either something wrong with the Certificate that was issued, or there's something wrong with your `TLSContext` or `Mapping` config.  Ambassador Edge Stack must present the correct certificate to the Kubernetes API and the RBAC usernames and the CN of the certificate have to be consistent with one another.
4. Do I have to use `kubelogin`?
  Technically no.  Any method of obtaining an ID or Access token from an Identity Provider will work.  You can then pass the token using `--token <jwt-token>` when running `kubectl`.  `kubelogin` simply automates the process of getting the ID token and attaching it to a `kubectl` request.

## Under the Hood

In this tutorial, we set up Ambassador Edge Stack to [impersonate a user](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation) to access the Kubernetes API. Requests get sent to Ambassador Edge Stack, which functions as an Authenticating Proxy. Ambassador Edge Stack uses its integrated authentication mechanism to authenticate the external request's identity and sets the User and Group based on Claims recieved by the `Filter`.

The general flow of the `kubectl` command is as follows: On making an unauthenticated kubectl command, `kubelogin` does a browser open/redirect in order to do OIDC token negotiation.  `kubelogin` obtains an OIDC Identity Token (notice this is not an access token) and sends it to Ambassador Edge Stack in an Authorization header.  Ambassador Edge Stack validates the Identity Token and parses Claims from it to put into `Impersonate-XXX` headers.  Ambassador Edge Stack then scrubs the Authorization header and replaces it with the Admin token we set up in step 1.  Ambassador Edge Stack then forwards this request with the new Authorization and Impersonate headers to the KubeAPI to first Authenticate, and then Authorize based on Kubernetes RBAC.