1. Project Overview & Quickstart (telepresenceio/telepresence)

File: README.md

Telepresence: Fast, Local Development for Kubernetes

Telepresence is a CNCF project that connects your
workstation to a Kubernetes cluster. Code and debug a service locally — with
your own IDE, debugger, and hot reload — while it receives real traffic from
the cluster, without the container build/push/deploy cycle.

Key Features

Cluster access - Your workstation reaches cluster Services and Pods as
if it were inside the cluster: cluster DNS, cluster IPs, no port-forwards.
Four attachment modes - Replace the remote container, intercept a
service port, wiretap a copy of the traffic, or ingest just the
environment and volumes.
Traffic filtering - Intercept only your own requests, selected by HTTP
header or path, so a whole team can share one cluster without collisions.
Sidecar or node-agent - The traffic-agent is either injected as a
sidecar (the default), or runs as a node-hosted pod that attaches to
workloads without modifying them: no injection, no pod restarts.
Remote environment, local process - Run with the remote container's
environment variables and volume mounts, so the code behaves like it does
in the cluster.

Getting Started

Quick Start Guide - Get up and running in minutes
Installation - Install the Telepresence client
Documentation - Full documentation

How It Works

When Telepresence connects, it creates a virtual network interface on your
workstation and routes cluster traffic through a traffic-manager deployed in
the cluster. Attaching to a workload adds a traffic-agent — a sidecar or a
node-agent — that hands the workload's traffic, environment, and volumes over
to your machine. See the
architecture overview
for the full picture.

Community

GitHub Discussions - Ask questions, share ideas, and help shape the roadmap
CNCF Slack - Join #telepresence-oss
Troubleshooting - Common issues and solutions

Contributing

See AGENTS.md for build instructions, architecture overview, and development guidelines.

License

Telepresence is licensed under the Apache License 2.0.

File: docs/discussion/how-it-works.md

How it works

Goals

Our goals for Telepresence are:

Transparency: make the local proxied process match the Kubernetes environment as closely as possible.
Isolation: only the proxied process has its environment modified.
This goal is much like that of a container: an isolated process-specific environment.
Cross-platform: Linux and macOS work the same way, when possible.
Linux provides far more capabilities (mount namespaces, bind mounts, network namespaces) than macOS.
Compatibility: works with any program.

Achieving all these goals at the same time is not always possible.
We've therefore chosen to support more than one method of proxying, with the different methods each having its own benefits and limitations.

How it works

Telepresence works by building a two-way network proxy (bootstrapped using kubectl port-forward or oc port-forward) between a custom pod running inside a remote (or local) Kubernetes cluster and a process running on your development machine.
The custom pod is substituted for your normal pod that would run in production.
Typically you'd want to do this to a testing or staging cluster, not your production cluster.

Environment variables from the remote pod are made available to your local process.
In addition, the local process has its networking transparently overridden such that DNS calls and TCP connections are routed over the proxy to the remote Kubernetes cluster.
This happens one of two ways:

When using --method vpn-tcp, the default, a VPN-like tunnel is created using a program called sshuttle, which tunnels the packets over the SSH connection, and forwards DNS queries to a DNS proxy in the cluster.
When using --method inject-tcp this is implemented using LD_PRELOAD/DYLD_INSERT_LIBRARIES mechanism on Linux/OSX, where a shared library can be injected into a process and override library calls.
In particular, it overrides DNS resolution and TCP connection and routes them via a SOCKS proxy to the cluster.
We wrote a blog post explaining LD_PRELOAD in more detail.

(Technically there is a third method, for proxying containers, that is based on the vpn-tcp method.)

Volumes are proxied using sshfs, with their location available to the container as an environment variable.

The result is that your local process has a similar environment to the remote Kubernetes cluster, while still being fully under your local control.

vpn-tcp method in detail

Telepresence figures out the CIDR for Kubernetes Pods and Services, and any cloud hosts specified with --also-proxy, and tells sshuttle to forward traffic to those IPs via the proxy Pod running in Kubernetes.
sshuttle effectively acts as a VPN for these packets.

For DNS the implementation is more complex, since base sshuttle is insufficient.
sshuttle will capture all packets going to your default nameservers, and instead forward them to a custom DNS server running inside the Telepresence Pod in the Kubernetes cluster.
For now we'll assume this is a remote cluster.
Some examples can help explain the process.

Whenever you do a DNS lookup your DNS client library may add a suffix, try to resolve that, and if that fails try the original hostname you provided.
For example, if you're working at the offices of Example.com then the DHCP server in your office may tell clients to add example.com to DNS lookups.
Thus, when you lookup the domain myservice your DNS client will first try myservice.example.com and then myservice if that doesn't work.

On startup:

telepresence does a lookup of hellotelepresence.
Your DNS client library turns this into a lookup of hellotelepresence.example.com.
sshuttle forwards this to the custom Telepresence DNS server.
The Telepresence DNS server recognizes the hellotelepresence marker, and so now it knows the suffix it needs to filter is example.com.

Next let's say you do:

console

$ curl http://myservice:8080

inside a Telepresence-proxied shell.

Your DNS client library turns this into a lookup of myservice.example.com.
sshuttle forwards this to the custom Telepresence DNS server.
The custom Telepresence DNS server strips off example.com, and does a local DNS lookup of myservice.
The Kubernetes DNS server replies with the IP of the myservice Service.
The custom Telepresence DNS server hands this back, sshuttle forwards it back, and eventually curl gets the Service IP.
curl opens connection to that IP, sshuttle forwards it to the Kubernetes cluster.

minikube/minishift and Docker Desktop

There is an additional complication when running a cluster locally in a VM, using something like minikube, minishift, or Docker Desktop.
Let's say you lookup google.com.

sshuttle forwards google.com to Kubernetes (via Telepresence DNS server).
Kubernetes DNS server doesn't know about any such Service, so it does normal DNS lookup.
The normal DNS lookup might get routed via the host machine.
sshuttle captures all DNS lookups going from the host machine.
Your DNS lookup is now in an infinite loop.

To solve this Telepresence will detect minikube, minishift, and Docker Desktop.
When it does, the Telepresence DNS server will forward DNS requests that aren't Kubernetes-specific to an external DNS server that is different than the ones your host machine is using.
E.g. it might use Google's public DNS if your host isn't.
As a result these DNS lookups aren't captured by sshuttle and the infinite loop is prevented.

inject-tcp method in detail

A custom SOCKS proxy is run on the Kubernetes pod, which uses Tor's extended SOCKSv5 protocol which adds support for DNS lookups.
kubectl port-forward creates a tunnel to this SOCKS proxy.

A subprocess is then run using torsocks, a library that uses LD_PRELOAD to override TCP and (most) DNS lookups so that they get routed via the SOCKSv5 proxy.

Docker method in detail

Telepresence can also proxy a Docker container, using a variant on the vpn-tcp method: sshuttle is run inside a Docker container.

The user's specified Docker container is then run using the same network namespace as the proxy container with sshuttle.
(In typical Docker usage, in contrast, each container gets its own separate network namespace by default.)

File: docs/discussion/overview.md

Introduction to Telepresence

Telepresence is an open source tool that lets you run a single service locally, while connecting that service to a remote Kubernetes cluster. This lets developers working on multi-service applications to:

Do fast local development of a single service, even if that service depends on other services in your cluster. Make a change to your service, save, and you can immediately see the new service in action.
Use any tool installed locally to test/debug/edit your service. For example, you can use a debugger or IDE!
Make your local development machine operate as if it's part of your Kubernetes cluster. If you've got an application on your machine that you want to run against a service in the cluster -- it's easy to do.

Telepresence works on both Mac OS X and Linux, with OS-native packages.

How it works

Telepresence deploys a two-way network proxy in a pod running in your Kubernetes cluster. This pod proxies data from your Kubernetes environment (e.g., TCP connections, environment variables, volumes) to the local process. The local process has its networking transparently overridden so that DNS calls and TCP connections are routed over the proxy to the remote Kubernetes cluster.

This approach gives:

your local service full access to other services in the remote cluster
your local service full access to Kubernetes environment variables, secrets, and ConfigMap
your remote services full access to your local service

How Telepresence works is discussed in more detail here.

Alternatives to Telepresence

Typical alternatives to Telepresence include:

running your entire multi-service application locally via Docker Compose. This gives you a fast dev/debug cycle. However, it's less realistic since you're not running your services actually inside Kubernetes, and there are cloud services you might use (e.g., a database) that might not be easy to use locally.
minikube. You can't do live coding/debugging with minikube by itself, but you can with Telepresence. The two work well together.
run everything in a remote Kubernetes cluster. Again, you can't do live coding/debugging in a remote Kubernetes cluster ... but you can with Telepresence.

Getting started

Telepresence offers a broad set of proxying options which have different strengths and weaknesses. Generally speaking, we recommend you:

Start with the container method, which provides the most consistent environment for your code. Here is a container quick start.
Use the VPN method, which lets you use an IDE or debugger with your code. Here is a quick start that uses the VPN method

File: docs/discussion/why-telepresence.md

Why Telepresence?

Let's assume you have a web service which listens on port 8080, and has a Dockerfile which gets built to an image called examplecom/servicename.
Your service depends on other Kubernetes Service instances (thing1 and thing2), and on a cloud database.

The Kubernetes staging environment looks like this:

graph LR
subgraph Kubernetes in Cloud
code["k8s.Pod: servicename"]
s1["k8s.Service: servicename"]---code
code---s2["k8s.Service: thing1"]
code---s3["k8s.Service: thing2"]
code---c1>"Cloud Database (AWS RDS)"]
end

The slow status quo

If you need that cloud database and those two services to directly test your software, you will need to do the following to test a change:

Change your code.
Build a Docker image.
Push the Docker image to a Docker registry in the cloud.
Update the staging Kubernetes cluster to use your new image.
Wait for the image to download.

This is slow.

graph TD
subgraph Laptop
code["Source code for servicename"]==>local["Docker image"]
kubectl
end
subgraph Kubernetes in Cloud
local==>registry["Docker registry"]
registry==>deployment["k8s.Deployment: servicename"]
kubectl==>deployment
s1["k8s.Service: servicename"]---deployment
deployment---s2["k8s.Service: thing1"]
deployment---s3["k8s.Service: thing2"]
deployment---c1>"Cloud Database (AWS RDS)"]
end

A fast development cycle with Telepresence

Telepresence works by running your code locally, as a normal local process, and then forwarding requests to/from the Kubernetes cluster.

graph TD
subgraph Laptop
code["Source code for servicename"]==>local["local process"]
local---client[Telepresence client]
end
subgraph Kubernetes in Cloud
client-.-proxy["k8s.Pod: Telepresence proxy"]
s1["k8s.Service: servicename"]---proxy
proxy---s2["k8s.Service: thing1"]
proxy---s3["k8s.Service: thing2"]
proxy---c1>"Cloud Database (AWS RDS)"]
end

This means development is fast: you only have to change your code and restart your process.
Many web frameworks also do automatic code reload, in which case you won't even need to restart.

Other options

For more discussion of the solution space for local development see the article Development Environments for Microservices.

File: docs/howto/golang.md

Go support

In this howto you'll learn how to use Telepresence with a Go program.
Because of the way Go is implemented you will need to use --method vpn-tcp with telepresence.

A Go program talking to Kubernetes

First, start a web server inside Kubernetes and expose it via a service:

console

$ kubectl create deployment hello-world --image=datawire/hello-world
$ kubectl expose deployment hello-world --port=8000

Next, install a neat little Go program called wuzz, an interactive HTTP client.

console

$ go get github.com/asciimoo/wuzz

Now we'll see how we can use wuzz to interact with a remote Kubernetes cluster.
telepresence will create a new Deployment inside Kubernetes that will act as a proxy, and then communication from the wuzz subprocess it runs will be forwarded to the cluster:

console

$ telepresence --run $GOPATH/bin/wuzz http://hello-world:8000/

Important: Go programs will not work with --method inject-tcp option.

The wuzz UI will appear with the URL http://hello-world:8000/.
Hit Enter and you should see the "Hello, World!" response from the Kubernetes service.
You can also interact with the Kubernetes API - change the URL to https://kubernetes/ (but typically you'll have problems with the custom certificate authority.)

Kubernetes talks to a Go program

You can also run a Go program as a local server and have requests to your Kubernetes Deployment forwarded to that process.
This is just the same as the example covered in the tutorial except that you use --method vpn-tcp, and run a Go process instead of a Python process.

For example, if you have a Deployment called myservice running in Kubernetes and listening on port 8080, you can temporarily swap it out for a local process and have traffic forwarded to your laptop:

console

$ telepresence --swap-deployment myservice --expose 8080 \
               --run ./yourgoserver --port=8080

Now requests to that remote Deployment will be routed to the yourgoserver process running on your machine.

You can learn more about the differences between --new-deployment and --swap-deployment in the relevant reference documentation.

File: docs/howto/volumes.md

Volume access

Volume support requires a small amount of work on your part.
The root directory where all the volumes can be found will be set to the TELEPRESENCE_ROOT environment variable in the shell run by telepresence.
You will then need to use that env variable as the root for volume paths you are opening.

Telepresence will attempt to gather the mount points that exist in the remote pod and list them in the TELEPRESENCE_MOUNTS environment variable, separated by : characters.
This allows automated discovery of remote volumes.

For example, all Kubernetes containers have a volume mounted at /var/run/secrets with the service account details.
Those files are accessible from Telepresence:

console

$ telepresence
T: [...]
T: Setup complete. Launching your command.
@tel-testing|bash-3.2$ echo $TELEPRESENCE_ROOT
/tmp/tel-6cjjs3ba/fs
@tel-testing|bash-3.2$ echo $TELEPRESENCE_MOUNTS
/var/run/secrets/kubernetes.io/serviceaccount
@tel-testing|bash-3.2$ ls $TELEPRESENCE_ROOT/var/run/secrets/kubernetes.io/serviceaccount/
ca.crt          namespace       token

The files are available at a different path than they are on the actual Kubernetes environment.

One way to deal with that is to change your application's code slightly.
For example, let's say you have a volume that mounts a file called /app/secrets.
Normally you would just open it in your code like so:

python

secret_file = open("/app/secrets")

To support volume proxying by Telepresence, you will need to change your code, for example:

python

volume_root = "/"
if "TELEPRESENCE_ROOT" in os.environ:
    volume_root = os.environ["TELEPRESENCE_ROOT"]
secret_file = open(os.path.join(volume_root, "app/secrets"))

By falling back to / when the environment variable is not set your code will continue to work in its normal Kubernetes setting.

This approach is unavailable if you do not control the code that accesses the mounted filesystem, such as if you use a third-party library.
However, many such libraries offer configuration to work around this.
For example, the Java Kubernetes client library allows configuration via environment variables.

To simplify this process, Telepresence optionally lets you set the value of TELEPRESENCE_ROOT to a known path using the --mount option.
Using a known value as the mount point (e.g., --mount=/tmp/tel_root) can you let you configure your tools and libraries once and rely on that configuration continuing to work across multiple Telepresence sessions.
When using the container method, the --mount option allows bind-mounting portions of the remote filesystem directly onto the usual paths.

For example, the kubectl command expects to find Kubernetes API service account credentials in /var/run/secrets.
This example shows kubectl successfully talking to the cluster while running in a local container:

shell

$ telepresence --mount=/tmp/known --docker-run --rm -it -v=/tmp/known/var/run/secrets:/var/run/secrets lachlanevenson/k8s-kubectl version --short
Volumes are rooted at $TELEPRESENCE_ROOT. See https://telepresence.io/howto/volumes.html for details.

Client Version: v1.10.1
Server Version: v1.7.14-gke.1

Another way you can do this is by using the proot utility on Linux, which allows you to do fake bind mounts without being root.
For example, presuming you've installed proot (apt install proot on Ubuntu), in the following example we bind $TELEPRESENCE_ROOT/var/run/secrets to /var/run/secrets.
That means code doesn't need to be modified as the paths are in the expected location:

console

@minikube|$ proot -b $TELEPRESENCE_ROOT/var/run/secrets/:/var/run/secrets bash
$ ls /var/run/secrets/kubernetes.io/serviceaccount/
ca.crt  namespace  token

Using the TELEPRESENCE_MOUNTS environment variable allows for automatic discovery and handling of mount points.
For example, the following Python code will create symlinks so that any mounted volumes appear in their normal locations:

python

def telepresence_remote_mounts():
    mounts = os.environ.get('TELEPRESENCE_MOUNTS')
    if not mounts:
        return

    tel_root = os.environ.get('TELEPRESENCE_ROOT')

    for mount in mounts.split(':'):
        dir_name, link_name = os.path.split(mount)
        os.makedirs(dir_name, exist_ok=True)

        link_src = os.path.join(tel_root, mount[1:])
        os.symlink(link_src, mount)

Volume access via Docker volume for the container method

It is possible to use the vieux/sshfs volume driver to access remote volumes if you are using the container method.
This makes it possible to use volumes even if you don't have mount privileges or capabilities on your main system, e.g. on Windows.
Use the --docker-mount option to specify an absolute path for volume mounts inside your container.
Note that --docker-mount is mutually exclusive with --mount and is available only with the container method.

When --docker-mount is specified, instead of using sshfs to mount the remote filesystem on your host, a randomly named docker volume is created using the vieux/sshfs volume driver.
The volume is mounted in the user container at the mount point specified by --docker-mount.
As a side benefit, sudo is no longer required for the container method (unless you need sudo to run Docker).

One downside is that it is not possible to mount a subdirectory of the remote volumes at another location in your container.
This means that the /var/run/secrets workaround described above cannot be done with --docker-mount.
Using $TELEPRESENCE_ROOT is required.

Ensuring write permissions

If you are experiencing errors such as Operation not permitted or permission denied your processes might lack write permissions.
This happens because the volume mounted by Telepresence is owner by the privileged user (root, admin, etc), whereas the image might not have the same privileges.

Since Telepresence doesn't allow to you specify using a privileged image explicitly, as a workaround, you can expose a low port, i.e. less than 1024: --expose 9999:81
This works because a low port requires privileged network access, Telepresence detects this and chooses the privileged image, causing all the proxy pod processes to run as root.

File: docs/reference/community.md

Community

Ensuring Telepresence works well for everyone

We're always looking to collect more feedback and learn more about your use case for Telepresence. If you have questions, ask them in our Slack chatroom or file an issue on GitHub.

What other people are saying about Telepresence

[Telepresence for Local Development](https://blog.openshift.com/telepresence-local-development/) - Michael Hausenblas

OpenShift makes it easy to deploy your containers, but it can also impact your development cycle. This post covers how you can use Telepresence with OpenShift for faster local development.

[Run a local process in Kubernetes with Telepresence](https://blog.codersociety.com/run-a-local-process-in-kubernetes-with-telepresence-dc16bd593b52) - Kentaro Wakayama

This post walks through how the engineers at Coder Society use Telepresence in their Kubernetes development stack to gain access to running Kubernetes services without having to expose them publicly.

[Smarter local development of Kubernetes microservices with Telepresence](http://blog.stacktical.com/cloud-computing/2017/09/27/telepresence-kubernes-development.html) - [Stacktical](https://stacktical.com)

Installing projects locally as a developer has always been tedious, and microservices make that even more complicated. In this post, Stacktical talks about how Telepresence makes it super fast and easy for them to work amongst existing Kubernetes microservices.

[5 open source tools for microservices developers on Kubernetes](https://hackernoon.com/5-open-source-tools-for-microservices-developers-on-kubernetes-4fac73c2c6ee) - Richard Li

Kubernetes has powerful features for container scheduling, service discovery, and load balancing, but what tools exist for developers who are developing software on Kubernetes? This post covers 5 popular open source tools, including Telepresence!

[Using Telepresence for Kubernetes Development](https://youtu.be/G8fA_JssUG8) - Michael Hausenblas & Abhay Saxena

In this video interview, Michael Hausenblas of Red Hat interviews Abhay Saxena, the lead engineer on Telepresence.

[10 open-source Kubernetes tools for highly effective SRE and Ops Teams](https://abhishek-tiwari.com/10-open-source-tools-for-highly-effective-kubernetes-sre-and-ops-teams/) - Abhishek Tiwari

Abhishek Tiwari, head of engineering at Isentia, reviews 10 open source Kubernetes tools, including Telepresence.

[Productive Dev Environments for Kubernetes](https://youtu.be/4cct9JgItYQ) - Philip Lombardi

This talk from the NYC Kubernetes meetup covers strategies and trade-offs for setting up productive and realistic development environments for microservices and includes a demo of Telepresence.

[Writing golang applications designed to run in Kubernetes](https://github.com/number101010/go-kubernetes-vscode-dev) - number101010

This repo is a template to get started writing golang applications designed to run in Kubernetes and uses Telepresence to mimic the in-cluster environment.

[Towards a better service development story](https://blog.ironcladapp.com/towards-a-better-service-development-story-bae19a9b3c5a) - Dylan Scott

The team at Ironclad is using Telepresnece to develop individual Kubernetes services and improve their development workflow on Kubernetes.

File: docs/reference/connecting.md

Connecting to the cluster

Setting up the proxy

To use Telepresence with a cluster (Kubernetes or OpenShift, local or remote) you need to run a proxy inside the cluster.
There are three ways of doing so.

Creating a new deployment

By using the --new-deployment option telepresence can create a new deployment for you.
It will be deleted when the local telepresence process exits.
This is the default if no deployment option is specified.

For example, this creates a Deployment called myserver:

console

telepresence --new-deployment myserver --run-shell

This will create two Kubernetes objects, a Deployment and a Service, both named myserver.
(On OpenShift a DeploymentConfig will be used instead of Deployment.)
Or, if you don't care what your new Deployment is called, you can do:

console

telepresence --run-shell

If telepresence crashes badly enough (e.g. you used kill -9) you will need to manually delete the Deployment and Service that Telepresence created.

Swapping out an existing deployment

If you already have your code running in the cluster you can use the --swap-deployment option to replace the existing deployment with the Telepresence proxy.
When the telepresence process exits it restores the earlier state of the Deployment (or DeploymentConfig on OpenShift).

console

telepresence --swap-deployment myserver --run-shell

If you have more than one container in the pods created by the deployment you can also specify the container name:

console

telepresence --swap-deployment myserver:containername --run-shell

If telepresence crashes badly enough (e.g. you used kill -9) you will need to manually restore the Deployment.

Running Telepresence manually

You can also choose to run the Telepresence manually by starting a Deployment that runs the proxy in a pod.

The Deployment should only have 1 replica, and use the Telepresence different image:

apiVersion: extensions/v1beta1 kind: Deployment metadata: name: myservice spec: replicas: 1 # only one replica template: metadata: labels: name: myservice spec: containers: - name: myservice image: datawire/telepresence-k8s:{{ book['version'] }} # new image

You should apply this file to your cluster:

console

kubectl apply -f telepresence-deployment.yaml

Next, you need to run the local Telepresence client on your machine, using --deployment to indicate the name of the Deployment object whose pod is running telepresence/datawire-k8s:

console

telepresence --deployment myservice --run-shell

Telepresence will leave the deployment untouched when it exits.

Kubernetes contexts and namespaces

kubectl context

By default Telepresence uses whatever the current context is for kubectl.
If you want to choose a specific context you can use the --context option to telepresence.
For example:

console

telepresence --context minikube --run-shell

You can choose any context listed in kubectl config get-contexts.

If you've set a namespace for the context then that namespace will be used to find/create the Deployment, but you can also choose a namespace explicitly, as shown in the next section.

Kubernetes namespaces

If you want to proxy to a Deployment in a non-default namespace you can pass the --namespace argument to Telepresence:

console

telepresence --namespace yournamespace --swap-deployment yourservice --run-shell

Cluster permissions

Telepresence uses kubectl or oc to manipulate your Kubernetes/OpenShift cluster.
This means the user who invokes Telepresence needs the appropriate authorization. For Kubernetes, the following Role should be sufficient:

yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: telepresence-role
  namespace: restricted-ns
rules:
- apiGroups: [""]
  resources: ["services"]
  verbs: ["list", "create", "delete"]
- apiGroups: ["", "apps", "extensions"]
  resources: ["deployments"]
  verbs: ["list", "create", "get", "update", "delete"]
- apiGroups: ["", "apps", "extensions"]
  resources: ["deployments/scale"]
  verbs: ["get", "update"]
- apiGroups: ["", "apps", "extensions"]
  resources: ["replicasets"]
  verbs: ["list", "get", "update", "delete"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list", "get"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods/portforward"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]

File: docs/reference/developing.md

Development Guide

Known issues

Docs get updated off of master, so documentation on site may reference unreleased features if you're not careful.
Until that's fixed, releases should happen as soon as new feature is merged to master.
See issue #4.

Setting up a development environment

The following instructions will gets the Telepresence source and sets up some of its dependencies (torsocks, gcloud).
It also creates a virtualenv and installs Telepresence's Python dependencies into it.
The arguments required for environment-setup.sh are Google Cloud configuration items which identify a GKE cluster which can be used for testing, plus the operating system.

console

$ git clone [email protected]:telepresenceio/telepresence.git
$ cd telepresence
$ ./environment-setup.sh $PROJECT $CLUSTER $ZONE <linux|osx>
$ make virtualenv

You may want to activate the virtualenv (for the duration of your shell):

console

$ . virtualenv/bin/activate

This will give you access to the Telepresence executables:

telepresence
sshuttle-telepresence.

You can test your modifications to Telepresence with the build tool:

console

$ make check TELEPRESENCE_REGISTRY=<Docker registry for tag and push> [PYTEST_ARGS=]

You can run a subset of the tests using the pytest features for selecting tests
(for example, -k and -m).
End-to-end tests are marked with the method and operation they exercise.
So, for example, you can run all of the tests in the vpn-tcp, swap-deployment configuration:

console

$ make check TELEPRESENCE_REGISTRY=<Docker registry for tag and push> PYTEST_ARGS="-m 'vpn_tcp and swap_deployment'"

Note that - must be replaced with _ due to pytest limitations.

See make help for details about how to run specific tests.

You can also build images and push them to a registry without running any tests:

console

$ make docker-push TELEPRESENCE_REGISTRY=<Docker registry for tag and push>

If you want to push images to the local registry, start the container first:

console

$ docker run -d -p 5000:5000 --restart=always --name docker-local-registry registry

and then simply use TELEPRESENCE_REGISTRY=localhost:5000.

Or if you want to build images using minikube (untested):

console

$ eval $(minikube docker-env --shell bash)
$ make docker-push TELEPRESENCE_REGISTRY=<Docker registry for tag and push>

Or using minishift (untested):

console

$ eval $(minishift docker-env --shell bash)
$ make docker-push TELEPRESENCE_REGISTRY=<Docker registry for tag and push>

End-to-End Testing

The Telepresence test suite includes a set of tests which run Telepresence as a real user would.
These tests launch Telepresence and have it communicate with a real Kubernetes cluster, running real pods and observing the results.
These tests are implemented in tests/test_endtoend.py and tests/test_endtoend_distinct.py.

While the test functions themselves are present in these test modules,
there are several additional support modules also involved.
tests/probe_endtoend.py is a Python program which the tests tell Telepresence to run.
tests/parameterize_utils.py is a support module for writing tests.
tests/conftest.py integrates the tests with pytest.
At points during end-to-end test development you may find yourself working with any of these sources.

With the aim of making it clear how to write your own end-to-end test, here is one dissected.

Test Probe

python

from .conftest import (
    with_probe,
)

@with_probe
def test_demonstration(probe):

The end-to-end tests are written using a number of pytest features.
The first is parameterized fixtures to make it easy to apply a test to all Telepresence execution modes.

Notice that the test function is defined to take an argument probe.
The argument must be named probe to select the correct pytest fixture.
The with_probe decorator parameterizes the probe fixture with all of the Telepresence execution modes.
This means that pytest will call this test function many times with different values for probe.
For example, the test function will be called with a probe associated with a run of Telepresence given the --method=container --new-deployment arguments.
The test function is called once for each combination of method arguments and operations (--new-deployment, --swap-deployment, etc).

Probe Result

python

probe_environment = probe.result().result["environ"]

probe.result() will be used in every end-to-end test.
This method returns an object - a ProbeResult - representing the Telepresence run.
This may initiate a new run of Telepresence but it may also re-use the Telepresence launched by an earlier test with the same configuration (the same probe).
This is the result of pytest fixture optimization and it allows the test suite to run Telepresence far fewer times than would otherwise be required (reducing the overall runtime of the test suite).

The Telepresence probe collects some information from the Telepresence execution context immediately upon starting.
The result attribute of the ProbeResult provides access to this information.
In this case, we retrieve the complete POSIX environment for inspection.

Probe Operation

python

if probe.operation.inherits_deployment_environment():

This test now prepares to make its first assertion.
This first assertion is guarded by a check against the result of a method of probe.operation.
probe.operation is a reference to an object representing the operation which the probe used.
Remember that a test decorated with with_probe will be run multiple times with different probe arguments.
Many of those probes will be configured with a different operation.
This attribute lets a test vary its behavior based on that operation.
This is useful because different operations may have different desired behavior and require different assertions in their tests.
For more details about what can be done with the operation object, see tests/parameterize_utils.py where operations are implemented.

In this case, inherits_deployment_environment is a method of the operation which returns a boolean.
The result indicates whether it is expired and desired that the Telepresence execution context's POSIX environment inherits environment variables that were set in a pre-existing Kubernetes Deployment resource.
Not all Telepresence configurations interact with a pre-existing Deployment - hence the need for this check.

Supposing we are running with a probe where this check succeeds:

python

desired = probe.DESIRED_ENVIRONMENT
    expected = {
        k: probe_environment.get(k, None)
        for k in probe.DESIRED_ENVIRONMENT
    }
    assert desired == expected

Here the test makes an assertion about the observed POSIX environment.
It looks up the value of the environment which ought to have been inherited - probe.DESIRED_ENVIRONMENT - and makes sure the items all appear in the observed environment.

For the else case of this branch, we might assert that desired does not appear in the observed environment.

Probe Method

Test can also inspect the method in use.

python

if probe.method.inherits_client_environment():
        assert probe.CLIENT_ENV_VAR in probe_environment

The idea here is similar.
Different behavior may be desired from different methods.
Inspection of probe.method provides a way to vary the test behavior based on this.
Methods are implemented in tests/parameterize_utils.py.

Probe Interaction

The Telepresence process associated with the probe continues to run while the tests run.
This means interactions with it are possible.
Simple messages can easily be exchanged with the probe.

python

probe_result.write("probe-also-proxy {}".format(hostname))
    success, request_ip = loads(probe_result.read())

This uses a command supported by the probe which makes it issue an HTTP request to a particular URL and return the response.
These commands are implemented in probe_endtoend.py.
In this case, the result is a two-tuple.
The first element indicates whether the HTTP request succeeded or not.
The second element gives some data from the HTTP response (if it succeeded).

Probe commands are useful for observing any state or behavior which is only visible in the Telepresence execution context.
They allow the test to retrieve the information so assertions can be made.

Final Thoughts

When writing end-to-end tests keep a few things in mind:

Shared Telepresence

The probe fixture re-uses Probe instances.
Tests should not modify the Probe passed in for the probe argument.
Doing so will invalidate the results of subsequent tests.

Likewise, the Probe instance has an associated Telepresence process.
Tests should not modify that process, either, or subsequent tests will be invalidated.
This should be fairly easy since there's not much that can be done to "modify" a running Telepresence process.
One very obvious example, though, is that the process can be killed.
Don't do that.

End-to-end Debugging

When such a test fails,
the default is for it to present a low-information failure in the test suite result.
This may be the test suite hanging and being killed by a pytest timeout.
Or it may be Telepresence crashing and the full Telepresence log being dumped.
These kind of test failures are challenging to debug.
Be sure to examine all of the information available.
If not enough information is available, add logging to Telepresence or the test suite.
Do write your tests first, observe them fail, and improve their failure behavior before making them pass.

Unit Tests

End-to-end tests provide a highly realistic model of real-world Telepresence behavior.
However, they are not the only option and not always the best option.
For subtle logic (particularly involving many possible outcomes), unit tests may provide a lower-cost option.
A single end-to-end test to verify a gross code path combined with many unit tests to exercise all of the subtleties can provide the best of both worlds.

Coding standard

Formatting is enforced by the installed yapf tool; to reformat the code, you can do:

console

$ make format

Releasing Telepresence

Overview

Every commit to the master branch results in CI building a set of deployable artifacts: Docker images, Linux packages, a JSON blob for Scout, and a markdown blob for announcing a release on Slack et al.
The artifacts are available for download as a tarball telepresence-dist.tbz from the CircleCI artifacts tab on the deploy job page.
The release process pushes a set of those artifacts into production.

At the moment, the Linux packages are not tested, other than a minor smoke test. Package repositories are not tested at all.

Theory of operation

Recreate your Python virtual environment from scratch and re-run the linters.
This avoids the frustration of having your release fail in the lint stage in CI, which rebuilds its virtualenv every time.
rm -r virtualenv && make lint
Make sure docs/reference/changelog.md has changelog entries for the next release, and today's release date.
If changelog entries are in the newsfragments directory, use towncrier to construct the changelog update.
towncrier's version management is incompatible with the rest of the universe; specify the new version explicitly.
Make sure to commit your changes.
virtualenv/bin/towncrier --version 0.xx
# Edit the change log
git add docs/reference/changelog.md
git commit -m "Prep changelog for release 0.xx"
Mark the new version number for Telepresence by tagging in Git.
git tag -a 0.xx -m "Release 0.xx"
Push the new commit and tag to GitHub.
git push origin master --tags
Wait for CircleCI to finish.
Make sure all the test pass.
Download the tarball of deployable artifacts and unarchive into container in your project directory. It will populate the dist subdirectory.
curl -s https://.../telepresence-dist.tbz | tar xjf -
Set up release credentials in the environment:
. /keybase/team/datawireio/tel-release-secrets.sh
- HOMEBREW_KEY to push to GitHub
- PACKAGECLOUD_TOKEN to push Linux packages
- AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for AWS S3 access
Run the release script.
ci/release.sh
Log in to Netlify and rebuild the Telepresence website.
Select "Deploys" then "Trigger Deploy" then "Clear cache and deploy site" to get a clean build.
Without this step, more often than not, the website will display an old version number.
Post the release announcement on Slack et al.
The release script outputs the announcement, or you can find it in dist/announcement.md.

What the release script does

Build and launch a Docker container with required tools.
Upload Linux packages to PackageCloud using the script that's generated by the deployment script.
Update the Homebrew formula in homebrew-blackbird.
The Homebrew formula refers to the tarball GitHub generates for each release.
Push the Scout blobs to the datawire-static-files S3 bucket.
In the future, Telepresence will be able to inform users that a new version is available using this data.

Running tests

Full test suites

In order to run all possible code paths in Telepresence, you need to do the following:

Test environment	How to run
Minikube	`make minikube-test`
Remote K8s cluster	Runs on Circle
Minishift	`make openshift-tests` with minishift kube context
Remote OS cluster	`make openshift-tests` with remote OpenShift context
Docker on Mac	`make minikube-test` on Mac with Docker
Other Mac	Runs on Circle

In practice running on remote OpenShift cluster usually doesn't happen.

CircleCI on Mac does not yet support Docker, which is why that needs to be done manually.

Running individual tests

When doing local development you will typically run all tests by doing:

make minikube-test

If you want to only run some tests you can pass arguments to the underlying py.test run using TELEPRESENCE_TESTS.
For example, to run all tests containing the string "fromcluster" and to exit immediately after first failed test:

TELEPRESENCE_TESTS="-x -k fromcluster" make minikube-test

See py.test --help for other options you might want to set in TELEPRESENCE_TESTS.

Running a local copy of `telepresence`

FIXME: This is out-of-date. The above section of setting up a development environment has the correct info, but lacks a clear example like this section has.

During local development, typically against minikube, you will want to manually run telepresence you are working on.
You need to:

Make sure minikube has latest server-side Docker image: make build-k8s-proxy-minikube will do this. It has issues on Mac, however, due to old version of make maybe? Read the Makefile to see what it does.
If you forget this step you will have problems with Minikube not finding the Docker image for telepresence-k8s.
If you're using --docker-run, your local Docker needs to have the latest Docker image: make build-local.
You need to run cli/telepresence with env variable telling it the version number it should be using; this will be used as the tag for Docker images you created in steps 1 and 2. You do this by setting TELEPRESENCE_VERSION to the output of make version. You also need to set PATH so sshuttle-telepresence is found.

For example:

console

$ cli/telepresence --version
0.61

$ make version
0.61-1-gadd8818

$ make build-k8s-proxy-minikube
...

$ env PATH=$PATH:$PWD/virtualenv/bin/ TELEPRESENCE_VERSION=$(make version) \
  cli/telepresence --version
0.61-1-gadd8818

$ env PATH=$PATH:$PWD/virtualenv/bin/ TELEPRESENCE_VERSION=$(make version) \
  cli/telepresence --run-shell
@minikube|$ ...

File: docs/reference/install.md

Installing Telepresence

{% import "../macros.html" as macros %}
{{ macros.installSpecific("reference-page") }}

Dependencies

If you install Telepresence using a pre-built package, dependencies other than kubectl are handled by the package manager. If you install from source, you will also need to install the following software.

kubectl (OpenShift users can use oc)
Python 3.5 or newer
OpenSSH (the ssh command)
sshfs to mount the pod's filesystem
conntrack and iptables on Linux for the vpn-tcp method
torsocks for the inject-tcp method
Docker for the container method
sudo to allow Telepresence to
- modify the local network (via sshuttle and pf/iptables) for the vpn-tcp method
- run the docker command in some configurations on Linux
- mount the remote filesystem for access in a Docker container

2. Official Technical Reference & Guides (telepresenceio/docs)

Telepresence documentation

The Telepresence documentation "lives" at several different locations,
embedded in various Git repositories via git subtree. This
repository acts as a rendezvous point for each of those locations.

In this repository you will find release/v${X} branches that contain
the documentation for that release line of Telepresence:

release/v1 The documentation for the legacy Telepresence 0.x line.
release/v2.Y The documentation for a given Telepresence 2 release line.
release/v2 The documentation for ongoing Telepresence 2
development; this is the "mainline" documentation of yet-unreleased
Telepresence 2 versions. Because of procedures around when
subtrees get synced, this may lag behind what is in other
repositories.

Creating a new minor version

When you are adding a new minor version, you need to do a little bit of additional work.

In this repository, create an orphan release/v2.y branch:

text

git checkout --orphan release/v2.y

create some file in the repo (it doesn't really matter what, it will be overwritten later)
and then push that branch.

Once this is done, in the Telepresence repo
you need to create the new sub-directory as a subrepo

text

git subrepo clone https://github.com/telepresenceio/docs docs/v2.y -b release/v2.y`

Now you can write docs in the Telepresence repo for release/v2.y and push those docs
to this repo using the mechanisms that exist there.

Docker & Self-Hosted Deployment Verified OCI

1. Project Overview & Quickstart (telepresenceio/telepresence) Copy Chapter

File: README.md Copy Chapter

Telepresence: Fast, Local Development for Kubernetes

Key Features Copy Chapter

Getting Started Copy Chapter

How It Works Copy Chapter

Community Copy Chapter

Sponsors Copy Chapter

Contributing Copy Chapter

License Copy Chapter

File: docs/discussion/how-it-works.md Copy Chapter

How it works

Goals Copy Section

How it works Copy Section

vpn-tcp method in detail Copy Section

minikube/minishift and Docker Desktop

inject-tcp method in detail Copy Section

Docker method in detail Copy Section

File: docs/discussion/overview.md Copy Chapter

Introduction to Telepresence

How it works Copy Chapter

Alternatives to Telepresence Copy Chapter

Getting started Copy Chapter

File: docs/discussion/why-telepresence.md Copy Chapter

Why Telepresence?

The slow status quo Copy Section

A fast development cycle with Telepresence Copy Section

Other options Copy Section

File: docs/howto/golang.md Copy Chapter

Go support

A Go program talking to Kubernetes Copy Section

Kubernetes talks to a Go program Copy Section

File: docs/howto/volumes.md Copy Chapter

Volume access

Volume access via Docker volume for the container method Copy Chapter

Ensuring write permissions Copy Chapter

File: docs/reference/community.md Copy Chapter

Community

Ensuring Telepresence works well for everyone Copy Section

What other people are saying about Telepresence Copy Section

[Telepresence for Local Development](https://blog.openshift.com/telepresence-local-development/) - Michael Hausenblas

[Run a local process in Kubernetes with Telepresence](https://blog.codersociety.com/run-a-local-process-in-kubernetes-with-telepresence-dc16bd593b52) - Kentaro Wakayama

[Smarter local development of Kubernetes microservices with Telepresence](http://blog.stacktical.com/cloud-computing/2017/09/27/telepresence-kubernes-development.html) - [Stacktical](https://stacktical.com)

[5 open source tools for microservices developers on Kubernetes](https://hackernoon.com/5-open-source-tools-for-microservices-developers-on-kubernetes-4fac73c2c6ee) - Richard Li

[Using Telepresence for Kubernetes Development](https://youtu.be/G8fA_JssUG8) - Michael Hausenblas & Abhay Saxena

[10 open-source Kubernetes tools for highly effective SRE and Ops Teams](https://abhishek-tiwari.com/10-open-source-tools-for-highly-effective-kubernetes-sre-and-ops-teams/) - Abhishek Tiwari

[Productive Dev Environments for Kubernetes](https://youtu.be/4cct9JgItYQ) - Philip Lombardi

[Writing golang applications designed to run in Kubernetes](https://github.com/number101010/go-kubernetes-vscode-dev) - number101010

[Towards a better service development story](https://blog.ironcladapp.com/towards-a-better-service-development-story-bae19a9b3c5a) - Dylan Scott

If you're interested in sharing your use case for Telepresence, we'd love to hear it! You can reach out to us in the [Slack chatroom](https://a8r.io/slack) or [submit a pull request](https://github.com/telepresenceio/telepresence/pulls).

File: docs/reference/connecting.md Copy Chapter

Connecting to the cluster

Setting up the proxy Copy Chapter

Creating a new deployment Copy Section

Swapping out an existing deployment Copy Section

Running Telepresence manually Copy Section

Kubernetes contexts and namespaces Copy Chapter

kubectl context Copy Section

Kubernetes namespaces Copy Section

Cluster permissions Copy Chapter

File: docs/reference/developing.md Copy Chapter

Development Guide

Known issues Copy Section

Setting up a development environment Copy Section

End-to-End Testing Copy Section

Test Probe

Probe Result

Probe Operation

Probe Method

Probe Interaction

Final Thoughts

Shared Telepresence

End-to-end Debugging

Unit Tests

Coding standard Copy Section

Releasing Telepresence Copy Section

Overview

Theory of operation

What the release script does

1. Project Overview & Quickstart (telepresenceio/telepresence)

File: README.md

Key Features

Getting Started

How It Works

Community

Sponsors

Contributing

License

File: docs/discussion/how-it-works.md

Goals

How it works

vpn-tcp method in detail

inject-tcp method in detail

Docker method in detail

File: docs/discussion/overview.md

How it works

Alternatives to Telepresence

Getting started

File: docs/discussion/why-telepresence.md

The slow status quo

A fast development cycle with Telepresence

Other options

File: docs/howto/golang.md

A Go program talking to Kubernetes

Kubernetes talks to a Go program

File: docs/howto/volumes.md

Volume access via Docker volume for the container method

Ensuring write permissions

File: docs/reference/community.md

Ensuring Telepresence works well for everyone

What other people are saying about Telepresence

File: docs/reference/connecting.md

Setting up the proxy

Creating a new deployment

Swapping out an existing deployment

Running Telepresence manually

Kubernetes contexts and namespaces

kubectl context

Kubernetes namespaces

Cluster permissions

File: docs/reference/developing.md

Known issues

Setting up a development environment

End-to-End Testing

Coding standard

Releasing Telepresence

Running tests

Running a local copy of `telepresence`

File: docs/reference/install.md

Dependencies

2. Official Technical Reference & Guides (telepresenceio/docs)