File: README.md
Kraken is a P2P-powered Docker registry that focuses on scalability and availability. It is
designed for Docker image management, replication, and distribution in a hybrid cloud environment.
With pluggable backend support, Kraken can easily integrate into existing Docker registry setups
as the distribution layer.
Kraken has been in production at Uber since early 2018. In our busiest cluster, Kraken distributes
more than 1 million blobs per day, including 100k 1G+ blobs. At its peak production load, Kraken
distributes 20K 100MB-1G blobs in under 30 sec.
Below is the visualization of a small Kraken cluster at work:
Table of Contents
- Features
- Design
- Architecture
- Benchmark
- Usage
- Comparison With Other Projects
- Limitations
- Contributing
- Contact
Features
Following are some highlights of Kraken:
- Highly scalable. Kraken is capable of distributing Docker images at > 50% of the max download
speed limit on every host. Cluster size and image size do not have a significant impact on
download speed.- Supports at least 15k hosts per cluster.
- Supports arbitrarily large blobs/layers. We normally limit max size to 20G for the best performance.
- Highly available. No component is a single point of failure.
- Secure. Support uploader authentication and data integrity protection through TLS.
- Pluggable storage options. Instead of managing data, Kraken plugs into reliable blob storage
options, like S3, GCS, ECR, HDFS or another registry. The storage interface is simple and new
options are easy to add. - Lossless cross-cluster replication. Kraken supports rule-based async replication between
clusters. - Minimal dependencies. Other than pluggable storage, Kraken only has an optional dependency on
DNS.
Design
The high-level idea of Kraken is to have a small number of dedicated hosts seeding content to a
network of agents running on each host in the cluster.
A central component, the tracker, will orchestrate all participants in the network to form a
pseudo-random regular graph.
Such a graph has high connectivity and a small diameter. As a result, even with only one seeder and
having thousands of peers joining in the same second, all participants can reach a minimum of 80%
max upload/download speed in theory (60% with current implementation), and performance doesn't
degrade much as the blob size and cluster size increase. For more details, see the team's tech
talk at KubeCon + CloudNativeCon.
Architecture
- Agent
- Deployed on every host
- Implements Docker registry interface
- Announces available content to tracker
- Connects to peers returned by the tracker to download content
- Origin
- Dedicated seeders
- Stores blobs as files on disk backed by pluggable storage (e.g. S3, GCS, ECR)
- Forms a self-healing hash ring to distribute the load
- Tracker
- Tracks which peers have what content (both in-progress and completed)
- Provides ordered lists of peers to connect to for any given blob
- Proxy
- Implements Docker registry interface
- Uploads each image layer to the responsible origin (remember, origins form a hash ring)
- Uploads tags to build-index
- Build-Index
- Mapping of the human-readable tag to blob digest
- No consistency guarantees: the client should use unique tags
- Powers image replication between clusters (simple duplicated queues with retry)
- Stores tags as files on disk backed by pluggable storage (e.g. S3, GCS, ECR)
Benchmark
The following data is from a test where a 3G Docker image with 2 layers is downloaded by 2600 hosts
concurrently (5200 blob downloads), with 300MB/s speed limit on all agents (using 5 trackers and
5 origins):
- p50 = 10s (at speed limit)
- p99 = 18s
- p99.9 = 22s
Usage
All Kraken components can be deployed as Docker containers. To build the Docker images:
$ make imagesFor information about how to configure and use Kraken, please refer to the documentation.
Kraken on Kubernetes
You can use our example Helm chart to deploy Kraken (with an example HTTP fileserver backend) on
your k8s cluster:
$ helm install --name=kraken-demo ./helmOnce deployed, every node will have a docker registry API exposed on localhost:30081.
For example pod spec that pulls images from Kraken agent, see example.
For more information on k8s setup, see README.
Devcluster
To start a herd container (which contains origin, tracker, build-index and proxy) and two agent
containers with development configuration:
$ make devclusterDocker-for-Mac is required for making dev-cluster work on your laptop.
For more information on devcluster, please check out devcluster README.
Comparison With Other Projects
Dragonfly from Alibaba
Dragonfly cluster has one or a few "supernodes" that coordinates the transfer of every 4MB chunk of data
in the cluster.
While the supernode would be able to make optimal decisions, the throughput of the whole cluster is
limited by the processing power of one or a few hosts, and the performance would degrade linearly as
either blob size or cluster size increases.
Kraken's tracker only helps orchestrate the connection graph and leaves the negotiation of actual data
transfer to individual peers, so Kraken scales better with large blobs.
On top of that, Kraken is HA and supports cross-cluster replication, both are required for a
reliable hybrid cloud setup.
BitTorrent
Kraken was initially built with a BitTorrent driver, however, we ended up implementing our P2P
driver based on BitTorrent protocol to allow for tighter integration with storage solutions and more
control over performance optimizations.
Kraken's problem space is slightly different than what BitTorrent was designed for. Kraken's goal is
to reduce global max download time and communication overhead in a stable environment, while
BitTorrent was designed for an unpredictable and adversarial environment, so it needs to preserve more
copies of scarce data and defend against malicious or bad behaving peers.
Despite the differences, we re-examine Kraken's protocol from time to time, and if it's feasible, we
hope to make it compatible with BitTorrent again.
Limitations
- If Docker registry throughput is not the bottleneck in your deployment workflow, switching to
Kraken will not magically speed up yourdocker pull. To speed updocker pull, consider
switching to Makisu to improve layer reusability at build time, or
tweak compression ratios, asdocker pullspends most of the time on data decompression. - Mutating tags (e.g. updating a
latesttag) is allowed, however, a few things will not work: tag
lookups immediately afterwards will still return the old value due to Nginx caching, and replication
probably won't trigger. We are working on supporting this functionality better. If you need tag
mutation support right now, please reduce the cache interval of the build-index component. If you also need
replication in a multi-cluster setup, please consider setting up another Docker registry as Kraken's
backend. - Theoretically, Kraken should distribute blobs of any size without significant performance
degradation, but at Uber, we enforce a 20G limit and cannot endorse the production use of
ultra-large blobs (i.e. 100G+). Peers enforce connection limits on a per blob basis, and new peers
might be starved for connections if no peers become seeders relatively soon. If you have ultra-large
blobs you'd like to distribute, we recommend breaking them into <10G chunks first.
Contributing
Please check out our guide.
Contact
To contact us, please join our Slack channel.
File: docs/ARCHITECTURE.md
Kraken Core
Central P2P components that's not specific to docker images:
Agent
- Deployed on every host
- Implements Docker registry interface
Origin
- Dedicated seeders
- Pluggable storage backend (e.g. S3)
- Self-healing hash ring
Tracker
- Tracks peers and seeders, instructs them to form a sparse graph
- Self-healing hash ring
Kraken Proxy and Build Index
Components responsible for image tags and replication to other clusters:
Proxy
- Handles image upload and direct download
Build Index
- Mapping of human readable tag to blob hash (digest)
- No consistency guarantees, client should use unique tags
- Powers image replication between clusters. Simple duplicated queues with retry
- Pluggable storage
- Self-healing hash ring
File: docs/CONFIGURATION.md
Table of Contents
- Examples
- Configuring Peer To Peer Download
- Configuring Hash Ring
- Configuring Storage Backend For Origin And Build-Index
Examples
Here are some example configuration files we used for dev cluster (which can be started by runningmake devcluster).
They are split into a base.yaml that contains configs that we have been using for test, development
and production, and a development.yaml that contains configs specifically needed for starting dev
cluster using Docker-for-Mac, and need to updated for production setups.
Origin
Tracker
Build-index
Proxy
Agent
More in examples/devcluster/README.md
Configuring Peer To Peer Download
Kraken's peer-to-peer network consists of agents, origins and trackers. Origins are special dedicated peers that seed data from a storage backend (HDFS, S3, etc). Agents are peers that download from each other and from origins. Agents periodically announce each torrent they are currently downloading to tracker, and in return, receive a list of peers that are also seeding the same torrent. More details in ARCHITECTURE.md
Tracker Peer TTL
tracker.yaml
yamlpeerstore: redis: peer_set_window_size: 1h max_peer_set_windows: 5
As peers announce periodically to a tracker, the tracker stores the announce requests into several time window bucket.
Each announce request expires in peer_set_window_size * max_peer_set_windows time.
Then, the tracker returns a random set of peers selecting from max_peer_set_windows number of time bucket.
Announce Interval `TODO(evelynl94)`
Bandwidth
Download and upload bandwidths are configurable to prevent peers from saturating the host network.
agent.yaml/origin.yaml
yamlscheduler: conn: bandwidth: enable: true egress_bits_per_sec: 1677721600 # 200*8 Mbit ingress_bits_per_sec: 2516582400 # 300*8 Mbit
Connection Limits
Number of connections per torrent can be limited by:
agent.yaml/origin.yaml
yamlscheduler: connstate: max_open_conn: 10
There is no limit on number of torrents a peer can download simultaneously.
Pipeline limit `TODO(evelynl94)`
Seeder TTI
SeederTTI (time-to-idle) is the duration a completed torrent will exist without being read from before being removed from in-memory archive.
agent.yaml/origin.yaml
yamlscheduler: seeder_tti: 5m
However, until it is deleted by periodic storage purge, completed torrents will remain on disk and can be re-opened on another peer's request.
Torrent TTI On Disk
Both agents and origins can be configured to cleanup idle torrents on disk periodically.
agent.yaml/origin.yaml
yamlstore: cache_cleanup: tti: 6h download_cleanup: tti: 6h
For origins, the number of files can also be limited as origins are dedicated seeders and hence normally caches files on disk for longer time.
origin.yaml
yamlstore: capacity: 1000000
Configuring Hash Ring
Both origin and tracker clusters are self-healing hash rings and both can be represented by either a dns name or a static list of hosts.
We use rendezvous hashing for constructing ring membership.
Take an origin cluster for example:
origin-static-hosts.yaml
yamlhashring: max_replica: 2 cluster: hosts: static: - origin1:15002 - origin2:15002 - origin3:15002origin-dns.yaml
yamlhashring: max_replica: 2 cluster: hosts: dns: origin.example.com:15002
Health Check For Hash Rings
When a node in the hash ring is considered as unhealthy, the ring client will route requests to the next healthy node with the highest score. There are two ways to do health check:
Active Health Check
Origins do health check for each other in the ring as the cluster is usually smaller.
origin.yaml
yamlcluster: healthcheck: filter: fails: 3 passes: 2 monitor: interval: 30s
Above configures health check ping from one origin to others every 30 seconds. If 3 or more consecutive health checks fail for an origin, it is marked as unhealthy. Later, if 2 or more consecutive health checks succeed for the same origin, it is marked as healthy again. Initially, all hosts are healthy.
Passive Health Check
Agents health checks tracker, piggybacking on the announce requests.
agent.yaml
yamltracker: cluster: healthcheck: fails: 3 fail_timeout: 5m
As shown in this example, if 3 announce requests to one tracker fail with network error within 5 minutes, the host is marked as unhealthy for 5 minutes. The agent will not send requests to this host until after timeout.
Configuring Storage Backend For Origin And Build-Index
Storage backends are used by Origin and Build-Index for data persistence. Kraken has support for S3, GCS, ECR, HDFS, http (readonly), and Docker Registry (readonly) as backends.
Multiple backends can be used at the same time, configured based on namespaces of requested blob and tag (for docker images, that means the part of image name before ":").
Example origin config that uses multiple backends:
origin.yaml
yamlbackends: - namespace: library/.* backend: registry_blob: address: index.docker.io timeout: 60s security: basic: username: "" password: "" - namespace: test-domain/.* backend: http: download_url: http://test-domain:9000/download?sha256=%s download_backoff: enabled: true - namespace: ecr-images/.* backend: registry_tag: address: 123456789012.dkr.ecr.<region>.amazonaws.com security: credsStore: 'ecr-login' - namespace: s3-images/.* backend: s3: region: us-west-1 bucket: test-bucket root_directory: /test-bucket/kraken/default/ name_path: sharded_docker_blob username: kraken-user - namespace: minio-images/.* backend: s3: region: us-east-1 bucket: self-hosted-bucket root_directory: /kraken/default/ name_path: sharded_docker_blob username: minio-user endpoint: http://172.17.0.1:9000 disable_ssl: true force_path_style: true bandwidth: enable: true - namespace: gcs-images/.* backend: gcs: username: kraken-user bucket: test-bucket root_directory: /test-bucket/kraken/default/ name_path: sharded_docker_blob bandwidth: enable: true auth: s3: kraken-user: s3: aws: kraken-user aws_access_key_id: <keyid> aws_secret_access_key: <key> minio-user: s3: aws: minio-user aws_access_key_id: <keyid> aws_secret_access_key: <key> gcs: kraken-user: gcs: access_blob: <service_account_key>
Read-Only Registry Backend
For simple local testing with an insecure registry (assuming it listens on host.docker.internal:5000), you can configure the backend for origin and build-index accordingly:
origin.yaml
yamlbackends: - namespace: .* backend: registry_blob: address: host.docker.internal:5000 security: tls: client: disabled: true
build-index.yaml
yamlbackends: - namespace: .* backend: registry_tag: address: host.docker.internal:5000 security: tls: client: disabled: true
Bandwidth on Origin
When transferring data from and to its storage backend, origins can be configured with download and upload bandwidths. This is useful when using cloud storage providers to prevent origins from saturating the network link.
origin.yaml
yamlbackends: - namespace: .* backend: s3: <omitted> bandwidth: enabled: true egress_bits_per_sec: 8589934592 # 8 Gbit ingress_bits_per_sec: 85899345920 # 10*8 Gbit
File: docs/CONTRIBUTING.md
Contributing To Kraken
Issues
Please feel free to submit new issues.
Contributing
Start by reading our coding style guidelines.
Please follow standard fork-and-pull workflow.
- Fork the repo on GitHub
- Clone the project locally
- Commit changes to your own branch
- Push the change back to your fork
- Submit a Pull request. We will review and merge your change.
Setup
Most tests and scripts assumes the developer to have Docker installed locally.
To install dependencies:
$ make vendorTo run unit tests:
$ make unit-testTo run integration tests:
$ make integrationTo build docker images:
$ make imagesLinting
Krake uses golangci-lint for checking Go code quality.
Prerequisite
Install golangci-lint and ensure it is in your PATH. (macOS example: brew install golangci-lint).
Official instructions: https://golangci-lint.run/docs/welcome/install/
Quick Checklist
- Install
golangci-lint. - Verify:
golangci-lint version. - Install Git hooks (runs lint automatically on each commit):
make install-hooks - Make changes and commit (the hook blocks commits on lint errors).
- If the hook fails, fix the reported issues and retry the commit.
File: docs/ENDPOINTS.md
Table of Contents
Push And Pull Docker Images
Kraken proxy implements all docker registry V2 endpoints.
Kraken agent only implements the GET and HEAD endpoints of docker registry.
Pushing Docker Images To Kraken Proxy
You can push images to proxy using normal docker push command:
docker push {proxy_host}:{proxy_port}/{repo}:{tag}Proxy will then upload the blobs and tag to origin and build-index.
Yon can also pull from proxy directly without going through the p2p network:
docker pull {proxy_host}:{proxy_port}/{repo}:{tag}Note: if you didn't configure Kraken with TLS, you might need to update your docker daemon config to whitelist kraken-proxy as an insecure registry first.
Pulling Docker Images From Kraken Agent
To pull docker images from local kraken agent, run:
docker pull localhost:{agent_registry_port}/{repo}:{tag}Note: kraken agent use different ports for docker registry endpoints and generic content addressable blobs. Please make sure you are using the port configured via agent_registry_port.
Upload and Download Generic Content Addressable Blobs
Kraken's usecase is not limited to docker images.
It exposes a separate set of endpoints for uploading and downloading generic content addressable blobs that's identified by SHA256 hash of the content.
Kraken proxy and build-index are not used in this usecase.
Uploads will be handled by Kraken origin, and downloads still go through Kraken agent.
Uploading Blobs To Kraken Origin
Blobs can be uploaded through the Kraken origin cluster and into your storage backend. All blobs
must be uploaded in chunks, using the following API:
POST /namespace/<namespace>/blobs/<digest>/uploadsStarts the upload. Returns an upload id in the "Location" response header, which is used for
uploading chunks of the blob.
PATCH /namespace/<namespace>/blobs/<digest>/uploads/Uploads a chunk of the blob, using the previously returned upload id. Supply the chunk bytes in the
request body. The "Content-Range" header in the request must specify the range of the chunk
being uploaded, for example:
Content-Range: 128,256This would upload the request body to bytes [128, 256) of the blob.
PUT /namespace/<namespace>/blobs/<digest>/uploads/?through=Commits the upload. If through is set to true, the blob will be uploaded through the origin
cluster and into the storage backend configured for namespace.
Downloading Blobs From Kraken Agent
GET /namespace/<namespace>/blobs/<digest>Once Kraken has been configured with your storage information and namespace, you can download your
blobs from any host with a Kraken agent running on it. The download endpoint takes your namespace
and the content digest of the blob you want to download, and blocks until agent has downloaded the
blob to its on-disk cache. Once the blob is downloaded locally, status 200 is returned and the
blob content is streamed over the response body.
Error codes:
- 404: Blob was not found in your storage backend.
- 5xx: Something went wrong. Check the response body for an error message, or reach out to the
Kraken team.
File: docs/index.md
Kraken is a P2P-powered Docker registry that focuses on scalability and availability. It is
designed for Docker image management, replication and distribution in a hybrid cloud environment.
With pluggable backend support, Kraken can easily integrate into existing Docker registry setups
as the distribution layer.
Kraken has been in production at Uber since early 2018. In our busiest cluster, Kraken distributes
more than 1 million blobs per day, including 100k 1G+ blobs. At its peak production load, Kraken
distributes 20K 100MB-1G blobs in under 30 sec.
Below is the visualization of a small Kraken cluster at work:
Table of Contents
- Features
- Design
- Architecture
- Benchmark
- Usage
- Comparison With Other Projects
- Limitations
- Contributing
- Contact
Features
Following are some highlights of Kraken:
- Highly scalable. Kraken is capable of distributing Docker images at > 50% of max download
speed limit on every host. Cluster size and image size do not have significant impact on download
speed.- Supports at least 15k hosts per cluster.
- Supports arbitrarily large blobs/layers. We normally limit max size to 20G for best performance.
- Highly available. No component is a single point of failure.
- Secure. Support uploader authentication and data integrity protection through TLS.
- Pluggable storage options. Instead of managing data, Kraken plugs into reliable blob storage
options, like S3, GCS, HDFS or another registry. The storage interface is simple and new options
are easy to add. - Lossless cross cluster replication. Kraken supports rule-based async replication between
clusters. - Minimal dependencies. Other than pluggable storage, Kraken only has an optional dependency on
DNS.
Design
The high level idea of Kraken is to have a small number of dedicated hosts seeding content to a
network of agents running on each host in the cluster.
A central component, the tracker, will orchestrate all participants in the network to form a
pseudo-random regular graph.
Such a graph has high connectivity and small diameter. As a result, even with only one seeder and
having thousands of peers joining in the same second, all participants can reach a mininum of 80%
max upload/download speed in theory (60% with current implementation), and performance doesn't
degrade much as the blob size and cluster size increase. For more details, see the team's tech
talk at KubeCon + CloudNativeCon.
Architecture
- Agent
- Deployed on every host
- Implements Docker registry interface
- Announces available content to tracker
- Connects to peers returned by tracker to download content
- Origin
- Dedicated seeders
- Stores blobs as files on disk backed by pluggable storage (e.g. S3, GCS)
- Forms a self-healing hash ring to distribute load
- Tracker
- Tracks which peers have what content (both in-progress and completed)
- Provides ordered lists of peers to connect to for any given blob
- Proxy
- Implements Docker registry interface
- Uploads each image layer to the responsible origin (remember, origins form a hash ring)
- Uploads tags to build-index
- Build-Index
- Mapping of human readable tag to blob digest
- No consistency guarantees: client should use unique tags
- Powers image replication between clusters (simple duplicated queues with retry)
- Stores tags as files on disk backed by pluggable storage (e.g. S3, GCS)
Benchmark
The following data is from a test where a 3G Docker image with 2 layers is downloaded by 2600 hosts
concurrently (5200 blob downloads), with 300MB/s speed limit on all agents (using 5 trackers and
5 origins):
- p50 = 10s (at speed limit)
- p99 = 18s
- p99.9 = 22s
Usage
All Kraken components can be deployed as Docker containers. To build the Docker images:
$ make imagesFor information about how to configure and use Kraken, please refer to the documentation.
Kraken on Kubernetes
You can use our example Helm chart to deploy Kraken (with an example http fileserver backend) on
your k8s cluster:
$ helm install --name=kraken-demo ./helmOnce deployed, each and every node will have a docker registry API exposed on localhost:30081.
For an example pod spec that pulls images from Kraken agent, see example.
For more information on k8s setup, see README.
Devcluster
To start a herd container (which contains origin, tracker, build-index and proxy) and two agent
containers with development configuration:
$ make devclusterDocker-for-Mac is required for making dev-cluster work on your laptop.
For more information on devcluster, please check out devcluster README.
Comparison With Other Projects
Dragonfly from Alibaba
Dragonfly cluster has one or a few "supernodes" that coordinates transfer of every 4MB chunk of data
in the cluster.
While the supernode would be able to make optimal decisions, the throughput of the whole cluster is
limited by the processing power of one or a few hosts, and the performance would degrade linearly as
either blob size or cluster size increases.
Kraken's tracker only helps orchestrate the connection graph, and leaves negotiation of actual data
transfer to individual peers, so Kraken scales better with large blobs.
On top of that, Kraken is HA and supports cross cluster replication, both are required for a
reliable hybrid cloud setup.
BitTorrent
Kraken was initially built with a BitTorrent driver, however we ended up implementing our own P2P
driver based on BitTorrent protocol to allow for tighter integration with storage solutions and more
control over performance optimizations.
Kraken's problem space is slightly different than what BitTorrent was designed for. Kraken's goal is
to reduce global max download time and communication overhead in a stable environment, while
BitTorrent was designed for an unpredictable and adversarial environment, so it needs to preserve more
copies of scarce data and defend against malicious or bad behaving peers.
Despite the differences, we re-examine Kraken's protocol from time to time, and if it's feasible, we
hope to make it compatible with BitTorrent again.
Limitations
- If Docker registry throughput is not the bottleneck in your deployment workflow, switching to
Kraken will not magically speed up yourdocker pull. To actually speed updocker pull, consider
switching to Makisu to improve layer reusability at build time, or
tweak compression ratios, asdocker pullspends most of the time on data decompression. - Mutating tags (e.g. updating a
latesttag) is allowed, however a few things will not work: tag
lookups immediately afterwards will still return the old value due to Nginx caching, and replication
probably won't trigger. We are working on supporting this functionality better. If you need tag
mutation support right now, please reduce cache interval of build-index component. If you also need
replication in a multi-cluster setup, please consider setting up another Docker registry as Kraken's
backend. - Theoretically, Kraken should distribute blobs of any size without significant performance
degradation, but at Uber we enforce a 20G limit and cannot endorse of the production use of
ultra-large blobs (i.e. 100G+). Peers enforce connection limits on a per blob basis, and new peers
might be starved for connections if no peers become seeders relatively soon. If you have ultra-large
blobs you'd like to distribute, we recommend breaking them into <10G chunks first.
Contributing
Please check out our guide.
Contact
To contact us, please join our Slack channel.
File: docs/INTEGRATEWITHHARBOR.md
Concept
Harbor is the only container repository project in CNCF. And it supports image management, replication, CVE scan and so on. It could be used in production.
Kraken is also a container repository with P2P distribution. It supports S3, HDFS, docker registry as backend storage.
I will introduce how to integrate Harbor and Kraken here. And using the image management function of Harbor and the P2P distribution function of Kraken both.
Architecture
So you can see that Harbor and Kraken share the same docker registry as backend storage. It will decouple Harbor and Kraken if they just share the storage but not higher level sharing. So we can push images into Harbor and pull them from Harbor or kraken.
Deployment
Here we use helm chart from Harbor community to deploy them. and also helm chart from Kraken community to deploy them.
We should modify two config file additionally,
- modify the backend of kraken-origin and kraken-buildindex and make it refer to harbor-registry.
- modify config file of harbor-registry. add another notification hook for kraken.
As following,
harbor-registry configmap
## add the second endpoint in notifications
notifications:
endpoints:
- name: harbor
disabled: false
url: http://harbor-core.repo.svc.cluster.local/service/notifications
timeout: 3000ms
threshold: 5
backoff: 1s
- name: kraken
disabled: false
url: http://kraken-proxy.p2p.svc.cluster.local:10050/registry/notifications
timeout: 3000ms
threshold: 5
backoff: 1skraken-origin,kraken-buildindex,
backends:
- namespace: .*
backend:
registry_blob:
address: harbor-registry.repo.svc.cluster.local:5000
security:
basic:
username: "admin"
password: "XXXXX"You should add more configuration about TLS if the domain name of your Harbor use a self-signed SSL certificate. It will throws a X509 error if not.
backends:
- namespace: .*
backend:
registry_blob:
address: harbor-registry.repo.svc.cluster.local:5000
security:
basic:
username: "admin"
password: "XXXXX"
tls:
client:
cert:
path: /etc/certs/XXX.crt
key:
path: /etc/certs/XXX.key
cas:
- path: /etc/certs/ca.crtYou can pull images using a localhost domain by P2P distribution after you deploy kraken-agent in your k8s nodes using daemonSet.
For example, there is an image called hub.harbor.com/library/debian:latest, then you can pull it using localhost:13000/library/debian:latest.
Work flow
I describe the work flow of pushing and pulling images briefly here,
- User push an image named
docker push hub.harbor.com/library/debian:latestto Harbor. - Harbor-registry will trigger a notification of pushManifest event to kraken-proxy.
- Kraken-proxy will fetch the manifest and notify kraken-origin to cache related blobs after receiving the pushManifest event.
- User will try to pull image by
docker pull localhost:13000/library/debian:latest - Then the P2P distribution work flow. but kraken-origin has cached these blobs before P2P distribution starts. it will save the time of fetching blobs from harbor-registry.
- Pulling image is completed.
Use Case
It has been a long time that this solution is used in Qingzhou Platform in NetEase Cloud.
Now we use Harbor for managing images and Kraken for distributing images, and it resolves the problem of distribution of huge number of images. there are thousands of images distributed everyday in our Qingzhou Platform.
Notes
It is important to limit the resource quota of kraken-agent in k8s for it is deployed with containers of business. It will affect the containers of business if kraken-agent takes too much resource.
There are two aspects for resource quota. One is the resource limit of k8s pod, the other one is the bandwidth.
File: docs/ROADMAP.md
Performance
Kraken's performance goal is to be capable of distributing a 100GB blob to 10k hosts concurrently at
80% of host nic limit.
Currently, Kraken works the best if blob is smaller than 10G, so some peers could become seeders
soon, and help reduce load on origin cluster.
If the blob is too big, peers joined later will spend extra effort to find openings in the network,
and the topology they form would not be very balanced, negatively impacting download speed.
To support extra large blobs, one possible solution is to let the cluster periodically rebalance
itself to converge to a random regular graph, which in theory could guarantee high download speed
for all participants.
Security
Kraken supports bi-directional TLS between all components for image tags, and bi-directional TLS
between image builder and Kraken for all data.
Uploaders can be autherized; and since blobs are content addressable, data integrity is protected.
However, blob content in torrent traffic is not encrypted, so it's still vulnerable to eavesdropping.
We plan to support TLS on torrent traffic.
Kubernetes Integration
We are looking into integrating Kraken with Kubernetes, so Kraken can preheat hosts at the beginning
of rolling upgrades, and speed up deployments.
This would require a Kubernetes scheduler that supports in-place upgrade.
Tag Mutation
Mutating tags is allowed, but the behavior is undefined. Replication probably won't trigger, and
most tag lookups could still return the old tag due to caching.
We plan to support tag mutation.
Visualization
Current visualization tool is based on agent logs. If feasible, we hope to support realtime
visualization.
BitTorrent Compatibility
Kraken's torrent library is based on a simplified version of BitTorrent, however it is not
compatible with the BitTorrent protocol. We may investigate BitTorrent compatibility in the future.
File: docs/STYLEGUIDE.md
Code style guidelines
This project adheres to the official Go style guidelines: https://github.com/golang/go/wiki/CodeReviewComments
Some things we are very picky about:
Variable naming
Please read: https://talks.golang.org/2014/names.slide#1
Comments
All comments, regardless of location, must have proper grammar, including proper capitalization and punctuation.
// this is a bad comment with bad grammar
// This is a good comment with good grammar.Avoid comments that state the obvious or repeat code logic. Comments can easily get out of sync with
the code and can mislead rather than help.
Line length
All lines of code should be kept under 100 characters.
Comments should be kept under 80 characters.
Breaking up long lines
Long function signatures should be broken up like so:
func Foo(
bar int,
baz bool,
blah []int) (string, error) {
...
}And callsites:
// If it fits on a 2nd line:
x, err := Foo(
1, false, []int{1, 2, 3})
// If it doesn't:
y, err := Foo(
1,
false,
[]int{1, 2, 3, ...})Whitespace
Be conservative with adding blank lines between blocks of code. Avoid cluttering vertical screen space
with blank lines when the code reads just fine without them.
Bad:
func (s *State) DeletePending(peerID core.PeerID, h core.InfoHash) {
k := connKey{peerID, h}
if !s.pending[k] {
return
}
delete(s.pending, k)
s.capacity[k.infoHash]++
s.log("peer", peerID, "hash", h).Infof(
"Deleted pending conn, capacity now at %d", s.capacity[k.infoHash])
}Good:
func (s *State) DeletePending(peerID core.PeerID, h core.InfoHash) {
k := connKey{peerID, h}
if !s.pending[k] {
return
}
delete(s.pending, k)
s.capacity[k.infoHash]++
s.log("peer", peerID, "hash", h).Infof(
"Deleted pending conn, capacity now at %d", s.capacity[k.infoHash])
}File: examples/k8s/README.md
1. Run
$ helm install --name=kraken-demo ./helm
This command starts 3 trackers, origins and build-index pods, 1 proxy pod and an agent daemonset.
Once deployed, every node will have a docker registry API exposed on port 30081. The port number is
configurable.
2. Pulling from Docker Hub Library
A simple registry storage backend is provided for read-only access to Docker registry. A library
image can be pulled from Kraken agent by specifying 127.0.0.1:30081 in the image name in pod
spec. For example spec, see example.
Note, this backend is used only for all library/.* repositories. library is the default
namespace for Docker Hub's standard public repositories. To use your own registry as the backend,
please update origin and build-index configs accordingly.