## 1. Project Overview & Quickstart (sigstore/cosign_trusted-root) # cosign_trusted-root Open-source repository sigstore/cosign_trusted-root ### Repository Details - **Repository:** [sigstore/cosign_trusted-root](https://github.com/sigstore/cosign_trusted-root) - **Primary Language:** Code *Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.* ## 2. Official Technical Reference & Guides (sigstore/docs) ## File: README.md # Sigstore Docs This repo contains the Sigstore documentation, which is hosted at [docs.sigstore.dev](https://docs.sigstore.dev/). If you are looking for the frontend development of the [Sigstore website](https://www.sigstore.dev/), please visit the [sigstore-website](https://github.com/sigstore/sigstore-website) repo. ## Contributing We welcome contributions on the docs site! ### Pull Request Process 1. Please first discuss the change you wish to make via an [issue](https://github.com/sigstore/docs/issues). 2. Fork the `docs` repository to your own GitHub account and clone it locally. 3. Hack on your changes. 4. Write a meaningful commit message (and [sign your commit](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits)). Review the [commit message guidelines](https://github.com/sigstore/sigstore-website/blob/main/CONTRIBUTORS.md#commit-message-guidelines). 5. Ensure that CI passes, if it fails, fix the failures. If you are making many commits into one PR, please [squash your commits](https://github.com/sigstore/sigstore-website/blob/main/CONTRIBUTORS.md#squash-commits). 6. Every pull request requires a review from the [core sigstore-website team](https://github.com/orgs/github.com/sigstore/teams/codeowners-sigstore-website) before merging. ## Setup First, [install Hugo](https://gohugo.io/installation/) following the instructions for your platform. Clone this repository and navigate to its directory. Install dependencies: ```bash npm install ``` ## Development Server You can run the development server with: ```bash npm run start ``` If needed, you can also build the site locally: ```bash npm run build ``` ## Site Information This site is built using the [Hugo static site generator](https://gohugo.io/) and the [Doks theme](https://getdoks.org/). Modifications have been made to the theme templates to change the landing page from the Doks default, to make section bundles more useful to visitors, and to give an option to display simplified titles in the sidebar. Full text search is provided by [FlexSearch](https://github.com/nextapps-de/flexsearch). ## Common Tasks To change the order of pages on the site, edit the `weight` variable in the front matter of individual pages. A smaller number means the page will appear first in the sidebar and in section bundles. To change the order of sections of content, refer to the `weight` variable in the `_index.html` file within the folder for that section. These `_index.html` files also enable you to change the names of sections. Both a `title` and `menuTitle` variable can be set in front matter. If a `menuTitle` variable is set, that text will be used in the left sidebar and on section bundles. If only a `title` is set, that text will be used for both the on-page title and in menus. The `menuTitle` variable should be used to shorten long page titles. The Doks theme provides additional functionality for linting and checking scripts that may be useful: [Doks Commands](https://getdoks.org/docs/overview/commands/) ## Technical writing There’s plenty of good style guides for writing technical documentation. [Microsoft’s Style Guide](https://docs.microsoft.com/en-us/style-guide/welcome/) or [Google's developer documentation](https://developers.google.com/style) are two of the most mature and well established guides, and these are especially useful to turn to when you’ve got a decision to make that might become convention. The most important thing though, especially as your documentation resources grow, is to follow your nose, discuss any decisions that might create consistency rules and keep track of the results, and do a little contextual review of what’s already been published to check for consistency. ### Tips - Always write like you’re onboarding a new colleague. You want to help them get up to speed. Prioritise your points and be action-focused, cut anything that’s not really necessary, and provide as much cross-linking as you can - this’ll help keep your reader knowing what to pay attention to, and provides just enough information at the right time without robbing them of the chance to learn more. - It’s good to scope out the beginning, middle and end of what each resource will be before you start writing - it’s easy for a quick guide to become a rambling tangent. Take good stock of what content might be living in the same area as this one, and always include links to build up your internal architecture. - Give a quick pass over any similar type of resource to see what kind of layout conventions you should leverage to keep things consistent and easy to understand. Best practice is to imagine you’re onboarding a new colleague. Respect their time, be friendly but direct, and focus on the end goal together. - Using insets, alert banners and callouts help make each page a little more visually interesting, which helps a reader engage with, follow and understand what you’re writing about. - Always check your assumptions. It’s easy to assume your reader knows exactly what you do. But that’s not always the case. Be careful writing about anything that might require more context, embed links in passing if it’ll help. If you’re introducing something difficult for the first time, state it, explain it as simply as you can, and then move to its application using consistent language - don’t be tempted to start using variations or introducing more tangents than is absolutely critical. --- ## File: archetypes/docs/lorem/ipsum/index.md --- title: "Ipsum" lead: "" date: 2022-01-25T14:41:39+01:00 lastmod: 2022-01-25T14:41:39+01:00 draft: false images: [] type: docs menu: {{ .Section }}: parent: "lorem" identifier: "{{ .Name }}-{{ delimit (shuffle (split (md5 .Name) "" )) "" }}" weight: 100 toc: true --- --- ## File: archetypes/docs/lorem/_index.md --- title: "Lorem" lead: "" date: 2022-01-25T14:41:21+01:00 lastmod: 2022-01-25T14:41:21+01:00 draft: false images: [] type: docs --- --- ## File: archetypes/docs/_index.md --- title: "Docs" lead: "" date: 2022-01-25T14:40:56+01:00 lastmod: 2022-01-25T14:40:56+01:00 draft: false images: [] type: docs --- --- ## File: content/en/logging/CLI.md --- type: docs category: Transparency Log title: CLI weight: 1825 --- The following guide is targeted towards developers / software maintainers who would like to make a provenance entry into the Rekor transparency log. The steps outlined below will show how to sign your software and use the Rekor CLI to make and verify an entry. It uses GPG to demonstrate, but other types are outlined in the [Signing and Uploading Other Types]({{< relref "logging/sign-upload">}}) page. ## Prerequisites You need to have Rekor CLI installed. See the [Installation]({{< relref "logging/installation">}}) page for instructions. ## Sign your release Before using Rekor, you are required to sign your release. The following example illustrates this using GPG. You may use either armored or plain binary: ```bash gpg --armor -u jdoe@example.com --output mysignature.asc --detach-sig myrelease.tar.gz ``` You will also need to export your public key ```bash gpg --export --armor "jdoe@example.com" > mypublickey.key ``` ## Upload an entry to Rekor The `upload` command sends your public key / signature and artifact URL to the Rekor transparency log. ```bash rekor-cli upload --rekor_server https://rekor.sigstore.dev --signature --public-key --artifact | ``` The Rekor command will first verify your public key and signature and download a local copy of the artifact. Then it will validate the artifact signing (no access to your private key is required). If the validations above pass correctly, the entry will be made to Rekor and an entry URL will be returned: ```bash Created entry at: https://rekor.sigstore.dev/api/v1/log/entries/b08416d417acdb0610d4a030d8f697f9d0a718024681a00fa0b9ba67072a38b5 ``` This URL contains the UUID entry / merkle tree hash (in the above case `b08416d417acdb0610d4a030d8f697f9d0a718024681a00fa0b9ba67072a38b5`). ## Verify Proof of Entry The `verify` command allows you to send a public key / signature and artifact to the Rekor transparency log for verification of entry. You would typically use this command as a means to verify an 'inclusion proof' showing that your artifact is stored within the transparency log. ```bash rekor-cli verify --rekor_server --signature --public-key --artifact | ``` > Note that alternatively you can use a local artifact path with `--artifact`. ## Get Entry An entry in the log can be retrieved by using the `get` command with either the log index or the artifact UUID: ```bash rekor-cli get --rekor_server https://rekor.sigstore.dev --log-index ``` ```bash rekor-cli get --rekor_server https://rekor.sigstore.dev --uuid ``` ## Log Info The `loginfo` command retrieves the public key of the transparency log (unless already declared within the client `~/.rekor/rekor.yaml`) and then uses this public key to verify the signature on the signed tree head. ```bash rekor-cli loginfo --rekor_server https://rekor.sigstore.dev ``` ## Search If running a redis instance within Rekor, the `search` command performs a redis lookup using a file or a public key. This command requires one of an artifact, a public key, or a SHA hash (should be prefixed by `sha256:`). ```bash rekor-cli search --rekor_server https://rekor.sigstore.dev --[artifact|public-key|sha] ``` For example: ```bash rekor-cli search --rekor_server https://rekor.sigstore.dev --sha sha256:e2e90d1a25f90a3156a27f00f3a4179578e3132ed4f010dc3498d09175b6071a ``` --- ## File: content/en/logging/event_stream.md --- type: docs category: Transparency Log menuTitle: Event Stream title: Using the Rekor Event Stream weight: 1850 --- The public Rekor instance provides an event stream of new entries added to the transparency log using GCP Pub/Sub. This can be used to monitor the log in real-time for events you are interested in. ## Pub/Sub details > **Important:** Pub/Sub usage is not free. Please familiarize yourself with the [pricing](https://cloud.google.com/pubsub/pricing) before proceeding. > **Tip:** You can avoid paying egress network costs by placing the workload that processes events from the subscription in the same region as the topic. The details needed to subscribe to the topic are below. Any authenticated GCP principal can create subscriptions to the topic. | Item | Value | |----------------------|-----------------| | GCP Project ID | `project-rekor` | | Pub/Sub Topic ID | `new-entry` | | Pub/Sub Topic Region | `us-central1` | ## Event format The events are published in both the [JSON](https://github.com/cloudevents/spec/blob/main/cloudevents/formats/json-format.md) and [Protobuf](https://github.com/cloudevents/spec/blob/main/cloudevents/formats/protobuf-format.md) formats of the CloudEvents specificaion. The `data` field is always a serialized [`TransparencyLogEntry`](https://github.com/sigstore/protobuf-specs/blob/b46b842040854ceab8f3a42547ae6e991793d0ef/protos/sigstore_rekor.proto#L92) proto message. For JSON formatted events, this follows the JSON serialization [specification](https://protobuf.dev/programming-guides/proto3/#json) for Protobufs. All published events have attributes attached to them. Attributes that are custom to Rekor and not a part of the CloudEvents specification are prefixed by `rekor_`. You can add [server-side filters](https://cloud.google.com/pubsub/docs/subscription-message-filter) based on these attributes to automatically acknowledge messages you are not interested in to reduce costs. | Attribute | Value | |--------------------------|-------------------------------------------------------------------------| | `datacontenttype` | `application/json` or `application/protobuf` | | `id` | The unique identifier for the entry in the Rekor log. | | `rekor_entry_kind` | The kind of the Rekor entry (`hashedrekord`, `intoto`, etc.) | | `rekor_signing_subjects` | A sorted, comma-delimeted list of the principals that signed the entry. | | `source` | `/createLogEntry` | | `specversion` | `1.0` | | `time` | The timestamp the entry was added to the log. | | `type` | `dev.sigstore.rekor.events.v1.NewEntry` | ## Example usage In this example scenario, we want to monitor all log entries added by GitHub Actions CI workers in the Sigstore GitHub org. We do this to verify that all CI runs that upload entries to Rekor are from the `main` branch only. Uploads from other branches could indicate a GitHub Actions misconfiguration that allowed a malicious actor to submit a pull request which triggered a CI run that allowed them to sign their artifacts using our identity. ### Setting up the subscription > **Note:** To perform the steps below, you will need the [gcloud CLI](https://cloud.google.com/sdk/docs/install). First, define some environment variables. You may pick any value you like for `SUBSCRIPTION_PROJECT_ID` and `SUBSCRIPTION_ID`. The value for `SUBSCRIPTION_PROJECT_ID` must be globally unique among all GCP projects. ```sh export SUBSCRIPTION_PROJECT_ID="your-gcp-project-id" export SUBSCRIPTION_ID="new-entry-subscription" export SUBSCRIPTION_CONTENT_TYPE="application/protobuf" ``` If the project does not exist, you can create it with the command below. ```sh gcloud projects create $SUBSCRIPTION_PROJECT_ID ``` Then, you can create the subscription, filtering on the content type and the signing subjects we want to receive messages about. ```sh export SUBSCRIPTION_FILTER='attributes.datacontenttype = "$SUBSCRIPTION_CONTENT_TYPE" AND hasPrefix(attributes.rekor_signing_subjects, "https://github.com/sigstore/")' ``` ```sh gcloud pubsub subscriptions create \ $SUBSCRIPTION_ID \ --project $SUBSCRIPTION_PROJECT_ID \ --topic new-entry \ --topic-project project-rekor \ --message-filter $SUBSCRIPTION_FILTER ``` The subscription is now set up and we can pull the events ### Processing log entries The first step is to create a program that pulls all messages from the subscription indefinitely. It passes all received messages to a `handleMsg` function. The code examples below omit imports from Go's standard library for brevity. ```go import ( "cloud.google.com/go/pubsub" ) func main() { ctx, done := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT) defer done() // Bootstrap the PubSub client. client, err := pubsub.NewClient(ctx, os.Getenv("SUBSCRIPTION_PROJECT_ID")) if err != nil { log.Fatal(err) } sub := client.Subscription(os.Getenv("SUBSCRIPTION_ID")) // Continuously read messages into a channel. msgs := make(chan *pubsub.Message) go func() { for { err := sub.Receive(ctx, func(_ context.Context, msg *pubsub.Message) { msgs <- msg }) if err != nil { log.Print(err) } } }() // Handle each message received until the process is terminated. for { select { case msg := <-msgs: if err := handleMsg(msg); err != nil { log.Print(err) } msg.Ack() case <-ctx.Done(): return } } } ``` The `handleMsg` function deserializes the event and then checks the signing subjects. ```go import ( "google.golang.org/protobuf/proto" eventspb "github.com/sigstore/protobuf-specs/gen/pb-go/events/v1" ) func handleMsg(msg *pubsub.Message) error { event := new(eventspb.CloudEvent) if err := proto.Unmarshal(msg.Data, event); err != nil { return fmt.Errorf("unmarshal msg %q: %w", msg.ID, err) } // In a real implementation, you should verify the entry before proceeding! attr, ok := event.Attributes["rekor_signing_subjects"] if !ok { return fmt.Errorf("entry %q is missing rekor_signing_subjects", event.GetId()) } subjects := strings.Split(attr.GetCeString(), ",") for _, subject := range subjects { if !strings.HasSuffix(subject, "@refs/heads/main") { return fmt.Errorf("entry %q was not run from the main branch: %s", event.GetId(), subject) } } return nil } ``` --- ## File: content/en/logging/installation.md --- type: docs category: Transparency Log title: Installation weight: 1805 --- There are several ways to install both the `rekor-cli` and `rekor-server`. ## Using Go install If you have [Go](https://go.dev/doc/install) installed, you can use Go to retrieve the `rekor-cli` binaries ```bash go install -v github.com/sigstore/rekor/cmd/rekor-cli@latest ``` You may also do the same for `rekor-server`, but **please note** that the Rekor server also requires Trillian and a database. (see below for setup instructions). ```bash go install -v github.com/sigstore/rekor/cmd/rekor-server@latest ``` ## From the release page Rekor releases are available on the [Release page](https://github.com/sigstore/rekor/releases). Releases are available for both `rekor-server` and `rekor-cli`. Review [Verifying Binaries]({{< relref "logging/verify-release">}}) for details on how to verify Rekor release binaries. ## Build Rekor CLI manually ```bash git clone https://github.com/sigstore/rekor.git rekor-cli cd rekor-cli make rekor-cli cp rekor-cli /usr/local/bin/ ``` ## Deploy a Rekor Server manually There are a few ways you can deploy a Rekor Server: 1. We have a [docker-compose](https://github.com/sigstore/rekor/blob/main/docker-compose.yml) file available. 2. Alternatively, you can build a Rekor server yourself. Note: The Rekor server manually creates a new Merkle tree (or shard) in the Trillian backend every time it starts up, unless an existing one is specified in via the `--trillian_log_server.tlog_id` flag. If you are building the server yourself and do not need [sharding]({{< relref "logging/sharding">}}) functionality, you can find the existing tree's TreeID by issuing this client command while the server is running: `CURRENT_TREE_ID=$(rekor-cli loginfo --format json | jq -r .TreeID)` Then pass in this TreeID at the next server startup to tell Rekor to use the same existing tree: `rekor-server serve --trillian_log_server.tlog_id=$CURRENT_TREE_ID` Setting this flag isn't necessary in an environment like `docker-compose`. ### Prerequisites You will need golang version 1.16 or greater and a `$GOPATH` set. You will also need a MySQL compatible database and, if you want to perform fast queries, an instance of redis running. Otherwise you must pass the `--enable_retrieve_api=false` flag when running `rekor-server` in the later steps of this page. ### Get Rekor Grab the Rekor source: `go get -u -t -v github.com/sigstore/rekor/cmd/rekor-server` > Should you prefer, you can also `git clone https://github.com/sigstore/rekor.git` ### Create database and populate tables Rekor requires a database. We use MariaDB for now (others to be explored later). Install and set up MariaDB on your machine. ```bash dnf install mariadb mariadb-server systemctl start mariadb systemctl enable mariadb mysql_secure_installation ``` The Rekor directory has a `scripts/createdb.sh` file that will set up a test database (default user: test; default password: zaphod) and populate the needed tables for Trillian. If you’re just trying out Rekor, keep the DB user name and password the same as in the script (test/zaphod). If you change these, you need to make the changes on Trillian’s side (visit the [Trillian repo](https://github.com/google/trillian) for details). ```bash cd $GOPATH/pkg/mod/github.com/sigstore/rekor@v0.4.0/scripts/ sh createdb.sh ``` ### Build Trillian You also need to build Trillian, an append-only log: ```bash go get -u -t -v github.com/google/trillian cd $GOPATH/src/github.com/google/trillian/cmd/trillian_log_server go build cp trillian_log_server /usr/local/bin/ cd $GOPATH/src/github.com/google/trillian/cmd/trillian_log_signer go build cp trillian_log_signer /usr/local/bin/ cd $GOPATH/src/github.com/google/trillian/cmd/createtree go build cp createtree /usr/local/bin/ ``` Next, run the Trillian log server: ```bash trillian_log_server --logtostderr ... ``` Run the signer: ```bash trillian_log_signer --logtostderr --force_master --rpc_endpoint=localhost:8190 -http_endpoint=localhost:8191 --batch_size=1000 --sequencer_guard_window=0 --sequencer_interval=200ms ``` > Note: you can log both to files and to stderr using `--alsologtostderr` Create the tree: ```bash createtree --admin_server=localhost:8090 ``` #### Build the Rekor Server With Trillian and MariaDB set up, you can now build the Rekor Server: ```bash cd $GOPATH/pkg/mod/github.com/sigstore/rekor@v0.4.0/cmd/rekor-server go build -v -o rekor-server cp rekor-server /usr/local/bin/ ``` #### Start the Rekor Server ```bash rekor-server serve --enable_retrieve_api=false 2020-09-12T16:32:22.705+0100 INFO cmd/root.go:87 Using config file: /Users/lukehinds/go/src/github.com/sigstore/rekor-server/rekor-server.yaml 2020-09-12T16:32:22.705+0100 INFO app/server.go:55 Starting server... 2020-09-12T16:32:22.705+0100 INFO app/server.go:61 Listening on 127.0.0.1:3000 ``` > If you have a redis server running to enable searching your Rekor server, remove the `enable_reprieve_api` flag #### Next Steps Congratulations! Your local Rekor server is now running. You can interact with it using the [Rekor CLI]({{< relref "logging/cli">}}). --- ## File: content/en/logging/overview.md --- type: docs category: Transparency Log menuTitle: Overview title: Rekor weight: 1800 --- Rekor aims to provide an immutable, tamper-resistant ledger of metadata generated within a software project’s supply chain. It enables software maintainers and build systems to record signed metadata to an immutable record. Other parties can then query this metadata, enabling them to make informed decisions on trust and non-repudiation of an object’s lifecycle. The Rekor project provides a restful API-based server for validation, and a transparency log for storage. A CLI application is available to make and verify entries, query the log for inclusion proof, integrity verification of the log or retrieval of entries (either by a public key or an artifact). Rekor fulfils the signature transparency role of Sigstore’s software signing infrastructure. It can also be run on its own, and it’s designed to be extendable to working with different manifest schemas and PKI tooling. ## Usage and installation You can download and setup the Rekor Server and Rekor CLI by following the instructions on the [Installation]({{< relref "logging/installation">}}) page. A public instance of Rekor can be found at [rekor.sigstore.dev](https://rekor.sigstore.dev). The public instance offers an SLO of 99.5% availability and is monitored by an oncall team. ## Auditing the Public Instance Rekor is built on top of a [verifiable data structure](https://transparency.dev/verifiable-data-structures/). Auditors can monitor the log for consistency, meaning that the log remains append-only and entries are never mutated or removed. Verifiers can also monitor the log for their identities. Learn more about transparency logs [here](https://transparency.dev/), and about binary transparency [here](https://binary.transparency.dev/). There are few options for auditing and monitoring the Rekor log. We've built a monitor that runs on GitHub Actions, [Rekor monitor](https://github.com/sigstore/rekor-monitor). Follow the instructions to set up a new repository and use the [provided reusable workflow](https://github.com/sigstore/rekor-monitor/blob/main/.github/workflows/reusable_monitoring.yml) to audit the log. You can also monitor the log for specified identities, though this feature is a work in progress and supports a limited set of identities and entry types. You can also run [omniwitness](https://github.com/transparency-dev/witness/tree/main/cmd/omniwitness) to audit the log, built by the team who created Trillian, which provides Rekor's verifiable log. --- ## File: content/en/logging/pluggable-types.md --- type: docs category: Transparency Log title: Pluggable Types weight: 1830 --- Rekor supports pluggable types (aka different schemas) for entries stored in the transparency log. This will allow you to develop your own manifest type in your preferred formatting style (json|yaml|xml). ## Currently supported types The list of currently supported types and their schema is [maintained in the repository](https://github.com/sigstore/rekor/tree/main/pkg/types#currently-supported-types). ## Base schema The base schema for all types is modeled off of the schema used by Kubernetes and can be found in `openapi.yaml` as `#/definitions/ProposedEntry`: ```bash definitions: ProposedEntry: type: object discriminator: kind properties: kind: type: string required: - kind ``` The `kind` property is a [discriminator](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#fixed-fields-13) that is used to differentiate between different pluggable types. Types can have one or more versions of the schema supported concurrently by the same Rekor instance; an example implementation can be seen in `rekord.go`. ## Adding support for a new type To add a new type (called `newType` in this example): 1. Add a new definition in `openapi.yaml` that is a derived type of ProposedEntry (expressed in the `allOf` list seen below); for example: ```yaml newType: type: object allOf: - $ref: "#/definitions/ProposedEntry" - properties: version: type: string metadata: type: object additionalProperties: true data: type: object $ref: "pkg/types/newType/newType_schema.json" required: - version - data additionalProperties: false ``` > Note: the `$ref` feature can be used to refer to an externally defined JSON schema document; however it is also permitted to describe the entirety of the type in valid Swagger (aka OpenAPI) v2 format within `openapi.yaml`. 2. Create a subdirectory under `pkg/types/` with your type name (e.g. `newType`) as a new Go package 3. In this new Go package, define a struct that implements the `TypeImpl` interface as defined in `pkg/types/types.go`: ```go type TypeImpl interface { CreateProposedEntry(context.Context, version string, ArtifactProperties) (models.ProposedEntry, error) DefaultVersion() string SupportedVersions() []string IsSupportedVersion(version string) bool UnmarshalEntry(pe models.ProposedEntry) (EntryImpl, error) } ``` - `CreateProposedEntry` creates an instance of a proposed entry based on the specified API version and the provided artifact properties - `DefaultVersion` returns the default API version string across all types (to be used if a caller does not specify an explicit version) - `SupportedVersions` returns the list of all API version strings that can currently be inserted into the log. - `IsSupportedVersion` returns a boolean denoting whether the specified version could be inserted into the log - `UnmarshalEntry` will be called with a pointer to a struct that was automatically generated for the type defined in `openapi.yaml` by the [go-swagger](http://github.com/go-swagger/go-swagger) tool used by Rekor - This struct will be defined in the generated file at `pkg/generated/models/newType.go` (where `newType` is replaced with the name of the type you are adding) - This method should return a pointer to an instance of a struct that implements the `EntryImpl` interface as defined in `pkg/types/types.go`, or a `nil` pointer with an error specified Also, the `Kind` constant must return the _exact_ same string as you named your new type in `openapi.yaml` (e.g. "`newType`") 4. Also in this Go package, provide an implementation of the `EntryImpl` interface as defined in `pkg/types/entries.go`: ```go type EntryImpl interface { APIVersion() string // the supported versions for this implementation IndexKeys() ([]string, error) // the keys that should be added to the external index for this entry Canonicalize(ctx context.Context) ([]byte, error) // marshal the canonical entry to be put into the tlog Unmarshal(e models.ProposedEntry) error // unmarshal the abstract entry into the specific struct for this versioned type CreateFromArtifactProperties(context.Context, ArtifactProperties) (models.ProposedEntry, error) Verifier() (pki.PublicKey, error) Insertable() (bool, error) // denotes whether the entry that was unmarshalled has the writeOnly fields required to validate and insert into the log } ``` - `APIVersion` should return a version string that identifies the version of the type supported by the Rekor server - `IndexKeys` should return a `[]string` containing the keys that are stored in the search index that should map to this log entry's ID. - `Canonicalize` should return a `[]byte` containing the canonicalized contents representing the entry. The canonicalization of contents is important as we should have one record per unique signed object in the transparency log. - `Unmarshal` will be called with a pointer to a struct that was automatically generated for the type defined in `openapi.yaml` by the [go-swagger](http://github.com/go-swagger/go-swagger) tool used by Rekor - This method should validate the contents of the struct to ensure any string or cross-field dependencies are met to successfully insert an entry of this type into the transparency log - `CreateFromArtifactProperties` returns a proposed entry of this specific entry implementation given the provided artifact properties - `Verifier` returns the verification material that was used to verify the digital signature - `Insertable` introspects the entry and determines if the object is sufficiently hydrated to make a new entry into the log. Entry instances that are created by reading the contents stored in the log may not be sufficiently hydrated. 5. In the Go package you have created for the new type, be sure to add an entry in the `TypeMap` in `github.com/sigstore/rekor/pkg/types` for your new type in the `init` method for your package. The key for the map is the unique string used to define your type in `openapi.yaml` (e.g. `newType`), and the value for the map is the name of a factory function for an instance of `TypeImpl`. ```go func init() { types.TypeMap.Set("newType", NewEntry) } ``` 6. Add an entry to `pluggableTypeMap` in `cmd/server/app/serve.go` that provides a reference to your package. This ensures that the `init` function of your type (and optionally, your version implementation) will be called before the server starts to process incoming requests and therefore will be added to the map that is used to route request processing for different types. 7. After adding sufficient unit & integration tests, submit a pull request to `sigstore/rekor` for review and addition to the codebase. ## Adding a new version of the `Rekord` type To add new version of the default `Rekord` type: 1. Create a new subdirectory under `pkg/types/rekord/` for the new version 2. If there are changes to the Rekord schema for this version, create a new JSON schema document and add a reference to it within the `oneOf` clause in `rekord_schema.json`. If there are no changes, skip this step. 3. Provide an implementation of the `EntryImpl` interface as defined in `pkg/types/types.go` for the new version. 4. In your package's `init` method, ensure there is a call to `SemVerToFacFnMap.Set()` which provides the link between the valid _semver_ ranges that your package can successfully process and the factory function that creates an instance of a struct for your new version. 5. Add an entry to `pluggableTypeMap` in `cmd/server/app/serve.go` that provides a reference to the Go package implementing the new version. This ensures that the `init` function will be called before the server starts to process incoming requests and therefore will be added to the map that is used to route request processing for different types. 6. After adding sufficient unit & integration tests, submit a pull request to `sigstore/rekor` for review and addition to the codebase. --- ## File: content/en/logging/sharding.md --- type: docs category: Transparency Log title: Sharding weight: 1820 --- This document covers what Rekor log sharding is and how to shard the log. ## What is sharding? When Rekor is started for the first time, its backend is a transparency log built on a single [Merkle Tree](https://en.wikipedia.org/wiki/Merkle_tree). This log can grow indefinitely as entries are added, which can present issues over time. To resolve some of these issues the log can be "sharded" into multiple Merkle Trees. ## Why do we shard the log? Sharding the log allows for: * Freezing the current log and rotating signing keys if needed * Easier and faster querying for entries from the tree * Easier scaling and platform migrations ## How does this impact user experience? It shouldn't! End users shouldn't notice any difference in their experience. They can still query via UUID, and Rekor will find the correct entry from whichever shard it's in. Querying by log index works as well, since log indices are distinct and increase across shards. ## More Details For more details around sharding, see the original [design doc](https://docs.google.com/document/d/1QBTyK-wquplNdeUB5_aqztQHigJOepCvd-4FL4H-zl8/edit?resourcekey=0-grdVbSltkTvpNvhj03laCQ#heading=h.al4txfo7pxwl)! _Note: You'll need to join the [sigstore-dev@googlegroups.com](https://groups.google.com/g/sigstore-dev/about) Google group for access to the doc._ ## How do I shard the Rekor log? **Sharding the Rekor log will require some downtime in your Rekor service.** This is necessary because you'll need the length of the current shard later on, so new entries can't be added while sharding is in progress. Follow these steps to shard the log: 1. Stop all traffic to Rekor so new entries can't be added to the log 2. Store the tree ID and length of the current active shard: ```bash CURRENT_TREE_ID=$(rekor-cli loginfo --format json | jq -r .TreeID) CURRENT_SHARD_LENGTH=$(rekor-cli loginfo --format json | jq -r .ActiveTreeSize) ``` 3. Connect to the production cluster. Port-forward the running `trillian_logserver` container and run the [createtree](https://github.com/google/trillian/blob/master/cmd/createtree/main.go) script. This will create a new Merkle Tree which will become the new active shard. ```bash kubectl port-forward -n trillian-system deploy/trillian-log-server 8090:8090 # This is the Tree ID of the new active shard NEW_TREE_ID=$(createtree --admin_server localhost:8090) ``` > **Note:** Sigstore Helm Charts use port 8091 for gRPC 4. Update the Rekor `sharding-config` ConfigMap with details of the inactive shard: ```bash kubectl edit configmap sharding-config -n rekor-system ``` Append the following onto the `sharding-config.yaml` key (it will be empty if this is the first shard): ```yaml - treeID: $CURRENT_TREE_ID treeLength: $CURRENT_SHARD_LENGTH ``` 5. In your rekor-server [Deployment](https://github.com/sigstore/rekor/blob/main/config/rekor.yaml), update the `--trillian_log_server.tlog_id` flag to point to the new Tree ID. ```bash "--trillian_log_server.tlog_id=$NEW_TREE_ID", ``` 6. Redeploy Rekor to the cluster with these changes. 7. Restart traffic to your Rekor service. 8. Congratulations, you've successfully sharded the log! ## Identifier Definitions: EntryID, UUID, LogID, Log Index An **EntryID** is the unique identifier for an artifact in Rekor. It is made up of two parts, the TreeID and UUID: `EntryID = TreeID (8 byte hex) + UUID (32 byte hex)` The **TreeID** refers to the specific trillian tree (also known as log or shard) where the artifact can be found. Rekor uses Trillian to implement each transparency log. The **UUID** refers to the specific artifact within a specified tree. The **LogIndex** of an artifact identifies the index or order in which the artifact was entered into a specific tree.