# Technical Documentation: google/go-cloud

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/google/go-cloud)

### Overview (`README.md`)

# The Go Cloud Development Kit (Go CDK)

_Write once, run on any cloud ☁️_

[](https://github.com/google/go-cloud/actions)
[][PkgGoDev]
[](https://codecov.io/gh/google/go-cloud)

[](https://opensource.org/licenses/Apache-2.0)

<p align="center">
  <img width="509" height="276" src="internal/website/static/go-cdk-logo-gopherblue.png" alt="">
</p>

The Go Cloud Development Kit (Go CDK) allows Go application developers to
seamlessly deploy cloud applications on any combination of cloud providers. It
does this by providing stable, idiomatic interfaces for common uses like storage
and databases. Think `database/sql` for cloud products.

Imagine writing this to read from blob storage (like Google Cloud Storage or
S3):

```go
ctx := context.Background()
bucket, err := blob.OpenBucket(ctx, "s3://my-bucket")
if err != nil {
    return err
}
defer bucket.Close()
blobReader, err := bucket.NewReader(ctx, "my-blob", nil)
if err != nil {
    return err
}
```

and being able to run that code on any cloud you want, avoiding all the ceremony
of cloud-specific authorization, tracing, SDKs and all the other code required
to make an application portable across cloud platforms.

The project works well with a code generator called
[Wire](https://github.com/google/wire/blob/master/README.md). It creates
human-readable code that only imports the cloud SDKs for services you use. This
allows the Go CDK to grow to support any number of cloud services, without
increasing compile times or binary sizes, and avoiding any side effects from
`init()` functions.

You can learn more about the project from our [announcement blog post][], or our
talk at Next 2018:

[][video]

[announcement blog post]: https://blog.golang.org/go-cloud
[PkgGoDev]: https://pkg.go.dev/mod/gocloud.dev
[travis]: https://travis-ci.com/google/go-cloud
[video]: https://www.youtube.com/watch?v=_2ZwhvIkgek

## Installation

```shell
# First "cd" into your project directory if you have one to ensure "go get" uses
# Go modules (or not) appropriately. See "go help modules" for more info.
go get gocloud.dev
```

The Go CDK builds at the latest stable release of Go. Previous Go versions may
compile but are not supported.

## Documentation

Documentation for the project lives primarily on https://gocloud.dev/, including
[tutorials][].

You can also browse Go package reference on [pkg.go.dev][PkgGoDev].

[tutorials]: https://gocloud.dev/tutorials/

## Project status

The APIs are still in alpha, but we think they are production-ready and are
actively looking for feedback from early adopters. If you have comments or
questions please open an issue.

At this time we prefer to focus on maintaining the existing APIs and drivers,
and are unlikely to accept new ones into the `go-cloud` repository. The modular
nature of the Go CDK makes it simple to host new APIs and drivers for existing
APIs externally, in separate repositories.

If you have a new API or driver that you believe are important and mature enough
to be included, feel free to open an issue to discuss this; our default will
likely be to suggest starting in a separate repository.

### Known External Drivers

*   [gocloud-ext](https://github.com/TuSKan/gocloud-ext) — `blob` drivers for HTTP and WebDAV (`httpblob`) and for SFTP (`sftpblob`).

## Current features

The Go CDK provides generic APIs for:

*   Unstructured binary (blob) storage
*   Publish/Subscribe (pubsub)
*   Variables that change at runtime (runtimevar)
*   Document storage (docstore)
*   Secrets (secrets)
*   Connecting to MySQL (including MariaDB) and PostgreSQL databases (mysql, postgres)
*   Server startup and diagnostics: request logging, tracing, and health
    checking (server)

## Contributing

Thank you for your interest in contributing to the Go Cloud Development
Kit! :heart:

Everyone is welcome to contribute, whether it's in the form of code,
documentation, bug reports, feature requests, or anything else. We encourage you
to experiment with the Go CDK and make contributions to help evolve it to meet
your needs!

The GitHub repository at [google/go-cloud][go-cloud] contains some driver
implementations for each portable API. We intend to include
[Google Cloud Platform][gcp], [Amazon Web Services][aws], and [Azure][azure]
implementations, as well as prominent open source services and at least one
implementation suitable for use in local testing. Unfortunately, we cannot
support every service directly from the project; however, we encourage
contributions in separate repositories.

If you create a repository that implements the Go CDK interfaces for other
services, let us know! We would be happy to link to it here and give you a
heads-up before making any breaking changes.

See [the contributing guide](./CONTRIBUTING.md) for more details.

[go-cloud]: https://github.com/google/go-cloud
[gcp]: http://cloud.google.com
[aws]: http://aws.amazon.com
[azure]: https://azure.microsoft.com/

## Community

This project is covered by the Go [Code of Conduct][].

[Code of Conduct]: ./CODE_OF_CONDUCT.md

## Legal disclaimer

The Go CDK is open-source and released under an [Apache 2.0
License](https://github.com/google/go-cloud/blob/master/LICENSE). Copyright ©
2018–2019 The Go Cloud Development Kit Authors.

If you are looking for the website of GoCloud Systems, which is unrelated to the
Go CDK, visit https://gocloud.systems.

---

### Overview (`wire/README.md`)

# Wire has moved!

Wire has moved to its own repository: [github.com/google/wire](https://github.com/google/wire)

Read the [announcement][] for more details.

[announcement]: https://groups.google.com/d/msg/go-cloud/4HuWfjDAkOY/Y2tUQdB_BQAJ

---

### Overview (`internal/docs/README.md`)

# Docs

This is the documentation for developers of the Go CDK, describing various
coding practices and project processes.

-   [Design Decisions](design.md)
-   [Releases](release.md)

---

### Overview (`internal/website/README.md`)

# gocloud.dev source

Source for the [gocloud.dev website][]. Powered by [Hugo][].

[gocloud.dev website]: https://gocloud.dev/
[Hugo]: https://gohugo.io/

## Local Development

Use local hugo server for preview. `cd` into this directory and run:

```
$ hugo server -D
```

This will run the Hugo server that auto-updates its output based on the source
contents. It will print out the `localhost:<PORT>` URL to point the browser to.

This was tested with Hugo 0.53 but should work with subsequent versions as well.

## Editing

Use `hugo new foo/page.md` to create `content/foo/page.md`. This will
automatically add the appropriate [Front Matter][] to the site. After modifying
an existing page, add the `lastmod` attribute with the current ISO date, which
you can obtain with `date -I`. For example:

```yaml
---
title: Foo
date: "2019-03-17T09:00:00-07:00"
lastmod: "2019-03-18T13:30:12-07:00"
---

...
```

[Front Matter]: https://gohugo.io/content-management/front-matter/

---

### Overview (`samples/appengine/README.md`)

# AppEngine Sample

This directory holds a simple "Hello world!" AppEngine app that uses
[server.Server](https://github.com/google/go-cloud/blob/master/server/server.go).

## Prerequisites

You will need to install the following software to run this sample:

-   [Go](https://golang.org/doc/install)
-   [gcloud CLI](https://cloud.google.com/sdk/downloads)

## Deploying

Run the following in this `samples/appengine` directory:

```shell
# Build the binary.
go build
# Deploy it to AppEngine.
gcloud app deploy
# Open a browser to the app.
gcloud app browse
```

Try browsing to the `/healthz/readiness` page that `server.Server` adds a
handler for.

---

### Overview (`samples/guestbook/README.md`)

# Guestbook Sample

Guestbook is a sample application that records visitors' messages, displays a
cloud banner, and an administrative message. The main business logic is written
in a cloud-agnostic manner using MySQL, the generic blob API, and the generic
runtimevar API. All platform-specific code is set up by
[Wire](https://github.com/google/wire).

The [online tutorial](https://gocloud.dev/tutorials/guestbook) will walk you through how to build, run, and deploy this sample locally, on Google Cloud Platform (GCP), on Amazon Web Servicess (AWS), or on Microsoft Azure.

## Gophers

The Go gopher was designed by Renee French and used under the
[Creative Commons 3.0 Attributions](https://creativecommons.org/licenses/by/3.0/)
license.

---

### Overview (`samples/tutorial/README.md`)

# Getting Started With The Go Cloud Development Kit

This is the source directory for the [command-line uploader tutorial][].

[command-line uploader tutorial]: https://gocloud.dev/tutorials/cli-uploader/

---

### Overview (`samples/wire/README.md`)

# Wire Tutorial has moved

The Wire Tutorial has moved to [the Wire repository](https://github.com/google/wire/tree/master/_tutorial).

---

### Overview (`docstore/internal/fields/README.md`)

This package is copied from cloud.google.com/go/internal/fields.

---

### Internal Docs Design (`internal/docs/design.md`)

# Design Decisions

This document outlines important design decisions made for this repository and
attempts to provide succinct rationales. Recording these decisions helps
maintain consistency across packages, especially as an open source project where
contributors can join at any point during development.

A broad design goal for the Go Cloud Development Kit (Go CDK) is for the API
style to be consistent. Consistency aids users in building a mental model of how
to use the APIs. As such, the design of individual packages must also consider
their impact on the Go CDK as a whole.

This is a [Living Document](https://en.wikipedia.org/wiki/Living_document). The
decisions in here are not set in stone, but simply describe our current thinking
about how to guide the Go Cloud Development Kit project. While it is useful to
link to this document when having discussions in an issue, it is not to be used
as a means of closing issues without discussion at all. Discussion on an issue
can lead to revisions of this document.

## Developers and Operators

The Go CDK is designed with two different personas in mind: the developer and
the operator. In the world of DevOps, these may be the same person. A developer
may be directly deploying their application into production, especially on
smaller teams. In a larger organization, these may be different teams entirely,
but working closely together. Regardless, these two personas have two very
different ways of looking at a Go program:

-   The developer persona wants to write business logic that is agnostic of
    underlying cloud provider. Their focus is on making software correct for the
    requirements at hand.
-   The operator persona wants to incorporate the business logic into the
    organization's policies and provision resources for the logic to run. Their
    focus is making software run predictably and reliably with the resources at
    hand.

The Go CDK uses Go interfaces at the boundary between these two personas: a
developer is meant to use an interface, and an operator is meant to provide an
implementation of that interface. This distinction prevents the Go CDK going
down a path of complexity that makes application portability difficult. The
[`blob.Bucket`][] type is a prime example: the API does not provide a way of
creating a new bucket. To properly and safely create such a bucket requires
careful consideration, getting something like ACLs wrong could lead to a
catastrophic data leak. To generate the ACLs correctly requires modeling of IAM
users and roles for each cloud platform, and some way of mapping those users and
roles across clouds. While not impossible, the level of complexity and the high
likelihood of a leaky abstraction leads us to believe this is not the right
direction for the Go CDK.

Instead of adding large amounts of leaky complexity to the Go CDK, we expect the
operator role to handle the management of non-portable platform-specific
resources. An implementor of the `Bucket` interface does not need to determine
the content type of incoming data, as that is a developer's concern. This
separation of concerns allows these two personas to communicate using a shared
language while focusing on their respective areas of expertise.

[`blob.Bucket`]: https://godoc.org/github.com/google/go-cloud/blob#Bucket

## Portable Types and Drivers

The portable APIs that the Go CDK exports (like [`blob.Bucket`][] or
[`runtimevar.Variable`][]) are concrete types, not interfaces. To understand
why, imagine if we used a plain interface:

![Diagram showing user code depending on blob.Bucket, which is implemented by
awsblob.Bucket.](img/user-facing-type-no-driver.png)

Consider the [`Bucket.NewWriter` method][], which infers the content type of the
blob based on the first bytes written to it. If `blob.Bucket` was an interface,
each implementation of `blob.Bucket` would have to replicate this behavior
precisely. This does not scale: conformance tests would be needed to ensure that
each interface method actually behaves in the way that the docs describe. This
makes the interfaces hard to implement, which runs counter to the goals of the
project.

Instead, we follow the example of [`database/sql`][] and separate out the
implementation-agnostic logic from the interface. The implementation-agnostic
logic-containing concrete type is the **portable type**. We call the interface
the **driver**. Visually, it looks like this:

![Diagram showing user code depending on blob.Bucket, which holds a
driver.Bucket implemented by awsblob.Bucket.](img/user-facing-type.png)

This has a number of benefits:

-   The portable type can perform higher level logic without making the
    interface complex to implement. In the blob example, the portable type's
    `NewWriter` method can do the content type detection and then pass the final
    result to the driver type.
-   Methods can be added to the portable type without breaking compatibility.
    Contrast with adding methods to an interface, which is a breaking change.
-   When new operations on the driver are added as new optional interfaces, the
    portable type can hide the need for type-assertions from the user.

As a rule, if a method `Foo` has the same inputs and semantics in the portable
type and the driver type, then the driver method may be called `Foo`, even
though the return signatures may differ. Otherwise, the driver method name
should be different to reduce confusion.

New Go CDK APIs should always follow this portable type and driver pattern.

[`runtimevar.Variable`]:
https://godoc.org/github.com/google/go-cloud/runtimevar#Variable
[`Bucket.NewWriter` method]:
https://godoc.org/github.com/google/go-cloud/blob#Bucket.NewWriter
[`database/sql`]: https://godoc.org/database/sql

## Minimize Global State

As a library, the Go CDK should not introduce global state. Global state is
difficult to reason about in large codebases, where it can be necessary for
different parts of the application to use different states. Instead of adding
global state, push responsibility to the application to inject the state where
it is needed.

The exception we permit is URL scheme registration as documented under
[URLs](#urls). The amount of boilerplate setup code required for URL muxes for
multiple drivers without use of a tool like Wire is an unreasonable burden for
users of Go CDK. We want the Go CDK to be usable both with and without Wire. A
global registry is acceptable as long as its use is not mandatory, but the
burden is to prove the benefit over the cost.

## Driver Package Naming Conventions

Inside this repository, we name packages that handle cloud services after the
service name, not the providing cloud (`s3blob` instead of `awsblob`). While a
cloud provider may provide a unique offering for a particular API, they may not
always provide only one, so distinguishing them in this way keeps the API
symbols stable over time.

The naming convention is `<provider><product><api>`, where:

*   `<provider>` is the provider name, like `aws` or `gcp` or `azure`.
    *   Omit for 3rd party/open source/local packages.
    *   May also be omitted in cases where the product name is sufficient (e.g.,
        `s3blob` not `awss3blob` since S3 is well-known, `gcsblob` not
        `gcpgcsblob` since GCS already references Google).
    *   Required if the product name is not unique across providers (e.g.,
        `gcpkms` and `awskms`).
*   `<product>`is the product/service name.
*   `<api>` is the portable API name.
    *   Include for local/test packages like (e.g., `fileblob`, `mempubsub`).
    *   May be omitted when it makes the package name too long (e.g. `awssnssqs`
        is long enough, don't add `pubsub`).
    *   Encouraged when it helps distinguish the package from the service's own
        package name (e.g., `s3blob` not `s3`).

## Portable Type Constructors

Portable type constructors are the functions defined in driver packages that end
users call to get an instance of the portable type. For example,
`gcsblob.OpenBucket`, which returns an instance of the `*blob.Bucket` portable
type backed by GCS.

-   Portable type constructors should be top-level functions that return the
    portable type directly. Avoid helpers (e.g., a `Client` struct with a
    function that returns the portable type instead of it being top-level) and
    wrappers (e.g., a `fooblob.Bucket` type returned from `fooblob.OpenBucket`
    that wraps the portable type). Top level functions without wrappers are
    easier to use, especially when we're consistent about it.
-   Order arguments that are less likely to change across multiple calls to the
    constructor before ones that are likely to change. For example, connection
    and authorization related arguments should go before names, so
    `OpenBucket(ctx, client, "mybucket")` instead of `OpenBucket(ctx,
    "mybucket", client)`.
-   All public constructors should take an `Options` struct (see next section).

### Option Structs

All public constructors should take an `Options` struct, even if it is currently
empty, to ensure that we can add arguments to the APIs in the future without
breaking backward compatibility.

-   This includes driver constructors (e.g., `gcsblob.OpenBucket`) as well as
    API functions (e.g., `blob.NewReader`). When in doubt, if you think it's
    possible that we'll add arguments, add `Options`.
-   The argument should be of type `*Options`, so that `nil` can be passed in
    the default case.
-   Name the `Options` struct appropriately. `Options` is usually fine for
    portable type constructors since the package generally only exposes a
    constructor. Inside a driver interface or in a portable type like `blob`,
    use more descriptive names like `ReaderOptions` or `WriterOptions`.
-   If a function already has a struct argument, don't add a separate `Options`
    struct. Example: the various `sql.Open` functions take a `Params` struct
    with connection parameters; we chose to add new options to `Params` instead
    of introducing a separate `Options` struct. This keeps the function
    signature simpler and avoid confusion about which struct new parameters
    should be added to.
-   When similar `Options` are part of a driver interface and also part of the
    portable type (e.g., `blob.WriterOptions`), duplicate the struct instead of
    aliasing or embedding it, and copy the struct fields explicitly where
    needed. This allows the godoc for each type to be tailored to the
    appropriate audience (e.g. end-users for the portable type, driver
    implementors for the driver interface), and also allows the structs to
    diverge over time if appropriate.
-   Required arguments must not be in an `Options` struct, and all fields of the
    `Options` struct must have reasonable defaults. Exception: struct arguments
    that don't have `Options` in the name can contain required arguments (e.g.,
    see the `Params` example for `sql.Open` above).

Regarding empty `Options` structs: we considered only adding them when the first
option is added, and using a separate constructor for compatibility (e.g., start
with `foo.New(...)` and later add `foo.NewWithOptions(..., opts *Options)` if
needed). However, this would result in inconsistent names over time (e.g., some
packages would expose `New` with an `Options`, while others would expose
`NewWithOptions`).

### Compound IDs

Many cloud providers have resource IDs that are made up of subcomponents in
some well-defined syntax. For example, [GCP KMS key IDs][] take the form
`projects/[PROJECT_ID]/locations/[LOCATION]/keyRings/[KEY_RING]/cryptoKeys/[KEY]`.
We call these _compound IDs_.

There are two broad compound ID usage patterns we have observed:

1. Applications will keep resources in the same location, so the application
   will build the ID from the subcomponents rather than passing the entire
   resource ID around.
2. Applications will pass a verbatim string from configuration down to the
   API, since this is what was easily copy-pasteable from the cloud console UI.

Go CDK constructors that take in compound IDs should take in a `string` with the
full compound ID. Helper functions to build these compound IDs from
subcomponents may be provided as needed. URL openers (described below) should
prefer to use the full compound ID in their URL format.

[GCP KMS key IDs]: https://cloud.google.com/kms/docs/object-hierarchy#key

### URLs

To enable the [Backing services factor][] of a Twelve-Factor Application, Go
Cloud includes the ability to construct each of its API objects using
identifying URLs. The portable type's package should include APIs like the
following:

```
/* Detailed source-code truncated for AI context efficiency. */
```

The repetition of `Widget` in the method names permits a type to handle multiple
resources within the API. Exporting the `URLMux` allows applications to build
their own muxes, potentially wrapping existing ones.

Driver packages should include their own `URLOpener` struct type which
implements all the relevant `WidgetURLOpener` methods. The URL should only serve
to identify which resource to open. Any credentials or other complex values
should be taken in as struct fields, not as input from URL. If the driver
package registers its `URLOpener` with the `DefaultURLMux`, then it should
populate these complex fields from environment variables. If doing so is
undesirable or expensive, then it should not register with the `DefaultURLMux`
and instead rely on users to create their own mux. If there already exists a
well-established URI format for the backend (like S3 URLs or database connection
URIs), then drivers should honor them where possible.

[Backing services factor]: https://12factor.net/backing-services

#### URL Examples

A `WidgetURLOpener` implementation for a hypothetical GCP service:

```go
package gcpfoo

// ...

const Scheme = "gcpwidget"

type URLOpener struct {
  Client  *gcp.HTTPClient
  Options Options
}

func (o *URLOpener) OpenWidgetURL(ctx context.Context, u *url.URL) (*foo.Widget, error) {
  // ...
  return OpenWidget(ctx, o.Client, u.Host, &o.Options)
}

type lazyURLOpener struct {
  init   sync.Once
  opener *URLOpener
  err    error
}

func (o *lazyURLOpener) OpenWidgetURL(ctx context.Context, u *url.URL) (*foo.Widget, error) {
  o.init.Once(func() {
    creds, err := gcp.DefaultCredentials(ctx)
    if err != nil {
      o.err = err
      return
    }
    o.opener = new(URLOpener)
    o.opener.Client, _ = gcp.NewHTTPClient(http.DefaultTransport, creds.TokenSource)
  })
  if o.err != nil {
    return nil, o.err
  }
  return o.opener.OpenWidgetURL(ctx, u)
}

func init() {
  foo.DefaultURLMux().Register(Scheme, new(lazyURLOpener))
}

// OpenWidget is the exported non-URL constructor.
func OpenWidget(ctx context.Context, c *gcp.HTTPClient, name string, opts *Options) (*foo.Widget, error) {
  // ...
}
```

Using the global default mux:

```go
import _ "gocloud.dev/foo/gcpfoo"

// ...

widget, err := foo.OpenWidget(context.Background(), "gcpwidget://xyzzy")
```

Using a custom mux created during server initialization:

```go
myMux := new(foo.URLMux)
myMux.Register(gcpfoo.Scheme, &gcpfoo.URLOpener{
  Client: client,
})
widget, err := myMux.OpenWidget(context.Background(), "gcpwidget://xyzzy")
```

## Errors

### General

-   The callee is expected to return `error`s with messages that include
    information about the particular call, as opposed to the caller adding this
    information. This aligns with common Go practice.

### Drivers

Driver implementations should:

-   Return the raw errors from the underlying service, and not wrap them in
    `fmt.Errorf` calls, so that they can be exposed to end users via `ErrorAs`.

### Portable Types

Portable types should:

-   Wrap errors returned from driver implementations before returning them to
    end users, so that users can't peek into driver-specific error details
    without using `As`. Make sure not to double-wrap.

-   Use `internal/gcerr.New` when wrapping driver errors, like so: `if err :=
    driver.Call(xyz); err != nil { return gcerr.New(code, err, 1, "blob") }` The
    first argument is an error code. See below for advice on choosing the
    appropriate code.

    The third argument is the distance in stack frames from the function whose
    location should be associated with the error. It should be `1` if you are
    calling `New` from the same function that made the driver call, `2` if you
    are calling new from a helper function, and so on. The fourth argument is an
    additional string that will display with the error. You should pass the API
    name.

-   By default, choose the code `Unknown`, keeping details of returned `error`s
    unspecified. The most common case is that the caller will only care whether
    an operation succeeds or not.

-   If certain `error`s are interesting for callers to distinguish, choose one
    of the other codes from the `gcerrors.ErrorCode` enum, so user programs can
    act on the kind of error without having to look at driver-specific errors.

    -   If more than one error code makes sense, choose the most specific one.
    -   If none make sense, choose `Unknown`.
    -   If none make sense but you want something more specific than `Unknown`:
        -   If you can generalize your code to make it applicable to more than
            just your API, add it to `gcerrors.ErrorCode`. Look at the
            [gRPC error codes](https://github.com/grpc/grpc-go/blob/master/codes/codes.go)
            for inspiration.
        -   Otherwise, you can define a custom code in your portable API
            package. Your code should use a negative integer.

-   For now, your package should expose an `ErrorAs` function to allow users to
    access driver-specific error types. We may review this choice if
    `golang.org/x/xerrors.As` becomes part of the standard library.

-   Handle transient network errors. Retry logic is best handled as low in the
    stack as possible to avoid [cascading failure][]. APIs should try to surface
    "permanent" errors (e.g. malformed request, bad permissions) where
    appropriate so that application logic does not attempt to retry
    non-idempotent operations, but the responsibility is largely on the library,
    not on the application.

[cascading failure]:
https://landing.google.com/sre/book/chapters/addressing-cascading-failures.html

## Escape Hatches using As

The Go CDK allows users to escape the abstraction as needed using `As`
functions, described in more detail in the
[concept guide](https://gocloud.dev/concepts/as/). `As` functions take an
`interface{}` and return a `bool`; they return `true` if the underlying concrete
type could be converted into the type provided as the `interface{}`.

An alternative approach would have been something like
[`os.ProcessState.Sys`](https://golang.org/pkg/os/#ProcessState.Sys), which
returns an `interface{}` that the user can then type cast/assert to
service-specific types.

We ended up going with `As` because:

1.  Most portable types have an `As` function for errors; choosing `As` results
    in an easy and natural implementation for chained errors once the
    [Go 2 proposal for errors](https://go.googlesource.com/proposal/+/master/design/29934-error-values.md)
    arrives. It is currently implemented in
    [xerrors](https://godoc.org/golang.org/x/xerrors#As), and we're already
    using that in some drivers.
2.  `As` adds more flexibility for drivers to support conversions to multiple
    types. Specifically, not the case where there are multiple possible
    underlying types, but rather that a single underlying type can be converted
    to multiple types.
    *   Chained errors is one example of this, where the top-level error may
        always be the same type, but may also represent a chain of other errors
        with different types.
    *   Another example is that a driver might choose to support `As`-level
        compatibility with another driver; e.g., driver `foo` could support all
        of the `As` types defined by `s3blob`, converting them internally, and
        then any code that runs with driver `s3blob` would also work with driver
        `foo` (even if it uses the `As` escape hatches).

## Enforcing Portability

The Go CDK APIs will end up exposing functionality that is not supported by all
services. In addition, some functionality details will differ across services.
Some theoretical examples using [`blob.Bucket`][]:

1.  **Top-level APIs**: There might be a service that supports reads, but not
    writes or deletes.
1.  **Data fields**. Some services may support key/value metadata associated
    with a blob, others may not.
1.  **Naming rules**. Different services may allow different name lengths, or
    allow/disallow non-ASCII unicode characters. See [Strings](#strings) below
    for more on handling string differences.
1.  **Semantic guarantees**. Different services may have different consistency
    guarantees; for example, S3 only provides eventually consistency while GCS
    provides strong consistency.

How can we maintain portability while these differences exist?

### Guiding Principle

Any incompatibilities between drivers should be visible to the user as soon as
possible. From best to worst:

1.  At compile time
1.  At configuration/app startup time (e.g., when the portable type is created)
1.  At runtime (e.g., when the incompatible behavior is accessed), via a non-nil
    error
1.  At runtime, via panic

### Approaches Considered

1.  **Documentation**. We could try to document non-uniform or optional
    functionality across drivers. Optional fields or functionality would
    return "not implemented" errors or zero values.
1.  **Restrict functionality to the intersection**. We could explicitly only
    support the intersection of all services. For example, if not all services
    allow unicode characters in names, then **blob** would not allow it either.
1.  **Enforced feature codes**: Go CDK APIs could enumerate the ways in which
    drivers differ as a `FeatureCode` enum.
    *   Drivers would declare which feature codes they support, enforced by
        extensions to the existing conformance tests.
    *   API users would declare which feature codes they need.
    *   Mismatches between what a user requests and what the driver supports
        would be enforced at initialization time.
    *   As much as possible, the API (via the portable type) would enforce that
        the user is only exposed to optional functionality that they asked for.
    *   For example, the default legal name for a blob might be ASCII only, with
        a `FeatureUnicodeNames` feature code. Users that don't request this
        feature code would only be able to use blobs with ASCII names, even if
        the underlying service supports unicode. If the user requested
        `FeatureUnicodeNames`, and their driver supports it, they could then
        use blobs with unicode; if their driver doesn't support it, they would
        get an initialization-time error.

```
b, err := blob.NewBucket(d, blob.FeatureUnicodeNames)
...
```

Design discussions regarding enforcing portability are ongoing; we welcome input
on the [mailing list](https://groups.google.com/forum/#!forum/go-cloud).

### Strings

Services often differ on what they accept in particular strings (e.g., blob
names, metadata keys, etc.). A couple of specific examples:

*   Azure Blob only
    [accepts C# identifiers](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata)
    as metadata keys.
*   S3 drops double slashes in blob names (e.g., `foo//bar` will end up being
    saved as `foo/bar`).

These differences lead to a loss of portability and predictability for users.

To resolve this issue, we insist that Go CDK can handle any UTF-8 string, and
force drivers to use escaping mechanisms to handle strings that the underlying
service can't handle. We enforce driver compliance with conformance tests.
Behavior for non-UTF-8 strings is undefined (but see
https://github.com/google/go-cloud/issues/1281 and
https://github.com/google/go-cloud/issues/1260).

We try to use URL encoding as the escaping mechanism where possible; however,
sometimes it is not and we'll use custom escaping. As an example, a driver for a
service that only allows underscores and ASCII alphanumeric characters might
escape the string `foo.bar` to `foo__0x2e__bar` (URL escaping won't work because
`%` isn't allowed).

Pros of this approach:

*   Go CDK APIs are internally consistent in that a user can write any string to
    any service and get the original string back when they read it back.
*   Go CDK APIs have visibility into all existing strings for all services.

Cons:

*   Go CDK could overwrite existing data if a Go CDK-written key escapes to an
    already-existing value (e.g., if the `foo__0x2e__bar` string already
    existed, it would be overwritten by a Go CDK write to `foo.bar`).
*   Escaping may push a string over the maximum allowed string length for a
    service. Escaping does not solve (and in fact may exacerbate) problems with
    different maximum string lengths across services.
*   Existing strings that happen to look like Go CDK-escaped strings will be
    unescaped by Go CDK (e.g., an existing string `foo__0x2e__bar` would appear
    as `foo.bar` when read through the Go CDK).
*   Strings that were written through the Go CDK and needed escaping will appear
    in their escaped form when viewed outside of Go CDK (e.g., `foo__0x2e__bar`
    would appear on the service's UI).

Most of these cons are mitigated by choosing unusual-looking escape mechanisms
that are unlikely to appear in existing data.

Drivers should escape strings when writing to the underlying service, and
unescape them when reading them back. The Go CDK will provide helpers for these
operations, as well as a test suite of strings for conformance tests.

Sample code for the helper for escaping strings:

```
// package escape provides helpers for escaping and unescaping strings.
package escape

// Escape returns s, with all runes for which shouldEscape returns true
// escaped to "__0xXXXX__", where XXXX is the hex representation of the rune
// value. For example, " " would escape to "__0x20__".
//
// Non-UTF-8 strings will have their non-UTF-8 characters escaped to
// unicode.ReplacementChar; the original value is lost. Please file an
// issue if you need non-UTF8 support.
//
// Note: shouldEscape takes the whole string as a slice of runes and an
// index. Passing it a single byte or a single rune doesn't provide
// enough context for some escape decisions; for example, the caller might
// want to escape the second "/" in "//" but not the first one.
// We pass a slice of runes instead of the string or a slice of bytes
// because some decisions will be made on a rune basis (e.g., encode
// all non-ASCII runes).
func Escape(s string, shouldEscape func(s []rune, i int) bool) string { ... }

// Unescape reverses Escape.
func Unescape(s string) string {...}
```

Sample code for how a driver might use it, using metadata keys for a `blob` as
the example string:

```
// When writing metadata keys, escape the keys:
// ... gcdkMetadata is the metadata passed to the GCDK API.
for k, v := range gcdkMetadata {
    e := escape.Escape(k, func (r []rune, i int) bool {...})
    if _, ok := serviceMetadata[e]; ok {
      return fmt.Errorf("duplicate keys after escaping: %q => %q", k, e)
    }
    serviceMetadata[e] = v
}
// ... write serviceMetadata to the service.

// When reading metadata keys, unescape them:
// ... serviceMetadata is the metadata read from the service.
for k, v := range serviceMetadata {
    gcdkMetadata[escape.Unescape(k)] = v
}
// ... return gcdkMetadata.
```

The details of what runes need to be escaped will vary from service to
service. The details of how to escape may also vary, although we expect to use
URL encoding where possible, and a common custom escaping where not. For the
custom escaping, we plan to escape each rune for which `shouldEscape` returns
true with `__0xXXX__`, where `XCX` is the hex representation of the rune value.

### Alternatives Considered

*   We considered restricting Go CDK's APIs to strings that all services
    support. For example, we could have asserted that Go CDK's `blob` only
    supports ASCII plus `/` for blob names (and no `//`!). However, such a rule
    would mean that we couldn't cleanly handle existing strings created through
    some mechanism other than through Go CDK APIs that violate the rule. For
    example, an existing blob in S3 with a unicode name. Filtering out such
    strings so that they aren't visible at all through the Go CDK would be both
    surprising and limiting, and could easily result in data loss (e.g., if a
    user read a set of metadata for a blob via the Go CDK, and some keys were
    filtered out, and then wrote the metadata back, the filtered keys would be
    lost). Not filtering such strings would mean that the Go CDK isn't
    internally consistent (i.e., you can read some strings but not write them).
    Overall, we decided that this approach is unacceptable.

*   We could expose the escaper used by drivers in their `Options` structs
    (including options like disabling it, overriding the set of bytes to be
    escaped, or overriding the escaping mechanism), but we'll wait to see if
    there's demand for that.

## Coding Conventions

We try to adhere to commonly accepted Go coding conventions, some of which are
described on the
[Code Review Comments](https://github.com/golang/go/wiki/CodeReviewComments)
wiki page. We also adopt the following guidelines:

-   Prefer `map[K]V{}` to `make(map[K]V)`. It's more concise.
-   When writing a loop appending to a slice `s`, prefer

    ```
      var s []T
      for ... {
        ...
        s = append(s, ...)
        ...
      }
    ```

    to

    ```
      s := make([]T, 0, N)
      for ... {
        ...
        s = append(s, ...)
        ...
      }
    ```

    or

    ```
      s := make([]T, N)
      for ... {
        ...
        s[i] = ...
        ...
      }
    ```

    (Exception: the loop body is trivial and the loop is performance-sensitive.)
    The first version is shorter and easier to read, and it is impossible to get
    the length wrong.

-   Prefer `log.Fatal` to `panic` in example tests.

-   Ensure you've run `goimports` on your code to properly group import
    statements.

-   Order arguments that are less likely to change across multiple calls to the
    constructor before ones that are likely to change. For example, connection
    and authorization related arguments should go before names, so
    `OpenBucket(ctx, client, "mybucket")` instead of `OpenBucket(ctx,
    "mybucket", client)`.

## Tests

### Conformance Tests

Since our goal is for users to be able to use drivers interchangeably, it is
critical that they behave similarly. To this end, each portable API (e.g.,
`blob`) must provide a suite of conformance tests that driver implementations
should run. The conformance tests should be comprehensive; drivers should not
need additional unit tests for the core driver semantics.

### Provisioning For Tests

Portable API integration tests require developer-specific resources to be
created and destroyed. We use [Terraform](http://terraform.io) to do so, and
record the resource info and network interactions so that they can be replayed
as fast and repeatable unit tests.

### Replay Mode

Tests normally run in replay mode. In this mode, they don't require any
provisioned resources or network interactions. Replay tests verify that:

-   The same test inputs produce the same requests (e.g., HTTP requests) to the
    cloud service. Some parts of the request may be dynamic (e.g., dates in the
    HTTP request headers), so the replay tests do some scrubbing when verifying
    that requests match. Some parts of this scrubbing are service-specific.

-   The replayed service responses produce the expected results from the
    portable API library.

### Record Mode

In `-record` mode, tests run as integration tests, making live requests to
backend servers and recording the requests/responses for later use in replay
mode.

To use `-record`:

1.  Provision resources.

    -   For example, the tests for the AWS implementation of Blob requires a
        bucket to be provisioned.
    -   TODO(issue #300): Use Terraform scripts to provision the resources
        needed for a given test.
    -   For now, do this manually.

2.  Run the test with `-record`.

    -   TODO(issue #300): The test will read the Terraform output to find its
        inputs.
    -   For now, pass the required resources via test-specific flags.
    -   When changing or adding tests, please only record the tests that are
        changed/affected by passing the `-run` flag to `go test` with the
        name of the test(s). Re-recording all tests of a driver creates a lot
        of noise and a large diff that's difficult to review.

3.  The test will save the network interactions for subsequent replays.

    -   TODO(issue #300): The test will save the Terraform output to a file in
        order to replay using the same inputs.
    -   Commit the new replay files along with your code change. Expect to see
        lots of diffs; see below for more discussion.

### Diffs in replay files

Each time portable API tests are run in `-record` mode, the resulting replay
files are different. Looking at diffs of these files isn't particularly useful.

We [considered](https://github.com/google/go-cloud/issues/276) trying to scrub
the files of dynamic data so that diffs would be useful. We ended up deciding
not to do this, for several reasons:

-   There's a lot of dynamic data, in structured data of various forms (e.g.,
    HTTP headers, XML/JSON body, etc.). It would be difficult and fragile to
    scrub it all.

-   The scrub process would also be fragile relative to changes in services
    (e.g., adding a new dynamic HTTP response header).

-   The scrub process would need to be implemented for every new service,
    increasing the barrier to entry for new implementations.

-   Scrubbing would likely be even more difficult for services using a
    non-HTTP-based protocol (e.g., gRPC).

-   Scrubbing the data decreases the fidelity of the replay test, since it
    wouldn't be operating on the original data.

Overall, massive diffs in the replay files are expected and fine. As part of a
code change, you may want to check for things like the number of RPCs made to
identify performance regressions.

---

### Internal Docs Release (`internal/docs/release.md`)

# Releases

To do a release:

-   Pick the new release name; it's probably `v0.x.0` where `x` is whatever the
    [last release](https://github.com/google/go-cloud/releases/latest) was plus
    one, but follow [semantic versioning](https://semver.org/).

-   Consider updating dependencies via `internal/testing/update_deps.sh` if it
    hasn't been done recently. Do this as a separte step before the release.

-   Create a new branch for the release (`git checkout -B prerelease`).

-   Update the `User-Agent` version in internal/useragent/useragent.go.

-   Run the release helper tool to remove `replace` lines from the `go.mod`
    files of submodules:

    ```bash
    $ go run internal/releasehelper/releasehelper.go dropreplace
    ```

-   Run the release helper tool to set the version in `require` directives of
    submodules to the new (yet unreleased) version:

    ```bash
    $ go run internal/releasehelper/releasehelper.go setversion v0.x.0
    ```

-   Commit and create a PR. Tests will fail for this PR because submodules depend on a
    version of the main module that wasn't tagged yet, so you may have to
    force-merge the PR. Note that this does not affect users, since a new
    version hasn't been tagged yet.

-   `git sync` your local client and move to the master branch.

-   Tag new versions by running `./internal/testing/git_tag_modules.sh v0.X.0`.

-   Push tags to upstream with `git push upstream --tags`

-   Go to [Releases](https://github.com/google/go-cloud/releases). Click `Draft
    a new release`, enter your release name, select your tag from the dropdown,
    and enter release notes by clicking "Generate Release Notes".

    -   Add a section for breaking changes, if any. They should be marked with
        `BREAKING_CHANGE` in the PR title; however, that's not enforced so do
        your best to look for them.
    -   Update the list of changes to remove anything that's not interesting
        (e.g., updating dependencies, prerelease, minor cleanups, etc.).

-   Send an email to
    [go-cloud@googlegroups.com](https://groups.google.com/forum/#!forum/go-cloud)
    announcing the release, and including the release notes.

-   Create a new branch for the postrelease (`git checkout -B postrelease`).

-   Add back `replace` lines:

    ```bash
    $ go run internal/releasehelper/releasehelper.go addreplace
    ```

    Run tests and send out a PR as usual.

---

### Internal Docs Pubsub Design (`internal/docs/pubsub/design.md`)

# Go CDK `pubsub` Design

## Summary

This document proposes a new `pubsub` package for the Go CDK.

## Motivation

A developer designing a new system with cross-cloud portability in mind could
choose a messaging system supporting pubsub, such as ZeroMQ, Kafka or RabbitMQ.
These pubsub systems run on AWS, Azure, GCP and others, so they pose no obstacle
to portability between clouds. They can also be run on-prem. Users wanting
managed pubsub could go with Confluent Cloud for Kafka (AWS, GCP), or CloudAMQP
for RabbitMQ (AWS, Azure) without losing much in the way of portability.

So what’s missing? The solution described above means being locked into a
particular implementation of pubsub. There is also a potential for lock-in when
building systems in terms of the cloud-specific services such as AWS SNS+SQS,
GCP PubSub or Azure Service Bus.

Developers may wish to compare different pubsub systems in terms of their
performance, reliability, cost or other factors, and they may want the option to
move between these systems without too much friction. A `pubsub` package in the
Go CDK could lower the cost of such experiments and migrations.

## Goals

*   Publish messages to an existing topic.
*   Receive messages from an existing subscription.
*   Perform not much worse than 90% compared to directly using the APIs of
    various pubsub systems.
*   Work well with managed pubsub services on AWS, Azure, GCP and the most used
    open source pubsub systems.

## Non-goals

*   Create new topics in the cloud. The Go CDK focuses on developer concerns,
    but topic creation is an
    [operator concern](https://github.com/google/go-cloud/blob/master/internal/docs/design.md#developers-and-operators).

*   Create new subscriptions in the cloud. The subscribers are assumed to
    correspond to components of a distributed system rather than to users of
    that system.

## Background

[Pubsub](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern) is a
frequently requested feature for the Go CDK project
\[[github issue](https://github.com/google/go-cloud/issues/312)]. A key use case
motivating these requests is to support
[event driven architectures](https://en.wikipedia.org/wiki/Event-driven_architecture).

There are several pubsub systems available that could be made to work with the
Go CDK by writing drivers for them. Here is a
[table](https://docs.google.com/a/google.com/spreadsheets/d/e/2PACX-1vQ2CML8muCrqhinxOeKTcWtwAeGk-RFFFMjB3O2u5DbbBt9R3YnUQcgRjRp6TySXe1CzSOtPVCsKACY/pubhtml)
comparing some of them.

## Design overview

### Developer’s perspective

Given a topic that has already been created on the pubsub server, messages can
be sent to that topic by calling `acmepubsub.OpenTopic` and calling the `Send`
method of the returned `Topic`, like this (assuming a fictional pubsub service
called "acme"):

```go
package main

import (
    "context"
    "log"
    "net/http"

    rawacmepubsub "github.com/acme/pubsub"
    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/acmepubsub"
)

func main() {
    log.Fatal(serve())
}

func serve() error {
    ctx := context.Background()
    client, err := rawacmepubsub.NewClient(ctx, "unicornvideohub")
    if err != nil {
        return err
    }
    t, err := acmepubsub.OpenTopic(ctx, client, "user-signup", nil)
    if err != nil {
        return err
    }
    defer t.Close()
    http.HandleFunc("/signup", func(w http.ResponseWriter, r *http.Request) {
        err := t.Send(r.Context(), pubsub.Message{Body: []byte("Someone signed up")})
        if err != nil {
            log.Println(err)
        }
    })
    return http.ListenAndServe(":8080", nil)
}
```

The call to `Send` will only return after the message has been sent to the
server or its sending has failed.

Messages can be received from an existing subscription to a topic by calling the
`Receive` method on a `Subscription` object returned from
`acmepubsub.OpenSubscription`, like this:

```go
package main

import (
    "context"
    "fmt"
    "log"

    rawacmepubsub "github.com/acme/pubsub"
    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/acmepubsub"
)

func main() {
    if err := receive(); err != nil {
        log.Fatal(err)
    }
}

func receive() error {
    ctx := context.Background()
    client, err := rawacmepubsub.NewClient(ctx, "unicornvideohub")
    if err != nil {
        return err
    }
    s, err := acmepubsub.OpenSubscription(ctx, client, "user-signup-minder", nil)
    if err != nil {
        return err
    }
    defer s.Close()
    msg, err := s.Receive(ctx)
    if err != nil {
        return err
    }
    // Do something with msg.
    fmt.Printf("Got message: %s\n", msg.Body)
    // Acknowledge that we handled the message.
    msg.Ack()
}
```

A more realistic subscriber client would process messages in a loop, like this:

```go
package main

import (
    "context"
    "log"
    "os"
    "os/signal"

    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/acmepubsub"
)

func main() {
    if err := receive(); err != nil {
        log.Fatal(err)
    }
}

func receive() error {
    ctx := context.Background()
    client, err := rawacmepubsub.NewClient(ctx, "unicornvideohub")
    if err != nil {
        return err
    }
    s, err := acmepubsub.OpenSubscription(ctx, client, "signup-minder", nil)
    if err != nil {
        return err
    }
    defer s.Close()

    // Process messages.
    for {
        msg, err := s.Receive(ctx)
        if err {
            return err
        }
        log.Printf("Got message: %s\n", msg.Body)
        msg.Ack()
    }
}
```

The messages can be processed concurrently with an
[inverted worker pool](https://www.youtube.com/watch?v=5zXAHh5tJqQ&t=26m58s),
like this:

```go
package main

import (
    "context"
    "log"
    "os"
    "os/signal"

    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/acmepubsub"
)

func main() {
    if err := receive(); err != nil {
        log.Fatal(err)
    }
}

func receive() error {
    ctx := context.Background()
    client, err := rawacmepubsub.NewClient(ctx, "unicornvideohub")
    if err != nil {
        return err
    }
    s, err := acmepubsub.OpenSubscription(ctx, client, "user-signup-minder", nil)
    if err != nil {
        return err
    }
    defer s.Close()

    // Process messages.
    const poolSize = 10
    // Use a buffered channel as a semaphore.
    sem := make(chan struct{}, poolSize)
    for {
        msg, err := s.Receive(ctx)
        if err {
            return err
        }
        sem <- struct{}{}
        go func() {
            log.Printf("Got message: %s", msg.Body)
            msg.Ack()
            <-sem
        }()
    }
    for n := poolSize; n > 0; n-- {
        sem <- struct{}{}
    }
}
```

### Driver implementer’s perspective

Adding support for a new pubsub system involves the following steps, continuing
with the "acme" example:

1.  Add a new package called `acmepubsub`.
2.  Add private `topic` and `subscription` types to `acmepubsub` implementing
    the corresponding interfaces in the `github.com/go-cloud/pubsub/driver`
    package.
3.  Add `func OpenTopic(...)` that creates an `acmepubsub.topic` and returns a
    concrete `pubsub.Topic` object made from it.
4.  Add `func OpenSubscription(...)` that creates an `acmepubsub.subscription`
    and returns a `pubsub.Subscription` object made from it.

Here is a sketch of what the `acmepubsub` package could look like:

```
/* Detailed source-code truncated for AI context efficiency. */
```

The driver interfaces are batch-oriented because some pubsub systems can more
efficiently deal with batches of messages than with one at a time. Streaming was
considered but it does not appear to provide enough of a performance gain to be
worth the additional complexity of supporting it across different pubsub systems
\[[benchmarks](https://github.com/ijt/pubsub/tree/master/benchmarks)].

The driver interfaces will be located in the
`github.com/google/go-cloud/pubsub/driver` package and will look something like
this:

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Detailed design

The developer experience of using Go CDK's pubsub involves sending, receiving
and acknowledging one message at a time, all in terms of synchronous calls.
Behind the scenes, the driver implementations deal with batches of messages and
acks. The concrete API, to be written by the Go CDK team, takes care of creating
the batches in the case of Send or Ack, and dealing out messages one at a time
in the case of Receive.

The concrete API will be located at `github.com/google/go-cloud/pubsub` and will
look something like this:

```
/* Detailed source-code truncated for AI context efficiency. */
```

Topics will gather messages into batches for sending. The batch size will be
dynamically tuned according to how many messages are being sent concurrently.

Subscriptions will gather message acks into batches the same way, also
dynamically tuning the batch size. If sending acks back to the server fails
transiently then it will be retried, most likely within a loop in the concrete
API. If an unrecoverable error occurs while sending acks then a flag will be set
on the `pubsub.Subscription` saying that the whole `Subscription` is no longer
usable. Calls to `Receive` will fail from then on.

## Alternative designs considered

### Batch oriented concrete API

In this alternative, the application code sends, receives and acknowledges
messages in batches. Here is an example of how it would look from the
developer's perspective, in a situation where not too many signups are happening
per second.

```go
package main

import (
    "context"
    "log"
    "net/http"

    rawacmepubsub "github.com/acme/pubsub"
    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/acmepubsub"
)

func main() {
    log.Fatal(serve())
}

func serve() error {
    ctx := context.Background()
    client, err := rawacmepubsub.NewClient(ctx, "unicornvideohub")
    if err != nil {
        return err
    }
    t, err := acmepubsub.OpenTopic(ctx, client, "user-signup", nil)
    if err != nil {
        return err
    }
    defer t.Close()
    http.HandleFunc("/signup", func(w http.ResponseWriter, r *http.Request) {
        err := t.Send(r.Context(), []pubsub.Message{{Body: []byte("Someone signed up")}})
        if err != nil {
            log.Println(err)
        }
    })
    return http.ListenAndServe(":8080", nil)
}
```

For a company experiencing explosive growth or enthusiastic spammers creating
more signups than this simple-minded implementation can handle, the app would
have to be adapted to create non-singleton batches, like this:

```go
package main

import (
    "context"
    "log"
    "net/http"

    rawacmepubsub "github.com/acme/pubsub"
    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/acmepubsub"
)

const batchSize = 1000

func main() {
    log.Fatal(serve())
}

func serve() error {
    ctx := context.Background()
    client, err := rawacmepubsub.NewClient(ctx, "unicornvideohub")
    if err != nil {
        return err
    }
    t, err := acmepubsub.OpenTopic(ctx, client, "user-signup", nil)
    if err != nil {
        return err
    }
    defer t.Close()
    c := make(chan *pubsub.Message)
    go sendBatches(ctx, t, c)
    http.HandleFunc("/signup", func(w http.ResponseWriter, r *http.Request) {
        c <- &pubsub.Message{Body: []byte("Someone signed up")}
    })
    return http.ListenAndServe(":8080", nil)
}

func sendBatches(ctx context.Context, t *pubsub.Topic, c chan *pubsub.Message) {
    batch := make([]*pubsub.Message, batchSize)
    for {
        for i := 0; i < batchSize; i++ {
            batch[i] = <-c
        }
        if err := t.Send(ctx, batch); err != nil {
            log.Println(err)
        }
    }
}
```

This shows how the complexity of batching has been pushed onto the application
code. Removing messages from the batch when HTTP/2 requests are canceled would
require the application code to be even more complex, adding more risk of bugs.

In this API, the application code has to either request batches of size 1,
meaning more network traffic, or it has to explicitly manage the batches of
messages it receives. Here is an example of how this API would be used for
serial message processing:

```go
package main

import (
    "context"
    "log"
    "os"
    "os/signal"

    rawacmepubsub "github.com/acme/pubsub"
    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/acmepubsub"
)

const batchSize = 10

func main() {
    if err := receive(); err != nil {
        log.Fatal(err)
    }
}

func receive() error {
    ctx := context.Background()
    client, err := rawacmepubsub.NewClient(ctx, "unicornvideohub")
    if err != nil {
        return err
    }
    s, err := acmepubsub.OpenSubscription(ctx, client, "signup-minder", nil)
    if err != nil {
        return err
    }
    defer s.Close()

    // Process messages.
    for {
        msgs, err := s.Receive(ctx, batchSize)
        if err {
            return err
        }
        acks := make([]pubsub.AckID, 0, batchSize)
        for _, msg := range msgs {
            // Do something with msg.
            fmt.Printf("Got message: %q\n", msg.Body)
            acks = append(acks, msg.AckID)
        }
        err := s.SendAcks(ctx, acks)
        if err != nil {
            return err
        }
    }
}
```

Here’s what it might look like to use this batch-only API with the inverted
worker pool pattern:

```
/* Detailed source-code truncated for AI context efficiency. */
```

Here are some trade-offs of this design:

Pro:

*   The semantics are simple, making it
    *   straightforward to implement the concrete API and the drivers for most
        pubsub services
    *   easy for developers to reason about how it will behave
    *   less risky that bugs will be present in the concrete API
*   Fairly efficient sending and receiving of messages is possible by tuning
    batch size and the number of goroutines sending or receiving messages.

Con:

*   This style of API makes the inverted worker pool pattern verbose.
*   Apps needing to send or receive a large volume of messages must have their
    own logic to create batches of size greater than 1.

### go-micro

Here is an example of what application code could look like for a pubsub API
inspired by [`go-micro`](https://github.com/micro/go-micro)'s `broker` package:

```go
b := somepubsub.NewBroker(...)
if err := b.Connect(); err != nil {
    /* handle err */
}
topic := "user-signups"
subID := "user-signups-subscription-1"
s, err := b.Subscription(ctx, topic, subID, func(pub broker.Publication) error {
    fmt.Printf("%s\n", pub.Message.Body)
    return nil
})
if err := b.Publish(ctx, topic, &broker.Message{ Body: []byte("alice signed up") }); err != nil {
    /* handle err */
}
// Sometime later:
if err := s.Unsubscribe(ctx); err != nil {
    /* handle err */
}
```

Pro:

*   The callback to the subscription returning an error to decide whether to
    acknowledge the message means the developer cannot forget to ack.

Con:

*   Go micro has code to auto-create
    [topics](https://github.com/micro/go-plugins/blob/f3fcfcdf77392b4e053c8d5b361abfabc0c623d3/broker/googlepubsub/googlepubsub.go#L152)
    and
    [subscriptions](https://github.com/micro/go-plugins/blob/f3fcfcdf77392b4e053c8d5b361abfabc0c623d3/broker/googlepubsub/googlepubsub.go#L185)
    as needed, but this is not consistent with the Go CDK’s design principle to
    not get involved in operations.
*   The subscription callback idea does not appear to be compatible with
    inverted worker pools.

## Acknowledgements

In pubsub systems with acknowledgement, messages are kept in a queue associated
with the subscription on the server. When a client receives one of these
messages, its counterpart on the server is marked as being processed. Once the
client finishes processing the message, it sends an acknowledgement (or "ack")
to the server and the server removes the message from the subscription queue.
There may be a deadline for the acknowledgement, past which the server unmarks
the message so that it can be received again for another try at processing.

Redis Pub/Sub and ZeroMQ don’t support acking, but many others do including GCP
PubSub, Azure Service Bus, RabbitMQ, and
[Redis Streams](https://redis.io/topics/streams-intro). Given the wide support
and usefulness, it makes sense to support message acking in the Go CDK.

As of this writing, it is an open question as to what should be done about
pubsub systems that do not support acks. Some possibilities have been discussed,
but no clear best option has emerged yet:

1.  simulating acknowledgement by constructing queues on the server. Con: the
    magically created queues would probably be a less than pleasant surprise for
    some users.
2.  making ack a no-op for systems that don't support it. With this, do we
    return a sentinel error from `Ack`, and if so then doesn't that unduly
    complicate the code for apps that never use non-acking systems? This option
    is also potentially misleading for developers who would naturally assume
    that un-acked messages would be redelivered.

### Rejected acknowledgement API: `Receive` method returns an `ack` func

In this alternative, the application code would look something like this:

```go
msg, ack, err := s.Receive(ctx)
log.Printf("Received message: %q", msg.Body)
ack(msg)
```

Pro:

*   The compiler will complain if the returned `ack` function is not used.

Con:

*   Receive has one more return value.
*   Passing `ack` around along with `msg` is inconvenient.

## Tests

### Unit tests for the concrete API (`github.com/go-cloud/pubsub`)

We can test that the batched sending, receiving and acking work as intended by
making mock implementations of the driver interfaces.

At least the following things should be tested:

*   Calling `pubsub.Message.Ack` causes `driver.Subscription.SendAcks` to be
    called.
*   Calling `pubsub.Topic.Send` causes `driver.Topic.SendBatch` to be called.
*   Calling `pubsub.Subscription.Receive` causes
    `driver.Subscription.ReceiveBatch` to be called.

### Conformance tests for specific implementations (*e.g.*, `github.com/go-cloud/pubsub/acmepubsub`)

*   Sent messages with random contents are received with the same contents.
*   Sent messages with random attributes are received with the same attributes.
*   Error occurs when making a local topic with an ID that doesn’t exist on the
    server.
*   Error occurs when making a subscription with an ID that doesn’t exist on the
    server.
*   Message gets sent again after ack deadline if a message is never
    acknowledged.
*   ~~Acked messages don't get received again after waiting twice the ack
    deadline.~~ :point_left: This test would probably be too flakey.

## Benchmarks

What is the throughput and latency of the Go CDK's `pubsub` package, relative to
directly using the APIs for various services?

*   send, for 1, 10, 100 topics, and for 1, 10, 100 goroutines sending messages
    to those topics
*   receive, for 1, 10, 100 subscriptions, and for 1, 10, 100 goroutines
    receiving from each subscription

## References

*   https://github.com/google/go-cloud/issues/312
*   http://queues.io/

---

--- METRICS ---
- Files Extracted: 12
- Estimated Token Budget: ~16420 tokens
- Canonical Reference: https://codewiki.google/github.com/google/go-cloud