go-cloud Documentation

File: README.md Copy Chapter

The Go Cloud Development Kit (Go CDK)

Write once, run on any cloud ☁️

The Go Cloud Development Kit (Go CDK) allows Go application developers to
seamlessly deploy cloud applications on any combination of cloud providers. It
does this by providing stable, idiomatic interfaces for common uses like storage
and databases. Think database/sql for cloud products.

Imagine writing this to read from blob storage (like Google Cloud Storage or
S3):

ctx := context.Background()
bucket, err := blob.OpenBucket(ctx, "s3://my-bucket")
if err != nil {
    return err
}
defer bucket.Close()
blobReader, err := bucket.NewReader(ctx, "my-blob", nil)
if err != nil {
    return err
}

and being able to run that code on any cloud you want, avoiding all the ceremony
of cloud-specific authorization, tracing, SDKs and all the other code required
to make an application portable across cloud platforms.

The project works well with a code generator called
Wire. It creates
human-readable code that only imports the cloud SDKs for services you use. This
allows the Go CDK to grow to support any number of cloud services, without
increasing compile times or binary sizes, and avoiding any side effects from
init() functions.

You can learn more about the project from our announcement blog post, or our
talk at Next 2018:

Installation Copy Chapter

# First "cd" into your project directory if you have one to ensure "go get" uses
# Go modules (or not) appropriately. See "go help modules" for more info.
go get gocloud.dev

The Go CDK builds at the latest stable release of Go. Previous Go versions may
compile but are not supported.

Documentation Copy Chapter

Documentation for the project lives primarily on https://gocloud.dev/, including
tutorials.

You can also browse Go package reference on pkg.go.dev.

Project status Copy Chapter

The APIs are still in alpha, but we think they are production-ready and are
actively looking for feedback from early adopters. If you have comments or
questions please open an issue.

At this time we prefer to focus on maintaining the existing APIs and drivers,
and are unlikely to accept new ones into the go-cloud repository. The modular
nature of the Go CDK makes it simple to host new APIs and drivers for existing
APIs externally, in separate repositories.

If you have a new API or driver that you believe are important and mature enough
to be included, feel free to open an issue to discuss this; our default will
likely be to suggest starting in a separate repository.

Known External Drivers Copy Section

• gocloud-ext — blob drivers for HTTP and WebDAV (httpblob) and for SFTP (sftpblob).

Current features Copy Chapter

The Go CDK provides generic APIs for:

• Unstructured binary (blob) storage
• Publish/Subscribe (pubsub)
• Variables that change at runtime (runtimevar)
• Document storage (docstore)
• Secrets (secrets)
• Connecting to MySQL (including MariaDB) and PostgreSQL databases (mysql, postgres)
• Server startup and diagnostics: request logging, tracing, and health
  checking (server)

Contributing Copy Chapter

Thank you for your interest in contributing to the Go Cloud Development
Kit! :heart:

Everyone is welcome to contribute, whether it's in the form of code,
documentation, bug reports, feature requests, or anything else. We encourage you
to experiment with the Go CDK and make contributions to help evolve it to meet
your needs!

The GitHub repository at google/go-cloud contains some driver
implementations for each portable API. We intend to include
Google Cloud Platform, Amazon Web Services, and Azure
implementations, as well as prominent open source services and at least one
implementation suitable for use in local testing. Unfortunately, we cannot
support every service directly from the project; however, we encourage
contributions in separate repositories.

If you create a repository that implements the Go CDK interfaces for other
services, let us know! We would be happy to link to it here and give you a
heads-up before making any breaking changes.

See the contributing guide for more details.

Community Copy Chapter

This project is covered by the Go Code of Conduct.

Legal disclaimer Copy Chapter

The Go CDK is open-source and released under an Apache 2.0
License. Copyright ©
2018–2019 The Go Cloud Development Kit Authors.

If you are looking for the website of GoCloud Systems, which is unrelated to the
Go CDK, visit https://gocloud.systems.

File: CODE_OF_CONDUCT.md Copy Chapter

Code of Conduct

This project is covered under the Go Code of Conduct. In summary:

• Treat everyone with respect and kindness.
• Be thoughtful in how you communicate.
• Don’t be destructive or inflammatory.
• If you encounter an issue, please mail [email protected].

File: CONTRIBUTING.md Copy Chapter

How to Contribute

We would love to accept your patches and contributions to this project. Here is
how you can help.

Filing issues Copy Chapter

Filing issues is an important way you can contribute to the Go Cloud Development
Kit. We want your feedback on things like bugs, desired API changes, or just
anything that isn't working for you.

Bugs Copy Section

If your issue is a bug, open one
here. The easiest way to file
an issue with all the right information is to run go bug. go bug will print
out a handy template of questions and system information that will help us get
to the root of the issue quicker. Please start the title of your bug with the
name of the affected package, followed by a colon, followed by a short summary
of the issue, like "blob/gcsblob: not blobby enough".

Changes Copy Section

Unlike the core Go project, we do not have a formal proposal process for
changes. If you have a change you would like to see in the Go CDK, please file
an issue with the necessary details.

Triaging Copy Section

The Go CDK team triages issues at least every two weeks, but usually within two
business days. Issues that we would like to address, but do not have time for
are placed into the Unplanned milestone.

Contributing Code Copy Chapter

We love accepting contributions! If your change is minor, please feel free
to submit a pull request.
If your change is larger, or adds a feature, please file an issue beforehand so
that we can discuss the change. You're welcome to file an implementation pull
request immediately as well, although we generally lean towards discussing the
change and then reviewing the implementation separately.

Be sure to take a look at the internal docs, which include more information
about conventions and design patterns found in the project.

Finding something to work on Copy Section

If you want to write some code, but don't know where to start or what you might
want to do, take a look at the Good First Issue label and our Unplanned
milestone. The latter is where you can find issues we would like to address, but
can't currently find time for. See if any of the latest ones look interesting!
If you need help before you can start work, you can comment on the issue, and we
will try to help as best we can.

Contributor License Agreement Copy Section

Contributions to this project can only be made by those who have signed Google's
Contributor License Agreement. You (or your employer) retain the copyright to
your contribution, so this simply gives us permission to use and redistribute your
contributions as part of the project. Head over to
https://cla.developers.google.com/ to see your current agreements on file or
to sign a new one.

As a personal contributor, you only need to sign the Google CLA once across all
Google projects. If you've already signed the CLA, there is no need to do it
again. If you are submitting code on behalf of your employer, there's
a separate corporate CLA that your employer manages for you.

A Warning About Tests Copy Section

Tests that interact with cloud providers are written using a replay method,
where the test is run and actually performs the operations, and the
requests/results of the operations are stored in a replay file. This replay file
is then read back in unit tests on Travis, so the tests get to operate with real
data. Unfortunately, while the Go CDK team can generate these replay files
against our test cloud infrastructure, it is not yet possible for external
contributors to do the same. We want to improve this process in the future and
are researching how we can do this. If you have any ideas, please
file an issue!

Writing and running tests against a cloud environment

If you can create cloud resources, setup your authentication using either aws
or gcloud and set the default project as the test project. Most tests will
have constants defining the resource names they use (for example, for blob
tests, the bucket name); update the constant and then run go test -record. New
replay files will be generated. This uses API quota and will create and delete
cloud resources. Replay files scrub sensitive information.
Send your PR without the replay files, and we can
generate new ones to be used by others.

Dependencies Copy Section

The Go CDK has a policy to depend only on code licensed under one of the
notice, permissive, or
unencumbered categories in the
Google Open Source Licenses documentation. This is enforced with a
Travis build check that verifies that every dependency is in the
alldeps file. Do not add new direct or indirect dependencies to the Go CDK
unless you have verified that the dependency is released under an acceptable
license.

Making a pull request Copy Chapter

• Follow the normal
  pull request flow.
• Build your changes using Go 1.11 with Go modules enabled. The Go CDK's
  continuous integration uses Go modules in order to ensure
  reproducible builds.
• Test your changes using go test ./.... Please add tests that show the
  change does what it says it does, even if there wasn't a test in the first
  place. Don't add the replay files to your commits.
• Feel free to make as many commits as you want; we will squash them all into
  a single commit before merging your change.
• Check the diffs, write a useful description (including something like
  Fixes #123 if it's fixing a bug) and send the PR out. Please start the
  title of your pull request with the name of the affected package, followed
  by a colon, followed by a short summary of the change, like "blob/gcsblob:
  add more tests".
• Travis CI will run tests against the PR. This should
  happen within 10 minutes or so. If a test fails, go back to the coding stage
  and try to fix the test and push the same branch again. You won't need to
  make a new pull request, the changes will be rolled directly into the PR you
  already opened. Wait for Travis again. There is no need to assign a reviewer
  to the PR, the project team will assign someone for review during the
  standard triage process.

Code review Copy Chapter

All submissions, including submissions by project members, require review. It is
almost never the case that a pull request is accepted without some changes
requested, so please do not be offended!

When you have finished making requested changes to your pull request, please
make a comment containing "PTAL" (Please Take Another Look) on your pull
request. GitHub notifications can be noisy, and it is unfortunately easy for
things to be lost in the shuffle.

Once your PR is approved (hooray!), the reviewer will squash your commits into a
single commit and then merge the commit onto the Go CDK master branch. Thank
you!

Github code review workflow conventions Copy Chapter

(For project members and frequent contributors.)

As a contributor:

• Try hard to make each Pull Request as small and focused as possible. In
  particular, this means that if a reviewer asks you to do something that is
  beyond the scope of the Pull Request, the best practice is to file another
  issue and reference it from the Pull Request rather than just adding more
  commits to the existing PR.
• Adding someone as a Reviewer means "please feel free to look and comment";
  the review is optional. Choose as many Reviewers as you'd like.
• Adding someone as an Assignee means that the Pull Request should not be
  submitted until they approve. If you choose multiple Assignees, wait until
  all of them approve. It is fine to ask someone if they are OK with being
  removed as an Assignee.
  • Note that if you don't select any assignees, ContributeBot will turn all
    of your Reviewers into Assignees.
• Make as many commits as you want locally, but try not to push them to Github
  until you've addressed comments; this allows the email notification about
  the push to be a signal to reviewers that the PR is ready to be looked at
  again.
• When there may be confusion about what should happen next for a PR, be
  explicit; add a "PTAL" comment if it is ready for review again, or a "Please
  hold off on reviewing for now" if you are still working on addressing
  comments.
• "Resolve" comments that you are sure you've addressed; let your reviewers
  resolve ones that you're not sure about.
• Do not use git push --force; this can cause comments from your reviewers
  that are associated with a specific commit to be lost. This implies that
  once you've sent a Pull Request, you should use git merge instead of git rebase to incorporate commits from the master branch.
• Travis checks will fail if you haven't run gofmt -w -s.
• Travis checks will fail if your PR has backwards-incompatible changes,
  unless one of your commits has the strings BREAKING_CHANGE_OK in the first
  line of the commit message.

As a reviewer:

• Be timely in your review process, especially if you are an Assignee.
• Try to use Start a Review instead of single comments, to reduce email
  spam.
• "Resolve" your own comments if they have been addressed.
• If you want your review to be blocking, and are not currently an Assignee,
  add yourself as an Assignee.

When squashing-and-merging:

• Ensure that all of the Assignees have approved.
• Do a final review of the one-line PR summary, ensuring that it meets the
  guidelines (e.g., "blob: add more blobbing") and accurately describes the
  change.
• Mark breaking changes with BREAKING_CHANGE in the commit message (e.g.,
  "blob: BREAKING_CHANGE remove old blob").
  • If the PR includes a breaking change, it will be declared via a commit
    with BREAKING_CHANGE_OK in it (see Contributor section above).
  • You can omit the marker if the change is technically breaking, but not
    expected to affect users (e.g., it's a breaking change to an object that
    wasn't in the last tagged release, or it's a change to a portable API
    function that's only expected to be used by driver implementations).
• Delete the automatically added commit lines; these are generally not
  interesting and make commit history harder to read.

File: .codecov.yml Copy Chapter

codecov:

Set via secrets, see .github/workflows/tests.yml.

token: ""
ignore:

• "**/drivertest/*.go"
• "samples"
  coverage:
  status:
  project:
  default:
  target: 0
  threshold: null
  base: auto
  
  patch:
  default:
  target: 0
  threshold: null
  base: auto

File: .golangci.yml Copy Chapter

version: "2"
linters:
settings:
govet:
disable:
# cannot inline call to reflect.PtrTo
- inline
staticcheck:
checks:
# Using a deprecated function, variable, constant or field.
# https://staticcheck.dev/docs/checks/#SA1019
- -SA1019

File: docstore/internal/fields/README.md Copy Chapter

This package is copied from cloud.google.com/go/internal/fields.

File: docstore/mongodocstore/docker-compose.yml Copy Chapter

services:
mongo:
image: mongo:4.4
container_name: mongo
restart: always
volumes:
- mongo_data:/data/db
ports:
- "27017:27017"
command: ["--replSet", "dbrs", "--bind_ip_all"]

mongosetup:
image: mongo:4.4
container_name: mongosetup
depends_on:
- mongo
entrypoint: >
bash -c "sleep 5 &&
mongo --host mongo:27017 --eval '
rs.initiate({
_id: "dbrs",
members: [
{ _id: 0, host: "mongo:27017"}
]
})
'"

volumes:
mongo_data:

File: internal/docs/README.md Copy Chapter

Docs

This is the documentation for developers of the Go CDK, describing various
coding practices and project processes.

• Design Decisions
• Releases

File: internal/docs/design.md Copy Chapter

Design Decisions

This document outlines important design decisions made for this repository and
attempts to provide succinct rationales. Recording these decisions helps
maintain consistency across packages, especially as an open source project where
contributors can join at any point during development.

A broad design goal for the Go Cloud Development Kit (Go CDK) is for the API
style to be consistent. Consistency aids users in building a mental model of how
to use the APIs. As such, the design of individual packages must also consider
their impact on the Go CDK as a whole.

This is a Living Document. The
decisions in here are not set in stone, but simply describe our current thinking
about how to guide the Go Cloud Development Kit project. While it is useful to
link to this document when having discussions in an issue, it is not to be used
as a means of closing issues without discussion at all. Discussion on an issue
can lead to revisions of this document.

Developers and Operators Copy Chapter

The Go CDK is designed with two different personas in mind: the developer and
the operator. In the world of DevOps, these may be the same person. A developer
may be directly deploying their application into production, especially on
smaller teams. In a larger organization, these may be different teams entirely,
but working closely together. Regardless, these two personas have two very
different ways of looking at a Go program:

• The developer persona wants to write business logic that is agnostic of
  underlying cloud provider. Their focus is on making software correct for the
  requirements at hand.
• The operator persona wants to incorporate the business logic into the
  organization's policies and provision resources for the logic to run. Their
  focus is making software run predictably and reliably with the resources at
  hand.

The Go CDK uses Go interfaces at the boundary between these two personas: a
developer is meant to use an interface, and an operator is meant to provide an
implementation of that interface. This distinction prevents the Go CDK going
down a path of complexity that makes application portability difficult. The
blob.Bucket type is a prime example: the API does not provide a way of
creating a new bucket. To properly and safely create such a bucket requires
careful consideration, getting something like ACLs wrong could lead to a
catastrophic data leak. To generate the ACLs correctly requires modeling of IAM
users and roles for each cloud platform, and some way of mapping those users and
roles across clouds. While not impossible, the level of complexity and the high
likelihood of a leaky abstraction leads us to believe this is not the right
direction for the Go CDK.

Instead of adding large amounts of leaky complexity to the Go CDK, we expect the
operator role to handle the management of non-portable platform-specific
resources. An implementor of the Bucket interface does not need to determine
the content type of incoming data, as that is a developer's concern. This
separation of concerns allows these two personas to communicate using a shared
language while focusing on their respective areas of expertise.

Portable Types and Drivers Copy Chapter

The portable APIs that the Go CDK exports (like blob.Bucket or
runtimevar.Variable) are concrete types, not interfaces. To understand
why, imagine if we used a plain interface:

Consider the Bucket.NewWriter method, which infers the content type of the
blob based on the first bytes written to it. If blob.Bucket was an interface,
each implementation of blob.Bucket would have to replicate this behavior
precisely. This does not scale: conformance tests would be needed to ensure that
each interface method actually behaves in the way that the docs describe. This
makes the interfaces hard to implement, which runs counter to the goals of the
project.

Instead, we follow the example of database/sql and separate out the
implementation-agnostic logic from the interface. The implementation-agnostic
logic-containing concrete type is the portable type. We call the interface
the driver. Visually, it looks like this:

This has a number of benefits:

• The portable type can perform higher level logic without making the
  interface complex to implement. In the blob example, the portable type's
  NewWriter method can do the content type detection and then pass the final
  result to the driver type.
• Methods can be added to the portable type without breaking compatibility.
  Contrast with adding methods to an interface, which is a breaking change.
• When new operations on the driver are added as new optional interfaces, the
  portable type can hide the need for type-assertions from the user.

As a rule, if a method Foo has the same inputs and semantics in the portable
type and the driver type, then the driver method may be called Foo, even
though the return signatures may differ. Otherwise, the driver method name
should be different to reduce confusion.

New Go CDK APIs should always follow this portable type and driver pattern.

Minimize Global State Copy Chapter

As a library, the Go CDK should not introduce global state. Global state is
difficult to reason about in large codebases, where it can be necessary for
different parts of the application to use different states. Instead of adding
global state, push responsibility to the application to inject the state where
it is needed.

The exception we permit is URL scheme registration as documented under
URLs. The amount of boilerplate setup code required for URL muxes for
multiple drivers without use of a tool like Wire is an unreasonable burden for
users of Go CDK. We want the Go CDK to be usable both with and without Wire. A
global registry is acceptable as long as its use is not mandatory, but the
burden is to prove the benefit over the cost.

Driver Package Naming Conventions Copy Chapter

Inside this repository, we name packages that handle cloud services after the
service name, not the providing cloud (s3blob instead of awsblob). While a
cloud provider may provide a unique offering for a particular API, they may not
always provide only one, so distinguishing them in this way keeps the API
symbols stable over time.

The naming convention is <api>, where:

• `` is the provider name, like aws or gcp or azure.
  • Omit for 3rd party/open source/local packages.
  • May also be omitted in cases where the product name is sufficient (e.g.,
    s3blob not awss3blob since S3 is well-known, gcsblob not
    gcpgcsblob since GCS already references Google).
  • Required if the product name is not unique across providers (e.g.,
    gcpkms and awskms).
• ``is the product/service name.
• <api> is the portable API name.
  • Include for local/test packages like (e.g., fileblob, mempubsub).
  • May be omitted when it makes the package name too long (e.g. awssnssqs
    is long enough, don't add pubsub).
  • Encouraged when it helps distinguish the package from the service's own
    package name (e.g., s3blob not s3).

Portable Type Constructors Copy Chapter

Portable type constructors are the functions defined in driver packages that end
users call to get an instance of the portable type. For example,
gcsblob.OpenBucket, which returns an instance of the *blob.Bucket portable
type backed by GCS.

• Portable type constructors should be top-level functions that return the
  portable type directly. Avoid helpers (e.g., a Client struct with a
  function that returns the portable type instead of it being top-level) and
  wrappers (e.g., a fooblob.Bucket type returned from fooblob.OpenBucket
  that wraps the portable type). Top level functions without wrappers are
  easier to use, especially when we're consistent about it.
• Order arguments that are less likely to change across multiple calls to the
  constructor before ones that are likely to change. For example, connection
  and authorization related arguments should go before names, so
  OpenBucket(ctx, client, "mybucket") instead of OpenBucket(ctx, "mybucket", client).
• All public constructors should take an Options struct (see next section).

Option Structs Copy Section

All public constructors should take an Options struct, even if it is currently
empty, to ensure that we can add arguments to the APIs in the future without
breaking backward compatibility.

• This includes driver constructors (e.g., gcsblob.OpenBucket) as well as
  API functions (e.g., blob.NewReader). When in doubt, if you think it's
  possible that we'll add arguments, add Options.
• The argument should be of type *Options, so that nil can be passed in
  the default case.
• Name the Options struct appropriately. Options is usually fine for
  portable type constructors since the package generally only exposes a
  constructor. Inside a driver interface or in a portable type like blob,
  use more descriptive names like ReaderOptions or WriterOptions.
• If a function already has a struct argument, don't add a separate Options
  struct. Example: the various sql.Open functions take a Params struct
  with connection parameters; we chose to add new options to Params instead
  of introducing a separate Options struct. This keeps the function
  signature simpler and avoid confusion about which struct new parameters
  should be added to.
• When similar Options are part of a driver interface and also part of the
  portable type (e.g., blob.WriterOptions), duplicate the struct instead of
  aliasing or embedding it, and copy the struct fields explicitly where
  needed. This allows the godoc for each type to be tailored to the
  appropriate audience (e.g. end-users for the portable type, driver
  implementors for the driver interface), and also allows the structs to
  diverge over time if appropriate.
• Required arguments must not be in an Options struct, and all fields of the
  Options struct must have reasonable defaults. Exception: struct arguments
  that don't have Options in the name can contain required arguments (e.g.,
  see the Params example for sql.Open above).

Regarding empty Options structs: we considered only adding them when the first
option is added, and using a separate constructor for compatibility (e.g., start
with foo.New(...) and later add foo.NewWithOptions(..., opts *Options) if
needed). However, this would result in inconsistent names over time (e.g., some
packages would expose New with an Options, while others would expose
NewWithOptions).

Compound IDs Copy Section

Many cloud providers have resource IDs that are made up of subcomponents in
some well-defined syntax. For example, GCP KMS key IDs take the form
projects/[PROJECT_ID]/locations/[LOCATION]/keyRings/[KEY_RING]/cryptoKeys/[KEY].
We call these compound IDs.

There are two broad compound ID usage patterns we have observed:

1. Applications will keep resources in the same location, so the application
   will build the ID from the subcomponents rather than passing the entire
   resource ID around.
2. Applications will pass a verbatim string from configuration down to the
   API, since this is what was easily copy-pasteable from the cloud console UI.

Go CDK constructors that take in compound IDs should take in a string with the
full compound ID. Helper functions to build these compound IDs from
subcomponents may be provided as needed. URL openers (described below) should
prefer to use the full compound ID in their URL format.

URLs Copy Section

To enable the Backing services factor of a Twelve-Factor Application, Go
Cloud includes the ability to construct each of its API objects using
identifying URLs. The portable type's package should include APIs like the
following:

// Package foo is a portable API. foo could be something like blob or pubsub.
//
// Throughout this example, Widget is used as a stand-in for a portable type
// inside foo, like Bucket or Subscription.
package foo

// A type that implements WidgetURLOpener can open widgets based on a URL.
// The opener must not modify the URL argument. OpenWidgetURL must be safe to
// call from multiple goroutines.
//
// WidgetURLOpeners should not assume that the URL has a particular scheme.
type WidgetURLOpener interface {
  OpenWidgetURL(ctx context.Context, u *url.URL) (*Widget, error)
}

// URLMux is a URL opener multiplexer. It matches the scheme of the URLs
// against a set of registered schemes and calls the opener that matches the
// URL's scheme.
//
// The zero value is a multiplexer with no registered schemes.
type URLMux struct {
  // ...
}

// RegisterWidget registers the opener with the given scheme. If an opener
// already exists for the scheme, RegisterWidget panics.
func (mux *URLMux) RegisterWidget(scheme string, opener WidgetURLOpener) {
  // ...
}

// OpenWidget calls OpenWidgetURL with the URL parsed from urlstr.
// OpenWidget is safe to call from multiple goroutines.
func (mux *URLMux) OpenWidget(ctx context.Context, urlstr string) (*Widget, error) {
  u, err := url.Parse(urlstr)
  if err != nil {
    return nil, fmt.Errorf("open widget: %v", err)
  }
  return mux.OpenWidgetURL(ctx, u)
}

// OpenWidgetURL dispatches the URL to the opener that is registered with the
// URL's scheme. OpenWidgetURL is safe to call from multiple goroutines.
func (mux *URLMux) OpenWidgetURL(ctx context.Context, u *url.URL) (*Widget, error) {
  // ...
}

// DefaultURLMux returns the URLMux used by OpenWidget.
func DefaultURLMux() *URLMux {
  return defaultURLMux
}

var defaultURLMux = new(URLMux)

// OpenWidget opens the Widget identified by the URL given. URL openers must be
// registered in the DefaultURLMux, which is typically done in driver
// packages' initialization.
func OpenWidget(ctx context.Context, urlstr string) (*Widget, error) {
  return DefaultURLMux().OpenWidget(urlstr)
}

The repetition of Widget in the method names permits a type to handle multiple
resources within the API. Exporting the URLMux allows applications to build
their own muxes, potentially wrapping existing ones.

Driver packages should include their own URLOpener struct type which
implements all the relevant WidgetURLOpener methods. The URL should only serve
to identify which resource to open. Any credentials or other complex values
should be taken in as struct fields, not as input from URL. If the driver
package registers its URLOpener with the DefaultURLMux, then it should
populate these complex fields from environment variables. If doing so is
undesirable or expensive, then it should not register with the DefaultURLMux
and instead rely on users to create their own mux. If there already exists a
well-established URI format for the backend (like S3 URLs or database connection
URIs), then drivers should honor them where possible.

URL Examples

A WidgetURLOpener implementation for a hypothetical GCP service:

package gcpfoo

// ...

const Scheme = "gcpwidget"

type URLOpener struct {
  Client  *gcp.HTTPClient
  Options Options
}

func (o *URLOpener) OpenWidgetURL(ctx context.Context, u *url.URL) (*foo.Widget, error) {
  // ...
  return OpenWidget(ctx, o.Client, u.Host, &o.Options)
}

type lazyURLOpener struct {
  init   sync.Once
  opener *URLOpener
  err    error
}

func (o *lazyURLOpener) OpenWidgetURL(ctx context.Context, u *url.URL) (*foo.Widget, error) {
  o.init.Once(func() {
    creds, err := gcp.DefaultCredentials(ctx)
    if err != nil {
      o.err = err
      return
    }
    o.opener = new(URLOpener)
    o.opener.Client, _ = gcp.NewHTTPClient(http.DefaultTransport, creds.TokenSource)
  })
  if o.err != nil {
    return nil, o.err
  }
  return o.opener.OpenWidgetURL(ctx, u)
}

func init() {
  foo.DefaultURLMux().Register(Scheme, new(lazyURLOpener))
}

// OpenWidget is the exported non-URL constructor.
func OpenWidget(ctx context.Context, c *gcp.HTTPClient, name string, opts *Options) (*foo.Widget, error) {
  // ...
}

Using the global default mux:

import _ "gocloud.dev/foo/gcpfoo"

// ...

widget, err := foo.OpenWidget(context.Background(), "gcpwidget://xyzzy")

Using a custom mux created during server initialization:

myMux := new(foo.URLMux)
myMux.Register(gcpfoo.Scheme, &gcpfoo.URLOpener{
  Client: client,
})
widget, err := myMux.OpenWidget(context.Background(), "gcpwidget://xyzzy")

Errors Copy Chapter

General Copy Section

• The callee is expected to return errors with messages that include
  information about the particular call, as opposed to the caller adding this
  information. This aligns with common Go practice.

Drivers Copy Section

Driver implementations should:

• Return the raw errors from the underlying service, and not wrap them in
  fmt.Errorf calls, so that they can be exposed to end users via ErrorAs.

Portable Types Copy Section

Portable types should:

• Wrap errors returned from driver implementations before returning them to
  end users, so that users can't peek into driver-specific error details
  without using As. Make sure not to double-wrap.

• Use internal/gcerr.New when wrapping driver errors, like so: if err := driver.Call(xyz); err != nil { return gcerr.New(code, err, 1, "blob") } The
  first argument is an error code. See below for advice on choosing the
  appropriate code.

  The third argument is the distance in stack frames from the function whose
  location should be associated with the error. It should be 1 if you are
  calling New from the same function that made the driver call, 2 if you
  are calling new from a helper function, and so on. The fourth argument is an
  additional string that will display with the error. You should pass the API
  name.

• By default, choose the code Unknown, keeping details of returned errors
  unspecified. The most common case is that the caller will only care whether
  an operation succeeds or not.

• If certain errors are interesting for callers to distinguish, choose one
  of the other codes from the gcerrors.ErrorCode enum, so user programs can
  act on the kind of error without having to look at driver-specific errors.

  • If more than one error code makes sense, choose the most specific one.
  • If none make sense, choose Unknown.
  • If none make sense but you want something more specific than Unknown:
    • If you can generalize your code to make it applicable to more than
      just your API, add it to gcerrors.ErrorCode. Look at the
      gRPC error codes
      for inspiration.
    • Otherwise, you can define a custom code in your portable API
      package. Your code should use a negative integer.

• For now, your package should expose an ErrorAs function to allow users to
  access driver-specific error types. We may review this choice if
  golang.org/x/xerrors.As becomes part of the standard library.

• Handle transient network errors. Retry logic is best handled as low in the
  stack as possible to avoid cascading failure. APIs should try to surface
  "permanent" errors (e.g. malformed request, bad permissions) where
  appropriate so that application logic does not attempt to retry
  non-idempotent operations, but the responsibility is largely on the library,
  not on the application.

Escape Hatches using As Copy Chapter

The Go CDK allows users to escape the abstraction as needed using As
functions, described in more detail in the
concept guide. As functions take an
interface{} and return a bool; they return true if the underlying concrete
type could be converted into the type provided as the interface{}.

An alternative approach would have been something like
os.ProcessState.Sys, which
returns an interface{} that the user can then type cast/assert to
service-specific types.

We ended up going with As because:

1. Most portable types have an As function for errors; choosing As results
   in an easy and natural implementation for chained errors once the
   Go 2 proposal for errors
   arrives. It is currently implemented in
   xerrors, and we're already
   using that in some drivers.
2. As adds more flexibility for drivers to support conversions to multiple
   types. Specifically, not the case where there are multiple possible
   underlying types, but rather that a single underlying type can be converted
   to multiple types.
   • Chained errors is one example of this, where the top-level error may
     always be the same type, but may also represent a chain of other errors
     with different types.
   • Another example is that a driver might choose to support As-level
     compatibility with another driver; e.g., driver foo could support all
     of the As types defined by s3blob, converting them internally, and
     then any code that runs with driver s3blob would also work with driver
     foo (even if it uses the As escape hatches).

Enforcing Portability Copy Chapter

The Go CDK APIs will end up exposing functionality that is not supported by all
services. In addition, some functionality details will differ across services.
Some theoretical examples using blob.Bucket:

1. Top-level APIs: There might be a service that supports reads, but not
   writes or deletes.
2. Data fields. Some services may support key/value metadata associated
   with a blob, others may not.
3. Naming rules. Different services may allow different name lengths, or
   allow/disallow non-ASCII unicode characters. See Strings below
   for more on handling string differences.
4. Semantic guarantees. Different services may have different consistency
   guarantees; for example, S3 only provides eventually consistency while GCS
   provides strong consistency.

How can we maintain portability while these differences exist?

Guiding Principle Copy Section

Any incompatibilities between drivers should be visible to the user as soon as
possible. From best to worst:

1. At compile time
2. At configuration/app startup time (e.g., when the portable type is created)
3. At runtime (e.g., when the incompatible behavior is accessed), via a non-nil
   error
4. At runtime, via panic

Approaches Considered Copy Section

1. Documentation. We could try to document non-uniform or optional
   functionality across drivers. Optional fields or functionality would
   return "not implemented" errors or zero values.
2. Restrict functionality to the intersection. We could explicitly only
   support the intersection of all services. For example, if not all services
   allow unicode characters in names, then blob would not allow it either.
3. Enforced feature codes: Go CDK APIs could enumerate the ways in which
   drivers differ as a FeatureCode enum.
   • Drivers would declare which feature codes they support, enforced by
     extensions to the existing conformance tests.
   • API users would declare which feature codes they need.
   • Mismatches between what a user requests and what the driver supports
     would be enforced at initialization time.
   • As much as possible, the API (via the portable type) would enforce that
     the user is only exposed to optional functionality that they asked for.
   • For example, the default legal name for a blob might be ASCII only, with
     a FeatureUnicodeNames feature code. Users that don't request this
     feature code would only be able to use blobs with ASCII names, even if
     the underlying service supports unicode. If the user requested
     FeatureUnicodeNames, and their driver supports it, they could then
     use blobs with unicode; if their driver doesn't support it, they would
     get an initialization-time error.

b, err := blob.NewBucket(d, blob.FeatureUnicodeNames)
...

Design discussions regarding enforcing portability are ongoing; we welcome input
on the mailing list.

Strings Copy Section

Services often differ on what they accept in particular strings (e.g., blob
names, metadata keys, etc.). A couple of specific examples:

• Azure Blob only
  accepts C# identifiers
  as metadata keys.
• S3 drops double slashes in blob names (e.g., foo//bar will end up being
  saved as foo/bar).

These differences lead to a loss of portability and predictability for users.

To resolve this issue, we insist that Go CDK can handle any UTF-8 string, and
force drivers to use escaping mechanisms to handle strings that the underlying
service can't handle. We enforce driver compliance with conformance tests.
Behavior for non-UTF-8 strings is undefined (but see
https://github.com/google/go-cloud/issues/1281 and
https://github.com/google/go-cloud/issues/1260).

We try to use URL encoding as the escaping mechanism where possible; however,
sometimes it is not and we'll use custom escaping. As an example, a driver for a
service that only allows underscores and ASCII alphanumeric characters might
escape the string foo.bar to foo__0x2e__bar (URL escaping won't work because
% isn't allowed).

Pros of this approach:

• Go CDK APIs are internally consistent in that a user can write any string to
  any service and get the original string back when they read it back.
• Go CDK APIs have visibility into all existing strings for all services.

Cons:

• Go CDK could overwrite existing data if a Go CDK-written key escapes to an
  already-existing value (e.g., if the foo__0x2e__bar string already
  existed, it would be overwritten by a Go CDK write to foo.bar).
• Escaping may push a string over the maximum allowed string length for a
  service. Escaping does not solve (and in fact may exacerbate) problems with
  different maximum string lengths across services.
• Existing strings that happen to look like Go CDK-escaped strings will be
  unescaped by Go CDK (e.g., an existing string foo__0x2e__bar would appear
  as foo.bar when read through the Go CDK).
• Strings that were written through the Go CDK and needed escaping will appear
  in their escaped form when viewed outside of Go CDK (e.g., foo__0x2e__bar
  would appear on the service's UI).

Most of these cons are mitigated by choosing unusual-looking escape mechanisms
that are unlikely to appear in existing data.

Drivers should escape strings when writing to the underlying service, and
unescape them when reading them back. The Go CDK will provide helpers for these
operations, as well as a test suite of strings for conformance tests.

Sample code for the helper for escaping strings:

// package escape provides helpers for escaping and unescaping strings.
package escape

// Escape returns s, with all runes for which shouldEscape returns true
// escaped to "__0xXXXX__", where XXXX is the hex representation of the rune
// value. For example, " " would escape to "__0x20__".
//
// Non-UTF-8 strings will have their non-UTF-8 characters escaped to
// unicode.ReplacementChar; the original value is lost. Please file an
// issue if you need non-UTF8 support.
//
// Note: shouldEscape takes the whole string as a slice of runes and an
// index. Passing it a single byte or a single rune doesn't provide
// enough context for some escape decisions; for example, the caller might
// want to escape the second "/" in "//" but not the first one.
// We pass a slice of runes instead of the string or a slice of bytes
// because some decisions will be made on a rune basis (e.g., encode
// all non-ASCII runes).
func Escape(s string, shouldEscape func(s []rune, i int) bool) string { ... }

// Unescape reverses Escape.
func Unescape(s string) string {...}

Sample code for how a driver might use it, using metadata keys for a blob as
the example string:

// When writing metadata keys, escape the keys:
// ... gcdkMetadata is the metadata passed to the GCDK API.
for k, v := range gcdkMetadata {
    e := escape.Escape(k, func (r []rune, i int) bool {...})
    if _, ok := serviceMetadata[e]; ok {
      return fmt.Errorf("duplicate keys after escaping: %q => %q", k, e)
    }
    serviceMetadata[e] = v
}
// ... write serviceMetadata to the service.

// When reading metadata keys, unescape them:
// ... serviceMetadata is the metadata read from the service.
for k, v := range serviceMetadata {
    gcdkMetadata[escape.Unescape(k)] = v
}
// ... return gcdkMetadata.

The details of what runes need to be escaped will vary from service to
service. The details of how to escape may also vary, although we expect to use
URL encoding where possible, and a common custom escaping where not. For the
custom escaping, we plan to escape each rune for which shouldEscape returns
true with __0xXXX__, where XCX is the hex representation of the rune value.

Alternatives Considered Copy Section

• We considered restricting Go CDK's APIs to strings that all services
  support. For example, we could have asserted that Go CDK's blob only
  supports ASCII plus / for blob names (and no //!). However, such a rule
  would mean that we couldn't cleanly handle existing strings created through
  some mechanism other than through Go CDK APIs that violate the rule. For
  example, an existing blob in S3 with a unicode name. Filtering out such
  strings so that they aren't visible at all through the Go CDK would be both
  surprising and limiting, and could easily result in data loss (e.g., if a
  user read a set of metadata for a blob via the Go CDK, and some keys were
  filtered out, and then wrote the metadata back, the filtered keys would be
  lost). Not filtering such strings would mean that the Go CDK isn't
  internally consistent (i.e., you can read some strings but not write them).
  Overall, we decided that this approach is unacceptable.

• We could expose the escaper used by drivers in their Options structs
  (including options like disabling it, overriding the set of bytes to be
  escaped, or overriding the escaping mechanism), but we'll wait to see if
  there's demand for that.

Coding Conventions Copy Chapter

We try to adhere to commonly accepted Go coding conventions, some of which are
described on the
Code Review Comments
wiki page. We also adopt the following guidelines:

• Prefer map[K]V{} to make(map[K]V). It's more concise.

• When writing a loop appending to a slice s, prefer

  text Copy

  var s []T
    for ... {
      ...
      s = append(s, ...)
      ...
    }

  to

  text Copy

  s := make([]T, 0, N)
    for ... {
      ...
      s = append(s, ...)
      ...
    }

  or

  text Copy

  s := make([]T, N)
    for ... {
      ...
      s[i] = ...
      ...
    }

  (Exception: the loop body is trivial and the loop is performance-sensitive.)
  The first version is shorter and easier to read, and it is impossible to get
  the length wrong.

• Prefer log.Fatal to panic in example tests.

• Ensure you've run goimports on your code to properly group import
  statements.

• Order arguments that are less likely to change across multiple calls to the
  constructor before ones that are likely to change. For example, connection
  and authorization related arguments should go before names, so
  OpenBucket(ctx, client, "mybucket") instead of OpenBucket(ctx, "mybucket", client).

Tests Copy Chapter

Conformance Tests Copy Section

Since our goal is for users to be able to use drivers interchangeably, it is
critical that they behave similarly. To this end, each portable API (e.g.,
blob) must provide a suite of conformance tests that driver implementations
should run. The conformance tests should be comprehensive; drivers should not
need additional unit tests for the core driver semantics.

Provisioning For Tests Copy Section

Portable API integration tests require developer-specific resources to be
created and destroyed. We use Terraform to do so, and
record the resource info and network interactions so that they can be replayed
as fast and repeatable unit tests.

Replay Mode Copy Section

Tests normally run in replay mode. In this mode, they don't require any
provisioned resources or network interactions. Replay tests verify that:

• The same test inputs produce the same requests (e.g., HTTP requests) to the
  cloud service. Some parts of the request may be dynamic (e.g., dates in the
  HTTP request headers), so the replay tests do some scrubbing when verifying
  that requests match. Some parts of this scrubbing are service-specific.

• The replayed service responses produce the expected results from the
  portable API library.

Record Mode Copy Section

In -record mode, tests run as integration tests, making live requests to
backend servers and recording the requests/responses for later use in replay
mode.

To use -record:

1. Provision resources.

   • For example, the tests for the AWS implementation of Blob requires a
     bucket to be provisioned.
   • TODO(issue #300): Use Terraform scripts to provision the resources
     needed for a given test.
   • For now, do this manually.

2. Run the test with -record.

   • TODO(issue #300): The test will read the Terraform output to find its
     inputs.
   • For now, pass the required resources via test-specific flags.
   • When changing or adding tests, please only record the tests that are
     changed/affected by passing the -run flag to go test with the
     name of the test(s). Re-recording all tests of a driver creates a lot
     of noise and a large diff that's difficult to review.

3. The test will save the network interactions for subsequent replays.

   • TODO(issue #300): The test will save the Terraform output to a file in
     order to replay using the same inputs.
   • Commit the new replay files along with your code change. Expect to see
     lots of diffs; see below for more discussion.

Diffs in replay files Copy Section

Each time portable API tests are run in -record mode, the resulting replay
files are different. Looking at diffs of these files isn't particularly useful.

We considered trying to scrub
the files of dynamic data so that diffs would be useful. We ended up deciding
not to do this, for several reasons:

• There's a lot of dynamic data, in structured data of various forms (e.g.,
  HTTP headers, XML/JSON body, etc.). It would be difficult and fragile to
  scrub it all.

• The scrub process would also be fragile relative to changes in services
  (e.g., adding a new dynamic HTTP response header).

• The scrub process would need to be implemented for every new service,
  increasing the barrier to entry for new implementations.

• Scrubbing would likely be even more difficult for services using a
  non-HTTP-based protocol (e.g., gRPC).

• Scrubbing the data decreases the fidelity of the replay test, since it
  wouldn't be operating on the original data.

Overall, massive diffs in the replay files are expected and fine. As part of a
code change, you may want to check for things like the number of RPCs made to
identify performance regressions.

File: internal/docs/release.md Copy Chapter

Releases

To do a release:

• Pick the new release name; it's probably v0.x.0 where x is whatever the
  last release was plus
  one, but follow semantic versioning.

• Consider updating dependencies via internal/testing/update_deps.sh if it
  hasn't been done recently. Do this as a separte step before the release.

• Create a new branch for the release (git checkout -B prerelease).

• Update the User-Agent version in internal/useragent/useragent.go.

• Run the release helper tool to remove replace lines from the go.mod
  files of submodules:

  bash Copy

  $ go run internal/releasehelper/releasehelper.go dropreplace

• Run the release helper tool to set the version in require directives of
  submodules to the new (yet unreleased) version:

  bash Copy

  $ go run internal/releasehelper/releasehelper.go setversion v0.x.0

• Commit and create a PR. Tests will fail for this PR because submodules depend on a
  version of the main module that wasn't tagged yet, so you may have to
  force-merge the PR. Note that this does not affect users, since a new
  version hasn't been tagged yet.

• git sync your local client and move to the master branch.

• Tag new versions by running ./internal/testing/git_tag_modules.sh v0.X.0.

• Push tags to upstream with git push upstream --tags

• Go to Releases. Click Draft a new release, enter your release name, select your tag from the dropdown,
  and enter release notes by clicking "Generate Release Notes".

  • Add a section for breaking changes, if any. They should be marked with
    BREAKING_CHANGE in the PR title; however, that's not enforced so do
    your best to look for them.
  • Update the list of changes to remove anything that's not interesting
    (e.g., updating dependencies, prerelease, minor cleanups, etc.).

• Send an email to
  [email protected]
  announcing the release, and including the release notes.

• Create a new branch for the postrelease (git checkout -B postrelease).

• Add back replace lines:

  bash Copy

  $ go run internal/releasehelper/releasehelper.go addreplace

  Run tests and send out a PR as usual.

File: internal/docs/pubsub/design.md Copy Chapter

Go CDK `pubsub` Design

Summary Copy Chapter

This document proposes a new pubsub package for the Go CDK.

Motivation Copy Chapter

A developer designing a new system with cross-cloud portability in mind could
choose a messaging system supporting pubsub, such as ZeroMQ, Kafka or RabbitMQ.
These pubsub systems run on AWS, Azure, GCP and others, so they pose no obstacle
to portability between clouds. They can also be run on-prem. Users wanting
managed pubsub could go with Confluent Cloud for Kafka (AWS, GCP), or CloudAMQP
for RabbitMQ (AWS, Azure) without losing much in the way of portability.

So what’s missing? The solution described above means being locked into a
particular implementation of pubsub. There is also a potential for lock-in when
building systems in terms of the cloud-specific services such as AWS SNS+SQS,
GCP PubSub or Azure Service Bus.

Developers may wish to compare different pubsub systems in terms of their
performance, reliability, cost or other factors, and they may want the option to
move between these systems without too much friction. A pubsub package in the
Go CDK could lower the cost of such experiments and migrations.

Goals Copy Chapter

• Publish messages to an existing topic.
• Receive messages from an existing subscription.
• Perform not much worse than 90% compared to directly using the APIs of
  various pubsub systems.
• Work well with managed pubsub services on AWS, Azure, GCP and the most used
  open source pubsub systems.

Non-goals Copy Chapter

• Create new topics in the cloud. The Go CDK focuses on developer concerns,
  but topic creation is an
  operator concern.

• Create new subscriptions in the cloud. The subscribers are assumed to
  correspond to components of a distributed system rather than to users of
  that system.

Background Copy Chapter

Pubsub is a
frequently requested feature for the Go CDK project
[github issue]. A key use case
motivating these requests is to support
event driven architectures.

There are several pubsub systems available that could be made to work with the
Go CDK by writing drivers for them. Here is a
table
comparing some of them.

Design overview Copy Chapter

Developer’s perspective Copy Section

Given a topic that has already been created on the pubsub server, messages can
be sent to that topic by calling acmepubsub.OpenTopic and calling the Send
method of the returned Topic, like this (assuming a fictional pubsub service
called "acme"):

package main

import (
    "context"
    "log"
    "net/http"

    rawacmepubsub "github.com/acme/pubsub"
    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/acmepubsub"
)

func main() {
    log.Fatal(serve())
}

func serve() error {
    ctx := context.Background()
    client, err := rawacmepubsub.NewClient(ctx, "unicornvideohub")
    if err != nil {
        return err
    }
    t, err := acmepubsub.OpenTopic(ctx, client, "user-signup", nil)
    if err != nil {
        return err
    }
    defer t.Close()
    http.HandleFunc("/signup", func(w http.ResponseWriter, r *http.Request) {
        err := t.Send(r.Context(), pubsub.Message{Body: []byte("Someone signed up")})
        if err != nil {
            log.Println(err)
        }
    })
    return http.ListenAndServe(":8080", nil)
}

The call to Send will only return after the message has been sent to the
server or its sending has failed.

Messages can be received from an existing subscription to a topic by calling the
Receive method on a Subscription object returned from
acmepubsub.OpenSubscription, like this:

package main

import (
    "context"
    "fmt"
    "log"

    rawacmepubsub "github.com/acme/pubsub"
    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/acmepubsub"
)

func main() {
    if err := receive(); err != nil {
        log.Fatal(err)
    }
}

func receive() error {
    ctx := context.Background()
    client, err := rawacmepubsub.NewClient(ctx, "unicornvideohub")
    if err != nil {
        return err
    }
    s, err := acmepubsub.OpenSubscription(ctx, client, "user-signup-minder", nil)
    if err != nil {
        return err
    }
    defer s.Close()
    msg, err := s.Receive(ctx)
    if err != nil {
        return err
    }
    // Do something with msg.
    fmt.Printf("Got message: %s\n", msg.Body)
    // Acknowledge that we handled the message.
    msg.Ack()
}

A more realistic subscriber client would process messages in a loop, like this:

package main

import (
    "context"
    "log"
    "os"
    "os/signal"

    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/acmepubsub"
)

func main() {
    if err := receive(); err != nil {
        log.Fatal(err)
    }
}

func receive() error {
    ctx := context.Background()
    client, err := rawacmepubsub.NewClient(ctx, "unicornvideohub")
    if err != nil {
        return err
    }
    s, err := acmepubsub.OpenSubscription(ctx, client, "signup-minder", nil)
    if err != nil {
        return err
    }
    defer s.Close()

    // Process messages.
    for {
        msg, err := s.Receive(ctx)
        if err {
            return err
        }
        log.Printf("Got message: %s\n", msg.Body)
        msg.Ack()
    }
}

The messages can be processed concurrently with an
inverted worker pool,
like this:

package main

import (
    "context"
    "log"
    "os"
    "os/signal"

    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/acmepubsub"
)

func main() {
    if err := receive(); err != nil {
        log.Fatal(err)
    }
}

func receive() error {
    ctx := context.Background()
    client, err := rawacmepubsub.NewClient(ctx, "unicornvideohub")
    if err != nil {
        return err
    }
    s, err := acmepubsub.OpenSubscription(ctx, client, "user-signup-minder", nil)
    if err != nil {
        return err
    }
    defer s.Close()

    // Process messages.
    const poolSize = 10
    // Use a buffered channel as a semaphore.
    sem := make(chan struct{}, poolSize)
    for {
        msg, err := s.Receive(ctx)
        if err {
            return err
        }
        sem <- struct{}{}
        go func() {
            log.Printf("Got message: %s", msg.Body)
            msg.Ack()
            <-sem
        }()
    }
    for n := poolSize; n > 0; n-- {
        sem <- struct{}{}
    }
}

Driver implementer’s perspective Copy Section

Adding support for a new pubsub system involves the following steps, continuing
with the "acme" example:

1. Add a new package called acmepubsub.
2. Add private topic and subscription types to acmepubsub implementing
   the corresponding interfaces in the github.com/go-cloud/pubsub/driver
   package.
3. Add func OpenTopic(...) that creates an acmepubsub.topic and returns a
   concrete pubsub.Topic object made from it.
4. Add func OpenSubscription(...) that creates an acmepubsub.subscription
   and returns a pubsub.Subscription object made from it.

Here is a sketch of what the acmepubsub package could look like:

package acmepubsub

import (
    "context"

    rawacmepubsub "github.com/acme/pubsub"
    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/driver"
)

// OpenTopic opens an existing topic on the pubsub server and returns a Topic
// that can be used to send messages to that topic.
func OpenTopic(ctx context.Context, client *rawacmepubsub.Client, topicName string) (*pubsub.Topic, error) {
    rt, err := client.Topic(ctx, topicName)
    if err != nil {
        return nil, err
    }
    rt, err := client.Topic(ctx, topicName)
    if err != nil {
        return err
    }
    t := &topic{ rawTopic: rt }
    return pubsub.NewTopic(t)
}

// OpenSubscription opens an existing subscription on the server and returns a
// Subscription that can be used to receive messages.
func OpenSubscription(ctx context.Context, client *rawacmepubsub.Client, subscriptionName string) (*pubsub.Subscription, error) {
    rs, err := client.Subscription(ctx, subscriptionName)
    if err != nil {
        return err
    }
    s := &subscription{ rawSub: rs }
    return pubsub.NewSubscription(s)
}

type topic struct {
    rawTopic    *rawacmepubsub.Topic
}

func (t *topic) SendBatch(ctx context.Context, []*pubsub.Message) error {
    // ...
}

func (t *topic) Close() error {
    // ...
}

type subscription struct {
    rawSub  *rawacmepubsub.Subscription
}

func (s *subscription) ReceiveBatch(ctx context.Context) ([]*pubsub.Message, error) {
    // ...
}

func (s *subscription) SendAcks(ctx context.Context, []pubsub.AckID) error {
    // ...
}

func (s *subscription) Close() error {
    // ...
}

The driver interfaces are batch-oriented because some pubsub systems can more
efficiently deal with batches of messages than with one at a time. Streaming was
considered but it does not appear to provide enough of a performance gain to be
worth the additional complexity of supporting it across different pubsub systems
[benchmarks].

The driver interfaces will be located in the
github.com/google/go-cloud/pubsub/driver package and will look something like
this:

package driver

type AckID interface{}

type Message struct {
    // Body contains the content of the message.
    Body []byte

    // Attributes has key/value metadata for the message.
    Attributes map[string]string

    // AckID identifies the message on the server.
    // It can be used to ack the message after it has been received.
    AckID AckID
}

// Topic publishes messages.
type Topic interface {
    // SendBatch publishes all the messages in ms.
    SendBatch(ctx context.Context, ms []*Message) error

    // Close disconnects the Topic.
    Close() error
}

// Subscription receives published messages.
type Subscription interface {
        // ReceiveBatch should return a batch of messages that have queued up
        // for the subscription on the server.
        //
        // If there is a transient failure, this method should not retry but
        // should return a nil slice and an error. The concrete API will take
        // care of retry logic.
        //
        // If the service returns no messages for some other reason, this
        // method should return the empty slice of messages and not attempt to
        // retry.
        //
        // ReceiveBatch is only called sequentially for individual
        // Subscriptions.
    ReceiveBatch(ctx context.Context) ([]*Message, error)

    // SendAcks acknowledges the messages with the given ackIDs on the
    // server so that they
    // will not be received again for this subscription. This method
    // returns only after all the ackIDs are sent.
    SendAcks(ctx context.Context, ackIDs []interface{}) error

    // Close disconnects the Subscription.
    Close() error
}

Detailed design Copy Chapter

The developer experience of using Go CDK's pubsub involves sending, receiving
and acknowledging one message at a time, all in terms of synchronous calls.
Behind the scenes, the driver implementations deal with batches of messages and
acks. The concrete API, to be written by the Go CDK team, takes care of creating
the batches in the case of Send or Ack, and dealing out messages one at a time
in the case of Receive.

The concrete API will be located at github.com/google/go-cloud/pubsub and will
look something like this:

package pubsub

import (
    "context"
    "github.com/google/go-cloud/pubsub/driver"
)

// Message contains data to be published.
type Message struct {
    // Body contains the content of the message.
    Body []byte

    // Attributes contains key/value pairs with metadata about the message.
    Attributes map[string]string

    // ackID is an ID for the message on the server, used for acking.
    ackID AckID

    // sub is the Subscription this message was received from.
    sub *Subscription

    // isAcked is true if Ack has been called on this message.
    isAcked bool
}

type AckID interface{}

// Ack acknowledges the message, telling the server that it does not need to
// be sent again to the associated Subscription. This method returns
// immediately. If Ack has already been called on the message, Ack panics.
func (m *Message) Ack() {
    // Send the ack ID back to the subscriber for batching.
        // The ack is sent to the server in a separate goroutine
        // managed by the Subscription from which this message was
        // received.
        // ...
}

// Topic publishes messages to all its subscribers.
type Topic struct {
    driver   driver.Topic
    mcChan   chan msgCtx
    doneChan chan struct{}
}

// msgCtx pairs a Message with the Context of its Send call.
type msgCtx struct {
    msg *Message
    ctx context.Context
}

// Send publishes a message. It only returns after the message has been
// sent, or failed to be sent. The call will fail if ctx is canceled.
// Send can be called from multiple goroutines at once.
func (t *Topic) Send(ctx context.Context, m *Message) error {
        // Send this message over t.mcChan and then wait for the batch including
        // this message to be sent to the server.
        // ...
}

// Close disconnects the Topic.
func (t *Topic) Close() error {
    close(t.doneChan)
    return t.driver.Close()
}

// NewTopic makes a pubsub.Topic from a driver.Topic.
func NewTopic(d driver.Topic) *Topic {
    t := &Topic{
        driver:   d,
        mcChan:   make(chan msgCtx),
        doneChan: make(chan struct{}),
    }
    go func() {
        // Pull messages from t.mcChan and put them in batches. Send the current
        // batch whenever it is large enough or enough time has elapsed since
        // the last send.
        // ...
    }()
    return t
}

// Subscription receives published messages.
type Subscription struct {
    driver driver.Subscription

    // ackChan conveys ackIDs from Message.Ack to the ack batcher goroutine.
    ackChan chan AckID

    // ackErrChan reports errors back to Message.Ack.
    ackErrChan chan error

    // doneChan tells the goroutine from startAckBatcher to finish.
    doneChan chan struct{}

    // q is the local queue of messages downloaded from the server.
    q []*Message
}

// Receive receives and returns the next message from the Subscription's queue,
// blocking if none are available. This method can be called concurrently from
// multiple goroutines. On systems that support acks, the Ack() method of the
// returned Message has to be called once the message has been processed, to
// prevent it from being received again.
func (s *Subscription) Receive(ctx context.Context) (*Message, error) {
    if len(s.q) == 0 {
        // Get the next batch of messages from the server.
        // ...
    }
    m := s.q[0]
    s.q = s.q[1:]
    return m, nil
}

// Close disconnects the Subscription.
func (s *Subscription) Close() error {
    close(s.doneChan)
    return s.driver.Close()
}

// NewSubscription creates a Subscription from a driver.Subscription and opts to
// tune sending and receiving of acks and messages. Behind the scenes,
// NewSubscription spins up a goroutine to gather acks into batches and
// periodically send them to the server.
func NewSubscription(s driver.Subscription) *Subscription {
    // Details similar to the body of NewTopic should go here.
}

Topics will gather messages into batches for sending. The batch size will be
dynamically tuned according to how many messages are being sent concurrently.

Subscriptions will gather message acks into batches the same way, also
dynamically tuning the batch size. If sending acks back to the server fails
transiently then it will be retried, most likely within a loop in the concrete
API. If an unrecoverable error occurs while sending acks then a flag will be set
on the pubsub.Subscription saying that the whole Subscription is no longer
usable. Calls to Receive will fail from then on.

Alternative designs considered Copy Chapter

Batch oriented concrete API Copy Section

In this alternative, the application code sends, receives and acknowledges
messages in batches. Here is an example of how it would look from the
developer's perspective, in a situation where not too many signups are happening
per second.

package main

import (
    "context"
    "log"
    "net/http"

    rawacmepubsub "github.com/acme/pubsub"
    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/acmepubsub"
)

func main() {
    log.Fatal(serve())
}

func serve() error {
    ctx := context.Background()
    client, err := rawacmepubsub.NewClient(ctx, "unicornvideohub")
    if err != nil {
        return err
    }
    t, err := acmepubsub.OpenTopic(ctx, client, "user-signup", nil)
    if err != nil {
        return err
    }
    defer t.Close()
    http.HandleFunc("/signup", func(w http.ResponseWriter, r *http.Request) {
        err := t.Send(r.Context(), []pubsub.Message{{Body: []byte("Someone signed up")}})
        if err != nil {
            log.Println(err)
        }
    })
    return http.ListenAndServe(":8080", nil)
}

For a company experiencing explosive growth or enthusiastic spammers creating
more signups than this simple-minded implementation can handle, the app would
have to be adapted to create non-singleton batches, like this:

package main

import (
    "context"
    "log"
    "net/http"

    rawacmepubsub "github.com/acme/pubsub"
    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/acmepubsub"
)

const batchSize = 1000

func main() {
    log.Fatal(serve())
}

func serve() error {
    ctx := context.Background()
    client, err := rawacmepubsub.NewClient(ctx, "unicornvideohub")
    if err != nil {
        return err
    }
    t, err := acmepubsub.OpenTopic(ctx, client, "user-signup", nil)
    if err != nil {
        return err
    }
    defer t.Close()
    c := make(chan *pubsub.Message)
    go sendBatches(ctx, t, c)
    http.HandleFunc("/signup", func(w http.ResponseWriter, r *http.Request) {
        c <- &pubsub.Message{Body: []byte("Someone signed up")}
    })
    return http.ListenAndServe(":8080", nil)
}

func sendBatches(ctx context.Context, t *pubsub.Topic, c chan *pubsub.Message) {
    batch := make([]*pubsub.Message, batchSize)
    for {
        for i := 0; i < batchSize; i++ {
            batch[i] = <-c
        }
        if err := t.Send(ctx, batch); err != nil {
            log.Println(err)
        }
    }
}

This shows how the complexity of batching has been pushed onto the application
code. Removing messages from the batch when HTTP/2 requests are canceled would
require the application code to be even more complex, adding more risk of bugs.

In this API, the application code has to either request batches of size 1,
meaning more network traffic, or it has to explicitly manage the batches of
messages it receives. Here is an example of how this API would be used for
serial message processing:

package main

import (
    "context"
    "log"
    "os"
    "os/signal"

    rawacmepubsub "github.com/acme/pubsub"
    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/acmepubsub"
)

const batchSize = 10

func main() {
    if err := receive(); err != nil {
        log.Fatal(err)
    }
}

func receive() error {
    ctx := context.Background()
    client, err := rawacmepubsub.NewClient(ctx, "unicornvideohub")
    if err != nil {
        return err
    }
    s, err := acmepubsub.OpenSubscription(ctx, client, "signup-minder", nil)
    if err != nil {
        return err
    }
    defer s.Close()

    // Process messages.
    for {
        msgs, err := s.Receive(ctx, batchSize)
        if err {
            return err
        }
        acks := make([]pubsub.AckID, 0, batchSize)
        for _, msg := range msgs {
            // Do something with msg.
            fmt.Printf("Got message: %q\n", msg.Body)
            acks = append(acks, msg.AckID)
        }
        err := s.SendAcks(ctx, acks)
        if err != nil {
            return err
        }
    }
}

Here’s what it might look like to use this batch-only API with the inverted
worker pool pattern:

package main

import (
    "context"
    "log"
    "os"
    "os/signal"

    rawacmepubsub "github.com/acme/pubsub"
    "github.com/google/go-cloud/pubsub"
    "github.com/google/go-cloud/pubsub/acmepubsub"
)

const batchSize = 100
const poolSize = 10

func main() {
    if err := receive(); err != nil {
        log.Fatal(err)
    }
}

func receive() error {
    ctx := context.Background()
    client, err := rawacmepubsub.NewClient(ctx, "unicornvideohub")
    if err != nil {
        return err
    }
    s, err := acmepubsub.OpenSubscription(ctx, client, "user-signup-minder", nil)
    if err != nil {
        return err
    }
    defer s.Close()

    // Receive the messages and forward them to a chan.
    msgsChan := make(chan *pubsub.Message)
    go func() {
        for {
            msgs, err := s.Receive(ctx, batchSize)
            if err {
                log.Fatal(err)
            }
            for _, m := range msgs {
                msgsChan <- m
            }
        }
    }

    // Get the acks from a chan and send them back to the
    // server in batches.
    acksChan := make(chan pubsub.AckID)
    go func() {
        for {
            batch := make([]pubsub.AckID, batchSize)
            for i := 0; i < len(batch); i++ {
                batch[i] = <-acksChan
            }
            if err := s.SendAcks(ctx, batch); err != nil {
                /* handle err */
            }
        }
    }

    // Use a buffered channel as a semaphore.
    sem := make(chan struct{}, poolSize)
    for msg := range msgsChan {
        sem <- struct{}{}
        go func(msg *pubsub.Message) {
            log.Printf("Got message: %s", msg.Body)
            acksChan <- msg.AckID
            <-sem
        }(msg)
    }
    for n := poolSize; n > 0; n-- {
        sem <- struct{}{}
    }
}

Here are some trade-offs of this design:

Pro:

• The semantics are simple, making it
  • straightforward to implement the concrete API and the drivers for most
    pubsub services
  • easy for developers to reason about how it will behave
  • less risky that bugs will be present in the concrete API
• Fairly efficient sending and receiving of messages is possible by tuning
  batch size and the number of goroutines sending or receiving messages.

Con:

• This style of API makes the inverted worker pool pattern verbose.
• Apps needing to send or receive a large volume of messages must have their
  own logic to create batches of size greater than 1.

go-micro Copy Section

Here is an example of what application code could look like for a pubsub API
inspired by go-micro's broker package:

b := somepubsub.NewBroker(...)
if err := b.Connect(); err != nil {
    /* handle err */
}
topic := "user-signups"
subID := "user-signups-subscription-1"
s, err := b.Subscription(ctx, topic, subID, func(pub broker.Publication) error {
    fmt.Printf("%s\n", pub.Message.Body)
    return nil
})
if err := b.Publish(ctx, topic, &broker.Message{ Body: []byte("alice signed up") }); err != nil {
    /* handle err */
}
// Sometime later:
if err := s.Unsubscribe(ctx); err != nil {
    /* handle err */
}

Pro:

• The callback to the subscription returning an error to decide whether to
  acknowledge the message means the developer cannot forget to ack.

Con:

• Go micro has code to auto-create
  topics
  and
  subscriptions
  as needed, but this is not consistent with the Go CDK’s design principle to
  not get involved in operations.
• The subscription callback idea does not appear to be compatible with
  inverted worker pools.

Acknowledgements Copy Chapter

In pubsub systems with acknowledgement, messages are kept in a queue associated
with the subscription on the server. When a client receives one of these
messages, its counterpart on the server is marked as being processed. Once the
client finishes processing the message, it sends an acknowledgement (or "ack")
to the server and the server removes the message from the subscription queue.
There may be a deadline for the acknowledgement, past which the server unmarks
the message so that it can be received again for another try at processing.

Redis Pub/Sub and ZeroMQ don’t support acking, but many others do including GCP
PubSub, Azure Service Bus, RabbitMQ, and
Redis Streams. Given the wide support
and usefulness, it makes sense to support message acking in the Go CDK.

As of this writing, it is an open question as to what should be done about
pubsub systems that do not support acks. Some possibilities have been discussed,
but no clear best option has emerged yet:

1. simulating acknowledgement by constructing queues on the server. Con: the
   magically created queues would probably be a less than pleasant surprise for
   some users.
2. making ack a no-op for systems that don't support it. With this, do we
   return a sentinel error from Ack, and if so then doesn't that unduly
   complicate the code for apps that never use non-acking systems? This option
   is also potentially misleading for developers who would naturally assume
   that un-acked messages would be redelivered.

Rejected acknowledgement API: `Receive` method returns an `ack` func Copy Section

In this alternative, the application code would look something like this:

msg, ack, err := s.Receive(ctx)
log.Printf("Received message: %q", msg.Body)
ack(msg)

Pro:

• The compiler will complain if the returned ack function is not used.

Con:

• Receive has one more return value.
• Passing ack around along with msg is inconvenient.

Tests Copy Chapter

Unit tests for the concrete API (`github.com/go-cloud/pubsub`) Copy Section

We can test that the batched sending, receiving and acking work as intended by
making mock implementations of the driver interfaces.

At least the following things should be tested:

• Calling pubsub.Message.Ack causes driver.Subscription.SendAcks to be
  called.
• Calling pubsub.Topic.Send causes driver.Topic.SendBatch to be called.
• Calling pubsub.Subscription.Receive causes
  driver.Subscription.ReceiveBatch to be called.

Conformance tests for specific implementations (*e.g.*, `github.com/go-cloud/pubsub/acmepubsub`) Copy Section

• Sent messages with random contents are received with the same contents.
• Sent messages with random attributes are received with the same attributes.
• Error occurs when making a local topic with an ID that doesn’t exist on the
  server.
• Error occurs when making a subscription with an ID that doesn’t exist on the
  server.
• Message gets sent again after ack deadline if a message is never
  acknowledged.
• Acked messages don't get received again after waiting twice the ack
  deadline. :point_left: This test would probably be too flakey.

Benchmarks Copy Chapter

What is the throughput and latency of the Go CDK's pubsub package, relative to
directly using the APIs for various services?

• send, for 1, 10, 100 topics, and for 1, 10, 100 goroutines sending messages
  to those topics
• receive, for 1, 10, 100 subscriptions, and for 1, 10, 100 goroutines
  receiving from each subscription

References Copy Chapter

• https://github.com/google/go-cloud/issues/312
• http://queues.io/

File: internal/website/README.md Copy Chapter

gocloud.dev source

Source for the gocloud.dev website. Powered by Hugo.

Local Development Copy Chapter

Use local hugo server for preview. cd into this directory and run:

$ hugo server -D

This will run the Hugo server that auto-updates its output based on the source
contents. It will print out the localhost: URL to point the browser to.

This was tested with Hugo 0.53 but should work with subsequent versions as well.

Editing Copy Chapter

Use hugo new foo/page.md to create content/foo/page.md. This will
automatically add the appropriate Front Matter to the site. After modifying
an existing page, add the lastmod attribute with the current ISO date, which
you can obtain with date -I. For example:

---
title: Foo
date: "2019-03-17T09:00:00-07:00"
lastmod: "2019-03-18T13:30:12-07:00"
---

...

File: internal/website/archetypes/default.md Copy Chapter

title: "{{ replace .Name "-" " " | title }}" date: {{ .Date }} draft: true showInSidenav: false # only for sections (any level) pagesInSidenav: false # only for top-level sections weight: 0 Copy Chapter

File: internal/website/archetypes/howto.md Copy Chapter

title: "{{ replace .Name "-" " " | title }}" date: {{ .Date }} draft: true showInSidenav: false # only for sections (any level) weight: 0 Copy Chapter

File: internal/website/content/_index.md Copy Chapter

title: "Go CDK" pkgmeta: true Copy Chapter

The Go Cloud Development Kit

The Go Cloud Development Kit (Go CDK) is an open source project building
libraries and tools to improve the experience of developing for the cloud with
Go.

Go CDK provides commonly used, vendor-neutral generic APIs that you can deploy
across cloud providers. The idea is to support hybrid cloud deployments while
combining on-prem (local) and cloud tools.

This project also lays the foundation for other open source projects to write
cloud libraries that work across providers. It does this by providing stable,
idiomatic interfaces for use cases like storage, events and databases.

For more background about the project, check out the
announcement blog post and
our talk from Cloud Next 2018.

If you're interested in contributing to the Go CDK or are interested in checking
out the code, head to our GitHub project
page.

Installing and getting started Copy Chapter

To start using the Go CDK, install it using go get:

go get gocloud.dev

Then follow the [Go CDK tutorial][]. Links to additional documentation and
samples are available below and in the site navigation bar.

[Go CDK tutorial]: {{< ref "/tutorials/cli-uploader.md" >}}

Portable Cloud APIs in Go Copy Chapter

At this time, the Go CDK focuses on a set of portable APIs for cloud
programming. We strive to implement these APIs for the leading Cloud providers:
AWS, GCP and Azure, as well as provide a local (on-prem) implementation.

Using the Go CDK you can write your application code once using these idiomatic
APIs, test locally using the local versions, and then deploy to a cloud provider
with only minimal setup-time changes.

Project status Copy Chapter

We're looking for early adopters to help us validate the APIs before releasing
a beta version. Please try it and provide feedback by filing a
GitHub issue.

Legal disclaimer Copy Chapter

The Go CDK is open-source and released under an Apache 2.0
License. Copyright ©
2018–2019 The Go Cloud Development Kit Authors.

If you are looking for the website of GoCloud Systems, which is unrelated to the
Go CDK, visit https://gocloud.systems.

File: internal/website/content/aws/_index.md Copy Chapter

title: gocloud.dev/aws type: pkg Copy Chapter

File: internal/website/content/aws/awscloud/_index.md Copy Chapter

title: gocloud.dev/aws/awscloud type: pkg Copy Chapter

File: internal/website/content/aws/rds/_index.md Copy Chapter

title: gocloud.dev/aws/rds type: pkg Copy Chapter

File: internal/website/content/azure/azurecloud/_index.md Copy Chapter

title: gocloud.dev/azure/azurecloud type: pkg Copy Chapter

File: internal/website/content/azure/azuredb/_index.md Copy Chapter

title: gocloud.dev/azure/azuredb type: pkg Copy Chapter

File: internal/website/content/blob/_index.md Copy Chapter

title: gocloud.dev/blob type: pkg Copy Chapter

File: internal/website/content/blob/azureblob/_index.md Copy Chapter

title: gocloud.dev/blob/azureblob type: pkg Copy Chapter

File: internal/website/content/blob/driver/_index.md Copy Chapter

title: gocloud.dev/blob/driver type: pkg Copy Chapter

File: internal/website/content/blob/drivertest/_index.md Copy Chapter

title: gocloud.dev/blob/drivertest type: pkg Copy Chapter

File: internal/website/content/blob/fileblob/_index.md Copy Chapter

title: gocloud.dev/blob/fileblob type: pkg Copy Chapter

File: internal/website/content/blob/gcsblob/_index.md Copy Chapter

title: gocloud.dev/blob/gcsblob type: pkg Copy Chapter

File: internal/website/content/blob/memblob/_index.md Copy Chapter

title: gocloud.dev/blob/memblob type: pkg Copy Chapter

File: internal/website/content/blob/s3blob/_index.md Copy Chapter

title: gocloud.dev/blob/s3blob type: pkg Copy Chapter

File: internal/website/content/concepts/_index.md Copy Chapter

title: "Concepts" date: 2019-05-06T09:52:00-07:00 showInSidenav: true pagesInSidenav: true weight: 4 Copy Chapter

The documents in this section describe higher level concepts in the Go CDK.

File: internal/website/content/concepts/as.md Copy Chapter

title: Using provider-specific APIs date: 2019-05-10T11:17:09-07:00 weight: 3 Copy Chapter

It is not feasible or desirable for APIs like blob.Bucket to encompass the
full functionality of every provider. Rather, we intend to provide a subset
of the most commonly used functionality. There will be cases where a
developer wants to access provider-specific functionality, such as unexposed
APIs or data fields, errors, or options. This can be accomplished using As
functions.

`As` {#as} Copy Chapter

As functions in the APIs provide the user a way to escape the Go CDK
abstraction to access provider-specific types. They might be used as an
interim solution until a feature request to the Go CDK is implemented. Or,
the Go CDK may choose not to support specific features, and the use of As
will be permanent.

Using As implies that the resulting code is no longer portable; the
provider-specific code will need to be ported in order to switch providers.
Therefore, it should be avoided if possible.

Each API includes examples demonstrating how to use its various As
functions, and each provider implementation documents what types it supports
for each.

Usage:

1. Declare a variable of the provider-specific type you want to access.
2. Pass a pointer to it to As.
3. If the type is supported, As will return true and copy the
   provider-specific type into your variable. Otherwise, it will return false.

Provider-specific types that are intended to be mutable will be exposed
as a pointer to the underlying type.

File: internal/website/content/concepts/urls.md Copy Chapter

title: "URLs" date: 2019-05-06T09:55:09-07:00 weight: 2 Copy Chapter

In addition to creating portable types via provider-specific constructors
(e.g., creating a *blob.Bucket using s3blob.OpenBucket), many portable types
can also be created using a URL. The scheme of the URL specifies the provider,
and each provider implementation has code to convert the URL into the data
needed to call its constructor. For example, calling
blob.OpenBucket("s3blob://my-bucket") will return a *blob.Bucket created
using s3blob.OpenBucket.

Each portable API package will document the types that it supports opening
by URL. For example, the blob package supports Buckets, while the pubsub
package supports Topics and Subscriptions. Each provider implementation will
document what scheme(s) it registers for, and what format of URL it expects.

Each portable type URL opener will accept URL schemes with an <api>+ prefix
(e.g. blob+file:///dir instead of file:///dir, as well as schemes with an
<api>+<type>+ prefix (e.g. blob+bucket+file:///dir).

Each portable API package should include an example using a URL, and
many providers will include provider-specific examples as well.

Muxes Copy Chapter

Each portable type that is openable via URL will have a top-level function
you can call, like blob.OpenBucket. This top-level function uses a default
instance of a URLMux multiplexer to map schemes to a provider-specific
opener for the type. For example, blob has a BucketURLOpener interface
that providers implement and then register using RegisterBucket on the
result of DefaultURLMux.

Many applications will work just fine using the default mux through the
top-level Open functions. However, if you want more control, you can create
your own URLMux and register the provider URLOpeners you need. Most
providers will export URLOpeners that give you more fine grained control over
the arguments needed by the constructor. In particular, portable types opened
via URL will often use default credentials from the environment. For example,
the AWS URL openers use the credentials saved by "aws login" (we don't want
to include credentials in the URL itself, since they are likely to be
sensitive).

1. Instantiate the provider's URLOpener with the specific fields you need.
   For example, s3blob.URLOpener{ConfigProvider: myAWSProvider} using a
   ConfigProvider that holds explicit AWS credentials.
2. Create your own instance of the URLMux. For example: mymux := new(blob.URLMux)
3. Register your custom URLOpener on your mux. For example:
   mymux.RegisterBucket(s3blob.Scheme, myS3URLOpener)
4. Now use your mux to open URLs: mymux.OpenBucket("s3://my-bucket")

File: internal/website/content/concepts/structure/index.md Copy Chapter

title: "Structuring Portable Code" date: 2019-06-03T07:34:22-07:00 weight: 1 Copy Chapter

The Go CDK's APIs are intentionally structured to make it easier to separate
your application's core logic from the details of the services it is using.

Motivation Copy Chapter

Consider the [uploader tutorial][]. Without the Go CDK, we would have had to
write a code path for Amazon's Simple Storage Service (S3) and another code
path for Google Cloud Storage (GCS). That would work, but it would be
tedious. We would have to learn the semantics of uploading files to both blob
storage services. Even worse, we would have two code paths that effectively
do the same thing, but would have to be maintained separately. It would be
much nicer if we could write the upload logic once and reuse it across
providers. That's exactly the kind of separation of concerns that the Go
CDK makes possible.

(More details available in the Go CDK design doc.)

[uploader tutorial]: {{< ref "/tutorials/cli-uploader.md" >}}

Portable Types and Drivers Copy Chapter

The portable APIs that the Go CDK exports (like blob.Bucket or
runtimevar.Variable) are concrete types, not interfaces. To understand
why, imagine if we used a plain interface:

{{< figure class="FullWidthFigure" src="portable-type-no-driver.png" link="portable-type-no-driver.png" alt="Diagram showing user code depending on blob.Bucket, which is implemented by awsblob.Bucket." >}}

Consider the Bucket.NewWriter method, which infers the content type of the
blob based on the first bytes written to it. If blob.Bucket was an interface,
each implementation of blob.Bucket would have to replicate this behavior
precisely. This does not scale: conformance tests would be needed to ensure that
each interface method actually behaves in the way that the docs describe. This
makes the interfaces hard to implement, which runs counter to the goals of the
project.

Instead, we follow the example of database/sql and separate out the
implementation-agnostic logic from the interface. The implementation-agnostic
logic-containing concrete type is the portable type. We call the interface
the driver. Visually, it looks like this:

{{< figure class="FullWidthFigure" src="portable-type.png" link="portable-type.png" alt="Diagram showing user code depending on blob.Bucket, which holds a driver.Bucket implemented by awsblob.Bucket." >}}

This has a number of benefits:

• The portable type can perform higher level logic without making the
  interface complex to implement. In the blob example, the portable type's
  NewWriter method can do the content type detection and then pass the final
  result to the driver type.
• Methods can be added to the portable type without breaking compatibility.
  Contrast with adding methods to an interface, which is a breaking change.
• When new operations on the driver are added as new optional interfaces, the
  portable type can hide the need for type-assertions from the user.

(More details available in the Go CDK design doc.)

Best Practices Copy Chapter

• Create portable types as close to program startup as possible. Since
  creation of a portable type requires using driver-specific setup, this
  separates your driver-specific details from the rest of your application.
• Pass portable types around as arguments or struct fields instead of as
  package variables. This allows you to easily swap out the portable type
  for a local implementation in unit tests. It also enables you to use
  dependency injection tools like [Wire][] to set up your application.
• Avoid using [As][] functions when possible. Using driver-specific
  options makes it harder to test your code with confidence or migrate to
  another driver later. If your application needs to use driver-specific
  options, try to make it so that other drivers fall back gracefully. For
  example, you may need to use a particular ACL setting for a write to a Google
  Cloud Storage bucket. When testing for the driver-specific write options,
  don't return an error if the As function doesn't have the right type. That
  way, when running against an in-memory bucket for tests, the write will still
  occur and can be observed. Leave provider-specific checks to integration
  tests.

[As]: {{< ref "/concepts/as.md#as" >}}
[Wire]: https://github.com/google/wire

File: internal/website/content/docstore/_index.md Copy Chapter

title: gocloud.dev/docstore type: pkg Copy Chapter

File: internal/website/content/docstore/awsdynamodb/v2/_index.md Copy Chapter

title: gocloud.dev/docstore/awsdynamodb/v2 type: pkg Copy Chapter

File: internal/website/content/docstore/driver/_index.md Copy Chapter

title: gocloud.dev/docstore/driver type: pkg Copy Chapter

File: internal/website/content/docstore/drivertest/_index.md Copy Chapter

title: gocloud.dev/docstore/drivertest type: pkg Copy Chapter

File: internal/website/content/docstore/dynamodocstore/_index.md Copy Chapter

title: gocloud.dev/docstore/dynamodocstore type: pkg Copy Chapter

File: internal/website/content/docstore/firedocstore/_index.md Copy Chapter

title: gocloud.dev/docstore/firedocstore type: pkg Copy Chapter

File: internal/website/content/docstore/gcpfirestore/_index.md Copy Chapter

title: gocloud.dev/docstore/gcpfirestore type: pkg Copy Chapter

File: internal/website/content/docstore/internal/fields/_index.md Copy Chapter

title: gocloud.dev/docstore/internal/fields type: pkg Copy Chapter

File: internal/website/content/docstore/memdocstore/_index.md Copy Chapter

title: gocloud.dev/docstore/memdocstore type: pkg Copy Chapter

File: internal/website/content/docstore/mongodocstore/_index.md Copy Chapter

title: gocloud.dev/docstore/mongodocstore type: pkg Copy Chapter

File: internal/website/content/gcerrors/_index.md Copy Chapter

title: gocloud.dev/gcerrors type: pkg Copy Chapter

File: internal/website/content/gcp/_index.md Copy Chapter

title: gocloud.dev/gcp type: pkg Copy Chapter

File: internal/website/content/gcp/cloudsql/_index.md Copy Chapter

title: gocloud.dev/gcp/cloudsql type: pkg Copy Chapter

File: internal/website/content/gcp/gcpcloud/_index.md Copy Chapter

title: gocloud.dev/gcp/gcpcloud type: pkg Copy Chapter

File: internal/website/content/health/_index.md Copy Chapter

title: gocloud.dev/health type: pkg Copy Chapter

File: internal/website/content/health/sqlhealth/_index.md Copy Chapter

title: gocloud.dev/health/sqlhealth type: pkg Copy Chapter

File: internal/website/content/howto/_index.md Copy Chapter

title: "How-To Guides" date: 2019-03-20T14:50:56-07:00 showInSidenav: true pagesInSidenav: true weight: 2 Copy Chapter

The guides in this section are aimed to help you solve common tasks with
the Go CDK. They are grouped by API.

File: internal/website/content/howto/blob/_index.md Copy Chapter

title: "Blob" date: 2019-07-09T16:46:29-07:00 lastmod: 2019-07-29T12:00:00-07:00 showInSidenav: true toc: true Copy Chapter

Blobs are a common abstraction for storing unstructured data on Cloud storage
services and accessing them via HTTP. This guide shows how to work with
blobs in the Go CDK.

The blob package supports operations like reading and writing blobs (using standard
io package interfaces), deleting blobs, and listing blobs in a bucket.

Subpackages contain driver implementations of blob for various services,
including Cloud and on-prem solutions. You can develop your application
locally using fileblob, then deploy it to multiple Cloud providers with
minimal initialization reconfiguration.

Opening a Bucket {#opening} Copy Chapter

The first step in interacting with unstructured storage is
to instantiate a portable *blob.Bucket for your storage service.

The easiest way to do so is to use blob.OpenBucket and a service-specific URL
pointing to the bucket, making sure you "blank import" the driver package to
link it in.

import (
	"gocloud.dev/blob"
	_ "gocloud.dev/blob/<driver>"
)
...
bucket, err := blob.OpenBucket(context.Background(), "<driver-url>")
if err != nil {
    return fmt.Errorf("could not open bucket: %v", err)
}
defer bucket.Close()
// bucket is a *blob.Bucket; see usage below
...

See [Concepts: URLs][] for general background and the [guide below][] for URL usage
for each supported service.

Alternatively, if you need
fine-grained control over the connection settings, you can call the constructor
function in the driver package directly.

import "gocloud.dev/blob/<driver>"
...
bucket, err := <driver>.OpenBucket(...)
...

You may find the wire package useful for managing your initialization code
when switching between different backing services.

See the [guide below][] for constructor usage for each supported service.

[Concepts: URLs]: {{< ref "/concepts/urls.md" >}}
[guide below]: {{< ref "#services" >}}

Prefixed Buckets {#prefix} Copy Section

You can wrap a *blob.Bucket to always operate on a subfolder of the bucket
using blob.PrefixedBucket:

{{< goexample "gocloud.dev/blob.ExamplePrefixedBucket" >}}

Alternatively, you can configure the prefix directly in the blob.OpenBucket
URL:

{{< goexample "gocloud.dev/blob.Example_openFromURLWithPrefix" >}}

Single Key Buckets {#singlekey} Copy Section

You can wrap a *blob.Bucket to always operate on a single key
using blob.SingleKeyBucket:

{{< goexample "gocloud.dev/blob.ExampleSingleKeyBucket" >}}

Alternatively, you can configure the single key directly in the blob.OpenBucket
URL:

{{< goexample "gocloud.dev/blob.Example_openFromURLWithSingleKey" >}}

The resulting bucket will ignore the key parameter to its functions,
and always refer to the single key. This can be useful to allow configuration
of a specific "file" via a single URL.

List functions will not work on single key buckets.

Using a Bucket {#using} Copy Chapter

Once you have opened a bucket for the storage provider you want, you can
store and access data from it using the standard Go I/O patterns described
below. Other operations like listing and reading metadata are documented in the
blob package documentation.

Writing Data to a Bucket {#writing} Copy Section

To write data to a bucket, you create a writer, write data to it, and then
close the writer. Closing the writer commits the write to the provider,
flushing any buffers, and releases any resources used while writing, so you
must always check the error of Close.

The writer implements io.Writer, so you can use any functions that take
an io.Writer like io.Copy or fmt.Fprintln.

{{< goexample src="gocloud.dev/blob.ExampleBucket_NewWriter" imports="0" >}}

In some cases, you may want to cancel an in-progress write to avoid the blob
being created or overwritten. A typical reason for wanting to cancel a write
is encountering an error in the stream your program is copying from. To abort
a write, you cancel the Context you pass to the writer. Again, you must
always Close the writer to release the resources, but in this case you can
ignore the error because the write's failure is expected.

{{< goexample src="gocloud.dev/blob.ExampleBucket_NewWriter_cancel" imports="0" >}}

Reading Data from a Bucket {#reading} Copy Section

Once you have written data to a bucket, you can read it back by creating a
reader. The reader implements io.Reader, so you can use any functions
that take an io.Reader like io.Copy or io/io.ReadAll. You must
always close a reader after using it to avoid leaking resources.

{{< goexample src="gocloud.dev/blob.ExampleBucket_NewReader" imports="0" >}}

Many storage providers provide efficient random-access to data in buckets. To
start reading from an arbitrary offset in the blob, use NewRangeReader.

{{< goexample src="gocloud.dev/blob.ExampleBucket_NewRangeReader" imports="0" >}}

Deleting a Bucket {#deleting} Copy Section

You can delete blobs using the Bucket.Delete method.

{{< goexample src="gocloud.dev/blob.ExampleBucket_Delete" imports="0" >}}

Other Usage Samples Copy Chapter

• [CLI Tutorial]({{< ref "/tutorials/cli-uploader.md" >}})
• CLI Sample
• Guestbook sample
• blob package examples

Supported Storage Services {#services} Copy Chapter

Google Cloud Storage {#gcs} Copy Section

Google Cloud Storage (GCS) URLs in the Go CDK closely resemble the URLs
you would see in the gsutil CLI.

blob.OpenBucket will use Application Default Credentials; if you have
authenticated via gcloud auth application-default login, it will use those credentials. See
Application Default Credentials to learn about authentication
alternatives, including using environment variables.

{{< goexample "gocloud.dev/blob/gcsblob.Example_openBucketFromURL" >}}

Full details about acceptable URLs can be found under the API reference for
gcsblob.URLOpener.

GCS Constructor {#gcs-ctor}

The gcsblob.OpenBucket constructor opens a GCS bucket. You must first
create a *net/http.Client that sends requests authorized by Google Cloud
Platform credentials. (You can reuse the same client for any
other API that takes in a *gcp.HTTPClient.) You can find functions in the
gocloud.dev/gcp package to set this up for you.

{{< goexample "gocloud.dev/blob/gcsblob.ExampleOpenBucket" >}}

S3 {#s3} Copy Section

S3 URLs in the Go CDK closely resemble the URLs you would see in the AWS CLI.
You should specify the region query parameter to ensure your application
connects to the correct region.

It will create an AWS Config based on the AWS SDK V2; see AWS V2 Config to learn more.

Full details about acceptable URLs can be found under the API reference for
s3blob.URLOpener.

{{< goexample "gocloud.dev/blob/s3blob.Example_openBucketFromURL" >}}

S3 Constructor {#s3-ctor}

The s3blob.OpenBucket constructor opens an S3 bucket. You must first
create an AWS Config with the same region as your bucket:

{{< goexample "gocloud.dev/blob/s3blob.ExampleOpenBucket" >}}

S3-Compatible Servers {#s3-compatible}

The Go CDK can also interact with S3-compatible storage servers that
recognize the same REST HTTP endpoints as S3, like Minio, Ceph, or
SeaweedFS. You can change the endpoint by changing the Endpoint field
on the *aws.Config you pass to s3blob.OpenBucket. If you are using
blob.OpenBucket, you can switch endpoints by using the S3 URL using query
parameters like so:

bucket, err := blob.OpenBucket("s3://mybucket?" +
    "endpoint=my.minio.local:8080&" +
    "disable_https=true&" +
    "s3ForcePathStyle=true")

See aws.V2ConfigFromURLParams for more details on supported URL options for S3.

Azure Blob Storage {#azure} Copy Section

Azure Blob Storage URLs in the Go CDK allow you to identify Azure Blob Storage containers
when opening a bucket with blob.OpenBucket. Go CDK uses the environment
variables AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_KEY, and
AZURE_STORAGE_SAS_TOKEN, among others, to configure the credentials.

{{< goexample "gocloud.dev/blob/azureblob.Example_openBucketFromURL" >}}

Full details about acceptable URLs can be found under the API reference for
azureblob.URLOpener.

Azure Blob Constructor {#azure-ctor}

The azureblob.OpenBucket constructor opens an Azure Blob Storage container.
azureblob operates on Azure Storage Block Blobs. You must first create
an Azure Service Client before you can open a container.

{{< goexample "gocloud.dev/blob/azureblob.ExampleOpenBucket" >}}

Local Storage {#local} Copy Section

The Go CDK provides blob drivers for storing data in memory and on the local
filesystem. These are primarily intended for testing and local development,
but may be useful in production scenarios where an NFS mount is used.

Local storage URLs take the form of either mem:// or file:/// URLs.
Memory URLs are always mem:// with no other information and always create a
new bucket. File URLs convert slashes to the operating system's native file
separator, so on Windows, C:\foo\bar would be written as
file:///C:/foo/bar.

import (
    "gocloud.dev/blob"
    _ "gocloud.dev/blob/fileblob"
    _ "gocloud.dev/blob/memblob"
)

// ...

bucket1, err := blob.OpenBucket(ctx, "mem://")
if err != nil {
    return err
}
defer bucket1.Close()

bucket2, err := blob.OpenBucket(ctx, "file:///path/to/dir")
if err != nil {
    return err
}
defer bucket2.Close()

Local Storage Constructors {#local-ctor}

You can create an in-memory bucket with memblob.OpenBucket:

{{< goexample "gocloud.dev/blob/memblob.ExampleOpenBucket" >}}

You can use a local filesystem directory with fileblob.OpenBucket:

{{< goexample "gocloud.dev/blob/fileblob.ExampleOpenBucket" >}}

File: internal/website/content/howto/docstore/_index.md Copy Chapter

title: "Docstore" date: 2019-06-08T15:11:57-04:00 lastmod: 2019-07-29T12:00:00-07:00 showInSidenav: true toc: true Copy Chapter

The docstore package provides an abstraction layer over common
document stores like
Google Cloud Firestore, Amazon DynamoDB and MongoDB. This guide shows how to
work with document stores in the Go CDK.

A document store is a service that stores data in semi-structured JSON-like
documents grouped into collections. Like other NoSQL databases, document stores
are schemaless.

The docstore package supports operations to add, retrieve, modify and
delete documents.

Subpackages contain driver implementations of docstore for various services,
including Cloud and on-prem solutions. You can develop your application
locally using memdocstore, then deploy it to multiple Cloud providers with
minimal initialization reconfiguration.

Opening a Collection {#opening} Copy Chapter

The first step in interacting with a document store is to instantiate
a portable *docstore.Collection for your service.

While every docstore service has the concept of a primary key that uniquely
distinguishes a document in a collection, each one specifies that key in its own
way. To be portable, Docstore requires that the key be part of the document's
contents. When you open a collection using one of the functions described here,
you specify how to find the provider's primary key in the document.

The easiest way to open a collection is using docstore.OpenCollection and
a service-specific URL pointing to it, making sure you "blank import"
the driver package to link it in.

import (
    "gocloud.dev/docstore"
    _ "gocloud.dev/docstore/<driver>"
)
...
coll, err := docstore.OpenCollection(context.Background(), "<driver-url>")
if err != nil {
    return fmt.Errorf("could not open collection: %v", err)
}
defer coll.Close()
// coll is a *docstore.Collection; see usage below
...

See [Concepts: URLs][] for general background and the [guide below][] for
URL usage for each supported service.

Alternatively, if you need
fine-grained control over the connection settings, you can call the constructor
function in the driver package directly (like mongodocstore.OpenCollection).

import "gocloud.dev/docstore/<driver>"
...
coll, err := <driver>.OpenCollection(...)
...

You may find the wire package useful for managing your initialization code
when switching between different backing services.

See the [guide below][] for constructor usage for each supported service

[Concepts: URLs]: {{< ref "/concepts/urls.md" >}}
[guide below]: {{< ref "#services" >}}
wire package: http://github.com/google/wire

Using a Collection {#using} Copy Chapter

Representing Documents {#rep-doc} Copy Section

We'll use a collection with documents represented by this Go struct:

type Player struct {
    Name             string
    Score            int
    DocstoreRevision interface{}
}

We recommend using structs for documents because they impose some structure on
your data, but Docstore also accepts map[string]interface{} values. See
the docstore package documentation
for more information.

The DocstoreRevision field holds information about the latest revision of the
document. We discuss it [below]({{< ref "#rev" >}}).

Actions {#actions} Copy Section

Once you have [opened a collection]( {{< ref "#opening" >}}), you can call
action methods on it to read, modify and write documents. You can execute a
single action, or run multiple actions together in an [_action list_]({{ ref
"act-list" }}).

Docstore supports six kinds of actions on documents:

• Get retrieves a document.
• Create creates a new document.
• Replace replaces an existing document.
• Put puts a document whether or not it already exists.
• Update applies a set of modifications to a document.
• Delete deletes a document.

You can create a single document with the Collection.Create method, we will
use coll as the variable holding the collection throughout the guide:

err := coll.Create(ctx, &Player{Name: "Pat", Score: 10})
if err != nil {
    return err
}

Action Lists {#act-list}

When you use an action list to perform multiple actions at once, drivers can
optimize action lists by using bulk RPCs, running the actions concurrently,
or employing a provider's special features to improve efficiency and reduce
cost. Here we create several documents using an action list.

{{< goexample "gocloud.dev/docstore.ExampleCollection_Actions_bulkWrite" >}}

ActionList has a fluent API, so you can build and execute a sequence of
actions in one line of code. Here we Put a document and immediately Get its
new contents.

{{< goexample "gocloud.dev/docstore.ExampleCollection_Actions_getAfterWrite" >}}

If the underlying provider is eventually consistent, the result of the Get
might not reflect the Put. Docstore only guarantees that it will perform the
Get after the Put completes.

See the documentation for docstore.ActionList for the semantics of action
list execution.

Updates {#act-update}

Use Update to modify individual fields of a document. The Update action
takes a set of modifications to document fields, and applies them all
atomically. You can change the value of a field, increment it, or delete it.

{{< goexample "gocloud.dev/docstore.ExampleCollection_Update" >}}

Queries {#queries} Copy Section

Docstore's Get action lets you retrieve a single document by its primary key.
Queries let you retrieve all documents that match some conditions. You can also
use queries to delete or update all documents that match the conditions.

Getting Documents {#qr-get}

Like [actions]({{< ref "#actions" >}}), queries are built up in a fluent
style. Just as a Get action returns one document, the Query.Get method
returns several documents, in the form of an iterator.

iter := coll.Query().Where("Score", ">", 20).Get(ctx)
defer iter.Stop() // Always call Stop on an iterator.

Repeatedly calling Next on the iterator will return all the matching
documents. Like the Get action, Next will populate an empty document that
you pass to it:

doc := &Player{}
err := iter.Next(ctx, doc)

The iteration is over when Next returns io.EOF.

{{< goexample "gocloud.dev/docstore.ExampleQuery_Get" >}}

You can pass a list of fields to Get to reduce the amount of data transmitted.

Queries support the following methods:

• Where describes a condition on a document. You can ask whether a field is
  equal to, greater than, or less than a value. The "not equals" comparison
  isn't supported, because it isn't portable across providers.
• OrderBy specifies the order of the resulting documents, by field and
  direction. For portability, you can specify at most one OrderBy, and its
  field must also be mentioned in a Where clause.
• Limit limits the number of documents in the result.

If a query returns an error, the message may help you fix the problem. Some
features, like full table scans, have to be enabled via constructor options,
because they can be expensive. Other queries may require that you manually
create an index on the collection.

Revisions {#rev} Copy Section

Docstore maintains a revision for every document. Whenever the document is
changed, the revision is too. By default, Docstore stores the revision in a
field named DocstoreRevision, but you can change the field name via an option
to a Collection constructor.

You can use revisions to perform optimistic locking, a technique for updating
a document atomically:

1. Get a document. This reads the current revision.
2. Modify the document contents on the client (but do not change the revision).
3. Replace the document. If the document was changed since it was retrieved
   in step 1, the revision will be different, and Docstore will return an error
   instead of overwriting the document.
4. If the Replace failed, start again from step 1.

{{< goexample "gocloud.dev/docstore.Example_optimisticLocking" >}}

See
the Revisions section of the package documentation
for more on revisions.

Other Usage Samples Copy Chapter

• CLI Sample
• Order Processor sample
• docstore package examples

Supported Docstore Services {#services} Copy Chapter

Google Cloud Firestore {#firestore} Copy Section

The gcpfirestore
package supports
Google Cloud Firestore. Firestore
documents are uniquely named by paths that are not part of the document content.
In Docstore, these unique names are represented as part of the document. You
must supply a way to extract a document's name from its contents. This can be
done by specifying a document field that holds the name, or by providing a
function to extract the name from a document.

Firestore URLs provide the project and collection, as well as the field that
holds the document name.

docstore.OpenCollection will use Application Default Credentials; if you have
authenticated via gcloud auth application-default login, it will use those credentials. See
Application Default Credentials to learn about authentication
alternatives, including using environment variables.

{{< goexample
"gocloud.dev/docstore/gcpfirestore.Example_openCollectionFromURL" >}}

Full details about acceptable URLs can be found under the API reference for
gcpfirestore.URLOpener.

Firestore Constructors {#firestore-ctor}

The gcpfirestore.OpenCollection constructor opens a Cloud Firestore
collection as a Docstore collection. You must first connect a Firestore client
using gcpfirestore.Dial or the
cloud.google.com/go/firestore/apiv1
package. In addition to a client, OpenCollection requires a Google Cloud
project ID, the path to the Firestore collection, and the name of the field that
holds the document name.

{{< goexample "gocloud.dev/docstore/gcpfirestore.ExampleOpenCollection" >}}

Instead of mapping the document name to a field, you can supply a function to
construct the name from the document contents with
gcpfirestore.OpenCollectionWithNameFunc. This can be useful for documents
whose name is the combination of two or more fields.

{{< goexample
"gocloud.dev/docstore/gcpfirestore.ExampleOpenCollectionWithNameFunc" >}}

Amazon DynamoDB {#dynamodb} Copy Section

The awsdynamodb package
supports Amazon DynamoDB. A Docstore
collection corresponds to a DynamoDB table.

DynamoDB URLs provide the table, partition key field and optionally the sort key
field for the collection.

docstore.OpenCollection will create a default AWS Session with the
SharedConfigEnable option enabled; if you have authenticated with the AWS CLI,
it will use those credentials. See AWS Session to learn about authentication
alternatives, including using environment variables.

{{< goexample
"gocloud.dev/docstore/awsdynamodb.Example_openCollectionFromURL" >}}

Full details about acceptable URLs can be found under the API reference for
awsdynamodb.URLOpener.

DynamoDB Constructor {#dynamodb-ctor}

The awsdynamodb.OpenCollection constructor opens a DynamoDB table as a
Docstore collection. You must first create an AWS session with the same
region as your collection:

{{< goexample "gocloud.dev/docstore/awsdynamodb.ExampleOpenCollection" >}}

Azure Cosmos DB {#cosmosdb} Copy Section

Azure Cosmos DB is compatible with the MongoDB API. You can use the
mongodocstore package to connect to Cosmos DB. You must
create an Azure Cosmos account and get the MongoDB connection string.

When you use MongoDB URLs to connect to Cosmos DB, specify the Mongo server
URL by setting the MONGO_SERVER_URL environment variable to the connection
string. See the [MongoDB section][] for more details and examples on how to
use the package.

[MongoDB section]: {{< ref "#mongo" >}}

Cosmos DB Constructors {#cosmosdb-ctor}

The mongodocstore.OpenCollection constructor can open a Cosmos DB
collection. You must first obtain a standard MongoDB Go client with your
Cosmos connections string. See the [MongoDB constructor section][] for more
details and examples.

[MongoDB constructor section]: {{< ref "#mongo-ctor" >}}

MongoDB {#mongo} Copy Section

The mongodocstore package supports the popular
MongoDB document store. MongoDB documents are uniquely
identified by a field called _id. In Docstore, you can choose a different
name for this field, or provide a function to extract the document ID from a
document.

MongoDB URLs provide the database and collection, and optionally the field that
holds the document ID. Specify the Mongo server URL by setting the
MONGO_SERVER_URL environment variable.

{{< goexample
"gocloud.dev/docstore/mongodocstore.Example_openCollectionFromURL" >}}

Full details about acceptable URLs can be found under the API reference for
mongodocstore.URLOpener.

MongoDB Constructors {#mongo-ctor}

The mongodocstore.OpenCollection constructor opens a MongoDB collection.
You must first obtain a standard MongoDB Go client using
mongodocstore.Dial or the package
go.mongodb.org/mongo-driver/mongo.
Obtain a *mongo.Collection from the client with
client.Database(dbName).Collection(collName). Then pass the result to
mongodocstore.OpenCollection along with the name of the ID field, or "" to
use _id.

{{< goexample "gocloud.dev/docstore/mongodocstore.ExampleOpenCollection" >}}

Instead of mapping the document ID to a field, you can supply a function to
construct the ID from the document contents with
mongodocstore.OpenCollectionWithIDFunc. This can be useful for documents
whose name is the combination of two or more fields.

{{< goexample
"gocloud.dev/docstore/mongodocstore.ExampleOpenCollectionWithIDFunc" >}}

In-Memory Document Store {#mem} Copy Section

The memdocstore
package implements an in-memory document store suitable for testing and
development.

URLs for the in-memory store have a mem: scheme. The URL host is used as the
the collection name, and the URL path is used as the name of the document field
to use as a primary key.

{{< goexample
"gocloud.dev/docstore/memdocstore.Example_openCollectionFromURL" >}}

Full details about acceptable URLs can be found under the API reference for
memdocstore.URLOpener.

Mem Constructors {#mem-ctor}

The memdocstore.OpenCollection constructor creates and opens a collection,
taking the name of the key field.

{{< goexample "gocloud.dev/docstore/memdocstore.ExampleOpenCollection" >}}

You can instead supply a function to construct the primary key from the document
contents with memdocstore.OpenCollectionWithKeyFunc. This can be useful
for documents whose name is the combination of two or more fields.

{{< goexample
"gocloud.dev/docstore/memdocstore.ExampleOpenCollectionWithKeyFunc" >}}

File: internal/website/content/howto/pubsub/_index.md Copy Chapter

title: "Pub/Sub" date: 2019-03-26T09:44:06-07:00 showInSidenav: true Copy Chapter

The pubsub package provides an easy and portable way to interact with
publish/subscribe systems. This guide shows how to work with pubsub
in the Go CDK.

The publish/subscribe model allows parts of a system to publish messages
that other parts of a system may subscribe to. This is commonly used to
arrange for work to happen at some point after an interactive frontend
request is finished or in other event-driven computing.

The pubsub package supports operations to publish messages to a topic and
to subscribe to receive messages from a topic.

Subpackages contain driver implementations of pubsub for various services,
including Cloud and on-prem solutions. You can develop your application
locally using mempubsub, then deploy it to multiple Cloud providers with
minimal initialization reconfiguration.

File: internal/website/content/howto/pubsub/publish.md Copy Chapter

title: "Publish Messages to a Topic" date: 2019-03-26T09:44:15-07:00 lastmod: 2019-07-29T12:00:00-07:00 weight: 1 toc: true Copy Chapter

Publishing a message to a topic with the Go CDK takes two steps:

1. [Open a topic][] with the Pub/Sub provider of your choice (once per topic).
2. [Send messages][] on the topic.

[Open a topic]: {{< ref "#opening" >}}
[Send messages]: {{< ref "#sending" >}}

Opening a Topic {#opening} Copy Chapter

The first step in publishing messages to a topic is to instantiate a
portable [*pubsub.Topic][] for your service.

The easiest way to do so is to use [pubsub.OpenTopic][] and a service-specific URL
pointing to the topic, making sure you "blank import" the driver package to
link it in.

import (
    "context"

    "gocloud.dev/pubsub"
    _ "gocloud.dev/pubsub/<driver>"
)
...
ctx := context.Background()
topic, err := pubsub.OpenTopic(ctx, "<driver-url>")
if err != nil {
    return fmt.Errorf("could not open topic: %v", err)
}
defer topic.Shutdown(ctx)
// topic is a *pubsub.Topic; see usage below
...

See [Concepts: URLs][] for general background and the [guide below][]
for URL usage for each supported service.

Alternatively, if you need fine-grained
control over the connection settings, you can call the constructor function in
the driver package directly (like gcppubsubv2.OpenTopic).

import "gocloud.dev/pubsub/<driver>"
...
topic, err := <driver>.OpenTopic(...)
...

You may find the wire package useful for managing your initialization code
when switching between different backing services.

See the [guide below][] for constructor usage for each supported service.

[guide below]: {{< ref "#services" >}}
[*pubsub.Topic]: https://godoc.org/gocloud.dev/pubsub#Topic
[pubsub.OpenTopic]:
https://godoc.org/gocloud.dev/pubsub#OpenTopic
"blank import": https://golang.org/doc/effective_go.html#blank_import
[Concepts: URLs]: {{< ref "/concepts/urls.md" >}}
wire package: http://github.com/google/wire

Sending Messages on a Topic {#sending} Copy Chapter

Sending a message on a Topic looks
like this:

{{< goexample src="gocloud.dev/pubsub.ExampleTopic_Send" imports="0" >}}

Note that the semantics of message delivery can vary by backing service.

Other Usage Samples Copy Chapter

• CLI Sample
• Order Processor sample
• pubsub package examples

Supported Pub/Sub Services {#services} Copy Chapter

Google Cloud Pub/Sub {#gcp} Copy Section

The Go CDK can publish to a Google Cloud Pub/Sub topic. The URLs use the
project ID and the topic ID.

pubsub.OpenTopic will use Application Default Credentials; if you have
authenticated via gcloud auth application-default login, it will use those credentials. See
Application Default Credentials to learn about authentication
alternatives, including using environment variables.

{{< goexample "gocloud.dev/pubsub/gcppubsubv2.Example_openTopicFromURL" >}}

Google Cloud Pub/Sub Constructor {#gcp-ctor}

The gcppubsubv2.OpenTopic constructor opens a Cloud Pub/Sub topic. You
must first obtain GCP credentials and then create a gRPC
connection to Cloud Pub/Sub. (This gRPC connection can be reused among
topics.)

{{< goexample "gocloud.dev/pubsub/gcppubsubv2.ExampleOpenTopic" >}}

Amazon Simple Notification Service {#sns} Copy Section

The Go CDK can publish to an Amazon Simple Notification Service (SNS)
topic. SNS URLs in the Go CDK use the Amazon Resource Name (ARN) to identify
the topic. You should specify the region query parameter to ensure your
application connects to the correct region.

It will create an AWS Config based on the AWS SDK V2; see AWS V2 Config to learn more.

{{< goexample "gocloud.dev/pubsub/awssnssqs.Example_openSNSTopicFromURL" >}}

SNS messages are restricted to UTF-8 clean payloads. If your application
sends a message that contains non-UTF-8 bytes, then the Go CDK will
automatically Base64 encode the message and add a base64encoded message
attribute. When subscribing to messages on the topic through the Go CDK,
these will be [automatically Base64 decoded][SQS Subscribe], but if you are
receiving messages from a topic in a program that does not use the Go CDK,
you may need to manually Base64 decode the message payload.

[SQS Subscribe]: {{< relref "./subscribe.md#sqs" >}}
AWS V2 Config: https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/

Amazon SNS Constructor {#sns-ctor}

The awssnssqs.OpenSNSTopic constructor opens an SNS topic. You must first
create an AWS Config with the same region as your topic:

{{< goexample "gocloud.dev/pubsub/awssnssqs.ExampleOpenSNSTopic" >}}

Amazon Simple Queue Service {#sqs} Copy Section

The Go CDK can publish to an Amazon Simple Queue Service (SQS)
topic. SQS URLs closely resemble the the queue URL, except the leading
https:// is replaced with awssqs://. You can specify the region
query parameter to ensure your application connects to the correct region, but
otherwise pubsub.OpenTopic will use the region found in the environment
variables or your AWS CLI configuration.

{{< goexample "gocloud.dev/pubsub/awssnssqs.Example_openSQSTopicFromURL" >}}

SQS messages are restricted to UTF-8 clean payloads. If your application
sends a message that contains non-UTF-8 bytes, then the Go CDK will
automatically Base64 encode the message and add a base64encoded message
attribute. When subscribing to messages on the topic through the Go CDK,
these will be [automatically Base64 decoded][SQS Subscribe], but if you are
receiving messages from a topic in a program that does not use the Go CDK,
you may need to manually Base64 decode the message payload.

[SQS Subscribe]: {{< relref "./subscribe.md#sqs" >}}
SQS: https://aws.amazon.com/sqs/

Amazon SQS Constructor {#sqs-ctor}

The awssnssqs.OpenSQSTopic constructor opens an SQS topic. You must first
create an AWS Config with the same region as your topic:

{{< goexample "gocloud.dev/pubsub/awssnssqs.ExampleOpenSQSTopic" >}}

Azure Service Bus {#azure} Copy Section

The Go CDK can publish to an Azure Service Bus topic.
The URL for publishing is the topic name. pubsub.OpenTopic will use the
environment variable SERVICEBUS_CONNECTION_STRING to obtain the Service Bus
connection string. The connection string can be obtained
from the Azure portal.

{{< goexample "gocloud.dev/pubsub/azuresb.Example_openTopicFromURL" >}}

Azure Service Bus Constructor {#azure-ctor}

The azuresb.OpenTopic constructor opens an Azure Service Bus topic. You
must first connect to the topic using the Azure Service Bus library and
then pass it to azuresb.OpenTopic. There are also helper functions in the
azuresb package to make this easier.

{{< goexample "gocloud.dev/pubsub/azuresb.ExampleOpenTopic" >}}

RabbitMQ {#rabbitmq} Copy Section

The Go CDK can publish to an AMQP 0.9.1 fanout exchange, the dialect of
AMQP spoken by RabbitMQ. A RabbitMQ URL only includes the exchange name.
The RabbitMQ's server is discovered from the RABBIT_SERVER_URL environment
variable (which is something like amqp://guest:guest@localhost:5672/).

{{< goexample "gocloud.dev/pubsub/rabbitpubsub.Example_openTopicFromURL" >}}

RabbitMQ Constructor {#rabbitmq-ctor}

The rabbitpubsub.OpenTopic constructor opens a RabbitMQ exchange. You
must first create an *amqp.Connection to your RabbitMQ instance.

{{< goexample "gocloud.dev/pubsub/rabbitpubsub.ExampleOpenTopic" >}}

NATS {#nats} Copy Section

The Go CDK can publish to a NATS subject. A NATS URL only includes the
subject name. The NATS server is discovered from the NATS_SERVER_URL
environment variable (which is something like nats://nats.example.com).

{{< goexample "gocloud.dev/pubsub/natspubsub.Example_openTopicFromURL" >}}

Because NATS does not natively support metadata, messages sent to NATS will
be encoded with gob.

NATS Constructor {#nats-ctor}

The natspubsub.OpenTopic constructor opens a NATS subject as a topic. You
must first create an *nats.Conn to your NATS instance.

{{< goexample "gocloud.dev/pubsub/natspubsub.ExampleOpenTopic" >}}

Kafka {#kafka} Copy Section

The Go CDK can publish to a Kafka cluster. A Kafka URL only includes the
topic name. The brokers in the Kafka cluster are discovered from the
KAFKA_BROKERS environment variable (which is a comma-delimited list of
hosts, something like 1.2.3.4:9092,5.6.7.8:9092).

{{< goexample "gocloud.dev/pubsub/kafkapubsub.Example_openTopicFromURL" >}}

Kafka Constructor {#kafka-ctor}

The kafkapubsub.OpenTopic constructor opens a Kafka topic to publish
messages to. Depending on your Kafka cluster configuration (see
auto.create.topics.enable), you may need to provision the topic beforehand.

In addition to the list of brokers, you'll need a *sarama.Config, which
exposes many knobs that can affect performance and semantics; review and set
them carefully. kafkapubsub.MinimalConfig provides a minimal config to get
you started.

{{< goexample "gocloud.dev/pubsub/kafkapubsub.ExampleOpenTopic" >}}

In-Memory {#mem} Copy Section

The Go CDK includes an in-memory Pub/Sub provider useful for local testing.
The names in mem:// URLs are a process-wide namespace, so subscriptions to
the same name will receive messages posted to that topic. This is detailed
more in the [subscription guide][subscribe-mem].

{{< goexample "gocloud.dev/pubsub/mempubsub.Example_openTopicFromURL" >}}

[subscribe-mem]: {{< ref "./subscribe.md#mem" >}}

In-Memory Constructor {#mem-ctor}

To create an in-memory Pub/Sub topic, use the mempubsub.NewTopic
function. You can use the returned topic to create in-memory
subscriptions, as detailed in the [subscription guide][subscribe-mem-ctor].

{{< goexample "gocloud.dev/pubsub/mempubsub.ExampleNewTopic" >}}

[subscribe-mem-ctor]: {{< ref "./subscribe.md#mem-ctor" >}}

File: internal/website/content/howto/pubsub/subscribe.md Copy Chapter

title: "Subscribe to Messages from a Topic" date: 2019-03-26T09:44:33-07:00 lastmod: 2019-07-29T12:00:00-07:00 weight: 2 toc: true Copy Chapter

Subscribing to receive messages from a topic with the Go CDK takes three steps:

1. [Open a subscription][] to a topic with the Pub/Sub service of your choice (once per
   subscription).
2. [Receive and acknowledge messages][] from the topic. After completing any
   work related to the message, use the Ack method to prevent it from being
   redelivered.

[Open a subscription]: {{< ref "#opening" >}}
[Receive and acknowledge messages]: {{< ref "#receiving" >}}

Opening a Subscription {#opening} Copy Chapter

The first step in subscribing to receive messages from a topic is
to instantiate a portable *pubsub.Subscription for your service.

The easiest way to do so is to use [pubsub.OpenSubscription][]
and a service-specific URL pointing to the topic, making sure you
"blank import" the driver package to link it in.

import (
    "context"

    "gocloud.dev/pubsub"
    _ "gocloud.dev/pubsub/<driver>"
)
...
ctx := context.Background()
subs, err := pubsub.OpenSubscription(ctx, "<driver-url>")
if err != nil {
    return fmt.Errorf("could not open topic subscription: %v", err)
}
defer subs.Shutdown(ctx)
// subs is a *pubsub.Subscription; see usage below
...

See [Concepts: URLs][] for general background and the [guide below][]
for URL usage for each supported service.

Alternatively, if you need fine-grained
control over the connection settings, you can call the constructor function in
the driver package directly (like gcppubsubv2.OpenSubscription).

import "gocloud.dev/pubsub/<driver>"
...
subs, err := <driver>.OpenSubscription(...)
...

You may find the wire package useful for managing your initialization code
when switching between different backing services.

See the [guide below][] for constructor usage for each supported service.

[guide below]: {{< ref "#services" >}}
[pubsub.OpenSubscription]:
https://godoc.org/gocloud.dev/pubsub#OpenTopic
"blank import": https://golang.org/doc/effective_go.html#blank_import
[Concepts: URLs]: {{< ref "/concepts/urls.md" >}}
wire package: http://github.com/google/wire

Receiving and Acknowledging Messages {#receiving} Copy Chapter

A simple subscriber that operates on
messages serially looks like
this:

{{< goexample src="gocloud.dev/pubsub.ExampleSubscription_Receive" imports="0" >}}

If you want your subscriber to operate on incoming messages concurrently,
you can start multiple goroutines:

{{< goexample src="gocloud.dev/pubsub.ExampleSubscription_Receive_concurrent" imports="0" >}}

Note that the semantics of message delivery can vary by backing service.

Other Usage Samples Copy Chapter

• CLI Sample
• Order Processor sample
• pubsub package examples

Supported Pub/Sub Services {#services} Copy Chapter

Google Cloud Pub/Sub {#gcp} Copy Section

The Go CDK can receive messages from a Google Cloud Pub/Sub subscription.
The URLs use the project ID and the subscription ID.

pubsub.OpenSubscription will use Application Default Credentials; if you have
authenticated via gcloud auth application-default login, it will use those credentials. See
Application Default Credentials to learn about authentication
alternatives, including using environment variables.

{{< goexample "gocloud.dev/pubsub/gcppubsubv2.Example_openSubscriptionFromURL" >}}

Google Cloud Pub/Sub Constructor {#gcp-ctor}

The gcppubsubv2.OpenSubscription constructor opens a Cloud Pub/Sub
subscription. You must first obtain GCP credentials and then
create a gRPC connection to Cloud Pub/Sub. (This gRPC connection can be
reused among subscriptions.)

{{< goexample "gocloud.dev/pubsub/gcppubsubv2.ExampleOpenSubscription" >}}

Amazon Simple Queueing Service {#sqs} Copy Section

The Go CDK can subscribe to an Amazon Simple Queueing Service (SQS)
topic. SQS URLs closely resemble the the queue URL, except the leading
https:// is replaced with awssqs://. You should specify the region
query parameter to ensure your application connects to the correct region.

pubsub.OpenSubscription will open a subscription using a default AWS Config.

{{< goexample "gocloud.dev/pubsub/awssnssqs.Example_openSubscriptionFromURL" >}}

If your messages are being sent to SQS directly, or if they are being delivered
via an SNS topic with RawMessageDelivery enabled, set a raw=true query
parameter in your URL, or set SubscriberOptions.Raw to true if you're using
the constructors. By default, the subscription will use heuristics to identify
whether the message bodies are raw or [SNS JSON][].

Messages with a base64encoded message attribute will be automatically
Base64 decoded before being returned. See the [SNS publishing guide][]
or the [SQS publishing guide][] for more details.

[SNS publishing guide]: {{< ref "./publish.md#sns" >}}
[SQS publishing guide]: {{< ref "./publish.md#sqs" >}}
[SNS JSON]: https://aws.amazon.com/sns/faqs/#Raw_message_delivery

Amazon SQS Constructor {#sqs-ctor}

The awssnssqs.OpenSubscription constructor opens an SQS queue. You must
first create an AWS Config with the same region as your topic:

{{< goexample "gocloud.dev/pubsub/awssnssqs.ExampleOpenSubscription" >}}

Azure Service Bus {#azure} Copy Section

The Go CDK can receive messages from an Azure Service Bus subscription.
The URL for subscribing is the topic name with the
subscription name in the subscription query parameter.
pubsub.OpenSubscription will use the environment variable
SERVICEBUS_CONNECTION_STRING to obtain the Service Bus Connection String
you need to copy from the Azure portal.

{{< goexample "gocloud.dev/pubsub/azuresb.Example_openSubscriptionFromURL" >}}

Azure Service Bus Constructor {#azure-ctor}

The azuresb.OpenSubscription constructor opens an Azure Service Bus
subscription. You must first connect to the topic and subscription using the
Azure Service Bus library and then pass the subscription to
azuresb.OpenSubscription. There are also helper functions in the azuresb
package to make this easier.

{{< goexample "gocloud.dev/pubsub/azuresb.ExampleOpenSubscription" >}}

RabbitMQ {#rabbitmq} Copy Section

The Go CDK can receive messages from an AMQP 0.9.1 queue, the dialect of
AMQP spoken by RabbitMQ. A RabbitMQ URL only includes the queue name.
The RabbitMQ's server is discovered from the RABBIT_SERVER_URL environment
variable (which is something like amqp://guest:guest@localhost:5672/).

{{< goexample "gocloud.dev/pubsub/rabbitpubsub.Example_openSubscriptionFromURL" >}}

RabbitMQ Constructor {#rabbitmq-ctor}

The rabbitpubsub.OpenSubscription constructor opens a RabbitMQ queue.
You must first create an *amqp.Connection to your RabbitMQ instance.

{{< goexample "gocloud.dev/pubsub/rabbitpubsub.ExampleOpenSubscription" >}}

NATS {#nats} Copy Section

The Go CDK can publish to a NATS subject. A NATS URL only includes the
subject name. The NATS server is discovered from the NATS_SERVER_URL
environment variable (which is something like nats://nats.example.com).

{{< goexample "gocloud.dev/pubsub/natspubsub.Example_openSubscriptionFromURL" >}}

NATS guarantees at-most-once delivery; it will never redeliver a message.
Therefore, Message.Ack is a no-op.

To parse messages [published via the Go CDK][publish#nats], the NATS driver
will first attempt to decode the payload using gob. Failing that, it will
return the message payload as the Data with no metadata to accomodate
subscribing to messages coming from a source not using the Go CDK.

[publish#nats]: {{< ref "./publish.md#nats" >}}

NATS Constructor {#nats-ctor}

The natspubsub.OpenSubscription constructor opens a NATS subject as a
topic. You must first create an *nats.Conn to your NATS instance.

{{< goexample "gocloud.dev/pubsub/natspubsub.ExampleOpenSubscription" >}}

Kafka {#kafka} Copy Section

The Go CDK can receive messages from a Kafka cluster.
A Kafka URL includes the consumer group name, plus at least one instance
of a query parameter specifying the topic to subscribe to.
The brokers in the Kafka cluster are discovered from the
KAFKA_BROKERS environment variable (which is a comma-delimited list of
hosts, something like 1.2.3.4:9092,5.6.7.8:9092).

{{< goexample "gocloud.dev/pubsub/kafkapubsub.Example_openSubscriptionFromURL" >}}

Kafka Constructor {#kafka-ctor}

The kafkapubsub.OpenSubscription constructor creates a consumer in a
consumer group, subscribed to one or more topics.

In addition to the list of brokers, you'll need a *sarama.Config, which
exposes many knobs that can affect performance and semantics; review and set
them carefully. kafkapubsub.MinimalConfig provides a minimal config to
get you started.

{{< goexample "gocloud.dev/pubsub/kafkapubsub.ExampleOpenSubscription" >}}

In-Memory {#mem} Copy Section

The Go CDK includes an in-memory Pub/Sub provider useful for local testing.
The names in mem:// URLs are a process-wide namespace, so subscriptions to
the same name will receive messages posted to that topic. For instance, if
you open a topic mem://topicA and open two subscriptions with
mem://topicA, you will have two subscriptions to the same topic.

{{< goexample "gocloud.dev/pubsub/mempubsub.Example_openSubscriptionFromURL" >}}

In-Memory Constructor {#mem-ctor}

To create a subscription to an in-memory Pub/Sub topic, pass the [topic you
created][publish-mem-ctor] into the mempubsub.NewSubscription function.
You will also need to pass an acknowledgement deadline: once a message is
received, if it is not acknowledged after the deadline elapses, then it will be
redelivered.

{{< goexample "gocloud.dev/pubsub/mempubsub.ExampleNewSubscription" >}}

[publish-mem-ctor]: {{< ref "./publish.md#mem-ctor" >}}

File: internal/website/content/howto/runtimevar/_index.md Copy Chapter

title: "Runtime Configuration" date: 2019-07-11T12:00:00-07:00 lastmod: 2020-12-23T12:00:00-07:00 showInSidenav: true toc: true Copy Chapter

The runtimevar package provides an easy and portable way to watch runtime
configuration variables. This guide shows how to work with runtime configuration
variables using the Go CDK.

Subpackages contain driver implementations of runtimevar for various services,
including Cloud and on-prem solutions. You can develop your application locally
using filevar or constantvar, then deploy it to multiple Cloud
providers with minimal initialization reconfiguration.

Opening a Variable {#opening} Copy Chapter

The first step in watching a variable is to instantiate a portable
*runtimevar.Variable for your service.

The easiest way to do so is to use runtimevar.OpenVariable and a service-specific URL pointing
to the variable, making sure you "blank import" the driver package to link
it in.

import (
	"gocloud.dev/runtimevar"
	_ "gocloud.dev/runtimevar/<driver>"
)
...
v, err := runtimevar.OpenVariable(context.Background(), "<driver-url>")
if err != nil {
    return fmt.Errorf("could not open variable: %v", err)
}
defer v.Close()
// v is a *runtimevar.Variable; see usage below
...

See [Concepts: URLs][] for general background and the [guide below][]
for URL usage for each supported service.

Alternatively, if you need fine-grained control
over the connection settings, you can call the constructor function in the
driver package directly (like etcdvar.OpenVariable).

import "gocloud.dev/runtimevar/<driver>"
...
v, err := <driver>.OpenVariable(...)
...

You may find the wire package useful for managing your initialization code
when switching between different backing services.

See the [guide below][] for constructor usage for each supported service.

When opening the variable, you can provide a [decoder][] parameter (either as a
[query parameter][] for URLs, or explicitly to the constructor) to specify
whether the raw value stored in the variable is interpreted as a string, a
[]byte, or as JSON. Here's an example of using a JSON encoder:

{{< goexample src="gocloud.dev/runtimevar.Example_jsonDecoder" imports="0" >}}

[Concepts: URLs]: {{< ref "/concepts/urls.md" >}}
[decoder]: https://godoc.org/gocloud.dev/runtimevar#Decoder
[guide below]: {{< ref "#services" >}}
[query parameter]: https://godoc.org/gocloud.dev/runtimevar#DecoderByName
wire package: http://github.com/google/wire

Using a Variable {#using} Copy Chapter

Once you have opened a runtimevar.Variable for the provider you want, you can
use it portably.

Latest {#latest} Copy Section

The easiest way to a Variable is to use the Variable.Latest method. It
returns the latest good Snapshot of the variable value, blocking if no
good value has ever been detected. The dynamic type of Snapshot.Value
depends on the decoder you provided when creating the Variable.

{{< goexample src="gocloud.dev/runtimevar.ExampleVariable_Latest" imports="0" >}}

To avoid blocking, you can pass an already-Done context. You can also use
Variable.CheckHealth, which reports as healthy when Latest will
return a value without blocking.

Watch {#watch} Copy Section

Variable also has a Watch method for obtaining the value of a variable;
it has different semantics than Latest and may be useful in some scenarios. We
recommend starting with Latest as it's conceptually simpler to work with.

Other Usage Samples Copy Chapter

• CLI Sample
• Guestbook sample
• runtimevar package examples

Supported Services {#services} Copy Chapter

GCP Runtime Configurator {#gcprc} Copy Section

To open a variable stored in GCP Runtime Configurator via a URL, you can use
the runtimevar.OpenVariable function as shown in the example below.

runtimevar.OpenVariable will use Application Default Credentials; if you have
authenticated via gcloud auth application-default login, it will use those credentials. See
Application Default Credentials to learn about authentication
alternatives, including using environment variables.

{{< goexample
"gocloud.dev/runtimevar/gcpruntimeconfig.Example_openVariableFromURL" >}}

GCP Constructor {#gcprc-ctor}

The gcpruntimeconfig.OpenVariable constructor opens a Runtime Configurator
variable.

{{< goexample "gocloud.dev/runtimevar/gcpruntimeconfig.ExampleOpenVariable" >}}

GCP Secret Manager {#gcpsm} Copy Section

To open a variable stored in GCP Secret Manager via a URL, you can use
the runtimevar.OpenVariable function as shown in the example below.

runtimevar.OpenVariable will use Application Default Credentials; if you have
authenticated via gcloud auth application-default login, it will use those credentials. See
Application Default Credentials to learn about authentication
alternatives, including using environment variables.

{{< goexample
"gocloud.dev/runtimevar/gcpsecretmanager.Example_openVariableFromURL" >}}

GCP Constructor {#gcpsm-ctor}

The gcpsecretmanager.OpenVariable constructor opens a Secret Manager
variable.

{{< goexample "gocloud.dev/runtimevar/gcpsecretmanager.ExampleOpenVariable" >}}

AWS Parameter Store {#awsps} Copy Section

To open a variable stored in AWS Parameter Store via a URL, you can use the
runtimevar.OpenVariable function as shown in the example below.

It will create an AWS Config based on the AWS SDK V2; see AWS V2 Config to learn more.

{{< goexample
"gocloud.dev/runtimevar/awsparamstore.Example_openVariableFromURL" >}}

AWS Constructor {#awsps-ctor}

The awsparamstore.OpenVariable constructor opens a Parameter Store
variable.

{{< goexample "gocloud.dev/runtimevar/awsparamstore.ExampleOpenVariable" >}}

AWS Secrets Manager {#awssm} Copy Section

To open a variable stored in AWS Secrets Manager via a URL, you can use the
runtimevar.OpenVariable function as shown in the example below.

It will create an AWS Config based on the AWS SDK V2; see AWS V2 Config to learn more.

{{< goexample
"gocloud.dev/runtimevar/awssecretsmanager.Example_openVariableFromURL" >}}

AWS Constructor {#awssm-ctor}

The awssecretsmanager.OpenVariable constructor opens a Secrets Manager
variable.

{{< goexample "gocloud.dev/runtimevar/awssecretsmanager.ExampleOpenVariable" >}}

Note that both secretsmanager:GetSecretValue and secretsmanager:DescribeSecret actions must be allowed in
the caller's IAM policy.

etcd {#etcd} Copy Section

NOTE: Support for etcd has been temporarily dropped due to dependency
issues. See https://github.com/google/go-cloud/issues/2914.

You can use runtimevar.etcd in Go CDK version v0.20.0 or earlier.

HTTP {#http} Copy Section

httpvar supports watching a variable via an HTTP request. Use
runtimevar.OpenVariable with a regular URL starting with http or https.
httpvar will periodically make an HTTP GET request to that URL, with the
decode URL parameter removed (if present).

{{< goexample "gocloud.dev/runtimevar/httpvar.Example_openVariableFromURL" >}}

HTTP Constructor {#http-ctor}

The httpvar.OpenVariable constructor opens a variable with a http.Client
and a URL.

{{< goexample "gocloud.dev/runtimevar/httpvar.ExampleOpenVariable" >}}

HashiCorp Vault {#hashivault} Copy Section

hashivault supports watching a variable stored in HashiCorp Vault's KV Secrets Engine.
Use runtimevar.OpenVariable with a URL starting with hashivault://.

The default URL opener will use the environment variables VAULT_SERVER_URL (or
VAULT_ADDR) for the server address, and VAULT_SERVER_TOKEN (or VAULT_TOKEN)
for authentication.

{{< goexample "gocloud.dev/runtimevar/hashivault.Example_openVariableFromURL" >}}

HashiCorp Vault Constructor {#hashivault-ctor}

The hashivault.OpenVariable constructor opens a variable with a Vault client.

{{< goexample "gocloud.dev/runtimevar/hashivault.ExampleOpenVariable" >}}

Blob {#blob} Copy Section

blobvar supports watching a variable based on the contents of a
Go CDK blob. Set the environment variable BLOBVAR_BUCKET_URL to the URL
of the bucket, and then use runtimevar.OpenVariable as shown below.
blobvar will periodically re-fetch the contents of the blob.

{{< goexample "gocloud.dev/runtimevar/blobvar.Example_openVariableFromURL" >}}

You can also use blobvar.OpenVariable.

Local {#local} Copy Section

You can create an in-memory variable (useful for testing) using constantvar:

{{< goexample "gocloud.dev/runtimevar/constantvar.Example_openVariableFromURL" >}}

Alternatively, you can create a variable based on the contents of a file using
filevar:

{{< goexample "gocloud.dev/runtimevar/filevar.Example_openVariableFromURL" >}}

File: internal/website/content/howto/secrets/_index.md Copy Chapter

title: "Secrets" date: 2019-03-21T17:42:18-07:00 lastmod: 2019-07-29T12:00:00-07:00 showInSidenav: true toc: true Copy Chapter

The secrets package provides access to key management services in a
portable way. This guide shows how to work with secrets in the Go CDK.

Cloud applications frequently need to store sensitive information like web
API credentials or encryption keys in a medium that is not fully secure. For
example, an application that interacts with GitHub needs to store its OAuth2
client secret and use it when obtaining end-user credentials. If this
information was compromised, it could allow someone else to impersonate the
application. In order to keep such information secret and secure, you can
encrypt the data, but then you need to worry about rotating the encryption
keys and distributing them securely to all of your application servers.
Most Cloud providers include a key management service to perform these tasks,
usually with hardware-level security and audit logging.

The secrets package supports encryption and decryption operations.

Subpackages contain driver implementations of secrets for various services,
including Cloud and on-prem solutions. You can develop your application
locally using localsecrets, then deploy it to multiple Cloud providers with
minimal initialization reconfiguration.

Opening a SecretsKeeper {#opening} Copy Chapter

The first step in working with your secrets is
to instantiate a portable *secrets.Keeper for your service.

The easiest way to do so is to use secrets.OpenKeeper and a service-specific
URL pointing to the keeper, making sure you "blank import" the driver
package to link it in.

import (
	"gocloud.dev/secrets"
	_ "gocloud.dev/secrets/<driver>"
)
...
keeper, err := secrets.OpenKeeper(context.Background(), "<driver-url>")
if err != nil {
    return fmt.Errorf("could not open keeper: %v", err)
}
defer keeper.Close()
// keeper is a *secrets.Keeper; see usage below
...

See [Concepts: URLs][] for general background and the [guide below][]
for URL usage for each supported service.

Alternatively, if you need
fine-grained control over the connection settings, you can call the constructor
function in the driver package directly (like awskms.OpenKeeper).

import "gocloud.dev/secrets/<driver>"
...
keeper, err := <driver>.OpenKeeper(...)
...

You may find the wire package useful for managing your initialization code
when switching between different backing services.

See the [guide below][] for constructor usage for each supported service.

[Concepts: URLs]: {{< ref "/concepts/urls.md" >}}
[guide below]: {{< ref "#services" >}}
wire package: http://github.com/google/wire

Using a SecretsKeeper {#using} Copy Chapter

Once you have [opened a secrets keeper][] for the secrets provider you want,
you can encrypt and decrypt small messages using the keeper.

[opened a secrets keeper]: {{< ref "#opening" >}}

Encrypting Data {#encrypt} Copy Section

To encrypt data with a keeper, you call Encrypt with the byte slice you
want to encrypt.

{{< goexample src="gocloud.dev/secrets.ExampleKeeper_Encrypt" imports="0" >}}

Decrypting Data {#decrypt} Copy Section

To decrypt data with a keeper, you call Decrypt with the byte slice you
want to decrypt. This should be data that you obtained from a previous call
to Encrypt with a keeper that uses the same secret material (e.g. two AWS
KMS keepers created with the same customer master key ID). The Decrypt
method will return an error if the input data is corrupted.

{{< goexample src="gocloud.dev/secrets.ExampleKeeper_Decrypt" imports="0" >}}

Large Messages {#large-messages} Copy Section

The secrets keeper API is designed to work with small messages (i.e. <10 KiB
in length.) Cloud key management services are high latency; using them for
encrypting or decrypting large amounts of data is prohibitively slow (and in
some providers not permitted). If you need your application to encrypt or
decrypt large amounts of data, you should:

1. Generate a key for the encryption algorithm (16KiB chunks with
   secretbox is a reasonable approach).
2. Encrypt the key with secret keeper.
3. Store the encrypted key somewhere accessible to the application.

When your application needs to encrypt or decrypt a large message:

1. Decrypt the key from storage using the secret keeper
2. Use the decrypted key to encrypt or decrypt the message inside your
   application.

Keep Secrets in Configuration {#runtimevar} Copy Section

Once you have [opened a secrets keeper][] for the secrets provider you want,
you can use a secrets keeper to access sensitive configuration stored in an
encrypted runtimevar.

First, you create a [*runtimevar.Decoder][] configured to use your secrets
keeper using [runtimevar.DecryptDecode][]. In this example, we assume the
data is a plain string, but the configuration could be a more structured
type.

{{< goexample src="gocloud.dev/runtimevar.ExampleDecryptDecode" imports="0" >}}

Then you can pass the decoder to the runtime configuration provider of your
choice. See the [Runtime Configuration How-To Guide][] for more on how to set up
runtime configuration.

[opened a secrets keeper]: {{< ref "#opening" >}}
[Runtime Configuration How-To Guide]: {{< ref "/howto/runtimevar/_index.md" >}}
[*runtimevar.Decoder]: https://godoc.org/gocloud.dev/runtimevar#Decoder
[runtimevar.DecryptDecode]: https://godoc.org/gocloud.dev/runtimevar#DecryptDecode

Other Usage Samples Copy Chapter

• CLI Sample
• Secrets package examples

Supported Services {#services} Copy Chapter

Google Cloud Key Management Service {#gcp} Copy Section

The Go CDK can use keys from Google Cloud Platform's Key Management
Service (GCP KMS) to keep information secret. GCP KMS URLs are
similar to key resource IDs.

secrets.OpenKeeper will use Application Default Credentials; if you have
authenticated via gcloud auth application-default login, it will use those credentials. See
Application Default Credentials to learn about authentication
alternatives, including using environment variables.

{{< goexample "gocloud.dev/secrets/gcpkms.Example_openFromURL" >}}

GCP Constructor {#gcp-ctor}

The gcpkms.OpenKeeper constructor opens a GCP KMS key. You must first
obtain GCP credentials and then create a gRPC connection to GCP KMS.

{{< goexample "gocloud.dev/secrets/gcpkms.ExampleOpenKeeper" >}}

AWS Key Management Service {#aws} Copy Section

The Go CDK can use customer master keys from Amazon Web Service's Key
Management Service (AWS KMS) to keep information secret. AWS KMS
URLs can use the key's ID, alias, or Amazon Resource Name (ARN) to identify
the key. You should specify the region query parameter to ensure your
application connects to the correct region.

It will create an AWS Config based on the AWS SDK V2; see AWS V2 Config to learn more.

{{< goexample "gocloud.dev/secrets/awskms.Example_openFromURL" >}}

AWS Constructor {#aws-ctor}

The awskms.OpenKeeper constructor opens a customer master key. You must
first create an AWS Config with the same region as your key and then
connect to KMS:

{{< goexample "gocloud.dev/secrets/awskms.ExampleOpenKeeper" >}}

Azure KeyVault {#azure} Copy Section

The Go CDK can use keys from Azure KeyVault to keep information secret.
secrets.OpenKeeper will use default credentials from the environment, unless you set the environment variable
AZURE_KEYVAULT_AUTH_VIA_CLI to true, in which case it will use
credentials from the az command line.

Azure KeyVault URLs are based on the Azure Key object identifer:

{{< goexample "gocloud.dev/secrets/azurekeyvault.Example_openFromURL" >}}

Azure Constructor {#azure-ctor}

The azurekeyvault.OpenKeeper constructor opens an Azure KeyVault key.

{{< goexample "gocloud.dev/secrets/azurekeyvault.ExampleOpenKeeper" >}}

HashiCorp Vault {#vault} Copy Section

The Go CDK can use the transit secrets engine in Vault to keep
information secret. Vault URLs only specify the key ID. The Vault server
endpoint and authentication token are specified using the environment
variables VAULT_SERVER_URL and VAULT_SERVER_TOKEN, respectively.

{{< goexample "gocloud.dev/secrets/hashivault.Example_openFromURL" >}}

HashiCorp Vault Constructor {#vault-ctor}

The hashivault.OpenKeeper constructor opens a transit secrets engine
key. You must first connect to your Vault instance.

{{< goexample "gocloud.dev/secrets/hashivault.ExampleOpenKeeper" >}}

Local Secrets {#local} Copy Section

The Go CDK can use local encryption for keeping secrets. Internally, it uses
the NaCl secret box algorithm to perform encryption and authentication.

{{< goexample "gocloud.dev/secrets/localsecrets.Example_openFromURL" >}}

Local Secrets Constructor {#local-ctor}

The localsecrets.NewKeeper constructor takes in its secret material as
a []byte.

{{< goexample "gocloud.dev/secrets/localsecrets.ExampleNewKeeper" >}}

File: internal/website/content/howto/server/_index.md Copy Chapter

title: "Server" date: 2019-06-21T10:36:43-07:00 showInSidenav: true toc: true Copy Chapter

The Go CDK's server package provides a pre-configured HTTP server with
diagnostic hooks for request logging, health checks, and trace exporting via
OpenCensus. This guide will show you how to start up and shut down the server,
as well as how to work with the request logging and health checks.

Starting up the server Copy Chapter

The Go CDK Server constructor takes an http.Handler and an Options struct.
The simplest way to start the server is to use http.DefaultServeMux and
pass nil for the options.

{{< goexample src="gocloud.dev/server.ExampleServer" >}}

Adding a request logger Copy Section

You can use the server.Options struct to specify a request logger.

The example is shown with the Go CDK requestlog package's NCSALogger.
To get logs in the Stackdriver JSON format, use NewStackdriverLogger in place
of NewNCSALogger.

{{< goexample src="gocloud.dev/server.ExampleServer_withRequestLogger" >}}

Adding health checks Copy Section

The Go CDK server package affords a hook for you to define health checks for
your application and see the results at /healthz/readiness. The server also
runs an endpoint at /healthz/liveness, which is a conventional name for a
liveness check and is where Kubernetes, if you are using it, will look.

Health checks are an important part of application monitoring, and readiness
checks are subtly different than liveness checks. The liveness check will return
200 OK if the server can serve requests. But because each application may have
a different definition of what it means to be "healthy" (perhaps your
application has a dependency on a back end service), you will need to define a
concrete type to implement the health.Checker interface and define a
CheckHealth method specific to your application for readiness checks.

// customHealthCheck is an example health check. It implements the
// health.Checker interface and reports the server is healthy when the healthy
// field is set to true.
type customHealthCheck struct {
	mu      sync.RWMutex
	healthy bool
}

// customHealthCheck implements the health.Checker interface because it has a
// CheckHealth method.
func (h *customHealthCheck) CheckHealth() error {
	h.mu.RLock()
	defer h.mu.RUnlock()
	if !h.healthy {
		return errors.New("not ready yet!")
	}
	return nil
}

{{< goexample src="gocloud.dev/server.ExampleServer_withHealthChecks" >}}

Other Usage Samples Copy Chapter

• Minimal server sample
• Guestbook sample

File: internal/website/content/howto/sql/_index.md Copy Chapter

title: "MySQL/PostgreSQL" date: 2019-06-21T09:26:56-07:00 showInSidenav: true toc: true Copy Chapter

Connecting to Cloud providers' hosted database services requires additional
steps to ensure the security of the connection. For example, each of GCP,
AWS and Azure require the use of custom certificate authorities to be
configured in the client. GCP requires a custom proxy with authentication
credentials. The Go CDK makes opening these connections easier while still
using the standard *sql.DB type.

Local or On-Premise {#local} Copy Chapter

The Go CDK uses the same [URL opener pattern][] as seen in other Go CDK APIs. It
differs from the standard library's sql.Open in that it automatically
instruments the connection with OpenCensus metrics.

The portable function for MySQL is mysql.Open:

{{< goexample "gocloud.dev/mysql.ExampleOpen" >}}

And the portable function for PostgreSQL is postgres.Open:

{{< goexample "gocloud.dev/postgres.ExampleOpen" >}}

[URL opener pattern]: {{< ref "/concepts/urls.md" >}}

GCP {#gcp} Copy Chapter

Users of GCP Cloud SQL for MySQL should import the gocloud.dev/mysql/gcpmysql package:

{{< goexample "gocloud.dev/mysql/gcpmysql.Example" >}}

Users of GCP Cloud SQL for PostgreSQL should import the gocloud.dev/postgres/gcppostgres package:

{{< goexample "gocloud.dev/postgres/gcppostgres.Example" >}}

AWS {#aws} Copy Chapter

Users of AWS RDS for MySQL should import the gocloud.dev/mysql/awsmysql package:

{{< goexample "gocloud.dev/mysql/awsmysql.Example" >}}

Users of AWS RDS for PostgreSQL should import the gocloud.dev/postgres/awspostgres package:

{{< goexample "gocloud.dev/postgres/awspostgres.Example" >}}

Azure {#azure} Copy Chapter

Users of Azure Database for MySQL should import the gocloud.dev/mysql/azuremysql package:

{{< goexample "gocloud.dev/mysql/azuremysql.Example" >}}

Other Usage Samples Copy Chapter

• Guestbook sample

File: internal/website/content/internal/batcher/_index.md Copy Chapter

title: gocloud.dev/internal/batcher type: pkg Copy Chapter

File: internal/website/content/internal/escape/_index.md Copy Chapter

title: gocloud.dev/internal/escape type: pkg Copy Chapter

File: internal/website/content/internal/gcerr/_index.md Copy Chapter

title: gocloud.dev/internal/gcerr type: pkg Copy Chapter

File: internal/website/content/internal/oc/_index.md Copy Chapter

title: gocloud.dev/internal/oc type: pkg Copy Chapter

File: internal/website/content/internal/openurl/_index.md Copy Chapter

title: gocloud.dev/internal/openurl type: pkg Copy Chapter

File: internal/website/content/internal/otel/_index.md Copy Chapter

title: gocloud.dev/internal/otel type: pkg Copy Chapter

Package otel provides OpenTelemetry integration for Go Cloud Development Kit, supporting tracing, metrics, and logging. It serves as the primary instrumentation framework, replacing the deprecated OpenCensus implementation.

File: internal/website/content/internal/releasehelper/_index.md Copy Chapter

title: gocloud.dev/internal/releasehelper type: pkg Copy Chapter

File: internal/website/content/internal/retry/_index.md Copy Chapter

title: gocloud.dev/internal/retry type: pkg Copy Chapter

File: internal/website/content/internal/testing/_index.md Copy Chapter

title: gocloud.dev/internal/testing type: pkg Copy Chapter

File: internal/website/content/internal/testing/cmdtest/_index.md Copy Chapter

title: gocloud.dev/internal/testing/cmdtest type: pkg Copy Chapter

File: internal/website/content/internal/testing/octest/_index.md Copy Chapter

title: gocloud.dev/internal/testing/octest type: pkg Copy Chapter

File: internal/website/content/internal/testing/oteltest/_index.md Copy Chapter

title: gocloud.dev/internal/testing/oteltest type: pkg Copy Chapter

File: internal/website/content/internal/testing/setup/_index.md Copy Chapter

title: gocloud.dev/internal/testing/setup type: pkg Copy Chapter

File: internal/website/content/internal/testing/terraform/_index.md Copy Chapter

title: gocloud.dev/internal/testing/terraform type: pkg Copy Chapter

File: internal/website/content/internal/testing/test-summary/_index.md Copy Chapter

title: gocloud.dev/internal/testing/test-summary type: pkg Copy Chapter

File: internal/website/content/internal/trace/_index.md Copy Chapter

title: gocloud.dev/internal/trace type: pkg Copy Chapter

File: internal/website/content/internal/useragent/_index.md Copy Chapter

title: gocloud.dev/internal/useragent type: pkg Copy Chapter

File: internal/website/content/internal/website/gatherexamples/_index.md Copy Chapter

title: gocloud.dev/internal/website/gatherexamples type: pkg Copy Chapter

File: internal/website/content/mysql/_index.md Copy Chapter

title: gocloud.dev/mysql type: pkg Copy Chapter

File: internal/website/content/mysql/awsmysql/_index.md Copy Chapter

title: gocloud.dev/mysql/awsmysql type: pkg Copy Chapter

File: internal/website/content/mysql/azuremysql/_index.md Copy Chapter

title: gocloud.dev/mysql/azuremysql type: pkg Copy Chapter

File: internal/website/content/mysql/cloudmysql/_index.md Copy Chapter

title: gocloud.dev/mysql/cloudmysql type: pkg Copy Chapter

File: internal/website/content/mysql/gcpmysql/_index.md Copy Chapter

title: gocloud.dev/mysql/gcpmysql type: pkg Copy Chapter

File: internal/website/content/mysql/rdsmysql/_index.md Copy Chapter

title: gocloud.dev/mysql/rdsmysql type: pkg Copy Chapter

File: internal/website/content/postgres/_index.md Copy Chapter

title: gocloud.dev/postgres type: pkg Copy Chapter

File: internal/website/content/postgres/awspostgres/_index.md Copy Chapter

title: gocloud.dev/postgres/awspostgres type: pkg Copy Chapter

File: internal/website/content/postgres/cloudpostgres/_index.md Copy Chapter

title: gocloud.dev/postgres/cloudpostgres type: pkg Copy Chapter

File: internal/website/content/postgres/gcppostgres/_index.md Copy Chapter

title: gocloud.dev/postgres/gcppostgres type: pkg Copy Chapter

File: internal/website/content/postgres/rdspostgres/_index.md Copy Chapter

title: gocloud.dev/postgres/rdspostgres type: pkg Copy Chapter

File: internal/website/content/pubsub/_index.md Copy Chapter

title: gocloud.dev/pubsub type: pkg Copy Chapter

File: internal/website/content/pubsub/awssnssqs/_index.md Copy Chapter

title: gocloud.dev/pubsub/awssnssqs type: pkg Copy Chapter

File: internal/website/content/pubsub/azurepubsub/_index.md Copy Chapter

title: gocloud.dev/pubsub/azurepubsub type: pkg Copy Chapter

File: internal/website/content/pubsub/azuresb/_index.md Copy Chapter

title: gocloud.dev/pubsub/azuresb type: pkg Copy Chapter

File: internal/website/content/pubsub/batcher/_index.md Copy Chapter

title: gocloud.dev/internal/batcher type: pkg Copy Chapter

File: internal/website/content/pubsub/driver/_index.md Copy Chapter

title: gocloud.dev/pubsub/driver type: pkg Copy Chapter

File: internal/website/content/pubsub/drivertest/_index.md Copy Chapter

title: gocloud.dev/pubsub/drivertest type: pkg Copy Chapter

File: internal/website/content/pubsub/gcppubsub/_index.md Copy Chapter

title: gocloud.dev/pubsub/gcppubsub type: pkg Copy Chapter

File: internal/website/content/pubsub/gcppubsubv2/_index.md Copy Chapter

title: gocloud.dev/pubsub/gcppubsubv2 type: pkg Copy Chapter

File: internal/website/content/pubsub/kafkapubsub/_index.md Copy Chapter

title: gocloud.dev/pubsub/kafkapubsub type: pkg Copy Chapter

File: internal/website/content/pubsub/mempubsub/_index.md Copy Chapter

title: gocloud.dev/pubsub/mempubsub type: pkg Copy Chapter

File: internal/website/content/pubsub/natspubsub/_index.md Copy Chapter

title: gocloud.dev/pubsub/natspubsub type: pkg Copy Chapter

File: internal/website/content/pubsub/rabbitpubsub/_index.md Copy Chapter

title: gocloud.dev/pubsub/rabbitpubsub type: pkg Copy Chapter

File: internal/website/content/requestlog/_index.md Copy Chapter

title: gocloud.dev/requestlog type: pkg Copy Chapter

File: internal/website/content/runtimevar/_index.md Copy Chapter

title: gocloud.dev/runtimevar type: pkg Copy Chapter

File: internal/website/content/runtimevar/awsparamstore/_index.md Copy Chapter

title: gocloud.dev/runtimevar/awsparamstore type: pkg Copy Chapter

File: internal/website/content/runtimevar/awssecretsmanager/_index.md Copy Chapter

title: gocloud.dev/runtimevar/awssecretsmanager type: pkg Copy Chapter

File: internal/website/content/runtimevar/blobvar/_index.md Copy Chapter

title: gocloud.dev/runtimevar/blobvar type: pkg Copy Chapter

File: internal/website/content/runtimevar/constantvar/_index.md Copy Chapter

title: gocloud.dev/runtimevar/constantvar type: pkg Copy Chapter

File: internal/website/content/runtimevar/driver/_index.md Copy Chapter

title: gocloud.dev/runtimevar/driver type: pkg Copy Chapter

File: internal/website/content/runtimevar/drivertest/_index.md Copy Chapter

title: gocloud.dev/runtimevar/drivertest type: pkg Copy Chapter

File: internal/website/content/runtimevar/etcdvar/_index.md Copy Chapter

title: gocloud.dev/runtimevar/etcdvar type: pkg Copy Chapter

File: internal/website/content/runtimevar/filevar/_index.md Copy Chapter

title: gocloud.dev/runtimevar/filevar type: pkg Copy Chapter

File: internal/website/content/runtimevar/filevar/_demo/_index.md Copy Chapter

title: gocloud.dev/runtimevar/filevar/_demo type: pkg Copy Chapter

File: internal/website/content/runtimevar/gcpruntimeconfig/_index.md Copy Chapter

title: gocloud.dev/runtimevar/gcpruntimeconfig type: pkg Copy Chapter

File: internal/website/content/runtimevar/gcpsecretmanager/_index.md Copy Chapter

title: gocloud.dev/runtimevar/gcpsecretmanager type: pkg Copy Chapter

File: internal/website/content/runtimevar/hashivault/_index.md Copy Chapter

title: gocloud.dev/runtimevar/hashivault type: pkg Copy Chapter

File: internal/website/content/runtimevar/httpvar/_index.md Copy Chapter

title: gocloud.dev/runtimevar/httpvar type: pkg Copy Chapter

File: internal/website/content/samples/appengine/_index.md Copy Chapter

title: gocloud.dev/samples/appengine type: pkg Copy Chapter

File: internal/website/content/samples/gocdk-blob/_index.md Copy Chapter

title: gocloud.dev/samples/gocdk-blob type: pkg Copy Chapter

File: internal/website/content/samples/gocdk-docstore/_index.md Copy Chapter

title: "Gocdk Docstore" date: 2019-06-11T14:33:50-07:00 showInSidenav: false # only for sections (any level) pagesInSidenav: false # only for top-level sections weight: 0 Copy Chapter

File: internal/website/content/samples/gocdk-pubsub/_index.md Copy Chapter

title: gocloud.dev/samples/gocdk-pubsub type: pkg Copy Chapter

File: internal/website/content/samples/gocdk-runtimevar/_index.md Copy Chapter

title: gocloud.dev/samples/gocdk-runtimevar type: pkg Copy Chapter

File: internal/website/content/samples/gocdk-secrets/_index.md Copy Chapter

title: gocloud.dev/samples/gocdk-secrets type: pkg Copy Chapter

File: internal/website/content/samples/guestbook/_index.md Copy Chapter

title: gocloud.dev/samples/guestbook type: pkg Copy Chapter

File: internal/website/content/samples/guestbook/aws/provision_db/_index.md Copy Chapter

title: gocloud.dev/samples/guestbook/aws/provision_db type: pkg Copy Chapter

File: internal/website/content/samples/guestbook/gcp/deploy/_index.md Copy Chapter

title: gocloud.dev/samples/guestbook/gcp/deploy type: pkg Copy Chapter

File: internal/website/content/samples/guestbook/gcp/provision_db/_index.md Copy Chapter

title: gocloud.dev/samples/guestbook/gcp/provision_db type: pkg Copy Chapter

File: internal/website/content/samples/guestbook/localdb/_index.md Copy Chapter

title: gocloud.dev/samples/guestbook/localdb type: pkg Copy Chapter

File: internal/website/content/samples/order/_index.md Copy Chapter

title: gocloud.dev/samples/order type: pkg Copy Chapter

File: internal/website/content/samples/server/_index.md Copy Chapter

title: gocloud.dev/samples/server type: pkg Copy Chapter

File: internal/website/content/samples/tutorial/_index.md Copy Chapter

title: gocloud.dev/samples/tutorial type: pkg Copy Chapter

File: internal/website/content/secrets/_index.md Copy Chapter

title: gocloud.dev/secrets type: pkg Copy Chapter

File: internal/website/content/secrets/awskms/_index.md Copy Chapter

title: gocloud.dev/secrets/awskms type: pkg Copy Chapter

File: internal/website/content/secrets/azurekeyvault/_index.md Copy Chapter

title: gocloud.dev/secrets/azurekeyvault type: pkg Copy Chapter

File: internal/website/content/secrets/driver/_index.md Copy Chapter

title: gocloud.dev/secrets/driver type: pkg Copy Chapter

File: internal/website/content/secrets/drivertest/_index.md Copy Chapter

title: gocloud.dev/secrets/drivertest type: pkg Copy Chapter

File: internal/website/content/secrets/gcpkms/_index.md Copy Chapter

title: gocloud.dev/secrets/gcpkms type: pkg Copy Chapter

File: internal/website/content/secrets/hashivault/_index.md Copy Chapter

title: gocloud.dev/secrets/hashivault type: pkg Copy Chapter

File: internal/website/content/secrets/localsecrets/_index.md Copy Chapter

title: gocloud.dev/secrets/localsecrets type: pkg Copy Chapter

File: internal/website/content/secrets/vault/_index.md Copy Chapter

title: gocloud.dev/secrets/vault type: pkg Copy Chapter

File: internal/website/content/server/_index.md Copy Chapter

title: gocloud.dev/server type: pkg Copy Chapter

File: internal/website/content/server/driver/_index.md Copy Chapter

title: gocloud.dev/server/driver type: pkg Copy Chapter

File: internal/website/content/server/health/_index.md Copy Chapter

title: gocloud.dev/server/health type: pkg Copy Chapter

File: internal/website/content/server/health/sqlhealth/_index.md Copy Chapter

title: gocloud.dev/server/health/sqlhealth type: pkg Copy Chapter

File: internal/website/content/server/requestlog/_index.md Copy Chapter

title: gocloud.dev/server/requestlog type: pkg Copy Chapter

File: internal/website/content/server/sdserver/_index.md Copy Chapter

title: gocloud.dev/server/sdserver type: pkg Copy Chapter

File: internal/website/content/server/xrayserver/_index.md Copy Chapter

title: gocloud.dev/server/xrayserver type: pkg Copy Chapter

File: internal/website/content/tests/aws/app/_index.md Copy Chapter

title: gocloud.dev/tests/aws/app type: pkg Copy Chapter

File: internal/website/content/tests/gcp/app/_index.md Copy Chapter

title: gocloud.dev/tests/gcp/app type: pkg Copy Chapter

File: internal/website/content/tests/internal/testutil/_index.md Copy Chapter

title: gocloud.dev/tests/internal/testutil type: pkg Copy Chapter

File: internal/website/content/tutorials/_index.md Copy Chapter

title: "Tutorials" date: 2019-03-19T18:45:15-07:00 showInSidenav: true pagesInSidenav: true weight: 1 Copy Chapter

File: internal/website/content/tutorials/cli-uploader.md Copy Chapter

title: "Tutorial: Command-Line Uploader" linkTitle: "Command-Line Uploader" date: 2019-03-19T18:45:53-07:00 weight: 1 Copy Chapter

This quickstart will build a command line application called upload that
uploads files to blob storage on GCP, AWS, and Azure. Blob storage stores binary
data under a string key, and is one of the most frequently used cloud
services.

When we're done, our command line application will work like this:

# uploads gopher.png to GCS
$ ./upload gs://go-cloud-bucket gopher.png

# uploads gopher.png to S3
$ ./upload s3://go-cloud-bucket gopher.png

# uploads gopher.png to Azure
$ ./upload azblob://go-cloud-bucket gopher.png

(You can download the finished tutorial from GitHub).

Setup Copy Chapter

We start with a skeleton for our program to read from command-line
arguments to configure the bucket URL.

// Command upload saves files to blob storage on GCP, AWS, and Azure.
package main

import (
    "log"
    "os"
)

func main() {
    // Define our input.
    if len(os.Args) != 3 {
        log.Fatal("usage: upload BUCKET_URL FILE")
    }
    bucketURL := os.Args[1]
    file := os.Args[2]
    _, _ = bucketURL, file
}

Now that we have a basic skeleton in place, let's start filling in the pieces.

Connecting to the bucket Copy Chapter

The easiest way to open a portable bucket API is with blob.OpenBucket.

package main

import (
    // previous imports omitted

    "gocloud.dev/blob"
)

func main() {
    // previous code omitted

    // Open a connection to the bucket.
    b, err := blob.OpenBucket(context.Background(), bucketURL)
    if err != nil {
        log.Fatalf("Failed to setup bucket: %s", err)
    }
    defer b.Close()
}

This is all we need in the main function to connect to the bucket. However,
as written, this function call will always fail: the Go CDK does not link in any
cloud-specific implementations of blob.OpenBucket unless you specifically
depend on them. This ensures you're only depending on the code you need.
To link in implementations, use blank imports:

package main

import (
    // previous imports omitted

    // Import the blob packages we want to be able to open.
    _ "gocloud.dev/blob/azureblob"
    _ "gocloud.dev/blob/gcsblob"
    _ "gocloud.dev/blob/s3blob"
)

func main() {
    // ...
}

With the setup done, we're ready to use the bucket connection. Note, as a design
principle of the Go CDK, blob.Bucket does not support creating a bucket and
instead focuses solely on interacting with it. This separates the concerns of
provisioning infrastructure and using infrastructure.

Reading the file Copy Chapter

We need to read our file into a slice of bytes before uploading it. The process
is the usual one:

package main

import (
    // previous imports omitted
    "os"
    "io/ioutil"
)

func main() {
    // ... previous code omitted

    // Prepare the file for upload.
    data, err := os.ReadFile(file)
    if err != nil {
        log.Fatalf("Failed to read file: %s", err)
    }
}

Writing the file to the bucket Copy Chapter

Now, we have data, our file in a slice of bytes. Let's get to the fun part and
write those bytes to the bucket!

package main

// No new imports.

func main() {
    // ...

    w, err := b.NewWriter(ctx, file, nil)
    if err != nil {
        log.Fatalf("Failed to obtain writer: %s", err)
    }
    _, err = w.Write(data)
    if err != nil {
        log.Fatalf("Failed to write to bucket: %s", err)
    }
    if err := w.Close(); err != nil {
        log.Fatalf("Failed to close: %s", err)
    }
}

First, we create a writer based on the bucket connection. In addition to a
context.Context type, the method takes the key under which the data will be
stored and the mime type of the data.

The call to NewWriter creates a blob.Writer, which implements io.Writer.
With the writer, we call Write passing in the data. In response, we get the
number of bytes written and any error that might have occurred.

Finally, we close the writer with Close and check the error.

Alternatively, we could have used the shortcut b.WriteAll(ctx, file, data, nil).

Uploading an image Copy Chapter

That's it! Let's try it out. As setup, we will need to create a
GCS bucket, an S3 bucket, and an
Azure container. In the code above, I called that bucket
go-cloud-bucket, but you can change that to match whatever your bucket is
called. Of course, you are free to try the code on any subset of Cloud
providers.

• For GCP, you will need to login with
  gcloud. If you do not want to
  install gcloud, see
  here for
  alternatives.
• For AWS, you will need an access key ID and a secret access key. See
  here
  for details. You then need to set the AWS_REGION environment variable to
  the region your bucket is in.
• For Azure, you will need to add your storage account name and key as
  environment variables (AZURE_STORAGE_ACCOUNT and
  AZURE_STORAGE_KEY, respectively). See
  here
  for details.

With our buckets created and our credentials set up, we'll build the program
first:

$ go build -o upload

Then, we will send gopher.png (in the same directory as this README) to GCS:

$ ./upload gs://go-cloud-bucket gopher.png

Then, we send that same gopher to S3:

$ ./upload s3://go-cloud-bucket?region=us-west-1 gopher.png

Finally, we send that same gopher to Azure:

$ ./upload azblob://go-cloud-bucket gopher.png

If we check the buckets, we should see our gopher in each of them! We're done!

Wrapping Up Copy Chapter

In conclusion, we have a program that can seamlessly switch between multiple
Cloud storage providers using just one code path. You can see the finished
tutorial on GitHub.

We hope this example demonstrates how having one type for multiple clouds is a
huge win for simplicity and maintainability. By writing an application using a
generic interface like *blob.Bucket, we retain the option of using
infrastructure in whichever cloud that best fits our needs all without having to
worry about a rewrite. If you want to learn more, you can read about
[Structuring Portable Code][].

If you want to see how to deploy a Go CDK application, see [other tutorials][].
If you want to see how to use Go CDK APIs in your application, see the
[how-to guides][].

[how-to guides]: {{< ref "/howto/_index.md" >}}
[other tutorials]: {{< ref "/tutorials/_index.md" >}}
[Structuring Portable Code]: {{< ref "/concepts/structure/index.md" >}}

File: internal/website/content/tutorials/guestbook.md Copy Chapter

title: "Tutorial: Guestbook Application" linkTitle: "Guestbook" date: 2019-07-10T16:58:00-07:00 weight: 2 toc: true Copy Chapter

In this tutorial, we will deploy an existing Go CDK application called Guestbook
locally, then on Google Cloud Platform (GCP), Amazon Web Services (AWS), and
Azure.

Guestbook is a sample application that records visitors' messages and displays a
cloud banner with an administrative message. The main business logic is written
in a cloud-agnostic manner using MySQL, the generic blob API, and the generic
runtimevar API. All platform-specific code is set up by
Wire.

Prerequisites Copy Chapter

You will need to install the following software for this tutorial:

• Git
• Go
• Wire
• Docker Desktop
• Terraform

Cloning the Sample Copy Section

Before you can do anything else, you need to clone the Go CDK repository. The
repository contains the Guestbook sample.

git clone https://github.com/google/go-cloud.git
cd go-cloud/samples/guestbook

Building Copy Chapter

Run the following in the samples/guestbook directory:

go generate && go build

Running Locally Copy Chapter

You will need to run a local MySQL database server and create a local message of
the day. localdb/main.go runs the local MySQL database server using Docker:

go get ./localdb/... # Get package dependencies.
go run localdb/main.go

In another terminal, run the guestbook application:

# Set a local Message of the Day.
echo 'Hello, World!' > motd.txt

# Run the server.
# For blob, it uses fileblob, pointing at the local directory ./blobs.
# For runtimevar, it uses filevar, pointing at the local file ./motd.txt.
#   You can update the ./motd.txt while the server is running, refresh
#   the page, and see it change.
./guestbook -env=local -bucket=blobs -motd_var=motd.txt

Your server is now running on http://localhost:8080/.

You can stop the MySQL database server with Ctrl-\. MySQL ignores Ctrl-C
(SIGINT).

Running on Google Cloud Platform (GCP) Copy Chapter

If you want to run this sample on GCP, you need to create a project, download
the gcloud CLI, install kubectl and log in.

# Install kubectl.
gcloud components install kubectl

# Opens a browser to log you into GCP.
gcloud auth login

Provision resources Copy Section

You can now use Terraform to set up your project.

gcloud auth application-default login
cd gcp
terraform init

# Terraform will prompt you for your GCP project ID, desired region,
# and desired zone.
terraform apply

Deploy the application Copy Section

This sample provides a script for building the Guestbook binary and deploying it
to the Kubernetes cluster created by Terraform.

go run deploy/main.go

The deploy script will display the URL of your running service.

Cleanup Copy Section

To clean up the created resources, run terraform destroy inside the gcp
directory using the same variables you entered during terraform apply.

Running on Amazon Web Services (AWS) Copy Chapter

If you want to run this sample on AWS, you need to set up an account, download
the AWS command line interface, and log in. There's
help here if you need it.

aws configure

Agree to the Debian Terms and Conditions Copy Section

You have to agree to the Debian Terms and Conditions in order to provision
the resources. Click through the "Continue to Subscribe" button at the top, then
log in to your AWS account and subscribe to Debian.

SSH Key Copy Section

You will also need an SSH key to SSH into the EC2 instance. If you don't already
have one, you can follow this guide from GitHub. Follow the
instructions for "Adding your key to the ssh-agent" if you want the key to
persist across terminal sessions.

ssh-add

Provision resources Copy Section

You can now use Terraform to set up your project. This will create an EC2
instance you can SSH to and run your binary.

# Build for deploying on the AWS Linux VM.
GOOS=linux GOARCH=amd64 go build

# Enter AWS directory from samples/guestbook.
cd aws
terraform init

# Provisioning can take up to 10 minutes.
# Keep track of the output of this command as it is needed later.
# You can replace us-west-1 with whatever region you want.
terraform apply -var region=us-west-1 -var ssh_public_key="$(cat ~/.ssh/id_rsa.pub)"

Deploy the application Copy Section

You now need to connect to the new remote server to execute the guestbook
binary. The final output of terraform apply lists the variables guestbook
requires as arguments. Here's an example, with actual strings replaced with
[redacted]:

# Output from "terraform apply" command....
<snip>

Outputs:

bucket = [redacted]
database_host = [redacted]
database_root_password = <sensitive>
instance_host = [redacted]
paramstore_var = /guestbook/motd
region = us-west-1

# Print out the database root password, since we'll need it below
# Terraform hides it by default in the Outputs above.
localhost$ terraform output database_root_password
[redacted]

# SSH into the EC2 instance.
localhost$ ssh "admin@$( terraform output instance_host )"

# Fill in each command-line argument with the values from the
# Terraform output above.
server$ AWS_REGION=<your region> ./guestbook \
  -env=aws \
  -bucket=<your bucket> \
  -db_host=<your database_host> \
  -db_user=root \
  -db_password=<your database_root_password> \
  -motd_var=/guestbook/motd

You can now visit the server at http://<your instance_host>:8080/.

Cleanup Copy Section

To clean up the created resources, run terraform destroy inside the aws
directory using the same variables you entered during terraform apply.

Running on Azure Copy Chapter

The Go CDK doesn't have support for SQL on Azure yet (#1305), so we'll run
MySQL and the binary locally. This tutorial will show how to use Azure storage
for the MOTD and Gopher logo.

You first need to set up an Azure account and install the Azure CLI. Then,
use the az CLI to log in.

az login

Provision resources Copy Section

We'll use Terraform to set up your project.

# Enter the Azure directory from samples/guestbook.
cd azure
terraform init

# Provisioning can take up to 10 minutes.
# Keep track of the output of this command as it is needed later.
terraform apply -var location="West US"

<snip>
Outputs:

access_key = [redacted]
storage_account = [redacated]
storage_container = [redacted]

Deploy the application Copy Section

You will need to run a local MySQL database server, similar to what we did for
running locally earlier. Open a new terminal window, and run:

cd .. # back up to samples/guestbook
go get ./localdb/... # Get package dependencies.
go run localdb/main.go

In the original terminal, add your Azure credentials to the environment and run
the guestbook application:

# You should be in the "samples/guestbook/azure" directory.

# Enter the storage_account from the Terraform output earlier.
export AZURE_STORAGE_ACCOUNT=<your storage_account>
# Enter the access_key from the Terraform output earlier.
export AZURE_STORAGE_KEY=<your access_key>

# Run the binary.
# Fill in the -bucket command-line argument with the value from the Terraform
# output.
#
./guestbook -env=azure -bucket=<your storage_container> -motd_var=motd

Your server is now running on http://localhost:8080/.

You can stop the MySQL database server with Ctrl-\. MySQL ignores Ctrl-C
(SIGINT).

Cleanup Copy Section

To clean up the created resources, run terraform destroy inside the azure
directory using the same variables you entered during terraform apply.

Gophers Copy Chapter

The Go gopher was designed by Renee French and used under the
Creative Commons 3.0 Attributions
license.

File: internal/website/content/tutorials/order.md Copy Chapter

title: "Order Processor" date: 2019-07-09T10:22:39-04:00 weight: 5 toc: true Copy Chapter

In this tutorial, we will run a Go CDK application called Order Processor
on a local machine.

Order Processor is a sample application that lets users place orders to convert
images to PNG format, and to view the results. The main business logic is
written in a cloud-agnostic manner using the generic APIs for blob, pubsub and
docstore.

The Order Processor application has two parts: a web frontend, and an
image-processing backend called a processor. They communicate over a pubsub
topic, store order information in a docstore collection, and store image files
in a blob bucket.

Prerequisites Copy Chapter

You will need to install the following software for this tutorial:

• Git
• Go

Then you need to clone the Go CDK repository. The
repository contains the Order Processor sample.

git clone https://github.com/google/go-cloud.git
cd go-cloud/samples/order

Building Copy Chapter

Run the following in the samples/order directory:

go build

Running Locally Copy Chapter

If you run order with no arguments, both the frontend and the processor will
run together in the same process.

./order

The frontend is now running on http://localhost:10538.

Visit the home page in your browser and click "Convert an Image".

Enter an email address (it need not be real) and select any image file from your
computer. Then click Submit.

Now visit the order list page by returning to the home page and clicking "List
Conversions". It may take a few seconds to process the order (thanks to an
artificial delay in the processor), so refresh the page until you see your order
in the list.

Then click on the output image link to see the converted image in your browser.

Running on a Cloud Provider Copy Chapter

To run the Order Processor application on a cloud provider like Google Cloud
Platform, Amazon AWS or Microsoft Azure, you will have to provision
some resources:

• A storage bucket, to hold the image files. (See the
  blob How-To Guide.)
• A Pub/Sub topic and subscription, for requests from the frontend to the
  processor. (See the pubsub How-To Guide.)
• A document store collection to store order metadata. (See the docstore How-To
  Guide.)

Then launch the order program with flags that provide the URLs to your
resources. Run order -help to see the list of flags.