go-plugin Documentation

File: README.md Copy Chapter

Go Plugin System over RPC

go-plugin is a Go (golang) plugin system over RPC. It is the plugin system
that has been in use by HashiCorp tooling for over 4 years. While initially
created for Packer, it is additionally in use by
Terraform, Nomad,
Vault,
Boundary,
and Waypoint.

While the plugin system is over RPC, it is currently only designed to work
over a local [reliable] network. Plugins over a real network are not supported
and will lead to unexpected behavior.

This plugin system has been used on millions of machines across many different
projects and has proven to be battle hardened and ready for production use.

Features Copy Chapter

The HashiCorp plugin system supports a number of features:

Plugins are Go interface implementations. This makes writing and consuming
plugins feel very natural. To a plugin author: you just implement an
interface as if it were going to run in the same process. For a plugin user:
you just use and call functions on an interface as if it were in the same
process. This plugin system handles the communication in between.

Cross-language support. Plugins can be written (and consumed) by
almost every major language. This library supports serving plugins via
gRPC. gRPC-based plugins enable plugins to be written
in any language.

Complex arguments and return values are supported. This library
provides APIs for handling complex arguments and return values such
as interfaces, io.Reader/Writer, etc. We do this by giving you a library
(MuxBroker) for creating new connections between the client/server to
serve additional interfaces or transfer raw data.

Bidirectional communication. Because the plugin system supports
complex arguments, the host process can send it interface implementations
and the plugin can call back into the host process.

Built-in Logging. Any plugins that use the log standard library
will have log data automatically sent to the host process. The host
process will mirror this output prefixed with the path to the plugin
binary. This makes debugging with plugins simple. If the host system
uses hclog then the log data
will be structured. If the plugin also uses hclog, logs from the plugin
will be sent to the host hclog and be structured.

Protocol Versioning. A very basic "protocol version" is supported that
can be incremented to invalidate any previous plugins. This is useful when
interface signatures are changing, protocol level changes are necessary,
etc. When a protocol version is incompatible, a human friendly error
message is shown to the end user.

Stdout/Stderr Syncing. While plugins are subprocesses, they can continue
to use stdout/stderr as usual and the output will get mirrored back to
the host process. The host process can control what io.Writer these
streams go to to prevent this from happening.

TTY Preservation. Plugin subprocesses are connected to the identical
stdin file descriptor as the host process, allowing software that requires
a TTY to work. For example, a plugin can execute ssh and even though there
are multiple subprocesses and RPC happening, it will look and act perfectly
to the end user.

Host upgrade while a plugin is running. Plugins can be "reattached"
so that the host process can be upgraded while the plugin is still running.
This requires the host/plugin to know this is possible and daemonize
properly. NewClient takes a ReattachConfig to determine if and how to
reattach.

Cryptographically Secure Plugins. Plugins can be verified with an expected
checksum and RPC communications can be configured to use TLS. The host process
must be properly secured to protect this configuration.

Architecture Copy Chapter

The HashiCorp plugin system works by launching subprocesses and communicating
over RPC (using standard net/rpc or gRPC). A single
connection is made between any plugin and the host process. For net/rpc-based
plugins, we use a connection multiplexing
library to multiplex any other connections on top. For gRPC-based plugins,
the HTTP2 protocol handles multiplexing.

This architecture has a number of benefits:

• Plugins can't crash your host process: A panic in a plugin doesn't
  panic the plugin user.

• Plugins are very easy to write: just write a Go application and go build.
  Or use any other language to write a gRPC server with a tiny amount of
  boilerplate to support go-plugin.

• Plugins are very easy to install: just put the binary in a location where
  the host will find it (depends on the host but this library also provides
  helpers), and the plugin host handles the rest.

• Plugins can be relatively secure: The plugin only has access to the
  interfaces and args given to it, not to the entire memory space of the
  process. Additionally, go-plugin can communicate with the plugin over
  TLS.

Usage Copy Chapter

To use the plugin system, you must take the following steps. These are
high-level steps that must be done. Examples are available in the
examples/ directory.

1. Choose the interface(s) you want to expose for plugins.

2. For each interface, implement an implementation of that interface
   that communicates over a net/rpc connection or over a
   gRPC connection or both. You'll have to implement
   both a client and server implementation.

3. Create a Plugin implementation that knows how to create the RPC
   client/server for a given plugin type.

4. Plugin authors call plugin.Serve to serve a plugin from the
   main function.

5. Plugin users use plugin.Client to launch a subprocess and request
   an interface implementation over RPC.

That's it! In practice, step 2 is the most tedious and time consuming step.
Even so, it isn't very difficult and you can see examples in the examples/
directory as well as throughout our various open source projects.

For complete API documentation, see GoDoc.

Roadmap Copy Chapter

Our plugin system is constantly evolving. As we use the plugin system for
new projects or for new features in existing projects, we constantly find
improvements we can make.

At this point in time, the roadmap for the plugin system is:

Semantic Versioning. Plugins will be able to implement a semantic version.
This plugin system will give host processes a system for constraining
versions. This is in addition to the protocol versioning already present
which is more for larger underlying changes.

What About Shared Libraries? Copy Chapter

When we started using plugins (late 2012, early 2013), plugins over RPC
were the only option since Go didn't support dynamic library loading. Today,
Go supports the plugin standard library with
a number of limitations. Since 2012, our plugin system has stabilized
from tens of millions of users using it, and has many benefits we've come to
value greatly.

For example, we use this plugin system in
Vault where dynamic library loading is
not acceptable for security reasons. That is an extreme
example, but we believe our library system has more upsides than downsides
over dynamic library loading and since we've had it built and tested for years,
we'll continue to use it.

Shared libraries have one major advantage over our system which is much
higher performance. In real world scenarios across our various tools,
we've never required any more performance out of our plugin system and it
has seen very high throughput, so this isn't a concern for us at the moment.

File: docs/extensive-go-plugin-tutorial.md Copy Chapter

Intro

If you don't know what go-plugin is, don't worry, here is a small introduction on the subject matter:

Back in the old days when Go didn't have the plugin package, HashiCorp was looking for a way to use plugins.

Mitchell had this brilliant idea of using RPC over the local network to serve a local interface as something that could easily be implemented with any other language that supported RPC. This sounds convoluted but has many benefits! For example, your code will never crash because of a plugin and the ability to use any language to implement a plugin. Not just Go.

It has been a battle-hardened solution for years now and is being actively used by Terraform, Vault, Consul, and especially Packer. All using go-plugin in order to provide a much needed flexibility. Writing a plugin is easy. Or so they say.

It can get complicated quickly, for example, if you are trying to use GRPC. You can lose sight of what exactly you'll need to implement, where and why; or utilizing various languages; or using go-plugins in your own project and extending your CLI with pluggable components.

These are all nothing to sneeze at. Suddenly you'll find yourself with hundreds of lines of code pasted from various examples and yet nothing works. Or worse, it DOES work but you have no idea how. Then you find yourself needing to extend it with a new capability, or you find an elusive bug and can't trace its origins.

Let's try to demystify things and draw a clearer picture about how it works and how the pieces fit together.

Basic plugin

Start by writing a simple Go GRPC plugin. In fact, we can go through the basic example in the go-plugin’s repository. We'll go step-by-step, and the switch to GRPC will be much easier!

Basic concepts Copy Chapter

Server Copy Section

In the case of plugins, the Server is the one serving the plugin's implementation. This means the server will have to provide the implementation to an interface.

Client Copy Section

The Client calls the server in order to execute the desired behaviour. The underlying logic will connect to the server running on localhost on a random higher port, call the wanted function’s implementation and wait for a response. Once the response is received provide that back to the calling Client.

Implementation Copy Chapter

The main function Copy Section

Logger

The plugins defined here use stdout in a special way. If you aren't writing a Go based plugin, you will have to do that yourself by outputting something like this:

1|1|tcp|127.0.0.1:1234|grpc

We'll come back to this later. Suffice to say the framework will pick this up and will connect to the plugin based on the output. In order to get some output back, we must define a special logger:

// Create an hclog.Logger
	logger := hclog.New(&hclog.LoggerOptions{
		Name:   "plugin",
		Output: os.Stdout,
		Level:  hclog.Debug,
	})

NewClient

// We're a host! Start by launching the plugin process.
	client := plugin.NewClient(&plugin.ClientConfig{
		HandshakeConfig: handshakeConfig,
		Plugins:         pluginMap,
		Cmd:             exec.Command("./plugin/greeter"),
		Logger:          logger,
	})
	defer client.Kill()

What is happening here? Let's see one by one:

HandshakeConfig: handshakeConfig,: This part is the handshake configuration of the plugin. It has a nice comment as well.

// handshakeConfigs are used to just do a basic handshake between
// a plugin and host. If the handshake fails, a user friendly error is shown.
// This prevents users from executing bad plugins or executing a plugin
// directory. It is a UX feature, not a security feature.
var handshakeConfig = plugin.HandshakeConfig{
	ProtocolVersion:  1,
	MagicCookieKey:   "BASIC_PLUGIN",
	MagicCookieValue: "hello",
}

The ProtocolVersion here is used in order to maintain compatibility with your current plugin versions. It's basically like an API version. If you increase this, you will have two options. Don't accept lower protocol versions or switch to the version number and use a different client implementation for a lower version than for a higher version. This way you will maintain backwards compatibility.

The MagicCookieKey and MagicCookieValue are used for a basic handshake which the comment is talking about. You have to set this ONCE for your application. Never change it again, for if you do, your plugins will no longer work. For uniqueness sake, I suggest using UUID.

Cmd is one of the most important parts about a plugin. Basically how plugins work is that they boil down to a compiled binary which starts an RPC server. This is where you will have to define the binary which will be executed and does all this. Since this is all happening locally, (please keep in mind that Go-plugins only support localhost, and for a good reason), these binaries will most likely sit next to your application's binary or in a pre-configured global location. Something like: ~/.config/my-app/plugins. This is individual for each plugin of course. The plugins can be autoloaded via a discovery function given a path and a glob.

And last but not least is the Plugins map. This map is used in order to identify a plugin called Dispense. This map is globally available and must stay consistent in order for all the plugins to work:

// pluginMap is the map of plugins we can dispense.
var pluginMap = map[string]plugin.Plugin{"greeter": &shared.GreeterPlugin{}}

You can see that the key is the name of the plugin and the value is the plugin.

We then proceed to create an RPC client:

// Connect via RPC
	rpcClient, err := client.Client()
	if err != nil {
		log.Fatal(err)
	}

Nothing fancy about this one...

The interesting part is this:

// Request the plugin
	raw, err := rpcClient.Dispense("greeter")
	if err != nil {
		log.Fatal(err)
	}

Dispense will look in the above created map and search for the plugin. If it cannot find it, it will throw an error. If it does find it, it will cast this plugin to an RPC or a GRPC type plugin. Then proceed to create an RPC or a GRPC client out of it.

There is no call yet. This is just creating a client and parsing it to a respective representation.

Now, the magic:

// We should have a Greeter now! This feels like a normal interface
	// implementation but is in fact over an RPC connection.
	greeter := raw.(shared.Greeter)
	fmt.Println(greeter.Greet())

Here, we are type asserting our raw GRPC client into our own plugin type. This is so we can call the respective function on the plugin! Once that's done we will have a {client,struct,implementation} that can be called like a simple function.

The implementation right now comes from greeter_impl.go, but that will change once protoc makes an appearance.

Behind the scenes, go-plugin will do a bunch of hat tricks with multiplexing TCP connections as well as a remote procedure call to our plugin. Our plugin then will run the function, generate some kind of output, and will then send that back for the waiting client.

The client will then proceed to parse the message into a given response type and will then return it back to the client’s callee.

This concludes main.go for now.

The Interface Copy Section

Now let’s investigate the Interface. The interface is used to provide calling details. This interface will be what defines our plugins’ capabilities. How does our Greeter look like?

// Greeter is the interface that we're exposing as a plugin.
type Greeter interface {
	Greet() string
}

This defines a function which will return a string typed value.

Now, we will need a couple of things for this to work. Firstly we need something which defines the RPC workings. go-plugin is working with net/http inside. It also uses something called Yamux for connection multiplexing, but we needn’t worry about this detail.

Implementing the RPC details looks like this:

// Here is an implementation that talks over RPC
type GreeterRPC struct {
    client *rpc.Client
}

func (g *GreeterRPC) Greet() string {
	var resp string
	err := g.client.Call("Plugin.Greet", new(interface{}), &resp)
	if err != nil {
		// You usually want your interfaces to return errors. If they don't,
		// there isn't much other choice here.
		panic(err)
	}

	return resp
}

Here the GreeterRPC struct is an RPC specific implementation that will handle communication over RPC. This is the Client in this setup.

In case of gRPC, it would look like this:

// GRPCClient is an implementation of KV that talks over RPC.
type GreeterGRPC struct{ client proto.GreeterClient }

func (m *GreeterGRPC) Greet() (string, error) {
    s, err := m.client.Greet(context.Background(), &proto.Empty{})
	return s, err
}

What's Proto and what is GreeterClient? GRPC uses Google's protoc library to serialize and unserialize data. proto.GreeterClient is generated Go code by protoc. This code is a skeleton for which implementation detail will be replaced on run time. Well, the actual result will be used and not replaced as such.

Back to our previous example. The RPC client calls a specific Plugin function called Greet for which the implementation will be provided by a Server that will be streamed back over the RPC protocol.

The server is pretty easy to follow:

// Here is the RPC server that GreeterRPC talks to, conforming to
// the requirements of net/rpc
type GreeterRPCServer struct {
	// This is the real implementation
	Impl Greeter
}

Impl is the concrete implementation that will be called in the Server's implementation of the Greet plugin. Now we must define Greet on the RPCServer in order for it to be able to call the remote code. This looks like as follows:

func (s *GreeterRPCServer) Greet(args interface{}, resp *string) error {
	*resp = s.Impl.Greet()
	return nil
}

This is all still boilerplate for the RPC works. Now comes the plugin. For this, the comment is actually quite good:

// This is the implementation of plugin.Plugin so we can serve/consume this
//
// This has two methods: Server must return an RPC server for this plugin
// type. We construct a GreeterRPCServer for this.
//
// Client must return an implementation of our interface that communicates
// over an RPC client. We return GreeterRPC for this.
//
// Ignore MuxBroker. That is used to create more multiplexed streams on our
// plugin connection and is a more advanced use case.
type GreeterPlugin struct {
	// Impl Injection
	Impl Greeter
}

func (p *GreeterPlugin) Server(*plugin.MuxBroker) (interface{}, error) {
	return &GreeterRPCServer{Impl: p.Impl}, nil
}

func (GreeterPlugin) Client(b *plugin.MuxBroker, c *rpc.Client) (interface{}, error) {
	return &GreeterRPC{client: c}, nil
}

So, remember: GreeterRPCServer is the one calling the actual implementation while Client is receiving the result of that call. The GreeterPlugin has the Greeter interface embedded just like the RPCServer. We will use the GreeterPlugin as a struct in the plugin map. This is the plugin that we will actually use.

This is all still common stuff. These things will need to be visible for both. The plugin's implementation will use the interface to see what it needs to implement. The Client will use it to see what to call and what APIs are available. Like, Greet.

The Implementation Copy Section

In a completely separate package, but which still has access to the interface definition, this plugin could be like this:

// Here is a real implementation of Greeter
type GreeterHello struct {
	logger hclog.Logger
}

func (g *GreeterHello) Greet() string {
	g.logger.Debug("message from GreeterHello.Greet")
	return "Hello!"
}

We create a struct and then add the function to it which is defined by the plugin's interface. This interface, since it's required by both parties, could well sit in a common package outside of both programs. Something like a SDK. Both code could import it and use it as a common dependency. This way we have separated the interface from the plugin and the calling client.

The main function could look something like this:

logger := hclog.New(&hclog.LoggerOptions{
    Level:      hclog.Trace,
    Output:     os.Stderr,
    JSONFormat: true,
})

greeter := &GreeterHello{
    logger: logger,
}
// pluginMap is the map of plugins we can dispense.
var pluginMap = map[string]plugin.Plugin{
    "greeter": &shared.GreeterPlugin{Impl: greeter},
}

logger.Debug("message from plugin", "foo", "bar")

plugin.Serve(&plugin.ServeConfig{
    HandshakeConfig: handshakeConfig,
    Plugins:         pluginMap,
})

Notice two things that we need. One is the handshakeConfig. You can either define it here, with the same cookie details as you defined in the client code, or you can extract the handshake information into the SDK. This is up to you.

Then the next interesting thing is the plugin.Serve method. The plugins open up a RPC communication socket and over a hijacked stdout, broadcasts its availability to the calling Client in this format:

CORE-PROTOCOL-VERSION | APP-PROTOCOL-VERSION | NETWORK-TYPE | NETWORK-ADDR | PROTOCOL

For Go plugins, you don't have to concern yourself with this. go-plugin takes care of all this for you. For non-Go versions, we must take this into account. And before calling serve, we need to output this information to stdout.

For example, a Python plugin must deal with this himself. Like this:

# Output information
print("1|1|tcp|127.0.0.1:1234|grpc")
sys.stdout.flush()

For GRPC plugins, it's also mandatory to implement a HealthChecker.

How would all this look like with GRPC?

It gets slightly more complicated but not too much. We need to use protoc to create a protocol description for our implementation, and then we will call that. Let's look at this now by converting the basic greeter example into GRPC.

GRPC Basic plugin

The example that's under GRPC is quite elaborate and perhaps you don't need the Python part. I will focus on the basic RPC example into a GRPC example. That should not be a problem.

The API Copy Chapter

First and foremost, you will need to define an API to implement with protoc. For our basic example, the protoc file could look like this:

syntax = "proto3";
package proto;

message GreetResponse {
    string message = 1;
}

message Empty {}

service GreeterService {
    rpc Greet(Empty) returns (GreetResponse);
}

The syntax is quite simple and readable. What this defines is a message, which is a response, that will contain a message with the type string. The service defines a service which has a method called Greet. The service definition is basically an interface for which we will be providing the concrete implementation through the plugin.

To read more about protoc, visit this page: Google Protocol Buffer.

Generate the code Copy Chapter

Now, with the protoc definition in hand, we need to generate the stubs that the local client implementation can call. That client call will then, through the remote procedure call, call the right function on the server which will have the concrete implementation at the ready. Run it and return the result in the specified format. Because the stub needs to be available by both parties, (the client AND the server), this needs to live in a shared location.

The client is calling the stub and the server is implementing the stub. Both need it in order to know what to call/implement.

To generate the code, run the following command:

protoc -I proto/ proto/greeter.proto --go_out=plugins=grpc:proto

I encourage you to read the generated code. Much will make little sense at first. It will have a bunch of structs and defined things that the GRPC package will use in order to server the function. The interesting bits and pieces are:

func (m *GreetResponse) GetMessage() string {
	if m != nil {
		return m.Message
	}
	return ""
}

Which will get use the message inside the response.

type GreeterServiceClient interface {
	Greet(ctx context.Context, in *Empty, opts ...grpc.CallOption) (*GreetResponse, error)
}

This is our ServiceClient interface which defines the Greet function’s topology.

And lastly, this guy:

func RegisterGreeterServiceServer(s *grpc.Server, srv GreeterServiceServer) {
	s.RegisterService(&_GreeterService_serviceDesc, srv)
}

Which we will need in order to register our implementation for the server. We can ignore the rest.

The interface Copy Chapter

Much like the RPC, we need to define an interface for the client and server to use. This must be in a shared place as both the server and the client need to know about it. You could put this into an SDK and your peers could just get the SDK and implement some function for define and done. The interface definition in the GRPC land could look something like this:

// Greeter is the interface that we're exposing as a plugin.
type Greeter interface {
	Greet() string
}

// This is the implementation of plugin.GRPCPlugin so we can serve/consume this.
type GreeterGRPCPlugin struct {
	// GRPCPlugin must still implement the Plugin interface
	plugin.Plugin
	// Concrete implementation, written in Go. This is only used for plugins
	// that are written in Go.
	Impl Greeter
}

func (p *GreeterGRPCPlugin) GRPCServer(broker *plugin.GRPCBroker, s *grpc.Server) error {
	proto.RegisterGreeterServer(s, &GRPCServer{Impl: p.Impl})
	return nil
}

func (p *GreeterGRPCPlugin) GRPCClient(ctx context.Context, broker *plugin.GRPCBroker, c *grpc.ClientConn) (interface{}, error) {
	return &GRPCClient{client: proto.NewGreeterClient(c)}, nil
}

With this we have the Plugin's implementation for hashicorp what needed to be done. The plugin will call the underlying implementation and serve/consume the plugin. We can now write the GRPC part of it.

Please note that proto is a shared library too where the protocol stubs reside. That needs to be somewhere on the path or in a separate SDK of some sort, but it must be visible.

Writing the GRPC Client Copy Chapter

Firstly we define the grpc client struct:

// GRPCClient is an implementation of Greeter that talks over RPC.
type GRPCClient struct{ client proto.GreeterClient }

Then we define how the client will call the remote function:

func (m *GRPCClient) Greet() string {
	ret, _ := m.client.Greet(context.Background(), &proto.Empty{})
	return ret.Message
}

This will take the client in the GRPCClient and will call the method on it. Once that's done we will return to the result Message property which will be Hello!. proto.Empty is an empty struct; we use this if there is no parameter for a defined method or no return value. We can't just leave it blank. protoc needs to be told explicitly that there is no parameter or return value.

Writing the GRPC Server Copy Chapter

The server implementation will also be similar. We call Impl here which will have our concrete plugin implementation.

// Here is the gRPC server that GRPCClient talks to.
type GRPCServer struct {
	// This is the real implementation
	Impl Greeter
}

func (m *GRPCServer) Greet(
	ctx context.Context,
	req *proto.Empty) *proto.GreeterResponse {
	v := m.Impl.Greet()
	return &proto.GreeterResponse{Message: v}
}

And we will use the protoc defined message response. v will have the response from Greet which will be Hello! provided by the concrete plugin's implementation. We then transform that into a protoc type by setting the Message property on the GreeterResponse struct provided by the automatically generated protoc stub code.

Easy, right?

Writing the plugin itself Copy Chapter

The whole thing looks much like the RPC implementation with just a few small modifications and changes. This can sit completely outside of everything, or can even be provided by a third party implementor.

// Here is a real implementation of KV that writes to a local file with
// the key name and the contents are the value of the key.
type Greeter struct{}

func (Greeter) Greet() error {
	return "Hello!"
}

func main() {
	plugin.Serve(&plugin.ServeConfig{
		HandshakeConfig: shared.Handshake,
		Plugins: map[string]plugin.Plugin{
			"greeter": &shared.GreeterGRPCPlugin{Impl: &Greeter{}},
		},

		// A non-nil value here enables gRPC serving for this plugin...
		GRPCServer: plugin.DefaultGRPCServer,
	})
}

Calling it all in the main Copy Chapter

Once all that is done, the main function looks the same as RPC's main but with some small modifications.

// We're a host. Start by launching the plugin process.
	client := plugin.NewClient(&plugin.ClientConfig{
		HandshakeConfig: shared.Handshake,
		Plugins:         shared.PluginMap,
		Cmd:             exec.Command("./plugin/greeter"),
		AllowedProtocols: []plugin.Protocol{plugin.ProtocolGRPC},
	})

The NewClient now defines AllowedProtocols to be ProtocolGRPC. The rest is the same as before calling Dispense and value hinting the plugin to the correct type then calling Greet().

Conclusion

This is it. We made it! Now our plugin works over GRPC with a defined API by protoc. The plugin's implementation can live where ever we want it, but it needs some shared data. These are:

• The generated code by protoc
• The defined plugin interface
• The GRPC Server and Client

These need to be visible by both the Client and the Server. The Server here is the plugin. If you are planning on making people be able to extend your application with go-plugin, you should make these available as a separate SDK. So people won't have to include your whole project just to implement an interface and use protoc. In fact, you could also extract the protoc definition into a separate repository so that your SDK can also pull it in.

You will have three repositories:

• Your application
• The SDK providing the interface and the GRPC Server and Client implementation
• The protoc definition file and generated skeleton ( for Go based plugins ).

Other languages will have to generate their own protoc code, and includ it into the plugin; like the Python implementation example located here: Go-plugin Python Example. Also, read this documentation carefully: non-go go-plugin. This document will also clarify what 1|1|tcp|127.0.0.1:1234|grpc means and will dissipate the confusion around how plugins work.

Lastly, if you would like to have an in-depth explanation about how go-plugin came to be, watch this video by Mitchell:

go-plugin explanation video.

That's it. I hope this has helped to clear the confusion around how to use go-plugin.

File: docs/guide-plugin-write-non-go.md Copy Chapter

Writing Plugins Without Go

This guide explains how to write a go-plugin compatible plugin using
a programming language other than Go. go-plugin supports plugins using
gRPC. This makes it relatively simple to write plugins
using other languages!

Minimal knowledge about gRPC is assumed. We recommend reading the
gRPC Go Tutorial. This
alone is enough gRPC knowledge to continue.

This guide will implement the kv example in Python.
Full source code for the examples present in this guide
is available in the examples/grpc folder.

1. Implement the Service Copy Chapter

The first step is to implement the gRPC server for the protocol buffers
service that your plugin defines. This is a standard gRPC server.
For the KV service, the service looks like this:

service KV {
    rpc Get(GetRequest) returns (GetResponse);
    rpc Put(PutRequest) returns (Empty);
}

We can implement that using Python as easily as:

class KVServicer(kv_pb2_grpc.KVServicer):
    """Implementation of KV service."""

    def Get(self, request, context):
        filename = "kv_"+request.key
        with open(filename, 'r') as f:
            result = kv_pb2.GetResponse()
            result.value = f.read()
            return result

    def Put(self, request, context):
        filename = "kv_"+request.key
        value = "{0}\n\nWritten from plugin-python".format(request.value)
        with open(filename, 'w') as f:
            f.write(value)

        return kv_pb2.Empty()

Great! With that, we have a fully functioning implementation of the service.
You can test this using standard gRPC testing mechanisms.

2. Serve the Service Copy Chapter

Next, we need to create a gRPC server and serve the service we just made.

In Python:

# Make the server
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))

# Add our service
kv_pb2_grpc.add_KVServicer_to_server(KVServicer(), server)

# Listen on a port
server.add_insecure_port(':1234')

# Start
server.start()

You can listen on any TCP address or Unix domain socket. go-plugin does
assume that connections are reliable (local), so you should not serve
your plugin across the network.

3. Add the gRPC Health Checking Service Copy Chapter

go-plugin requires the
gRPC Health Checking Service
to be registered on your server. You must register the status of "plugin" to be SERVING.

The health checking service is used by go-plugin to determine if everything
is healthy with the connection. If you don't implement this service, your
process may be abruptly restarted and your plugins are likely to be unreliable.

health = HealthServicer()
health.set("plugin", health_pb2.HealthCheckResponse.ServingStatus.Value('SERVING'))
health_pb2_grpc.add_HealthServicer_to_server(health, server)

4. Output Handshake Information Copy Chapter

The final step is to output the handshake information to stdout. go-plugin
reads a single line from stdout to determine how to connect to your plugin,
what protocol it is using, etc.

The structure is:

CORE-PROTOCOL-VERSION | APP-PROTOCOL-VERSION | NETWORK-TYPE | NETWORK-ADDR | PROTOCOL

Where:

• CORE-PROTOCOL-VERSION is the protocol version for go-plugin itself.
  The current value is 1. Please use this value. Any other value will
  cause your plugin to not load.

• APP-PROTOCOL-VERSION is the protocol version for the application data.
  This is determined by the application. You must reference the documentation
  for your application to determine the desired value.

• NETWORK-TYPE and NETWORK-ADDR are the networking information for
  connecting to this plugin. The type must be "unix" or "tcp". The address
  is a path to the Unix socket for "unix" and an IP address for "tcp".

• PROTOCOL is the named protocol that the connection will use. If this
  is omitted (older versions), this is "netrpc" for Go net/rpc. This can
  also be "grpc". This is the protocol that the plugin wants to speak to
  the host process with.

For our example that is:

1|1|tcp|127.0.0.1:1234|grpc

The only element you'll have to be careful about is the second one (the
APP-PROTOCOL-VERISON). This will depend on the application you're
building a plugin for. Please reference their documentation for more
information.

5. Done! Copy Chapter

And we're done!

Configure the host application (the application you're writing a plugin
for) to execute your Python application. Configuring plugins is specific
to the host application.

For our example, we used an environmental variable, and it looks like this:

$ export KV_PLUGIN="python plugin.py"

File: docs/internals.md Copy Chapter

go-plugin Internals

This section discusses the internals of how go-plugin works.

go-plugin operates by either serving a plugin or being a client
connecting to a remote plugin. The "client" is the host process or the
process that itself uses plugins. The "server" is the plugin process.

For a server:

1. Output handshake to stdout
2. Wait for connection on control address
3. Serve plugins over control address

For a client:

1. Launch a plugin binary
2. Read and verify handshake from plugin stdout
3. Connect to plugin control address using desired protocol
4. Dispense plugins using control connection

Handshake Copy Chapter

The handshake is the initial communication between a plugin and a host
process to determine how the host process can connect and communicate to
the plugin. This handshake is done over the plugin process's stdout.

The go-plugin library itself handles the handshake when using the
Server to serve a plugin. You do not need to understand the internals
of the handshake, unless you're building a go-plugin compatible plugin
in another language.

The handshake is a single line of data terminated with a newline character
\n. It looks like the following:

1|3|unix|/path/to/socket|grpc

The structure is:

CORE-PROTOCOL-VERSION | APP-PROTOCOL-VERSION | NETWORK-TYPE | NETWORK-ADDR | PROTOCOL

Where:

• CORE-PROTOCOL-VERSION is the protocol version for go-plugin itself.
  The current value is 1. Please use this value. Any other value will
  cause your plugin to not load.

• APP-PROTOCOL-VERSION is the protocol version for the application data.
  This is determined by the application. You must reference the documentation
  for your application to determine the desired value.

• NETWORK-TYPE and NETWORK-ADDR are the networking information for
  connecting to this plugin. The type must be "unix" or "tcp". The address
  is a path to the Unix socket for "unix" and an IP address for "tcp".

• PROTOCOL is the named protocol that the connection will use. If this
  is omitted (older versions), this is "netrpc" for Go net/rpc. This can
  also be "grpc". This is the protocol that the plugin wants to speak to
  the host process with.

Environment Variables Copy Chapter

When serving a plugin over TCP, the following environment variables can be
specified to restrict the port that will be assigned to be from within a
specific range. If not values are provided, the port will be randomly assigned
by the operating system.

• PLUGIN_MIN_PORT: Specifies the minimum port value that will be assigned to the listener.

• PLUGIN_MAX_PORT: Specifies the maximum port value that will be assigned to the listener.

File: docs/README.md Copy Chapter

go-plugin Documentation

This directory contains documentation and guides for go-plugin and how
to integrate it into your projects. It is assumed that you know what
go-plugin is and why you would want to use it. If not, please see the
README.

Table of Contents Copy Chapter

Writing Plugins Without Go.
This shows how to write a plugin using a programming language other than
Go.

Extensive Tutorial on Usage.
A guide showing the usage and structure of go-plugin including a detailed
walkthrough of setting up the plugins under the examples folder.

File: examples/negotiated/README.md Copy Chapter

Negotiated version KV Example

This example builds a simple key/value store CLI where the plugin version can
be negotiated between client and server.

# This builds the main CLI
$ go build -o kv

# This builds the plugin written in Go
$ go build -o kv-plugin ./plugin-go

# Write a value using proto version 3 and grpc
$ KV_PROTO=grpc ./kv put hello world

# Read it back using proto version 2 and netrpc
$ KV_PROTO=netrpc ./kv get hello
world

Written from plugin version 3
Read by plugin version 2

Negotiated Protocol

The Client sends the list of available plugin versions to the server. When
presented with a list of plugin versions, the server iterates over them in
reverse, and uses the highest numbered match to choose the plugins to execute.
If a legacy client is used and no versions are sent to the server, the server
will default to the oldest version in its configuration.

File: examples/grpc/README.md Copy Chapter

KV Example

This example builds a simple key/value store CLI where the mechanism
for storing and retrieving keys is pluggable. To build this example:

# This builds the main CLI
$ go build -o kv

# This builds the plugin written in Go
$ go build -o kv-go-grpc ./plugin-go-grpc

# This tells the KV binary to use the "kv-go-grpc" binary over gRPC
$ export KV_PROTO=grpc
$ export KV_PLUGIN="./kv-go-grpc"

# Read and write
$ ./kv put hello world

$ ./kv get hello
world

Plugin: plugin-go-grpc Copy Section

This plugin uses gRPC to serve a plugin that is written in Go:

# This builds the plugin written in Go
$ go build -o kv-go-grpc ./plugin-go-grpc

# This tells the KV binary to use the "kv-go-grpc" binary over gRPC
$ export KV_PROTO=grpc
$ export KV_PLUGIN="./kv-go-grpc"

Plugin: plugin-go-netrpc Copy Section

This plugin uses the builtin Go net/rpc mechanism to serve the plugin:

# This builds the plugin written in Go
$ go build -o kv-go-netrpc ./plugin-go-netrpc

# This tells the KV binary to use the "kv-go-netrpc" binary over net/rpc
$ export KV_PROTO=netrpc
$ export KV_PLUGIN="./kv-go-netrpc"

Plugin: plugin-python Copy Section

This plugin is written in Python:

$ python -m venv plugin-python/.venv
$ source plugin-python/.venv/bin/activate
$ pip install -r plugin-python/requirements.txt
$ export KV_PROTO=grpc
$ export KV_PLUGIN="python plugin-python/plugin.py"

Updating the Protocol Copy Chapter

If you update the protocol buffers file, you can regenerate the file
using the following command from this directory. You do not need to run
this if you're just trying the example.

$ buf generate

File: examples/bidirectional/README.md Copy Chapter

Counter Example

This example builds a simple key/counter store CLI where the mechanism
for storing and retrieving keys is pluggable. However, in this example we don't
trust the plugin to do the summation work. We use bi-directional plugins to
call back into the main proccess to do the sum of two numbers. To build this example:

# This builds the main CLI
$ go build -o counter

# This builds the plugin written in Go
$ go build -o counter-go-grpc ./plugin-go-grpc

# This tells the Counter binary to use the "counter-go-grpc" binary
$ export COUNTER_PLUGIN="./counter-go-grpc"

# Read and write
$ ./counter put hello 1
$ ./counter put hello 1

$ ./counter get hello
2

Plugin: plugin-go-grpc Copy Section

This plugin uses gRPC to serve a plugin that is written in Go:

# This builds the plugin written in Go
$ go build -o counter-go-grpc ./plugin-go-grpc

# This tells the KV binary to use the "kv-go-grpc" binary
$ export COUNTER_PLUGIN="./counter-go-grpc"

Updating the Protocol Copy Chapter

If you update the protocol buffers file, you can regenerate the file
using the following command from this directory. You do not need to run
this if you're just trying the example.

$ buf generate

File: examples/basic/README.md Copy Chapter

Plugin Example Copy Chapter

Compile the plugin itself via:

go build -o ./plugin/greeter ./plugin/greeter_impl.go

Compile this driver via:

go build -o basic .

You can then launch the plugin sample via:

./basic

File: internal/cmdrunner/testdata/README.md Copy Chapter

This folder contains a minimal Go program so that we can obtain example binaries of programs for various architectures for use in tests.