tools Documentation

1. Project Overview & Quickstart (golang/tools) Copy Chapter

File: README.md Copy Chapter

Go Tools

This repository provides the golang.org/x/tools module, comprising
various tools and packages mostly for static analysis of Go programs,
some of which are listed below.
Use the "Go reference" link above for more information about any package.

It also contains the
golang.org/x/tools/gopls
module, whose root package is a language-server protocol (LSP) server for Go.
An LSP server analyses the source code of a project and
responds to requests from a wide range of editors such as VSCode and
Vim, allowing them to support IDE-like functionality.

Selected commands:

• cmd/goimports formats a Go program like go fmt and additionally
  inserts import statements for any packages required by the file
  after it is edited.
• cmd/callgraph prints the call graph of a Go program.
• cmd/digraph is a utility for manipulating directed graphs in textual notation.
• cmd/stringer generates declarations (including a String method) for "enum" types.
• cmd/toolstash is a utility to simplify working with multiple versions of the Go toolchain.

These commands may be fetched with a command such as

go install golang.org/x/tools/cmd/goimports@latest

Selected packages:

• go/ssa provides a static single-assignment form (SSA) intermediate
  representation (IR) for Go programs, similar to a typical compiler,
  for use by analysis tools.

• go/packages provides a simple interface for loading, parsing, and
  type checking a complete Go program from source code.

• go/analysis provides a framework for modular static analysis of Go
  programs.

• go/callgraph provides call graphs of Go programs using a variety
  of algorithms with different trade-offs.

• go/ast/inspector provides an optimized means of traversing a Go
  parse tree for use in analysis tools.

• go/cfg provides a simple control-flow graph (CFG) for a Go function.

• go/gcexportdata and go/gccgoexportdata read and write the binary
  files containing type information used by the standard and gccgo compilers.

• go/types/objectpath provides a stable naming scheme for named
  entities ("objects") in the go/types API.

Numerous other packages provide more esoteric functionality.

Contributing Copy Chapter

This repository uses Gerrit for code changes.
To learn how to submit changes, see https://go.dev/doc/contribute.

The git repository is https://go.googlesource.com/tools.

The main issue tracker for the tools repository is located at
https://go.dev/issues. Prefix your issue with "x/tools/(your
subdir):" in the subject line, so it is easy to find.

JavaScript and CSS Formatting Copy Section

This repository uses prettier to format JS and CSS files.

The version of prettier used is 1.18.2.

It is encouraged that all JS and CSS code be run through this before submitting
a change. However, it is not a strict requirement enforced by CI.

File: cmd/signature-fuzzer/README.md Copy Chapter

signature-fuzzer

This directory contains utilities for fuzz testing of Go function signatures, for use in developing/testing a Go compiler.

The basic idea of the fuzzer is that it emits source code for a stand-alone Go program; this generated program is a series of pairs of functions, a "Caller" function and a "Checker" function. The signature of the Checker function is generated randomly (random number of parameters and returns, each with randomly chosen types). The "Caller" func contains invocations of the "Checker" function, each passing randomly chosen values to the params of the "Checker", then the caller verifies that expected values are returned correctly. The "Checker" function in turn has code to verify that the expected values (more details below).

There are three main parts to the fuzzer: a generator package, a driver package, and a runner package.

The "generator" contains the guts of the fuzzer, the bits that actually emit the random code.

The "driver" is a stand-alone program that invokes the generator to create a single test program. It is not terribly useful on its own (since it doesn't actually build or run the generated program), but it is handy for debugging the generator or looking at examples of the emitted code.

The "runner" is a more complete test harness; it repeatedly runs the generator to create a new test program, builds the test program, then runs it (checking for errors along the way). If at any point a build or test fails, the "runner" harness attempts a minimization process to try to narrow down the failure to a single package and/or function.

What the generated code looks like Copy Chapter

Generated Go functions will have an "interesting" set of signatures (mix of
arrays, scalars, structs), intended to pick out corner cases and odd bits in the
Go compiler's code that handles function calls and returns.

The first generated file is genChecker.go, which contains function that look something
like this (simplified):

type StructF4S0 struct {
F0 float64
F1 int16
F2 uint16
}

// 0 returns 2 params
func Test4(p0 int8, p1 StructF4S0)  {
  c0 := int8(-1)
  if p0 != c0 {
    NoteFailure(4, "parm", 0)
  }
  c1 := StructF4S0{float64(2), int16(-3), uint16(4)}
  if p1 != c1 {
    NoteFailure(4, "parm", 1)
  }
  return
}

Here the test generator has randomly selected 0 return values and 2 params, then randomly generated types for the params.

The generator then emits code on the calling side into the file "genCaller.go", which might look like:

func Caller4() {
var p0 int8
p0 = int8(-1)
var p1 StructF4S0
p1 = StructF4S0{float64(2), int16(-3), uint16(4)}
// 0 returns 2 params
Test4(p0, p1)
}

The generator then emits some utility functions (ex: NoteFailure) and a main routine that cycles through all of the tests.

Trying a single run of the generator Copy Chapter

To generate a set of source files just to see what they look like, you can build and run the test generator as follows. This creates a new directory "cabiTest" containing generated test files:

$ git clone https://golang.org/x/tools
$ cd tools/cmd/signature-fuzzer/fuzz-driver
$ go build .
$ ./fuzz-driver -numpkgs 3 -numfcns 5 -seed 12345 -outdir /tmp/sigfuzzTest -pkgpath foobar
$ cd /tmp/sigfuzzTest
$ find . -type f -print
./genCaller1/genCaller1.go
./genUtils/genUtils.go
./genChecker1/genChecker1.go
./genChecker0/genChecker0.go
./genCaller2/genCaller2.go
./genCaller0/genCaller0.go
./genMain.go
./go.mod
./genChecker2/genChecker2.go
$

You can build and run the generated files in the usual way:

$ cd /tmp/sigfuzzTest
$ go build .
$ ./foobar
starting main
finished 15 tests
$

Example usage for the test runner Copy Chapter

The test runner orchestrates multiple runs of the fuzzer, iteratively emitting code, building it, and testing the resulting binary. To use the runner, build and invoke it with a specific number of iterations; it will select a new random seed on each invocation. The runner will terminate as soon as it finds a failure. Example:

$ git clone https://golang.org/x/tools
$ cd tools/cmd/signature-fuzzer/fuzz-runner
$ go build .
$ ./fuzz-runner -numit=3
... begin iteration 0 with current seed 67104558
starting main
finished 1000 tests
... begin iteration 1 with current seed 67104659
starting main
finished 1000 tests
... begin iteration 2 with current seed 67104760
starting main
finished 1000 tests
$

If the runner encounters a failure, it will try to perform test-case "minimization", e.g. attempt to isolate the failure

$ cd tools/cmd/signature-fuzzer/fuzz-runner
$ go build .
$ ./fuzz-runner -n=10
./fuzz-runner -n=10
... begin iteration 0 with current seed 40661762
Error: fail [reflect] |20|3|1| =Checker3.Test1= return 1
error executing cmd ./fzTest: exit status 1
... starting minimization for failed directory /tmp/fuzzrun1005327337/fuzzTest
package minimization succeeded: found bad pkg 3
function minimization succeeded: found bad fcn 1
$

Here the runner has generates a failure, minimized it down to a single function and package, and left the resulting program in the output directory /tmp/fuzzrun1005327337/fuzzTest.

Limitations, future work Copy Chapter

No support yet for variadic functions.

The set of generated types is still a bit thin; it has fairly limited support for interface values, and doesn't include channels.

Todos:

• better interface value coverage

• implement testing of reflect.MakeFunc

• extend to work with generic code of various types

• extend to work in a debugging scenario (e.g. instead of just emitting code,
  emit a script of debugger commands to run the program with expected
  responses from the debugger)

• rework things so that instead of always checking all of a given parameter
  value, we sometimes skip over elements (or just check the length of a slice
  or string as opposed to looking at its value)

• consider adding runtime.GC() calls at some points in the generated code

File: cmd/bisect/testdata/README.md Copy Chapter

This directory contains test inputs for the bisect command.

Each text file is a txtar archive (see https://pkg.go.dev/golang.org/x/tools/txtar
or go doc txtar).

The comment at the top of the archive is a JSON object describing a
target behavior. Specifically, the Fail key gives a boolean expression
that should provoke a failure. Bisect's job is to discover this
condition.

The Bisect key describes settings in the Bisect struct that we want to
change, to simulate the use of various command-line options.

The txtar archive files should be "stdout" and "stderr", giving the
expected standard output and standard error. If the bisect command
should exit with a non-zero status, the stderr in the archive will end
with the line "".

Running go test -update will rewrite the stdout and stderr files in
each testdata archive to match the current state of the tool. This is
a useful command when the logging prints from bisect change or when
writing a new test.

To use go test -update to write a new test:

• Create a new .txt file with just a JSON object at the top,
  specifying what you want to test.
• Run go test -update.
• Reload the .txt file and read the stdout and stderr to see if you agree.

File: go/analysis/doc/suggested_fixes.md Copy Chapter

Suggested Fixes in the Analysis Framework

The Purpose of Suggested Fixes Copy Chapter

The analysis framework is planned to add a facility to output
suggested fixes. Suggested fixes in the analysis framework
are meant to address two common use cases. The first is the
natural use case of allowing the user to quickly fix errors or issues
pointed out by analyzers through their editor or analysis tool.
An editor, when showing a diagnostic for an issue, can propose
code to fix that issue. Users can accept the proposal and have
the editor apply the fix for them. The second case is to allow
for defining refactorings. An analyzer meant to perform a
refactoring can produce suggested fixes equivalent to the diff
of the refactoring. Then, an analysis driver meant to apply
refactorings can automatically apply all the diffs that
are produced by the analysis as suggested fixes.

Proposed Suggested Fix API Copy Chapter

Suggested fixes will be defined using the following structs:

// A SuggestedFix is a code change associated with a Diagnostic that a user can choose
// to apply to their code. Usually the SuggestedFix is meant to fix the issue flagged
// by the diagnostic.
type SuggestedFix struct {
	// A description for this suggested fix to be shown to a user deciding
	// whether to accept it.
	Message   string
	TextEdits []TextEdit
}

// A TextEdit represents the replacement of the code between Pos and End with the new text.
type TextEdit struct {
	// For a pure insertion, End can either be set to Pos or token.NoPos.
	Pos     token.Pos
	End     token.Pos
	NewText []byte
}

A suggested fix needs a message field so it can specify what it will do.
Some analyses may not have clear cut fixes, and a suggested fix may need
to provide additional information to help users specify whether they
should be added.

Suggested fixes are allowed to make multiple
edits in a file, because some logical changes may affect otherwise
unrelated parts of the AST.

A TextEdit specifies a Pos and End: these will usually be the Pos
and End of an AST node that will be replaced.

Finally, the replacements themselves are represented as []bytes.

Suggested fixes themselves will be added as a field in the
Diagnostic struct:

type Diagnostic struct {
	...
	SuggestedFixes []SuggestedFix  // this is an optional field
}

Requirements for SuggestedFixes Copy Section

SuggestedFixes will be required to conform to several requirements:

• TextEdits for a SuggestedFix should not overlap.
• TextEdits for SuggestedFixes should not contain edits for other packages.
• Each TextEdit should apply to a single file.

These requirements guarantee that suggested fixes can be cleanly applied.
Because a driver may only analyze, or be able to modify, the current package,
we restrict edits to the current package. In general this restriction should
not be a big problem for users because other packages might not belong to the
same module and so will not be safe to modify in a single change.

On the other hand, analyzers will not be required to produce gofmt-compliant
code. Analysis drivers will be expected to apply gofmt to the results of
a SuggestedFix application.

SuggestedFix integration points Copy Chapter

```checker -fix``` Copy Section

Singlechecker and multichecker have the -fix flag, which will automatically
apply all fixes suggested by their analysis or analyses. This is intended to
be used primarily by refactoring tools, because in general, like diagnostics,
suggested fixes will need to be examined by a human who can decide whether
they are relevant.

gopls Copy Section

Suggested fixes have been integrated into gopls, and editors can choose
to display the suggested fixes to the user as they type, so that they can be
accepted to fix diagnostics immediately.

Code Review Tools (Future Work) Copy Section

Suggested fixes can be integrated into programs that are integrated with
code review systems to suggest fixes that users can apply from their code review tools.

Alternatives Copy Chapter

Performing transformations directly on the AST Copy Section

Even though it may be more convenient
for authors of refactorings to perform transformations directly on
the AST, allowing mutations on the AST would mean that a copy of the AST
would need to be made every time a transformation was produced, to avoid
having transformations interfere with each other.

This is primarily an issue with the current design of the Go AST and
it's possible that a new future version of the AST might make this a more
viable option.

Supplying AST nodes directly Copy Section

Another possibility would be for SuggestedFixes to supply the replacement
ASTs directly. There is one primary limitation to this: that because
comments to ASTs specify their location using token.Pos values, it's very
difficult to place any comments in the right place.

In general, it's also more difficult to generate the AST structures for
some code than to generate the text for that code. So we prefer to allow
the flexibility to do the latter.

Because users can call format.Node to produce the text for any
AST node, users will always be able to produce a SuggestedFix from AST
nodes. In future, we may choose to add a convenience method that does this for users.

File: gopls/doc/design/design.md Copy Chapter

title: "Gopls: Design" Copy Chapter

_A note from the future_ Copy Chapter

What follows below is the original design document for gopls, aggregated from
various sources spanning 2018 and 2019. Since then, all of the features listed
below have been implemented, along with many others. The first two goals have
been achieved: gopls is a full implementation of the LSP, and the default
backend for VS Code Go and many other editors. The third goal has only been
partially realized: while gopls has gained many features, it is not extensible
in the sense used in this document: the only way to extend gopls is to modify
gopls. The fourth goal is not achieved: while some notable companies are able
to use gopls with Bazel, the experience is subpar, and the Go command is the
only officially supported build system.

On the other hand, two of the explicit non-goals have been reconsidered. One is
minor: syntax highlighting is now supported in the LSP by way of semantic
tokens. The other is major: as gopls gained popularity, it became apparent that
its memory footprint was a problem. The size of developer workspaces was
increasing faster than the RAM available in typically development environments
(particularly with containerized development). Gopls now uses a hybrid of
on-disk indexes and in-memory caches, described in more detail in our
blog post on scalability.

Notably, in anticipating difficulties this doc turned out to be prescient.
Gopls has indeed struggled against the core standary library packages upon
which it is built, and its user experience is still limited by the LSP.
Nevertheless, sticking with the standard library and LSP was the right
approach, as despite our small team these decisions have helped gopls keep up
with the evolving Go language (i.e. generics), and to integrate with many new
text editors.

Gopls development continues, more than four years later, with a focus on
simplicity, reliability, and extensibility. The new, opt-in
Go telemetry
will help us attain a higher standard of stability in our releases than we've
been able to achieve through Github issues alone. Furthermore, telemetry will
allow us to focus on high-priority features, and deprecate historical
workarounds that burden the codebase. With greater velocity, we look forward
to working with the community on improved refactoring, static analysis, and
whatever else the future brings.

• Rob Findley ([email protected]), 2023

Goals Copy Chapter

• gopls should become the default editor backend for the major editors used by Go programmers, fully supported by the Go team.
• gopls will be a full implementation of LSP, as described in the LSP specification, to standardize as many of its features as possible.
• gopls will be clean and extensible so that it can encompass additional features in the future, allowing Go tooling to become best in class once more.
• gopls will support alternate build systems and file layouts, allowing Go development to be simpler and more powerful in any environment.

Context Copy Chapter

While Go has a number of excellent and useful command-line tools that enhance the developer experience, it has become clear that integrating these tools with IDEs can pose challenges.

Support of these tools has relied on the goodwill of community members, and they have been put under a large burden of support at times as the language, toolchain and environments change. As a result many tools have ceased to work, have had support problems, or become confusing with forks and replacements, or provided an experience that is not as good as it could be.
See the section below on existing solutions for more problems and details.

This is fine for tools used occasionally, but for core IDE features, this is not acceptable.
Autocompletion, jump to definition, formatting, and other such features should always work, as they are key for Go development.

The Go team will create an editor backend that works in any build system.
It will also be able to improve upon the latency of Go tools, since each tool will no longer have to individually run the type-checker on each invocation, instead there will be a long-running process and data can be shared between the definitions, completions, diagnostics, and other features.

By taking ownership of these tools and packaging them together in the form of gopls, the Go team will ensure that the Go development experience isn’t unnecessarily complicated for Go users.
Having one editor backend will simplify the lives of Go developers, the Go team, and the maintainers of Go editor plugins.

See Rebecca's excellent GopherCon keynote talk and slides for some more context.

Non-Goals Copy Chapter

• Command line speed

  Although gopls will have a command line mode, it will be optimized for long running and not command responsiveness, as such it may not be the right tool for things like CI systems.
  For such cases there will have to be an alternate tool using the same underlying libraries for consistency.

• Low memory environments

  In order to do a good job of processing large projects with very low latencies gopls will be holding a lot of information in memory.
  It is presumed that developers are normally working on systems with significant RAM and this will not be a problem.
  In general this is upheld by the large memory usage of existing IDE solutions (like IntelliJ)

• Syntax highlighting

  At the moment there is no editor that delegates this functionality to a separate binary, and no standard way of doing it.

Existing solutions Copy Chapter

Every year the Go team conducts a survey, asking developers about their experiences with the language.

One question that is asked is “How do you feel about your editor?”.

The responses told a very negative story. Some categorized quotes:

• Setup
  • "Hard to install and configure"
  • "Inadequate documentation"
• Performance
  • "Performance is very poor"
  • "Pretty slow in large projects"
• Reliability
  • "Features work one day, but not the next"
  • "Tooling is not updated with new language features"

Each editor has its own plugin that shells out to a variety of tools, many of which break with new Go releases or because they are no longer maintained.

The individual tools each have to do the work to understand the code and all its transitive dependencies.

Each feature is a different tool, with a different set of patterns for its command line, a different way to accept input and parse output, a different way of specifying source code locations.
To support its existing feature set, VSCode installed 24 different command line tools, many of which have options or forks to configure. When looking at the set of tools that needed to be migrated to modules, across all the editors, there were 63 separate tools.

All these tools need to understand the code, and they use the same standard libraries to do it. Those libraries are optimized for these kinds of tools, but even so processing that much code takes a lot of time time. Almost none of the tools are capable of returning results within 100ms.
As developers type in their editor, multiple of these features need to activate, which means they are not just paying the cost once, but many times. The overall effect is an editing experience that feels sluggish, and features that are either not enabled or sometimes produce results that appear so slowly they are no longer useful when they arrive. This is a problem that increases with the size of the code base, which means it is getting worse over time, and is especially bad for the kinds of large code bases companies are dealing with as they use Go for more major tasks.

Requirements Copy Chapter

Complete feature set Copy Section

For gopls to be considered a success it has to implement the full feature set discussed below.
This is the set of features that users need in order to feel as productive as they were with the tooling it is replacing. It does not include every feature of previous implementations, there are some features that are almost never used that should be dropped (like guru's pointer analysis) and some other features that do not easily fit and will have to be worked around (replacing the save hook/linter).

Equivalent or better experience Copy Section

For all of those features, the user experience must match or exceed the current one available in all editors.
This is an easy statement to make, but a hard one to validate or measure. Many of the possible measures fail to capture the experience.

For instance, if an attempt was made to measure the latency of a jump to definition call, the results would be fairly consistent from the old godef tool. From the gopls implementation there may be a much larger range of latencies, with the best being orders of magnitude faster, and the worse slightly worse, because gopls attempts to do far more work, but manages to cache it across calls.

Or for a completion call, it might be slower but produce a better first match such that users accept it more often, resulting in an overall better experience.

For the most part this has to rely on user reports. If users are refusing to switch because the experience is not better, it is clearly not done, if they are switching but most people are complaining, there are probably enough areas that are better to make the switch compelling but other areas which are worse. If most people are switching and either staying silent or being positive, it is probably done. When writing tools, the user is all that matters.

Solid community of contributors Copy Section

The scope and scale of the problem gopls is trying to solve is untenable for the core Go team, it is going to require a strong community to make it all happen.

This implies the code must be easy to contribute to, and easy for many developers to work on in parallel. The functionality needs to be well decoupled, and have a thorough testing story.

Latencies that fall within user tolerance Copy Section

There has been a lot of research on acceptable latencies for user actions.

The main result that affects gopls is that feedback in direct response to continuous user actions needs to be under 100ms to be imperceptible, and anything above 200ms aggravates the user.
This means in general the aim has to be <100ms for anything that happens as the developer types.
There will always be cases where gopls fails to meet this deadline, and there needs to be ways to make the user experience okay in those cases, but in general the point of this deadline is to inform the basic architecture design, any solution that cannot theoretically meet this goal in the long term is the wrong answer.

Easy to configure Copy Section

Developers are very particular, and have very differing desires in their coding experience. gopls is going to have to support a significant amount of flexibility, in order to meet those desires.
The default settings however with no configuration at all must be the one that is best experience for most users, and where possible the features must be flexible without configuration so that the client can easily make the choices about treatment without changing its communication with gopls.

Difficulties Copy Chapter

Volume of data Copy Section

• Small:
• Medium:
• Large:
• Corporate mono-repo: Much much bigger

Parsing and type checking large amounts of code is quite expensive, and the converted forms use a lot of space. As gopls has to keep updating this information while the developer types, it needs to manage how it caches the converted forms very carefully to balance memory use vs speed.

Cache invalidation Copy Section

The basic unit of operation for the type checking is the package, but the basic unit of operation for an editor is the file.
gopls needs to be able to map files to packages efficiently, so that when files change it knows which packages need to be updated (along with any other packages that transitively depended on them).
This is made especially difficult by the fact that changing the content of a file can modify which packages it is considered part of (either by changing the package declaration or the build tags), a file can be in more than one package, and changes can be made to files without using the editor, in which case it will not notify us of the changes.

Inappropriate core functionality Copy Section

The base libraries for Go (things like go/token, go/ast and go/types) are all designed for compiler-like applications.
They tend to worry more about throughput than memory use, they have structures that are intended to grow and then be thrown away at program exit, and they are not designed to keep going in the presence of errors in the source they are handling.
They also have no abilities to do incremental changes.

Making a long running service work well with those libraries is a very large challenge, but writing new libraries would be far more work, and cause a significant long term cost as both sets of libraries would have to be maintained. Right now it is more important to get a working tool into the hands of users. In the long term this decision may have to be revisited, new low level libraries may be the only way to keep pushing the capabilities forwards.

Build system capabilities Copy Section

gopls is supposed to be build system agnostic, but it must use the build system to discover how files map to packages. When it tries to do so, even when the functionality is the same, the costs (in time, CPU and memory) are very different, and can significantly impact the user experience. Designing how gopls interacts with the build system to try to minimize or hide these differences is hard.

Build tags Copy Section

The build tag system in Go is quite powerful, and has many use cases. Source files can exclude themselves using powerful boolean logic on the set of active tags.
It is however designed for specifying the set of active tags on the command line, and the libraries are all designed to cope with only one valid combination at a time. There is also no way to work out the set of valid combinations.

Type checking a file requires knowledge of all the other files in the same package, and that set of files is modified by the build tags. The set of exported identifiers of a package is also affected by which files are in the package, and thus its build tags.

This means that even for files or packages that have no build tag controls it is not possible to produce correct results without knowing the set of build tags to consider.
This makes it very hard to produce useful results when viewing a file.

Features not supported by LSP Copy Section

There are some things it would be good to be able to do that do not fit easily into the existing LSP protocol.
For instance, displaying control flow information, automatic struct tags, complex refactoring...

Each feature will have to be considered carefully, and either propose a change to LSP, or add a way to have gopls specific extensions to the protocol that are still easy to use in all the editor plugins.

To avoid these at the start, only core LSP features will be implemented, as they are sufficient to meet the baseline requirements anyway, but the potential features need to be kept in mind in the core architecture.

Distribution Copy Section

Making sure that users are using the right version of gopls is going to be a problem. Each editor plugin is probably going to install the tools in its own way, some will choose to install it system wide, some will keep their own copy.

Because it is a brand new tool, it will be changing rapidly. If users are not informed they are on an old version they will be experiencing problems that have already been fixed, which is worse for them, and then probably reporting them, which wastes time for the gopls team. There needs to be a mechanism for gopls to check if is up to date, and a recommended way to install an up to date version.

Debugging user problems Copy Section

gopls is essentially a very stateful long running server on the developer's machine. Its basic operation is affected by many things, from the users environment to the contents of the local build cache. The data it is operating on is often a confidential code base that cannot be shared.
All of these things make it hard for users to report a bug usefully, or create a minimal reproduction.

There needs to be easy ways for users to report what information they can, and ways to attempt to reproduce problems without their entire state. This is also needed to produce regression tests.

Basic design decisions Copy Chapter

There are some fundamental architecture decisions that affect much of the rest of the design of the tool, making fundamental trade offs that impact the user experience.

Process lifetime: *managed by the editor* Copy Section

Processing a large code base to fully type check and then analyze it within the latency requirements is not feasible, and is one of the primary problems with the existing solutions. This remains true even if the computed information was cached on disk, as running analyzers and type checkers ends up requiring the full AST of all files in the dependency graph.
It is theoretically possible to do better, but only with a major re-write of the existing parsing and type checking libraries, something that is not feasible at this time.

This implies that gopls should be a long running process, that is able to cache and pre-calculate results in memory so that when a request arrives it can produce the answer much faster.

It could run as a daemon on the user's machine, but there are a lot of issues with managing a daemon. It may well be the right choice in the long term, and it should be allowed for in the fundamental architecture design, but to start with it will instead have a process that lasts as long as the editor that starts it, and that can easily be restarted.

Caching: *in memory* Copy Section

Persistent disk caches are very expensive to maintain, and require solving a lot of extra problems.
Although building the information required is expensive compared to the latencies required of the requests, it is fairly minor compared to the startup times of an editor, so it is expected that rebuilding the information when gopls is restarted will be acceptable.

The advantage gained from this is that gopls becomes stateless across restarts which means if it has issues or gets its state confused, a simple restart will often fix the problem.
It also means that when users report problems, the entire state of the on disk cache is not needed to diagnose and reproduce the issue.

Communication: *stdin/stdout JSON* Copy Section

The LSP specification defines the JSON messages that are normally used, but it does not define how those message should be sent, and there are implementations of the LSP that do not use JSON (for instance, Protocol buffers are an option).

The constraints on gopls are that it must be easy to integrate into every editor on all operating systems, and that it should not have large external dependencies.

JSON is part of the Go standard library, and is also the native language of LSP, so it makes the most sense. By far the best supported communication mechanism is the standard input and output of a process, and the common client implementations all have ways of using JSON rpc 2 in this mode. There were no complete and low dependency implementations of this protocol in Go, but it is a fairly small protocol on top of the JSON library that can be implemented with a moderate effort, and would be a generally useful library to have anyway.

In the future it is expected to run in separated client server mode, so writing it in a way that could use sockets instead of stdin/stdout from the start was the best way to make sure it remained possible. It was also a huge debugging aid to be able to run the gopls server by hand and watch/debug it outside the editor.

Running other tools: *no* Copy Section

Features Copy Chapter

There is a set of features that gopls needs to expose to be a comprehensive IDE solution.
The following is the minimum set of features, along with their existing solutions and how they should map to the LSP.

Introspection Copy Section

Introspection features tell developers information about their code while they work. They do not make or suggest changes.

Navigation Copy Section

Navigation features are designed to make it easier for a developer to find their way round a code base.

Edit assistance Copy Section

These features suggest or apply edits to the code for the user, including refactoring features, for which there are many potential use cases.
Refactoring is one of the places where Go tools could potentially be very strong, but have not been so far, and thus there is huge potential for improvements in the developer experience.
There is not yet a clear understanding of the kinds of refactoring people need or how they should express them however, and there are weaknesses in the LSP protocol around this.
This means it may be much more of a research project.

File: gopls/doc/design/implementation.md Copy Chapter

title: "Gopls: Implementation" Copy Chapter

Last major update: Jan 16 2024

This doc presents a high-level overview of the structure of gopls to
help new contributors find their way. It is not intended to be a
complete description of the implementation, nor even of any key
components; for that, the package documentation (linked below) and
other comments within the code are a better guide.

The diagram below shows selected components of the gopls module and
their relationship to each other according to the Go import graph.
Tests and test infrastructure are not shown, nor are utility packages,
nor packages from the x/tools module. For brevity, packages are
referred to by their last segment, which is usually unambiguous.

The height of each blob corresponds loosely to its technical depth.
Some blocks are wide and shallow, such as protocol, which declares
Go types for the entire LSP protocol. Others are deep, such as cache
and golang, as they contain a lot of dense logic and algorithms.

Starting from the bottom, we'll describe the various components.

The lowest layer defines the request and response types of the
Language Server Protocol:

• The protocol package defines the standard protocol; it is mostly
  generated mechanically from the schema definition provided by
  Microsoft.
  The most important type is DocumentURI, which represents a file:
  URL that identifies a client editor document. It also provides
  Mapper, which maps between the different coordinate systems used
  for source positions: UTF-8, UTF-16, and token.Pos.

• The command package defines Gopls's non-standard commands, which
  are all invoked through the workspace/executeCommand extension
  mechanism. These commands are typically returned by the server as
  continuations of Code Actions or Code Lenses; most clients do not
  construct calls to them directly.

The next layer defines a number of important and very widely used data structures:

• The file package defines the primary abstractions of a client
  file: its Identity (URI and content hash), and its Handle (which
  additionally provides the version and content of a particular
  snapshot of the file.

• The parsego package defines File, the parsed form of a Go source
  file, including its content, syntax tree, and coordinary mappings
  (Mapper and token.File). The package performs various kinds of tree
  repair to work around error-recovery shortcomings of the Go parser.

• The metadata package defines Package, an abstraction of the
  metadata of a Go package, similar to the output of go list -json.
  Metadata is produced from go/packages, which takes
  care of invoking go list. (Users report that it works to some extent
  with a GOPACKAGESDRIVER for Bazel, though we maintain no tests for this
  scenario.)

  The package also provides Graph, the complete import graph for a
  workspace; each graph node is a Package.

The settings layer defines the data structure (effectively a large
tree) for gopls configuration options, along with its JSON encoding.

The cache layer is the largest and most complex component of gopls.
It is concerned with state management, dependency analysis, and invalidation:
the Session of communication with the client;
the Folders that the client has opened;
the View of a particular workspace tree with particular build
options;
the Snapshot of the state of all files in the workspace after a
particular edit operation;
the contents of all files, whether saved to disk (DiskFile) or
edited and unsaved (Overlay);
the Cache of in-memory memoized computations,
such as parsing go.mod files or build the symbol index;
and the Package, which holds the results of type checking a package
from Go syntax.

The cache layer depends on various auxiliary packages, including:

• The filecache package, which manages gopls' persistent, transactional,
  file-based key/value store.

• The xrefs, methodsets, and typerefs packages define algorithms
  for constructing indexes of information derived from type-checking,
  and for encoding and decoding these serializable indexes in the file
  cache.

  Together these packages enable the fast restart, reduced memory
  consumption, and synergy across processes that were delivered by the
  v0.12 redesign and described in "Scaling gopls for the growing Go
  ecosystem".

The cache also defines gopls's go/analysis driver, which runs
modular analysis (similar to go vet) across the workspace.
Gopls also includes a number of analysis passes that are not part of vet.

The next layer defines four packages, each for handling files in a
particular language:
mod for go.mod files;
work for go.work files;
template for files in text/template syntax; and
golang, for files in Go itself.
This package, by far the largest, provides the main features of gopls:
navigation, analysis, and refactoring of Go code.
As most users imagine it, this package is gopls.

The server package defines the LSP service implementation, with one
handler method per LSP request type. Each handler switches on the type
of the file and dispatches to one of the four language-specific
packages.

The lsprpc package connects the service interface to our jsonrpc2 server.

Bear in mind that the diagram is a dependency graph, a "static"
viewpoint of the program's structure. A more dynamic viewpoint would
order the packages based on the sequence in which they are encountered
during processing of a particular request; in such a view, the bottom
layer would represent the "wire" (protocol and command), the next
layer up would hold the RPC-related packages (lsprpc and server), and
features (e.g. golang, mod, work, template) would be at the top.

The cmd package defines the command-line interface of the gopls
command, around which gopls's main package is just a trivial wrapper.
It is usually run without arguments, causing it to start a server and
listen indefinitely.
It also provides a number of subcommands that start a server, make a
single request to it, and exit, providing traditional batch-command
access to server functionality. These subcommands are primarily
provided as a debugging aid; but see https://go.dev/issue/63693.

File: gopls/doc/design/integrating.md Copy Chapter

title: "Documentation for plugin authors" Copy Chapter

If you are integrating gopls into an editor by writing an editor plugin, there are quite a few semantics of the communication between the editor and gopls that are not specified by the LSP specification.

We attempt to document those details along with any other information that has been helpful to other plugin authors here.

If you are implementing a plugin yourself and have questions this page does not answer, please reach out to us to ask, and then also contribute your findings back to this page.

Supported features Copy Chapter

For the most part you should look at the Index of features
to know whether gopls supports a feature.

For a truly authoritative answer you should check the result of the initialize request, where gopls enumerates its support in the ServerCapabilities.

Positions and ranges Copy Chapter

Many LSP requests pass position or range information. This is described in the LSP specification:

A position inside a document (see Position definition below) is expressed as a zero-based line and character offset. The offsets are based on a UTF-16 string representation. So a string of the form a𐐀b the character offset of the character a is 0, the character offset of 𐐀 is 1 and the character offset of b is 3 since 𐐀 is represented using two code units in UTF-16.

This means that integrators will need to calculate UTF-16 based column offsets.
Use protocol.Mapper for all the conversions.

Edits Copy Chapter

In order to deliver changes from gopls to the editor, the LSP supports arrays of TextEdits in responses.
The spec specifies exactly how these should be applied:

All text edits ranges refer to positions in the original document. Text edits ranges must never overlap, that means no part of the original document must be manipulated by more than one edit. However, it is possible that multiple edits have the same start position: multiple inserts, or any number of inserts followed by a single remove or replace edit. If multiple inserts have the same position, the order in the array defines the order in which the inserted strings appear in the resulting text.

All []TextEdit are sorted such that applying the array of deltas received in reverse order achieves the desired result that holds with the spec.

Errors Copy Chapter

Various error codes are described in the LSP specification. We are still determining what it means for a method to return an error; are errors only for low-level LSP/transport issues or can other conditions cause errors to be returned? See some of this discussion on #31526.

The method chosen is currently influenced by the exact treatment in the currently popular editor integrations. It may well change, and ideally would become more coherent across requests.

• textDocument/codeAction: Return error if there was an error computing code actions.
• textDocument/completion: Log errors, return empty result list.
• textDocument/definition: Return error if there was an error computing the definition for the position.
• textDocument/typeDefinition: Return error if there was an error computing the type definition for the position.
• textDocument/formatting: Return error if there was an error formatting the file.
• textDocument/highlight: Log errors, return empty result.
• textDocument/hover: Return empty result.
• textDocument/documentLink: Log errors, return nil result.
• textDocument/publishDiagnostics: Log errors if there were any while computing diagnostics.
• textDocument/references: Log errors, return empty result.
• textDocument/rename: Return error if there was an error computing renames.
• textDocument/signatureHelp: Log errors, return nil result.
• textDocument/documentSymbols: Return error if there was an error computing document symbols.

Watching files Copy Chapter

It is fairly normal for files that affect gopls to be modified outside of the editor it is associated with.

For instance, files that are needed to do correct type checking are modified by switching branches in git, or updated by a code generator.

Monitoring files inside gopls directly has a lot of awkward problems, but the LSP specification has methods that allow gopls to request that the client notify it of file system changes, specifically workspace/didChangeWatchedFiles.
This is currently being added to gopls by a community member, and tracked in #31553

File: gopls/doc/features/diagnostics.md Copy Chapter

title: "Gopls: Diagnostics" Copy Chapter

Gopls continuously annotates all your open files of source code with a
variety of diagnostics. Every time you edit a file or make a
configuration change, gopls asynchronously recomputes these
diagnostics and sends them to the client using the LSP
publishDiagnostics
notification, giving you real-time feedback that reduces the cost of
common mistakes.

Diagnostics come from two main sources: compilation errors and analysis findings.

• Compilation errors are those that you would obtain from running go build. Gopls doesn't actually run the compiler; that would be too
  slow. Instead it runs go list (when needed) to compute the
  metadata of the compilation, then processes those packages in a similar
  manner to the compiler front-end: reading, scanning, and parsing the
  source files, then type-checking them. Each of these steps can
  produce errors that gopls will surface as a diagnostic.

  The source field of the LSP Diagnostic record indicates where
  the diagnostic came from: those with source "go list" come from
  the go list command, and those with source "compiler" come from
  gopls' parsing or type checking phases, which are similar to those
  used in the Go compiler.

  The example above shows a string + int addition, causes the type
  checker to report a MismatchedTypes error. The diagnostic contains
  a link to the documentation about this class of type error.

• Analysis findings come from the Go analysis
  framework, the system
  used by go vet to apply a variety of additional static checks to
  your Go code. The best-known example is the printf
  analyzer,
  which reports calls to fmt.Printf
  where the format "verb" doesn't match the argument, such as
  fmt.Printf("%d", "three").

  Gopls provides dozens of analyzers aggregated from a variety of
  suites; see Analyzers for the complete list. The
  source field of each diagnostic produced by an analyzer records
  the name of the analyzer that produced it.

  The example above shows a printf formatting mistake. The diagnostic contains
  a link to the documentation for the printf analyzer.

There is an optional third source of diagnostics:

• Compiler optimization details are diagnostics that report
  details relevant to optimization decisions made by the Go
  compiler, such as whether a variable escapes or a slice index
  requires a bounds check.

  Optimization decisions include:
  whether a variable escapes, and how escape is inferred;
  whether a nil-pointer check is implied or eliminated; and
  whether a function can be inlined.

  This source is disabled by default but can be enabled on a
  package-by-package basis by invoking the
  source.toggleCompilerOptDetails ("{Show,Hide} compiler optimization
  details") code action.

  Remember that the compiler's optimizer runs only on packages that
  are transitively free from errors, so optimization diagnostics
  will not be shown on packages that do not build.

Recomputation of diagnostics Copy Chapter

By default, diagnostics are automatically recomputed each time the source files
are edited.

Compilation errors in open files are updated after a very short delay
(tens of milliseconds) after each file change, potentially after every keystroke.
This ensures rapid feedback of syntax and type errors while editing.

Compilation and analysis diagnostics for the whole workspace are much
more expensive to compute, so they are usually recomputed after a
short idle period (around 1s) following an edit.

The diagnosticsDelay setting determines
this period.
Alternatively, diagnostics may be triggered only after an edited file
is saved, using the
diagnosticsTrigger setting.

When initialized with "pullDiagnostics": true, gopls also supports
"pull diagnostics",
an alternative mechanism for recomputing diagnostics in which the client
requests diagnostics from gopls explicitly using the textDocument/diagnostic
request. This feature is off by default until the performance of pull
diagnostics is comparable to push diagnostics.

Quick fixes Copy Chapter

Each analyzer diagnostic may suggest one or more alternative
ways to fix the problem by editing the code.
For example, when a return statement has too few operands,
the fillreturns analyzer
suggests a fix that heuristically fills in the missing ones
with suitable values. Applying the fix eliminates the compilation error.

The screenshot above shows VS Code's Quick Fix menu for an "unused
parameter" analysis diagnostic with two alternative fixes.
(See Remove unused parameter for more detail.)

Suggested fixes that are indisputably safe are code
actions whose kind is
"source.fixAll".
Many client editors have a shortcut to apply all such fixes.

TODO(adonovan): audit all the analyzers to ensure that their
documentation is up-to-date w.r.t. any fixes they suggest.

Settings:

• The diagnosticsDelay setting determines
  the idle period after an edit before diagnostics are recomputed.
• The diagnosticsTriggerr setting determines
  what events cause recomputation of diagnostics.
• The linkTarget setting specifies
  the base URI for Go package links in the Diagnostic.CodeDescription field.

Client support:

• VS Code: Each diagnostic appears as a squiggly underline.
  Hovering reveals the details, along with any suggested fixes.
• Emacs + eglot: Each diagnostic appears as a squiggly underline.
  Hovering reveals the details. Use M-x eglot-code-action-quickfix
  to apply available fixes; it will prompt if there are more than one.
• Vim + coc.nvim: ??
• CLI: gopls check file.go

`stubMissingInterfaceMethods`: Declare missing methods of I Copy Section

When a value of a concrete type is assigned to a variable of an
interface type, but the concrete type does not possess all the
necessary methods, the type checker will report a "missing method"
error.

In this situation, gopls offers a quick fix to add stub declarations
of all the missing methods to the concrete type so that it implements
the interface.

For example, this function will not compile because the value
NegativeErr{} does not implement the "error" interface:

func sqrt(x float64) (float64, error) {
	if x < 0 {
		return 0, NegativeErr{} // error: missing method
	}
	...
}

type NegativeErr struct{}

Gopls will offer a quick fix to declare this method:

// Error implements [error.Error].
func (NegativeErr) Error() string {
	panic("unimplemented")
}

Beware that the new declarations appear alongside the concrete type,
which may be in a different file or even package from the cursor
position.
(Perhaps gopls should send a showDocument request to navigate the
client there, or a progress notification indicating that something
happened.)

`StubMissingCalledFunction`: Declare missing method T.f Copy Section

When you attempt to call a method on a type that does not have that method,
the compiler will report an error such as "type X has no field or method Y".
In this scenario, gopls now offers a quick fix to generate a stub declaration of
the missing method, inferring its type from the call.

Consider the following code where Foo does not have a method bar:

type Foo struct{}

func main() {
  var s string
  f := Foo{}
  s = f.bar("str", 42) // error: f.bar undefined (type Foo has no field or method bar)
}

Gopls will offer a quick fix, "Declare missing method Foo.bar".
When invoked, it creates the following declaration:

func (f Foo) bar(s string, i int) string {
	panic("unimplemented")
}

`CreateUndeclared`: Create missing declaration for "undeclared name: X" Copy Section

A Go compiler error "undeclared name: X" indicates that a variable or function is being used before
it has been declared in the current scope. In this scenario, gopls offers a quick fix to create the declaration.

Declare a new variable

When you reference a variable that hasn't been declared:

func main() {
  x := 42
  min(x, y) // error: undefined: y
}

The quick fix would insert a declaration with a default
value inferring its type from the context:

func main() {
  x := 42
  y := 0
  min(x, y)
}

Declare a new function

Similarly, if you call a function that hasn't been declared:

func main() {
  var s string
  s = doSomething(42) // error: undefined: doSomething
}

Gopls will insert a new function declaration below,
inferring its type from the call:

func main() {
  var s string
  s = doSomething(42)
}

func doSomething(i int) string {
  panic("unimplemented")
}

2. Official Technical Reference & Guides (golang/website) Copy Chapter

Go website

This repo holds content and serving programs for the go.dev and golang.org web sites.

Content is in _content/ (go.dev) and tour/ (go.dev/tour).
Server code is in cmd/ and internal/.

To run the combined go.dev+golang.org server to preview local content changes, use:

go run ./cmd/golangorg

The supporting programs cmd/admingolangorg and cmd/googlegolangorg
are the servers for admin.golang.org and google.golang.org.
(They do not use the _content/ directories.)

Each command directory has its own README.md explaining deployment.

JS/TS/CSS Formatting Copy Chapter

This repository uses eslint to format JS and TS files,
and stylelint to format CSS files.

See also:

• CSS
• JavaScript
• TypeScript

It is encouraged that all JS, TS, and CSS code be run through formatters before
submitting a change. However, it is not a strict requirement enforced by CI.

Installing npm Dependencies: Copy Section

1. Install docker
2. Create a .gitignore file at repo root
3. Add .gitignore and node_modules to .gitignore
4. Run ./npm install

Run ESlint Copy Section

./npx eslint [options] [file] [dir]

Run Stylelint Copy Section

./npx stylelint [input] [options]

TypeScript Support Copy Chapter

TypeScript files served from _content are transformed into JavaScript.
Reference .ts files in html templates as module code.

<script type="module" src="/ts/filename.ts">

Write unit tests for TypeScript code using the jest
testing framework.

Run Jest Copy Section

./npx jest [TestPathPattern]

Deploying Copy Chapter

Each time a CL is reviewed and submitted, the code is deployed to App Engine.
See cmd/golangorg/README.md for details.

Report Issues / Send Patches Copy Chapter

This repository uses Gerrit for code changes. To learn how to submit changes to
this repository, see https://go.dev/doc/contribute.

The git repository is https://go.googlesource.com/website.

The main issue tracker for the website repository is located at
https://go.dev/issues. Prefix your issue with "x/website:" in the
subject line, so it is easy to find.