protobuf Documentation

Technical Documentation: protocolbuffers/protobuf

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Protocol Buffers - Google's data interchange format

Copyright 2008 Google LLC

Overview Copy Chapter

Protocol Buffers (a.k.a., protobuf) are Google's language-neutral,
platform-neutral, extensible mechanism for serializing structured data. You
can learn more about it in protobuf's documentation.

This README file contains protobuf installation instructions. To install
protobuf, you need to install the protocol compiler (used to compile .proto
files) and the protobuf runtime for your chosen programming language.

Working With Protobuf Source Code Copy Chapter

Most users will find working from
supported releases to be
the easiest path.

If you choose to work from the head revision of the main branch your build will
occasionally be broken by source-incompatible changes and insufficiently-tested
(and therefore broken) behavior.

If you are using C++ or otherwise need to build protobuf from source as a part
of your project, you should pin to a release commit on a release branch.

This is because even release branches can experience some instability in between
release commits.

Bazel with Bzlmod Copy Section

Protobuf supports
Bzlmod with Bazel 8 +.
Users should specify a dependency on protobuf in their MODULE.bazel file as
follows.

bazel_dep(name = "protobuf", version = <VERSION>)

Users can optionally override the repo name, such as for compatibility with
WORKSPACE.

bazel_dep(name = "protobuf", version = <VERSION>, repo_name = "com_google_protobuf")

Bazel with WORKSPACE Copy Section

Users can also add the following to their legacy
WORKSPACE file.

Note that with the release of 30.x there are a few more load statements to
properly set up rules_java and rules_python.

http_archive(
    name = "com_google_protobuf",
    strip_prefix = "protobuf-VERSION",
    sha256 = ...,
    url = ...,
)

load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")

protobuf_deps()

load("@rules_java//java:rules_java_deps.bzl", "rules_java_dependencies")

rules_java_dependencies()

load("@rules_java//java:repositories.bzl", "rules_java_toolchains")

rules_java_toolchains()

load("@rules_python//python:repositories.bzl", "py_repositories")

py_repositories()

Protobuf Compiler Installation Copy Chapter

The protobuf compiler is written in C++. If you are using C++, please follow
the C++ Installation Instructions to install protoc along
with the C++ runtime.

For non-C++ users, the simplest way to install the protocol compiler is to
download a pre-built binary from our GitHub release page.

In the downloads section of each release, you can find pre-built binaries in
zip packages: protoc-$VERSION-$PLATFORM.zip. It contains the protoc binary
as well as a set of standard .proto files distributed along with protobuf.

If you are looking for an old version that is not available in the release
page, check out the Maven repository.

These pre-built binaries are only provided for released versions. If you want
to use the github main version at HEAD, or you need to modify protobuf code,
or you are using C++, it's recommended to build your own protoc binary from
source.

If you would like to build protoc binary from source, see the C++ Installation Instructions.

Protobuf Runtime Installation Copy Chapter

Protobuf supports several different programming languages. For each programming
language, you can find instructions in the corresponding source directory about
how to install protobuf runtime for that specific language:

Quick Start Copy Chapter

The best way to learn how to use protobuf is to follow the tutorials in our
developer guide.

If you want to learn from code examples, take a look at the examples in the
examples directory.

Documentation Copy Chapter

The complete documentation is available at the Protocol Buffers doc site.

Support Policy Copy Chapter

Read about our version support policy
to stay current on support timeframes for the language libraries.

Developer Community Copy Chapter

To be alerted to upcoming changes in Protocol Buffers and connect with protobuf developers and users,
join the Google Group.

Overview (`.bazelci/README.md`) Copy Section

Bazel CI test

The .bazelci/presubmit.yml file is used by https://buildkite.com/bazel/protobuf
to test building protobuf with Bazel on Bazel CI. It should contain the same
set of tests as .bcr/presubmit.yml which is the tests that will run before
publishing protobuf to BCR.

Overview (`.bcr/README.md`) Copy Section

Bazel Central Registry

When protobuf is released, we want it to be published to the Bazel Central
Registry automatically: https://registry.bazel.build

This folder contains configuration files to automate the publish step. See
https://github.com/bazel-contrib/publish-to-bcr/blob/main/templates/README.md
for authoritative documentation about these files.

Overview (`ci/README.md`) Copy Section

This directory contains CI-specific tooling.

Clang wrappers

CMake allows for compiler wrappers to be injected such as ccache, which
intercepts compiler calls and short-circuits on cache-hits. This can be done
by specifying CMAKE_C_COMPILER_LAUNCHER and CMAKE_CXX_COMPILER_LAUNCHER
during CMake's configure step. Unfortunately, X-Code doesn't provide anything
like this, so we use basic wrapper scripts to invoke ccache + clang.

Bazelrc files

In order to allow platform-specific .bazelrc flags during testing, we keep
3 different versions here along with a shared common.bazelrc that they all
include. Our GHA infrastructure will select the appropriate file for any test
and overwrite the default .bazelrc in our workspace, which is intended for
development only.

Overview (`cmake/README.md`) Copy Section

This directory contains CMake files that can be used to build protobuf.

You need to have CMake and Git
installed on your computer before proceeding. If you don't have
Abseil installed, it will be fetched
from GitHub. We currently support CMake 3.22 and newer.

Most of the instructions will be given using CMake's command-line interface, but
the same actions can be performed using appropriate GUI tools.

Getting Sources

You can get the latest stable source packages from the release page:

https://github.com/protocolbuffers/protobuf/releases/latest

Or you can use git to clone the protobuf git repository.

git clone -b [release_tag] https://github.com/protocolbuffers/protobuf.git

Where [release_tag] is a git tag like v32.0-rc1 or a branch name like main
if you want to get the latest code.

Basic Build

It is considered good practice not to build CMake projects in the source tree
but in a separate folder. The following commands show a basic, cross-platform
way to configure, build, and install protobuf.

# From the protobuf source directory
cmake -S . -B build \
  -DCMAKE_INSTALL_PREFIX=../install \
  -DCMAKE_BUILD_TYPE=Release

# Compile the code
cmake --build build --parallel 10

# Run tests
ctest --test-dir build --verbose

# Install the libraries and headers
cmake --install build

This will install the compiled libraries, headers, and protoc binary into the
../install directory, relative to the source root.

CMake Configuration Flags

The following flags can be passed to cmake during the configuration step
(e.g., cmake -S . -B build -D<FLAG>=<VALUE>) to customize the build.

C++ Version Copy Chapter

By default, CMake will use whatever C++ version is the system default. Since
protobuf requires C++17 or newer, sometimes you will need to explicitly override
this.

# Configure protobuf to be built with C++17
cmake . -DCMAKE_CXX_STANDARD=17

See the CMake documentation for
CXX_STANDARD{.external}
for all available options.

Dependency Management Copy Chapter

Abseil and Google Test Copy Section

During configuration, you can optionally specify where CMake should expect to
find your Abseil, Google Test, and jsoncpp installations, if they are installed.
If they aren't installed, they will be fetched from GitHub. To specify the
locations, set -DCMAKE_PREFIX_PATH to the path where you installed them.

# Path to where Abseil and GTest are installed
cmake . -DCMAKE_PREFIX_PATH=/path/to/my/dependencies

If the installation of a dependency can't be found, CMake will default to
downloading and building a copy from GitHub. To prevent this and make it an
error condition, you can optionally set:
-Dprotobuf_LOCAL_DEPENDENCIES_ONLY=ON.

Enabling Tests Copy Section

To enable building the unit tests, set the following flag:

• -Dprotobuf_BUILD_TESTS=ON

ZLib Support Copy Section

If you want to include GzipInputStream and GzipOutputStream in libprotobuf,
you need to have ZLib installed. Ensure ZLib headers and libraries are in a
standard system location or a custom install prefix.

If ZLib is installed in a non-standard location, you can help CMake find it by
setting:

• -DZLIB_INCLUDE_DIR=/path/to/zlib/headers
• -DZLIB_LIBRARY=/path/to/zlib/library.lib

Build Options Copy Chapter

DLLs vs. Static Linking Copy Section

Static linking is the default. To build shared libraries (DLLs on Windows), add
the following flag during configuration:

• -Dprotobuf_BUILD_SHARED_LIBS=ON

When you build your own project against protobuf, you must also define #define PROTOBUF_USE_DLLS. For more platform-specific details, see the
Windows Builds section.

Compiling

The standard, cross-platform way to compile a CMake project is:

cmake --build <build_directory>

For example:

cmake --build build --parallel 10

If your generator supports multiple configurations (like Visual Studio), you
must also specify which one to build:

cmake --build build --config Release

Testing

To run unit tests, first compile protobuf as described above. Then run ctest:

ctest --test-dir build --progress --output-on-failure

You can also build the check target, which will compile and run the tests:

cmake --build build --target check

To run specific tests, you need to pass arguments to the test program itself,
which requires finding the test executable in your build directory.

Installing

To install protobuf to the directory specified by CMAKE_INSTALL_PREFIX during
configuration, build the install target:

cmake --build <build_directory> --target install

This will create the following folders under the install location:

• bin - contains protobuf protoc compiler
• include - contains C++ headers and .proto files
• lib - contains linking libraries and CMake package configuration files.

Platform-Specific Details

Windows Builds {#windows-builds} Copy Chapter

On Windows, you can build using Visual Studio's command-line tools or IDE.

Generators Copy Section

Of most interest to Windows programmers are the following
generators:

• Visual Studio: Generates a multi-configuration .sln file. Example: -G "Visual Studio 16 2019".
• Ninja: Uses the external tool Ninja to
  build. This is often the fastest solution.

Environment Setup Copy Section

Open the appropriate Command Prompt from the Start menu (e.g., x86 Native
Tools Command Prompt for VS 2019) to ensure cl.exe and other build tools are
in your PATH.

DLLs vs. Static Linking on Windows Copy Section

While shared libraries can be built, static linking is strongly recommended on
Windows. This is due to issues with Win32's use of a separate heap for each DLL
and binary compatibility issues between different versions of MSVC's STL
library.

If you are distributing your software, do NOT install libprotobuf.dll or
libprotoc.dll to a shared location. Keep them in your application's own
install directory.

Notes on Compiler Warnings Copy Section

The following MSVC warnings have been disabled while building the protobuf
libraries. You may need to disable them in your own project as well.

• C4065 - # switch statement contains 'default' but no 'case' labels
• C4146 - # unary minus operator applied to unsigned type
• C4244 - # 'conversion' conversion from 'type1' to 'type2', possible loss
  of data
• C4251 - # 'identifier' : class 'type' needs to have dll-interface to be
  used by clients of class 'type2'
• C4267 - # 'var' : conversion from 'size_t' to 'type', possible loss of
  data
• C4305 - # 'identifier' : truncation from 'type1' to 'type2'
• C4307 - # 'operator' : integral constant overflow
• C4309 - # 'conversion' : truncation of constant value
• C4334 - # 'operator' : result of 32-bit shift implicitly converted to 64
  bits (was 64-bit shift intended?)
• C4355 - # 'this' : used in base member initializer list
• C4506 - # no definition for inline function 'function'
• C4800 - # 'type' : forcing value to bool 'true' or 'false' (performance
  warning)
• C4996 - # The compiler encountered a deprecated declaration.

C4251 is particularly notable if you build protobuf as a DLL. See the old
version of this document for a longer explanation if needed.

Linux Builds Copy Chapter

Building with CMake works very similarly on Linux. You will need to have gcc
or clang installed. CMake will generate Makefiles by default, but can also be
configured to use Ninja.

After building, you can install the files system-wide using sudo:

sudo cmake --install build

Or directly with Makefiles:

cd build
 sudo make install

Overview (`conformance/README.md`) Copy Section

Protocol Buffers - Google's data interchange format

Copyright 2008 Google LLC.

This directory contains conformance tests for testing completeness and
correctness of Protocol Buffers implementations. These tests are designed
to be easy to run against any Protocol Buffers implementation.

This directory contains the tester process conformance-test, which
contains all of the tests themselves. Then separate programs written
in whatever language you want to test communicate with the tester
program over a pipe.

If you're not using Bazel to run these tests, make sure you build the C++
tester code beforehand, e.g. from the base directory:

$ cmake . -Dprotobuf_BUILD_CONFORMANCE=ON && cmake --build .

This will produce a conformance_test_runner binary that can be used to run
conformance tests on any executable. Pass it --help for more information.

Running the tests for C++ Copy Chapter

To run the tests against the C++ implementation, run:

$ bazel test //src:conformance_test

Or alternatively with CMake:

$ ctest -R conformance_cpp_test

Running the tests for other languages Copy Chapter

All of the languages in the Protobuf source tree are set up to run conformance
tests using similar patterns. You can either use Bazel to run the
conformance_test target defined in the language's root BUILD.bazel file,
or create an executable for a custom test and pass it to
conformance_test_runner.

Note: CMake can be used to build the conformance test runner, but not any of
the conformance test executables outside C++. So if you aren't using Bazel
you'll need to create the executable you pass to conformance_test_runner via
some alternate build system.

While we plan to model all our supported languages more completely in Bazel,
today some of them are a bit tricky to run. Below is a list of the commands
(and prerequisites) to run each language's conformance tests.

C#:

$ which dotnet || echo "You must have dotnet installed!"
$ bazel test //csharp:conformance_test \
    --action_env=DOTNET_CLI_TELEMETRY_OPTOUT=1 --test_env=DOTNET_CLI_HOME=~ \
    --action_env=DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1

Java:

$ bazel test //java/core:conformance_test //java/lite:conformance_test

Objective-C (Mac only):

$ bazel test //objectivec:conformance_test --macos_minimum_os=12.0

PHP:

$ bazel test //php:conformance_test

PHP (C):

$ which gcc     || echo "gcc is required!"
$ which libtool || echo "libtool is required!"
$ which make    || echo "make is required!"
$ which pear    || echo "pear is required! It might require a development version of PHP such as a php-dev package"
$ which pecl    || echo "pecl is required! It might require a development version of PHP such as a php-dev package"
$ which phpize  || echo "phpize is required! It might require a development version of PHP such as a php-dev package"
$ bazel test //php:conformance_test_c

Python:

$ bazel test //python:conformance_test

Python (C++):

$ bazel test //python:conformance_test_cpp --define=use_fast_cpp_protos=true

Ruby:

$ [[ $(ruby --version) == "ruby"* ]] || echo "Select a C Ruby!"
$ bazel test //ruby:conformance_test --define=ruby_platform=c \
    --action_env=PATH --action_env=GEM_PATH --action_env=GEM_HOME

Ruby (JRuby):

$ [[ $(ruby --version) == "jruby"* ]] || echo "Switch to Java Ruby!"
$ bazel test //ruby:conformance_test_jruby --define=ruby_platform=java \
    --action_env=PATH --action_env=GEM_PATH --action_env=GEM_HOME

Testing other Protocol Buffer implementations Copy Chapter

To run these tests against a new Protocol Buffers implementation, write a
program in your language that uses the protobuf implementation you want
to test. This program should implement the testing protocol defined in
conformance.proto.
This is designed to be as easy as possible: the C++ version is only
150 lines and is a good example for what this program should look like
(see conformance_cpp.cc).
The program only needs to be able to read from stdin and write to stdout.

Portability Copy Chapter

Note that the test runner currently does not work on Windows. Patches
to fix this are welcome! (But please get in touch first to settle on
a general implementation strategy).

Overview (`csharp/README.md`) Copy Section

This directory contains the C# Protocol Buffers runtime library.

Usage

The easiest way how to use C# protobufs is via the Google.Protobuf
NuGet package. Just add the NuGet package to your VS project.

You will also want to install the Google.Protobuf.Tools NuGet package, which
contains precompiled version of protoc.exe and a copy of well known .proto
files under the package's tools directory.

To generate C# files from your .proto files, invoke protoc with the
--csharp_out option.

Supported platforms

The runtime library is built as a class library, supporting targets of:

• .NET 4.5+ (net45)
• .NET Standard 1.1 and 2.0 (netstandard1.1 and netstandard2.0)
• .NET 5+ (net50)

You should be able to use Protocol Buffers in Visual Studio 2012 and
all later versions. This includes all code generated by protoc,
which only uses features from C# 3 and earlier. When compiling generated
code with old compilers (before C# 7.2) you need to define the
GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE symbol in your project
so that the generated classes don't implement IBufferMessage, which uses
ref struct types.

Building

Open the src/Google.Protobuf.sln solution in Visual Studio 2022 or
later.

Although users of this project are only expected to have Visual
Studio 2012 or later, developers of the library are required to
have Visual Studio 2022 or later, as the library uses C# 10 features
in its implementation and runs tests under .NET 6. These features
have no impact when using the compiled code - they're only relevant
when building the Google.Protobuf assembly.

Testing

The unit tests use NUnit 3. Tests can be
run using the Visual Studio Test Explorer or dotnet test.

.NET 3.5

We don't support .NET 3.5. It used to be feasible to build this library
targeting .NET 3.5, but a number of changes requiring newer runtime/framework
features have been added over time. While it would no doubt be possible to
rework the current implementation to allow most of the functionality to be built
in .NET 3.5, this would create an undue maintenance burden.

History of C# protobufs

This subtree was originally imported from https://github.com/jskeet/protobuf-csharp-port
and represents the latest development version of C# protobufs, that will now be developed
and maintained by Google. All the development will be done in open, under this repository
(https://github.com/protocolbuffers/protobuf).

The previous project differs from this project in a number of ways:

• The old code only supported proto2; the new code initially only supported
  proto3 (so no unknown fields, no required/optional distinction, no
  extensions); since then proto2 support has been added
• The old code was based on immutable message types and builders for
  them
• The old code did not support maps or oneof
• The old code had its own JSON representation, whereas the new code
  uses the standard protobuf JSON representation
• The old code had no notion of the "well-known types" which have
  special support in the new code
• The old project supported some older platforms (such as older
  versions of Silverlight) which are not currently supported in the
  new project

Overview (`examples/README.md`) Copy Section

Protocol Buffers - Code Example

This directory contains example code that uses Protocol Buffers to manage an
address book. Two programs are provided for each supported language. The
add_person example adds a new person to an address book, prompting the user to
input the person's information. The list_people example lists people already in
the address book. The examples use the exact same format in all three languages,
so you can, for example, use add_person_java to create an address book and then
use list_people_python to read it.

These examples are part of the Protocol Buffers tutorial, located at:
https://developers.google.com/protocol-buffers/docs/tutorials

Build the example using bazel Copy Chapter

The example requires bazel 0.5.4 or newer to build. You can download/install
the latest version of bazel from bazel's release page:

https://github.com/bazelbuild/bazel/releases

Once you have bazel installed, simply run the following command in this examples
directory to build the code:

$ bazel build :all

Then you can run the built binary:

$ bazel-bin/add_person_cpp addressbook.data

To use protobuf in your own bazel project, please follow instructions in the
BUILD.bazel file and WORKSPACE file.

Build the example using make Copy Chapter

You must install the protobuf package before you can build it using make. The
minimum requirement is to install protocol compiler (i.e., the protoc binary)
and the protobuf runtime for the language you want to build.

You can simply run "make" to build the example for all languages (except for
Go). However, since different languages have different installation requirements,
it will likely fail. It's better to follow individual instructions below to
build only the language you are interested in.

C++ Copy Section

You can follow instructions in ../src/README.md to install
protoc from source.

Then run "make cpp" in this examples directory to build the C++ example. It
will create two executables: add_person_cpp and list_people_cpp. These programs
simply take an address book file as their parameter. The add_person_cpp
programs will create the file if it doesn't already exist.

To run the examples:

$ ./add_person_cpp addressbook.data
$ ./list_people_cpp addressbook.data

Note that on some platforms you may have to edit the Makefile and remove
"-lpthread" from the linker commands (perhaps replacing it with something else).
We didn't do this automatically because we wanted to keep the example simple.

Python Copy Section

Follow instructions in ../README.md to install protoc and then
follow ../python/README.md to install protobuf python
runtime from source. You can also install python runtime using pip:

$ pip install protobuf

Make sure the runtime version is the same as protoc binary, or it may not work.

After you have install both protoc and python runtime, run "make python" to
build two executables (shell scripts actually): add_person_python and
list_people_python. They work the same way as the C++ executables.

Java Copy Section

Follow instructions in ../README.md to install protoc and then
download protobuf Java runtime .jar file from maven:

https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java

Then run the following:

$ export CLASSPATH=/path/to/protobuf-java-[version].jar
$ make java

This will create the add_person_java/list_people_java executables (shell
scripts) and can be used to create/display an address book data file.

Go Copy Section

Follow instructions in ../README.md to install protoc. Then
install the Go protoc plugin (protoc-gen-go):

$ go install google.golang.org/protobuf/cmd/protoc-gen-go@latest

The "go install" command will install protoc-gen-go into the GOBIN
directory. You can set the $GOBIN environment variable before
running "go install" to change the install location. Make sure the
install directory is in your shell $PATH.

Build the Go samples with "make go". This creates the following
executable files in the current directory:

add_person_go      list_people_go

To run the example:

./add_person_go addressbook.data

to add a person to the protocol buffer encoded file addressbook.data. The file
is created if it does not exist. To view the data, run:

./list_people_go addressbook.data

Observe that the C++, Python, Java, and Dart examples in this directory run in a
similar way and can view/modify files created by the Go example and vice
versa.

Dart Copy Section

First, follow the instructions in ../README.md to install the Protocol Buffer Compiler (protoc).

Then, install the Dart Protocol Buffer plugin as described here.
Note, the executable bin/protoc-gen-dart must be in your PATH for protoc to find it.

Build the Dart samples in this directory with make dart.

To run the examples:

$ dart add_person.dart addressbook.data
$ dart list_people.dart addressbook.data

The two programs take a protocol buffer encoded file as their parameter.
The first can be used to add a person to the file. The file is created
if it does not exist. The second displays the data in the file.

Overview (`hpb_generator/README.md`) Copy Section

hpb Generator

This directory contains the generator for the hpb
API, an
experimental C++ protobuf implementation. Most users should use the standard
C++ implementation
here.

Overview (`java/README.md`) Copy Section

Protocol Buffers - Google's data interchange format

Copyright 2008 Google Inc.

https://developers.google.com/protocol-buffers/

Use Java Protocol Buffers Copy Chapter

To use protobuf in Java, first obtain the protocol compiler (a.k.a., protoc,
see instructions in the toplevel README.md) and use it to
generate Java code for your .proto files:

$ protoc --java_out=${OUTPUT_DIR} path/to/your/proto/file

Include the generated Java files in your project and add a dependency on the
protobuf Java runtime.

Maven Copy Section

If you are using Maven, use the following:

<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-java</artifactId>
  <version></version>
</dependency>

And replace `` with a version from the
Maven Protocol Buffers Repository.
For example, 4.28.2.

Make sure the version number of the runtime matches (or is newer than) the
version number of the protoc.

If you want to use features like protobuf JsonFormat, add a dependency on the
protobuf-java-util package:

<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-java-util</artifactId>
  <version></version>
</dependency>

Use Java Protocol Buffers on Android Copy Section

For Android users, it's recommended to use protobuf Java Lite runtime because
of its smaller code size. Java Lite runtime also works better with Proguard
because it doesn't rely on Java reflection and is optimized to allow as much
code stripping as possible. You can following these instructions to use Java
Lite runtime.

Use Java Protocol Buffers with Bazel Copy Section

Bazel has native build rules to work with protobuf. For Java, you can use the
java_proto_library rule for server and the java_lite_proto_library rule for
Android. Check out our build files examples to learn
how to use them.

Build from Source Copy Chapter

Most users should follow the instructions above to use protobuf Java runtime.
If you are contributing code to protobuf or want to use a protobuf version
that hasn't been officially released yet, you can follow the instructions
below to build protobuf from source code.

Build from Source Copy Section

You may follow these instructions to build from source. This does not require
Maven to be installed. Note that these instructions skip running unit tests and
only describes how to install the core protobuf library (without the util
package).

1. Build the C++ code, or obtain a binary distribution of protoc (see
   the toplevel README.md). If you install a binary
   distribution, make sure that it is the same version as this package.
   If in doubt, run:

   $ protoc --version

   If you built the C++ code without installing, the compiler binary
   should be located in ../src.

2. Invoke protoc to build DescriptorProtos.java:

   $ protoc --java_out=core/src/main/java -I../src
   ../src/google/protobuf/descriptor.proto

3. Compile the code in core/src/main/java using whatever means you prefer.

4. Install the classes wherever you prefer.

Kotlin Protocol Buffers Copy Chapter

This directory also provides support for Kotlin protocol buffers, which are
built on top of Java protocol buffers. Kotlin protocol buffers require a
dependency on Java protocol buffers, and both Java and Kotlin protocol buffer
code must be generated for every proto file.

The main goal of Kotlin protobuf is to provide idiomatic ways to build and read
protocol buffers in Kotlin. Learn more about Kotlin protobufs in our
documentation.

Use Kotlin Protocol Buffers Copy Section

To use protobuf in Kotlin, first install the protocol compiler (protoc --
instructions for installing are in the top-level README.md) and
use it to generate Java and Kotlin code for your .proto files:

$ protoc --java_out=${OUTPUT_DIR} --kotlin_out=${OUTPUT_DIR} path/to/your/proto/file

Include the generated Java and Kotlin files in your project and add a dependency on the
protobuf Java and Kotlin runtime.

Maven Copy Section

If you are using Maven, use the following:

<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-java</artifactId>
  <version></version>
</dependency>

<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-kotlin</artifactId>
  <version></version>
</dependency>

Replace `` with a version from the
Maven Protocol Buffers Repository,
such as 4.28.2.

Make sure the version number of the runtimes match each other and match (or are
newer than) the version number of the protoc.

Use Kotlin Protocol Buffers on Android Copy Section

For Android users, it's recommended to use the Java Lite runtime for its smaller
code size. We provide a protobuf-kotlin-lite package in Maven and Bazel to
pair with the Java Lite runtime. Use these if you want to use Kotlin on
Android or in another context where you want to use Java Lite. Similar to the
full runtime, protobuf-kotlin-lite requires a dependency on
protobuf-java-lite.

Compatibility Notice Copy Chapter

• Protobuf minor version releases are backwards-compatible. If your code
  can build/run against the old version, it's expected to build/run against
  the new version as well. Both binary compatibility and source compatibility
  are guaranteed for minor version releases if the user follows the guideline
  described in this section.

• Protobuf major version releases may also be backwards-compatible with the
  last release of the previous major version. See the release notice for more
  details.

• APIs marked with the @ExperimentalApi annotation are subject to change. They
  can be modified in any way, or even removed, at any time. Don't use them if
  compatibility is needed. If your code is a library itself (i.e. it is used on
  the CLASSPATH of users outside your own control), you should not use
  experimental APIs, unless you repackage them (e.g. using ProGuard).

• Deprecated non-experimental APIs will be removed two years after the release
  in which they are first deprecated. You must fix your references before this
  time. If you don't, any manner of breakage could result (you are not
  guaranteed a compilation error).

• Protobuf message interfaces/classes are designed to be subclassed by protobuf
  generated code only. Do not subclass these message interfaces/classes
  yourself. We may add new methods to the message interfaces/classes which will
  break your own subclasses.

• Don't use any method/class that is marked as "used by generated code only".
  Such methods/classes are subject to change.

• Protobuf LITE runtime APIs are not stable yet. They are subject to change even
  in minor version releases.

Documentation Copy Chapter

The complete documentation for Protocol Buffers is available via the
web at:

https://developers.google.com/protocol-buffers/

Overview (`lua/README.md`) Copy Section

upb Lua bindings

These are some bare-bones upb bindings for Lua.

These bindings exist primarily for experimentation and testing.
They are incomplete and are not really intended for use in any application.
This is by no means a complete or supported protobuf library, and in fact
we don't even claim it to be functional.

Overview (`objectivec/README.md`) Copy Section

Protocol Buffers - Google's data interchange format

Copyright 2008 Google Inc.

This directory contains the Objective C Protocol Buffers runtime library.

Requirements Copy Chapter

The Objective C implementation requires:

• Objective C 2.0 Runtime (32bit & 64bit iOS, 64bit OS X).
• Xcode 13.3.1 (or later).
• The library code does not use ARC (for performance reasons), but it all can
  be called from ARC code.

Installation Copy Chapter

The distribution pulled from github includes the sources for both the
compiler (protoc) and the runtime (this directory). After cloning the distribution
and needed submodules (see the src directory's README),
to build the compiler and run the runtime tests, you can use:

$ objectivec/DevTools/full_mac_build.sh

This will generate the protoc binary.

Building Copy Chapter

There are two ways to include the Runtime sources in your project:

Add objectivec/*.h, objectivec/google/protobuf/*.pbobjc.h, and
objectivec/GPBProtocolBuffers.m to your project.

or

Add objectivec/*.h, objectivec/google/protobuf/*.pbobjc.h,
objectivec/google/protobuf/*.pbobjc.m, and objectivec/*.m except for
objectivec/GPBProtocolBuffers.m to your project.

If the target is using ARC, remember to turn off ARC (-fno-objc-arc) for the
.m files.

The files generated by protoc for the *.proto files (*.pbobjc.h and
*.pbobjc.m) are then also added to the target.

Usage Copy Chapter

The objects generated for messages should work like any other Objective C
object. They are mutable objects, but if you don't change them, they are safe
to share between threads (similar to passing an NSMutableDictionary between
threads/queues; as long as no one mutates it, things are fine).

There are a few behaviors worth calling out:

A property that is type NSString* will never return nil. If the value is
unset, it will return an empty string (@""). This is inpart to align things
with the Protocol Buffers spec which says the default for strings is an empty
string, but also so you can always safely pass them to isEqual:/compare:, etc.
and have deterministic results.

A property that is type NSData* also won't return nil, it will return an empty
data ([NSData data]). The reasoning is the same as for NSString not returning
nil.

A property that is another GPBMessage class also will not return nil. If the
field wasn't already set, you will get a instance of the correct class. This
instance will be a temporary instance unless you mutate it, at which point it
will be attached to its parent object. We call this pattern autocreators.
Similar to NSString and NSData properties it makes things a little safer when
using them with isEqual:/etc.; but more importantly, this allows you to write
code that uses Objective C's property dot notation to walk into nested objects
and access and/or assign things without having to check that they are not nil
and create them each step along the way. You can write this:

- (void)updateRecord:(MyMessage *)msg {
  ...
  // Note: You don't have to check subMessage and otherMessage for nil and
  // alloc/init/assign them back along the way.
  msg.subMessage.otherMessage.lastName = @"Smith";
  ...
}

If you want to check if a GPBMessage property is present, there is always as
has\[NAME\] property to go with the main property to check if it is set.

A property that is of an Array or Dictionary type also provides autocreator
behavior and will never return nil. This provides all the same benefits you
see for the message properties. Again, you can write:

- (void)updateRecord:(MyMessage *)msg {
  ...
  // Note: Just like above, you don't have to check subMessage and otherMessage
  // for nil and alloc/init/assign them back along the way. You also don't have
  // to create the siblingsArray, you can safely just append to it.
  [msg.subMessage.otherMessage.siblingsArray addObject:@"Pat"];
  ...
}

If you are inspecting a message you got from some other place (server, disk,
etc), you may want to check if the Array or Dictionary has entries without
causing it to be created for you. For this, there is always a \[NAME\]_Count
property also provided that can return zero or the real count, but won't trigger
the creation.

All message fields always have a value when accessed. For primitive type
fields (ints, floats, bools, enum) there the concept of presence, that allows
you to tell the difference between when the field is the default value because
it was not set and when it was set, but explicitly to the default
value for the field. For fields with that do support presence, you can test
if the value was explicitly set via the has\[NAME\] property. If the value
has been set, and you want to clear it, you can set the has\[NAME\] to NO.

The Objective C classes/enums can be used from Swift code.

Objective C Generator Proto File Options Copy Chapter

*objc_class_prefix=* (no default)

This options allow you to provide a custom prefix for all the symbols generated
from a proto file (classes (from message), enums, the Root for extension
support).

If not set, the generation options package_to_prefix_mappings_path and
use_package_as_prefix (documented below) controls what is used instead. Since
Objective C uses a global namespace for all of its classes, there can be collisions.
use_package_as_prefix=yes should avoid collisions since proto package are used to
scope/name things in other languages, but this option can be used to get shorter
names instead. Convention is to base the explicit prefix on the proto package.

Objective C Generator `protoc` Options Copy Chapter

When generating Objective C code, protoc supports a --objc_opt argument; the
argument is comma-delimited name/value pairs (key=value,key2=value2). The
keys are used to change the behavior during generation. The currently
supported keys are:

• generate_for_named_framework: The value used for this key will be used
  when generating the #import statements in the generated code. Instead
  of being plain #import "some/path/file.pbobjc.h" lines, they will be
  framework based, i.e. - #import <VALUE/file.pbobjc.h>.

  NOTE: If this is used with named_framework_to_proto_path_mappings_path,
  then this is effectively the default to use for everything that wasn't
  mapped by the other.

• named_framework_to_proto_path_mappings_path: The value used for this key
  is a path to a file containing the listing of framework names and proto
  files. The generator uses this to decide if another proto file referenced
  should use a framework style import vs. a user level import
  (#import <FRAMEWORK/file.pbobjc.h> vs #import "dir/file.pbobjc.h").

  The format of the file is:

  • An entry is a line of frameworkName: file.proto, dir/file2.proto.
  • Comments start with #.
  • A comment can go on a line after an entry.
    (i.e. - frameworkName: file.proto # comment)

  Any number of files can be listed for a framework, just separate them with
  commas.

  There can be multiple lines listing the same frameworkName in case it has a
  lot of proto files included in it; and having multiple lines makes things
  easier to read.

• runtime_import_prefix: The value used for this key to be used as a
  prefix on #imports of runtime provided headers in the generated files.
  When integrating ObjC protos into a build system, this can be used to avoid
  having to add the runtime directory to the header search path since the
  generate #import will be more complete.

• package_to_prefix_mappings_path: The value used for this key is a
  path to a file containing a list of proto packages and prefixes.
  The generator will use this to locate which ObjC class prefix to use when
  generating sources unless the objc_class_prefix file option is set.
  This option can be useful if multiple apps consume a common set of
  proto files but wish to use a different prefix for the generated sources
  between them. This option takes precedent over the use_package_as_prefix
  option.

  The format of the file is:

  • An entry is a line of "package=prefix".
  • Comments start with #.
  • A comment can go on a line after a expected package/prefix pair.
    (i.e. - "package=prefix # comment")
  • For files that do NOT have a proto package (not recommended), an
    entry can be made as "no_package:PATH=prefix", where PATH is the
    path for the .proto file.

• use_package_as_prefix, package_as_prefix_forced_prefix and
  proto_package_prefix_exceptions_path: The value for
  use_package_as_prefix can be yes or no, and indicates if a prefix
  should be derived from the proto package for all the symbols for files that
  don't have the objc_class_prefix file option (mentioned above). This helps
  ensure the symbols are more unique and means there is less chance of ObjC
  class name collisions.

  To help in migrating code to using this support,
  proto_package_prefix_exceptions_path can be used to provide the path
  to a file that contains proto package names (one per line, comments allowed
  if prefixed with #). These package won't get the derived prefix, allowing
  migrations to the behavior one proto package at a time across a code base.

  package_as_prefix_forced_prefix can be used to provide a value that will
  be used before all prefixes derived from the packages to help group all of
  these types with a common prefix. Thus it only makes sense to use it when
  use_package_as_prefix is also enabled. For example, setting this to
  "XYZ_" and generating a file with the package "something" defining
  "MyMessage", would have Objective-C class be XYZ_Something_MyMessage.

  use_package_as_prefix currently defaults to no (existing behavior), but
  that could change in the future as it helps avoid collisions when more
  protos get added to the build. Note that this would be a breaking change.

• headers_use_forward_declarations: The value for this can be yes or
  no, and indicates if the generated headers use forward declarations for
  Message and Enum types from other .proto files or if the files should be
  imported into the generated header instead.

  By using forward declarations, less code is likely to recompile when the
  files do change, but Swift generally doesn't like forward declarations and
  will fail to include properties when the concrete definition of the type is
  known at import time. If your proto usages span modules, this can be a
  problem.

  headers_use_forward_declarations currently defaults to yes (existing
  behavior), but in a future release, that default may change to provide
  better Swift support by default.

Contributing Copy Chapter

Please make updates to the tests along with changes. If just changing the
runtime, the Xcode projects can be used to build and run tests. If your change
also requires changes to the generated code,
objectivec/DevTools/full_mac_build.sh can be used to easily rebuild and test
changes. Passing -h to the script will show the addition options that could
be useful.

Documentation Copy Chapter

The complete documentation for Protocol Buffers is available via the
web at:

https://developers.google.com/protocol-buffers/

Overview (`php/README.md`) Copy Section

This directory contains the Protocol Buffers runtime implementation via both a
pure PHP package and a native c extension. The pure PHP package is intended to
provide usability to wider range of PHP platforms, while the c extension is
intended to provide higher performance. Both implementations provide the same
runtime APIs and share the same generated code. Users don’t need to re-generate
code for the same proto definition when they want to switch the implementation
later.

Both implementations make use of generated PHP code that defines message and
enum types in PHP. We strongly recommend using protoc's PHP generation support
with .proto files. The build process in this directory only installs the
extension/package; you need to install protoc as well to have PHP code
generation functionality.

Requirements Copy Chapter

For information on supported language versions and how the support levels for
PHP versions will change over time, see
Supported PHP versions.

Installation Copy Chapter

C Extension Copy Section

Prerequirements

To install the c extension, the following tools are needed: * gcc * libtool *
make * pear * pecl * phpize

On Ubuntu, you can install them with:

sudo apt-get install -y php-pear php-dev libtool make gcc

On other platforms, please use the corresponding package managing tool to
install them before proceeding.

Installation from Source (Building extension)

To build the c extension, run the following command:

cd ext/google/protobuf
pear package
sudo pecl install protobuf-{VERSION}.tgz

Installation from PECL

When we release a version of Protocol Buffers, we will upload the extension to
PECL. To use this pre-packaged extension, simply
install it as you would any other extension:

sudo pecl install protobuf-{VERSION}

PHP Package Copy Section

Installation from composer

Simply add "google/protobuf" to the 'require' section of composer.json in your
project.

To use the pure PHP implementation, you need to install bcmath.

Protoc Copy Section

Once the extension or package is installed, if you wish to generate PHP code
from a .proto file, you will also want to install the Protocol Buffers
compiler (protoc), as described in this repository's main README file. The
version of protoc included in the latest release supports the --php_out
option to generate PHP code:

protoc --php_out=out_dir test.proto

Usage Copy Chapter

For generated code:
https://developers.google.com/protocol-buffers/docs/reference/php-generated

Known Issues Copy Chapter

• Missing native support for well known types.
• Missing support for proto2.
• No API provided for clear/copy messages.
• No API provided for encoding/decoding with stream.
• Map fields may not be garbage-collected if there is cycle reference.
• No debug information for messages in c extension.
• HHVM not tested.
• C extension not tested on windows, mac, php 7.0.
• Message name cannot be Empty.

Development Copy Chapter

Test Native PHP Copy Section

# Install Dependencies (Linux)
apt-get install bazel composer php-dev

# Download protobuf
git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf

# Build protoc
bazel build :protoc

# Test native php
cd php
composer install
composer test

Test C Extension Copy Section

After you have finished testing the native php, you can test the c extension:

cd tests
./test.sh 5.6 # The php runtime version.
              # We provide 5.5, 5.5-zts, 5.6, 5.6-zts, 7.0, 7.0-zts, 7.1, 7.1-zts, 7.2, 7.2-zts, 7.3 and 7.3-zts
              # ls /usr/local for more details

If you want to use gdb to debug the c extension, you can do:

./gdb_test.sh

Overview (`pkg/README.md`) Copy Section

Protobuf packaging

This directory contains Bazel rules for building packaging and distribution
artifacts.

Everything in this directory should be considered internal and subject to
change.

Protocol compiler binary packaging Copy Chapter

The protocol compiler is used in binary form in various places. There are rules
which package it, along with commonly used .proto files, for distribution.

Source distribution packaging Copy Chapter

Protobuf releases include source distributions, sliced by target language (C++,
Java, etc.). There are rules in this package to define those source archives.
These depend upon pkg_files rules elsewhere in the repo to get the contents.

The source distribution files should include the outputs from autogen.sh, but
this isn't something we can reliably do from Bazel. To produce fully functioning
source distributions, run autogen.sh before building the archives (this
populates the necessary files directly into the source tree).

C++ runtime binary distribution Copy Chapter

The cc_dist_library rule creates composite libraries from several other
cc_library targets. Bazel uses a "fine-grained" library model, where each
cc_library produces its own library artifacts, without transitive
dependencies. The cc_dist_library rule combines several other libraries
together, creating a single library that may be suitable for distribution.

Overview (`python/README.md`) Copy Section

Protocol Buffers Python

This directory contains the Protobuf library for Python.

For user documentation about how to use Protobuf Python, see
https://protobuf.dev/getting-started/pythontutorial/

Installation

In most cases you should install the library using pip or another package
manager:

$ pip install protobuf

The packages released on https://pypi.org/project/protobuf/#files include both a
source distribution and binary wheels.

Building packages from this repo Copy Chapter

If for some reason you wish to build the packages directly from this repo, you
can use the following Bazel commands:

$ bazel build //python/dist:source_wheel
$ bazel build //python/dist:binary_wheel

The binary wheel will build against whatever version of Python is installed on
your system. The source package is always the same and does not depend on a
local version of Python.

Building from `setup.py` Copy Chapter

We support building from setup.py, but only from a Python source package.
You cannot build from setup.py using the GitHub repo or the GitHub source
tarball.

To build a source package from this repo, see the instructions in the previous
section.

Implementation backends

There are three separate implementations of Python Protobuf. All of them offer
the same API and are thus functionally the same, though they have very different
performance characteristics.

The runtime library contains a switching layer that can choose between these
backends at runtime. Normally it will choose between them automatically, using
priority-ordered list, and skipping any backends that are not available. However
you can request a specific backend by setting the
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION environment variable to one of the
following values:

1. upb: Built on the
   upb C library,
   this is a new extension module
   released in 4.21.0. It offers
   better performance than any of the previous backends, and it is now the
   default. It is distributed in our PyPI packages, and requires no special
   installation. The code for this module lives in this directory.
2. cpp: This extension module wraps the C++ protobuf library. It is
   deprecated and is no longer released in our PyPI packages, however it is
   still used in some legacy cases where apps want to perform zero-copy message
   sharing between Python and C++. It must be installed separately before it
   can be used. The code for this module lives in
   google/protobuf/pyext.
3. python: The pure-Python backend, this does not require any extension
   module to be present on the system. The code for the pure-Python backend
   lives in google/protobuf/internal

The order given above is the general priority order, with upb being preferred
the most and the python backend preferred the least. However this ordering can
be overridden by the presence of a
google.protobuf.internal._api_implementation module. See the logic in
api_implementation.py
for details.

You can check which backend you are using with the following snippet:

$ python
Python 3.10.9 (main, Dec  7 2022, 13:47:07) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.protobuf.internal import api_implementation
>>> print(api_implementation.Type())
upb

This is not an officially-supported or stable API, but it is useful for ad hoc
diagnostics.

More information about sharing messages between Python and C++ is available
here: https://protobuf.dev/reference/python/python-generated/#sharing-messages

Code generator

The code for the Protobuf Python code generator lives in
//src/google/protobuf/compiler/python.
The code generator can output two different files for each proto foo.proto:

• foo_pb2.py: The module you import to actually use the protos.
• foo_pb2.pyi: A stub file that describes the interface of the protos.

The foo_pb2.pyi file is useful for IDEs or for users who want to read the
output file. The foo_pb2.py file is optimized for fast loading and is not
readable at all.

Note that the pyi file is only generated if you pass the pyi_out option to
protoc:

$ protoc --python_out=pyi_out:output_dir

Overview (`ruby/README.md`) Copy Section

This directory contains the Ruby extension that implements Protocol Buffers
functionality in Ruby.

The Ruby extension makes use of generated Ruby code that defines message and
enum types in a Ruby DSL. You may write definitions in this DSL directly, but we
recommend using protoc's Ruby generation support with .proto files. The build
process in this directory only installs the extension; you need to install
protoc as well to have Ruby code generation functionality. You can build protoc
from source using bazel build //:protoc.

Installation from Gem Copy Chapter

In Gemfile (Please check which version of Protocol Buffers you need: RubyGems):

gem 'google-protobuf'

Or for using this pre-packaged gem, simply install it as you would any other gem:

$ gem install [--prerelease] google-protobuf

Once the gem is installed, you may or may not need protoc. If you write your
message type descriptions directly in the Ruby DSL, you do not need it.
However, if you wish to generate the Ruby DSL from a .proto file, you will
also want to install Protocol Buffers itself, as described in this repository's
main README file. The version of protoc included in the latest release
supports the --ruby_out option to generate Ruby code.

A simple example of using the Ruby extension follows. More extensive
documentation may be found on
rubydoc.info.

require 'google/protobuf'

# generated from my_proto_types.proto with protoc:
#  $ protoc --ruby_out=. my_proto_types.proto
require 'my_proto_types'

mymessage = MyTestMessage.new(:field1 => 42, :field2 => ["a", "b", "c"])
mymessage.field1 = 43
mymessage.field2.push("d")
mymessage.field3 = SubMessage.new(:foo => 100)

encoded_data = MyTestMessage.encode(mymessage)
decoded = MyTestMessage.decode(encoded_data)
assert_equal mymessage, decoded
puts "JSON:"
puts MyTestMessage.encode_json(mymessage)

Installation from Source (Building Gem) Copy Chapter

Protocol Buffers has a new experimental backend that uses the
ffi gem to provide a unified C-based
implementation across Ruby interpreters based on
UPB. For now, use
of the FFI implementation is opt-in. If any of the following are true, the
traditional platform-native implementations (MRI-ruby based on CRuby, Java based
on JRuby) are used instead of the new FFI-based implementation: 1. ffi and
ffi-compiler gems are not installed 2. PROTOCOL_BUFFERS_RUBY_IMPLEMENTATION
environment variable has a value other than FFI (case-insensitive). 3. FFI is
unable to load the native library at runtime.

To build this Ruby extension, you will need:

• Rake
• Bundler
• Ruby development headers
• a C compiler

To Build the JRuby extension, you will need:

• Maven
• The latest version of the protobuf java library (see ../java/README.md)
• Install JRuby via rbenv or RVM

First switch to the desired platform with rbenv or RVM.

Then install the required Ruby gems:

$ gem install bundler
$ bundle

Then build the Gem:

$ rake
$ rake clobber_package gem
$ gem install `ls pkg/google-protobuf-*.gem`

If you intend to debug the protobuf_c Ruby bindings with gdb, you can also
build a version with debug symbols enabled by setting the PROTOBUF_CONFIG
environment variable when you build the native extension:

$ PROTOBUF_CONFIG=dbg rake

To run the specs:

$ rake test

To run the specs while using the FFI-based implementation:

$ PROTOCOL_BUFFERS_RUBY_IMPLEMENTATION=FFI rake test

This gem includes the upb parsing and serialization library as a single-file
amalgamation. It is up-to-date with upb git commit
535bc2fe2f2b467f59347ffc9449e11e47791257.

Alternatively, you can use Bazel to build and to run tests.

From the project root (rather than the ruby directory):

$ bazel test //ruby/tests/...

To run tests against the FFI implementation:

$ bazel test //ruby/tests/... //ruby:ffi_enabled --test_env=PROTOCOL_BUFFERS_RUBY_IMPLEMENTATION=FFI

Version Number Scheme Copy Chapter

We are using a version number scheme that is a hybrid of Protocol Buffers'
overall version number and some Ruby-specific rules. Gem does not allow
re-uploads of a gem with the same version number, so we add a sequence number
("upload version") to the version. We also format alphabetical tags (alpha,
pre, ...) slightly differently, and we avoid hyphens. In more detail:

• First, we determine the prefix: a Protocol Buffers version "3.0.0-alpha-2"
  becomes "3.0.0.alpha.2". When we release 3.0.0, this prefix will be simply
  "3.0.0".
• We then append the upload version: "3.0.0.alpha.2.0" or "3.0.0.0". If we need
  to upload a new version of the gem to fix an issue, the version becomes
  "3.0.0.alpha.2.1" or "3.0.0.1".
• If we are working on a prerelease version, we append a prerelease tag:
  "3.0.0.alpha.3.0.pre". The prerelease tag comes at the end so that when
  version numbers are sorted, any prerelease builds are ordered between the
  prior version and current version.

These rules are designed to work with the sorting rules for
Gem::Version:
release numbers should sort in actual release order.

Overview (`src/README.md`) Copy Section

Protocol Buffers - Google's data interchange format

Copyright 2008 Google Inc.

https://protobuf.dev.

CMake Installation Copy Chapter

To compile or install protobuf from source using CMake, see
cmake/README.md.

C++ Protobuf - Unix Copy Chapter

To build protobuf from source, the following tools are needed:

• Bazel
• git
• g++
• Abseil

On Ubuntu/Debian, for example, you can install them with:

sudo apt-get install g++ git bazel

On other platforms, use the corresponding package managing tool to install them
before proceeding. See https://bazel.build/install for further instructions on
installing Bazel, or to build from source using CMake, see
cmake/README.md. See https://github.com/abseil/abseil-cpp
for instructions on installing Abseil.

To get the source, download the release .tar.gz or .zip package in the
release page:

https://github.com/protocolbuffers/protobuf/releases/latest

You can also get the source by "git clone" our git repository. Make sure you
have also cloned the submodules and generated the configure script (skip this
if you are using a release .tar.gz or .zip package):

git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
git submodule update --init --recursive

To build the C++ Protocol Buffer runtime and the Protocol Buffer compiler
(protoc) execute the following:

bazel build :protoc :protobuf

You can then install the compiler, for example on Linux:

cp bazel-bin/protoc /usr/local/bin

For more usage information on Bazel, see http://bazel.build.

Compiling dependent packages

To compile a package that uses Protocol Buffers, you need to setup a Bazel
WORKSPACE that's hooked up to the protobuf repository and loads its
dependencies. For an example, see WORKSPACE.

Note for Mac users

For a Mac system, Unix tools are not available by default. You will first need
to install Xcode from the Mac AppStore and then run the following command from
a terminal:

sudo xcode-select --install

To install Unix tools, you can install "port" following the instructions at
https://www.macports.org . This will reside in /opt/local/bin/port for most
Mac installations.

sudo /opt/local/bin/port install bazel

Alternative for Homebrew users:

brew install bazel

Then follow the Unix instructions above.

C++ Protobuf - Windows Copy Chapter

If you only need the protoc binary, you can download it from the release
page:

https://github.com/protocolbuffers/protobuf/releases/latest

In the downloads section, download the zip file protoc-$VERSION-win32.zip. It
contains the protoc binary as well as public proto files of protobuf library.
The
protoc binary and the gencode version must match.

You can install protobuf and its dependencies directly by using vcpkg:

>vcpkg install protobuf protobuf:x64-windows

If you want to use zlib, you'll also need to install the zlib feature:

>vcpkg install protobuf[zlib] protobuf[zlib]:x64-windows

See https://github.com/Microsoft/vcpkg for more information.

To build from source using Microsoft Visual C++, see cmake/README.md.

To build from source using Cygwin or MinGW, follow the Unix installation
instructions, above.

Binary Compatibility Warning Copy Chapter

Due to the nature of C++, it is unlikely that any two versions of the
Protocol Buffers C++ runtime libraries will have compatible ABIs.
That is, if you linked an executable against an older version of
libprotobuf, it is unlikely to work with a newer version without
re-compiling. This problem, when it occurs, will normally be detected
immediately on startup of your app. Still, you may want to consider
using static linkage. You can configure this in your cc_binary Bazel rules
by specifying:

linkstatic=True

Usage Copy Chapter

The complete documentation for Protocol Buffers is available at:

https://protobuf.dev/

Overview (`upb/README.md`) Copy Section

μpb: small, fast C protos

μpb (often written 'upb') is a small
protobuf implementation written
in C.

upb is the core runtime for protobuf languages extensions in
Ruby,
PHP, and
Python.

While upb offers a C API, the C API & ABI are not stable. For this reason,
upb is not generally offered as a C library for direct consumption, and there
are no releases.

Features Copy Chapter

upb has comparable speed to protobuf C++, but is an order of magnitude smaller
in code size.

Like the main protobuf implementation in C++, it supports:

• a generated API (in C)
• reflection
• binary & JSON wire formats
• text format serialization
• all standard features of protobufs (oneofs, maps, unknown fields, extensions,
  etc.)
• full conformance with the protobuf conformance tests

upb also supports some features that C++ does not:

• optional reflection: generated messages are agnostic to whether
  reflection will be linked in or not.
• no global state: no pre-main registration or other global state.
• fast reflection-based parsing: messages loaded at runtime parse
  just as fast as compiled-in messages.

However there are a few features it does not support:

• text format parsing
• deep descriptor verification: upb's descriptor validation is not as exhaustive
  as protoc.

Install Copy Chapter

For Ruby, use RubyGems:

$ gem install google-protobuf

For PHP, use PECL:

$ sudo pecl install protobuf

For Python, use PyPI:

$ sudo pip install protobuf

Alternatively, you can build and install upb using
vcpkg dependency manager:

git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.sh
./vcpkg integrate install
./vcpkg install upb

The upb port in vcpkg is kept up to date by microsoft team members and community
contributors.

If the version is out of date, please
create an issue or pull request on the
vcpkg repository.

Contributing Copy Chapter

Please see CONTRIBUTING.md.

Overview (`.github/workflows/README.md`) Copy Section

This directory contains all of our automatically triggered workflows.

Test runner

Our top level test_runner.yml is responsible for kicking off all tests, which
are represented as reusable workflows. This is carefully constructed to satisfy
the design laid out in go/protobuf-gha-protected-resources (see below), and
duplicating it across every workflow file would be difficult to maintain. As an
added bonus, we can manually dispatch our full test suite with a single button
and monitor the progress of all of them simultaneously in GitHub's actions UI.

There are five ways our test suite can be triggered:

• Post-submit tests (push): These are run over newly submitted code
  that we can assume has been thoroughly reviewed. There are no additional
  security concerns here and these jobs can be given highly privileged access to
  our internal resources and caches.

• Pre-submit tests from a branch (push_request): These are run over
  every PR as changes are made. Since they are coming from branches in our
  repository, they have secret access by default and can also be given highly
  privileged access. However, we expect many of these events per change,
  and likely many from abandoned/exploratory changes. Given the much higher
  frequency, we restrict the ability to write to our more expensive caches.

• Pre-submit tests from a fork (pull_request_target): These are run
  over every PR from a forked repository as changes are made. These have much
  more restricted access, since they could be coming from anywhere. To protect
  our secret keys and our resources, tests will not run until a commit has been
  labeled safe to submit. Further commits will require further approvals to
  run our test suite. Once marked as safe, we will provide read-only access to
  our caches and Docker images, but will generally disallow any writes to shared
  resources.

• Continuous tests (schedule): These are run on a fixed schedule. We
  currently have them set up to run daily, and can help identify non-hermetic
  issues in tests that don't get run often (such as due to test caching) or during
  slow periods like weekends and holidays. Similar to post-submit tests, these
  are run over submitted code and are highly privileged in the resources they
  can use.

• Manual testing (workflow_dispatch): Our test runner can be triggered
  manually over any branch. This is treated similarly to pre-submit tests,
  which should be highly privileged because they can only be triggered by the
  protobuf team.

Staleness handling

While Bazel handles code generation seamlessly, we do support build systems that
don't. There are a handful of cases where we need to check in generated files
that can become stale over time. In order to provide a good developer
experience, we've implemented a system to make this more manageable.

• Stale files should have a corresponding staleness_test Bazel target. This
  should be marked manual to avoid getting picked up in CI, but will fail if
  files become stale. It also provides a --fix flag to update the stale files.

• Bazel tests will never depend on the checked-in versions, and will generate
  new ones on-the-fly during build.

• Non-Bazel tests will always regenerate necessary files before starting. This
  is done using our bash and docker actions, which should be used for any
  non-Bazel tests. This way, no tests will fail due to stale files.

• A post-submit job will immediately regenerate any stale files and commit them
  if they've changed.

• A scheduled job will run late at night every day to make sure the post-submit
  is working as expected (that is, it will run all the staleness tests).

The regenerate_stale_files.sh script is the central script responsible for all
the re-generation of stale files.

Forked PRs

Because we need secret access to run our tests, we use the pull_request_target
event for PRs coming from forked repositories. We do checkout the code from the
PR's head, but the workflow files themselves are always fetched from the base
branch (that is, the branch we're merging to). Therefore, any changes to these
files won't be tested, so we explicitly ban PRs that touch these files.

Caches

We have a number of different caching strategies to help speed up tests. These
live either in GCP buckets or in our GitHub repository cache. The former has
a lot of resources available and we don't have to worry as much about bloat.
On the other hand, the GitHub repository cache is limited to 10GB, and will
start pruning old caches when it exceeds that threshold. Therefore, we need
to be very careful about the size and quantity of our caches in order to
maximize the gains.

Bazel remote cache Copy Chapter

As described in https://bazel.build/remote/caching, remote caching allows us to
offload a lot of our build steps to a remote server that holds a cache of
previous builds. We use our GCP project for this storage, and configure
every Bazel call to use it. This provides substantial performance
improvements at minimal cost.

We do not allow forked PRs to upload updates to our Bazel caches, but they
do use them. Every other event is given read/write access to the caches.
Because Bazel behaves poorly under certain environment changes (such as
toolchain, operating system), we try to use finely-grained caches. Each job
should typically have its own cache to avoid cross-pollution.

Bazel repository cache Copy Chapter

When Bazel starts up, it downloads all the external dependencies for a given
build and stores them in the repository cache. This cache is separate from
the remote cache, and only exists locally. Because we have so many Bazel
dependencies, this can be a source of frequent flakes due to network issues.

To avoid this, we keep a cached version of the repository cache in GitHub's
action cache. Our full set of repository dependencies ends up being ~300MB,
which is fairly expensive given our 10GB maximum. The most expensive ones seem
to come from Java, which has some very large downstream dependencies.

Given the cost, we take a more conservative approach for this cache. Only push
events will ever write to this cache, but all events can read from them.
Additionally, we only store three caches for any given commit, one per platform.
This means that multiple jobs are trying to update the same cache, leading to a
race. GitHub rejects all but one of these updates, so we designed the system so
that caches are only updated if they've actually changed. That way, over time
(and multiple pushes) the repository caches will incrementally grow to encompass
all of our dependencies. A scheduled job will run monthly to clear these caches
to prevent unbounded growth as our dependencies evolve.

ccache Copy Chapter

In order to speed up non-Bazel builds to be on par with Bazel, we make use of
ccache. This intercepts all calls to the compiler, and
caches the result. Subsequent calls with a cache-hit will very quickly
short-circuit and return the already computed result. This has minimal affect
on any single job, since we typically only run a single build. However, by
caching the ccache results in GitHub's action cache we can substantially
decrease the build time of subsequent runs.

One useful feature of ccache is that you can set a maximum cache size, and it
will automatically prune older results to keep below that limit. On Linux and
Mac cmake builds, we generally get 30MB caches and set a 100MB cache limit. On
Windows, with debug symbol stripping we get ~70MB and set a 200MB cache limit.

Because CMake build tend to be our slowest, bottlenecking the entire CI process,
we use a fairly expensive strategy with ccache. All events will cache their
ccache directory, keyed by the commit and the branch. This means that each
PR and each branch will write its own set of caches. When looking up which
cache to use initially, each job will first look for a recent cache in its
current branch. If it can't find one, it will accept a cache from the base
branch (for example, PRs will initially use the latest cache from their target
branch).

While the ccache caches quickly over-run our GitHub action cache, they also
quickly become useless. Since GitHub prunes caches based on the time they were
last used, this just means that we'll see quicker turnover.

sccache Copy Chapter

An alternative to ccache is sccache. The
two tools are very similar in function, but sccache requires (and allows) much
less configuration and supports GCS storage right out of the box. By hooking
this up to our project that we already use for Bazel caching, we're able to get
even bigger CMake wins in CI because we're no longer constrained by GitHub's
10GB cache limit.

Similar to the Bazel remote cache, we give read access to every CI run, but
disallow writing in PRs from forks.

Bazelisk Copy Chapter

Bazelisk will automatically download a pinned version of Bazel on first use.
This can lead to flakes, and to avoid that we cache the result keyed on the
Bazel version. Only push events will write to this cache, but it's unlikely
to change very often.

Docker images Copy Chapter

Instead of downloading a fresh Docker image for every test run, we can save it
as a tar and cache it using docker image save and later restore using
docker image load. This can decrease download times and also reduce flakes.
Note, Docker's load can actually be significantly slower than a pull in certain
situations. Therefore, we should reserve this strategy for only Docker images
that are causing noticeable flakes.

Pip dependencies Copy Chapter

The actions/setup-python action we use for Python supports automated caching
of pip dependencies. We enable this to avoid having to download these
dependencies on every run, which can lead to flakes.

Custom actions

We've defined a number of custom actions to abstract out shared pieces of our
workflows.

• Bazel use this for running all Bazel tests. It can take either a single
  Bazel command or a more general bash command. In the latter case, it provides
  environment variables for running Bazel with all our standardized settings.

• Bazel-Docker nearly identical to the Bazel action, this additionally
  runs everything in a specified Docker image.

• Bash use this for running non-Bazel tests. It takes a bash command and
  runs it verbatim. It also handles the regeneration of stale files (which does
  use Bazel), which non-Bazel tests might depend on.

• Docker nearly identical to the Bash action, this additionally runs
  everything in a specified Docker image.

• ccache this sets up a ccache environment, and initializes some
  environment variables for standardized usage of ccache.

• Cross-compile protoc this abstracts out the compilation of protoc using
  our cross-compilation infrastructure. It will set a PROTOC environment
  variable that gets automatically picked up by a lot of our infrastructure.
  This is most useful in conjunction with the Bash action with non-Bazel
  tests.

Overview (`csharp/keys/README.md`) Copy Section

Contents Copy Chapter

• Google.Protobuf.public.snk:
  Public key to verify strong name of Google.Protobuf assemblies.
• Google.Protobuf.snk:
  Signing key to provide strong name of Google.Protobuf assemblies.
  As per Microsoft guidance
  signing key should be checked into the repository.

Overview (`csharp/protos/README.md`) Copy Section

This directory contains unit test protos adapted from those in
src/google/protobuf, and C#-specific test protos for regression
tests against bugs found in the C# codegen or library.

Overview (`java/protoc/README.md`) Copy Section

Publish pre-compiled protoc artifacts

protoc is the compiler for .proto files. It generates language bindings
for the messages and/or RPC services from .proto files.

Because protoc is a native executable, the scripts under this directory
publish a protoc executable (a.k.a. artifact) to Maven repositories. The
artifact can be used by build automation tools so that users would not need to
compile and install protoc for their systems.

If you would like us to publish protoc artifact for a new platform, please
open an issue to request it.

Maven Location Copy Chapter

The published protoc artifacts are available on Maven here:

https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/

Versioning Copy Chapter

The version of the protoc artifact must be the same as the version of the
Protobuf project.

Artifact name Copy Chapter

The name of a published protoc artifact is in the following format:
protoc-<version>-<os>-<arch>.exe, e.g., protoc-3.6.1-linux-x86_64.exe.

Note that artifacts for linux/macos also have the .exe suffix but they are
not windows binaries.

System requirement Copy Chapter

Install Apache Maven if you don't have it.

The scripts only work under Unix-like environments, e.g., Linux, MacOSX, and
Cygwin or MinGW for Windows. Please see README.md of the Protobuf project
for how to set up the build environment.

Tested build environments Copy Chapter

We have successfully built artifacts on the following environments:

• Linux x86_32 and x86_64:
  • Centos 6.9 (within Docker 1.6.1)
  • Ubuntu 14.04.5 64-bit
• Linux aarch_64: Cross compiled with g++-aarch64-linux-gnu on Ubuntu 14.04.5 64-bit
• Mac OS X x86_32 and x86_64: Mac OS X 10.9.5

Overview (`python/protobuf_distutils/README.md`) Copy Section

Python setuptools extension

This is an extension for Python setuptools which uses an installed protobuf
compiler (protoc) to generate Python sources.

Installing Copy Chapter

To use this extension, it needs to be installed so it can be imported by other
projects' setup.py.

$ python setup.py build
$ python -m pip install .

(If you want to test changes to the extension, you can use python setup.py develop.)

Usage Copy Chapter

Example setup.py configuration Copy Section

from setuptools import setup
setup(
    # ...
    name='example_project',

    # Require this package, but only for setup (not installation):
    setup_requires=['protobuf_distutils'],

    options={
        # See below for details.
        'generate_py_protobufs': {
            'source_dir':        'path/to/protos',
            'extra_proto_paths': ['path/to/other/project/protos'],
            'output_dir':        'path/to/project/sources',  # default '.'
            'proto_files':       ['relative/path/to/just_this_file.proto'],
            'protoc':            'path/to/protoc.exe',
        },
    },
)

Example build invocation Copy Section

These steps will generate protobuf sources so they are included when building
and installing example_project (see above):

$ python setup.py generate_py_protobufs
$ python setup.py build
$ python -m pip install .

Options Copy Chapter

• source_dir:

  This is the directory holding .proto files to be processed.

  The default behavior is to generate sources for all .proto files found under
  source_dir, recursively. This behavior can be controlled with options below.

• proto_root_path:

  This is the root path for resolving imports in source .proto files.

  The default is the shortest prefix of source_dir among [source_dir] + self.extra_proto_paths.

• extra_proto_paths:

  Specifies additional paths that should be used to find imports, in
  addition to source_dir.

  This option can be used to specify the path to other protobuf sources,
  which are imported by files under source_dir. No Python code will
  be generated for .proto files under extra_proto_paths.

• output_dir:

  Specifies where generated code should be placed.

  Typically, this should be the root package that generated Python modules
  should be below.

  The generated files will be placed under output_dir according to the
  relative source paths under proto_root_path. For example, the source file
  ${proto_root_path}/subdir/message.proto will be generated as the Python
  module ${output_dir}/subdir/message_pb2.py.

• proto_files:

  A list of strings, specific .proto file paths for generating code, instead of
  searching for all .proto files under source_path.

  These paths are relative to source_dir. For example, to generate code
  for just ${source_dir}/subdir/message.proto, specify
  ['subdir/message.proto'].

• protoc:

  By default, the protoc binary (the Protobuf compiler) is found by
  searching the environment path. To use a specific protoc binary, its
  path can be specified. Resolution of the protoc value is as follows:

  1. If the --protoc=VALUE flag is passed to generate_py_protobufs,
     then VALUE will be used.
     For example:
     shell Copy

     $ python setup.py generate_py_protobufs --protoc=/path/to/protoc

  2. Otherwise, if a value was set in the options, it will be used.
     (See "Example setup.py configuration," above.)
  3. Otherwise, if the PROTOC environment variable is set, it will be
     used. For example:
     For example:
     shell Copy

     $ PROTOC=/path/to/protoc python setup.py generate_py_protobufs

  4. Otherwise, $PATH will be searched.

Overview (`rust/release_crates/README.md`) Copy Section

This directory contains one directory per crate that we release.

Each directory contains a pkg_tar() which packages up the crate into a tar which
can be pushed to crates.io.

New development happens in the google_protobuf* crates. The legacy
protobuf* crates are kept as re-exports of their google_protobuf*
equivalents so that existing users keep building, but they set
publish = false and are no longer pushed to crates.io.

Overview (`rust/upb/README.md`) Copy Section

Rust upb bindings

This directory contains Rust bindings for upb, attempting to be the a direct
mapping to expose upb to Rust. This should not be used directly by application
code, instead is an implementation detail for Rust Protobuf, which is still
under development but can be found here:
here.

Overview (`third_party/utf8_range/README.md`) Copy Section

Fast UTF-8 validation with Range algorithm (NEON+SSE4+AVX2)

This is a brand new algorithm to leverage SIMD for fast UTF-8 string validation. Both NEON(armv8a) and SSE4 versions are implemented. AVX2 implementation contributed by ioioioio.

Four UTF-8 validation methods are compared on both x86 and Arm platforms. Benchmark result shows range base algorithm is the best solution on Arm, and achieves same performance as Lemire's approach on x86.

• Range based algorithm
  • range-neon.c: NEON version
  • range-sse.c: SSE4 version
  • range-avx2.c: AVX2 version
  • range2-neon.c, range2-sse.c: Process two blocks in one iteration
• Lemire's SIMD implementation
  • lemire-sse.c: SSE4 version
  • lemire-avx2.c: AVX2 version
  • lemire-neon.c: NEON porting
• naive.c: Naive UTF-8 validation byte by byte
• lookup.c: Lookup-table method

About the code Copy Chapter

• Run "make" to build. Built and tested with gcc-7.3.
• Run "./utf8" to see all command line options.
• Benchmark
  • Run "./utf8 bench" to benchmark all algorithms with default test file.
  • Run "./utf8 bench size NUM" to benchmark specified string size.
• Run "./utf8 test" to test all algorithms with positive and negative test cases.
• To benchmark or test specific algorithm, run something like "./utf8 bench range".

Benchmark result (MB/s) Copy Chapter

Method Copy Section

1. Generate UTF-8 test buffer per test file or buffer size.
2. Call validation sub-routines in a loop until 1G bytes are checked.
3. Calculate speed(MB/s) of validating UTF-8 strings.

NEON(armv8a) Copy Section

SSE4(E5-2650) Copy Section

Range algorithm analysis Copy Chapter

Basic idea:

• Load 16 bytes
• Leverage SIMD to calculate value range for each byte efficiently
• Validate 16 bytes at once

UTF-8 coding format Copy Section

http://www.unicode.org/versions/Unicode6.0.0/ch03.pdf, page 94

Table 3-7. Well-Formed UTF-8 Byte Sequences

To summarise UTF-8 encoding:

• Depending on First Byte, one legal character can be 1, 2, 3, 4 bytes
  • For First Byte within C0..DF, character length = 2
  • For First Byte within E0..EF, character length = 3
  • For First Byte within F0..F4, character length = 4
• C0, C1, F5..FF are not allowed
• Second,Third,Fourth Bytes must lie in 80..BF.
• There are four special cases for Second Byte, shown bold italic in above table.

Range table Copy Section

Range table maps range index 0 ~ 15 to minimal and maximum values allowed. Our task is to observe input string, find the pattern and set correct range index for each byte, then validate input string.

Calculate byte ranges (ignore special cases) Copy Section

Ignoring the four special cases(E0,ED,F0,F4), how should we set range index for each byte?

• Set range index to 0(00..7F) for all bytes by default
• Find non-ASCII First Byte (C0..FF), set their range index to 8(C2..F4)
• For First Byte within C0..DF, set next byte's range index to 1(80..BF)
• For First Byte within E0..EF, set next two byte's range index to 2,1(80..BF) in sequence
• For First Byte within F0..FF, set next three byte's range index to 3,2,1(80..BF) in sequence

To implement above operations efficiently with SIMD:

• For 16 input bytes, use lookup table to map C0..DF to 1, E0..EF to 2, F0..FF to 3, others to 0. Save to first_len.
• Map C0..FF to 8, we get range indices for First Byte.
• Shift first_len one byte, we get range indices for Second Byte.
• Saturate subtract first_len by one(3->2, 2->1, 1->0, 0->0), then shift two bytes, we get range indices for Third Byte.
• Saturate subtract first_len by two(3->1, 2->0, 1->0, 0->0), then shift three bytes, we get range indices for Fourth Byte.

Example(assume no previous data)

Range_index = First_Byte | Second_Byte | Third_Byte | Fourth_Byte

Error handling

• C0,C1,F5..FF are not included in range table and will always be detected.
• Illegal 80..BF will have range index 0(00..7F) and be detected.
• Based on First Byte, according Second, Third and Fourth Bytes will have range index 1/2/3, to make sure they must lie in 80..BF.
• If non-ASCII First Byte overlaps, above algorithm will set range index of the latter First Byte to 9,10,11, which are illegal ranges. E.g, Input = F1 80 C2 90 --> Range index = 8 3 10 1, where 10 indicates error. See table below.

Overlapped non-ASCII First Byte

Adjust Second Byte range for special cases Copy Section

Range index adjustment for four special cases

Range index adjustment can be reduced to below problem:

Given 16 bytes, replace E0 with 2, ED with 3, F0 with 3, F4 with 4, others with 0.

A naive SIMD approach:

1. Compare 16 bytes with E0, get the mask for eacy byte (FF if equal, 00 otherwise)
2. And the mask with 2 to get adjustment for E0
3. Repeat step 1,2 for ED,F0,F4

At least eight operations are required for naive approach.

Observing special bytes(E0,ED,F0,F4) are close to each other, we can do much better using lookup table.

NEON

NEON tbl instruction is very convenient for table lookup:

• Table can be up to 16x4 bytes in size
• Return zero if index is out of range

Leverage these features, we can solve the problem with as few as two operations:

• Precreate a 16x2 lookup table, where table[0]=2, table[13]=3, table[16]=3, table[20]=4, table[others]=0.
• Subtract input bytes with E0 (E0 -> 0, ED -> 13, F0 -> 16, F4 -> 20).
• Use the subtracted byte as index of lookup table and get range adjustment directly.
  • For indices less than 32, we get zero or required adjustment value per input byte
  • For out of bound indices, we get zero per tbl behaviour

SSE

SSE pshufb instruction is not as friendly as NEON tbl in this case:

• Table can only be 16 bytes in size
• Out of bound indices are handled this way:
  • If 7-th bit of index is 0, least four bits are used as index (E.g, index 0x73 returns 3rd element)
  • If 7-th bit of index is 1, return 0 (E.g, index 0x83 returns 0)

We can still leverage these features to solve the problem in five operations:

• Precreate two tables:
  • table_df[1] = 2, table_df[14] = 3, table_df[others] = 0
  • table_ef[1] = 3, table_ef[5] = 4, table_ef[others] = 0
• Subtract input bytes with EF (E0 -> 241, ED -> 254, F0 -> 1, F4 -> 5) to get the temporary indices
• Get range index for E0,ED
  • Saturate subtract temporary indices with 240 (E0 -> 1, ED -> 14, all values below 240 becomes 0)
  • Use subtracted indices to look up table_df, get the correct adjustment
• Get range index for F0,F4
  • Saturate add temporary indices with 112(0x70) (F0 -> 0x71, F4 -> 0x75, all values above 16 will be larger than 128(7-th bit set))
  • Use added indices to look up table_ef, get the correct adjustment (index 0x71,0x75 returns 1st,5th elements, per pshufb behaviour)

Error handling

• For overlapped non-ASCII First Byte, range index before adjustment is 9,10,11. After adjustment (adds 2,3,4 or 0), the range index will be 9 to 15, which is still illegal in range table. So the error will be detected.

Handling remaining bytes Copy Section

For remaining input less than 16 bytes, we will fallback to naive byte by byte approach to validate them, which is actually faster than SIMD processing.

• Look back last 16 bytes buffer to find First Byte. At most three bytes need to look back. Otherwise we either happen to be at character boundray, or there are some errors we already detected.
• Validate string byte by byte starting from the First Byte.

Tests Copy Chapter

It's necessary to design test cases to cover corner cases as more as possible.

Positive cases Copy Section

1. Prepare correct characters
2. Validate correct characters
3. Validate long strings
   • Round concatenate characters starting from first character to 1024 bytes
   • Validate 1024 bytes string
   • Shift 1 byte, validate 1025 bytes string
   • Shift 2 bytes, Validate 1026 bytes string
   • ...
   • Shift 16 bytes, validate 1040 bytes string
4. Repeat step3, test buffer starting from second character
5. Repeat step3, test buffer starting from third character
6. ...

Negative cases Copy Section

1. Prepare bad characters and bad strings
   • Bad character
   • Bad character cross 16 bytes boundary
   • Bad character cross last 16 bytes and remaining bytes boundary
2. Test long strings
   • Prepare correct long strings same as positive cases
   • Append bad characters
   • Shift one byte for each iteration
   • Validate each shift

Code breakdown Copy Chapter

Below table shows how 16 bytes input are processed step by step. See range-neon.c for according code.

Overview (`upb/util/README.md`) Copy Section

upb util library

The libraries in this directory contain useful functionality that is layered
on top of the main upb APIs. In other words, the APIs in this directory have
no special access to upb internals; you could easily implement the same things
yourself.

Overview (`upb_generator/common/README.md`) Copy Section

This directory contains APIs that are used by multiple code generators, but not
public to users.

Overview (`bazel/private/oss/README.md`) Copy Section

Bazel-specific Build Rules

This directory contains rule implementations that are specific to Bazel (ie.
where Blaze and Bazel rules have diverged).

Overview (`docs/design/editions/README.md`) Copy Section

Protocol Buffers - Protobuf Editions design documents

This directory contains historical design documents that describe plans for
implementing Protobuf Editions. For an up-to-date overview of this feature of
Protocol Buffers, see
Protobuf Editions Overview.

These files represent the state that the original documents were in at the time
that they were published to this repository. While some updates may be made to
the files after their initial upload, you should consider the possibility that
they are outdated as you read them. These are purely for historical value and
should not be treated as documentation of the current state.

Note that the authors listed in the topics were the authors of the original
version of the document; it may have changed since they last worked on the
document.

Design Documents Copy Chapter

The following topics are in this repository:

• What are Protobuf Editions?
• Life of an Edition
• Edition Lifetimes
• Protobuf Editions Design: Features
• Editions: Life of a Featureset
• Edition Zero Features
• Minimum Required Edition
• Protobuf Editions for Schema Producers
• Stricter Schemas with Editions
• Edition Zero: JSON Handling
• Edition Zero: Converged Semantics
• Edition Zero Feature: Enum Field Closedness
• C++ APIs for Edition Zero
• Java Lite For Editions
• Edition Evolution
• Edition Naming
• Editions Feature Visibility
• Legacy Syntax Editions
• Editions: Feature Extension Layout
• Editions: Group Migration Issues

Overview (`docs/design/prototiller/README.md`) Copy Section

Protocol Buffers - Prototiller design documents

This directory contains historical design documents that describe plans for
implementing Prototiller, a tool for updating proto files to be conformant with Protobuf Editions functionality. For design documents related to Protobuf Editions, see the README.

These files represent the state that the original documents were in at the time
that they were published to this repository. While some updates may be made to
the files after their initial upload, you should consider the possibility that
they are outdated as you read them. These are purely for historical value and
should not be treated as documentation of the current state.

Note that the authors listed in the topics were the authors of the original
version of the document; it may have changed since they last worked on the
document.

Design Documents Copy Chapter

The following topics are in this repository:

• Prototiller Requirements for Editions
• Prototiller Requirements for Edition Zero
• Editions Tooling

Overview (`rust/release_crates/google_protobuf/README.md`) Copy Section

The runtime of the official Google Rust Protobuf implementation.

This is currently a beta release: the API is subject to change, and there may be
some rough edges, including missing documentation and features.

An example for how to use this crate can be found in the
google-protobuf-example crate

Ownership and implementation change

This crate is officially supported by the Protobuf team at Google.

This is a completely new implementation with a different API, as well as a
fundamentally different approach than prior versions of the protobuf crate. It
focuses on delivering a high-quality Rust API which is backed by either a pure C
implementation (upb) or the Protobuf C++ implementation. This choice was made
for performance, feature parity, development velocity, and security reasons.

It is not planned for the pure Rust lineage to be actively developed going
forward. While it is not expected to receive significant further development, as
a stable and high quality pure Rust implementation, many open source projects
may reasonably continue to stay on the protobuf crate.

Development was previously done under the protobuf crate by a community
project by stepancheg who generously donated
the crate name to Google. V4.36.0 of the protobuf crate corresponds to V0.36.0
of this crate.

How to get a compatible version of protoc

The protoc binary that you use to generate code needs to have a version that
exactly matches the version of the protobuf crate you are using. More
specifically, if you are using Rust protobuf x.y.z then you need to use protoc
y.z. See here for more
details on our versioning scheme.

The easiest way to get ahold of protoc is to download a prebuilt binary from the
matching release here.
Just make sure protoc is on your $PATH when you run cargo.

Overview (`rust/release_crates/google_protobuf_codegen/README.md`) Copy Section

The build time dependency crate for generating code for the official Google Rust
Protobuf implementation.

The code generated by this is expected to be used in conjunction with the
official Google Protobuf crate.

This is currently a beta release: the API is subject to change,
and there may be some rough edges, including missing documentation and features.

How to get a compatible version of protoc

The protoc binary that you use to generate code needs to have a version that
exactly matches the version of the protobuf crate you are using. More
specifically, if you are using Rust protobuf x.y.z then you need to use protoc
y.z. See here for more
details on our versioning scheme.

The easiest way to get ahold of protoc is to download a prebuilt binary from the
matching release here.
Just make sure protoc is on your $PATH when you run cargo.

Overview (`rust/release_crates/google_protobuf_example/README.md`) Copy Section

An example that demonstrates how to use the google-protobuf and
google-protobuf-codegen crates together.

The source of the latest version of this example can be read
here.

How to get a compatible version of protoc

The protoc binary that you use to generate code needs to have a version that
exactly matches the version of the protobuf crate you are using. More
specifically, if you are using Rust protobuf x.y.z then you need to use protoc
y.z. See here for more
details on our versioning scheme.

The easiest way to get ahold of protoc is to download a prebuilt binary from the
matching release here.
Just make sure protoc is on your $PATH when you run cargo.

Overview (`rust/release_crates/google_protobuf_macros/README.md`) Copy Section

Internal crate. Use google-protobuf
instead.

Overview (`rust/release_crates/google_protobuf_well_known_types/README.md`) Copy Section

This crate contains the protobuf
well-known types for
the official Google Protobuf crate.

Overview (`rust/release_crates/protobuf/README.md`) Copy Section

The runtime of the official Google Rust Protobuf implementation.

This is currently a beta release: the API is subject to change, and there may be
some rough edges, including missing documentation and features.

An example for how to use this crate can be found in the
protobuf_example crate

This crate is superseded by
google-protobuf. New development
happens there; please migrate there to get the latest releases.

V4 ownership and implementation change

V4 of this crate is officially supported by the Protobuf team at Google. Prior
major versions were developed by as a community project by
stepancheg who generously donated the crate
name to Google.

V4 is a completely new implementation with a different API, as well as a
fundamentally different approach than prior versions of this crate. It focuses
on delivering a high-quality Rust API which is backed by either a pure C
implementation (upb) or the Protobuf C++ implementation. This choice was made
for performance, feature parity, development velocity, and security reasons.
More discussion about the rationale and design philosophy can be found at
https://protobuf.dev/reference/rust/.

It is not planned for the V3 pure Rust lineage to be actively developed going
forward. While it is not expected to receive significant further development, as
a stable and high quality pure Rust implementation, many open source projects
may reasonably continue to stay on the V3 API.

Overview (`rust/release_crates/protobuf_codegen/README.md`) Copy Section

The build time dependency crate for generating code for the official Google Rust
Protobuf implementation.

This crate is superseded by
google-protobuf-codegen.
New development happens there; please migrate there to get the latest
releases.

The code generated by this is expected to be used in conjunction with the
official Protobuf crate.

This is currently a beta release: the API is subject to change,
and there may be some rough edges, including missing documentation and features.

How to get a compatible version of protoc

The protoc binary that you use to generate code needs to have a version that
exactly matches the version of the protobuf crate you are using. More
specifically, if you are using Rust protobuf x.y.z then you need to use protoc
y.z. See here for more
details on our versioning scheme.

The easiest way to get ahold of protoc is to download a prebuilt binary from the
matching release here.
Just make sure protoc is on your $PATH when you run cargo.

Overview (`rust/release_crates/protobuf_example/README.md`) Copy Section

An example that demonstrates how to use the protobuf and protobuf_codegen
crates together.

The source of the latest version of this example can be read
here.

How to get a compatible version of protoc

The protoc binary that you use to generate code needs to have a version that
exactly matches the version of the protobuf crate you are using. More
specifically, if you are using Rust protobuf x.y.z then you need to use protoc
y.z. See here for more
details on our versioning scheme.

The easiest way to get ahold of protoc is to download a prebuilt binary from the
matching release here.
Just make sure protoc is on your $PATH when you run cargo.

Overview (`rust/release_crates/protobuf_macros/README.md`) Copy Section

Internal crate. Use protobuf instead.

This crate is superseded by
google-protobuf-macros,
which it re-exports. New development happens there.

Overview (`rust/release_crates/protobuf_well_known_types/README.md`) Copy Section

This crate contains the protobuf
well-known types.

This crate is superseded by
google-protobuf-well-known-types,
which it re-exports. New development happens there; please migrate there to
get the latest releases.

Overview (`src/google/protobuf/compiler/kotlin/README.md`) Copy Section

Kotlin DSL Generator

This code generator implements the Kotlin DSL. The Kotlin DSL sits on top of
another proto implementation (written in Java or Kotlin) and adds convenient
support for building proto messages using DSL syntax, as documented in
Kotlin Generated Code Guide.

This code generator is invoked by passing --kotlin_out to protoc.

When using Kotlin on the JVM, you will also need to pass --java_out to
generate the Java code that implements the generated classes themselves.

When using Kotlin on other platforms (eg. Kotlin Native), there is currently no
support for generating message classes, so it is not possible to use the Kotlin
DSL at this time.

Overview (`src/google/protobuf/util/python/README.md`) Copy Section

Python wrappers protobuf/util

The field_mask_util wrappers are maintained by the
Risk AI Team for their own
benefit. Other teams may add themselves to the visibility rule, but have to sign
up as OWNERs in return.

Cmake Protobuf Generate (`docs/cmake_protobuf_generate.md`) Copy Section

How to use `protobuf_generate`

This document explains how to use the function protobuf_generate which is
provided by protobuf's CMake module.

Usage Copy Chapter

In the same directory that called find_package(protobuf CONFIG) and any of its
subdirectories, the CMake function protobuf_generate is made available by
protobuf-generate.cmake. It can be used to
automatically generate source files from .proto schema files at build time.

Basic example Copy Section

Let us see how protobuf_generate can be used to generate and compile the
source files of a proto schema whenever an object target called proto-objects
is built.

Given the following directory structure:

• proto/helloworld/helloworld.proto
• CMakeLists.txt

where helloworld.proto is a protobuf schema file and CMakeLists.txt
contains:

find_package(protobuf CONFIG REQUIRED)

add_library(proto-objects OBJECT "${CMAKE_CURRENT_LIST_DIR}/proto/helloworld/helloworld.proto")

target_link_libraries(proto-objects PUBLIC protobuf::libprotobuf)

set(PROTO_BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}/generated")

target_include_directories(proto-objects PUBLIC "$<BUILD_INTERFACE:${PROTO_BINARY_DIR}>")

protobuf_generate(
    TARGET proto-objects
    IMPORT_DIRS "${CMAKE_CURRENT_LIST_DIR}/proto"
    PROTOC_OUT_DIR "${PROTO_BINARY_DIR}")

Building the target proto-objects will generate the files:

• ${CMAKE_CURRENT_BINARY_DIR}/generated/helloworld/helloworld.pb.h
• ${CMAKE_CURRENT_BINARY_DIR}/generated/helloworld/helloworld.pb.cc

and (depending on the build system) output:

[build] [1/2] Running cpp protocol buffer compiler on /proto/helloworld/helloworld.proto
[build] [2/2] Building CXX object /build/generated/helloworld/helloworld.pb.cc.o

gRPC example Copy Section

protobuf_generate can also be customized to invoke plugins like gRPC's
grpc_cpp_plugin. Given the same directory structure as in the
basic example and let CMakeLists.txt contain:

find_package(gRPC CONFIG REQUIRED)

add_library(proto-objects OBJECT "${CMAKE_CURRENT_LIST_DIR}/proto/helloworld/helloworld.proto")

target_link_libraries(proto-objects PUBLIC protobuf::libprotobuf gRPC::grpc++)

set(PROTO_BINARY_DIR "${CMAKE_CURRENT_BINARY_DIR}/generated")
set(PROTO_IMPORT_DIRS "${CMAKE_CURRENT_LIST_DIR}/proto")

target_include_directories(proto-objects PUBLIC "$<BUILD_INTERFACE:${PROTO_BINARY_DIR}>")

protobuf_generate(
    TARGET proto-objects
    IMPORT_DIRS ${PROTO_IMPORT_DIRS}
    PROTOC_OUT_DIR "${PROTO_BINARY_DIR}")

protobuf_generate(
    TARGET proto-objects
    LANGUAGE grpc
    GENERATE_EXTENSIONS .grpc.pb.h .grpc.pb.cc
    PLUGIN "protoc-gen-grpc=\$<TARGET_FILE:gRPC::grpc_cpp_plugin>"
    IMPORT_DIRS ${PROTO_IMPORT_DIRS}
    PROTOC_OUT_DIR "${PROTO_BINARY_DIR}")

Then building proto-objects will generate and compile:

• ${CMAKE_CURRENT_BINARY_DIR}/generated/helloworld/helloworld.pb.h
• ${CMAKE_CURRENT_BINARY_DIR}/generated/helloworld/helloworld.pb.cc
• ${CMAKE_CURRENT_BINARY_DIR}/generated/helloworld/helloworld.grpc.pb.h
• ${CMAKE_CURRENT_BINARY_DIR}/generated/helloworld/helloworld.grpc.pb.cc

And protoc will automatically be re-run whenever the schema files change and
proto-objects is built.

Note on unity builds Copy Section

Since protobuf's generated source files are unsuited for
jumbo/unity builds
it is recommended to exclude them from such builds which can be achieved by
adjusting their properties:

protobuf_generate(
    OUT_VAR PROTO_GENERATED_FILES
    ...)

set_source_files_properties(${PROTO_GENERATED_FILES} PROPERTIES SKIP_UNITY_BUILD_INCLUSION on)

How it works Copy Chapter

For each source file ending in proto of the argument provided to TARGET or
each file provided through PROTOS, protobuf_generate will set up a
add_custom_command
which depends on protobuf::protoc and the proto files. It declares the
generated source files as OUTPUT which means that any target that depends on
them will automatically cause the custom command to execute when it is brought
up to date. The command itself is made up of the arguments for protoc, like
the output directory, the schema files, the language to generate for, the
plugins to use, etc.

Reference Copy Chapter

Arguments accepted by protobuf_generate.

Flag arguments:

• APPEND_PATH — A flag that causes the base path of all proto schema files
  to be added to IMPORT_DIRS.

Single-value arguments:

• LANGUAGE — A single value: cpp or python. Determines what kind of source
  files are being generated.
• OUT_VAR — Name of a CMake variable that will be filled with the paths to
  the generated source files.
• EXPORT_MACRO — Name of a macro that is applied to all generated Protobuf
  message classes and extern variables. It can, for example, be used to
  declare DLL exports.
• PROTOC_EXE — Command name, path, or CMake executable used to run protoc
  commands. Defaults to protobuf::protoc.
• PROTOC_OUT_DIR — Output directory of generated source files. Defaults to
  CMAKE_CURRENT_BINARY_DIR.
• PLUGIN — An optional plugin executable. This could, for example, be the
  path to grpc_cpp_plugin.
• PLUGIN_OPTIONS — Additional options provided to the plugin, such as
  generate_mock_code=true for the gRPC cpp plugin.
• DEPENDENCIES — Arguments forwarded to the DEPENDS of the underlying
  add_custom_command invocation.
• TARGET — CMake target that will have the generated files added as sources.

Multi-value arguments:

• PROTOS — List of proto schema files. If omitted, then every source file
  ending in proto of TARGET will be used.
• IMPORT_DIRS — A common parent directory for the schema files. For example,
  if the schema file is proto/helloworld/helloworld.proto and the import
  directory proto/ then the generated files are
  ${PROTOC_OUT_DIR}/helloworld/helloworld.pb.h and
  ${PROTOC_OUT_DIR}/helloworld/helloworld.pb.cc.
• GENERATE_EXTENSIONS — If LANGUAGE is omitted then this must be set to the
  extensions that protoc generates.
• PROTOC_OPTIONS — Additional arguments that are forwarded to protoc. Don't
  use it for flags that protobuf_generate sets from its own arguments:
  import paths (-I/--proto_path; use IMPORT_DIRS or APPEND_PATH), the
  output directory and generator options (--<lang>_out; use
  PROTOC_OUT_DIR and PLUGIN_OPTIONS) or plugins (--plugin; use
  PLUGIN). The generated file names are worked out from those arguments,
  so flags passed here aren't taken into account.

Cpp Build Systems (`docs/cpp_build_systems.md`) Copy Section

How Protobuf supports multiple C++ build systems

This document explains how the Protobuf project supports multiple C++ build
systems.

Background Copy Chapter

Protobuf primarily uses Bazel to build the Protobuf C++
runtime and Protobuf compiler[^historical_sot]. However, there are several
different build systems in common use for C++, each one of which requires
essentially a complete copy of the same build definitions.

[^historical_sot]: On a historical note, prior to its
release as Open Source Software,
the Protobuf project was developed using Google's internal
build system, which was the predecessor to Bazel (the vast
majority of Google's contributions continue to be developed
this way). The Open Source Protobuf project, however,
historically used Autoconf to build the C++ implementation.
Over time, other build systems (including Bazel) have been
added, thanks in large part to substantial contributions from
the Open Source community. Since the Protobuf project deals
with multiple languages (all of which ultimately rely upon
C++, for the Protobuf compiler), Bazel is a natural choice
for a project-wide build system -- in fact, Bazel (and its
predecessor, Blaze) was designed in large part to support
exactly this type of rich, multi-language build.

Currently, C++ Protobuf can be built with Bazel and CMake. Each of these build
systems has different semantics and structure, but share in common the list of
files needed to build the runtime and compiler.

Design Copy Chapter

Extracting information from Bazel Copy Section

Bazel's Starlark API provides aspects to
traverse the build graph, inspect build rules, define additional actions, and
expose information through
providers. For example, the
cc_proto_library rule uses an aspect to traverse the dependency graph of
proto_library rules, and dynamically attaches actions to generate C++ code
using the Protobuf compiler and compile using the C++ compiler.

In order to support multiple build systems, the overall build structure is
defined once for each system, and expose frequently-changing metadata from Bazel
in a way that can be included from the build definition. Primarily, this means
exposing the list of source files in a way that can be included in other build
definitions.

Two aspects are used to extract this information from the Bazel build
definitions:

• cc_file_list_aspect extracts srcs, hdrs, and textual_hdrs from build
  rules like cc_library. The sources are exposed through a provider named
  CcFileList.
• proto_file_list_aspect extracts the srcs from a proto_library, and
  also generates the expected filenames that would be generated by the
  Protobuf compiler. This information is exposed through a provider named
  ProtoFileList.

On their own, these aspects have limited utility. However, they can be
instantiated by custom rules, so that an ordinary BUILD.bazel target can
produce outputs based on the information gleaned from these aspects.

(Aside) Distribution libraries Copy Section

Bazel's native cc_library rule is typically used on a "fine-grained" level, so
that, for example, lightweight unit tests can be written with narrow scope.
Although Bazel does build library artifacts (such as .so and .a files on
Linux), they correspond to cc_library rules.

Since the entire "Protobuf library" includes many constituent cc_library
rules, a special rule, cc_dist_library, combines several fine-grained
libraries into a single, monolithic library.

For the Protobuf project, these "distribution libraries" are intended to match
the granularity of the CMake-based builds. Since the Bazel-built distribution
library covers the rules with the source files needed by other builds, the
cc_dist_library rule invokes the cc_file_list_aspect on its input libraries.
The result is that a cc_dist_library rule not only produces composite library
artifacts, but also collect and provide the list of sources that were inputs.

For example:

$ cat cc_dist_library_example/BUILD.bazel
load("@rules_cc//cc:defs.bzl", "cc_library")
load("//pkg:cc_dist_library.bzl", "cc_dist_library")

cc_library(
    name = "a",
    srcs = ["a.cc"],
)

cc_library(
    name = "b",
    srcs = ["b.cc"],
    deps = [":c"],
)

# N.B.: not part of the cc_dist_library, even though it is in the deps of 'b':
cc_library(
    name = "c",
    srcs = ["c.cc"],
)

cc_dist_library(
    name = "lib",
    deps = [
        ":a",
        ":b",
    ],
    visibility = ["//visibility:public"],
)

# Note: the output below has been formatted for clarity:
$ bazel cquery //cc_dist_library_example:lib \
    --output=starlark \
    --starlark:expr='providers(target)["//pkg:cc_dist_library.bzl%CcFileList"]'
struct(
    hdrs = depset([]),
    internal_hdrs = depset([]),
    srcs = depset([
        <source file cc_dist_library_example/a.cc>,
        <source file cc_dist_library_example/b.cc>,
    ]),
    textual_hdrs = depset([]),
)

The upshot is that the "coarse-grained" library can be defined by the Bazel
build, and then export the list of source files that are needed to reproduce the
library in a different build system.

One major difference from most Bazel rule types is that the file list aspects do
not propagate. In other words, they only expose the immediate dependency's
sources, not transitive sources. This is for two reasons:

1. Immediate dependencies are conceptually simple, while transitivity requires
   substantially more thought. For example, if transitive dependencies were
   considered, then some way would be needed to exclude dependencies that
   should not be part of the final library (for example, a distribution library
   for //:protobuf could be defined not to include all of
   //:protobuf_lite). While dependency elision is an interesting design
   problem, the protobuf library is small enough that directly listing
   dependencies should not be problematic.
2. Dealing only with immediate dependencies gives finer-grained control over
   what goes into the composite library. For example, a Starlark select()
   could conditionally add fine-grained libraries to some builds, but not
   others.

Another subtlety for tests is due to Bazel internals. Internally, a slightly
different configuration is used when evaluating cc_test rules as compared to
cc_dist_library. If cc_test targets are included in a cc_dist_library
rule, and both are evaluated by Bazel, this can result in a build-time error:
the config used for the test contains additional options that tell Bazel how to
execute the test that the cc_file_list_aspect build config does not. Bazel
detects this as two conflicting actions generating the same outputs. (For
cc_test rules, the simplest workaround is to provide sources through a
filegroup or similar.)

File list generation Copy Section

Lists of input files are generated by Bazel in a format that can be imported to
other build systems. Currently only CMake-style files can be generated.

The lists of files are derived from Bazel build targets. The sources can be: *
cc_dist_library rules (as described above) * proto_library rules *
individual files * filegroup rules * pkg_files or pkg_filegroup rules from
https://github.com/bazelbuild/rules_pkg

For example:

/* Detailed source-code truncated for AI context efficiency. */

A hand-written CMake build rule could then use the generated file to define
libraries, such as:

include(source_lists.cmake)
add_library(distlib ${distlib_srcs} ${buff_srcs})

Protobuf usage Copy Section

The main C++ runtimes (lite and full) and the Protobuf compiler use their
corresponding cc_dist_library rules to generate file lists. For
proto_library targets, the file list generation can extract the source files
directly. For other targets, notably cc_test targets, the file list generators
use filegroup rules.

In general, adding new targets to a non-Bazel build system in Protobuf (or
adding a new build system altogether) requires some one-time setup:

1. The overall structure of the new build system has to be defined. It should
   import lists of files and refer to them by variable, instead of listing
   files directly.
2. (Only if the build system is new) A new rule type has to be added to
   //pkg:build_systems.bzl. Most of the implementation is shared, but a
   "fragment generator" is need to declare a file list variable, and the rule
   type itself has to be defined and call the shared implementation.

When files are added or deleted, or when the Protobuf Bazel structure is
changed, these changes may need to be reflected in the file list logic. These
are some example scenarios:

• Files are added to (or removed from) the srcs of an existing cc_library:
  no changes needed. If the cc_library is already part of a
  cc_dist_library, then regenerating the source lists will reflect the
  change.
• A cc_library is added: the new target may need to be added to the Protobuf
  cc_dist_library targets, as appropriate.
• A cc_library is deleted: if a cc_dist_library depends upon the deleted
  target, then a build-time error will result. The library needs to be removed
  from the cc_dist_library.
• A cc_test is added or deleted: test sources are handled by filegroup
  rules defined in the same package as the cc_test rule. The filegroups
  are usually given a name like "test_srcs", and often use glob() to find
  sources. This means that adding or removing a test may not require any extra
  work, but this can be verified within the same package as the test rule.
• Test-only proto files are added: the proto_library might need to be added
  to the file list map in //pkg:BUILD.bazel, and then the file added to
  various build systems. However, most test-only protos are already exposed
  through libraries like //src/google/protobuf:test_protos.

If there are changes, then the regenerated file lists need to be copied back
into the repo. That way, the corresponding build systems can be used with a git
checkout, without needing to run Bazel first.

(Aside) Distribution archives Copy Section

A very similar set of rules is defined in //pkg to build source distribution
archives for releases. In addition to the full sources, Protobuf releases also
include source archives sliced by language, so that, for example, a Ruby-based
project can get just the sources needed to build the Ruby runtime. (The
per-language slices also include sources needed to build the protobuf compiler,
so they all effectively include the C++ runtime.)

These archives are defined using rules from the
rules_pkg project. Although they are
similar to cc_dist_library and the file list generation rules, the goals are
different: the build system file lists described above only apply to C++, and
are organized according to what should or should not be included in different
parts of the build (e.g., no tests are included in the main library). On the
other hand, the distribution archives deal with languages other than C++, and
contain all the files that need to be distributed as part of a release (even for
C++, this is more than just the C++ sources).

While it might be possible to use information from the CcFileList and
ProtoFileList providers to define the distribution files, additional files
(such as the various BUILD.bazel files) are also needed in the distribution
archive. The lists of distribution files can usually be generated by glob(),
anyhow, so sharing logic with the file list aspects may not be beneficial.

Currently, all of the file lists are checked in. However, it would be possible
to build the file lists on-the-fly and include them in the distribution
archives, rather than checking them in.

Field Presence (`docs/field_presence.md`) Copy Section

Application note: Field presence

This application note explains the various presence tracking disciplines for
protobuf fields. It also explains the behaviour of explicit presence tracking
for singular proto3 fields with basic types.

Background Copy Chapter

Field presence is the notion of whether a protobuf field has a value. There
are two different manifestations of presence for protobufs: no presence, where
the generated message API stores field values (only), and explicit presence,
where the API also stores whether or not a field has been set.

Historically, proto2 has mostly followed explicit presence, while proto3
exposes only no presence semantics. Singular proto3 fields of basic types
(numeric, string, bytes, and enums) which are defined with the optional label
have explicit presence, like proto2 (this feature is enabled by default as
release 3.15).

Presence disciplines Copy Section

Presence disciplines define the semantics for translating between the API
representation and the serialized representation. The no presence
discipline relies upon the field value itself to make decisions at
(de)serialization time, while the explicit presence discipline relies upon the
explicit tracking state instead.

Presence in *tag-value stream* (wire format) serialization Copy Section

The wire format is a stream of tagged, self-delimiting values. By definition,
the wire format represents a sequence of present values. In other words, every
value found within a serialization represents a present field; furthermore,
the serialization contains no information about not-present values.

The generated API for a proto message includes (de)serialization definitions
which translate between API types and a stream of definitionally present (tag,
value) pairs. This translation is designed to be forward- and
backward-compatible across changes to the message definition; however, this
compatibility introduces some (perhaps surprising) considerations when
deserializing wire-formatted messages:

• When serializing, fields with no presence are not serialized if they
  contain their default value.
  • For numeric types, the default is 0.
  • For enums, the default is the zero-valued enumerator.
  • For strings, bytes, and repeated fields, the default is the zero-length
    value.
  • For messages, the default is the language-specific null value.
• "Empty" length-delimited values (such as empty strings) can be validly
  represented in serialized values: the field is "present," in the sense that
  it appears in the wire format. However, if the generated API does not track
  presence, then these values may not be re-serialized; i.e., the empty field
  may be "not present" after a serialization round-trip.
• When deserializing, duplicate field values may be handled in different ways
  depending on the field definition.
  • Duplicate repeated fields are typically appended to the field's API
    representation. (Note that serializing a packed repeated field
    produces only one, length-delimited value in the tag stream.)
  • Duplicate optional field values follow the rule that "the last one
    wins."
• oneof fields expose the API-level invariant that only one field is set at
  a time. However, the wire format may include multiple (tag, value) pairs
  which notionally belong to the oneof. Similar to optional fields, the
  generated API follows the "last one wins" rule.
• Out-of-range values are not returned for enum fields in generated proto2
  APIs. However, out-of-range values may be stored as unknown fields in the
  API, even though the wire-format tag was recognized.

Presence in *named-field mapping* formats Copy Section

Protobufs can be represented in human-readable, textual forms. Two notable
formats are TextFormat (the output format produced by generated message
DebugString methods) and JSON.

These formats have correctness requirements of their own, and are generally
stricter than tagged-value stream formats. However, TextFormat more closely
mimics the semantics of the wire format, and does, in certain cases, provide
similar semantics (for example, appending repeated name-value mappings to a
repeated field). In particular, similar to the wire format, TextFormat only
includes fields which are present.

JSON is a much stricter format, however, and cannot validly represent some
semantics of the wire format or TextFormat.

• Notably, JSON elements are semantically unordered, and each member must
  have a unique name. This is different from TextFormat rules for repeated
  fields.
• JSON may include fields that are "not present," unlike the no presence
  discipline for other formats:
  • JSON defines a null value, which may be used to represent a defined
    but not-present field.
  • Repeated field values may be included in the formatted output, even if
    they are equal to the default (an empty list).
• Because JSON elements are unordered, there is no way to unambiguously
  interpret the "last one wins" rule.
  • In most cases, this is fine: JSON elements must have unique names:
    repeated field values are not valid JSON, so they do not need to be
    resolved as they are for TextFormat.
  • However, this means that it may not be possible to interpret oneof
    fields unambiguously: if multiple cases are present, they are unordered.

In theory, JSON can represent presence in a semantic-preserving fashion. In
practice, however, presence correctness can vary depending upon implementation
choices, especially if JSON was chosen as a means to interoperate with clients
not using protobufs.

Presence in proto2 APIs Copy Section

This table outlines whether presence is tracked for fields in proto2 APIs (both
for generated APIs and using dynamic reflection):

Singular fields (of all types) track presence explicitly in the generated API.
The generated message interface includes methods to query presence of fields.
For example, the field foo has a corresponding has_foo method. (The specific
name follows the same language-specific naming convention as the field
accessors.) These methods are sometimes referred to as "hazzers" within the
protobuf implementation.

Similar to singular fields, oneof fields explicitly track which one of the
members, if any, contains a value. For example, consider this example oneof:

oneof foo {
  int32 a = 1;
  float b = 2;
}

Depending on the target language, the generated API would generally include
several methods:

• A hazzer for the oneof: has_foo
• A oneof case method: foo
• Hazzers for the members: has_a, has_b
• Getters for the members: a, b

Repeated fields and maps do not track presence: there is no distinction between
an empty and a not-present repeated field.

Presence in proto3 APIs Copy Section

This table outlines whether presence is tracked for fields in proto3 APIs (both
for generated APIs and using dynamic reflection):

Similar to proto2 APIs, proto3 does not track presence explicitly for repeated
fields. Without the optional label, proto3 APIs do not track presence for
basic types (numeric, string, bytes, and enums), either. Oneof fields
affirmatively expose presence, although the same set of hazzer methods may not
generated as in proto2 APIs.

Under the no presence discipline, the default value is synonymous with "not
present" for purposes of serialization. To notionally "clear" a field (so it
won't be serialized), an API user would set it to the default value.

The default value for enum-typed fields under no presence is the corresponding
0-valued enumerator. Under proto3 syntax rules, all enum types are required to
have an enumerator value which maps to 0. By convention, this is an UNKNOWN or
similarly-named enumerator. If the zero value is notionally outside the domain
of valid values for the application, this behavior can be thought of as
tantamount to explicit presence.

Semantic differences Copy Chapter

The no presence serialization discipline results in visible differences from
the explicit presence tracking discipline, when the default value is set. For
a singular field with numeric, enum, or string type:

• No presence discipline:
  • Default values are not serialized.
  • Default values are not merged-from.
  • To "clear" a field, it is set to its default value.
  • The default value may mean:
    • the field was explicitly set to its default value, which is valid in
      the application-specific domain of values;
    • the field was notionally "cleared" by setting its default; or
    • the field was never set.
• Explicit presence discipline:
  • Explicitly set values are always serialized, including default values.
  • Un-set fields are never merged-from.
  • Explicitly set fields -- including default values -- are merged-from.
  • A generated has_foo method indicates whether or not the field foo
    has been set (and not cleared).
  • A generated clear_foo method must be used to clear (i.e., un-set) the
    value.

Considerations for merging Copy Section

Under the no presence rules, it is effectively impossible for a target field
to merge-from its default value (using the protobuf's API merging functions).
This is because default values are skipped, similar to the no presence
serialization discipline. Merging only updates the target (merged-to) message
using the non-skipped values from the update (merged-from) message.

The difference in merging behavior has further implications for protocols which
rely on partial "patch" updates. If field presence is not tracked, then an
update patch alone cannot represent an update to the default value, because only
non-default values are merged-from.

Updating to set a default value in this case requires some external mechanism,
such as FieldMask. However, if presence is tracked, then all explicitly-set
values -- even default values -- will be merged into the target.

Considerations for change-compatibility Copy Section

Changing a field between explicit presence and no presence is a
binary-compatible change for serialized values in wire format. However, the
serialized representation of the message may differ, depending on which version
of the message definition was used for serialization. Specifically, when a
"sender" explicitly sets a field to its default value:

• The serialized value following no presence discipline does not contain the
  default value, even though it was explicitly set.
• The serialized value following explicit presence discipline contains every
  "present" field, even if it contains the default value.

This change may or may not be safe, depending on the application's semantics.
For example, consider two clients with different versions of a message
definition.

Client A uses this definition of the message, which follows the explicit
presence serialization discipline for field foo:

syntax = "proto3";
message Msg {
  optional int32 foo = 1;
}

Client B uses a definition of the same message, except that it follows the no
presence discipline:

syntax = "proto3";
message Msg {
  int32 foo = 1;
}

Now, consider a scenario where client A observes foo's presence as the clients
repeatedly exchange the "same" message by deserializing and reserializing:

// Client A:
Msg m_a;
m_a.set_foo(1);                  // non-default value
assert(m_a.has_foo());           // OK
Send(m_a.SerializeAsString());   // to client B

// Client B:
Msg m_b;
m_b.ParseFromString(Receive());  // from client A
assert(m_b.foo() == 1);          // OK
Send(m_b.SerializeAsString());   // to client A

// Client A:
m_a.ParseFromString(Receive());  // from client B
assert(m_a.foo() == 1);          // OK
assert(m_a.has_foo());           // OK
m_a.set_foo(0);                  // default value
Send(m_a.SerializeAsString());   // to client B

// Client B:
Msg m_b;
m_b.ParseFromString(Receive());  // from client A
assert(m_b.foo() == 0);          // OK
Send(m_b.SerializeAsString());   // to client A

// Client A:
m_a.ParseFromString(Receive());  // from client B
assert(m_a.foo() == 0);          // OK
assert(m_a.has_foo());           // FAIL

If client A depends on explicit presence for foo, then a "round trip"
through client B will be lossy from the perspective of client A. In the example,
this is not a safe change: client A requires (by assert) that the field is
present; even without any modifications through the API, that requirement fails
in a value- and peer-dependent case.

How to enable *explicit presence* in proto3 Copy Chapter

These are the general steps to use field tracking support for proto3:

1. Add an optional field to a .proto file.
2. Run protoc (at least v3.15, or v3.12 using
   --experimental_allow_proto3_optional flag).
3. Use the generated "hazzer" methods and "clear" methods in application code,
   instead of comparing or setting default values.

`.proto` file changes Copy Section

This is an example of a proto3 message with fields which follow both no
presence and explicit presence semantics:

syntax = "proto3";
package example;

message MyMessage {
  // No presence:
  int32 not_tracked = 1;

  // Explicit presence:
  optional int32 tracked = 2;
}

`protoc` invocation Copy Section

Presence tracking for proto3 messages is enabled by default
since v3.15.0
release, formerly up until
v3.12.0 the
--experimental_allow_proto3_optional flag was required when using presence
tracking with protoc.

Using the generated code Copy Section

The generated code for proto3 fields with explicit presence (the optional
label) will be the same as it would be in a proto2 file.

This is the definition used in the "no presence" examples below:

syntax = "proto3";
package example;
message Msg {
  int32 foo = 1;
}

This is the definition used in the "explicit presence" examples below:

syntax = "proto3";
package example;
message Msg {
  optional int32 foo = 1;
}

In the examples, a function GetProto constructs and returns a message of type
Msg with unspecified contents.

C++ example

No presence:

Msg m = GetProto();
if (m.foo() != 0) {
  // "Clear" the field:
  m.set_foo(0);
} else {
  // Default value: field may not have been present.
  m.set_foo(1);
}

Explicit presence:

Msg m = GetProto();
if (m.has_foo()) {
  // Clear the field:
  m.clear_foo();
} else {
  // Field is not present, so set it.
  m.set_foo(1);
}

C# example

No presence:

var m = GetProto();
if (m.Foo != 0) {
  // "Clear" the field:
  m.Foo = 0;
} else {
  // Default value: field may not have been present.
  m.Foo = 1;
}

Explicit presence:

var m = GetProto();
if (m.HasFoo) {
  // Clear the field:
  m.ClearFoo();
} else {
  // Field is not present, so set it.
  m.Foo = 1;
}

Go example

No presence:

m := GetProto()
if m.Foo != 0 {
  // "Clear" the field:
  m.Foo = 0
} else {
  // Default value: field may not have been present.
  m.Foo = 1
}

Explicit presence:

m := GetProto()
if m.Foo != nil {
  // Clear the field:
  m.Foo = nil
} else {
  // Field is not present, so set it.
  m.Foo = proto.Int32(1)
}

Java example

These examples use a Builder to demonstrate clearing. Simply checking presence
and getting values from a Builder follows the same API as the message type.

No presence:

Msg.Builder m = GetProto().toBuilder();
if (m.getFoo() != 0) {
  // "Clear" the field:
  m.setFoo(0);
} else {
  // Default value: field may not have been present.
  m.setFoo(1);
}

Explicit presence:

Msg.Builder m = GetProto().toBuilder();
if (m.hasFoo()) {
  // Clear the field:
  m.clearFoo()
} else {
  // Field is not present, so set it.
  m.setFoo(1);
}

Python example

No presence:

m = example.Msg()
if m.foo != 0:
  # "Clear" the field:
  m.foo = 0
else:
  # Default value: field may not have been present.
  m.foo = 1

Explicit presence:

m = example.Msg()
if m.HasField('foo'):
  # Clear the field:
  m.ClearField('foo')
else:
  # Field is not present, so set it.
  m.foo = 1

Ruby example

No presence:

m = Msg.new
if m.foo != 0
  # "Clear" the field:
  m.foo = 0
else
  # Default value: field may not have been present.
  m.foo = 1
end

Explicit presence:

m = Msg.new
if m.has_foo?
  # Clear the field:
  m.clear_foo
else
  # Field is not present, so set it.
  m.foo = 1
end

Javascript example

No presence:

var m = new Msg();
if (m.getFoo() != 0) {
  // "Clear" the field:
  m.setFoo(0);
} else {
  // Default value: field may not have been present.
  m.setFoo(1);
}

Explicit presence:

var m = new Msg();
if (m.hasFoo()) {
  // Clear the field:
  m.clearFoo()
} else {
  // Field is not present, so set it.
  m.setFoo(1);
}

Objective C example

No presence:

Msg *m = [[Msg alloc] init];
if (m.foo != 0) {
  // "Clear" the field:
  m.foo = 0;
} else {
  // Default value: field may not have been present.
  m.foo = 1;
}

Explicit presence:

Msg *m = [[Msg alloc] init];
if (m.hasFoo()) {
  // Clear the field:
  [m clearFoo];
} else {
  // Field is not present, so set it.
  [m setFoo:1];
}

Implementing Proto3 Presence (`docs/implementing_proto3_presence.md`) Copy Section

How To Implement Field Presence for Proto3

Protobuf release 3.12 adds experimental support for optional fields in proto3.
Proto3 optional fields track presence like in proto2. For background information
about what presence tracking means, please see
docs/field_presence.

Document Summary Copy Chapter

This document is targeted at developers who own or maintain protobuf code
generators. All code generators will need to be updated to support proto3
optional fields. First-party code generators developed by Google are being
updated already. However third-party code generators will need to be updated
independently by their authors. This includes:

• implementations of Protocol Buffers for other languages.
• alternate implementations of Protocol Buffers that target specialized use
  cases.
• RPC code generators that create generated APIs for service calls.
• code generators that implement some utility code on top of protobuf
  generated classes.

While this document speaks in terms of "code generators", these same principles
apply to implementations that dynamically generate a protocol buffer API "on the
fly", directly from a descriptor, in languages that support this kind of usage.

Background Copy Chapter

Presence tracking was added to proto3 in response to user feedback, both from
inside Google and
from open-source users.
The
proto3 wrapper types
were previously the only supported presence mechanism for proto3. Users have
pointed to both efficiency and usability issues with the wrapper types.

Presence in proto3 uses exactly the same syntax and semantics as in proto2.
Proto3 Fields marked optional will track presence like proto2, while fields
without any label (known as "singular fields"), will continue to omit presence
information. The optional keyword was chosen to minimize differences with
proto2.

Unfortunately, for the current descriptor protos and Descriptor API (as of
3.11.4) it is not possible to use the same representation as proto2. Proto3
descriptors already use LABEL_OPTIONAL for proto3 singular fields, which do
not track presence. There is a lot of existing code that reflects over proto3
protos and assumes that LABEL_OPTIONAL in proto3 means "no presence." Changing
the semantics now would be risky, since old software would likely drop proto3
presence information, which would be a data loss bug.

To minimize this risk we chose a descriptor representation that is semantically
compatible with existing proto3 reflection. Every proto3 optional field is
placed into a one-field oneof. We call this a "synthetic" oneof, as it was not
present in the source .proto file.

Since oneof fields in proto3 already track presence, existing proto3
reflection-based algorithms should correctly preserve presence for proto3
optional fields with no code changes. For example, the JSON and TextFormat
parsers/serializers in C++ and Java did not require any changes to support
proto3 presence. This is the major benefit of synthetic oneofs.

This design does leave some cruft in descriptors. Synthetic oneofs are a
compatibility measure that we can hopefully clean up in the future. For now
though, it is important to preserve them across different descriptor formats and
APIs. It is never safe to drop synthetic oneofs from a proto schema. Code
generators can (and should) skip synthetic oneofs when generating a user-facing
API or user-facing documentation. But for any schema representation that is
consumed programmatically, it is important to keep the synthetic oneofs around.

In APIs it can be helpful to offer separate accessors that refer to "real"
oneofs (see API Changes below). This is a convenient way to omit
synthetic oneofs in code generators.

Updating a Code Generator Copy Chapter

When a user adds an optional field to proto3, this is internally rewritten as
a one-field oneof, for backward-compatibility with reflection-based algorithms:

syntax = "proto3";

message Foo {
  // Experimental feature, not generally supported yet!
  optional int32 foo = 1;

  // Internally rewritten to:
  // oneof _foo {
  //   int32 foo = 1 [proto3_optional=true];
  // }
  //
  // We call _foo a "synthetic" oneof, since it was not created by the user.
}

As a result, the main two goals when updating a code generator are:

1. Give optional fields like foo normal field presence, as described in
   docs/field_presence If your implementation already
   supports proto2, a proto3 optional field should use exactly the same API
   and internal implementation as proto2 optional.
2. Avoid generating any oneof-based accessors for the synthetic oneof. Its only
   purpose is to make reflection-based algorithms work properly if they are not
   aware of proto3 presence. The synthetic oneof should not appear anywhere in
   the generated API.

Satisfying the Experimental Check Copy Section

If you try to run protoc on a file with proto3 optional fields, you will get
an error because the feature is still experimental:

$ cat test.proto
syntax = "proto3";

message Foo {
  // Experimental feature, not generally supported yet!
  optional int32 a = 1;
}
$ protoc --cpp_out=. test.proto
test.proto: This file contains proto3 optional fields, but --experimental_allow_proto3_optional was not set.

There are two options for getting around this error:

1. Pass --experimental_allow_proto3_optional to protoc.
2. Make your filename (or a directory name) contain the string
   test_proto3_optional. This indicates that the proto file is specifically
   for testing proto3 optional support, so the check is suppressed.

These options are demonstrated below:

# One option:
$ protoc test.proto --cpp_out=. --experimental_allow_proto3_optional

# Another option:
$ cp test.proto test_proto3_optional.proto
$ protoc test_proto3_optional.proto --cpp_out=.
$

The experimental check will be removed in a future release, once we are ready to
make this feature generally available. Ideally this will happen for the 3.13
release of protobuf, sometime in mid-2020, but there is not a specific date set
for this yet. Some of the timing will depend on feedback we get from the
community, so if you have questions or concerns please get in touch via a GitHub
issue.

Signaling That Your Code Generator Supports Proto3 Optional Copy Section

If you now try to invoke your own code generator with the test proto, you will
run into a different error:

$ protoc test_proto3_optional.proto --my_codegen_out=.
test_proto3_optional.proto: is a proto3 file that contains optional fields, but
code generator --my_codegen_out hasn't been updated to support optional fields in
proto3. Please ask the owner of this code generator to support proto3 optional.

This check exists to make sure that code generators get a chance to update
before they are used with proto3 optional fields. Without this check an old
code generator might emit obsolete generated APIs (like accessors for a
synthetic oneof) and users could start depending on these. That would create a
legacy migration burden once a code generator actually implements the feature.

To signal that your code generator supports optional fields in proto3, you
need to tell protoc what features you support. The method for doing this
depends on whether you are using the C++
google::protobuf::compiler::CodeGenerator framework or not.

If you are using the CodeGenerator framework:

class MyCodeGenerator : public google::protobuf::compiler::CodeGenerator {
  // Add this method.
  uint64_t GetSupportedFeatures() const override {
    // Indicate that this code generator supports proto3 optional fields.
    // (Note: don't release your code generator with this flag set until you
    // have actually added and tested your proto3 support!)
    return FEATURE_PROTO3_OPTIONAL;
  }
}

If you are generating code using raw CodeGeneratorRequest and
CodeGeneratorResponse messages from plugin.proto, the change will be very
similar:

void GenerateResponse() {
  CodeGeneratorResponse response;
  response.set_supported_features(CodeGeneratorResponse::FEATURE_PROTO3_OPTIONAL);

  // Generate code...
}

Once you have added this, you should now be able to successfully use your code
generator to generate a file containing proto3 optional fields:

$ protoc test_proto3_optional.proto --my_codegen_out=.

Updating Your Code Generator Copy Section

Now to actually add support for proto3 optional to your code generator. The goal
is to recognize proto3 optional fields as optional, and suppress any output from
synthetic oneofs.

If your code generator does not currently support proto2, you will need to
design an API and implementation for supporting presence in scalar fields.
Generally this means:

• allocating a bit inside the generated class to represent whether a given
  field is present or not.
• exposing a has_foo() method for each field to return the value of this
  bit.
• make the parser set this bit when a value is parsed from the wire.
• make the serializer test this bit to decide whether to serialize.

If your code generator already supports proto2, then most of your work is
already done. All you need to do is make sure that proto3 optional fields have
exactly the same API and behave in exactly the same way as proto2 optional
fields.

From experience updating several of Google's code generators, most of the
updates that are required fall into one of several patterns. Here we will show
the patterns in terms of the C++ CodeGenerator framework. If you are using
CodeGeneratorRequest and CodeGeneratorReply directly, you can translate the
C++ examples to your own language, referencing the C++ implementation of these
methods where required.

To test whether a field should have presence

Old:

bool MessageHasPresence(const google::protobuf::FieldDescriptor* field) {
  return field->has_presence();
}

New:

// Presence is no longer a property of a message, it's a property of individual
// fields.
bool FieldHasPresence(const google::protobuf::FieldDescriptor* field) {
  return field->has_presence();
  // Note, the above will return true for fields in a oneof.
  // If you want to filter out oneof fields, write this instead:
  //   return field->has_presence && !field->real_containing_oneof()
}

To test whether a field is a member of a oneof

Old:

bool FieldIsInOneof(const google::protobuf::FieldDescriptor* field) {
  return field->containing_oneof() != nullptr;
}

New:

bool FieldIsInOneof(const google::protobuf::FieldDescriptor* field) {
  // real_containing_oneof() returns nullptr for synthetic oneofs.
  return field->real_containing_oneof() != nullptr;
}

To iterate over all oneofs

Old:

bool IterateOverOneofs(const google::protobuf::Descriptor* message) {
  for (int i = 0; i < message->oneof_decl_count(); i++) {
    const google::protobuf::OneofDescriptor* oneof = message->oneof(i);
    // ...
  }
}

New:

bool IterateOverOneofs(const google::protobuf::Descriptor* message) {
  // Real oneofs are always first, and real_oneof_decl_count() will return the
  // total number of oneofs, excluding synthetic oneofs.
  for (int i = 0; i < message->real_oneof_decl_count(); i++) {
    const google::protobuf::OneofDescriptor* oneof = message->oneof(i);
    // ...
  }
}

Updating Reflection Copy Chapter

If your implementation offers reflection, there are a few other changes to make:

API Changes Copy Section

The API for reflecting over fields and oneofs should make the following changes.
These match the changes implemented in C++ reflection.

1. Add a FieldDescriptor::has_presence() method returning bool (adjusted to
   your language's naming convention). This should return true for all fields
   that have explicit presence, as documented in
   docs/field_presence. In particular, this includes
   fields in a oneof, proto2 scalar fields, and proto3 optional fields. This
   accessor will allow users to query what fields have presence without
   thinking about the difference between proto2 and proto3.
2. As a corollary of (1), please do not expose an accessor for the
   FieldDescriptorProto.proto3_optional field. We want to avoid having users
   implement any proto2/proto3-specific logic. Users should use the
   has_presence() function instead.
3. You may also wish to add a FieldDescriptor::has_optional_keyword() method
   returning bool, which indicates whether the optional keyword is present.
   Message fields will always return true for has_presence(), so this
   method can allow a user to know whether the user wrote optional or not. It
   can occasionally be useful to have this information, even though it does not
   change the presence semantics of the field.
4. If your reflection API may be used for a code generator, you may wish to
   implement methods to help users tell the difference between real and
   synthetic oneofs. In particular:
   • OneofDescriptor::is_synthetic(): returns true if this is a synthetic
     oneof.
   • FieldDescriptor::real_containing_oneof(): like containing_oneof(),
     but returns nullptr if the oneof is synthetic.
   • Descriptor::real_oneof_decl_count(): like oneof_decl_count(), but
     returns the number of real oneofs only.

Implementation Changes Copy Section

Proto3 optional fields and synthetic oneofs must work correctly when reflected
on. Specifically:

1. Reflection for synthetic oneofs should work properly. Even though synthetic
   oneofs do not really exist in the message, you can still make reflection
   work as if they did. In particular, you can make a method like
   Reflection::HasOneof() or Reflection::GetOneofFieldDescriptor() look at
   the hasbit to determine if the oneof is present or not.
2. Reflection for proto3 optional fields should work properly. For example, a
   method like Reflection::HasField() should know to look for the hasbit for
   a proto3 optional field. It should not be fooled by the synthetic oneof
   into thinking that there is a case member for the oneof.

Once you have updated reflection to work properly with proto3 optional and
synthetic oneofs, any code that uses your reflection interface should work
properly with no changes. This is the benefit of using synthetic oneofs.

In particular, if you have a reflection-based implementation of protobuf text
format or JSON, it should properly support proto3 optional fields without any
changes to the code. The fields will look like they all belong to a one-field
oneof, and existing proto3 reflection code should know how to test presence for
fields in a oneof.

So the best way to test your reflection changes is to try round-tripping a
message through text format, JSON, or some other reflection-based parser and
serializer, if you have one.

Validating Descriptors Copy Section

If your reflection implementation supports loading descriptors at runtime, you
must verify that all synthetic oneofs are ordered after all "real" oneofs.

Here is the code that implements this validation step in C++, for inspiration:

// Validation that runs for each message.
  // Synthetic oneofs must be last.
  int first_synthetic = -1;
  for (int i = 0; i < message->oneof_decl_count(); i++) {
    const OneofDescriptor* oneof = message->oneof_decl(i);
    if (oneof->is_synthetic()) {
      if (first_synthetic == -1) {
        first_synthetic = i;
      }
    } else {
      if (first_synthetic != -1) {
        AddError(message->full_name(), proto.oneof_decl(i),
                 DescriptorPool::ErrorCollector::OTHER,
                 "Synthetic oneofs must be after all other oneofs");
      }
    }
  }

  if (first_synthetic == -1) {
    message->real_oneof_decl_count_ = message->oneof_decl_count_;
  } else {
    message->real_oneof_decl_count_ = first_synthetic;
  }

Jvm Aot (`docs/jvm_aot.md`) Copy Section

Ahead Of Time (AOT) compilation for the Java Virtual Machine (JVM)

Ahead Of Time (AOT) compilation build tools such as those provided by
GraalVM's native-image
can require some configuration when using protobuf. Protobuf for the JVM uses
reflection, and some of its target classes are not possible to determine in
advance. This situation is unlikely to change.

The Lite version of protobuf for the JVM
avoids reflection and may be better suited for use with AOT compilation tooling.
This Lite version was originally targeted for use on Android which has similar
AOT compilation goals as GraalVM's native-image tool.

GraalVM native-image Copy Chapter

The GraalVM native-image tool can be configured with options for the
Java Native Interface
(JNI), http proxying, reflection, and other resources. While these configuration
options can be manually specified in JSON files, we recommend that you exercise
the application with
the assisted configuration agent.
The agent will generate native-image.properties files that you add to the
project's META-INF/native-image/ directory. The Native Image builder uses
configuration options provided in META-INF/native-image/ to construct
native-image command line arguments.

Options (`docs/options.md`) Copy Section

Protobuf Global Extension Registry

This file contains a global registry of known extensions for descriptor.proto,
so that any developer who wishes to use multiple 3rd party projects, each with
their own extensions, can be confident that there won't be collisions in
extension numbers.

If you need an extension number for your custom option (see
custom options),
please
send us a pull request to
add an entry to this doc, or
create an issue with info
about your project (name and website) so we can add an entry for you.

Existing Registered Extensions Copy Chapter

1. C# port of protocol buffers

   • Website: https://github.com/jskeet/protobuf-csharp-port
   • Extensions: 1000

2. Perl/XS port of protocol buffers

   • Website: http://code.google.com/p/protobuf-perlxs
   • Extensions: 1001

3. Objective-C port of protocol buffers

   • Website: http://code.google.com/p/protobuf-objc
   • Extensions: 1002

4. Google Wave Federation Protocol open-source release (FedOne)

   • Website: http://code.google.com/p/wave-protocol
   • Extensions: 1003

5. PHP code generator plugin

   • Website: ???
   • Extensions: 1004

6. GWT code generator plugin (third-party!)

   • Website: http://code.google.com/p/protobuf-gwt/
   • Extensions: 1005

7. Unix Domain RPC code generator plugin

   • Website: http://go/udrpc
   • Extensions: 1006

8. Object-C generator plugin (Plausible Labs)

   • Website: http://www.plausible.coop
   • Extensions: 1007

9. TBD (code42.com)

   • Website: ???
   • Extensions: 1008

10. Goby Underwater Autonomy Project

    • Website: https://github.com/GobySoft/goby
    • Extensions: 1009

11. Nanopb

    • Website: http://kapsi.fi/~jpa/nanopb
    • Extensions: 1010

12. Bluefin AUV Communication Extensions

    • Website: http://www.bluefinrobotics.com
    • Extensions: 1011

13. Dynamic Compact Control Language

    • Website: http://github.com/GobySoft/dccl
    • Extensions: 1012

14. ScaleOut StateServer® Native C++ API

    • Website: http://www.scaleoutsoftware.com
    • Extensions: 1013

15. FoundationDB SQL Layer

    • Website: https://github.com/FoundationDB/sql-layer
    • Extensions: 1014

16. Fender

    • Website: https://github.com/hassox/fender
    • Extensions: 1015

17. Vortex

    • Website: http://www.prismtech.com/vortex
    • Extensions: 1016

18. tresorit

    • Website: https://tresorit.com/
    • Extensions: 1017

19. CRIU (Checkpoint Restore In Userspace)

    • Website: http://criu.org/Main_Page
    • Extensions: 1018

20. protobuf-c

    • Website: https://github.com/protobuf-c/protobuf-c
    • Extensions: 1019

21. ScalaPB

    • Website: https://scalapb.github.io/
    • Extensions: 1020

22. protoc-gen-bq-schema

    • Website: https://github.com/GoogleCloudPlatform/protoc-gen-bq-schema
    • Extensions: 1021

23. grpc-gateway

    • Website: https://github.com/gengo/grpc-gateway
    • Extensions: 1022

24. Certificate Transparency

    • Website: https://github.com/google/certificate-transparency
    • Extensions: 1023

25. JUNOS Router Telemetry

    • Website: http://www.juniper.net
    • Extensions: 1024

26. Spine Event Engine

    • Website: https://github.com/SpineEventEngine/core-java
    • Extensions: 1025

27. Aruba cloud platform

    • Website: ???
    • Extensions: 1026 -> 1030

28. Voltha

    • Website: ???
    • Extensions: 1031 -> 1033

29. gator

    • Website: ???
    • Extensions: 1034

30. protoc-gen-flowtypes

    • Website:
      https://github.com/tmc/grpcutil/tree/master/protoc-gen-flowtypes
    • Extensions: 1035

31. ProfaneDB

    • Website: https://gitlab.com/ProfaneDB/ProfaneDB
    • Extensions: 1036

32. protobuf-net

    • Website: https://github.com/mgravell/protobuf-net
    • Extensions: 1037

33. FICO / StreamEngine

    • Website: http://www.fico.com/
    • Extensions: 1038

34. GopherJS

    • Website: https://github.com/johanbrandhorst/protobuf
    • Extensions: 1039

35. ygot

    • Website: https://github.com/openconfig/ygot
    • Extensions: 1040, 1179, 1180

36. go-grpcmw

    • Website: https://github.com/MarquisIO/go-grpcmw
    • Extensions: 1041

37. grpc-gateway protoc-gen-openapiv2

    • Website: https://github.com/grpc-ecosystem/grpc-gateway
    • Extensions: 1042

38. AN Message

    • Website: TBD
    • Extensions: 1043

39. protofire

    • Website: https://github.com/ribrdb/protofire
    • Extensions: 1044

40. Gravity

    • Website: https://github.com/aphysci/gravity
    • Extensions: 1045

41. SEMI Standards – I&C Technical Committee

    • Website:
      http://downloads.semi.org/web/wstdsbal.nsf/9c2b317e76523cca88257641005a47f5/88a5863a580e323088256e7b00707489!OpenDocument
    • Extensions: 1046

42. Elixir plugin

    • Website: https://github.com/tony612/grpc-elixir
    • Extensions: 1047

43. API client generators

    • Website: ???
    • Extensions: 1048-1056

44. Netifi Proteus

    • Website: https://github.com/netifi-proteus
    • Extensions: 1057

45. CGSN Mooring Project

    • Website: https://bitbucket.org/ooicgsn/cgsn-mooring
    • Extensions: 1058

46. Container Storage Interface

    • Website: https://github.com/container-storage-interface/spec
    • Extensions: 1059-1069

47. TwirpQL Plugin

    • Website: https://twirpql.dev
    • Extensions: 1070

48. Protoc-gen-validate

    • Website: https://github.com/bufbuild/protoc-gen-validate
    • Extensions: 1071

49. Protokt (pre 1.0.0)

    • Website: https://github.com/open-toast/protokt
    • Extensions: 1072

50. Dart port of protocol buffers

    • Website https://github.com/dart-lang/protobuf
    • Extensions: 1073

51. Ocaml-protoc-plugin

    • Website: https://github.com/issuu/ocaml-protoc-plugin
    • Extensions: 1074

52. Analyze Re Graphene

    • Website: https://analyzere.com
    • Extensions: 1075

53. Wire since and until

    • Website: https://square.github.io/wire/
    • Extensions: 1076, 1077

54. Bazel, Failure Details

    • Website: https://github.com/bazelbuild/bazel
    • Extensions: 1078

55. grpc-graphql-gateway

    • Website: https://github.com/ysugimoto/grpc-graphql-gateway
    • Extensions: 1079

56. Cloudstate

    • Website: https://cloudstate.io
    • Extensions: 1080-1084

57. SummaFT protoc-plugins

    • Website: https://summaft.com/
    • Extensions: 1085

58. ADLINK EdgeSDK

    • Website: https://www.adlinktech.com/en/Edge-SDK-IoT
    • Extensions: 1086

59. Wire wire_package

    • Website: https://square.github.io/wire/
    • Extensions: 1087

60. Confluent Schema Registry

    • Website: https://github.com/confluentinc/schema-registry
    • Extensions: 1088

61. ScalaPB Validate

    • Website: https://scalapb.github.io/docs/validation
    • Extension: 1089

62. Astounding (Currently Private)

    • Website: https://github.com/PbPipes/Astounding
    • Extension: 1090

63. Protoc-gen-psql

    • Website: https://github.com/Intrinsec/protoc-gen-psql
    • Extension: 1091-1101

64. Protoc-gen-sanitize

    • Website: https://github.com/Intrinsec/protoc-gen-sanitize
    • Extension: 1102-1106

65. Coach Client Connect (planned release in March 2021)

    • Website: https://www.coachclientconnect.com
    • Extension: 1107

66. Kratos API Errors

    • Website: https://go-kratos.dev
    • Extension: 1108

67. Glitchdot (Currently Private)

    • Website: https://go.glitchdot.com
    • Extension: 1109

68. eigr/protocol

    • Website: https://eigr.io
    • Extension: 1110-1114

69. Container Object Storage Interface (COSI)

    • Website:
      https://github.com/kubernetes-sigs/container-object-storage-interface-spec
    • Extension: 1115-1124

70. Protoc-gen-jsonschema

    • Website: https://github.com/chrusty/protoc-gen-jsonschema
    • Extension: 1125-1129

71. Protoc-gen-checker

    • Website: https://github.com/Intrinsec/protoc-gen-checker
    • Extension: 1130-1139

72. Protoc-gen-go-svc

    • Website: https://github.com/dane/protoc-gen-go-svc
    • Extension: 1140

73. Embedded Proto

    • Website: https://EmbeddedProto.com
    • Extension: 1141

74. Protoc-gen-fieldmask

    • Website: https://github.com/yeqown/protoc-gen-fieldmask
    • Extension: 1142

75. Google Gnostic

    • Website: https://github.com/google/gnostic
    • Extension: 1143

76. Protoc-gen-go-micro

    • Website: https://github.com/unistack-org/protoc-gen-go-micro
    • Extension: 1144

77. Protoc-gen-authz

    • Website: https://github.com/Neakxs/protoc-gen-authz
    • Extension: 1145

78. Protonium

    • Website: https://github.com/zyp/protonium
    • Extension: 1146

79. Protoc-gen-xo

    • Website: https://github.com/xo/ecosystem
    • Extension: 1147

80. Ballerina gRPC

    • Website: https://github.com/ballerina-platform/module-ballerina-grpc
    • Extension: 1148

81. Protoc-gen-referential-integrity

    • Website:
      https://github.com/ComponentCorp/protoc-gen-referential-integrity
    • Extension: 1149

82. Oclea Service Layer RPC

    • Website: https://oclea.com/
    • Extension: 1150

83. mypy-protobuf

    • Website: https://github.com/nipunn1313/mypy-protobuf
    • Extension: 1151-1154

84. Pigweed protobuf compiler

    • Website: https://pigweed.dev/pw_protobuf
    • Extension: 1155

85. Perfetto

    • Website: https://perfetto.dev
    • Extension: 1156

86. Buf

    • Website: http://buf.build/
    • Extension: 1157-1166

87. Connect RPC

    • Website: http://connectrpc.com/
    • Extension: 1167-1176

88. protocel

    • Website: https://github.com/Neakxs/protocel
    • Extension: 1177-1178

89. Cybozu

    • Website: https://github.com/cybozu/protobuf
    • Extension: 1179

90. EngFlow

    • Website: https://github.com/EngFlow/engflowapis
    • Extensions: 1181

91. Proto-telemetry

    • Website: https://github.com/clly/proto-telemetry
    • Extensions: 1182

92. Digital Twins Definition Language (DTDL)

    • Website: https://github.com/Azure/opendigitaltwins-dtdl
    • Extensions: 1183

93. RabbitMQ

    • Website: https://github.com/guihouchang/protoc-gen-go-event
    • Extensions: 1184

94. Wire use_array

    • Website: https://square.github.io/wire/
    • Extensions: 1185

95. Protons

    • Website: https://github.com/ipfs/protons
    • Extensions: 1186

96. gRPC Federation

    • Website: https://github.com/mercari/grpc-federation
    • Extensions: 1187

97. protoc-gen-openapi

    • Website: https://github.com/solo-io/protoc-gen-openapi
    • Extensions: 1188-1189

98. Wire enumMode

    • Website: https://square.github.io/wire/
    • Extensions: 1190

99. Nebius AI

    • Website: https://nebius.ai
    • Extensions: 1191-1199

100. Buf

     • Website: http://buf.build/
     • Extension: 1200-1239

101. userver

     • Website: https://userver.tech/
     • Extensions: 1240-1249

102. protoc-gen-default / protoc-gen-env (confpb)

     • Website: https://github.com/MarnixBouhuis/confpb
     • Extensions: 1250-1251

103. JaiaBot

     • Website: https://github.com/jaiarobotics/jaiabot
     • Extensions: 1252

104. Protokt (1.0.0 and up)

     • Website: https://github.com/open-toast/protokt
     • Extensions: 1253-1263

105. aep.dev Extensions

     • Website: https://github.com/aep-dev
     • Extensions: 1264-1274

106. Thermo Fisher Scientific TEM Extensions

     • Website: https://www.thermofisher.com/
     • Extensions: 1275-1284

107. LiveKit

     • Website: https://livekit.io
     • Extensions: 1285-1294

108. grpc-gateway protoc-gen-openapiv3

     • Website: https://github.com/grpc-ecosystem/grpc-gateway
     • Extensions: 1295

109. Protosource

     • Website: https://github.com/funinthecloud/protosource
     • Extensions: 1296-1305

110. protosearch

     • Website: https://github.com/benwebber/protosearch
     • Extensions: 1306

111. proto2type

     • Website: https://github.com/protocgen/proto2type
     • Extensions: 1307

112. protoc-gen-openapi

     • Website: https://github.com/engine-public/protoc-gen-openapi
     • Extensions: 1308

113. Motif

     • Website: https://motif.io
     • Extensions: 1309

114. Teleport

     • Website: https://goteleport.com
     • Extensions: 1310

115. proto2mcp

     • Website: https://github.com/protocgen/proto2mcp
     • Extensions: 1311-1313

116. ProtoWire

     • Website: https://protowire.org
     • Extensions: 1314-1363

117. Flyte

     • Website: https://github.com/flyteorg/flyte
     • Extensions: 1364-1373

118. durable

     • Website: https://github.com/dangra/durable
     • Extensions: 1374-1375

119. firestore-proto-codec

     • Website: https://github.com/willbinge/firestore-proto-codec
     • Extensions: 1376-1380

120. buffa

     • Website: https://github.com/anthropics/buffa
     • Extensions: 1381-1390

Third Party (`docs/third_party.md`) Copy Section

Third-Party Add-ons for Protocol Buffers

This page lists code related to Protocol Buffers which is developed and
maintained by third parties. You may find this code useful, but note that
these projects are not affiliated with or endorsed by Google (unless
explicitly marked); try them at your own risk. Also note that many projects
here are in the early stages of development and not production-ready.

If you have a project that should be listed here, please
send us a pull request to
update this page.

Programming Languages Copy Chapter

These are projects we know about implementing Protocol Buffers for other
programming languages:

• Action Script: https://github.com/matrix3d/JProtoc
• Action Script: https://github.com/zhongfq/protobuf-as3/
• Ada: https://github.com/reznikmm/protobuf
• C: https://github.com/protobuf-c/protobuf-c
• C: https://koti.kapsi.fi/jpa/nanopb/
• C: https://github.com/cloudwu/pbc/
• C: https://github.com/haberman/upb/wiki
• C: https://github.com/squidfunk/protobluff
• C: https://github.com/eerimoq/pbtools
• C++: https://github.com/protocolbuffers/protobuf (Google-official
  implementation)
• C++: https://github.com/tonda-kriz/simple-protobuf
• C++: https://EmbeddedProto.com
• C++: https://github.com/yksten/struct2x
• C/C++: http://spbc.sf.net/
• C#: https://silentorbit.com/protobuf/
• Clojure: http://github.com/ninjudd/clojure-protobuf
• Clojure: https://github.com/clojusc/protobuf
• Clojure: https://protojure.readthedocs.io
• Clojure: https://github.com/s-expresso/clojobuf
• ClojureScript: https://github.com/s-expresso/clojobuf
• Common Lisp: http://github.com/brown/protobuf
• Common Lisp: http://github.com/qitab/cl-protobuf
• D: https://github.com/dcarp/protobuf-d
• D: https://github.com/msoucy/dproto
• D: https://github.com/opticron/ProtocolBuffer
• Dart: https://github.com/dart-lang/dart-protobuf (runtime)
  https://github.com/dart-lang/dart-protoc-plugin (code generator)
• Delphi: http://sourceforge.net/projects/protobuf-delphi/
• Delphi: http://fundementals.sourceforge.net/dl.html
• Elixir: https://github.com/jeremyong/exprotoc
• Elixir: https://github.com/tony612/protobuf-elixir
• Elixir: https://github.com/ahamez/protox
• Elm: https://github.com/tiziano88/elm-protobuf
• Erlang: https://github.com/tomas-abrahamsson/gpb
• Erlang: http://piqi.org/
• Erlang: https://github.com/basho/erlang_protobuffs (no longer maintained,
  use gpb instead)
• Hacklang/HHVM: https://github.com/y3llowcake/proto-hack
• GDScript: https://github.com/oniksan/godobuf (Godot v3 engine plugin)
• Go: https://github.com/golang/protobuf (Google-official implementation)
• Go: https://github.com/akunspy/gopbuf
• Go: https://github.com/gogo/protobuf
• GopherJS: https://github.com/johanbrandhorst/protobuf
• Haskell: https://hackage.haskell.org/package/hprotoc
• Haskell: https://github.com/google/proto-lens (Google-unofficial
  implementation)
• Haskell: https://github.com/awakesecurity/proto3-suite (code generator)
  https://github.com/awakesecurity/proto3-wire (binary
  serializer/deserializer)
• Haxe: https://github.com/Atry/protoc-gen-haxe
• Java: https://github.com/protocolbuffers/protobuf (Google-official
  implementation)
• Java/Android: https://github.com/square/wire
• Java: https://github.com/HebiRobotics/QuickBuffers/
• Java ME: http://swingme.sourceforge.net/encode.shtml
• Javascript: http://github.com/bufbuild/protobuf-es
• Javascript: http://github.com/sirikata/protojs
• Javascript: https://github.com/dcodeIO/ProtoBuf.js
• Javascript: https://github.com/seishun/node-protoc-plugin (Node.js port of
  plugin.h)
• Javascript: https://github.com/seishun/node-protoc-gen-javascript (Node.js
  port of the Google-official implementation)
• Javascript: https://github.com/ButterCam/sisyphus-js
• Julia: https://github.com/JuliaIO/ProtoBuf.jl
• Kotlin: https://github.com/marcoferrer/kroto-plus
• Kotlin: https://github.com/Kotlin/kotlinx.serialization
• Kotlin: https://github.com/ButterCam/sisyphus
• Kotlin: https://github.com/open-toast/protokt
• Kotlin Multiplatform: https://github.com/streem/pbandk
• Lua: http://github.com/indygreg/lua-protobuf
• Lua: https://github.com/Neopallium/lua-pb
• Objective C: https://github.com/alexeyxo/protobuf-objc
• OCaml: http://piqi.org/
• Perl: http://groups.google.com/group/protobuf-perl
• Perl: https://metacpan.org/pod/Google::ProtocolBuffers
• Perl: https://metacpan.org/pod/Google::ProtocolBuffers::Dynamic
• PHP: https://github.com/allegro/php-protobuf/
• PHP: https://github.com/chobie/php-protocolbuffers
• Protobuf: https://github.com/mycroftjr/protoc-gen-proto
• Prolog: http://www.swi-prolog.org/pldoc/package/protobufs.html
• Purescript: https://github.com/xc-jp/purescript-protobuf
• Python: https://github.com/protocolbuffers/protobuf (Google-official
  implementation)
• Python: https://github.com/eigenein/protobuf
• Python: https://github.com/danielgtaylor/python-betterproto
• R: http://cran.r-project.org/package=RProtoBuf
• Ruby: http://github.com/mozy/ruby-protocol-buffers
• Ruby: https://github.com/bmizerany/beefcake/tree/master/lib/beefcake
• Ruby: https://github.com/localshred/protobuf
• Rust: https://github.com/tokio-rs/prost
• Rust: https://github.com/stepancheg/rust-protobuf/
• Rust: https://github.com/tafia/quick-protobuf
• Scala: http://github.com/jeffplaisance/scala-protobuf
• Scala: https://github.com/SandroGrzicic/ScalaBuff
• Scala: https://scalapb.github.io
• Solidity: https://github.com/celer-network/pb3-gen-sol
• Swift: https://github.com/alexeyxo/protobuf-swift
• Swift: https://github.com/apple/swift-protobuf/
• Typescript: http://github.com/bufbuild/protobuf-es
• Typescript: https://github.com/improbable-eng/ts-protoc-gen
• Typescript: https://github.com/thesayyn/protoc-gen-ts
• Typescript: https://github.com/pbkit/pbkit
• Vala: https://launchpad.net/protobuf-vala
• Zig: https://github.com/Arwalk/zig-protobuf

RPC Implementations Copy Chapter

GRPC (http://www.grpc.io/) is Google's RPC implementation for Protocol Buffers.
There are other third-party RPC implementations as well. Some of these actually
work with Protocol Buffers service definitions (defined using the service
keyword in .proto files) while others just use Protocol Buffers message
objects.

• https://github.com/grpc/grpc (C++, Node.js, Python, Ruby, Objective-C, PHP,
  C#, Google-official implementation)
• https://connectrpc.com (Go, Javascript, Typescript, Python, Dart, Swift,
  Kotlin)
• https://zeroc.com/products/ice (Multiple languages)
• https://github.com/protobuf-net/protobuf-net (C#/.NET/WCF/VB)
• http://www.deltavsoft.com/doc/_external_serialization.html#Protobufs (C++)
• https://protojure.github.io (Clojure)
• https://github.com/baidu/sofa-pbrpc (C++)
• https://github.com/madhon/protobuf-csharp-rpc (C#)
• https://github.com/madwyn/libpbrpc (C++)
• https://github.com/SeriousMa/grpc-protobuf-validation (Java)
• https://github.com/elixir-grpc/grpc (Elixir)
• https://github.com/johanbrandhorst/protobuf (GopherJS)
• https://github.com/awakesecurity/gRPC-haskell (Haskell)
• https://github.com/Yeolar/raster (C++)
• https://github.com/jnordberg/wsrpc (JavaScript Node.js/Browser)
• https://github.com/pbkit/npm-packages/blob/main/frpc-test/src/index.spec.ts
  (TypeScript Node.js/Browser)
• https://github.com/ppissias/xsrpcj (Java)
• https://github.com/twitchtv/twirp (Multiple languages)
• https://github.com/icerpc/icerpc-csharp (C#)
• https://philips-software.github.io/amp-embedded-infra-lib (C++, C#, Java)
• https://github.com/tdaira/blerpc (C/nanopb, Python, Kotlin, Swift, Dart,
  TypeScript; RPC over Bluetooth Low Energy)

Inactive:

• https://code.google.com/p/protobuf-actionscript3/ (Action Script)
• https://code.google.com/p/protoc-gen-as3/ (Action Script)
• https://code.google.com/p/protobuf-csharp-port (C#)
• https://code.google.com/p/protobuf-net/ (C#/.NET/WCF/VB)
• https://code.google.com/p/protobuf-javame/ (Java ME)
• https://code.google.com/p/protobuf-rpc-pro/ (Java)
• https://launchpad.net/txprotobuf/ (Python)
• https://github.com/modeswitch/protobuf-rpc (Python)
• https://github.com/sdeo/protobuf-socket-rpc (Java, Python)
• https://code.google.com/p/proto-streamer/ (Java)
• https://code.google.com/p/server1/ (C++)
• https://code.google.com/p/protobuf-mina-rpc/ (Python client, Java server)
• https://code.google.com/p/casocklib/ (C++)
• https://code.google.com/p/cxf-protobuf/ (Java)
• https://code.google.com/p/protobuf-remote/ (C++/C#)
• https://code.google.com/p/eneter-protobuf-serializer/ (Java/.NET)
• https://github.com/robbinfan/claire/tree/master/protorpc (C++)
• https://github.com/ebencheung/arab (C++)
• https://code.google.com/p/protobuf-csharp-rpc/ (C#)
• https://github.com/thesamet/rpcz (C++/Python, based on ZeroMQ)
• https://github.com/w359405949/libmaid (C++, Python)
• https://code.google.com/p/protobuf-js/ (Javascript)
• https://code.google.com/p/protobuf-for-node/ (Javascript)
• https://code.google.com/p/protostuff/ (Javascript)
• https://code.google.com/p/protoc-gen-lua/ (Lua)
• https://code.google.com/p/protobuf-matlab/ (Matlab)
• https://code.google.com/p/protobuf-mercury/ (Mercury)
• https://code.google.com/p/protobuf-objc/ (ObjC)
• https://code.google.com/p/protobuf-perlxs/ (Perl/XS)
• https://code.google.com/p/pb4php/ (PHP)
• https://code.google.com/p/ruby-protobuf/ (Ruby)
• https://code.google.com/p/protobuf-scala (Scala)
• https://code.google.com/p/protobuf-net/ (Visual Basic)
• https://code.google.com/p/protoc-gen-docbook/ (DocBook generator)

Build Copy Chapter

• Bazel Build
  • rules_closure
    js-closure
  • rules_go go
  • rules_protobuf java c++
c# go js-closure js-node python ruby
• Maven Protobuf Compiler Plugin
  • By xolstice.org
    (Documentation)
    (Source)
    
  • https://code.google.com/p/maven-protoc-plugin/
  • https://github.com/os72/protoc-jar-maven-plugin
  • https://ascopes.github.io/protobuf-maven-plugin
• Protobuf Plugin for Gradle
• Sbt plugin for Protocol Buffers

IDE Copy Chapter

• Buf for Visual Studio Code
• Visual Studio Code Support for Protocol Buffers
• Visual Studio Language Service support for Protocol Buffers
• IntelliJ IDEA plugin
• IntelliJ Protobuf Plugin
• TextMate syntax highlighting
• Notepad++ Syntax Highlighting for .proto files
• Eclipse editor for protobuf (from Google)
• NetBeans IDE plugin
• Editor for serialized protobufs

Documentation Copy Chapter

• Documentation generator plugin (Markdown/HTML/DocBook/...)

Other Utilities Copy Chapter

There are miscellaneous other things you may find useful as a Protocol Buffers
developer.

• Buf CLI
  • Code generation
  • Linting
  • Breaking change detection
  • Formatting
  • Calling gRPC APIs
  • Editor integration and LSP
• Wireshark/Ethereal packet sniffer plugin
• Alternate encodings (JSON, XML, HTML) for Java protobufs
• Another JSON encoder/decoder for Java
• Oracle PL SQL plugin
• C++ Builder compatible protobuf
• Protobuf for nginx module
• RSpec matchers and Cucumber step defs for testing Protocol Buffers
• Multi-platform executable JAR and Java API for protoc
• Python scripts to convert between Protocol Buffers and JSON
• C++ library for serialization/de-serialization between Protocol Buffers and
  JSON.
• ProtoBuf with Java EE7 Expression Language 3.0; pure Java ProtoBuf Parser
  and Builder.
• Linter for .proto files
• Protocol Buffers Dynamic Schema - create protobuf schemas programmatically
  (Java)
• Make protoc plugins in NodeJS
• ProfaneDB - A Protocol Buffers database
• Protocol Buffer property-based testing utility and example message
  generator (Python /
  Hypothesis)
• Protolock - CLI utility to prevent backward-incompatible changes to .proto
  files
• Optional GRPC - GRPC for testable microservices (Python)
• Protobuf Parser - Yet another Go package which parses a Protocol Buffer
  file (proto2+proto3)
• Protolint - A tool to enforce Protocol Buffer style and conventions.
  • vscode-protolint: A protobuf linter for visual studio code
  • intellij-protolint: A protobuf linter for JetBrains IDEs
  • vim-protolint: A protobuf linter for Vim
• super-linter: Protocol Buffer lint as GitHub Action
• protoc-gen-fieldmask - A plugin to generate static type fieldmask paths
• protoc-gen-bq-schema - A protoc plugin to generate BigQuery schema files
• protobuf-decoder-explainer - Web page which decodes and explains pasted
  binary Protocol
  Buffers
• rubberbuf - dynamically transpile protobuf into AST
  clojure clojurescript
• clojobuf-serdes - low level serialize/deserialize library for protobuf
  binary format clojure
clojurescript
• gRPC Federation - generates a gRPC server by writing a custom option in
  Protocol Buffers
• Proto Boiler - A plugin to generate boilerplate code from .proto files
  using templates

Csharp Proto2 (`docs/csharp/proto2.md`) Copy Section

As part of the 3.10 release of Google.Protobuf, experimental proto2 support has been released. This document outlines the new changes brought about to include proto2 support. This does not break existing proto3 support and users may continue to use proto3 features without changing their current code. Again the generated code and public API associated with proto2 is experimental and subject to change in the future. APIs for proto2 may be added, removed, or adjusted as feedback is received.
Generated code for proto2 may also be modified by adding, removing, or adjusting APIs as feedback is received.

Enabling proto2 features Copy Section

For information about specific proto2 features, please read the proto2 language guide.

Much like other languages, proto2 features are used with proto2 files with the syntax declaration syntax = "proto2";. However, please note, proto3 is still the recommended version of protobuf and proto2 support is meant for legacy system interop and advanced uses.

Generated code

Messages Copy Section

Messages in proto2 files are very similar to their proto3 counterparts. They expose the usual property for getting and setting, but they also include properties and methods to handle field presence.

For optional/required field XYZ, a HasXYZ property is included for checking presence and a ClearXYZ method is included for clearing the value.

message Foo {
    optional Bar bar = 1;
    required Baz baz = 2;
}

var foo = new Foo();
Assert.IsNull(foo.Bar);
Assert.False(foo.HasBar);
foo.Bar = new Bar();
Assert.True(foo.HasBar);
foo.ClearBar();

Messages with extension ranges Copy Section

Messages which define extension ranges implement the IExtendableMessage interface as shown below.
See inline comments for more info.

public interface IExtendableMessage<T> : IMessage<T> where T : IExtendableMessage<T>
{
    // Gets the value of a single value extension. If the extension isn't present, this returns the default value.
    TValue GetExtension<TValue>(Extension<T, TValue> extension);
    // Gets the value of a repeated extension. If the extension hasn't been set, this returns null to prevent unnecessary allocations.
    RepeatedField<TValue> GetExtension<TValue>(RepeatedExtension<T, TValue> extension);
    // Gets the value of a repeated extension. This will initialize the value of the repeated field and will never return null.
    RepeatedField<TValue> GetOrInitializeExtension<TValue>(RepeatedExtension<T, TValue> extension);
    // Sets the value of the extension
    void SetExtension<TValue>(Extension<T, TValue> extension, TValue value);
    // Returns whether the extension is present in the message
    bool HasExtension<TValue>(Extension<T, TValue> extension);
    // Clears the value of the extension, removing it from the message
    void ClearExtension<TValue>(Extension<T, TValue> extension);
    // Clears the value of the repeated extension, removing it from the message. Calling GetExtension after this will always return null.
    void ClearExtension<TValue>(RepeatedExtension<T, TValue> extension);
}

Extensions Copy Section

Extensions are generated in static containers like reflection classes and type classes.
For example for a file called foo.proto containing extensions in the file scope, a
FooExtensions class is created containing the extensions defined in the file scope.
For easy access, this class can be used with using static to bring all extensions into scope.

option csharp_namespace = "FooBar";
extend Foo {
    optional Baz foo_ext = 124;
}
message Baz {
    extend Foo {
        repeated Baz repeated_foo_ext = 125;
    }
}

public static partial class FooExtensions {
    public static readonly Extension<Foo, Baz> FooExt = /* initialization */;
}

public partial class Baz {
    public partial static class Extensions {
        public static readonly RepeatedExtension<Foo, Baz> RepeatedFooExt = /* initialization */;
    }
}

using static FooBar.FooExtensions;
using static FooBar.Baz.Extensions;

var foo = new Foo();
foo.SetExtension(FooExt, new Baz());
foo.GetOrInitializeExtension(RepeatedFooExt).Add(new Baz());

APIs

Message initialization Copy Section

Initialization refers to checking the status of required fields in a proto2 message. If a message is uninitialized, not all required fields are set in either the message itself or any of its submessages. In other languages, missing required fields throw errors depending on the merge method used. This could cause unforeseen errors at runtime if the incorrect method is used.
However, in this implementation, parsers and input streams don't check messages for initialization on their own and throw errors. Instead it's up to you to handle messages with missing required fields in whatever way you see fit.
Checking message initialization can be done manually via the IsInitialized extension method in MessageExtensions.

Extension registries Copy Section

Just like in Java, extension registries can be constructed to parse extensions when reading new messages
from input streams. The API is fairly similar to the Java API with some added bonuses with C# syntax sugars.

message Baz {
    extend Foo {
        optional Baz foo_ext = 124;
    }
}

var registry = new ExtensionRegistry() 
{ 
    Baz.Extensions.FooExt
};
var foo = Foo.Factory.WithExtensionRegistry(registry).ParseFrom(input);
Assert.True(foo.HasExtension(Bas.Extensions.FooExt));
var fooNoRegistry = Foo.Factory.ParseFrom(input);
Assert.False(foo.HasExtension(Bas.Extensions.FooExt));

Custom options Copy Section

Due to their limited use and lack of type safety, the original CustomOptions APIs are now deprecated. Using the new generated extension identifiers, you can access extensions safely through the GetOption APIs. Note that cloneable values such as
repeated fields and messages will be deep cloned.

Example based on custom options usage example here.

foreach (var service in input.Services)
{
    Console.WriteLine($"  {service.Name}");
    foreach (var method in service.Methods)
    {
        var rule = method.GetOption(AnnotationsExtensions.Http);
        if (rule != null)
        {
            Console.WriteLine($"    {method.Name}: {rule}");
        }
        else
        {
            Console.WriteLine($"    {method.Name}: no HTTP binding");
        }
    }
}

Reflection Copy Section

Reflection APIs have been extended to enable accessing the new proto2 portions of the library and generated code.

• FieldDescriptor.Extension
  • Gets the extension identifier behind an extension field, allowing it to be added to an ExtensionRegistry
• FieldDescriptor.IsExtension
  • Returns whether a field is an extension of another type.
• FieldDescriptor.ExtendeeType
  • Returns the extended type of an extension field
• IFieldAccessor.HasValue
  • Returns whether a field's value is set. For proto3 fields, throws an InvalidOperationException.
• FileDescriptor.Syntax
  • Gets the syntax of a file
• FileDescriptor.Extensions
  • An immutable list of extensions defined in the file
• MessageDescriptor.Extensions
  • An immutable list of extensions defined in the message

var extensions = Baz.Descriptor.Extensions.GetExtensionsInDeclarationOrder(Foo.Descriptor);
var registry = new ExtensionRegistry();
registry.AddRange(extensions.Select(f => f.Extension));

var baz = Foo.Descriptor.Parser.WithExtensionRegistry(registry).ParseFrom(input);
foreach (var field in extensions)
{
    if (field.Accessor.HasValue(baz))
    {
        Console.WriteLine($"{field.Name}: {field.Accessor.GetValue(baz)}");
    }
}

Upb Arena Fusion (`docs/upb/arena_fusion.md`) Copy Section

μpb Arena Fusion and References

μpb generally follows a thread-compatibility model where only operations on a
const pointer may occur concurrently from multiple threads; non-const
operations must not race with each other or operations on const pointers.

Fusion, via upb_Arena_Fuse, binds the lifetime of two arenas together, such
that none are freed until all of the transitively fused arenas reach refcount 0.
This is useful to avoid dangling pointers when a message in one arena references
a message in another - such as when a child message is constructed in isolation
then set onto a parent message - by fusing the parent's arena with the child's,
the child's lifetime is tied to the parent's without needing to copy it.

Both modifying the refcount and fusing are thread safe; if you want to
conceptually perform allocations concurrently from multiple threads with the
same arena lifetime, one way to achieve that would be to share a const upb_Arena* parent plus a dedicated arena per thread, and fuse the
thread-specific arena with the parent arena. By contrast, reference counting
cannot help to achieve concurrent allocations on multiple threads, it only helps
in synchronizing lifetimes from multiple things observing the same arena (which
may be a non-const pointer for multiple writers in a single threaded context,
or a const pointer for multiple readers in a multithreaded context).

To be usable everywhere, fusion has a lock-free implementation, documented
below.

Data Structure Copy Chapter

Each arena has three pointer sized members used to track the relationship
between arenas. Together, they implement a hybrid disjoint set forest and
doubly-linked list.

// Tagged pointer - tracked as black arrows in diagrams
UPB_ATOMIC(uintptr_t) parent_or_count;
// Linked list - tracked as red arrows in diagrams
UPB_ATOMIC(struct upb_ArenaInternal*) next;
// Linked list - previous tracked as blue arrows in diagrams, tail as dashed
UPB_ATOMIC(uintptr_t) previous_or_tail;

The disjoint set is used to check whether two arenas are already fused and to
provide agreement on a single reference count across all fused arenas. The
linked list is used to implement freeing arenas once the refcount reaches 0, and
to implement upb_Arena_SpaceAllocated.

Two arenas are considered fused when the disjoint set says they are; the linked
list is guaranteed to converge before the refcount reaches 0, but may only track
a subset of fused arenas while racing with fuse calls.

Finding the root Copy Chapter

Each set of fused arenas is uniquely identified by the root node of its tree. To
find the root for a given arena, traverse the parent_or_count pointer until
you reach a node with a count instead of a parent pointer - that's the root. To
avoid expensive lookups if frequently repeated, this data structure uses path
splitting to halve the distance from the root for every node traversed.
Repeatedly finding the root for a series of nodes that are part of the same
fused set will converge to O(1) quickly. Let's find the root of D, starting
with:

digraph {
C [label="C (next)"]
D [label="D (current)"]
A -> B [dir=back]
B -> C [dir=back]
C -> D [dir=back]
}

We traverse our parents; each time we pass one we link it to its grandparent, so
that future lookups are faster.

digraph {
B [label="B (next)"]
C [label="C (current)"]
A -> B [dir=back]
B -> C [dir=back]
B -> D [dir=back]
}

Then:

digraph {
A [label="A (next)"]
B [label="B (current)"]
A -> B [dir=back]
A -> C [dir=back]
B -> D [dir=back]
}

D and C's paths got shorter; if we query D's root again, all subsequent lookups
will only require one step.

Fusing Copy Chapter

Fusing starts by identifying the roots of each arena to fuse - if they already
share a root, there's nothing to be done. We'll work through the example of
fusing C and D - which is the same as fusing A and B, as they're the roots.

digraph {
A [label="A (refs=2)"]
B [label="B (refs=3)"]
A -> C [dir=back]
A -> C [weight=0.001, color="red"]
C -> A [weight=0.001, color="blue"]
A -> C [style=dashed]
B -> D [dir=back]
B -> D [weight=0.001, color="red"]
D -> B [weight=0.001, color="blue"]
B -> D [style=dashed]
}

A and B are roots, so they store a refcount in their parent_or_count and the
start of the linked list in their next pointers; C and D are not, so they
store pointers to their parent node in parent_or_count. C and D store NULL
in next as they happen to be the last nodes in their set.

Pass refcount Copy Section

As soon as the parent changes, all future refcount operations will switch to the
new root. To avoid decrementing to 0 while the old root still has active
references, we add the lower-addressed node's refcount to the new root.

digraph {
A [label="A (refs=5)"]
B [label="B (refs=3)"]
A -> C [dir=back]
A -> C [weight=0.001, color="red"]
C -> A [weight=0.001, color="blue"]
A -> C [style=dashed]
B -> D [dir=back]
B -> D [weight=0.001, color="red"]
D -> B [weight=0.001, color="blue"]
B -> D [style=dashed]
}

The total transferred refcount is tracked throughout the operation so it can be
fixed up at the end if retries resulted in over-transfer.

Union Copy Section

Now the arena with a lower address becomes the new root (A in this case), by
compare-and-exchange the higher address arena's parent_or_count pointer.

digraph {
A [label="A (refs=5)"]
A -> C [dir=back]
A -> C [weight=0.001, color="red"]
C -> A [weight=0.001, color="blue"]
A -> C [style=dashed]
B -> D [dir=back]
B -> D [weight=0.001, color="red"]
D -> B [weight=0.001, color="blue"]
B -> D [style=dashed]
A -> B [dir=back]
}

This atomically eliminates B's refcount and makes it one of A's children. Any
change to B's refcount (or fusing it to a different arena with a lower address)
will make the swap fail, and the process starts again from the beginning.

Refcount fixups Copy Section

If B's refcount changed while performing the union, we increase or decrease A's
refcount by the difference between the initial refcount we added and B's final
refcount, observed as part of the compare-and-exchange of B's parent_or_count.

Fuse linked lists Copy Section

To fuse the linked lists, we need to make A's tail point to B. The root node is
always the start of the linked list, so that the list can't contain cycles. From
A's tail pointer, we traverse until we hit the end (C in this case), and CAS its
next to B.

digraph {
A [label="A (refs=5)"]
A -> C [dir=back]
A -> C [weight=0.001, color="red"]
C -> A [weight=0.001, color="blue"]
A -> C [style=dashed]
B -> D [dir=back]
B -> D [weight=0.001, color="red"]
D -> B [weight=0.001, color="blue"]
B -> D [style=dashed]
A -> B [dir=back]
C -> B [weight=0.001, color="red"]
}

With this, B is now reachable from the root and the final free operation will be
able to see it, traversing A->C->B->D.

Update tail pointer Copy Section

If we stopped at the previous step, it's easy to see how fusing many arenas
could result in O(n^2) behavior, as we'd spend all of our time traversing the
linked list from the root to find the tail. To avoid this, we update A's tail
pointer to point to B's tail - so the next fuse operation will not have to
traverse C or B.

digraph {
A [label="A (refs=5)"]
A -> C [dir=back]
A -> C [weight=0.001, color="red"]
C -> A [weight=0.001, color="blue"]
A -> D [style=dashed]
B -> D [dir=back]
B -> D [weight=0.001, color="red"]
D -> B [weight=0.001, color="blue"]
B -> D [style=dashed]
A -> B [dir=back]
C -> B [weight=0.001, color="red"]
}

Update back links Copy Section

To make it possible to always traverse the list in both directions for
upb_Arena_SpaceAllocated, we need to make the matching back link for our
doubly-linked list. Since we linked C to B, we need to link B to C via B's
previous_or_tail pointer.

digraph {
A [label="A (refs=5)"]
A -> C [dir=back]
A -> C [weight=0.001, color="red"]
C -> A [weight=0.001, color="blue"]
A -> D [style=dashed]
B -> D [dir=back]
B -> D [weight=0.001, color="red"]
D -> B [weight=0.001, color="blue"]
B -> C [weight=0.001, color="blue"]
A -> B [dir=back]
C -> B [weight=0.001, color="red"]
}

One-Way References Copy Chapter

Fusion creates a bi-directional lifetime dependency between arenas: fused arenas
will have the same lifetime. In some cases, only a one-way dependency is needed.
If messages in arena A contain pointers to messages in arena B but not the
reverse, then arena B only needs to live at least as long as arena A.

upb_Arena_RefArena(A, B) establishes such a one-way dependency by having arena
A increment B's refcount. When arena A is freed, it releases its reference
on B. This is implemented by allocating a special block in A that holds a
pointer to B; this block is processed during upb_Arena_Free(A). Care is
taken to ensure that these blocks are processed before the memory blocks
containing them are freed from the underlying allocator.

Unlike fusion, upb_Arena_RefArena(A, B) is not thread-safe when called
concurrently with other operations on arena A, but it is thread-safe on B.

It is an error to create cycles of references (e.g., upb_Arena_RefArena(A, B); upb_Arena_RefArena(B, A)), or to create a reference between arenas that are
fused. Since fusion creates a bi-directional dependency, fusing arenas A and B
and calling upb_Arena_RefArena(B, A) would create a cycle A <-> B -> A. In
debug builds, upb checks for such cycles, with the implementation documented
below.

Counting allocated space Copy Chapter

Traversing the linked list while the arenas are still live can be tricky, as we
can't necessarily reach our own node via the linked list from the root. Instead,
we scan the linked list in both directions starting from the caller-specified
node. This makes space counting weakly consistent - it's possible to see that A
and B are fused, but not see A's space reflected in SpaceAllocated(B) if the
fuse operation that joined them is still in progress. But counting allocated
space is always consistent with itself and any fully complete fuses - nodes are
only appended or prepended to the list, so starting the traversal at the same
point each time guarantees that future scans see a superset of previous ones.
Referenced arenas are not included in the count of allocated space.

Detecting reference cycles Copy Chapter

To help prevent memory leaks from uncollectable arenas, upb checks for cycles in
debug builds when creating a reference or fusing arenas. A cycle can be formed
purely by references (eg. A->B->A) or by a combination of references and
fusions (eg. Fuse(A, B), then RefArena(B, A) creates the cycle A<->B->A).
If such cycles were permitted, the arenas involved could never be freed.

After performing a fusion or adding a reference, the cycle detector runs. This
check is performed after the operation because cycle detection cannot be
performed atomically; if the check was done prior to the fusion or reference, we
could miss a case where two concurrent operations resulted in a cycle. Both
would check for a possible cycle, find none, and then proceed.

Consider arenas with references A->B->C.

digraph {
  rankdir=LR;
  A; B; C;
  A -> B [label="ref", color=blue];
  B -> C [label="ref", color=blue];
}

If we attempt RefArena(C, A), the call will add ref C->A, then check that
C is not reachable from A.

digraph {
  rankdir=LR;
  A -> B [label="ref", color=blue];
  B -> C [label="ref", color=blue];
  C -> A [label="ref", color=blue];
}

The traversal will reveal A -> B -> C and trigger an assertion failure.

If instead attempt Fuse(A, C), fusion will occur:

digraph {
  rankdir=LR;
  node [shape=box, style=rounded];
  A -> B [label="ref", color=blue];
  B -> C [label="ref", color=blue];
  C -> A [label="fuse", dir=both, color=red];
}

The traversal will find C <-> A -> B -> C, and trigger an assertion failure.

Cycle detection algorithm Copy Section

Cycles are detected with a non-memoizing recursive depth-first search (DFS). A
path can traverse fusions bi-directionally and references uni-directionally to
find a cycle that contains at least one unidirectional edge. While this is not
asymptotically optimal as it may traverse the same set of nodes numerous times,
it doesn't require allocating memory, making it less intrusive as a debug-only
check. It also may result in infinite recursion if a cycle is created on one
thread while another thread is performing cycle checks, but that would have
caused an assertion failure anyway.

Fast check for fusion

If a directional ref was added and from and to are already fused
(upb_Arena_IsFused(from, to) returns true), they are mutually reachable, so a
path containing a directed edge exists and the assertion fails.

Traversing fusion group members

To check all references originating from from's fusion group, we must visit
every arena fused with from. Because fusion operations can race with this
check, we cannot rely on traversing from the fusion root, which might change.
Instead, like upb_Arena_SpaceAllocated, the algorithm finds members of
from's fusion group by first traversing backwards from from using
previous_or_tail, then iterates forwards.

Traverse fusion group and DFS on references

From the head of the list segment, we traverse forwards using next to visit
all nodes in from's fusion group. For each arena X in this group, we iterate
through all of its outgoing references (the list of arenas Y such that
RefArena(X, Y) was called). For each such reference X -> Y:

• If Y == to, then to is directly referenced, so a path exists. We return
  true and eventually trigger an assertion failure.
• If Y != to, we recurse to continue the depth-first search from Y,
  finding members of its fusion group and inspecting their outgoing
  references. If this recursive call returns true, then to is reachable
  via Y, and we return true to trigger an assertion failure.

If we explore all arenas in the fusion group and all their transitive references
without finding a path to to, no path exists, and we return false.

Upb Design (`docs/upb/design.md`) Copy Section

upb Design

[TOC]

upb is a protobuf kernel written in C. It is a fast and conformant
implementation of protobuf, with a low-level C API that is designed to be
wrapped in other languages.

upb is not designed to be used by applications directly. The C API is very
low-level, unsafe, and changes frequently. It is important that upb is able to
make breaking API changes as necessary, to avoid taking on technical debt that
would compromise upb's goals of small code size and fast performance.

Design goals Copy Chapter

Goals:

• Full protobuf conformance
• Small code size
• Fast performance (without compromising code size)
• Easy to wrap in language runtimes
• Easy to adapt to different memory management schemes (refcounting, GC, etc)

Non-Goals:

• Stable API
• Safe API
• Ergonomic API for applications

Parameters:

• C99
• 32 or 64-bit CPU (assumes 4 or 8 byte pointers)
• Uses pointer tagging, but avoids other implementation-defined behavior
• Aims to never invoke undefined behavior (tests with ASAN, UBSAN, etc)
• No global state, fully re-entrant

Arenas Copy Chapter

All memory management in upb uses arenas, using the type upb_Arena. Arenas are
an alternative to malloc() and free() that significantly reduces the costs
of memory allocation.

Arenas obtain blocks of memory using some underlying allocator (likely
malloc() and free()), and satisfy allocations using a simple bump allocator
that walks through each block in linear order. Allocations cannot be freed
individually: it is only possible to free the arena as a whole, which frees all
of the underlying blocks.

Here is an example of using the upb_Arena type:

upb_Arena* arena = upb_Arena_New();

  // Perform some allocations.
  int* x = upb_Arena_Malloc(arena, sizeof(*x));
  int* y = upb_Arena_Malloc(arena, sizeof(*y));

  // We cannot free `x` and `y` separately, we can only free the arena
  // as a whole.
  upb_Arena_Free(arena);

upb uses arenas for all memory management, and this fact is reflected in the API
for all upb data structures. All upb functions that allocate take a upb_Arena*
parameter and perform allocations using that arena rather than calling
malloc() or free().

// upb API to create a message.
UPB_API upb_Message* upb_Message_New(const upb_MiniTable* mini_table,
                                     upb_Arena* arena);

void MakeMessage(const upb_MiniTable* mini_table) {
  upb_Arena* arena = upb_Arena_New();

  // This message is allocated on our arena.
  upb_Message* msg = upb_Message_New(mini_table, arena);

  // We can free the arena whenever we want, but we cannot free the
  // message separately from the arena.
  upb_Arena_Free(arena);

  // msg is now deleted.
}

Arenas are a key part of upb's performance story. Parsing a large protobuf
payload usually involves rapidly creating a series of messages, arrays (repeated
fields), and maps. It is crucial for parsing performance that these allocations
are as fast as possible. Equally important, freeing the tree of messages should
be as fast as possible, and arenas can reduce this cost from O(n) to O(lg n).

Avoiding Dangling Pointers Copy Section

Objects allocated on an arena will frequently contain pointers to other
arena-allocated objects. For example, a upb_Message will have pointers to
sub-messages that are also arena-allocated.

Unlike unique ownership schemes (such as unique_ptr<>), arenas cannot provide
automatic safety from dangling pointers. Instead, upb provides tools to help
bridge between higher-level memory management schemes (GC, refcounting, RAII,
borrow checkers) and arenas.

If there is only one arena, dangling pointers within the arena are impossible,
because all objects are freed at the same time. This is the simplest case. The
user must still be careful not to keep dangling pointers that point at arena
memory after it has been freed, but dangling pointers between the arena
objects will be impossible.

But what if there are multiple arenas? If we have a pointer from one arena to
another, how do we ensure that this will not become a dangling pointer?

To help with the multiple arena case, upb provides a primitive called "fuse".

// Fuses the lifetimes of `a` and `b`.  None of the blocks from `a` or `b`
// will be freed until both arenas are freed.
UPB_API bool upb_Arena_Fuse(const upb_Arena* a, const upb_Arena* b);

When two arenas are fused together, their lifetimes are irreversibly joined,
such that none of the arena blocks in either arena will be freed until both
arenas are freed with upb_Arena_Free(). This means that dangling pointers
between the two arenas will no longer be possible.

Fuse is useful when joining two messages from separate arenas (making one a
sub-message of the other). Fuse is a relatively cheap operation, on the order of
150ns, and is very nearly O(1) in the number of arenas being fused (the true
complexity is the inverse Ackermann function, which grows extremely slowly).

Each arena does consume some memory, so repeatedly creating and fusing an
additional arena is not free, but the CPU cost of fusing two arenas together is
modest.

Initial Block and Custom Allocators Copy Section

upb_Arena normally uses malloc() and free() to allocate and return its
underlying blocks. But this default strategy can be customized to support the
needs of a particular language.

The lowest-level function for creating a upb_Arena is:

// Creates an arena from the given initial block (if any -- n may be 0).
// Additional blocks will be allocated from |alloc|.  If |alloc| is NULL,
// this is a fixed-size arena and cannot grow.
UPB_API upb_Arena* upb_Arena_Init(void* mem, size_t n, upb_alloc* alloc);

The buffer [mem, n] will be used as an "initial block", which is used to
satisfy allocations before calling any underlying allocation function. Note that
the upb_Arena itself will be allocated from the initial block if possible, so
the amount of memory available for allocation from the arena will be less than
n.

The alloc parameter specifies a custom memory allocation function which will
be used once the initial block is exhausted. The user can pass NULL as the
allocation function, in which case the initial block is the only memory
available in the arena. This can allow upb to be used even in situations where
there is no heap.

It follows that upb_Arena_Malloc() is a fallible operation, and all allocating
operations like upb_Message_New() should be checked for failure if there is
any possibility that a fixed size arena is in use.

Schemas Copy Chapter

Nearly all operations in upb require that you have a schema. A protobuf schema
is a data structure that contains all of the message, field, enum, etc.
definitions that are specified in a .proto file. To create, parse, serialize,
or access a message you must have a schema. For this reason, loading a schema is
generally the first thing you must do when you use upb. [^0]

[^0]: This requirement comes from the protobuf wire format itself, which is a
deep insight about the nature of protobuf (or at least the existing wire
format). Unlike JSON, protobuf cannot be parsed or manipulated in a
schema-less way. This is because the binary wire format does not
distinguish between strings and sub-messages, so a generic parser that is
oblivious to the schema is not possible. If a future version of the wire
format did distinguish between these, it could be possible to have a
schema-agnostic data representation, parser, and serializer.

upb has two main data structures that represent a protobuf schema:

• MiniTables are a compact, stripped down version of the schema that
  contains only the information necessary for parsing and serializing the
  binary wire format.
• Reflection contains basically all of the data from a .proto file,
  including the original names of all messages/fields/etc., and all options.

The table below summarizes the main differences between these two:

MiniTables are useful if you only need the binary wire format, because they are
much lighter weight than full reflection.

Reflection is useful if you need to parse JSON or TextFormat, or you need access
to options that were specified in the proto file. Note that reflection also
includes MiniTables, so if you have reflection, you also have MiniTables
available.

MiniTables Copy Section

MiniTables are represented by a set of data structures with names like
upb_MiniTable (representing a message), upb_MiniTableField,
upb_MiniTableFile, etc. Whenever you see one of these types in a function
signature, you know that this particular operation requires a MiniTable. For
example:

// Parses the wire format data in the given buffer `[buf, size]` and writes it
// to the message `msg`, which has the type `mt`.
UPB_API upb_DecodeStatus upb_Decode(const char* buf, size_t size,
                                    upb_Message* msg, const upb_MiniTable* mt,
                                    const upb_ExtensionRegistry* extreg,
                                    int options, upb_Arena* arena);

The subset of upb that requires only MiniTables can be thought of as "upb lite,"
because both the code size and the runtime memory overhead will be less than
"upb full" (the parts that use reflection).

Loading

There are three main ways of loading a MiniTable:

1. From C generated code: The upb code generator can emit
   .upb_minitable.c files that contain the MiniTables as global constant
   variables. When the main program links against these, the MiniTable will be
   placed into .rodata (or .data.rel.ro) in the binary. The MiniTable can
   then be obtained from a generated function. In Blaze/Bazel these files can
   be generated and linked using the upb_minitable_proto_library() rule.
2. From MiniDescriptors: The user can build MiniDescriptors into MiniTables
   at runtime. MiniDescriptors are a compact upb-specific wire format designed
   specially for this purpose. The user can call upb_MiniTable_Build() at
   runtime to convert MiniDescriptors to MiniTables.
3. From reflection: If you have already built reflection data structures
   for your type, then you can obtain the upb_MiniTable corresponding to a
   upb_MessageDef using upb_MessageDef_MiniTable().

For languages that are already using reflection, (3) is an obvious choice.

For languages that are avoiding reflection, here is a general guideline for
choosing between (1) and (2): if the language being wrapped participates in the
standard binary linking model on a given platform (in particular, if it is
generally linked using ld), then it is better to use (1), which is also known
as "static loading".

Static loading of MiniTables has the benefit of requiring no runtime
initialization[^2], leading to faster startup. Static loading of MiniTables also
facilitates cross-language sharing of proto messages, because sharing generally
requires that both languages are using exactly the same MiniTables.

The main downside of static loading is that it requires the user to generate one
.upb.c file per .proto and link against the transitive closure of .upb.c
files. Blaze and Bazel make this reasonably easy, but for other build systems it
can be more of a challenge.

[^2]: aside from possible pointer relocations performed by the ELF/Mach-O loader
if the library or binary is position-independent

Loading from MiniDescriptors, as in option (2), has the advantage that it does
not require per-message linking of C code. For many language toolchains,
generating and linking some custom C code for every protobuf file or message
type would be a burdensome requirement. MiniDescriptors are a convenient way of
loading MiniTables without needing to cross the FFI boundary outside the core
runtime.

A common pattern when using dynamic loading is to embed strings containing
MiniDescriptors directly into generated code. For example, the generated code in
Dart for a message with only primitive fields currently looks something like:

const desc = r'$(+),*-#$%&! /10';
  _accessor = $pb.instance.registry.newMessageAccessor(desc);

The implementation of newMessageAccessor() is mainly just a wrapper around
upb_MiniTable_Build(), which builds a MiniTable from a MiniDescriptor. In the
code generator, the MiniDescriptor can be obtained from the
upb_MessageDef_MiniDescriptorEncode() API; users should never need to encode a
MiniDescriptor manually.

Linking

When building MiniTables dynamically, it is the user's responsibility to link
each message to the to the appropriate sub-messages and or enums. Each message
must have its message and closed enum fields linked using
upb_MiniTable_SetSubMessage() and upb_MiniTable_SetSubEnum(), respectively.

A higher-level function that links all fields at the same time is also
available, as upb_MiniTable_Link(). This function pairs well with
upb_MiniTable_GetSubList() which can be used in a code generator to get a list
of all the messages and enums which must be passed to upb_MiniTable_Link().

A common pattern is to embed the link() calls directly into the generated
code. For example, here is an example from Dart of building a MiniTable that
contains sub-messages and enums:

const desc = r'$3334';
  _accessor = $pb.instance.registry.newMessageAccessor(desc);
  _accessor!.link(
      [
        M2.$_accessor,
        M3.$_accessor,
        M4.$_accessor,
      ],
      [
        E.$_accessor,
      ],
  );

In this case, upb_MiniTable_GetSubList() was used in the code generator to
discover the 3 sub-message fields and 1 sub-enum field that require linking. At
runtime, these lists of MiniTables are passed to the link() function, which
will internally call upb_MiniTable_Link().

Note that in some cases, the application may choose to delay or even skip the
registration of sub-message types, as part of a tree shaking strategy.

When using static MiniTables, a manual link step is not necessary, as linking is
performed automatically by ld.

Enums

MiniTables primarily carry data about messages, fields, and extensions. However
for closed enums, we must also have a upb_MiniTableEnum structure that stores
the set of all numbers that are defined in the enum. This is because closed
enums have the unfortunate behavior of putting unknown enum values into the
unknown field set.

Over time, closed enums will hopefully be phased out via editions, and the
relevance and overhead of upb_MiniTableEnum will shrink and eventually
disappear.

Reflection Copy Section

Reflection uses types like upb_MessageDef and upb_FieldDef to represent the
full contents of a .proto file at runtime. These types are upb's direct
equivalents of google::protobuf::Descriptor, google::protobuf::FieldDescriptor, etc. [^1]

[^1]: upb consistently uses Def where C++ would use Descriptor in type
names. This introduces divergence with C++; the rationale was to conserve
horizontal line length, as Def is less than 1/3 the length of
Descriptor. This is more relevant in C, where the type name is repeated
in every function, eg. upb_FieldDef_Name() vs.
upb_FieldDescriptor_Name().

Whenever you see one of these types in a function signature, you know that the
given operation requires reflection. For example:

// Parses JSON format into a message object, using reflection.
UPB_API bool upb_JsonDecode(const char* buf, size_t size, upb_Message* msg,
                            const upb_MessageDef* m, const upb_DefPool* symtab,
                            int options, upb_Arena* arena, upb_Status* status);

The part of upb that requires reflection can be thought of as "upb full." These
parts of the library cannot be used if a given application has only loaded
MiniTables. There is no way to convert a MiniTable into reflection.

The upb_DefPool type is the top-level container that builds and owns some set
of defs. This type is a close analogue of google::protobuf::DescriptorPool in C++. The
user must always ensure that the upb_DefPool outlives any def objects that it
owns.

Loading

As with MiniTable loading, we have multiple options for how to load full
reflection:

1. From C generated code: The upb code generator can create foo.upbdefs.c
   files that embed the descriptors and exports generated C functions for
   adding them to a user-provided upb_DefPool.
2. From descriptors: The user can make manual calls to
   upb_DefPool_AddFile(), using descriptors obtained at runtime. Defs for
   individual messages can then be obtained using
   upb_DefPool_FindMessageByName().

Unlike MiniTables, loading from generated code requires runtime initialization,
as reflection data structures like upb_MessageDef are not capable of being
emitted directly into .rodata like upb_MiniTable is. Instead, the generated
code embeds serialized descriptor protos into .rodata which are then built
into heap objects at runtime.

From this you might conclude that option (1) is nothing but a convenience
wrapper around option (2), but that is not quite correct either. Option (1)
does link against the static .upb.c structures for the MiniTables, whereas
option (2) will build the MiniTables from scratch on the heap. So option (1)
will use marginally less CPU and RAM when the descriptors are loaded into a
upb_DefPool. More importantly, the resulting descriptors will be capable of
reflecting over any messages built from the generated .upb.c MiniTables,
whereas descriptors built using option (2) will have distinct MiniTables that
cannot reflect over messages that use the generated MiniTables.

A common pattern for dynamic languages like PHP, Ruby, or Python, is to use
option (2) with descriptors that are embedded into the generated code. For
example, the generated code in Python currently looks something like:

from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf.internal import builder as _builder

_desc = b'\n\x1aprotoc_explorer/main.proto\x12\x03pkg'

DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(_desc)
_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'google3.protoc_explorer.main_pb2', _globals)

The AddSerializedFile() API above is mainly just a thin wrapper around
upb_DefPool_AddFile().

Upb Style Guide (`docs/upb/style-guide.md`) Copy Section

C style guide

Since upb is written in pure C, we supplement the
Google C++ style guide with
some C-specific guidance.

Everything written here is intended to follow the spirit of the C++ style guide.

upb is currently inconsistent about following these conventions. It is intended
that all code will be updated to match these guidelines. The priority is
converting public interfaces as these are more difficult to change later.

Naming Copy Chapter

Functions and Types Copy Section

C does not have namespaces. Anywhere you would normally use a namespace
separator (::) in C++, we use an underscore (_) in C:

// C equivalent for upb::Arena::New()
upb_Arena* upb_Arena_New();

Since we rely on _ to be our namespace separator, we never use it to merely
separate words in function or type names:

// BAD: this would be interpreted as upb::FieldDef::has::default().
bool upb_FieldDef_has_default(const upb_FieldDef* f);

// GOOD: this is equivalent to upb::FieldDef::HasDefault().
bool upb_FieldDef_HasDefault(const upb_FieldDef* f);

For multi-word namespaces, we use PascalCase:

// `PyUpb` is the namespace.
PyObject* PyUpb_CMessage_GetAttr(PyObject* _self, PyObject* attr);

Private Functions and Members Copy Section

Since we do not have private in C, we use the UPB_PRIVATE() macro to mark
internal functions and variables that should only be accessed from upb:

// Internal-only function.
int64_t UPB_PRIVATE(upb_Int64_FromLL)();

// Internal-only members. Underscore prefixes are only necessary when the
// structure is defined in a header file.
typedef struct {
  const int32_t* UPB_PRIVATE(values);
  uint64_t UPB_PRIVATE(mask);
  int UPB_PRIVATE(value_count);
} upb_MiniTableEnum;

// Using these members in an internal function.
int upb_SomeFunction(const upb_MiniTableEnum* e) {
  return e->UPB_PRIVATE(value_count);
}

This prevents users from accessing these symbols, because UPB_PRIVATE()
is defined in port_def.inc and undefined in port_undef.inc, and these
textual headers are not accessible by users.

It is only necessary to use UPB_PRIVATE() for things defined in headers.
For symbols in .c files, it is sufficient to mark private functions with
static.

Upb Vs Cpp Protos (`docs/upb/vs-cpp-protos.md`) Copy Section

upb vs. C++ Protobuf Design

upb is a small C
protobuf library. While some of the design follows in the footsteps of the C++
Protobuf Library, upb departs from C++'s design in several key ways. This
document compares and contrasts the two libraries on several design points.

Design Goals Copy Chapter

Before we begin, it is worth calling out that upb and C++ have different design
goals, and this motivates some of the differences we will see.