mysql-server Documentation

Technical Documentation: mysql/mysql-server

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Extra Opentelemetry Proto Opentelemetry Proto 1.10.0 Docs README (`extra/opentelemetry-proto/opentelemetry-proto-1.10.0/docs/README.md`) Copy Section

OpenTelemetry Protocol (OTLP)

This is the specification of the OpenTelemetry Protocol (OTLP).

• Design Goals
• Requirements
• Specification
• SDK Exporter

Extra Googletest Googletest 1.17.0 Googlemock Docs README (`extra/googletest/googletest-1.17.0/googlemock/docs/README.md`) Copy Section

Content Moved

We are working on updates to the GoogleTest documentation, which has moved to
the top-level docs directory.

Extra Googletest Googletest 1.17.0 Googletest Docs README (`extra/googletest/googletest-1.17.0/googletest/docs/README.md`) Copy Section

Content Moved

We are working on updates to the GoogleTest documentation, which has moved to
the top-level docs directory.

Docs Sp Imp Spec (`Docs/sp-imp-spec.txt`) Copy Section

Copyright (c) 2003, 2026, Oracle and/or its affiliates.

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License, version 2.0,
as published by the Free Software Foundation.

This program is designed to work with certain software (including
but not limited to OpenSSL) that is licensed under separate terms,
as designated in a particular file or component or in included license
documentation. The authors of MySQL hereby grant you an additional
permission to link the program and your derivative works with the
separately licensed software that they have either included with
the program or referenced in the documentation.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License, version 2.0, for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

Implementation specification for Stored Procedures
          ==================================================

• How parsing and execution of queries work

  In order to execute a query, the function sql_parse.cc:dispatch_sql_command()
  is called, which in turn calls the parser (yyparse()) with an updated Lex
  structure as the result. dispatch_sql_command() then calls
  mysql_execute_command() which dispatches on the command code (in Lex) to the
  corresponding code for executing that particular query.

  There are three structures involved in the execution of a query which are of
  interest to the stored procedure implementation:

  • Lex (mentioned above) is the "compiled" query, that is the output from
    the parser and what is then interpreted to do the actual work.
    It constains an enum value (sql_command) which is the query type, and
    all the data collected by the parser needed for the execution (table
    names, fields, values, etc).
  • THD is the "run-time" state of a connection, containing all that is
    needed for a particular client connection, and, among other things, the
    Lex structure currently being executed.
  • Item_*: During parsing, all data is translated into "items", objects of
    the subclasses of "Item", such as Item_int, Item_real, Item_string, etc,
    for basic datatypes, and also various more specialized Item types for
    expressions to be evaluated (Item_func objects).

• How to fit Stored Procedure into this scheme

  • An overview of the classes and files for stored procedures
    (More detailed APIs at the end of this file)

    • class sp_head (sp_head.{cc,h})
      This contains, among other things, an array of "instructions" and the
      method for executing the procedure.

    • class sp_pcontext (sp_pcontext.{cc,h}
      This is the parse context for the procedure. It's primarily used during
      parsing to keep track of local parameters, variables and labels, but
      it's also used at CALL time do find parameters mode (IN, OUT or INOUT)
      and type when setting up the runtime context.

    • class sp_instr (sp_head.{cc,h})
      This is the base class for "instructions", that is, what is generated
      by the parser. It turns out that we only need a minimum of 5 different
      sub classes:

      • sp_instr_stmt
        Execute a statement. This is the "call-out" any normal SQL statement,
        like a SELECT, INSERT etc. It contains the Lex structure for the
        statement in question.
      • sp_instr_set
        Set the value of a local variable (or parameter)
      • sp_instr_jump
        An unconditional jump.
      • sp_instr_jump_if_not
        Jump if condition is not true. It turns out that the negative test is
        most convenient when generating the code for the flow control
        constructs.
      • sp_instr_freturn
        Return a value from a FUNCTION and exit.
        For condition HANDLERs some special instructions are also needed, see
        that section below.

    • class sp_rcontext (sp_rcontext.h)
      This is the runtime context in the THD structure.
      It contains an array of items, the parameters and local variables for
      the currently executing stored procedure.
      This means that variable value lookup is in runtime is constant time,
      a simple index operation.

    • class Item_splocal (Item.{cc,h})
      This is a subclass of Item. Its sole purpose is to hide the fact that
      the real Item is actually in the current frame (runtime context).
      It contains the frame offset and defers all methods to the real Item
      in the frame. This is what the parser generates for local variables.

    • Utility functions (sp.{cc,h})
      This contains functions for creating, dropping and finding a stored
      procedure in the mysql.proc table (or the internal cache).

  • Parsing CREATE PROCEDURE ...

    When parsing a CREATE PROCEDURE the parser first initializes the
    sphead and spcont (runtime context) fields in the Lex.
    The sql_command code for the result of parsing a is
    SQLCOM_CREATE_PROCEDURE.

    The parsing of the parameter list and body is relatively
    straight-forward:

    • Parameters:
      name, type and mode (IN/OUT/INOUT) is pushed to spcont

    • Declared local variables:
      Same as parameters (mode is then IN)

    • Local Variable references:
      If an identifier is found in in spcont, an Item_splocal is created
      with the variable's frame index, otherwise an Item_field or Item_ref
      is created (as before).

    • Statements:
      The Lex in THD is replaced by a new Lex structure and the statement,
      is parsed as usual. A sp_instr_stmt is created, containing the new
      Lex, and added to added to the instructions in sphead.
      Afterwards, the procedure's Lex is restored in THD.

    • SET var:
      Setting a local variable generates a sp_instr_set instruction,
      containing the variable's frame offset, the expression (an Item),
      and the type.

    • Flow control:
      Flow control constructs like, IF, WHILE, etc, generate a conditional
      and unconditional jumps in the "obvious" way, but a few notes may
      be required:

      • Forward jumps: When jumping forward, the exact destination is not
        known at the time of the creation of the jump instruction. The
        sphead therefore contains list of instruction-label pairs for
        each forward reference. When the position later is known, the
        instructions in the list are updated with the correct location.
      • Loop constructs have optional labels. If a loop doesn't have a
        label, an anonymous label is generated to simplify the parsing.
      • There are two types of CASE. The "simple" case is implemented
        with an anonymous variable bound to the value to be tested.

    • A simple example

      Parsing the procedure:

      create procedure a(s char(16))
      begin
      declare x int;
      set x = 3;
      while x > 0 do
      set x = x-1;
      insert into db.tab values (x, s);
      
      end while;
      end

      would generate the following structures:
      ______
      
      thd: | | _________
      | lex -+--->| | ___________________
      || | spcont -+------------------->| "s",in,char(16):0 |
      | sphead -+------ |("x",in,int :1)|
      || | |_|
      V
      | m_name: "a" |
      | m_instr: ... |
      |______________________|

      Note that the contents of the spcont is changing during the parsing,
      at all times reflecting the state of the would-be runtime frame.
      The m_instr is an array of instructions:

      Pos. Instruction
      0 sp_instr_set(1, '3')
      1 sp_instr_jump_if_not(5, 'x>0')
      2 sp_instr_set(1, 'x-1')
      3 sp_instr_stmt('insert into ...')
      4 sp_instr_jump(1)
      5

    Here, '3', 'x>0', etc, represent the Items or Lex for the respective
    expressions or statements.

  • Parsing CREATE FUNCTION ...

    Creating a functions is essensially the same thing as for a PROCEDURE,
    with the addition that a FUNCTION has a return type and a RETURN
    statement, but no OUT or INOUT parameters.

    The main difference during parsing is that we store the result type
    in the sp_head. However, there are big differences when it comes to
    invoking a FUNCTION. (See below.)

  • Storing, caching, dropping...

    As seen above, the entired definition string, including the "CREATE
    PROCEDURE" (or "FUNCTION") is kept. The procedure definition string is
    stored in the table mysql.proc with the name and type as the key, the
    type being one of the enum ("procedure","function").

    A PROCEDURE is just stored in the mysql.proc table. A FUNCTION has an
    additional requirement. They will be called in expressions with the same
    syntax as UDFs, so UDFs and stored FUNCTIONs share the namespace. Thus,
    we must make sure that we do not have UDFs and FUNCTIONs with the same
    name (even if they are storded in different places).

    This means that we can reparse the procedure as many time as we want.
    The first time, the resulting Lex is used to store the procedure in
    the database (using the function sp.c:sp_create_procedure()).

    The simplest way would be to just leave it at that, and re-read the
    procedure from the database each time it is called. (And in fact, that's
    the way the earliest implementation will work.)
    However, this is not very efficient, and we can do better. The full
    implementation should work like this:

    1. Upon creation time, parse and store the procedure. Note that we still
       need to parse it to catch syntax errors, but we can't check if called
       procedures exists for instance.
    2. Upon first CALL, read from the database, parse it, and cache the
       resulting Lex in memory. This time we can do more error checking.
    3. Upon subsequent CALLs, use the cached Lex.

    Note that this implies that the Lex structure with its sphead must be
    reentrant, that is, reusable and shareable between different threads
    and calls. The runtime state for a procedure is kept in the sp_rcontext
    in THD.

    The mechanisms of storing, finding, and dropping procedures are
    encapsulated in the files sp.{cc,h}.

  • CALLing a procedure

    A CALL is parsed just like any statement. The resulting Lex has the
    sql_command SQLCOM_CALL, the procedure's name and the parameters are
    pushed to the Lex' value_list.

    sql_parse.cc:mysql_execute_command() then uses sp.cc:sp_find() to
    get the sp_head for the procedure (which may have been read from the
    database or feetched from the in-memory cache) and calls the sp_head's
    method execute().
    Note: It's important that substatements called by the procedure do not
    do send_ok(). Fortunately, there is a flag in THD->net to disable
    this during CALLs. If a substatement fails, it will however send
    an error back to the client, so the CALL mechanism must return
    immediately and without sending an error.

    The sp_head::execute() method works as follows:

    1. Keep a pointer to the old runtime context in THD (if any)
    2. Create a new runtime context. The information about the required size
       is in sp_head's parse time context.
    3. Push each parameter (from the CALL's Lex->value_list) to the new
       context. If it's an OUT or INOUT parameter, the parameter's offset
       in the caller's frame is set in the new context as well.
    4. For each instruction, call its execute() method.
       The result is a pointer to the next instruction to execute (or NULL)
       if an error occurred.
    5. On success, set the new values of the OUT and INOUT parameters in
       the caller's frame.

    • USE database

      Before executing the instruction we also keeps the current default
      database (if any). If this was changed during execution (i.e. a "USE"
      statement has been executed), we restore the current database to the
      original.

      This is the most useful way to handle USE in procedures. If we didn't,
      the caller would find himself in a different database after calling
      a function, which can be confusing.
      Restoring the database also gives full freedom to the procedure writer:

      • It's possible to write "general" procedures that are independent of
        the actual database name.
      • It's possible to write procedures that work on a particular database
        by calling USE, without having to use fully qualified table names
        everywhere (which doesn't help if you want to call other, "general",
        procedures anyway).

    • Evaluating Items

      There are three occasions where we need to evaluate an expression:

      • When SETing a variable
      • When CALLing a procedure
      • When testing an expression for a branch (in IF, WHILE, etc)

      The semantics in stored procedures is "call-by-value", so we have to
      evaluate any "func" Items at the point of the CALL or SET, otherwise
      we would get a kind of "lazy" evaluation with unexpected results with
      respect to OUT parameters for instance.
      For this the support function, sp_head.cc:eval_func_item() is needed.

  • Calling a FUNCTION

    Functions don't have an explicit call keyword like procedures. Instead,
    they appear in expressions with the conventional syntax "fun(arg, ...)".
    The problem is that we already have User Defined Functions (UDFs) which
    are called the same way. A UDF is detected by the lexical analyzer (not
    the parser!), in the find_keyword() function, and returns a UDF_*FUNC
    or UDA*_SUM token with the udf_func object as the yylval.

    So, stored functions must be handled in a simpilar way, and as a
    consequence, UDFs and functions must not have the same name.

    • Detecting and parsing a FUNCTION invocation

      The existance of UDFs are checked during the lexical analysis (in
      sql_lex.cc:find_keyword()). This has the drawback that they must
      exist before they are refered to, which was ok before SPs existed,
      but then it becomes a problem. The first implementation of SP FUNCTIONs
      will work the same way, but this should be fixed a.s.a.p. (This will
      required some reworking of the way UDFs are handled, which is why it's
      not done from the start.)
      For the time being, a FUNCTION is detected the same way, and returns
      the token SP_FUNC. During the parsing we only check for the existance
      of the function, we don't parse it, since wa can't call the parser
      recursively.

      When encountering a SP_FUNC with parameters in the expression parser,
      an instance of the new Item_func_sp class is created. Unlike UDFs, we
      don't have different classes for different return types, since we at
      this point don't know the type.

    • Collecting FUNCTIONs to invoke

      A FUNCTION differs from a PROCEDURE in one important aspect: Whereas a
      PROCEDURE is CALLed as statement by itself, a FUNCTION is invoked
      "on-the-fly" during the execution of another statement.
      This makes things a lot more complicated compared to CALL:

      • We can't read and parse the FUNCTION from the mysql.proc table at the
        point of invocation; the server requires that all tables used are
        opened and locked at the beginning of the query execution.
        One "obvious" solution would be to simply push "mysql.proc" to the list
        of tables used by the query, but this implies a "join" with this table
        if the query is a select, so it doesn't work (and we can't exclude this
        table easily; since a priviledged used might in fact want to search
        the proc table).
        Another solution would of course be to allow the opening and closing
        of the mysql.proc table during a query execution, but this it not
        possible at the present.

      So, the solution is to collect the names of the refered FUNCTIONs during
      parsing in the lex.
      Then, before doing anything else in mysql_execute_command(), read all
      functions from the database an keep them in the THD, where the function
      sp_find_function() can find them during the execution.
      Note: Even with an in-memory cache, we must still make sure that the
      functions are indeed read and cached at this point.
      The code that read and cache functions from the database must also be
      invoked recursively for each read FUNCTION to make sure we have all the
      functions we need.

  • Parsing DROP PROCEDURE/FUNCTION

    The procedure name is pushed to Lex->value_list.
    The sql_command code for the result of parsing a is
    SQLCOM_DROP_PROCEDURE/SQLCOM_DROP_FUNCTION.

    Dropping is done by simply getting the procedure with the sp_find()
    function and calling sp_drop() (both in sp.{cc,h}).

    DROP PROCEDURE/FUNCTION also supports the non-standard "IF EXISTS",
    analogous to other DROP statements in MySQL.

  • Condition and Handlers

    Condition names are lexical entities and are kept in the parser context
    just like variables. But, condition are just "aliases" for SQLSTATE
    strings, or mysqld error codes (which is a non-standard extension in
    MySQL), and are only used during parsing.

    Handlers comes in three types, CONTINUE, EXIT and UNDO. The latter is
    like an EXIT handler with an implicit rollback, and is currently not
    implemented.
    The EXIT handler jumps to the end of its BEGIN-END block when finished.
    The CONTINUE handler returns to the statement following that which
    invoked the handler.

    The handlers in effect at any point is part of each thread's runtime
    state, so we need to push and pop handlers in the sp_rcontext during
    execution. We use special instructions for this:

    • sp_instr_hpush_jump
      Push a handler. The instruction contains the necessary information,
      like which conditions we handle and the location of the handler.
      The jump takes us to the location after the handler code.
    • sp_instr_hpop
      Pop the handlers of the current frame (which we are just leaving).

    It might seems strange to jump past the handlers like that, but there's
    no extra cost in doing this, and for technical reasons it's easiest for
    the parser to generate the handler instructions when they occur in the
    source.

    When an error occurs, one of the error routines is called and an error
    message is normally sent back to the client immediately.
    Catching a condition must be done in these error routines (there are
    quite a few) to prevent them from doing this. We do this by calling
    a method in the THD's sp_rcontext (if there is one). If a handler is
    found, this is recorded in the context and the routine returns without
    sending the error message.
    The exectution loop (sp_head::execute()) checks for this after each
    statement and invokes the handler that has been found. If several
    errors or warnings occurs during one statement, only the first is
    caught, the rest are ignored.

    Invoking and returning from a handler is trivial in the EXIT case.
    We simply jump to it, and it will have an sp_instr_jump as its last
    instruction.

    Calling and returning from a CONTINUE handler poses some special
    problems. Since we need to return to the point after its invokation,
    we push the return location on a stack in the sp_rcontext (this is
    done by the exectution loop). The handler then ends with a special
    instruction, sp_instr_hreturn, which returns to this location.

    CONTINUE handlers have one additional problem: They are parsed at
    the lexical level where they occur, so variable offsets will assume
    that it's actually called at that level. However, a handler might be
    invoked from a sub-block where additional local variables have been
    declared, which will then share the location of any local variables
    in the handler itself. So, when calling a CONTINUE handler, we need
    to save any local variables above the handler's frame offset, and
    restore them upon return. (This is not a problem for EXIT handlers,
    since they will leave the block anyway.)
    This is taken care of by the execution loop and the sp_instr_hreturn
    instruction.

    • Examples:

      • EXIT handler
        begin
        declare x int default 0;

        begin
        declare exit handler for 'XXXXX' set x = 1;

        (statement1);
        (statement2);
        end;
        (statement3);
        end

      Pos. Instruction
      0 sp_instr_set(0, '0')
      1 sp_instr_hpush_jump(4, 1) # location and frame size
      2 sp_instr_set(0, '1')
      3 sp_instr_jump(6)
      4 sp_instr_stmt('statement1')
      5 sp_instr_stmt('statement2')
      6 sp_instr_hpop(1)
      7 sp_instr_stmt('statement3')

      • CONTINUE handler
        create procedure hndlr1(val int)
        begin
        declare x int default 0;
        declare foo condition for 1146;
        declare continue handler for foo set x = 1;

        insert into t3 values ("hndlr1", val); # Non-existing table?
        if x>0 then
        insert into t1 values ("hndlr1", val); # This instead then
        end if;
        end|

      Pos. Instruction
      0 sp_instr_set(1, '0')
      1 sp_instr_hpush_jump(4, 2)
      2 sp_instr_set(1, '1')
      3 sp_instr_hreturn(2) # frame size
      4 sp_instr_stmt('insert ... t3 ...')
      5 sp_instr_jump_if_not(7, 'x>0')
      6 sp_instr_stmt('insert ... t1 ...')
      7 sp_instr_hpop(2)

  • Cursors

    For stored procedures to be really useful, you want to have cursors.
    MySQL doesn't yet have "real" cursor support (with API and ODBC support,
    allowing updating, arbitrary scrolling, etc), but a simple asensitive,
    non-scrolling, read-only cursor can be implemented in SPs using the
    class Protocol_cursor.
    This class intecepts the creation and sending of results sets and instead
    stores it in-memory, as MYSQL_FIELDS and MYSQL_ROWS (as in the client API).

    To support this, we need the usual name binding support in sp_pcontext
    (similar to variables and conditions) to keep track on declared cursor
    names, and a corresponding run-time mechanism in sp_rcontext.
    Cursors are lexically scoped like everything with a body or BEGIN/END
    block, so they are pushed and poped as usual (see conditions and variables
    above).
    The basic operations on a cursor are OPEN, FETCH and CLOSE, which will
    each have a corresponding instruction. In addition, we need instructions
    to push a new cursor (this will encapsulate the LEX of the SELECT statement
    of the cursor), and a pop instruction:

    • sp_instr_cpush
      Push a cursor to the sp_rcontext. This instruction contains the LEX
      for the select statement
    • sp_instr_cpop
      Pop a number of cursors from the sp_rcontext.
    • sp_instr_copen
      Open a cursor: This will execute the select and get the result set
      in a sepeate memroot.
    • sp_instr_cfetch
      Fetch the next row from the in-memory result set. The instruction
      contains a list of the variables (frame offsets) to set.
    • sp_instr_cclose
      Free the result set.

    A cursor is a separate class, sp_cursor (defined in sp_rcontex.h) which
    encapsulates the basic operations used by the above instructions.
    This class contains the LEX, Protocol_cursor object, and its memroot,
    as well as the cursor's current state.
    Compiling and executing is fairly straight-forward. sp_instr_copen is
    a subclass of sp_instr_stmt and uses its mechanism to execute a
    substatement.

    • Example:

      begin
      declare x int;
      declare c cursor for select a from t1;

      open c;
      fetch c into x;
      close c;
      end

      Pos. Instruction
      0 sp_instr_cpush('select a from ...')
      1 sp_instr_copen(0) # The 0'th cursor
      2 sp_instr_cfetch(0) # Contains the variable list
      3 sp_instr_cclose(0)
      4 sp_instr_cpop(1)

  • The SP cache

    There are two ways to cache SPs:

    1. one global cache, share by all threads/connections,
    2. one cache per thread.

    There are pros and cons with both methods:

    1. Pros: Save memory, each SP only read from table once,
       Cons: Needs locking (= serialization at access), requires thread-safe
       data structures,
    2. Pros: Fast, no locking required (almost), limited thread-safe
       requirement,
       
       Cons: Uses more memory, each SP read from table once per thread.

    Unfortunately, we cannot use alternative 1 for the time being, as most
    of the datastructures to be cached (lex and items) are not reentrant
    and thread-safe. (Things are modifed at execution, we have THD pointers
    stored everywhere, etc.)
    This leaves us with alternative 2, one cache per thread; or actually
    two, since we keep FUNCTIONs and PROCEDUREs in separate caches.
    This is not that terrible; the only case when it will perform
    significantly worse than a global cache is when we have an application
    where new threads are connecting, calling a procedure, and disconnecting,
    over and over again.

    The cache implementation itself is simple and straightforward, a hashtable
    wrapped in a class and a C API (see APIs below).

    There is however one issue with multiple caches: dropping and altering
    procedures. Normally, this should be a very rare event in a running
    system; it's typically something you do during development and testing,
    so it's not unthinkable that we would simply ignore the issue and let
    any threads running with a cached version of an SP keep doing so until
    its disconnected.
    But assuming we want to keep the caches consistent with respect to drop
    and alter, it can be done:

    1. A global counter is needed, initialized to 0 at start.
    2. At each DROP or ALTER, increase the counter by one.
    3. Each cache has its own copy of the counter, copied at the last read.
    4. When looking up a name in the cache, first check if the global counter
       is larger than the local copy.
       If so, clear the cache and return "not found", and update the local
       counter; otherwise, lookup as usual.

    This minimizes the cost to a single brief lock for the access of an
    integer when operating normally. Only in the event of an actual drop or
    alter, is the cache cleared. This may seem to be drastic, but since we
    assume that this is a rare event, it's not a problem.
    It would of course be possible to have a much more fine-grained solution,
    keeping track of each SP, but the overhead of doing so is not worth the
    effort.

  • Class and function APIs
    This is an outline of the key types. Some types and other details
    in the actual files have been omitted for readability.

    • The parser context: sp_pcontext.h

      typedef enum
      {
      sp_param_in,
      sp_param_out,
      sp_param_inout
      } sp_param_mode_t;

      typedef struct
      {
      LEX_STRING name;
      enum enum_field_types type;
      sp_param_mode_t mode;
      uint offset; // Offset in current frame
      bool isset;
      } sp_pvar_t;

      typedef struct sp_cond_type
      {
      enum { number, state, warning, notfound, exception } type;
      char sqlstate[6];
      uint mysqlerr;
      } sp_cond_type_t;

      class sp_pcontext
      {
      sp_pcontext();

      // Return the maximum frame size
      uint max_framesize();

      // Return the current frame size
      uint current_framesize();

      // Return the number of parameters
      uint params();

      // Set the number of parameters to the current frame size
      void set_params();

      // Set type of the variable at offset 'i' in the frame
      void set_type(uint i, enum enum_field_types type);

      // Mark the i:th variable to "set" (i.e. having a value) with
      // 'val' true.
      void set_isset(uint i, bool val);

      // Push the variable 'name' to the frame.
      void push_var(LEX_STRING *name,
      enum enum_field_types type, sp_param_mode_t mode);

      // Pop 'num' variables from the frame.
      void pop_var(uint num = 1);

      // Find variable by name
      sp_pvar_t *find_pvar(LEX_STRING *name);

      // Find variable by index
      sp_pvar_t *find_pvar(uint i);

      // Push label 'name' of instruction index 'ip' to the label context
      sp_label_t *push_label(char *name, uint ip);

      // Find label 'name' in the context
      sp_label_t *find_label(char *name);

      // Return the last pushed label
      sp_label_t *last_label();

      // Return and remove the last pushed label.
      sp_label_t *pop_label();

      // Push a condition to the context
      void push_cond(LEX_STRING *name, sp_cond_type_t *val);

      // Pop a 'num' condition from the context
      void pop_cond(uint num);

      // Find a condition in the context
      sp_cond_type_t *find_cond(LEX_STRING *name);

      // Increase the handler count
      void add_handler();

      // Returns the handler count
      uint handlers();

    // Push a cursor
    void push_cursor(LEX_STRING *name);

    // Find a cursor
    bool find_cursor(LEX_STRING *name, uint *poff);

    // Pop 'num' cursors
    void pop_cursor(uint num);

    // Return the number of cursors
    uint cursors();
    }

    • The run-time context (call frame): sp_rcontext.h

    #define SP_HANDLER_NONE 0
    #define SP_HANDLER_EXIT 1
    #define SP_HANDLER_CONTINUE 2
    #define SP_HANDLER_UNDO 3

    typedef struct
    {
    struct sp_cond_type *cond;
    uint handler; // Location of handler
    int type;
    uint foffset; // Frame offset for the handlers declare level
    } sp_handler_t;

    class sp_rcontext
    {
    // 'fsize' is the max size of the context, 'hmax' the number of handlers,
    // 'cmax' the number of cursors
    sp_rcontext(uint fsize, uint hmax, , uint cmax);

    // Push value (parameter) 'i' to the frame
    void push_item(Item *i);

    // Set slot 'idx' to value 'i'
    void set_item(uint idx, Item *i);

    // Return the item in slot 'idx'
    Item *get_item(uint idx);

    // Set the "out" index 'oidx' for slot 'idx. If it's an IN slot,
    // use 'oidx' -1.
    void set_oindex(uint idx, int oidx);

    // Return the "out" index for slot 'idx'
    int get_oindex(uint idx);

    // Set the FUNCTION result
    void set_result(Item *i);

    // Get the FUNCTION result
    Item *get_result();

    // Push handler at location 'h' for condition 'cond'. 'f' is the
    // current variable frame size.
    void push_handler(sp_cond_type_t *cond, uint h, int type, uint f);

    // Pop 'count' handlers
    void pop_handlers(uint count);

    // Find a handler for this error. This sets the state for a found
    // handler in the context. If called repeatedly without clearing,
    // only the first call's state is kept.
    int find_handler(uint sql_errno);

    // Returns 1 if a handler has been found, with '*ip' and '*fp' set
    // to the handler location and frame size respectively.
    int found_handler(uint *ip, uint *fp);

    // Clear the found handler state.
    void clear_handler();

    // Push a return address for a CONTINUE handler
    void push_hstack(uint ip);

    // Pop the CONTINUE handler return stack
    uint pop_hstack();

    // Save variables from frame index 'fp' and up.
    void save_variables(uint fp);

    // Restore saved variables from to frame index 'fp' and up.
    void restore_variables(uint fp);

    // Push a cursor for the statement (lex)
    void push_cursor(LEX *lex);

    // Pop 'count' cursors
    void pop_cursors(uint count);

    // Pop all cursors
    void pop_all_cursors();

    // Get the 'i'th cursor
    sp_cursor *get_cursor(uint i);

    }

    • The procedure: sp_head.h

      #define TYPE_ENUM_FUNCTION 1
      #define TYPE_ENUM_PROCEDURE 2

      class sp_head
      {
      int m_type; // TYPE_ENUM_FUNCTION or TYPE_ENUM_PROCEDURE

      sp_head();

      void init(LEX_STRING *name, LEX *lex, LEX_STRING *comment, char suid);

      // Store this procedure in the database. This is a wrapper around
      // the function sp_create_procedure().
      int create(THD *);

      // Invoke a FUNCTION
      int
      execute_function(THD *thd, Item **args, uint argcount, Item **resp);

      // CALL a PROCEDURE
      int
      execute_procedure(THD *thd, List *args);

      // Add the instruction to this procedure.
      void add_instr(sp_instr *);

      // Returns the number of instructions.
      uint instructions();

      // Returns the last instruction
      sp_instr *last_instruction();

      // Resets lex in 'thd' and keeps a copy of the old one.
      void reset_lex(THD *);

      // Restores lex in 'thd' from our copy, but keeps some status from the
      // one in 'thd', like ptr, tables, fields, etc.
      void restore_lex(THD *);

      // Put the instruction on the backpatch list, associated with
      // the label.
      void push_backpatch(sp_instr *, struct sp_label *);

      // Update all instruction with this label in the backpatch list to
      // the current position.
      void backpatch(struct sp_label *);

      // Returns the SP name (with optional length in '*lenp').
      char *name(uint *lenp = 0);

      // Returns the result type for a function
      Item_result result();

      // Sets various attributes
      void sp_set_info(char *creator, uint creatorlen,
      longlong created, longlong modified,
      bool suid, char *comment, uint commentlen);
      
      }

    • Instructions

      • The base class:
        class sp_instr
        {
        // 'ip' is the index of this instruction
        sp_instr(uint ip);

        // Execute this instrution.
        // '*nextp' will be set to the index of the next instruction
        // to execute. (For most instruction this will be the
        // instruction following this one.)
        // Returns 0 on success, non-zero if some error occurred.
        virtual int execute(THD *, uint *nextp)
        }

      • Statement instruction:
        class sp_instr_stmt : public sp_instr
        {
        sp_instr_stmt(uint ip);

        int execute(THD *, uint *nextp);

        // Set the statement's Lex
        void set_lex(LEX *);

        // Return the statement's Lex
        LEX *get_lex();
        }

    • SET instruction:
      class sp_instr_set : public sp_instr
      {
      // 'offset' is the variable's frame offset, 'val' the value,
      // and 'type' the variable type.
      sp_instr_set(uint ip,
      uint offset, Item *val, enum enum_field_types type);

      int execute(THD *, uint *nextp);
      }

    • Unconditional jump
      class sp_instr_jump : public sp_instr
      {
      // No destination, must be set.
      sp_instr_jump(uint ip);

      // 'dest' is the destination instruction index.
      sp_instr_jump(uint ip, uint dest);

      int execute(THD *, uint *nextp);

      // Set the destination instruction 'dest'.
      void set_destination(uint dest);
      }

    • Conditional jump
      class sp_instr_jump_if_not : public sp_instr_jump
      {
      // Jump if 'i' evaluates to false. Destination not set yet.
      sp_instr_jump_if_not(uint ip, Item *i);

      // Jump to 'dest' if 'i' evaluates to false.
      sp_instr_jump_if_not(uint ip, Item *i, uint dest)

      int execute(THD *, uint *nextp);
      }

    • Return a function value
      class sp_instr_freturn : public sp_instr
      {
      // Return the value 'val'
      sp_instr_freturn(uint ip, Item *val, enum enum_field_types type);

      int execute(THD *thd, uint *nextp);
      }

    • Push a handler and jump
      class sp_instr_hpush_jump : public sp_instr_jump
      {
      // Push handler of type 'htype', with current frame size 'fp'
      sp_instr_hpush_jump(uint ip, int htype, uint fp);

      int execute(THD *thd, uint *nextp);

      // Add condition for this handler
      void add_condition(struct sp_cond_type *cond);
      }

    • Pops handlers
      class sp_instr_hpop : public sp_instr
      {
      // Pop 'count' handlers
      sp_instr_hpop(uint ip, uint count);

      int execute(THD *thd, uint *nextp);
      }

    • Return from a CONTINUE handler
      class sp_instr_hreturn : public sp_instr
      {
      // Return from handler, and restore variables to 'fp'.
      sp_instr_hreturn(uint ip, uint fp);

      int execute(THD *thd, uint *nextp);
      }

    • Push a CURSOR
      class sp_instr_cpush : public sp_instr_stmt
      {
      // Push a cursor for statement 'lex'
      sp_instr_cpush(uint ip, LEX *lex)

    int execute(THD *thd, uint *nextp);
    }

    • Pop CURSORs
      class sp_instr_cpop : public sp_instr_stmt
      {
      // Pop 'count' cursors
      sp_instr_cpop(uint ip, uint count)

    int execute(THD *thd, uint *nextp);
    }

    • Open a CURSOR
      class sp_instr_copen : public sp_instr_stmt
      {
      // Open the 'c'th cursor
      sp_instr_copen(uint ip, uint c);

    int execute(THD *thd, uint *nextp);
    }

    • Close a CURSOR
      class sp_instr_cclose : public sp_instr
      {
      // Close the 'c'th cursor
      sp_instr_cclose(uint ip, uint c);

    int execute(THD *thd, uint *nextp);
    }

    • Fetch a row with CURSOR
      class sp_instr_cfetch : public sp_instr
      {
      // Fetch next with the 'c'th cursor
      sp_instr_cfetch(uint ip, uint c);

    int execute(THD *thd, uint *nextp);

    // Add a target variable for the fetch
    void add_to_varlist(struct sp_pvar *var);
    }

    • Utility functions: sp.h

      #define SP_OK 0
      #define SP_KEY_NOT_FOUND -1
      #define SP_OPEN_TABLE_FAILED -2
      #define SP_WRITE_ROW_FAILED -3
      #define SP_DELETE_ROW_FAILED -4
      #define SP_GET_FIELD_FAILED -5
      #define SP_PARSE_ERROR -6

      // Finds a stored procedure given its name. Returns NULL if not found.
      sp_head *sp_find_procedure(THD *, LEX_STRING *name);

      // Store the procedure 'name' in the database. 'def' is the complete
      // definition string ("create procedure ...").
      int sp_create_procedure(THD *,
      char *name, uint namelen,
      char *def, uint deflen,
      char *comment, uint commentlen, bool suid);

      // Drop the procedure 'name' from the database.
      int sp_drop_procedure(THD *, char *name, uint namelen);

      // Finds a stored function given its name. Returns NULL if not found.
      sp_head *sp_find_function(THD *, LEX_STRING *name);

      // Store the function 'name' in the database. 'def' is the complete
      // definition string ("create function ...").
      int sp_create_function(THD *,
      char *name, uint namelen,
      char *def, uint deflen,
      char *comment, uint commentlen, bool suid);

      // Drop the function 'name' from the database.
      int sp_drop_function(THD *, char *name, uint namelen);

    • The cache: sp_cache.h

      /* Initialize the SP caching once at startup */
      void sp_cache_init();

      /* Clear the cache *cp and set *cp to NULL */
      void sp_cache_clear(sp_cache **cp);

      /* Insert an SP to cache. If **cp points to NULL, it's set to a
      new cache */
      void sp_cache_insert(sp_cache **cp, sp_head *sp);

      /* Lookup an SP in cache */
      sp_head *sp_cache_lookup(sp_cache **cp, char *name, uint namelen);

      /* Remove an SP from cache */
      void sp_cache_remove(sp_cache **cp, sp_head *sp);

  • The mysql.proc schema:

    CREATE TABLE proc (
    db char(64) binary DEFAULT '' NOT NULL,
    name char(64) DEFAULT '' NOT NULL,
    type enum('FUNCTION','PROCEDURE') NOT NULL,
    specific_name char(64) DEFAULT '' NOT NULL,
    language enum('SQL') DEFAULT 'SQL' NOT NULL,
    sql_data_access enum('CONTAINS_SQL') DEFAULT 'CONTAINS_SQL' NOT NULL,
    is_deterministic enum('YES','NO') DEFAULT 'NO' NOT NULL,
    security_type enum('INVOKER','DEFINER') DEFAULT 'DEFINER' NOT NULL,
    param_list blob DEFAULT '' NOT NULL,
    returns char(64) DEFAULT '' NOT NULL,
    body blob DEFAULT '' NOT NULL,
    definer char(77) binary DEFAULT '' NOT NULL,
    created timestamp,
    modified timestamp,
    sql_mode set(
    'REAL_AS_FLOAT',
    'PIPES_AS_CONCAT',
    'ANSI_QUOTES',
    'IGNORE_SPACE',
    'NOT_USED',
    'ONLY_FULL_GROUP_BY',
    'NO_UNSIGNED_SUBTRACTION',
    'NO_DIR_IN_CREATE',
    'POSTGRESQL',
    'ORACLE',
    'MSSQL',
    'DB2',
    'MAXDB',
    'NO_KEY_OPTIONS',
    'NO_TABLE_OPTIONS',
    'NO_FIELD_OPTIONS',
    'MYSQL323',
    'MYSQL40',
    'ANSI',
    'NO_AUTO_VALUE_ON_ZERO'
    ) DEFAULT 0 NOT NULL,
    
    comment char(64) binary DEFAULT '' NOT NULL,
    PRIMARY KEY (db,name,type)
    ) comment='Stored Procedures';

--

Storage Ndb Docs Wl2077 (`storage/ndb/docs/wl2077.txt`) Copy Section

100' * (select 1 from T1 (1M rows) where key = rand());
1 host, 1 ndbd, api co-hosted
results in 1000 rows / sec

wo/reset bounds  w/ rb

4.1-read committed a) 4.9 b) 7.4
4.1-read hold lock c) 4.7 d) 6.7

wl2077-read committed 6.4 (+30%) 10.8 (+45%)
wl2077-read hold lock 4.6 (-1%) 6.7 (+ 0%)

5.0-ndb batch read committed f) 50' (+680%) g) 50' (+360%)
5.0-ndb batch read hold lock h) 12' (+160%) i) 13' (+79%)

shm-mem read committed (cmp. wl2077) a) 9.5' (+48%) b) 14' (+30%)
read hold lock c) 6.7' (+45%) d) 9.8' (+46%)

-- Comparision e) shm
serial pk: 10.9' 20' (+83%)
batched (1000): 59' 62' (+5%)
serial uniq index: 8.4' 14' (+66%)
batched (1000): 33' 36' (+9%)
index range (1000): 186'

load) testScanPerf -c 1 -d 1 T1
a) testScanPerf -s 100000 -c 0 -d 0 -a 1 -l 0 -r 2 -q 0 T1
b) testScanPerf -s 100000 -c 0 -d 0 -a 1 -l 0 -r 2 -q 1 T1
c) testScanPerf -s 100000 -c 0 -d 0 -a 1 -l 1 -r 2 -q 0 T1
d) testScanPerf -s 100000 -c 0 -d 0 -a 1 -l 1 -r 2 -q 1 T1
e) testReadPerf -i 25 -c 0 -d 0 T1
f) testScanPerf -s 100000 -c 0 -d 0 -a 1 -l 0 -r 3 -q 0 -m 1000 -i 10 T1
g) testScanPerf -s 100000 -c 0 -d 0 -a 1 -l 0 -r 3 -q 1 -m 1000 -i 10 T1

--- music join 1db-co 2db-co

4.1 13s 14s
4.1 wo/ blobs 1.7s 3.2s

wl2077 12s 14s
wl2077 wo/ blobs 1.2s (-30%) 2.5s (-22%)

pekka-blob-fix 1.3s

shm 1.2s 2.0s
shm wo/ blobs 1.1s 2.0s

Extra Opentelemetry Proto Opentelemetry Proto 1.10.0 Docs Design Goals (`extra/opentelemetry-proto/opentelemetry-proto-1.10.0/docs/design-goals.md`) Copy Section

Design Goals for OpenTelemetry Wire Protocol

We want to design a telemetry data exchange protocol that has the following characteristics:

• Be suitable for use between all of the following node types: instrumented applications, telemetry backends, local agents, stand-alone collectors/forwarders.

• Have high reliability of data delivery and clear visibility when the data cannot be delivered.

• Have low CPU usage for serialization and deserialization.

• Impose minimal pressure on memory manager, including pass-through scenarios, where deserialized data is short-lived and must be serialized as-is shortly after and where such short-lived data is created and discarded at high frequency (think telemetry data forwarders).

• Support ability to efficiently modify deserialized data and serialize again to pass further. This is related but slightly different from the previous requirement.

• Ensure high throughput (within the available bandwidth) in high latency networks (e.g. scenarios where telemetry source and the backend are separated by high latency network).

• Allow backpressure signalling.

• Be load-balancer friendly (do not hinder re-balancing).

Extra Opentelemetry Proto Opentelemetry Proto 1.10.0 Docs Requirements (`extra/opentelemetry-proto/opentelemetry-proto-1.10.0/docs/requirements.md`) Copy Section

OpenTelemetry Protocol Requirements

This document will drive OpenTelemetry Protocol design and RFC.

Goals Copy Chapter

See the goals of OpenTelemetry Protocol design here.

Vocabulary Copy Chapter

There are 2 parties involved in telemetry data exchange. In this document the party that is the source of telemetry data is called the Client, the party that is the destination of telemetry data is called the Server.

Examples of a Client are instrumented applications or sending side of telemetry collectors, examples of Servers are telemetry backends or receiving side of telemetry collectors (so a Collector is typically both a Client and a Server depending on which side you look from).

Known Issues with Existing Protocols Copy Chapter

Our experience with OpenCensus and other protocols has been that many of them have one or more of the following drawbacks:

• High CPU consumption for serialization and especially deserialization of received telemetry data.
• High and frequent CPU consumption by Garbage Collector.
• Lack of delivery guarantees for certain protocols (e.g. stream-based gRPC OpenCensus protocol) which makes troubleshooting of telemetry pipelines difficult.
• Not aware / not cooperating with load balancers resulting in potentially large imbalances in horizontally scaled backends.
• Support either traces or metrics but not both.

Our goal is to avoid or mitigate these known issues in the new protocol.

Requirements Copy Chapter

The following are OpenTelemetry protocol requirements.

Supported Node Types Copy Section

The protocol must be suitable for use between all of the following node types: instrumented applications, telemetry backends, telemetry agents running as local daemons, stand-alone collector/forwarder services.

Supported Data Types Copy Section

The protocol must support traces and metrics as data types.

Reliability of Delivery Copy Section

The protocol must ensure reliable data delivery and clear visibility when the data cannot be delivered. This should be achieved by sending data acknowledgements from the Server to the Client.

Note that acknowledgements alone are not sufficient to guarantee that: a) no data will be lost and b) no data will be duplicated. Acknowledgements can help to guarantee a) but not b). Guaranteeing both at the same is difficult. Because it is usually preferable for telemetry data to be duplicated than to lose it, we choose to guarantee that there are no data losses while potentially allowing duplicate data.

Duplicates can typically happen in edge cases (e.g. on reconnections, network interruptions, etc) when the client has no way of knowing if last sent data was delivered. In these cases the client will usually choose to re-send the data to guarantee the delivery which in turn may result in duplicate data on the server side.

To avoid having duplicates the client and the server could track sent and delivered items using uniquely identifying ids. The exact mechanism for tracking the ids and performing data de-duplication may be defined at the layer above the protocol layer and is outside the scope of this document.

For this reason we have slightly relaxed requirements and consider duplicate data acceptable in rare cases.

Note: this protocol is concerned with reliability of delivery between one pair of client/server nodes and aims to ensure that no data is lost in-transit between the client and the server. Many telemetry collection systems have multiple nodes that the data must travel across until reaching the final destination (e.g. application -> agent -> collector -> backend). End-to-end delivery guarantees in such systems is outside of the scope for this document. The acknowledgements described in this protocol happen between a single client/server pair and do not span multiple nodes in multi-hop delivery paths.

Throughput Copy Section

The protocol must ensure high throughput in high latency networks when the client and the server are not in the same data center.

This requirement may rule out half-duplex protocols. The throughput of half-duplex protocols is highly dependent on network roundtrip time and request size. To achieve good throughput request sizes may be too large to be practical.

Compression Copy Section

The protocol must achieve high compression ratios for telemetry data. The protocol design must consider batching of telemetry data and grouping of similar data (both can help to achieve better compression using common compression algorithms).

Encryption Copy Section

Industry standard encryption (e.g. TLS/HTTPS) must be supported.

Backpressure Signalling and Throttling Copy Section

The protocol must allow backpressure signalling.

If the server is unable to keep up with the pace of data it receives from the client then it must be able to signal that fact to the client. The client may then throttle itself to avoid overwhelming the server.

If the underlying transport is a stream that has its own flow control mechanism then the backpressure could be applied by delaying the reading of data from the server’s endpoint which could then be signalled to the client via underlying flow-control. However this approach makes it difficult for the client to distinguish server overloading from network delays (due to e.g. network losses). Such distinction is important for observability reasons. Because of this it is required for the protocol to allow to explicitly and clearly signal backpressure from the server to the client without relying on implicit signalling using underlying flow-control mechanisms.

The backpressure signal should include a hint to the client about desirable reduced rate of data.

Serialization Performance Copy Section

The protocol must have fast data serialization and deserialization characteristics.

Ideally it must also support very fast pass-through mode (when no modifications to the data are needed), fast “augmenting” or “tagging” of data and partial inspection of data (e.g. check for presence of specific tag). These requirements help to create fast Agents and Collectors.

Memory Usage Profile Copy Section

The protocol must impose minimal pressure on memory manager, including pass-through scenarios, when deserialized data is short-lived and must be serialized as-is shortly after and when such short-lived data is created and discarded at high frequency (think telemetry data forwarders).

The implementation of telemetry protocol must aim to minimize the number of memory allocations and deallocations performed during serialization and deserialization and aim to minimize the pressure on Garbage Collection (for GC languages).

Level 7 Load Balancer Friendly Copy Section

The protocol must allow Level 7 load balancers such as Envoy to re-balance the traffic for each batch of telemetry data. The traffic should not get pinned by a load balancer to one server for the entire duration of telemetry data sending, thus potentially leading to imbalanced load of servers located behind the load balancer.

Backwards Compatibility Copy Section

The protocol should be possible to evolve over time. It should be possible for nodes that implement different versions of OpenTelemetry protocol to interoperate (while possibly regressing to the lowest common denominator from functional perspective).

General Requirements Copy Section

The protocol must use well-known, mature encoding and transport mechanisms with ubiquitous availability of implementations in wide selection of languages that are supported by OpenTelemetry.

Extra Opentelemetry Proto Opentelemetry Proto 1.10.0 Docs Specification (`extra/opentelemetry-proto/opentelemetry-proto-1.10.0/docs/specification.md`) Copy Section

OpenTelemetry Protocol Specification

Status:

• Stable for the trace, metric and log signals.
• Development for the profiles signal.

The OpenTelemetry Protocol (OTLP) specification describes the encoding, transport,
and delivery mechanism of telemetry data between telemetry sources, intermediate
nodes such as collectors and telemetry backends.

Table of Contents

• Protocol Details
  • OTLP/gRPC
    • OTLP/gRPC Concurrent Requests
    • OTLP/gRPC Response
      • Full Success
      • Partial Success
      • Failures
    • OTLP/gRPC Throttling
    • OTLP/gRPC Service and Protobuf Definitions
    • OTLP/gRPC Default Port
  • OTLP/HTTP
    • Binary Protobuf Encoding
    • JSON Protobuf Encoding
    • OTLP/HTTP Request
    • OTLP/HTTP Response
      • Full Success
      • Partial Success
      • Failures
      • Bad Data
      • OTLP/HTTP Throttling
      • All Other Responses
    • OTLP/HTTP Connection
    • OTLP/HTTP Concurrent Requests
    • OTLP/HTTP Default Port
• Implementation Recommendations
  • Multi-Destination Exporting
• Known Limitations
  • Request Acknowledgements
    • Duplicate Data
• Future Versions and Interoperability
• Glossary
• References

OTLP is a general-purpose telemetry data delivery protocol designed in the scope
of the OpenTelemetry project.

Protocol Details Copy Chapter

OTLP defines the encoding of telemetry data and the protocol used to exchange
data between the client and the server.

This specification defines how OTLP is implemented over
gRPC and HTTP transports and specifies
Protocol Buffers schema
that is used for the payloads.

OTLP is a request/response style protocol: the clients send requests, and the
server replies with corresponding responses. This document defines one request
and response type: Export.

All server components MUST support the following transport compression options:

• No compression, denoted by none.
• Gzip compression, denoted by gzip.

OTLP/gRPC Copy Section

After establishing the underlying gRPC transport, the client starts sending
telemetry data using unary requests using
Export*ServiceRequest
messages (ExportLogsServiceRequest for logs,
ExportMetricsServiceRequest for metrics,
ExportTraceServiceRequest for traces,
ExportProfilesServiceRequest for profiles).
The client continuously sends a sequence of requests to the server and expects
to receive a response to each request:

Note: this protocol is concerned with the reliability of delivery between one
pair of client/server nodes and aims to ensure that no data is lost in transit
between the client and the server. Many telemetry collection systems have
intermediary nodes that the data must travel across until reaching the final
destination (e.g. application -> agent -> collector -> backend). End-to-end
delivery guarantees in such systems is outside of the scope of OTLP. The
acknowledgements described in this protocol happen between a single
client/server pair and do not span intermediary nodes in multi-hop delivery
paths.

OTLP/gRPC Concurrent Requests

After sending the request the client MAY wait until the response is received
from the server. In that case there will be at most only one request in flight
that is not yet acknowledged by the server.

Sequential operation is recommended when simplicity of implementation is
desirable, and when the client and the server are connected via very low-latency
network, such as when the client is an instrumented application and
the server is an OpenTelemetry Collector running as a local daemon (agent).

The implementations that need to achieve high throughput SHOULD support
concurrent Unary calls to achieve higher throughput. The client SHOULD send new
requests without waiting for the response to the earlier sent requests,
essentially creating a pipeline of requests that are currently in flight that
are not acknowledged.

The number of concurrent requests SHOULD be configurable.

The maximum achievable throughput is
max_concurrent_requests * max_request_size / (network_latency + server_response_time).
For example, if the request can contain at most 100 spans, network roundtrip
latency is 200ms, and server response time is 300 ms, then the maximum achievable
throughput with one concurrent request is 100 spans / (200ms+300ms) or 200
spans per second. It is easy to see that in high latency networks or when the
server response time is high to achieve good throughput, the requests need to be
very big or a lot concurrent requests must be done.

If the client is shutting down (e.g. when the containing process wants to exit)
the client will optionally wait until all pending acknowledgements are received
or until an implementation-specific timeout expires. This ensures the reliable
delivery of telemetry data. The client implementation SHOULD expose an option to
turn on and off the waiting during a shutdown.

If the client is unable to deliver a certain request (e.g. a timer expired while
waiting for acknowledgements) the client SHOULD record the fact that the data
was not delivered.

OTLP/gRPC Response

The response MUST be the appropriate message (see below for
the specific message to use in the Full Success,
Partial Success and Failure cases).

Full Success

The success response indicates telemetry data is successfully accepted by the
server.

If the server receives an empty request (a request that does not carry
any telemetry data) the server SHOULD respond with success.

On success, the server response MUST be a
Export<signal>ServiceResponse message
(ExportTraceServiceResponse for traces, ExportMetricsServiceResponse for
metrics, ExportLogsServiceResponse for logs and
ExportProfilesServiceResponse for profiles).

The server MUST leave the partial_success field unset
in case of a successful response.

Partial Success

If the request is only partially accepted
(i.e. when the server accepts only parts of the data and rejects the rest), the
server response MUST be the same
Export<signal>ServiceResponse
message as in the Full Success case.

Additionally, the server MUST initialize the partial_success field
(ExportTracePartialSuccess message for traces, ExportMetricsPartialSuccess
message for metrics, ExportLogsPartialSuccess message for logs and
ExportProfilesPartialSuccess for profiles), and it MUST set the respective
rejected_spans, rejected_data_points, rejected_log_records or
rejected_profiles field with the number of spans/data points/log
records/profiles it rejected.

The server SHOULD populate the error_message field with a human-readable
error message in English. The message should explain why the
server rejected parts of the data and might offer guidance on how users
can address the issues.
The protocol does not attempt to define the structure of the error message.

Servers MAY also use the partial_success field to convey
warnings/suggestions to clients even when the server fully accepts the request.
In such cases, the rejected_<signal> field MUST have a value of 0, and
the error_message field MUST be non-empty.

The client MUST NOT retry the request when it receives a partial success
response where the partial_success is populated.

Failures

When the server returns an error, it falls into 2 broad categories:
retryable and not-retryable:

• Retryable errors indicate that telemetry data processing failed, and the
  client SHOULD record the error and may retry exporting the same data.
  For example, this can happen when the server is temporarily unable to
  process the data.

• Not-retryable errors indicate that telemetry data processing failed, and the
  client MUST NOT retry sending the same telemetry data. The client MUST drop
  the telemetry data.
  For example, this can happen, when the request contains bad data
  and cannot be deserialized or processed by the server. The client
  SHOULD maintain a counter of such dropped data.

The server SHOULD indicate retryable errors using code
Unavailable and MAY supply
additional
details via status
using
RetryInfo.
Here is a sample Go code to illustrate:

// Do this on server side.
  st, err := status.New(codes.Unavailable, "Server is unavailable").
    WithDetails(&errdetails.RetryInfo{RetryDelay: &duration.Duration{Seconds: 5}})
  if err != nil {
    log.Fatal(err)
  }

  return st.Err()

To indicate not-retryable errors, the server is recommended to use code
InvalidArgument and MAY supply
additional
details via status
using
BadRequest.
If more appropriate, another gRPC status code may be used. Here is a
snippet of sample Go code to illustrate:

// Do this on the server side.
  st, err := status.New(codes.InvalidArgument, "Invalid Argument").
    WithDetails(&errdetails.BadRequest{})
  if err != nil {
    log.Fatal(err)
  }

  return st.Err()

The server MAY use other gRPC codes to indicate retryable and not-retryable
errors if those other gRPC codes are more appropriate for a particular erroneous
situation. The client SHOULD interpret gRPC status codes as retryable or
not-retryable according to the following table:

When retrying, the client SHOULD implement an exponential backoff strategy. An
exception to this is the Throttling case explained below, which provides
explicit instructions about retrying interval.

The client SHOULD interpret RESOURCE_EXHAUSTED code as retryable only if the
server signals that the recovery from resource exhaustion is possible.
This is signaled by the server by returning
a status containing
RetryInfo.
In this case the behavior of the server and the client is exactly as described in
OTLP/gRPC Throttling section. If no such status is returned,
then the RESOURCE_EXHAUSTED code SHOULD be treated as non-retryable.

OTLP/gRPC Throttling

OTLP allows backpressure signaling.

If the server is unable to keep up with the pace of data it receives from the
client then it SHOULD signal that fact to the client. The client MUST then
throttle itself to avoid overwhelming the server.

To signal backpressure when using gRPC transport, the server SHOULD return an
error with code Unavailable
and MAY supply additional
details via status
using
RetryInfo.
Here is a snippet of sample Go code to illustrate:

// Do this on the server side.
  st, err := status.New(codes.Unavailable, "Server is unavailable").
    WithDetails(&errdetails.RetryInfo{RetryDelay: &duration.Duration{Seconds: 30}})
  if err != nil {
    log.Fatal(err)
  }

  return st.Err()

  ...

  // Do this on the client side.
  st := status.Convert(err)
  for _, detail := range st.Details() {
    switch t := detail.(type) {
    case *errdetails.RetryInfo:
      if t.RetryDelay.Seconds > 0 || t.RetryDelay.Nanos > 0 {
        // Wait before retrying.
      }
    }
  }

When the client receives this signal, it SHOULD follow the recommendations
outlined in documentation for
RetryInfo:

// Describes when the clients can retry a failed request. Clients could ignore
// the recommendation here or retry when this information is missing from the error
// responses.
//
// It's always recommended that clients should use exponential backoff when
// retrying.
//
// Clients should wait until `retry_delay` amount of time has passed since
// receiving the error response before retrying.  If retrying requests also
// fail, clients should use an exponential backoff scheme to increase gradually
// the delay between retries based on `retry_delay` until either a maximum
// number of retries has been reached, or a maximum retry delay cap has been
// reached.

The value of retry_delay is determined by the server and is implementation
dependant. The server SHOULD choose a retry_delay value that is big enough to
give the server time to recover yet is not too big to cause the client to drop
data while being throttled.

OTLP/gRPC Service and Protobuf Definitions

gRPC service definitions are here.

Protobuf definitions for requests and responses are here.

Please make sure to check the proto version and maturity level.
Schemas for different signals may be at different maturity level - some stable,
some in beta.

OTLP/gRPC Default Port

The default network port for OTLP/gRPC is 4317.

OTLP/HTTP Copy Section

OTLP/HTTP uses Protobuf payloads encoded either in
binary format or in JSON format.
Regardless of the encoding the Protobuf schema of the messages is the same for
OTLP/HTTP and OTLP/gRPC as defined here.

OTLP/HTTP uses HTTP POST requests to send telemetry data from clients to
servers. Implementations MAY use HTTP/1.1 or HTTP/2 transports. Implementations
that use HTTP/2 transport SHOULD fallback to HTTP/1.1 transport if HTTP/2
connection cannot be established.

Binary Protobuf Encoding

Binary Protobuf encoded payloads use proto3
encoding standard.

The client and the server MUST set "Content-Type: application/x-protobuf"
request and response headers when sending binary Protobuf encoded payload.

JSON Protobuf Encoding

JSON Protobuf encoded payloads use proto3 standard defined
JSON Mapping
for mapping between Protobuf and JSON, with the following deviations from that mapping:

• The traceId and spanId byte arrays are represented as
  case-insensitive hex-encoded strings;
  they are not base64-encoded as is defined in the standard
  Protobuf JSON Mapping.
  Hex encoding is used for traceId and spanId fields in all OTLP
  Protobuf messages, e.g., the Span, Link, LogRecord, etc. messages.
  For example, the traceId field in a Span can be represented like this:
  { "traceId": "5B8EFFF798038103D269B633813FC60C", ... }

• Values of enum fields MUST be encoded as integer values. Unlike the standard
  Protobuf JSON Mapping,
  which allows values of enum fields to be encoded as either integer values or
  as enum name strings, only integer enum values are allowed in OTLP JSON
  Protobuf Encoding; the enum name strings MUST NOT be used. For example, the
  kind field with a value of SPAN_KIND_SERVER in a Span can be represented
  like this: { "kind": 2, ... }

• OTLP/JSON receivers MUST ignore message fields with unknown names and MUST
  unmarshal the message as if the unknown field was not present in the payload.
  This aligns with the behavior of the Binary Protobuf unmarshaler and ensures
  that adding new fields to OTLP messages does not break existing receivers.

• The keys of JSON objects are field names converted to lowerCamelCase. Original
  field names are not valid to use as keys for JSON objects.
  For example, this is a valid JSON representation of a Resource:
  { "attributes": {...}, "droppedAttributesCount": 123 }, and this is NOT a valid
  representation:
  { "attributes": {...}, "dropped_attributes_count": 123 }.

Note that according to Protobuf specs 64-bit integer
numbers in JSON-encoded payloads are encoded as decimal strings, and either
numbers or strings are accepted when decoding.

The client and the server MUST set "Content-Type: application/json" request and
response headers when sending JSON Protobuf encoded payload.

For JSON payload examples see: OTLP JSON request examples

OTLP/HTTP Request

Telemetry data is sent via HTTP POST request. The body of the POST request is a
payload either in binary-encoded Protobuf format or in JSON-encoded Protobuf
format.

The default URL path for requests that carry trace data is /v1/traces (for
example the full URL when connecting to "example.com" server will be
https://example.com/v1/traces). The request body is a Protobuf-encoded
ExportTraceServiceRequest message.

The default URL path for requests that carry metric data is /v1/metrics and
the request body is a Protobuf-encoded ExportMetricsServiceRequest message.

The default URL path for requests that carry log data is /v1/logs and the
request body is a Protobuf-encoded ExportLogsServiceRequest message.

The default URL path for requests that carry profiling data is
/v1development/profiles and the request body is a Protobuf-encoded
ExportProfilesServiceRequest message.

The client MAY gzip the content and in that case MUST include
"Content-Encoding: gzip" request header. The client MAY include
"Accept-Encoding: gzip" request header if it can receive gzip-encoded responses.

Non-default URL paths for requests MAY be configured on the client and server
sides.

OTLP/HTTP Response

The response body MUST be the appropriate serialized Protobuf message (see
below for the specific message to use in the Full Success,
Partial Success and Failure cases).

The server MUST set "Content-Type: application/x-protobuf" header if the
response body is binary-encoded Protobuf payload. The server MUST set
"Content-Type: application/json" if the response is JSON-encoded Protobuf
payload. The server MUST use the same "Content-Type" in the response as it
received in the request.

If the request header "Accept-Encoding: gzip" is present in the request the
server MAY gzip-encode the response and set "Content-Encoding: gzip" response
header.

Full Success

The success response indicates telemetry data is successfully accepted by the
server.

If the server receives an empty request (a request that does not carry
any telemetry data) the server SHOULD respond with success.

On success, the server MUST respond with HTTP 200 OK. The response body MUST
be a Protobuf-encoded
Export<signal>ServiceResponse message
(ExportTraceServiceResponse for traces, ExportMetricsServiceResponse for
metrics, ExportLogsServiceResponse for logs and
ExportProfilesServiceResponse for profiles).

The server MUST leave the partial_success field unset
in case of a successful response.

Partial Success

If the request is only partially accepted
(i.e. when the server accepts only parts of the data and rejects the rest), the
server MUST respond with HTTP 200 OK. The response body MUST be the same
Export<signal>ServiceResponse
message as in the Full Success case.

Additionally, the server MUST initialize the partial_success field
(ExportTracePartialSuccess message for traces, ExportMetricsPartialSuccess
message for metrics, ExportLogsPartialSuccess message for logs and
ExportProfilesPartialSuccess for profiles), and it MUST set the respective
rejected_spans, rejected_data_points, rejected_log_records or rejected_profiles field with
the number of spans/data points/log records it rejected.

The server SHOULD populate the error_message field with a human-readable
error message in English. The message should explain why the
server rejected parts of the data and might offer guidance on how users
can address the issues.
The protocol does not attempt to define the structure of the error message.

Servers MAY also use the partial_success field to convey
warnings/suggestions to clients even when it fully accepts the request.
In such cases, the rejected_<signal> field MUST have a value of 0, and
the error_message field MUST be non-empty.

The client MUST NOT retry the request when it receives a partial success
response where the partial_success is populated.

Failures

If the processing of the request fails, the server MUST respond with appropriate
HTTP 4xx or HTTP 5xx status code. See the sections below for more details about
specific failure cases and HTTP status codes that should be used.

The response body for all HTTP 4xx and HTTP 5xx responses MUST be a
Protobuf-encoded
Status
message that describes the problem.

This specification does not use Status.code field and the server MAY omit
Status.code field. The clients are not expected to alter their behavior based
on Status.code field but MAY record it for troubleshooting purposes.

The Status.message field SHOULD contain a developer-facing error message as
defined in Status message schema.

The server MAY include Status.details field with additional details. Read
below about what this field can contain in each specific failure case.

The server SHOULD use HTTP response status codes to indicate
retryable and not-retryable errors for a particular erroneous situation. The
client SHOULD honour HTTP response status codes as retryable or not-retryable.

Retryable Response Codes

The requests that receive a response status code listed in following table SHOULD
be retried.
All other 4xx or 5xx response status codes MUST NOT be retried.

Bad Data

If the processing of the request fails because the request contains data that
cannot be decoded or is otherwise invalid and such failure is permanent, then the
server MUST respond with HTTP 400 Bad Request. The Status.details field in
the response SHOULD contain a
BadRequest
that describes the bad data.

The client MUST NOT retry the request when it receives HTTP 400 Bad Request
response.

OTLP/HTTP Throttling

If the server receives more requests than the client is allowed or the server is
overloaded, the server SHOULD respond with HTTP 429 Too Many Requests or
HTTP 503 Service Unavailable and MAY include
"Retry-After" header with a
recommended time interval in seconds to wait before retrying.

The client SHOULD honour the waiting interval specified in the "Retry-After"
header if it is present. If the client receives a retryable error code (see
table above) and the "Retry-After" header is
not present in the response, then the client SHOULD implement an exponential backoff
strategy between retries.

All Other Responses

All other HTTP responses that are not explicitly listed in this document should
be treated according to HTTP specifications.

If the server disconnects without returning a response, the client SHOULD retry
and send the same request. The client SHOULD implement an exponential backoff
strategy between retries to avoid overwhelming the server.

OTLP/HTTP Connection

If the client cannot connect to the server, the client SHOULD retry the
connection using an exponential backoff strategy between retries. The interval
between retries must have a random jitter.

The client SHOULD keep the connection alive between requests.

Server implementations SHOULD accept OTLP/HTTP with binary-encoded Protobuf
payload and OTLP/HTTP with JSON-encoded Protobuf payload requests on the same
port and multiplex the requests to the corresponding payload decoder based on
the "Content-Type" request header.

Server implementations MAY accept OTLP/gRPC and OTLP/HTTP requests on the same
port and multiplex the connections to the corresponding transport handler based
on the "Content-Type" request header.

OTLP/HTTP Concurrent Requests

To achieve higher total throughput, the client MAY send requests using several
parallel HTTP connections. In that case, the maximum number of parallel
connections SHOULD be configurable.

OTLP/HTTP Default Port

The default network port for OTLP/HTTP is 4318.

Implementation Recommendations Copy Chapter

Multi-Destination Exporting Copy Section

An additional complication must be accounted for when one client must send
telemetry data to more than one destination server. When one of the servers
acknowledges the data and the other server does not (yet), the client needs
to decide how to move forward.

In such a situation, the client SHOULD implement queuing, acknowledgment
handling, and retrying logic per destination. This ensures that servers do not
block each other. The queues SHOULD reference shared, immutable data to be sent,
thus minimizing the memory overhead caused by having multiple queues.

This ensures that all destination servers receive the data regardless of their
speed of reception (within the available limits imposed by the size of the
client-side queue).

Empty Telemetry Envelopes Copy Section

Under certain circumstances, it is possible to have a telemetry envelope with
no contents. Some examples would be a ResourceMetrics with no ScopeMetrics
inside it, a ResourceMetrics with no Metrics inside it, or the equivalents for
Logs or Spans. One way this might happen would be for filtering rules to remove
all the contained data points, though there are others.

In practice, such empty envelopes are often discarded by existing
implementations. Given that, senders SHOULD NOT create empty envelopes (OTLP
payloads that contain zero spans, zero metric points or zero log records),
receivers MAY ignore empty envelopes, and implementations that receive and send
(forward) OTLP payloads MAY drop empty envelopes.

Known Limitations Copy Chapter

Request Acknowledgements Copy Section

Duplicate Data

In edge cases (e.g. on reconnections, network interruptions, etc) the client has
no way of knowing if recently sent data was delivered if no acknowledgement was
received yet. The client will typically choose to re-send such data to guarantee
delivery, which may result in duplicate data on the server side. This is a
deliberate choice and is considered to be the right tradeoff for telemetry data.

Future Versions and Interoperability Copy Chapter

OTLP will evolve and change over time. Future versions of OTLP must be designed
and implemented in a way that ensures that clients and servers that implement
different versions of OTLP can interoperate and exchange telemetry data. Old
clients must be able to talk to new servers and vice versa. Suppose new versions
of OTLP introduce new functionality that cannot be understood and supported by
nodes implementing the old versions of OTLP. In that case, the protocol must
regress to the lowest common denominator from a functional perspective.

When possible, the interoperability MUST be ensured between all versions of
OTLP that are not declared obsolete.

OTLP does not use explicit protocol version numbering. OTLP's interoperability
of clients and servers of different versions is based on the following concepts:

1. OTLP (current and future versions) defines a set of capabilities, some of
   which are mandatory, while others are optional. Clients and servers must implement
   mandatory capabilities and can choose to implement only a subset of optional
   capabilities.

2. For minor changes to the protocol, future versions and extensions of OTLP are
   encouraged to use the Protobuf's ability to evolve the message schema in
   a backward-compatible manner. Newer versions of OTLP may add new fields to
   messages that will be ignored by clients and servers that do not understand
   these fields. In many cases, careful design of such schema changes and correct
   choice of default values for new fields is enough to ensure interoperability
   of different versions without nodes explicitly detecting that their peer node
   has different capabilities.

3. More significant changes must be explicitly defined as new optional
   capabilities in future OTEPs. Such capabilities SHOULD be discovered by
   client and server implementations after establishing the underlying
   transport. The exact discovery mechanism SHOULD be described in future OTEPs,
   which define the new capabilities and typically can be implemented by making
   a discovery request/response message exchange from the client to server. The
   mandatory capabilities defined by this specification are implied and do not
   require discovery. The implementation which supports a new, optional
   capability MUST adjust its behavior to match the expectation of a peer that
   does not support a particular capability.

Glossary Copy Chapter

There are 2 parties involved in telemetry data exchange. In this document the
party that is the source of telemetry data is called the Client, the party
that is the destination of telemetry data is called the Server.

Examples of a Client are instrumented applications or sending side of telemetry
collectors, examples of Servers are telemetry backends or receiving side of
telemetry collectors (so a Collector is typically both a Client and a Server
depending on which side you look from).

Both the Client and the Server are also a Node. This term is used in the
document when referring to either one.

References Copy Chapter

• OTEP 0035 OpenTelemetry Protocol Specification
• OTEP 0099 OTLP/HTTP: HTTP Transport Extension for OTLP
• OTEP 0122 OTLP: JSON Encoding for OTLP/HTTP

Extra Protobuf Protobuf 24.4 Python Docs Index (`extra/protobuf/protobuf-24.4/python/docs/index.rst`) Copy Section

.. Protocol Buffers documentation master file, created by
sphinx-quickstart on Thu Aug 15 13:56:43 2019.
You can adapt this file completely to your liking, but it should at least
contain the root toctree directive.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/master/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

Protocol Buffers Python API Reference

The complete documentation for Protocol Buffers is available via the web at:

https://developers.google.com/protocol-buffers/

Modules and Packages Copy Chapter

.. START REFTOC, generated by generate_docs.py.
.. toctree::

google/protobuf
google/protobuf/any_pb2
google/protobuf/descriptor
google/protobuf/descriptor_database
google/protobuf/descriptor_pb2
google/protobuf/descriptor_pool
google/protobuf/duration_pb2
google/protobuf/empty_pb2
google/protobuf/field_mask_pb2
google/protobuf/internal/containers
google/protobuf/json_format
google/protobuf/message
google/protobuf/message_factory
google/protobuf/proto_builder
google/protobuf/reflection
google/protobuf/service
google/protobuf/service_reflection
google/protobuf/struct_pb2
google/protobuf/symbol_database
google/protobuf/text_encoding
google/protobuf/text_format
google/protobuf/timestamp_pb2
google/protobuf/type_pb2
google/protobuf/unknown_fields
google/protobuf/wrappers_pb2

.. END REFTOC.

Indices and tables Copy Chapter

• :ref:genindex
• :ref:modindex

Extra Protobuf Protobuf 24.4 Python Docs Requirements (`extra/protobuf/protobuf-24.4/python/docs/requirements.txt`) Copy Section

googleapis-common-protos==1.56.1
jinja2==3.0.0
sphinx==3.0.4
sphinx_rtd_theme==0.4.3
sphinxcontrib-napoleon==0.7

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf

.. automodule:: google.protobuf
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Any Pb2 (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/any_pb2.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.any_pb2

.. automodule:: google.protobuf.any_pb2
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Descriptor (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/descriptor.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.descriptor

.. automodule:: google.protobuf.descriptor
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Descriptor Database (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/descriptor_database.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.descriptor_database

.. automodule:: google.protobuf.descriptor_database
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Descriptor Pb2 (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/descriptor_pb2.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.descriptor_pb2

.. automodule:: google.protobuf.descriptor_pb2
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Descriptor Pool (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/descriptor_pool.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.descriptor_pool

.. automodule:: google.protobuf.descriptor_pool
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Duration Pb2 (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/duration_pb2.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.duration_pb2

.. automodule:: google.protobuf.duration_pb2
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Empty Pb2 (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/empty_pb2.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.empty_pb2

.. automodule:: google.protobuf.empty_pb2
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Field Mask Pb2 (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/field_mask_pb2.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.field_mask_pb2

.. automodule:: google.protobuf.field_mask_pb2
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Json Format (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/json_format.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.json_format

.. automodule:: google.protobuf.json_format
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Message (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/message.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.message

.. automodule:: google.protobuf.message
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Message Factory (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/message_factory.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.message_factory

.. automodule:: google.protobuf.message_factory
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Proto Builder (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/proto_builder.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.proto_builder

.. automodule:: google.protobuf.proto_builder
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Reflection (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/reflection.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.reflection

.. automodule:: google.protobuf.reflection
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Service (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/service.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.service

.. automodule:: google.protobuf.service
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Service Reflection (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/service_reflection.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.service_reflection

.. automodule:: google.protobuf.service_reflection
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Struct Pb2 (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/struct_pb2.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.struct_pb2

.. automodule:: google.protobuf.struct_pb2
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Symbol Database (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/symbol_database.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.symbol_database

.. automodule:: google.protobuf.symbol_database
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Text Encoding (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/text_encoding.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.text_encoding

.. automodule:: google.protobuf.text_encoding
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Text Format (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/text_format.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.text_format

.. automodule:: google.protobuf.text_format
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Timestamp Pb2 (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/timestamp_pb2.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.timestamp_pb2

.. automodule:: google.protobuf.timestamp_pb2
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Type Pb2 (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/type_pb2.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.type_pb2

.. automodule:: google.protobuf.type_pb2
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Unknown Fields (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/unknown_fields.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.unknown_fields

.. automodule:: google.protobuf.unknown_fields
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Wrappers Pb2 (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/wrappers_pb2.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.wrappers_pb2

.. automodule:: google.protobuf.wrappers_pb2
:members:
:inherited-members:
:undoc-members:

Extra Protobuf Protobuf 24.4 Python Docs Google Protobuf Internal Containers (`extra/protobuf/protobuf-24.4/python/docs/google/protobuf/internal/containers.rst`) Copy Section

.. DO NOT EDIT, generated by generate_docs.py.

.. ifconfig:: build_env == 'readthedocs'

.. warning::

You are reading the documentation for the `latest committed changes
  <https://github.com/protocolbuffers/protobuf/tree/main/python>`_ of
  the `Protocol Buffers package for Python
  <https://developers.google.com/protocol-buffers/docs/pythontutorial>`_.
  Some features may not yet be released. Read the documentation for the
  latest released package at `googleapis.dev
  <https://googleapis.dev/python/protobuf/latest/>`_.

google.protobuf.internal.containers

.. automodule:: google.protobuf.internal.containers
:members:
:inherited-members:
:undoc-members:

--- METRICS ---

• Files Extracted: 35
• Estimated Token Budget: ~27875 tokens
• Canonical Reference: https://codewiki.google/github.com/mysql/mysql-server