# Technical Documentation: llvm/llvm-project > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/llvm/llvm-project) ### Overview (README.md) # The LLVM Compiler Infrastructure [](https://securityscorecards.dev/viewer/?uri=github.com/llvm/llvm-project) [](https://www.bestpractices.dev/projects/8273) [](https://github.com/llvm/llvm-project/actions/workflows/libcxx-pr-conformance-tests.yaml?query=event%3Aschedule) Welcome to the LLVM project! This repository contains the source code for LLVM, a toolkit for the construction of highly optimized compilers, optimizers, and run-time environments. The LLVM project has multiple components. The core of the project is itself called "LLVM". This contains all of the tools, libraries, and header files needed to process intermediate representations and convert them into object files. Tools include an assembler, disassembler, bitcode analyzer, and bitcode optimizer. C-like languages use the [Clang](https://clang.llvm.org/) frontend. This component compiles C, C++, Objective-C, and Objective-C++ code into LLVM bitcode -- and from there into object files, using LLVM. Other components include: the [libc++ C++ standard library](https://libcxx.llvm.org), the [LLD linker](https://lld.llvm.org), and more. ## Getting the Source Code and Building LLVM Consult the [Getting Started with LLVM](https://llvm.org/docs/GettingStarted.html#getting-the-source-code-and-building-llvm) page for information on building and running LLVM. For information on how to contribute to the LLVM project, please take a look at the [Contributing to LLVM](https://llvm.org/docs/Contributing.html) guide. ## Getting in touch Join the [LLVM Discourse forums](https://discourse.llvm.org/), [Discord chat](https://discord.gg/xS7Z362), [LLVM Office Hours](https://llvm.org/docs/GettingInvolved.html#office-hours) or [Regular sync-ups](https://llvm.org/docs/GettingInvolved.html#online-sync-ups). The LLVM project has adopted a [code of conduct](https://llvm.org/docs/CodeOfConduct.html) for participants to all modes of communication within the project. --- ### Bolt Docs Index (`bolt/docs/index.md`) # BOLT ```{toctree} :hidden: true BOLTAArch64OptimizationStatus ``` BOLT is a post-link optimizer developed to speed up large applications. It achieves the improvements by optimizing application’s code layout based on execution profile gathered by sampling profiler, such as Linux `perf` tool. An overview of the ideas implemented in BOLT along with a discussion of its potential and current results is available in [CGO’19 paper](https://research.fb.com/publications/bolt-a-practical-binary-optimizer-for-data-centers-and-beyond/). # User Guides ```{toctree} :maxdepth: 1 GettingStarted OptimizingClang OptimizingLinux Heatmaps ``` # Reference ```{toctree} :maxdepth: 1 CommandLineArgumentReference profiles BOLTAArch64OptimizationStatus ``` # Design Documentation ```{toctree} :maxdepth: 1 BAT BinaryAnalysis NewBackend PointerAuthDesign RuntimeLibrary ``` --- ### Clang Tools Extra Docs Index (`clang-tools-extra/docs/index.md`) ```{title} Welcome to Extra Clang Tools's documentation! ``` # Introduction Welcome to the clang-tools-extra project which contains extra tools built using Clang's tooling APIs. ```{toctree} :maxdepth: 1 ReleaseNotes ``` # Contents ```{toctree} :maxdepth: 2 clang-tidy/index clang-include-fixer clang-change-namespace clang-reorder-fields modularize pp-trace clangd clang-doc Maintainers ``` # Doxygen Documentation The Doxygen documentation describes the **internal** software that makes up the tools of clang-tools-extra, not the **external** use of these tools. The Doxygen documentation contains no instructions about how to use the tools, only the APIs that make up the software. For usage instructions, please see the user's guide or reference manual for each tool. - Doxygen documentation :::{note} This documentation is generated directly from the source code with doxygen. Since the tools of clang-tools-extra are constantly under active development, what you're about to read is out of date! ::: # Indices and tables - {ref}`genindex` - {ref}`search` --- ### Clang Docs DataFlowAnalysisIntro (`clang/docs/DataFlowAnalysisIntro.md`) # Data flow analysis: an informal introduction ## Abstract This document introduces data flow analysis in an informal way. The goal is to give the reader an intuitive understanding of how it works, and show how it applies to a range of refactoring and bug finding problems. Data flow analysis is a well-established technique; it is described in many papers, books, and videos. If you would like a more formal, or a more thorough explanation of the concepts mentioned in this document, please refer to the following resources: * [The Lattice article in Wikipedia](https://en.wikipedia.org/wiki/Lattice_$order$). * Videos on the PacketPrep YouTube channel that introduce lattices and the necessary background information: [#20](https://www.youtube.com/watch?v=73j_FXBXGm8), [#21](https://www.youtube.com/watch?v=b5sDjo9tfE8), [#22](https://www.youtube.com/watch?v=saOG7Uooeho), [#23](https://www.youtube.com/watch?v=3EAYX-wZH0g), [#24](https://www.youtube.com/watch?v=KRkHwQtW6Cc), [#25](https://www.youtube.com/watch?v=7Gwzsc4rAgw). * [Introduction to Dataflow Analysis](https://www.youtube.com/watch?v=OROXJ9-wUQE) * [Introduction to abstract interpretation](http://www.cs.tau.ac.il/~msagiv/courses/asv/absint-1.pdf). * [Introduction to symbolic execution](https://www.cs.umd.edu/~mwh/se-tutorial/symbolic-exec.pdf). * [Static Program Analysis by Anders Møller and Michael I. Schwartzbach](https://cs.au.dk/~amoeller/spa/). * [EXE: automatically generating inputs of death](https://css.csail.mit.edu/6.858/2020/readings/exe.pdf) (a paper that successfully applies symbolic execution to real-world software). ## Data flow analysis ### The purpose of data flow analysis Data flow analysis is a static analysis technique that proves facts about a program or its fragment. It can make conclusions about all paths through the program, while taking control flow into account and scaling to large programs. The basic idea is propagating facts about the program through the edges of the control flow graph (CFG) until a fixpoint is reached. ### Sample problem and an ad-hoc solution We would like to explain data flow analysis while discussing an example. Let's imagine that we want to track possible values of an integer variable in our program. Here is how a human could annotate the code: ```c++ void Example(int n) { int x = 0; // x is {0} if (n > 0) { x = 5; // x is {5} } else { x = 42; // x is {42} } // x is {5; 42} print(x); } ``` We use sets of integers to represent possible values of `x`. Local variables have unambiguous values between statements, so we annotate program points between statements with sets of possible values. Here is how we arrived at these annotations. Assigning a constant to `x` allows us to make a conclusion that `x` can only have one value. When control flow from the "then" and "else" branches joins, `x` can have either value. Abstract algebra provides a nice formalism that models this kind of structure, namely, a lattice. A join-semilattice is a partially ordered set, in which every two elements have a least upper bound (called a *join*). ``` join(a, b) ⩾ a and join(a, b) ⩾ b and join(x, x) = x ``` For this problem we will use the lattice of subsets of integers, with set inclusion relation as ordering and set union as a join. Lattices are often represented visually as Hasse diagrams. Here is a Hasse diagram for our lattice that tracks subsets of integers: :::{figure} DataFlowAnalysisIntroImages/IntegerSetsInfiniteLattice.svg :class: light-background :alt: Hasse diagram for a lattice of integer sets ::: Computing the join in the lattice corresponds to finding the lowest common ancestor (LCA) between two nodes in its Hasse diagram. There is a vast amount of literature on efficiently implementing LCA queries for a DAG, however Efficient Implementation of Lattice Operations (1989) ([CiteSeerX](https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.106.4911), [doi](https://doi.org/10.1145%2F59287.59293)) describes a scheme that particularly well-suited for programmatic implementation. ### Too much information and "top" values Let's try to find the possible sets of values of `x` in a function that modifies `x` in a loop: ```c++ void ExampleOfInfiniteSets() { int x = 0; // x is {0} while (condition()) { x += 1; // x is {0; 1; 2; …} } print(x); // x is {0; 1; 2; …} } ``` We have an issue: `x` can have any value greater than zero; that's an infinite set of values, if the program operated on mathematical integers. In C++ `int` is limited by `INT_MAX` so technically we have a set `{0; 1; …; INT_MAX}` which is still really big. To make our analysis practical to compute, we have to limit the amount of information that we track. In this case, we can, for example, arbitrarily limit the size of sets to 3 elements. If at a certain program point `x` has more than 3 possible values, we stop tracking specific values at that program point. Instead, we denote possible values of `x` with the symbol `⊤` (pronounced "top" according to a convention in abstract algebra). ```c++ void ExampleOfTopWithALoop() { int x = 0; // x is {0} while (condition()) { x += 1; // x is ⊤ } print(x); // x is ⊤ } ``` The statement "at this program point, `x`'s possible values are `⊤`" is understood as "at this program point `x` can have any value because we have too much information, or the information is conflicting". Note that we can get more than 3 possible values even without a loop: ```c++ void ExampleOfTopWithoutLoops(int n) { int x = 0; // x is {0} switch(n) { case 0: x = 1; break; // x is {1} case 1: x = 9; break; // x is {9} case 2: x = 7; break; // x is {7} default: x = 3; break; // x is {3} } // x is ⊤ } ``` ### Uninitialized variables and "bottom" values When `x` is declared but not initialized, it has no possible values. We represent this fact symbolically as `⊥` (pronounced "bottom"). ```c++ void ExampleOfBottom() { int x; // x is ⊥ x = 42; // x is {42} print(x); } ``` Note that using values read from uninitialized variables is undefined behaviour in C++. Generally, compilers and static analysis tools can assume undefined behavior does not happen. We must model uninitialized variables only when we are implementing a checker that specifically is trying to find uninitialized reads. In this example we show how to model uninitialized variables only to demonstrate the concept of "bottom", and how it applies to possible value analysis. We describe an analysis that finds uninitialized reads in a section below. ### A practical lattice that tracks sets of concrete values Taking into account all corner cases covered above, we can put together a lattice that we can use in practice to track possible values of integer variables. This lattice represents sets of integers with 1, 2, or 3 elements, as well as top and bottom. Here is a Hasse diagram for it: :::{figure} DataFlowAnalysisIntroImages/IntegerSetsFiniteLattice.svg :class: light-background :alt: Hasse diagram for a lattice of integer sets ::: ### Formalization Let's consider a slightly more complex example, and think about how we can compute the sets of possible values algorithmically. ```c++ void Example(int n) { int x; // x is ⊥ if (n > 0) { if (n == 42) { x = 44; // x is {44} } else { x = 5; // x is {5} } print(x); // x is {44; 5} } else { x = n; // x is ⊤ } print(x); // x is ⊤ } ``` As humans, we understand the control flow from the program text. We used our understanding of control flow to find program points where two flows join. Formally, control flow is represented by a CFG (control flow graph): :::{figure} DataFlowAnalysisIntroImages/CFGExample.svg :class: light-background :alt: CFG for the code above ::: We can compute sets of possible values by propagating them through the CFG of the function: * When `x` is declared but not initialized, its possible values are `{}`. The empty set plays the role of `⊥` in this lattice. * When `x` is assigned a concrete value, its possible set of values contains just that specific value. * When `x` is assigned some unknown value, it can have any value. We represent this fact as `⊤`. * When two control flow paths join, we compute the set union of incoming values (limiting the number of elements to 3, representing larger sets as `⊤`). The sets of possible values are influenced by: * Statements, for example, assignments. * Joins in control flow, for example, ones that appear at the end of "if" statements. **Effects of statements** are modeled by what is formally known as a transfer function. A transfer function takes two arguments: the statement, and the state of `x` at the previous program point. It produces the state of `x` at the next program point. For example, the transfer function for assignment ignores the state at the previous program point: ```c++ // GIVEN: x is {42; 44} x = 0; // CONCLUSION: x is {0} ``` The transfer function for `+` performs arithmetic on every set member: ```c++ // GIVEN: x is {42, 44} x = x + 100; // CONCLUSION: x is {142, 144} ``` **Effects of control flow** are modeled by joining the knowledge from all possible previous program points. ```c++ if (...) { ... // GIVEN: x is {42} } else { ... // GIVEN: x is {44} } // CONCLUSION: x is {42; 44} ``` ```c++ // GIVEN: x is {42} while (...) { ... // GIVEN: x is {44} } // CONCLUSION: {42; 44} ``` The predicate that we marked "given" is usually called a precondition, and the conclusion is called a postcondition. In terms of the CFG, we join the information from all predecessor basic blocks. :::{figure} DataFlowAnalysisIntroImages/CFGJoinRule.svg :class: light-background :alt: Modeling the effects of a CFG basic block ::: Putting it all together, to model the effects of a basic block we compute: ``` out = transfer(basic_block, join(in_1, in_2, ..., in_n)) ``` (Note that there are other ways to write this equation that produce higher precision analysis results. The trick is to keep exploring the execution paths separately and delay joining until later. However, we won't discuss those variations here.) To make a conclusion about all paths through the program, we repeat this computation on all basic blocks until we reach a fixpoint. In other words, we keep propagating information through the CFG until the computed sets of values stop changing. If the lattice has a finite height and transfer functions are monotonic the algorithm is guaranteed to terminate. Each iteration of the algorithm can change computed values only to larger values from the lattice. In the worst case, all computed values become `⊤`, which is not very useful, but at least the analysis terminates at that point, because it can't change any of the values. Fixpoint iteration can be optimised by only reprocessing basic blocks which had one of their inputs changed on the previous iteration. This is typically implemented using a worklist queue. With this optimisation the time complexity becomes `O(m * |L|)`, where `m` is the number of basic blocks in the CFG and `|L|` is the size of lattice used by the analysis. ## Symbolic execution: a very short informal introduction ### Symbolic values In the previous example where we tried to figure out what values a variable can have, the analysis had to be seeded with a concrete value. What if there are no assignments of concrete values in the program? We can still deduce some interesting information by representing unknown input values symbolically, and computing results as symbolic expressions: ```c++ void PrintAbs(int x) { int result; if (x >= 0) { result = x; // result is {x} } else { result = -x; // result is {-x} } print(result); // result is {x; -x} } ``` We can't say what specific value gets printed, but we know that it is either `x` or `-x`. Dataflow analysis is an instance of abstract interpretation, and does not dictate how exactly the lattice and transfer functions should be designed, beyond the necessary conditions for the analysis to converge. Nevertheless, we can use symbolic execution ideas to guide our design of the lattice and transfer functions: lattice values can be symbolic expressions, and transfer functions can construct more complex symbolic expressions from symbolic expressions that represent arguments. See [this StackOverflow discussion](https://cstheory.stackexchange.com/questions/19708/symbolic-execution-is-a-case-of-abstract-interpretation) for a further comparison of abstract interpretation and symbolic execution. ### Flow condition A human can say about the previous example that the function returns `x` when `x >= 0`, and `-x` when `x < 0`. We can make this conclusion programmatically by tracking a flow condition. A flow condition is a predicate written in terms of the program state that is true at a specific program point regardless of the execution path that led to this statement. For example, the flow condition for the program point right before evaluating `result = x` is `x >= 0`. If we enhance the lattice to be a set of pairs of values and predicates, the dataflow analysis computes the following values: ```c++ void PrintAbs(int x) { int result; if (x >= 0) { // Flow condition: x >= 0. result = x; // result is {x if x >= 0} } else { // Flow condition: x < 0. result = -x; // result is {-x if x < 0} } print(result); // result is {x if x >= 0; -x if x < 0} } ``` Of course, in a program with loops, symbolic expressions for flow conditions can grow unbounded. A practical static analysis system must control this growth to keep the symbolic representations manageable and ensure that the data flow analysis terminates. For example, it can use a constraint solver to prune impossible flow conditions, and/or it can abstract them, losing precision, after their symbolic representations grow beyond some threshold. This is similar to how we had to limit the sizes of computed sets of possible values to 3 elements. ### Symbolic pointers This approach proves to be particularly useful for modeling pointer values, since we don't care about specific addresses but just want to give a unique identifier to a memory location. ```c++ void ExampleOfSymbolicPointers(bool b) { int x = 0; // x is {0} int* ptr = &x; // x is {0} ptr is {&x} if (b) { *ptr = 42; // x is {42} ptr is {&x} } print(x); // x is {0; 42} ptr is {&x} } ``` ## Example: finding output parameters Let's explore how data flow analysis can help with a problem that is hard to solve with other tools in Clang. ### Problem description Output parameters are function parameters of pointer or reference type whose pointee is completely overwritten by the function, and not read before it is overwritten. They are common in pre-C++11 code due to the absence of move semantics. In modern C++ output parameters are non-idiomatic, and return values are used instead. Imagine that we would like to refactor output parameters to return values to modernize old code. The first step is to identify refactoring candidates through static analysis. For example, in the following code snippet the pointer `c` is an output parameter: ```c++ struct Customer { int account_id; std::string name; } void GetCustomer(Customer *c) { c->account_id = ...; if (...) { c->name = ...; } else { c->name = ...; } } ``` We would like to refactor this code into: ```c++ Customer GetCustomer() { Customer c; c.account_id = ...; if (...) { c.name = ...; } else { c.name = ...; } return c; } ``` However, in the function below the parameter `c` is not an output parameter because its field `name` is not overwritten on every path through the function. ```c++ void GetCustomer(Customer *c) { c->account_id = ...; if (...) { c->name = ...; } } ``` The code also cannot read the value of the parameter before overwriting it: ```c++ void GetCustomer(Customer *c) { use(c->account_id); c->name = ...; c->account_id = ...; } ``` Functions that escape the pointer also block the refactoring: ```c++ Customer* kGlobalCustomer; void GetCustomer(Customer *c) { c->name = ...; c->account_id = ...; kGlobalCustomer = c; } ``` To identify a candidate function for refactoring, we need to do the following: * Find a function with a non-const pointer or reference parameter. * Find the definition of that function. * Prove that the function completely overwrites the pointee on all paths before returning. * Prove that the function reads the pointee only after overwriting it. * Prove that the function does not persist the pointer in a data structure that is live after the function returns. There are also requirements that all usage sites of the candidate function must satisfy, for example, that function arguments do not alias, that users are not taking the address of the function, and so on. Let's consider verifying usage site conditions to be a separate static analysis problem. ### Lattice design To analyze the function body we can use a lattice which consists of normal states and failure states. A normal state describes program points where we are sure that no behaviors that block the refactoring have occurred. Normal states keep track of all parameter's member fields that are known to be overwritten on every path from function entry to the corresponding program point. Failure states accumulate observed violations (unsafe reads and pointer escapes) that block the refactoring. In the partial order of the lattice failure states compare greater than normal states, which guarantees that they "win" when joined with normal states. Order between failure states is determined by inclusion relation on the set of accumulated violations (lattice's `⩽` is `⊆` on the set of violations). Order between normal states is determined by reversed inclusion relation on the set of overwritten parameter's member fields (lattice's `⩽` is `⊇` on the set of overwritten fields). :::{figure} DataFlowAnalysisIntroImages/OutputParameterIdentificationLattice.svg :class: light-background :alt: Lattice for data flow analysis that identifies output parameters ::: To determine whether a statement reads or writes a field we can implement symbolic evaluation of `DeclRefExpr`s, `LValueToRValue` casts, pointer dereference operator and `MemberExpr`s. ### Using data flow results to identify output parameters Let's take a look at how we use data flow analysis to identify an output parameter. The refactoring can be safely done when the data flow algorithm computes a normal state with all of the fields proven to be overwritten in the exit basic block of the function. ```c++ struct Customer { int account_id; std::string name; }; void GetCustomer(Customer* c) { // Overwritten: {} c->account_id = ...; // Overwritten: {c->account_id} if (...) { c->name = ...; // Overwritten: {c->account_id, c->name} } else { c->name = ...; // Overwritten: {c->account_id, c->name} } // Overwritten: {c->account_id, c->name} } ``` When the data flow algorithm computes a normal state, but not all fields are proven to be overwritten we can't perform the refactoring. ```c++ void target(bool b, Customer* c) { // Overwritten: {} if (b) { c->account_id = 42; // Overwritten: {c->account_id} } else { c->name = "Konrad"; // Overwritten: {c->name} } // Overwritten: {} } ``` Similarly, when the data flow algorithm computes a failure state, we also can't perform the refactoring. ```c++ Customer* kGlobalCustomer; void GetCustomer(Customer* c) { // Overwritten: {} c->account_id = ...; // Overwritten: {c->account_id} if (...) { print(c->name); // Unsafe read } else { kGlobalCustomer = c; // Pointer escape } // Unsafe read, Pointer escape } ``` ## Example: finding dead stores Let's say we want to find redundant stores, because they indicate potential bugs. ```c++ x = GetX(); x = GetY(); ``` The first store to `x` is never read, probably there is a bug. The implementation of dead store analysis is very similar to output parameter analysis: we need to track stores and loads, and find stores that were never read. [Liveness analysis](https://en.wikipedia.org/wiki/Live_variable_analysis) is a generalization of this idea, which is often used to answer many related questions, for example: * finding dead stores, * finding uninitialized variables, * finding a good point to deallocate memory, * finding out if it would be safe to move an object. ## Example: definitive initialization Definitive initialization proves that variables are known to be initialized when read. If we find a variable which is read when not initialized then we generate a warning. ```c++ void Init() { int x; // x is uninitialized if (cond()) { x = 10; // x is initialized } else { x = 20; // x is initialized } print(x); // x is initialized } ``` ```c++ void Uninit() { int x; // x is uninitialized if (cond()) { x = 10; // x is initialized } print(x); // x is maybe uninitialized, x is being read, report a bug. } ``` For this purpose we can use lattice in a form of a mapping from variable declarations to initialization states; each initialization state is represented by the following lattice: :::{figure} DataFlowAnalysisIntroImages/DefinitiveInitializationLattice.svg :class: light-background :alt: Lattice for definitive initialization analysis ::: A lattice element could also capture the source locations of the branches that lead us to the corresponding program point. Diagnostics would use this information to show a sample buggy code path to the user. ## Example: refactoring raw pointers to `unique_ptr` Modern idiomatic C++ uses smart pointers to express memory ownership, however in pre-C++11 code one can often find raw pointers that own heap memory blocks. Imagine that we would like to refactor raw pointers that own memory to `unique_ptr`. There are multiple ways to design a data flow analysis for this problem; let's look at one way to do it. For example, we would like to refactor the following code that uses raw pointers: ```c++ void UniqueOwnership1() { int *pi = new int; if (...) { Borrow(pi); delete pi; } else { TakeOwnership(pi); } } ``` into code that uses `unique_ptr`: ```c++ void UniqueOwnership1() { auto pi = std::make_unique(); if (...) { Borrow(pi.get()); } else { TakeOwnership(pi.release()); } } ``` This problem can be solved with a lattice in form of map from value declarations to pointer states: :::{figure} DataFlowAnalysisIntroImages/UniquePtrLattice.svg :class: light-background :alt: Lattice that identifies candidates for unique_ptr refactoring ::: We can perform the refactoring if at the exit of a function `pi` is `Compatible`. ```c++ void UniqueOwnership1() { int *pi; // pi is Compatible pi = new int; // pi is Defined if (...) { Borrow(pi); // pi is Defined delete pi; // pi is Compatible } else { TakeOwnership(pi); // pi is Compatible } // pi is Compatible } ``` Let's look at an example where the raw pointer owns two different memory blocks: ```c++ void UniqueOwnership2() { int *pi = new int; // pi is Defined Borrow(pi); delete pi; // pi is Compatible if (smth) { pi = new int; // pi is Defined Borrow(pi); delete pi; // pi is Compatible } // pi is Compatible } ``` It can be refactored to use `unique_ptr` like this: ```c++ void UniqueOwnership2() { auto pi = make_unique(); Borrow(pi); if (smth) { pi = make_unique(); Borrow(pi); } } ``` In the following example, the raw pointer is used to access the heap object after the ownership has been transferred. ```c++ void UniqueOwnership3() { int *pi = new int; // pi is Defined if (...) { Borrow(pi); delete pi; // pi is Compatible } else { vector> v = {std::unique_ptr(pi)}; // pi is Compatible print(*pi); use(v); } // pi is Compatible } ``` We can refactor this code to use `unique_ptr`, however we would have to introduce a non-owning pointer variable, since we can't use the moved-from `unique_ptr` to access the object: ```c++ void UniqueOwnership3() { std::unique_ptr pi = std::make_unique(); if (...) { Borrow(pi); } else { int *pi_non_owning = pi.get(); vector> v = {std::move(pi)}; print(*pi_non_owning); use(v); } } ``` If the original code didn't call `delete` at the very end of the function, then our refactoring may change the point at which we run the destructor and release memory. Specifically, if there is some user code after `delete`, then extending the lifetime of the object until the end of the function may hold locks for longer than necessary, introduce memory overhead etc. One solution is to always replace `delete` with a call to `reset()`, and then perform another analysis that removes unnecessary `reset()` calls. ```c++ void AddedMemoryOverhead() { HugeObject *ho = new HugeObject(); use(ho); delete ho; // Release the large amount of memory quickly. LongRunningFunction(); } ``` This analysis will refuse to refactor code that mixes borrowed pointer values and unique ownership. In the following code, `GetPtr()` returns a borrowed pointer, which is assigned to `pi`. Then, `pi` is used to hold a uniquely-owned pointer. We don't distinguish between these two assignments, and we want each assignment to be paired with a corresponding sink; otherwise, we transition the pointer to a `Conflicting` state, like in this example. ```c++ void ConflictingOwnership() { int *pi; // pi is Compatible pi = GetPtr(); // pi is Defined Borrow(pi); // pi is Defined pi = new int; // pi is Conflicting Borrow(pi); delete pi; // pi is Conflicting } ``` We could still handle this case by finding a maximal range in the code where `pi` could be in the Compatible state, and only refactoring that part. ```c++ void ConflictingOwnership() { int *pi; pi = GetPtr(); Borrow(pi); std::unique_ptr pi_unique = std::make_unique(); Borrow(pi_unique.get()); } ``` ## Example: finding redundant branch conditions In the code below `b1` should not be checked in both the outer and inner "if" statements. It is likely there is a bug in this code. ```c++ int F(bool b1, bool b2) { if (b1) { f(); if (b1 && b2) { // Check `b1` again -- unnecessary! g(); } } } ``` A checker that finds this pattern syntactically is already implemented in ClangTidy using AST matchers (`bugprone-redundant-branch-condition`). To implement it using the data flow analysis framework, we can produce a warning if any part of the branch condition is implied by the flow condition. ```c++ int F(bool b1, bool b2) { // Flow condition: true. if (b1) { // Flow condition: b1. f(); if (b1 && b2) { // `b1` is implied by the flow condition. g(); } } } ``` One way to check this implication is to use a SAT solver. Without a SAT solver, we could keep the flow condition in the CNF form and then it would be easy to check the implication. ## Example: finding unchecked `std::optional` unwraps Calling `optional::value()` is only valid if `optional::has_value()` is true. We want to show that when `x.value()` is executed, the flow condition implies `x.has_value()`. In the example below `x.value()` is accessed safely because it is guarded by the `x.has_value()` check. ```c++ void Example(std::optional &x) { if (x.has_value()) { use(x.value()); } } ``` While entering the if branch we deduce that `x.has_value()` is implied by the flow condition. ```c++ void Example(std::optional x) { // Flow condition: true. if (x.has_value()) { // Flow condition: x.has_value() == true. use(x.value()); } // Flow condition: true. } ``` We also need to prove that `x` is not modified between check and value access. The modification of `x` may be very subtle: ```c++ void F(std::optional &x); void Example(std::optional &x) { if (x.has_value()) { // Flow condition: x.has_value() == true. unknown_function(x); // may change x. // Flow condition: true. use(x.value()); } } ``` ## Example: finding dead code behind A/B experiment flags Finding dead code is a classic application of data flow analysis. Unused flags for A/B experiment hide dead code. However, this flavor of dead code is invisible to the compiler because the flag can be turned on at any moment. We could make a tool that deletes experiment flags. The user tells us which flag they want to delete, and we assume that the it's value is a given constant. For example, the user could use the tool to remove `example_flag` from this code: ```c++ DEFINE_FLAG(std::string, example_flag, "", "A sample flag."); void Example() { bool x = GetFlag(FLAGS_example_flag).empty(); f(); if (x) { g(); } else { h(); } } ``` The tool would simplify the code to: ```c++ void Example() { f(); g(); } ``` We can solve this problem with a classic constant propagation lattice combined with symbolic evaluation. ## Example: finding inefficient usages of associative containers Real-world code often accidentally performs repeated lookups in associative containers: ```c++ map xs; xs[42]->name = "..."; xs[42]->title = "..."; ``` To find the above inefficiency we can use the available expressions analysis to understand that `m[42]` is evaluated twice. ```c++ map xs; Employee &e = xs[42]; e->name = "..."; e->title = "..."; ``` We can also track the `m.contains()` check in the flow condition to find redundant checks, like in the example below. ```c++ std::map xs; if (!xs.contains(42)) { xs.insert({42, someEmployee}); } ``` ## Example: refactoring types that implicitly convert to each other Refactoring one strong type to another is difficult, but the compiler can help: once you refactor one reference to the type, the compiler will flag other places where this information flows with type mismatch errors. Unfortunately this strategy does not work when you are refactoring types that implicitly convert to each other, for example, replacing `int32_t` with `int64_t`. Imagine that we want to change user IDs from 32 to 64-bit integers. In other words, we need to find all integers tainted with user IDs. We can use data flow analysis to implement taint analysis. ```c++ void UseUser(int32_t user_id) { int32_t id = user_id; // Variable `id` is tainted with a user ID. ... } ``` Taint analysis is very well suited to this problem because the program rarely branches on user IDs, and almost certainly does not perform any computation (like arithmetic). --- ### Clang Docs Index (`clang/docs/index.md`) % Clang documentation master file, created by % sphinx-quickstart on Sun Dec 9 20:01:55 2012. % You can adapt this file completely to your liking, but it should at least % contain the root `toctree` directive. ```{title} Welcome to Clang's documentation! ``` ``` /* Detailed source-code truncated for AI context efficiency. */ ``` * {doc}`ReleaseNotes` # Using Clang as a Compiler * {doc}`UsersManual` * {doc}`Toolchain` * {doc}`LanguageExtensions` * {doc}`ClangCommandLineReference` * {doc}`AttributeReference` * {doc}`DiagnosticsReference` * {doc}`AMDGPUBuiltinReference` * {doc}`WarningSuppressionMappings` * {doc}`CrossCompilation` * {doc}`ClangStaticAnalyzer` * {doc}`ThreadSafetyAnalysis` * {doc}`LifetimeSafety` * {doc}`SafeBuffers` * {doc}`ScalableStaticAnalysis/index` * {doc}`DataFlowAnalysisIntro` * {doc}`FunctionEffectAnalysis` * {doc}`AddressSanitizer` * {doc}`HardwareAssistedAddressSanitizer` * {doc}`ThreadSanitizer` * {doc}`MemorySanitizer` * {doc}`UndefinedBehaviorSanitizer` * {doc}`DataFlowSanitizer` * {doc}`LeakSanitizer` * {doc}`TypeSanitizer` * {doc}`RealtimeSanitizer` * {doc}`SanitizerCoverage` * {doc}`SanitizerStats` * {doc}`SanitizerSpecialCaseList` * {doc}`AllocToken` * {doc}`OverflowBehaviorTypes` * {doc}`BoundsSafety` * {doc}`BoundsSafetyAdoptionGuide` * {doc}`BoundsSafetyImplPlans` * {doc}`ControlFlowIntegrity` * {doc}`LTOVisibility` * {doc}`PointerAuthentication` * {doc}`ReturnAddressAuthenticationHardening` * {doc}`SafeStack` * {doc}`ShadowCallStack` * {doc}`StructureProtection` * {doc}`SourceBasedCodeCoverage` * {doc}`StandardCPlusPlusModules` * {doc}`Modules` * {doc}`MSVCCompatibility` * {doc}`MisExpect` * {doc}`OpenCLSupport` * {doc}`OpenMPSupport` * {doc}`SYCLSupport` * {doc}`HIPSupport` * {doc}`HLSL/HLSLDocs` * {doc}`ThinLTO` * {doc}`APINotes` * {doc}`DebuggingCoroutines` * {doc}`AMDGPUSupport` * {doc}`RISCVSupport` * {doc}`CXXTypeAwareAllocators` * {doc}`CIR/index` * {doc}`CommandGuide/index` * {doc}`FAQ` # Using Clang as a Library * {doc}`Tooling` * {doc}`ExternalClangExamples` * {doc}`IntroductionToTheClangAST` * {doc}`LibTooling` * {doc}`LibClang` * {doc}`LibFormat` * {doc}`ClangPlugins` * {doc}`RAVFrontendAction` * {doc}`LibASTMatchersTutorial` * {doc}`LibASTMatchers` * {doc}`ClangTransformerTutorial` * {doc}`LibASTImporter` * {doc}`HowToSetupToolingForLLVM` * {doc}`JSONCompilationDatabase` * {doc}`RefactoringEngine` # Using Clang Tools * {doc}`ClangTools` * {doc}`ClangCheck` * {doc}`ClangFormat` * {doc}`ClangFormatStyleOptions` * {doc}`ClangLinkerWrapper` * {doc}`ClangNVLinkWrapper` * {doc}`ClangOffloadBundler` * {doc}`ClangRepl` * {doc}`ClangSYCLLinker` # Design Documents * {doc}`Maintainers` * {doc}`InternalsManual` * {doc}`DriverInternals` * {doc}`Multilib` * {doc}`OffloadingDesign` * {doc}`PCHInternals` * {doc}`ItaniumMangleAbiTags` * {doc}`ControlFlowIntegrityDesign` * {doc}`HardwareAssistedAddressSanitizerDesign` * {doc}`ConstantInterpreter` * {doc}`LLVMExceptionHandlingCodeGen` # Indices and tables - {ref}`genindex` - {ref}`search` --- ### Flang Docs Overview (`flang/docs/Overview.md`) # Overview of Compiler Phases The Flang compiler transforms Fortran source code into an executable file. This transformation proceeds in three high level phases -- analysis, lowering, and code generation/linking. The first high level phase (analysis) transforms Fortran source code into a decorated parse tree and a symbol table. During this phase, all user related errors are detected and reported. The second high level phase (lowering), changes the decorated parse tree and symbol table into the Fortran Intermediate Representation (FIR), which is a dialect of LLVM's Multi-Level Intermediate Representation or MLIR. It then runs a series of passes on the FIR code which verify its validity, perform a series of optimizations, and finally transform it into LLVM's Intermediate Representation, or LLVM IR The third high level phase generates machine code and invokes a linker to produce an executable file. This document describes the first two high level phases. Each of these is described in more detailed phases. Each detailed phase is described -- its inputs and outputs along with how to produce a readable version of the outputs. Each detailed phase produces either correct output or fatal errors. ## Analysis This high level phase validates that the program is correct and creates all of the information needed for lowering. ### Prescan and Preprocess See [Preprocessing.md](Preprocessing.md). **Input:** Fortran source and header files, command line macro definitions, set of enabled compiler directives (to be treated as directives rather than comments). **Output:** - A "cooked" character stream: the entire program as a contiguous stream of normalized Fortran source. Extraneous whitespace and comments are removed (except comments that are compiler directives that are not disabled) and case is normalized. Also, directives are processed and macros expanded. - Provenance information mapping each character back to the source it came from. This is used in subsequent phases that need source locations. This includes error messages, optimization reports, and debugging information. **Entry point:** `parser::Parsing::Prescan` **Commands:** - `flang -fc1 -E src.f90` dumps the cooked character stream - `flang -fc1 -fdebug-dump-provenance src.f90` dumps provenance information ### Parsing **Input:** Cooked character stream **Output:** A parse tree for each Fortran program unit in the source code representing a syntactically correct program, rooted at the program unit. See: [Parsing.md](Parsing.md) and [ParserCombinators.md](ParserCombinators.md). **Entry point:** `parser::Parsing::Parse` **Commands:** - `flang -fc1 -fdebug-dump-parse-tree-no-sema src.f90` dumps the parse tree - `flang -fc1 -fdebug-unparse src.f90` converts the parse tree to normalized Fortran - `flang -fc1 -fdebug-dump-parsing-log src.f90` runs an instrumented parse and dumps the log - `flang -fc1 -fdebug-measure-parse-tree src.f90` measures the parse tree ### Semantic processing **Input:** the parse tree, the cooked character stream, and provenance information **Output:** * a symbol table * modified parse tree * module files, (see: [ModFiles.md](ModFiles.md)) * the intrinsic procedure table * the target characteristics * the runtime derived type derived type tables (see: [RuntimeTypeInfo.md](RuntimeTypeInfo.md)) **Entry point:** `semantics::Semantics::Perform` For more detail on semantic analysis, see: [Semantics.md](Semantics.md). Semantic processing performs several tasks: * validates labels, see: [LabelResolution.md](LabelResolution.md). * canonicalizes DO statements, * canonicalizes OpenACC and OpenMP code * resolves names, building a tree of scopes and symbols * rewrites the parse tree to correct parsing mistakes (when needed) once semantic information is available to clarify the program's meaning * checks the validity of declarations * analyzes expressions and statements, emitting error messages where appropriate * creates module files if the source code contains modules, see [ModFiles.md](ModFiles.md). In the course of semantic analysis, the compiler: * creates the symbol table * decorates the parse tree with semantic information (such as pointers into the symbol table) * creates the intrinsic procedure table * folds constant expressions At the end of semantic processing, all validation of the user's program is complete. This is the last detailed phase of analysis processing. **Commands:** - `flang -fc1 -fdebug-dump-parse-tree src.f90` dumps the parse tree after semantic analysis - `flang -fc1 -fdebug-dump-symbols src.f90` dumps the symbol table - `flang -fc1 -fdebug-dump-all src.f90` dumps both the parse tree and the symbol table ## Lowering Lowering takes the parse tree and symbol table produced by analysis and produces LLVM IR. ### Create the lowering bridge **Inputs:** - the parse tree - the symbol table - The default KINDs for intrinsic types (specified by default or command line option) - The intrinsic procedure table (created in semantics processing) - The target characteristics (created during semantics processing) - The cooked character stream - The target triple -- CPU type, vendor, operating system - The mapping between Fortran KIND values to FIR KIND values The lowering bridge is a container that holds all of the information needed for lowering. **Output:** A container with all of the information needed for lowering **Entry point:** lower::LoweringBridge::create ### Initial lowering **Input:** the lowering bridge **Output:** A Fortran IR (FIR) representation of the program. **Entry point:** `lower::LoweringBridge::lower` The compiler then takes the information in the lowering bridge and creates a pre-FIR tree or PFT. The PFT is a list of programs and modules. The programs and modules contain lists of function-like units. The function-like units contain a list of evaluations. All of these contain pointers back into the parse tree. The compiler walks the PFT generating FIR. **Commands:** - `flang -fc1 -fdebug-dump-pft src.f90` dumps the pre-FIR tree - `flang -fc1 -emit-mlir src.f90` dumps the FIR to the files src.mlir ### Transformation passes **Input:** initial version of the FIR code **Output:** An LLVM IR representation of the program **Entry point:** `mlir::PassManager::run` The compiler then runs a series of passes over the FIR code. The first is a verification pass. It's followed by a series of transformation passes that perform various optimizations and transformations. The final pass creates an LLVM IR representation of the program. **Commands:** - `flang -fc1 -emit-llvm src.f90` dumps the LLVM IR to src.ll - `flang -mmlir --mlir-print-ir-after-all -S src.f90` dumps the FIR code after each pass to standard error - `flang -mmlir --mlir-print-ir-before= -S src.f90` dumps the FIR code before a specific pass to standard error - `flang -mmlir --mlir-print-ir-after= -S src.f90` dumps the FIR code after a specific pass to standard error Note: The exact set of passes depends on compilation options. For example, `stack-arrays` only appears when `-fstack-arrays` is enabled. To see the complete list for a given compilation, use `--mlir-print-ir-before-all` and look at the `IR Dump Before` headers. ## Object code generation and linking After the LLVM IR is created, the flang driver invokes LLVM's existing infrastructure to generate object code and invoke a linker to create the executable file. --- ### Flang Docs Index (`flang/docs/index.md`) # Welcome to Flang's documentation Flang is LLVM's Fortran frontend that can be found [here](https://github.com/llvm/llvm-project/tree/main/flang). It is often referred to as "LLVM Flang" to differentiate itself from ["Classic Flang"](https://github.com/flang-compiler/flang) - these are two separate and independent Fortran compilers. LLVM Flang is under active development. While it is capable of generating executables for a number of examples, some functionality is still missing. See [Getting Involved](GettingInvolved.md) for tips on how to get in touch with us and to learn more about the current status. ```{eval-rst} .. toctree:: :titlesonly: ReleaseNotes ``` # Using Flang ```{eval-rst} .. toctree:: :titlesonly: GettingStarted FlangCommandLineReference FortranStandardsSupport Extensions Directives OpenMPSupport OpenMP-extensions Real16MathSupport Unsigned FAQ ``` # Contributing to Flang ```{eval-rst} .. toctree:: :titlesonly: C++17 C++style DesignGuideline FortranForCProgrammers GettingInvolved ImplementingASemanticCheck PullRequestChecklist ``` # Design Documents ```{eval-rst} .. toctree:: :titlesonly: Aliasing AliasingAnalysisFIR ArrayComposition ArrayRepacking AssumedRank BijectiveInternalNameUniquing Calls Character ComplexOperations ControlFlowGraph DebugGeneration DoConcurrent DoConcurrentConversionToOpenMP F202X FIRLangRef FlangDriver FortranFeatureHistory FortranIR FortranLLVMTestSuite HighLevelFIR IORuntimeInternals InlineRuntimeChecks InternalProcedureTrampolines Intrinsics IntrinsicTypes LabelResolution ModFiles OpenACC OpenACC-extensions OpenACC-descriptor-management.md OpenMP-4.5-grammar.md OpenMP-declare-target OpenMP-descriptor-management OpenMP-semantics OptionComparison Overview ParallelMultiImageFortranRuntime ParameterizedDerivedTypes ParserCombinators Parsing PolymorphicEntities Preprocessing ProcedurePointer RuntimeDescriptor RuntimeEnvironment RuntimeTypeInfo Semantics f2018-grammar.md fstack-arrays ``` # Historical References ```{eval-rst} .. toctree:: :titlesonly: HLFIRTransition ``` # Indices and tables ```{eval-rst} * :ref:`genindex` * :ref:`search` ``` --- ### Libc Docs Index (`libc/docs/index.md`) ```{image} ./_static/llvm_libc_logo.svg :alt: The LLVM-libc logo, drawn by Jordan Abrahams-Whitehead :align: right :width: 25% ``` # The LLVM C Library LLVM-libc is a from-scratch implementation of the C standard library, built as part of the LLVM project. It is designed to be **modular** (any piece can be used independently), **multiplatform** (Linux, GPU, baremetal embedded, UEFI, macOS, Windows), and written in modern C++ for correctness, performance, and safety. The logo was drawn by Jordan Abrahams-Whitehead ## What Works Today LLVM-libc is **actively used in production** for a targeted set of use cases, though coverage is still growing and many programs that depend on the full C standard library (regex, locale, wide-character I/O, etc.) will not yet compile against it: - **Static-linked Linux servers and containers** — used in production at Google (servers and Pixel Buds) on x86-64 and AArch64. - **GPU compute (AMDGPU, NVPTX)** — libc functions available in GPU kernels via LLVM's offloading runtime. {doc}`GPU docs ` - **Baremetal embedded (ARM, RISC-V, AArch64)** — minimal footprint builds for microcontrollers and custom hardware. - **UEFI applications** — experimental support for firmware development. {doc}`UEFI docs ` - **LLVM ecosystem internals** — libc++ and the offloading runtime consume LLVM-libc directly via {doc}`Hand-in-Hand `. - **Toolchain integrations** — pieces of LLVM-libc are used in Android Bionic, Fuchsia, Emscripten, and the ARM embedded toolchain. Coverage is still growing. See the {doc}`implementation status ` pages for per-header detail, and the {doc}`platform support ` page for OS/architecture coverage. ## Getting Started If you are new to LLVM-libc, {doc}`getting_started` is the right first stop. It covers cloning, building, testing, and verifying your installation in one place. Want to use LLVM-libc *alongside* your system libc instead of replacing it? See {doc}`overlay_mode`. ## Get Involved LLVM-libc is an active project and welcomes contributors of all experience levels. See {doc}`contributing` to learn how to help. - [Source code](https://github.com/llvm/llvm-project/tree/main/libc) - [Bug reports](https://github.com/llvm/llvm-project/labels/libc) - [Discourse](https://discourse.llvm.org/c/runtimes/libc) - [Discord](https://discord.com/channels/636084430946959380/636732994891284500) ([invite](https://discord.gg/xS7Z362)) - [Buildbot](https://lab.llvm.org/buildbot/#/builders?tags=libc) ```{toctree} :caption: Using LLVM-libc :hidden: true :maxdepth: 1 getting_started build_concepts overlay_mode full_host_build full_cross_build configure hand_in_hand ``` ```{toctree} :caption: Platforms :hidden: true :maxdepth: 1 gpu/index uefi/index ``` ```{toctree} :caption: Implementation Status :hidden: true :maxdepth: 1 headers/index arch_support platform_support compiler_support ``` ```{toctree} :caption: Development :hidden: true :maxdepth: 1 contributing build_and_test dev/index porting Maintainers ``` ```{toctree} :caption: Links :hidden: true :maxdepth: 1 talks Source Code Bug Reports Discourse Join the Discord Discord Channel Buildbot ``` --- ### Clang Docs CIR Index (`clang/docs/CIR/index.md`) # ClangIR :::{warning} The project of upstreaming ClangIR support from the incubator repository is still in progress, and ClangIR is not included in a default clang build. The documentation may be incomplete and out-of-date. ::: ClangIR is a high-level representation in Clang that reflects aspects of the C/C++ languages and their extensions. It is implemented using MLIR and occupies a position between Clang's AST and LLVM IR. ## ClangIR Design Documents ```{toctree} :maxdepth: 1 :numbered: ABILowering CleanupAndEHDesign CodeDuplication ``` --- ### Clang Docs CommandGuide Index (`clang/docs/CommandGuide/index.md`) # Clang "man" pages The following documents are command descriptions for all of the Clang tools. These pages describe how to use the Clang commands and what their options are. Note that these pages do not describe all of the options available for all tools. To get a complete listing, pass the `--help` (general options) or `--help-hidden` (general and debugging options) arguments to the tool you are interested in. ## Basic Commands ```{toctree} :maxdepth: 1 clang diagtool ``` --- ### Clang Docs ScalableStaticAnalysis Index (`clang/docs/ScalableStaticAnalysis/index.md`) # Scalable Static Analysis Framework This is a framework for writing cross-translation unit analyses in a scalable and extensible setting. ```{toctree} :glob: true :maxdepth: 2 :numbered: user-docs/* ``` {doc}`Developer documentation ` ```{toctree} :hidden: true developer-docs/index ``` --- ### Flang Docs CommandGuide Index (`flang/docs/CommandGuide/index.md`) # flang - the Flang Fortran compiler ## SYNOPSIS {program}`flang` \[*options*\] *filename ...* ## DESCRIPTION {program}`flang` is a Fortran compiler which supports all of the Fortran 95 and many newer language features. Flang supports OpenMP and has some support for OpenACC and CUDA. It encompasses preprocessing, parsing, optimization, code generation, assembly, and linking. Depending on the options passed in, Flang will perform only some, or all, of these actions. While Flang is highly integrated, it is important to understand the stages of compilation in order to understand how to invoke it. These stages are: Driver : The flang executable is actually a small driver that orchestrates the execution of other tools such as the compiler, assembler and linker. Typically you do not need to interact with the driver, but you transparently use it to run the other tools. Preprocessing : This stage performs tokenization of the input source file, macro expansion, #include expansion and handles other preprocessor directives. Parsing and Semantic Analysis : This stage parses the input file, translating preprocessor tokens into a parse tree. Once in the form of a parse tree, it applies semantic analysis to compute types for expressions and determine whether the code is well formed. Parse errors and most compiler warnings are generated by this stage. Code Generation and Optimization : This stage translates the parse tree into intermediate code (known as "LLVM IR") and, ultimately, machine code. It also optimizes this intermediate code and handles target-specific code generation. The output of this stage is typically a ".s" file, referred to as an "assembly" file. Flang also supports the use of an integrated assembler, in which the code generator produces object files directly. This avoids the overhead of generating the ".s" file and calling the target assembler explicitly. Assembler : This stage runs the target assembler to translate the output of the compiler into a target object file. The output of this stage is typically a ".o" file, referred to as an "object" file. Linker : This stage runs the target linker to merge multiple object files into an executable or dynamic library. The output of this stage is typically an "a.out", ".dylib" or ".so" file. ## OPTIONS :::{toctree} :maxdepth: 1 FlangCommandLineOptions ::: --- ### Flang Docs MeetingNotes README (`flang/docs/MeetingNotes/README.md`) # Combined Flang Community Call Notes ## About these notes This directory combines the notes from the Flang Technical and Community Calls. This was done to make the call notes more easily available than before. All references to Flang in these documents refer to the Fortran compiler being developed as part of the LLVM project and located in the LLVM monorepo ([github.com/llvm/llvm-project](http://github.com/llvm/llvm-project)). Information about "Classic Flang", a separate and independent Fortran compiler, can be found at [github.com/flang-compiler/flang](http://github.com/flang-compiler/flang). ## Call Information - Details about the call can be found in the [Getting Involved](https://flang.llvm.org/docs/GettingInvolved.html#flang-biweekly-call) page in flang's documentation. ## Flang Community * **Websites:** * **Flang:** [https://flang.llvm.org](https://flang.llvm.org) * **Issues:** * Issues for Flang should be reported in GitHub: [https://github.com/llvm/llvm-project/issues](https://github.com/llvm/llvm-project/issues). Please use the "flang" tag for flang-related issues. * **LLVM mailing lists and Discourse channels:** There are now two communication channels hosted by llvm.org: * The Flang Discourse channel (used primarily for discussions): [https://discourse.llvm.org/c/subprojects/flang/33](https://discourse.llvm.org/c/subprojects/flang/33) * The flang-commits mailing list (used primarily for routine notifications): subscribe at [https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-commits](https://lists.llvm.org/cgi-bin/mailman/listinfo/flang-commits) * **Slack:** The [flang-compiler.slack.com](http://flang-compiler.slack.com) workspace is used for informal real-time communication in the Flang community. See the [Getting Involved](https://flang.llvm.org/docs/GettingInvolved.html#flang-slack-workspace) page in flang's documentation for instructions on joining the workspace. * **Flang Development:** * On GitHub: The source code for Flang is available on [https://github.com/llvm/llvm-project/tree/main/flang](https://github.com/llvm/llvm-project/tree/main/flang). It is recommended to start in the "docs" directory. --- ### Libc Docs Dev Index (`libc/docs/dev/index.md`) (developer_guides)= # Developer Guides Navigate to the links below for information on the respective topics: ```{toctree} :maxdepth: 1 building_docs code_style source_tree_layout entrypoints implementing_a_function config_options date_and_time fuzzing header_generation implementation_standard undefined_behavior printf_behavior builtin_compatibility syscall_wrapper_refactor modular_format code_coverage ``` --- ### Clang Docs ScalableStaticAnalysis Developer Docs Index (`clang/docs/ScalableStaticAnalysis/developer-docs/index.md`) # Developer Documentation ```{toctree} :glob: true :maxdepth: 2 :numbered: * ``` --- ### Clang Tools Extra Docs Clang Tidy Checks Cppcoreguidelines Pro Bounds Constant Array Index (`clang-tools-extra/docs/clang-tidy/checks/cppcoreguidelines/pro-bounds-constant-array-index.md`) ```{title} clang-tidy - cppcoreguidelines-pro-bounds-constant-array-index ``` # cppcoreguidelines-pro-bounds-constant-array-index This check flags all array subscript expressions on static arrays and `std::arrays` that either do not have a constant integer expression index or are out of bounds (for `std::array`). For out-of-bounds checking of static arrays, see the `-Warray-bounds` Clang diagnostic. This rule is part of the [Bounds safety (Bounds 2)](https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#pro-bounds-arrayindex) profile from the C++ Core Guidelines. Optionally, this check can generate fixes using `gsl::at` for indexing. ## Options ```{option} GslHeader The check can generate fixes after this option has been set to the name of the include file that contains `gsl::at()`, e.g. `"gsl/gsl.h"`. Default is an empty string. ``` ```{option} IncludeStyle A string specifying which include-style is used, `llvm` or `google`. Default is `llvm`. ``` --- ### Clang Tools Extra Docs Clang Tidy Checks Readability Misplaced Array Index (`clang-tools-extra/docs/clang-tidy/checks/readability/misplaced-array-index.md`) ```{title} clang-tidy - readability-misplaced-array-index ``` # readability-misplaced-array-index This check warns for unusual array index syntax. The following code has unusual array index syntax: ```cpp void f(int *X, int Y) { Y[X] = 0; } ``` becomes ```cpp void f(int *X, int Y) { X[Y] = 0; } ``` The check warns about such unusual syntax for readability reasons: - There are programmers that are not familiar with this unusual syntax. - It is possible that variables are mixed up. --- ### Bolt Docs BAT (`bolt/docs/BAT.md`) # BOLT Address Translation (BAT) # Purpose A regular profile collection for BOLT involves collecting samples from unoptimized binary. BOLT Address Translation allows collecting profile from BOLT-optimized binary and using it for optimizing the input (pre-BOLT) binary. # Overview BOLT Address Translation is an extra section (`.note.bolt_bat`) inserted by BOLT into the output binary containing translation tables and split functions linkage information. This information enables mapping the profile back from optimized binary onto the original binary. # Usage `--enable-bat` flag controls the generation of BAT section. Sampled profile needs to be passed along with the optimized binary containing BAT section to `perf2bolt` which reads BAT section and produces profile for the original binary. # Internals ## Section contents The section is organized as follows: - Hot functions table - Address translation tables - Cold functions table ## Construction and parsing BAT section is created from `BoltAddressTranslation` class which captures address translation information provided by BOLT linker. It is then encoded as a note section in the output binary. During profile conversion when BAT-enabled binary is passed to perf2bolt, `BoltAddressTranslation` class is populated from BAT section. The class is then queried by `DataAggregator` during sample processing to reconstruct addresses/ offsets in the input binary. ## Encoding format The encoding is specified in [BoltAddressTranslation.h](https://github.com/llvm/llvm-project/blob/main/bolt/include/bolt/Profile/BoltAddressTranslation.h) and [BoltAddressTranslation.cpp](https://github.com/llvm/llvm-project/blob/main/bolt/lib/Profile/BoltAddressTranslation.cpp). ### Layout The general layout is as follows: ``` Hot functions table Cold functions table Functions table: |------------------| | Function entry | | | | Address | | translation | | table | | | | Secondary entry | | points | |------------------| ``` ### Functions table Hot and cold functions tables share the encoding except differences marked below. Header: | Entry | Encoding | Description | | ------ | ----- | ----------- | | `NumFuncs` | ULEB128 | Number of functions in the functions table | The header is followed by Functions table with `NumFuncs` entries. Output binary addresses are delta encoded, meaning that only the difference with the last previous output address is stored. Addresses implicitly start at zero. Output addresses are continuous through function start addresses and function internal offsets, and between hot and cold fragments, to better spread deltas and save space. Hot indices are delta encoded, implicitly starting at zero. | Entry | Encoding | Description | Hot/Cold | | ------ | ------| ----------- | ------ | | `Address` | Continuous, Delta, ULEB128 | Function address in the output binary | Both | | `HotIndex` | Delta, ULEB128 | Index of corresponding hot function in hot functions table | Cold | | `FuncHash` | 8b | Function hash for input function | Hot | | `NumBlocks` | ULEB128 | Number of basic blocks in the original function | Hot | | `NumSecEntryPoints` | ULEB128 | Number of secondary entry points in the original function | Hot | | `ColdInputSkew` | ULEB128 | Skew to apply to all input offsets | Cold | | `NumEntries` | ULEB128 | Number of address translation entries for a function | Both | | `EqualElems` | ULEB128 | Number of equal offsets in the beginning of a function | Both | | `BranchEntries` | Bitmask, `alignTo(EqualElems, 8)` bits | If `EqualElems` is non-zero, bitmask denoting entries with `BRANCHENTRY` bit | Both | Function header is followed by *Address Translation Table* with `NumEntries` total entries, and *Secondary Entry Points* table with `NumSecEntryPoints` entries (hot functions only). ### Address translation table Delta encoding means that only the difference with the previous corresponding entry is encoded. Input offsets implicitly start at zero. | Entry | Encoding | Description | Branch/BB | | ------ | ------| ----------- | ------ | | `OutputOffset` | Continuous, Delta, ULEB128 | Function offset in output binary | Both | | `InputOffset` | Optional, Delta, SLEB128 | Function offset in input binary with `BRANCHENTRY` LSB bit | Both | | `BBHash` | Optional, 8b | Basic block hash in input binary | BB | | `BBIdx` | Optional, Delta, ULEB128 | Basic block index in input binary | BB | The table omits the first `EqualElems` input offsets where the input offset equals output offset. `BRANCHENTRY` bit denotes whether a given offset pair is a control flow source (branch or call instruction). If not set, it signifies a control flow target (basic block offset). `InputAddr` is omitted for equal offsets in input and output function. In this case, `BRANCHENTRY` bits are encoded separately in a `BranchEntries` bitvector. Deleted basic blocks are emitted as having `OutputOffset` equal to the size of the function. They don't affect address translation and only participate in input basic block mapping. ### Secondary Entry Points table The table is emitted for hot fragments only. It contains `NumSecEntryPoints` offsets denoting secondary entry points, delta encoded, implicitly starting at zero. | Entry | Encoding | Description | | ----- | -------- | ----------- | | `SecEntryPoint` | Delta, ULEB128 | Secondary entry point offset | --- ### Bolt Docs BOLTAArch64OptimizationStatus (`bolt/docs/BOLTAArch64OptimizationStatus.md`) # AArch64 Optimization and Flags Status ## Overview This page summarizes default-off BOLT optimization flags that users may explicitly enable when optimizing AArch64 binaries. BOLT is to be used with binaries linked with relocations (`--emit-relocs` or `-Wl,-q`) and representative profile data. ## Main Code-Layout Optimizations The following code-layout optimizations are typically the first options to consider when optimizing AArch64 binaries with representative profile data. They typically provide the largest performance gains among BOLT optimizations. :::{list-table} :header-rows: 1 :widths: 34 42 :align: left * - Flag - Optimization * - `--reorder-functions=exec-count|hfsort|cdsort|pettis-hansen|random|user` `--function-order=` - Reorder functions * - `--reorder-blocks=normal|ext-tsp|cache|branch-predictor|reverse|cluster-shuffle` - Reorder basic blocks * - `--split-functions` `--split-strategy=profile2|random2|randomN|all` `--split-all-cold` `--split-eh` - Split hot and cold code ::: ## Other Supported Optimizations The following optimizations are also supported for AArch64. :::{list-table} :header-rows: 1 :widths: 34 42 :align: left * - Flag - Optimization * - `--align-blocks` `--block-alignment=` - Align basic blocks * - `--tail-duplication=aggressive|moderate|cache` - Duplicate branch tails * - `--peepholes=double-jumps|tailcall-traps|useless-branches|all` - Run peephole optimizations * - `--inline-all` `--inline-small-functions` Related options: `--inline-ap` `--inline-limit=` `--inline-small-functions-bytes=` - Inline functions * - `--icf=safe|all` - Fold identical functions ::: ## Supported Flags With Limitations The following flags are implemented for AArch64, but require specific runtime or option conditions. Enabling them without the required conditions may report an error or perform no transformation. :::{list-table} :header-rows: 1 :widths: 30 28 44 :align: left * - Flag - Optimization - Notes * - `--inline-memcpy` - Inline fixed-size `memcpy` calls - Only applies when the copy size is a known constant; AArch64 skips sizes over 64 bytes. * - `--plt=hot|all` - Optimize PLT calls - Requires immediate binding. If BOLT cannot update the binary, relink with `-znow`. * - `--hugify` - Place hot code on huge pages - Applies to binaries with a recognized entry point; skipped when `--instrument` is used. * - `--reorder-data=` `--reorder-data-algo=count|funcs` - Reorder data sections - `move`, `split` and `aggressive` disable data reordering. * - `--split-strategy=cdsplit` - Split functions using cache-directed splitting - Requires `--compact-code-model` on AArch64. ::: ## Unsupported Flags The following flags are not available for AArch64. `Not applicable to AArch64` means the optimization targets architectural features or mechanisms that do not apply to AArch64. `Not implemented for AArch64` means the optimization could be relevant, but is not currently implemented for this target. :::{list-table} :header-rows: 1 :widths: 30 28 42 :align: left * - Flag - Optimization - Notes * - `--jt-footprint-reduction` - Reduce jump-table footprint - Not implemented for AArch64. * - `--three-way-branch` - Reorder three-way branches - Not implemented for AArch64. * - `--simplify-rodata-loads` - Replace read-only data loads with constants - Not implemented for AArch64. * - `--frame-opt=hot|all` - Optimize stack-frame accesses - Not implemented for AArch64. * - `--indirect-call-promotion=calls|jump-tables|all` - Promote indirect calls - Not implemented for AArch64. * - `--memcpy1-spec=` - Specialize one-byte `memcpy` calls - Not implemented for AArch64. * - `--reg-reassign` - Reassign registers to reduce encoding size - Not applicable to AArch64. * - `--cmov-conversion` - Convert branches to conditional moves - Not applicable to AArch64. * - `--stoke` `--stoke-out` - Emit STOKE optimization data - Not applicable to AArch64. * - `--insert-retpolines` - Insert retpolines - Not applicable to AArch64. ::: --- ### Bolt Docs BinaryAnalysis (`bolt/docs/BinaryAnalysis.md`) # BOLT-based binary analysis As part of post-link-time optimization, BOLT needs to perform a range of analyses on binaries such as reconstructing control flow graphs, and more. The `llvm-bolt-binary-analysis` tool enables running requested analyses on binaries, and generating reports. It does this by building on top of the analyses implemented in the BOLT libraries. For now, the only analysis implemented is validation of Pointer Authentication hardening on AArch64, more types of analyses can be added later. ## Contents 1. [Background and motivation](#background-and-motivation) 2. [Usage](#usage) 3. [Pointer Authentication validator](#pointer-authentication-validator) 4. [How to add your own analysis](#how-to-add-your-own-analysis) ## Background and motivation ### Security scanners For the past 25 years, a large numbers of exploits have been built and used in the wild to undermine computer security. The majority of these exploits abuse memory vulnerabilities in programs, see evidence from [Microsoft](https://youtu.be/PjbGojjnBZQ?t=836), [Chromium](https://www.chromium.org/Home/chromium-security/memory-safety/) and [Android](https://security.googleblog.com/2021/01/data-driven-security-hardening-in.html). It is not surprising therefore, that a large number of mitigations have been added to instruction sets and toolchains to make it harder to build an exploit using a memory vulnerability. Examples are: stack canaries, stack clash, pac-ret, shadow stacks, arm64e, and many more. These mitigations guarantee a so-called "security property" on the binaries they produce. For example, for stack canaries, the security property is roughly that a canary is located on the stack between the set of saved registers and the set of local variables. For pac-ret, it is roughly that either the return address is never stored/retrieved to/from memory; or, there are no writes to the register containing the return address between an instruction authenticating it and a return instruction using it. From time to time, however, a bug gets found in the implementation of such mitigations in toolchains. Also, code that is manually written in assembly requires the developer to ensure these security properties by hand. In short, it is sometimes found that a few places in the binary code are not protected as well as expected given the requested mitigations. Attackers could make use of those places (sometimes called gadgets) to circumvent the protection that the mitigation should give. One of the reasons that such gadgets, or holes in the mitigation implementation, exist is that typically the amount of testing and verification for these security properties is limited to checking results on specific examples. In comparison, for testing functional correctness, or for testing performance, toolchain and software in general typically get tested with large test suites and benchmarks. In contrast, this typically does not get done for testing the security properties of binary code. Unlike functional correctness where compilation errors result in test failures, and performance where speed and size differences are measurable, broken security properties cannot be easily observed using existing testing and benchmarking tools. The security scanners implemented in `llvm-bolt-binary-analysis` aim to enable the testing of security hardening in arbitrary programs and not just specific examples. ### Pointer Authentication [Pointer Authentication](https://clang.llvm.org/docs/PointerAuthentication.html) is intended to make it harder for an attacker to replace pointers at run time. This is achieved by making it possible for the compiler or the programmer to produce a *signed* pointer from a raw one, and then to probabilistically *authenticate the signature* at another site in the program. On AArch64 this is achieved by injecting a cryptographic hash, called a ["Pointer Authentication Code" (PAC)](https://llsoftsec.github.io/llsoftsecbook/#sec:pointer-authentication), to the upper bits of the pointer. While this approach can be applied to any pointers in the program, the most frequent use case, at least in C and C++, is protecting the code pointers. The language rules for such pointers are more restrictive, thus allowing the compiler to implement various hardenings transparently to the programmer. Probably the most simple variant of hardening based on Pointer Authentication is [`pac-ret`](https://llsoftsec.github.io/llsoftsecbook/#sec:pac-ret), a security hardening scheme implemented in compilers such as GCC and Clang, which can be enabled with a command line option like `-mbranch-protection=pac-ret`. On AArch64, `pac-ret` hardening is enabled by default on most widely used Linux distributions. The hardening scheme mitigates [Return-Oriented Programming (ROP)](https://llsoftsec.github.io/llsoftsecbook/#return-oriented-programming) attacks by making sure that return addresses are only ever stored to memory in a signed form. This makes it substantially harder for attackers to divert control flow by overwriting a return address with a different value. It is possible to link object files with different pac-ret hardening modes together, as each particular function is responsible for signing the LR value in the prologue and authenticating it in the epilogue. Other hardening variants exist that break ABI compatibility, see the description of [PAuth ABI Extensions](https://github.com/ARM-software/abi-aa/blob/main/pauthabielf64/pauthabielf64.rst) for details. While pac-ret hardens (signs and authenticates) return addresses, PAuth ABI may harden most other code pointers as well, such as function pointers and labels used as the destination of computed goto. The approach to validation of Pointer Authentication hardening implemented in `llvm-bolt-binary-analysis` is tracking register properties using dataflow analysis. At each program point it is computed whether the particular register can be controlled and whether it can be inspected by an attacker under the [Pointer Authentication threat model](https://clang.llvm.org/docs/PointerAuthentication.html#theory-of-operation). Then, for a number of sensitive instruction kinds (such as function calls and pointer signing instructions), the properties of input or output operands are inspected to check if the particular instruction is emitted in a safe manner. As an example, for a return instruction this usually means that either the link register (`x30`, which is also referred to as LR on AArch64) was never clobbered in the function, or that it was authenticated at some point and never written to since then: ``` foo: ; x30 is assumed to be trusted on function entry cbnz x0, .L1 ; For every possible execution path leading to this point, x30 was never ; clobbered and is thus safe to be used by `ret`. ret .L1: paciasp stp x29, x30, [sp, #-16]! mov x29, sp bl bar ldp x29, x30, [sp], #16 autiasp ; At this point, x30 was never written to since `autiasp`. ret ``` The real rules being used by `llvm-bolt-binary-analysis` are somewhat more complex, see the below sections for the detailed explanation. ## Usage ``` llvm-bolt-binary-analysis --scanners= [options] ``` The `--scanners=` option accepts a comma-separated list of analyses to run on the provided binary. The binary to be analyzed can be either an ELF executable or a shared object. Similar to other BOLT tools, `llvm-bolt-binary-analysis` expects `binary` to be unstripped and preferably linked with `--emit-relocs` linker option. In addition to options printed by `llvm-bolt-binary-analysis --help-hidden`, other relevant BOLT options can generally be passed, see `llvm-bolt --help-hidden`. Incomplete help message is a known issue and is tracked in [#176969](https://github.com/llvm/llvm-project/issues/176969). The only analysis which is currently implemented is validation of a number of Pointer Authentication-based hardening schemes such as pac-ret and PAuth ABI. The specific set of gadget kinds which are searched for depends on command line options. Each gadget found by PtrAuth gadget scanner results in a plain text report printed at the end of the analysis. Furthermore, an attempt is made to provide additional information on the instructions that made the register unsafe. Please note that this extra information is provided on a best-effort basis and is not expected to be as accurate as the reports themselves. Here is an example of the report: ``` GS-PAUTH: signing oracle found in function function_name, basic block .LBB08, at address 102b8 The instruction is 000102b8: pacda x0, x1 The 1 instructions that write to the affected registers after any authentication are: 1. 000102b4: ldr x0, [x1] This happens in the following basic block: 000102b4: ldr x0, [x1] 000102b8: pacda x0, x1 000102bc: ret ``` A similar report without the associated extra information looks like this: ``` GS-PAUTH: signing oracle found in function function_name, basic block .LBB016, at address 10384 The instruction is 00010384: pacda x0, x1 The 0 instructions that write to the affected registers after any authentication are: ``` ## Pointer Authentication validator Pointer Authentication analysis is able to search for a number of gadget kinds, with the specific set depending on command line options: * [`ptrauth-pac-ret`](#return-address-protection-ptrauth-pac-ret) - non-protected return instruction * [`ptrauth-tail-calls`](#return-address-protection-before-tail-call-ptrauth-tail-calls) - performing a tail call with an untrusted value in the link register * [`ptrauth-forward-cf`](#ptrauth-forward-cf) - non-protected destination of branch or call instruction * [`ptrauth-sign-oracles`](#signing-oracles-ptrauth-sign-oracles) - signing of untrusted value (signing oracle) * [`ptrauth-auth-oracles`](#authentication-oracles-ptrauth-auth-oracles) - revealing the result of authentication without crashing the program on failure (authentication oracle) Validation is performed by `llvm-bolt-binary-analysis` on a per-function basis. First, the register properties are computed by analyzing the function as a whole. Then, the instructions are considered in isolation. For each kind of gadget, the set of susceptible instructions is computed. The properties of input or output registers of each such instruction are analyzed and reports are produced for each detected gadget. Each gadget kind that is searched for can be characterized by the combination of * the set of instructions to analyze * the properties of input or output operands to check Currently, three properties can be computed for each register at any given program point: * **"trusted"** - the register is known not to be attacker-controlled, either because it successfully passed authentication or because its value was materialized using an instruction sequence that an attacker cannot tamper with * **"safe-to-dereference"** (sometimes referred to as "s-t-d" below) - a weaker property that the register can be controlled by an attacker to some extent, but any memory access using a value crafted by an attacker is known to result in an access to an unmapped memory ("segmentation fault"). This makes it possible for authentication instructions to return an invalid address on failure as long as it is known to crash the program on accessing memory, but may require extra care to be taken when implementing operations like re-signing a pointer with a different signing schema without accessing that address in-between. If any failed authentication instruction is guaranteed to terminate the program abnormally, then "safe-to-dereference" and "trusted" properties are equivalent. * **"cannot escape unchecked"** - at every possible execution path after this point, it is known to be impossible for an attacker to determine that the value is a result of a failed authentication operation (for example, the register is zeroed, or its value is checked to be valid, so that failure results in immediate abnormal program termination). Generally, BOLT strives to reconstruct the control-flow graph of each function, which is important for dataflow analysis. However, when BOLT fails to recognize some control flow in the particular function, that function ends up being represented as a flat list of instructions - in such cases `llvm-bolt-binary-analysis` computes register properties using a fallback analysis implementation, which is less precise. The below sub-sections describe the particular detectors. Please note that while the descriptions refer to AArch64 for simplicity, the implementation of gadget detectors in `llvm-bolt-binary-analysis` attempts to be target-neutral by isolating AArch64 specifics in target-dependent hooks. ### Return address protection (`ptrauth-pac-ret`) **Instructions:** Return instructions without built-in authentication: either `ret` (implicit `x30` register) or `ret `, but not `retaa` and similar instructions. **Property:** The register holding the return address must be safe-to-dereference. **Notes:** Cross-exception-level return instructions (`eret`) are not analyzed yet. A report is generated for a return instruction whose destination is possibly attacker-controlled. **Examples:** ```asm authenticated_return: pacibsp ; ... ; ... some code here ... ; ... retab ; Built-in authentication, thus out of scope. good_leaf_function: ; x30 is implicitly safe-to-dereference (s-t-d) and trusted at function entry. mov x0, #42 ; x30 was not written to by this function, thus remains s-t-d. ret good_non_leaf_function: pacibsp ; Spilling signed return address. stp x29, x30, [sp, #-16]! mov x29, sp bl callee ; Re-loading signed return address. ; LDP writes to x30 and thus resets it to neither s-t-d nor trusted state. ldp x29, x30, [sp], #16 ; Checking that signature is valid. ; AUTIBSP sets "s-t-d" property of x30, but not "trusted" (unless FEAT_FPAC ; is known to be implemented). autibsp ; x30 is s-t-d at this point. ret bad_spill: ; x30 is implicitly s-t-d at function entry. stp x29, x30, [sp, #-16]! mov x29, sp bl callee ; Spilled x30 may have been overwritten on stack. ; Writing to x30 resets its s-t-d property. ldp x29, x30, [sp], #16 ; x30 is unsafe by the time it is used by ret, thus generating a report. ret bad_clobber: pacibsp ; ... ; ... some code here ... ; ... autibsp mov x30, x1 ; The value in LR is unsafe, even though there was autibsp above. ret ``` ### Return address protection before tail call (`ptrauth-tail-calls`) **Instructions:** Branch instructions (both direct and indirect, regular or with built-in authentication), classified as tail calls either by BOLT or by PtrAuth gadget scanner's heuristic. **Property:** link register (`x30` on AArch64) must be trusted. **Notes:** Heuristics are involved to classify instructions either as a tail call or as another kind of branch (such as jump table or computed goto). A report is generated if a tail call is performed with an untrusted link register. This basically means that the tail-called function would have the link register untrusted on its entry (unlike the inherently correct address placed in the link register by one of the `bl*` instructions when a non-tail call is performed). ```asm non_protected_tail_call: stp x29, x30, [sp, #-16]! mov x29, sp bl callee ldp x29, x30, [sp], #16 ; x30 is neither trusted nor safe-to-dereference at this point. b tail_callee non_checked_tail_call: pacibsp stp x29, x30, [sp, #-16]! mov x29, sp bl callee ldp x29, x30, [sp], #16 autibsp ; x30 is safe-to-dereference, but not fully trusted at this point. b tail_callee tail_callee: pacibsp ; ... ``` Even though `x30` is likely to be safe-to-dereference before exit from a function (whether via return or tail call) in a consistently pac-ret-protected program, with respect to this gadget kind it further must be fully "trusted". This corresponds to the way tail calls are emitted when `--aarch64-authenticated-lr-check-method=` option is specified with an argument other than `none`, or when such checks are requested for the particular subtarget by default. Properly mitigating this issue would usually require inserting an explicit check after a regular authentication instruction, which may be either too expensive (if a fully-generic XPAC-based sequence is being used) on one hand, or not required at all (if `FEAT_FPAC` is known to be implemented) on the other hand. (ptrauth-forward-cf)= ### Indirect branch / call target protection (`ptrauth-forward-cf`) **Instructions:** Indirect call and branch instructions without built-in authentication: either `blr ` or `br `, but not `blraa`, `braa` and similar instructions. **Property:** The call or branch target register must be safe-to-dereference. A report is generated for an indirect branch or call instruction whose destination is possibly attacker-controlled. **Examples:** ```asm direct_call: ; ... bl callee ; Direct call, thus out of scope. ; ... authenticated_call: ; ... ldr x2, [x1] blraa x2, x1 ; Built-in authentication, thus out of scope. ; ... good_call: ; ... ldr x2, [x1] autia x2, x1 blr x2 ; ... bad_call: ; ... ldr x2, [x1] autia x2, x1 ; Store unprotected address. str x2, [x3] ; ... ; The callee address may have been overwritten in memory. ldr x2, [x3] blr x2 ; ... good_call_dataflow: cbz x0, .L1 ldr x2, [x1] autia x2, x1 b .L2 .L1: adrp x2, callee add x2, x2, :lo12:callee .L2: ; Dataflow analysis can deduce that x2 is s-t-d on any possible execution ; path leading to the below "br x2" instruction. br x2 bad_call_dataflow: cbz x0, .L3 adrp x2, callee add x2, x2, :lo12:callee .L3: ; x2 is untrusted if x0 is 0. br x2 ``` ### Signing oracles (`ptrauth-sign-oracles`) **Instructions:** Address-signing instructions. **Property:** The address being signed must be trusted. Reports signing of untrusted values, as this could make arbitrary and possibly attacker-controlled values indistinguishable from perfectly trusted and protected ones. Note that in absence of `--auth-traps-on-failure` command line option, this detector reports auth+sign pairs unless an explicit check is emitted to make sure the authentication operation succeeded. This corresponds to the way LLVM emits instructions on AArch64. **Examples:** ```asm good_sign_constant: ; ... adrp x0, sym add x0, x0, :lo12:sym pacda x0, x1 ; ... good_resign: ; ... autda x0, x1 ; x0 is s-t-d here. ldr x2, [x0] ; If we got here without crashing on the above ldr, x0 is fully trusted. pacdb x0, x1 ; ... bad_resign_if_not_fpac: ; ... autda x0, x1 ; x0 is only s-t-d, but not trusted here, unless autda raises an error on failure. pacdb x0, x1 ; ... very_bad_function: pacda x0, x1 ret ``` ### Authentication oracles (`ptrauth-auth-oracles`) **Instructions:** Standalone authentication instructions: `autda`, `autdb`, etc. (i.e. not those built into corresponding memory-accessing instructions, such as `ldraa` or `blraa`). **Property:** The **result** of authentication must be written to a register that cannot escape unchecked. Reports authentication instructions, whose result (success or failure) can be observed by the attacker and used to guess the correct PAC field by trial-and-error. The authentication oracles searched for by this detector are impossible if all authentication instructions are known to generate an irrecoverable error on failure. On AArch64 this is the case if CPU is known to implement `FEAT_FPAC` (though recoverability depends on the OS: for example, on Linux such errors result in signals that can be handled if configured accordingly). This check is disabled by `--auth-traps-on-failure` command line option. **Examples:** ```asm ; The descriptions assume FEAT_FPAC is not implemented. good_auth_call: paciasp stp x29, x30, [sp, #-16]! mov x29, sp ; The result of authentication is inevitably dereferenced by blr. ; If autia returns a non-canonical address due to incorrect signature, ; the program crashes when the resulting address is jumped-to by blr. cbz x2, .L1 autia x0, x1 blr x0 .L1: ldp x29, x30, [sp], #16 autiasp ret bad_auth_call: paciasp stp x29, x30, [sp, #-16]! mov x29, sp ; x0 may be observed by the caller of bad_auth_call if x2 is 0. autia x0, x1 cbz x2, .L2 blr x0 .L2: ldp x29, x30, [sp], #16 autiasp ret bad_leaks_to_callee: paciasp stp x29, x30, [sp, #-16]! mov x29, sp ldr x20, [x0] autda x20, x0 ; The result of authentication is leaked to the called function. bl callee ; The below ldr instruction would properly check x20 if placed above the call. ldr x0, [x20] ldp x29, x30, [sp], #16 autiasp ret ``` ### Known issues and missing features #### Control-flow graph availability The analysis quality is degraded if BOLT is unable to reconstruct control-flow graph of the function correctly. When reconstructing CFG for a particular function, it is possible that BOLT finds a code pattern that it is unable to handle. In that case PtrAuth validator processes a function containing a flat list of instructions (as opposed to interlinked basic blocks) and the reports (if any) are printed without the `, basic block ` part in the first line. Furthermore, it is possible that CFG information is returned by BOLT for a function, even though the graph is imprecise. Inaccurate CFG information may result in false positives and false negatives, thus PtrAuth gadget scanner produces warning messages (at most once per function) for non-entry basic blocks without any predecessors in CFG: while unreachable basic blocks are technically correct, truly unreachable blocks are unlikely to exist in an optimized code. Known gadget scanner issues related to CFG reconstruction are tracked in [#177761](https://github.com/llvm/llvm-project/issues/177761), [#178058](https://github.com/llvm/llvm-project/issues/178058), [#178232](https://github.com/llvm/llvm-project/issues/178232). #### Last writing instructions Some of the reports contain extra information along these lines ``` The 1 instructions that write to the affected registers after any authentication are: 1. 000102b4: ldr x0, [x1] This happens in the following basic block: 000102b4: ldr x0, [x1] 000102b8: pacda x0, x1 000102bc: ret ``` This information is provided on a best-effort basis and is less reliable than gadget reports themselves. #### Feature: scan for unsafe computation of discriminators On AArch64, signing and authentication operations are parameterized by the pair of the key identifier and a 64-bit *discriminator* value. The key is one of `IA`, `IB`, `DA`, or `DB`, and its identifier is directly encoded into the instruction (the `GA` key differs significantly and is thus out of scope here). The discriminator, on the other hand, is computed at run-time and is intended to be [derived independently](https://clang.llvm.org/docs/PointerAuthentication.html#discriminators) at the signing and authentication sites. There is a common pattern on AArch64 to compute the discriminator as a blend of an address and an integer modifier by inserting a compile-time constant value into the top 16 bits of the storage address. It is not necessarily possible to prevent an attacker from modifying the storage address, but the insertion of 16-bit constant modifier can always be performed immediately before the discriminator value is used by signing or authentication instruction. While not as bad as signing an arbitrary untrusted pointer, spilling a "ready-to-use" discriminator and then reloading a potentially modified value later is something we would rather avoid. On the other hand, using an arbitrary value as the discriminator should probably be allowed, making it hard to distinguish the below patterns: ```asm ; Valid and not reported. good_store_with_address_and_constant_diversity: mov x16, x1 movk x16, #1234, lsl #48 pacda x0, x16 str x0, [x1] ret ; Valid and not reported. good_store_with_address_diversity: pacda x0, x1 str x0, [x1] ret ; Spilled discriminator. Not critically wrong, but could rather be avoided. ; Not reported, but probably should (false negative). bad_spilling: mov x16, x1 movk x16, #1234, lsl #48 ; Spilling the discriminator to memory. str x16, [x2] ; Reloaded value could have been modified by an attacker. ldr x16, [x2] pacda x0, x16 str x0, [x1] ret ; Not reported (and should probably not). better_spilling: mov x16, x1 str x16, [x2] ; Reloaded value could have been modified by an attacker. ldr x16, [x2] movk x16, #1234, lsl #48 pacda x0, x16 str x0, [x1] ret ``` #### Handling of constants While (PC-relative) address constants are tracked as "trusted" register state by `SrcSafetyAnalysis`, constant values are not generally accounted for. This results in false-positives like reporting ```asm ; Let assume FEAT_FPAC is implemented. autda x16, x22 mov x17, #0x128 add x16, x16, x17 pacda x16, x22 ``` as a signing oracle, even though ```asm ; Let assume FEAT_FPAC is implemented. autda x16, x22 add x16, x16, #8 pacda x16, x22 ``` is not reported, because `add Xdst, Xsrc, #imm` is recognized as a safe address computation. As an example of false-negative, it is possible that an instruction like `add x0, x0, #1` could be called in a loop with an attacker-controlled number of iterations, making it technically possible for an attacker to replace a valid pointer with a pointer to an arbitrary address. #### Tail call detection `ptrauth-tail-calls` detector uses a heuristic to guess which branch instructions (both direct and indirect) correspond to performing tail calls, as opposed to other kinds of control flow. Most other parts of BOLT should not break code when rewriting. Unlike them, this analyzer tries to keep reasonable balance between false positive reports and missed issues. For this reason, it inspects some other branch instructions in addition to those BOLT is sure about. While it should provide reasonable balance between false positives and false negatives on general code, this may not be the case for some specific patterns. An example of a generally uncommon code pattern that is likely to yield false positives is [labels-as-values GCC extension](https://gcc.gnu.org/onlinedocs/gcc/Labels-as-Values.html) which is also supported by Clang. #### Other known issues * No lightweight variant of [`ptrauth-tail-calls`](#return-address-protection-before-tail-call-ptrauth-tail-calls), see issue [#186204](https://github.com/llvm/llvm-project/issues/186204) for the details. * Not handling "no-return" functions. See issue [#115154](https://github.com/llvm/llvm-project/issues/115154) for details and pointers to open PRs to fix this. * Scanning of binaries compiled by Clang at `-Oz` optimization level produces a lot of reports due to outlining. Many such reports could probably be considered false negatives as long as an attacker is unable to call `OUTLINED_FUNCTION`s as ROP gadgets in the first place. * False positives are possible due to multi-instruction pointer-checking sequences not being detected without CFG. * While obviously "checking" the result of pointer authentication, store instructions do not transition their address operand register from safe-to- dereference to trusted state yet. This does not affect scanning regular code hardened neither by `pac-ret`, nor by `arm64e` or `pauthtest`. ## How to add your own analysis ### Pointer Authentication validator To implement the detection of a new gadget kind, add new `shouldReport*Gadget` function to `bolt/lib/Passes/PAuthGadgetScanner.cpp` and call it either from `FunctionAnalysisContext::findUnsafeUses` or `FunctionAnalysisContext::findUnsafeDefs`. To improve overall analysis quality by better computing register properties, either modify one of `*SafetyAnalysis` classes in `PAuthGadgetScanner.cpp` (if the improvement is target-neutral), or one of target-specific hooks in the subclass of `MCPlusBuilder` corresponding to your target (if an analysis of target-specific instruction patterns is to be improved). To add support for a new target, if one eventually implements similar pointer protection technique, implement PtrAuth-related hooks in the subclass of `MCPlusBuilder` corresponding to your target. Ideally, no changes should be needed in `PAuthGadgetScanner.cpp`, as it is intended to be reasonably target- independent, though it is possible that some amount of further generalization may be required. ### New types of analyses _TODO: this section needs to be written. Ideally, we should have a simple "example" or "template" analysis that can be the starting point for implementing custom analyses_ --- ### Bolt Docs CMakeLists (`bolt/docs/CMakeLists.txt`) if (DOXYGEN_FOUND) if (LLVM_ENABLE_DOXYGEN) set(abs_top_srcdir ${CMAKE_CURRENT_SOURCE_DIR}) set(abs_top_builddir ${CMAKE_CURRENT_BINARY_DIR}) if (HAVE_DOT) set(DOT ${LLVM_PATH_DOT}) endif() if (LLVM_DOXYGEN_EXTERNAL_SEARCH) set(enable_searchengine "YES") set(searchengine_url "${LLVM_DOXYGEN_SEARCHENGINE_URL}") set(enable_server_based_search "YES") set(enable_external_search "YES") set(extra_search_mappings "${LLVM_DOXYGEN_SEARCH_MAPPINGS}") else() set(enable_searchengine "NO") set(searchengine_url "") set(enable_server_based_search "NO") set(enable_external_search "NO") set(extra_search_mappings "") endif() # If asked, configure doxygen for the creation of a Qt Compressed Help file. if (LLVM_ENABLE_DOXYGEN_QT_HELP) set(BOLT_DOXYGEN_QCH_FILENAME "org.llvm.bolt.qch" CACHE STRING "Filename of the Qt Compressed help file") set(BOLT_DOXYGEN_QHP_NAMESPACE "org.llvm.bolt" CACHE STRING "Namespace under which the intermediate Qt Help Project file lives") set(BOLT_DOXYGEN_QHP_CUST_FILTER_NAME "Clang ${BOLT_VERSION}" CACHE STRING "See http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom-filters") set(BOLT_DOXYGEN_QHP_CUST_FILTER_ATTRS "Clang,${BOLT_VERSION}" CACHE STRING "See http://qt-project.org/doc/qt-4.8/qthelpproject.html#filter-attributes") set(bolt_doxygen_generate_qhp "YES") set(bolt_doxygen_qch_filename "${BOLT_DOXYGEN_QCH_FILENAME}") set(bolt_doxygen_qhp_namespace "${BOLT_DOXYGEN_QHP_NAMESPACE}") set(bolt_doxygen_qhelpgenerator_path "${LLVM_DOXYGEN_QHELPGENERATOR_PATH}") set(bolt_doxygen_qhp_cust_filter_name "${BOLT_DOXYGEN_QHP_CUST_FILTER_NAME}") set(bolt_doxygen_qhp_cust_filter_attrs "${BOLT_DOXYGEN_QHP_CUST_FILTER_ATTRS}") else() set(bolt_doxygen_generate_qhp "NO") set(bolt_doxygen_qch_filename "") set(bolt_doxygen_qhp_namespace "") set(bolt_doxygen_qhelpgenerator_path "") set(bolt_doxygen_qhp_cust_filter_name "") set(bolt_doxygen_qhp_cust_filter_attrs "") endif() llvm_configure_doxygen() set(abs_top_srcdir) set(abs_top_builddir) set(DOT) set(enable_searchengine) set(searchengine_url) set(enable_server_based_search) set(enable_external_search) set(extra_search_mappings) set(bolt_doxygen_generate_qhp) set(bolt_doxygen_qch_filename) set(bolt_doxygen_qhp_namespace) set(bolt_doxygen_qhelpgenerator_path) set(bolt_doxygen_qhp_cust_filter_name) set(bolt_doxygen_qhp_cust_filter_attrs) add_custom_target(doxygen-bolt COMMAND ${DOXYGEN_EXECUTABLE} ${CMAKE_CURRENT_BINARY_DIR}/doxygen.cfg WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} COMMENT "Generating bolt doxygen documentation." VERBATIM) set_target_properties(doxygen-bolt PROPERTIES FOLDER "BOLT/Docs") if (LLVM_BUILD_DOCS) add_dependencies(doxygen doxygen-bolt) endif() if (NOT LLVM_INSTALL_TOOLCHAIN_ONLY AND LLVM_BUILD_DOCS) install(DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/doxygen/html DESTINATION docs/html) endif() endif() endif() if (LLVM_ENABLE_SPHINX) include(AddSphinxTarget) if (SPHINX_FOUND) if (${SPHINX_OUTPUT_HTML}) add_sphinx_target(html bolt) endif() endif() endif() --- ### Bolt Docs CommandLineArgumentReference (`bolt/docs/CommandLineArgumentReference.md`) # BOLT - a post-link optimizer developed to speed up large applications ## SYNOPSIS `llvm-bolt [-o outputfile] .bolt [-data=perf.fdata] [options]` ## OPTIONS ### Generic options: - `-h` Alias for --help - `--help` Display available options (--help-hidden for more) - `--help-hidden` Display all available options - `--help-list` Display list of available options (--help-list-hidden for more) - `--help-list-hidden` Display list of all available options - `--version` Display the version of this program ### Output options: - `--bolt-info` Write bolt info section in the output binary - `-o ` output file - `-w ` Save recorded profile to a file ### BOLT generic options: - `--align-text=` Alignment of .text section - `--allow-stripped` Allow processing of stripped binaries - `--alt-inst-feature-size=` Size of feature field in .altinstructions - `--alt-inst-has-padlen` Specify that .altinstructions has padlen field - `--asm-dump[=]` Dump function into assembly - `-b` Alias for -data - `--bolt-id=` Add any string to tag this execution in the output binary via bolt info section - `--break-funcs=` List of functions to core dump on (debugging) - `--check-encoding` Perform verification of LLVM instruction encoding/decoding. Every instruction in the input is decoded and re-encoded. If the resulting bytes do not match the input, a warning message is printed. - `--comp-dir-override=` Overrides DW_AT_comp_dir, and provides an alternative base location, which is used with DW_AT_dwo_name to construct a path to *.dwo files. - `--create-debug-names-section` Creates .debug_names section, if the input binary doesn't have it already, for DWARF5 CU/TUs. - `--cu-processing-batch-size=` Specifies the size of batches for processing CUs. Higher number has better performance, but more memory usage. Default value is 1. - `--data=` data file - `--data2=` data file - `--debug-skeleton-cu` Prints out offsets for abbrev and debug_info of Skeleton CUs that get patched. - `--debug-thread-count=` Specifies the number of threads to be used when processing DWO debug information. - `--dot-tooltip-code` Add basic block instructions as tool tips on nodes - `--dump-alt-instructions` Dump Linux alternative instructions info - `--dump-cg=` Dump callgraph to the given file - `--dump-data` Dump parsed bolt data for debugging - `--dump-dot-all` Dump function CFGs to graphviz format after each stage;enable '-print-loops' for color-coded blocks - `--dump-dot-func=` Dump function CFGs to graphviz format for specified functions only; takes function name patterns (regex supported). Note: C++ function names must be passed using their mangled names - `--dump-linux-exceptions` Dump Linux kernel exception table - `--dump-orc` Dump raw ORC unwind information (sorted) - `--dump-para-sites` Dump Linux kernel paravitual patch sites - `--dump-pci-fixups` Dump Linux kernel PCI fixup table - `--dump-smp-locks` Dump Linux kernel SMP locks - `--dump-static-calls` Dump Linux kernel static calls - `--dump-static-keys` Dump Linux kernel static keys jump table - `--dwarf-output-path=` Path to where .dwo files or dwp file will be written out to. - `--dwp=` Path and name to DWP file. - `--dyno-stats` Print execution info based on profile - `--dyno-stats-all` Print dyno stats after each stage - `--dyno-stats-scale=` Scale to be applied while reporting dyno stats - `--enable-bat` Write BOLT Address Translation tables - `--fix-branches-with-liveness` Use liveness analysis to decide whether AArch64 branch inversion must preserve condition flags - `--force-data-relocations` Force relocations to data sections to always be processed - `--force-patch` Force patching of original entry points - `--funcs=` Limit optimizations to functions from the list - `--funcs-file=` File with list of functions to optimize - `--funcs-file-no-regex=` File with list of functions to optimize (non-regex) - `--funcs-no-regex=` Limit optimizations to functions from the list (non-regex) - `--hot-data` Hot data symbols support (relocation mode) - `--hot-functions-at-end` If reorder-functions is used, order functions putting hottest last - `--hot-text` Generate hot text symbols. Apply this option to a precompiled binary that manually calls into hugify, such that at runtime hugify call will put hot code into 2M pages. This requires relocation. - `--hot-text-move-sections=` List of sections containing functions used for hugifying hot text. BOLT makes sure these functions are not placed on the same page as the hot text. (default='.stub,.mover'). - `--insert-retpolines` Run retpoline insertion pass - `--keep-aranges` Keep or generate .debug_aranges section if .gdb_index is written - `--keep-tmp` Preserve intermediate .o file - `--lite` Skip processing of cold functions - `--log-file=` Redirect journaling to a file instead of stdout/stderr - `--long-jump-labels` Always use long jumps/nops for Linux kernel static keys - `--match-profile-with-function-hash` Match profile with function hash - `--max-data-relocations=` Maximum number of data relocations to process - `--max-funcs=` Maximum number of functions to process - `--no-huge-pages` Use regular size pages for code alignment - `--no-threads` Disable multithreading - `--pad-funcs=` List of functions to pad with amount of bytes - `--print-mappings` Print mappings in the legend, between characters/blocks and text sections (default false). - `--profile-format=` Format to dump profile output in aggregation mode, default is fdata - `fdata`: offset-based plaintext format - `yaml`: dense YAML representation - `--r11-availability=` Determine the availability of r11 before indirect branches - `never`: r11 not available - `always`: r11 available before calls and jumps - `abi`: r11 available before calls but not before jumps - `--relocs` Use relocations in the binary (default=autodetect) - `--remove-symtab` Remove .symtab section - `--reorder-skip-symbols=` List of symbol names that cannot be reordered - `--reorder-symbols=` List of symbol names that can be reordered - `--retpoline-lfence` Determine if lfence instruction should exist in the retpoline - `--skip-funcs=` List of functions to skip - `--skip-funcs-file=` File with list of functions to skip - `--strict` Trust the input to be from a well-formed source - `--tasks-per-thread=` Number of tasks to be created per thread - `--terminal-trap` Assume that execution stops at trap instruction - `--thread-count=` Number of threads - `--top-called-limit=` Maximum number of functions to print in top called functions section - `--trap-avx512` In relocation mode trap upon entry to any function that uses AVX-512 instructions - `--trap-old-code` Insert traps in old function bodies (relocation mode) - `--update-debug-sections` Update DWARF debug sections of the executable - `--use-gnu-stack` Use GNU_STACK program header for new segment (workaround for issues with strip/objcopy) - `--use-old-text` Reuse space in old .text if possible (relocation mode) - `-v ` Set verbosity level for diagnostic output ### BOLT optimization options: - `--align-blocks` Align basic blocks - `--align-blocks-min-size=` Minimal size of the basic block that should be aligned - `--align-blocks-threshold=` Align only blocks with frequency larger than containing function execution frequency specified in percent. E.g. 1000 means aligning blocks that are 10 times more frequently executed than the containing function. - `--align-functions=` Align functions at a given value (relocation mode) - `--align-functions-max-bytes=` Maximum number of bytes to use to align functions - `--assume-abi` Assume the ABI is never violated - `--block-alignment=` Boundary to use for alignment of basic blocks - `--bolt-seed=` Seed for randomization - `--cg-from-perf-data` Use perf data directly when constructing the call graph for stale functions - `--cg-ignore-recursive-calls` Ignore recursive calls when constructing the call graph - `--cg-use-split-hot-size` Use hot/cold data on basic blocks to determine hot sizes for call graph functions - `--cold-threshold=` Tenths of percents of main entry frequency to use as a threshold when evaluating whether a basic block is cold (0 means it is only considered cold if the block has zero samples). Default: 0 - `--elim-link-veneers` Run veneer elimination pass - `--eliminate-unreachable` Eliminate unreachable code - `--equalize-bb-counts` Use same count for BBs that should have equivalent count (used in non-LBR and shrink wrapping) - `--execution-count-threshold=` Perform profiling accuracy-sensitive optimizations only if function execution count >= the threshold (default: 0) - `--fix-block-counts` Adjust block counts based on outgoing branch counts - `--fix-func-counts` Adjust function counts based on basic blocks execution count - `--force-inline=` List of functions to always consider for inlining - `--frame-opt=` Optimize stack frame accesses - `none`: do not perform frame optimization - `hot`: perform FOP on hot functions - `all`: perform FOP on all functions - `--frame-opt-rm-stores` Apply additional analysis to remove stores (experimental) - `--function-order=` File containing an ordered list of functions to use for function reordering - `--generate-function-order=` File to dump the ordered list of functions to use for function reordering - `--generate-link-sections=` Generate a list of function sections in a format suitable for inclusion in a linker script - `--group-stubs` Share stubs across functions - `--hugify` Automatically put hot code on 2MB page(s) (hugify) at runtime. No manual call to hugify is needed in the binary (which is what --hot-text relies on). - `--icf=` Fold functions with identical code - `all`: Enable identical code folding - `none`: Disable identical code folding (default) - `safe`: Enable safe identical code folding - `--icp` Alias for --indirect-call-promotion - `--icp-calls-remaining-percent-threshold=` The percentage threshold against remaining unpromoted indirect call count for the promotion for calls - `--icp-calls-topn` Alias for --indirect-call-promotion-calls-topn - `--icp-calls-total-percent-threshold=` The percentage threshold against total count for the promotion for calls - `--icp-eliminate-loads` Enable load elimination using memory profiling data when performing ICP - `--icp-funcs=` List of functions to enable ICP for - `--icp-inline` Only promote call targets eligible for inlining - `--icp-jt-remaining-percent-threshold=` The percentage threshold against remaining unpromoted indirect call count for the promotion for jump tables - `--icp-jt-targets` Alias for --icp-jump-tables-targets - `--icp-jt-topn` Alias for --indirect-call-promotion-jump-tables-topn - `--icp-jt-total-percent-threshold=` The percentage threshold against total count for the promotion for jump tables - `--icp-jump-tables-targets` For jump tables, optimize indirect jmp targets instead of indices - `--icp-mp-threshold` Alias for --indirect-call-promotion-mispredict-threshold - `--icp-old-code-sequence` Use old code sequence for promoted calls - `--icp-top-callsites=` Optimize hottest calls until at least this percentage of all indirect calls frequency is covered. 0 = all callsites - `--icp-topn` Alias for --indirect-call-promotion-topn - `--icp-use-mp` Alias for --indirect-call-promotion-use-mispredicts - `--indirect-call-promotion=` Indirect call promotion - `none`: do not perform indirect call promotion - `calls`: perform ICP on indirect calls - `jump-tables`: perform ICP on jump tables - `all`: perform ICP on calls and jump tables - `--indirect-call-promotion-calls-topn=` Limit number of targets to consider when doing indirect call promotion on calls. 0 = no limit - `--indirect-call-promotion-jump-tables-topn=` Limit number of targets to consider when doing indirect call promotion on jump tables. 0 = no limit - `--indirect-call-promotion-topn=` Limit number of targets to consider when doing indirect call promotion. 0 = no limit - `--indirect-call-promotion-use-mispredicts` Use misprediction frequency for determining whether or not ICP should be applied at a callsite. The -indirect-call-promotion-mispredict-threshold value will be used by this heuristic - `--infer-fall-throughs` Infer execution count for fall-through blocks - `--infer-stale-profile` Infer counts from stale profile data. - `--inline-all` Inline all functions - `--inline-ap` Adjust function profile after inlining - `--inline-limit=` Maximum number of call sites to inline - `--inline-max-iters=` Maximum number of inline iterations - `--inline-memcpy` Inline memcpy using optimized instruction sequences (X86: 'rep movsb', AArch64: width-optimized register operations) - `--inline-small-functions` Inline functions if increase in size is less than defined by -inline-small- functions-bytes - `--inline-small-functions-bytes=` Max number of bytes for the function to be considered small for inlining purposes - `--instrument` Instrument code to generate accurate profile data - `--iterative-guess` In non-LBR mode, guess edge counts using iterative technique - `--jt-footprint-optimize-for-icache` With jt-footprint-reduction, only process PIC jumptables and turn off other transformations that increase code size - `--jt-footprint-reduction` Make jump tables size smaller at the cost of using more instructions at jump sites - `--jump-tables=` Jump tables support (default=basic) - `none`: do not optimize functions with jump tables - `basic`: optimize functions with jump tables - `move`: move jump tables to a separate section - `split`: split jump tables section into hot and cold based on function execution frequency - `aggressive`: aggressively split jump tables section based on usage of the tables - `--keep-nops` Keep no-op instructions. By default they are removed. - `--lite-threshold-count=` Similar to '-lite-threshold-pct' but specify threshold using absolute function call count. I.e. limit processing to functions executed at least the specified number of times. - `--lite-threshold-pct=` Threshold (in percent) for selecting functions to process in lite mode. Higher threshold means fewer functions to process. E.g threshold of 90 means only top 10 percent of functions with profile will be processed. - `--match-with-call-graph` Match functions with call graph - `--memcpy1-spec=` List of functions with call sites for which to specialize memcpy() for size 1 - `--min-branch-clusters` Use a modified clustering algorithm geared towards minimizing branches - `--name-similarity-function-matching-threshold=` Match functions using namespace and edit distance. - `--no-inline` Disable all inlining (overrides other inlining options) - `--no-scan` Do not scan cold functions for external references (may result in slower binary) - `--peepholes=` Enable peephole optimizations - `none`: disable peepholes - `double-jumps`: remove double jumps when able - `tailcall-traps`: insert tail call traps - `useless-branches`: remove useless conditional branches - `all`: enable all peephole optimizations - `--plt=` Optimize PLT calls (requires linking with -znow on non-x86 architectures) - `none`: do not optimize PLT calls - `hot`: optimize executed (hot) PLT calls - `all`: optimize all PLT calls - `--preserve-blocks-alignment` Try to preserve basic block alignment - `--profile-ignore-hash` Ignore hash while reading function profile - `--profile-use-dfs` Use DFS order for YAML profile - `--reg-reassign` Reassign registers so as to avoid using REX prefixes in hot code - `--reorder-blocks=` Change layout of basic blocks in a function - `none`: do not reorder basic blocks - `reverse`: layout blocks in reverse order - `normal`: perform optimal layout based on profile - `branch-predictor`: perform optimal layout prioritizing branch predictions - `cache`: perform optimal layout prioritizing I-cache behavior - `cache+`: perform layout optimizing I-cache behavior - `ext-tsp`: perform layout optimizing I-cache behavior - `cluster-shuffle`: perform random layout of clusters - `--reorder-data=` List of sections to reorder - `--reorder-data-algo=` Algorithm used to reorder data sections - `count`: sort hot data by read counts - `funcs`: sort hot data by hot function usage and count - `--reorder-data-inplace` Reorder data sections in place - `--reorder-data-max-bytes=` Maximum number of bytes to reorder - `--reorder-data-max-symbols=` Maximum number of symbols to reorder - `--reorder-functions=` Reorder and cluster functions (works only with relocations) - `none`: do not reorder functions - `exec-count`: order by execution count - `hfsort`: use hfsort algorithm - `hfsort+`: use cache-directed sort - `cdsort`: use cache-directed sort - `pettis-hansen`: use Pettis-Hansen algorithm - `random`: reorder functions randomly - `user`: use function order specified by -function-order - `--reorder-functions-use-hot-size` Use a function's hot size when doing clustering - `--report-bad-layout=` Print top functions with suboptimal code layout on input - `--report-stale` Print the list of functions with stale profile - `--runtime-hugify-lib=` Specify file name of the runtime hugify library - `--runtime-instrumentation-lib=` Specify file name of the runtime instrumentation library - `--runtime-lib-init-hook=` Primary target for hooking runtime library initialization, used in fallback order of availability in input binary (entry_point -> init -> init_array) (default: entry_point) - `entry_point`: use ELF Header Entry Point - `init`: use ELF DT_INIT entry - `init_array`: use ELF .init_array entry - `--sctc-mode=` Mode for simplify conditional tail calls - `always`: always perform sctc - `preserve`: only perform sctc when branch direction is preserved - `heuristic`: use branch prediction data to control sctc - `--sequential-disassembly` Performs disassembly sequentially - `--shrink-wrapping-threshold=` Percentage of prologue execution count to use as threshold when evaluating whether a block is cold enough to be profitable to move eligible spills there - `--simplify-conditional-tail-calls` Simplify conditional tail calls by removing unnecessary jumps - `--simplify-rodata-loads` Simplify loads from read-only sections by replacing the memory operand with the constant found in the corresponding section - `--split-align-threshold=` When deciding to split a function, apply this alignment while doing the size comparison (see -split-threshold). Default value: 2. - `--split-all-cold` Outline as many cold basic blocks as possible - `--split-eh` Split C++ exception handling code - `--split-functions` Split functions into fragments - `--split-strategy=` Strategy used to partition blocks into fragments - `profile2`: split each function into a hot and cold fragment using profiling information - `cdsplit`: split each function into a hot, warm, and cold fragment using profiling information - `random2`: split each function into a hot and cold fragment at a randomly chosen split point (ignoring any available profiling information) - `randomN`: split each function into N fragments at a randomly chosen split points (ignoring any available profiling information) - `all`: split all basic blocks of each function into fragments such that each fragment contains exactly a single basic block - `--split-threshold=` Split function only if its main size is reduced by more than given amount of bytes. Default value: 0, i.e. split iff the size is reduced. Note that on some architectures the size can increase after splitting. - `--stale-matching-max-func-size=` The maximum size of a function to consider for inference. - `--stale-matching-min-matched-block=` Percentage threshold of matched basic blocks at which stale profile inference is executed. - `--stale-threshold=` Maximum percentage of stale functions to tolerate (default: 100) - `--stoke` Turn on the stoke analysis - `--strip-rep-ret` Strip 'repz' prefix from 'repz retq' sequence (on by default) - `--tail-duplication=` Duplicate unconditional branches that cross a cache line - `none`: do not apply - `aggressive`: aggressive strategy - `moderate`: moderate strategy - `cache`: cache-aware duplication strategy - `--tsp-threshold=` Maximum number of hot basic blocks in a function for which to use a precise TSP solution while re-ordering basic blocks - `--use-aggr-reg-reassign` Use register liveness analysis to try to find more opportunities for -reg- reassign optimization - `--use-compact-aligner` Use compact approach for aligning functions - `--use-edge-counts` Use edge count data when doing clustering - `--verify-cfg` Verify the CFG after every pass - `--x86-align-branch-boundary-hot-only` Only apply branch boundary alignment in hot code - `--x86-strip-redundant-address-size` Remove redundant Address-Size override prefix ### BOLT instrumentation options: `llvm-bolt -instrument [-o outputfile] ` - `--conservative-instrumentation` Disable instrumentation optimizations that sacrifice profile accuracy (for debugging, default: false) - `--instrument-calls` Record profile for inter-function control flow activity (default: true) - `--instrument-hot-only` Only insert instrumentation on hot functions (needs profile, default: false) - `--instrumentation-binpath=` Path to instrumented binary in case if /proc/self/map_files is not accessible due to access restriction issues - `--instrumentation-file=` File name where instrumented profile will be saved (default: /tmp/prof.fdata) - `--instrumentation-file-append-pid` Append PID to saved profile file name (default: false) - `--instrumentation-no-counters-clear` Don't clear counters across dumps (use with instrumentation-sleep-time option) - `--instrumentation-sleep-time=` Interval between profile writes (default: 0 = write only at program end). This is useful for service workloads when you want to dump profile every X minutes or if you are killing the program and the profile is not being dumped at the end. - `--instrumentation-wait-forks` Wait until all forks of instrumented process will finish (use with instrumentation-sleep-time option) ### BOLT printing options: - `--print-aliases` Print aliases when printing objects - `--print-all` Print functions after each stage - `--print-cfg` Print functions after CFG construction - `--print-debug-info` Print debug info when printing functions - `--print-disasm` Print function after disassembly - `--print-dyno-opcode-stats=` Print per instruction opcode dyno stats and the functionnames:BB offsets of the nth highest execution counts - `--print-dyno-stats-only` While printing functions output dyno-stats and skip instructions - `--print-exceptions` Print exception handling data - `--print-globals` Print global symbols after disassembly - `--print-jump-tables` Print jump tables - `--print-loops` Print loop related information - `--print-mem-data` Print memory data annotations when printing functions - `--print-normalized` Print functions after CFG is normalized - `--print-only=` List of functions to print - `--print-orc` Print ORC unwind information for instructions - `--print-profile` Print functions after attaching profile - `--print-profile-stats` Print profile quality/bias analysis - `--print-pseudo-probes=` Print pseudo probe info - `decode`: decode probes section from binary - `address_conversion`: update address2ProbesMap with output block address - `encoded_probes`: display the encoded probes in binary section - `all`: enable all debugging printout - `--print-relocations` Print relocations when printing functions/objects - `--print-reordered-data` Print section contents after reordering - `--print-retpoline-insertion` Print functions after retpoline insertion pass - `--print-sdt` Print all SDT markers - `--print-sections` Print all registered sections - `--print-unknown` Print names of functions with unknown control flow - `--time-build` Print time spent constructing binary functions - `--time-rewrite` Print time spent in rewriting passes - `--print-after-branch-fixup` Print function after fixing local branches - `--print-after-jt-footprint-reduction` Print function after jt-footprint-reduction pass - `--print-after-lowering` Print function after instruction lowering - `--print-cache-metrics` Calculate and print various metrics for instruction cache - `--print-clusters` Print clusters - `--print-estimate-edge-counts` Print function after edge counts are set for no-LBR profile - `--print-finalized` Print function after CFG is finalized - `--print-fix-relaxations` Print functions after fix relaxations pass - `--print-fix-riscv-calls` Print functions after fix RISCV calls pass - `--print-fop` Print functions after frame optimizer pass - `--print-function-statistics=` Print statistics about basic block ordering - `--print-icf` Print functions after ICF optimization - `--print-icp` Print functions after indirect call promotion - `--print-inline` Print functions after inlining optimization - `--print-large-functions` Print functions that could not be overwritten due to excessive size - `--print-longjmp` Print functions after longjmp pass - `--print-optimize-bodyless` Print functions after bodyless optimization - `--print-output-address-range` Print output address range for each basic block in the function whenBinaryFunction::print is called - `--print-peepholes` Print functions after peephole optimization - `--print-plt` Print functions after PLT optimization - `--print-regreassign` Print functions after regreassign pass - `--print-reordered` Print functions after layout optimization - `--print-reordered-functions` Print functions after clustering - `--print-sctc` Print functions after conditional tail call simplification - `--print-simplify-rodata-loads` Print functions after simplification of RO data loads - `--print-sorted-by=` Print functions sorted by order of dyno stats - `executed-forward-branches`: executed forward branches - `taken-forward-branches`: taken forward branches - `executed-backward-branches`: executed backward branches - `taken-backward-branches`: taken backward branches - `executed-unconditional-branches`: executed unconditional branches - `all-function-calls`: all function calls - `indirect-calls`: indirect calls - `PLT-calls`: PLT calls - `executed-instructions`: executed instructions - `executed-load-instructions`: executed load instructions - `executed-store-instructions`: executed store instructions - `taken-jump-table-branches`: taken jump table branches - `taken-unknown-indirect-branches`: taken unknown indirect branches - `total-branches`: total branches - `taken-branches`: taken branches - `non-taken-conditional-branches`: non-taken conditional branches - `taken-conditional-branches`: taken conditional branches - `all-conditional-branches`: all conditional branches - `linker-inserted-veneer-calls`: linker-inserted veneer calls - `all`: sorted by all names - `--print-sorted-by-order=` Use ascending or descending order when printing functions ordered by dyno stats - `--print-split` Print functions after code splitting - `--print-stoke` Print functions after stoke analysis - `--print-uce` Print functions after unreachable code elimination - `--print-veneer-elimination` Print functions after veneer elimination pass - `--time-opts` Print time spent in each optimization - `--print-all-options` Print all option values after command line parsing - `--print-options` Print non-default options after command line parsing --- ### Bolt Docs GettingStarted (`bolt/docs/GettingStarted.md`) # Getting Started with BOLT ## Input Binary Requirements BOLT operates on X86-64 and AArch64 ELF binaries. At the minimum, the binaries should have an unstripped symbol table, and, to get maximum performance gains, they should be linked with relocations (`--emit-relocs` or `-q` linker flag). BOLT disassembles functions and reconstructs the control flow graph (CFG) before it runs optimizations. Since this is a nontrivial task, especially when indirect branches are present, we rely on certain heuristics to accomplish it. These heuristics have been tested on a code generated with Clang and GCC compilers. The main requirement for C/C++ code is not to rely on code layout properties, such as function pointer deltas. Assembly code can be processed too. Requirements for it include a clear separation of code and data, with data objects being placed into data sections/segments. If indirect jumps are used for intra-function control transfer (e.g., jump tables), the code patterns should be matching those generated by Clang/GCC. NOTE: BOLT is currently incompatible with the `-freorder-blocks-and-partition` compiler option. Since GCC8 enables this option by default, you have to explicitly disable it by adding `-fno-reorder-blocks-and-partition` flag if you are compiling with GCC8 or above. NOTE2: DWARF v5 is the new debugging format generated by the latest LLVM and GCC compilers. It offers several benefits over the previous DWARF v4. Currently, the support for v5 is a work in progress for BOLT. While you will be able to optimize binaries produced by the latest compilers, until the support is complete, you will not be able to update the debug info with `-update-debug-sections`. To temporarily work around the issue, we recommend compiling binaries with `-gdwarf-4` option that forces DWARF v4 output. PIE and .so support has been added recently. Please report bugs if you encounter any issues. ## Installation ### Docker Image You can build and use the docker image containing BOLT using our [docker file](https://github.com/llvm/llvm-project/blob/main/bolt/utils/docker/Dockerfile). Alternatively, you can build BOLT manually using the steps below. ### Manual Build BOLT heavily uses LLVM libraries, and by design, it is built as one of LLVM tools. The build process is not much different from a regular LLVM build. The following instructions are assuming that you are running under Linux. Start with cloning LLVM repo: ```console > git clone https://github.com/llvm/llvm-project.git > mkdir build > cd build > cmake -G Ninja ../llvm-project/llvm -DLLVM_TARGETS_TO_BUILD="X86;AArch64" -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS="bolt" > ninja bolt ``` `llvm-bolt` will be available under `bin/`. Add this directory to your path to ensure the rest of the commands in this tutorial work. ## Optimizing BOLT’s Performance BOLT runs many internal passes in parallel. If you foresee heavy usage of BOLT, you can improve the processing time by linking against one of memory allocation libraries with good support for concurrency. E.g. to use jemalloc: ```console > sudo yum install jemalloc-devel > LD_PRELOAD=/usr/lib64/libjemalloc.so llvm-bolt .... ``` Or if you rather use tcmalloc: ```console > sudo yum install gperftools-devel > LD_PRELOAD=/usr/lib64/libtcmalloc_minimal.so llvm-bolt .... ``` ## Usage For a complete practical guide of using BOLT see {doc}`OptimizingClang`. ### Step 0 In order to allow BOLT to re-arrange functions (in addition to re-arranging code within functions) in your program, it needs a little help from the linker. Add `--emit-relocs` to the final link step of your application. You can verify the presence of relocations by checking for `.rela.text` section in the binary. BOLT will also report if it detects relocations while processing the binary. ### Step 1: Collect Profile This step is different for different kinds of executables. If you can invoke your program to run on a representative input from a command line, then check **For Applications** section below. If your program typically runs as a server/service, then skip to **For Services** section. The version of `perf` command used for the following steps has to support `-F brstack` option. We recommend using `perf` version 4.5 or later. #### For Applications This assumes you can run your program from a command line with a typical input. In this case, simply prepend the command line invocation with `perf`: ```console $ perf record -e cycles:u -j any,u -o perf.data -- ... ``` #### For Services Once you get the service deployed and warmed-up, it is time to collect perf data with LBR (branch information). The exact perf command to use will depend on the service. E.g., to collect the data for all processes running on the server for the next 3 minutes use: ```console $ perf record -e cycles:u -j any,u -a -o perf.data -- sleep 180 ``` Depending on the application, you may need more samples to be included with your profile. It’s hard to tell upfront what would be a sweet spot for your application. We recommend the profile to cover 1B instructions as reported by BOLT `-dyno-stats` option. If you need to increase the number of samples in the profile, you can either run the `sleep` command for longer and use `-F` option with `perf` to increase sampling frequency. Note that for profile collection we recommend using cycle events and not `BR_INST_RETIRED.*`. Empirically we found it to produce better results. If the collection of a profile with branches is not available, e.g., when you run on a VM or on hardware that does not support it, then you can use only sample events, such as cycles. In this case, the quality of the profile information would not be as good, and performance gains with BOLT are expected to be lower. #### With instrumentation If perf record is not available to you, you may collect profile by first instrumenting the binary with BOLT and then running it. ```console llvm-bolt -instrument -o ``` After you run instrumented-executable with the desired workload, its BOLT profile should be ready for you in `/tmp/prof.fdata` and you can skip **Step 2**. Run BOLT with the `-help` option and check the category “BOLT instrumentation options” for a quick reference on instrumentation knobs. ### Step 2: Convert Profile to BOLT Format NOTE: you can skip this step and feed `perf.data` directly to BOLT using experimental `-p perf.data` option. For this step, you will need `perf.data` file collected from the previous step and a copy of the binary that was running. The binary has to be either unstripped, or should have a symbol table intact (i.e., running `strip -g` is okay). Make sure `perf` is in your `PATH`, and execute `perf2bolt`: ```console $ perf2bolt -p perf.data -o perf.fdata ``` This command will aggregate branch data from `perf.data` and store it in a format that is both more compact and more resilient to binary modifications. If the profile was collected without brstacks, you will need to add `-ba` flag to the command line above. ### Step 3: Optimize with BOLT Once you have `perf.fdata` ready, you can use it for optimizations with BOLT. Assuming your environment is setup to include the right path, execute `llvm-bolt`: ```console $ llvm-bolt -o .bolt -data=perf.fdata -reorder-blocks=ext-tsp -reorder-functions=hfsort -split-functions -split-all-cold -split-eh -dyno-stats ``` If you do need an updated debug info, then add `-update-debug-sections` option to the command above. The processing time will be slightly longer. For a full list of options see `-help`/`-help-hidden` output. The input binary for this step does not have to 100% match the binary used for profile collection in **Step 1**. This could happen when you are doing active development, and the source code constantly changes, yet you want to benefit from profile-guided optimizations. However, since the binary is not precisely the same, the profile information could become invalid or stale, and BOLT will report the number of functions with a stale profile. The higher the number, the less performance improvement should be expected. Thus, it is crucial to update `.fdata` for release branches. ## Multiple Profiles Suppose your application can run in different modes, and you can generate multiple profiles for each one of them. To generate a single binary that can benefit all modes (assuming the profiles don’t contradict each other) you can use `merge-fdata` tool: ```console $ merge-fdata *.fdata > combined.fdata ``` Use `combined.fdata` for **Step 3** above to generate a universally optimized binary. ## Profile Formats See {doc}`profiles` for comprehensive documentation of all profile formats accepted by BOLT: perf.data, fdata, YAML, and pre-aggregated. ## License BOLT is licensed under the [Apache License v2.0 with LLVM Exceptions](https://github.com/llvm/llvm-project/blob/main/LICENSE.TXT). --- ### Bolt Docs Heatmaps (`bolt/docs/Heatmaps.md`) # Code Heatmaps BOLT has gained the ability to print code heatmaps based on sampling-based profiles generated by `perf`, either with `brstack` data or not. The output is produced in colored ASCII to be displayed in a color-capable terminal. It looks something like this: Heatmaps can be generated for BOLTed and non-BOLTed binaries. You can use them to compare the code layout before and after optimizations. To generate a heatmap, start with running your app under `perf`: ```bash $ perf record -e cycles:u -j any,u -- ``` or if you want to monitor the existing process(es): ```bash $ perf record -e cycles:u -j any,u [-p PID|-a] -- sleep ``` Running with brstack (`-j any,u` or `-b`) is recommended. Heatmaps can be generated from basic events by using the llvm-bolt-heatmap option `-ba` (basic events) but such heatmaps do not have the coverage provided by brstack and may only be useful for finding event hotspots at larger code block granularities. Once the run is complete, and `perf.data` is generated, run llvm-bolt-heatmap: ```bash $ llvm-bolt-heatmap -p perf.data ``` By default the heatmap will be dumped to *stdout*. You can change it with `-o ` option. If you prefer to look at the data in a browser (or would like to share it that way), then you can use an HTML conversion tool. E.g.: ```bash $ aha -b -f > .html ``` --- ## Background on heatmaps: A heatmap is effectively a histogram that is rendered into a grid for better visualization. In theory we can generate a heatmap using any binary and a perf profile. Each block/character in the heatmap shows the execution data accumulated for corresponding 64 bytes of code. You can change this granularity with a `-block-size` option. E.g. set it to 4096 to see code usage grouped by 4K pages. When a block is shown as a dot, it means that no samples were found for that address. When it is shown as a letter, it indicates a captured sample on a particular text section of the binary. To show a mapping between letters and text sections in the legend, use `-print-mappings`. When a sampled address does not belong to any of the text sections, the characters 'o' or 'O' will be shown. The legend shows by default the ranges in the heatmap according to the number of samples per block. A color is assigned per range, except the first two ranges that distinguished by lower and upper case letters. On the Y axis, each row/line starts with an actual address of the binary. Consecutive lines in the heatmap advance by the same amount, with the binary size covered by a line dependent on the block size and the line size. An empty new line is inserted for larger gaps between samples. On the X axis, the horizontally emitted hex numbers can help *estimate* where in the line the samples lie, but they cannot be combined to provide a full address, as they are relative to both the bucket and line sizes. In the example below, the highlighted `0x100` column is not an offset to each row's address, but instead, it points to the middle of the line. For the generation, the default bucket size was used with a line size of 128. Some useful options are: ``` -line-size= - number of entries per line (default 256). Use a smaller value (e.g. 128) if the heatmap doesn't fit the screen horizontally. -block-size=[,,...] - heatmap bucket size, optionally followed by zoom-out sizes to produce coarse- grained heatmaps. Size can be specified in human-readable format with [kKmMgG][i][B] suffix. Default 64B, 4K, 256K. -max-address= - maximum address considered valid for heatmap (default 4GB) -print-mappings - print mappings in the legend, between characters/blocks and text sections (default false) ``` --- ### Bolt Docs NewBackend (`bolt/docs/NewBackend.md`) # Adding a New BOLT Backend A new BOLT backend should follow LLVM's [policy for adding a new target](https://llvm.org/docs/DeveloperPolicy.html#adding-a-new-target). New BOLT targets are experimental and are not built by default. Most target-specific work should be isolated in an `MCPlusBuilder` implementation under `bolt/lib/Target/`, together with platform-specific relocation handling in `bolt/lib/Core/Relocation.cpp`. Changes to common infrastructure, particularly `MCPlusBuilder`, must be separate from the target changes and land first. Such changes require thorough tests and must state their processing-time and peak-memory impact on existing targets. Measurements should use large binaries, including at least Clang itself. A backend does not need to implement every `MCPlusBuilder` interface initially. However, omitted interfaces must be checked against their callers. Analyses and passes that depend on missing functionality, such as stack frame analysis or register liveness, must remain disabled. BOLT passes are not cleanly partitioned into common and target-specific code. Enable them for a new target only after auditing their target assumptions and proving them stable with target-specific and end-to-end tests. Otherwise, keep them disabled. RISC targets generally need the `LongJmp` pass for branch relaxation. This requires correct branch-range checks and short- and long-jump construction, with tests covering range boundaries, calls, tail calls, and linker relaxation. The initial backend should include focused tests for instruction analysis, relocations, CFG construction, rewriting, and each enabled pass. Validation must include rewriting and executing large real-world binaries, with Clang as the minimum large-binary test. --- ### Bolt Docs OptimizingClang (`bolt/docs/OptimizingClang.md`) # Optimizing Clang : A Practical Example of Applying BOLT ## Preface *BOLT* (Binary Optimization and Layout Tool) is designed to improve the application performance by laying out code in a manner that helps CPU better utilize its caching and branch predicting resources. The most obvious candidates for BOLT optimizations are programs that suffer from many instruction cache and iTLB misses, such as large applications measuring over hundreds of megabytes in size. However, medium-sized programs can benefit too. Clang, one of the most popular open-source C/C++ compilers, is a good example of the latter. Its code size could easily be in the order of tens of megabytes. As we will see, the Clang binary suffers from many instruction cache misses and can be significantly improved with BOLT, even on top of profile-guided and link-time optimizations. In this tutorial we will first build Clang with PGO and LTO, and then will show steps on how to apply BOLT optimizations to make Clang up to 15% faster. We will also analyze where the compile-time performance gains are coming from, and verify that the speed-ups are sustainable while building other applications. ## Building Clang The process of getting Clang sources and performing the build is very similar to the one described at http://clang.llvm.org/get_started.html. For completeness, we provide the detailed steps on how to obtain and build Clang in [Bootstrapping Clang-7 with PGO and LTO](#bootstrapping-clang-7-with-pgo-and-lto) section. The only difference from the standard Clang build is that we require the `-Wl,-q` flag to be present during the final link. This option saves relocation metadata in the executable file, but does not affect the generated code in any way. ## Optimizing Clang with BOLT We will use the setup described in [Bootstrapping Clang-7 with PGO and LTO](#bootstrapping-clang-7-with-pgo-and-lto). Adjust the steps accordingly if you skipped that section. We will also assume that `llvm-bolt` is present in your `$PATH`. Before we can run BOLT optimizations, we need to collect the profile for Clang, and we will use Clang/LLVM sources for that. Collecting accurate profile requires running `perf` on a hardware that implements taken branch sampling (`-b/-j` flag). For that reason, it may not be possible to collect the accurate profile in a virtualized environment, e.g. in the cloud. We do support regular sampling profiles, but the performance improvements are expected to be more modest. ```bash $ mkdir ${TOPLEV}/stage3 $ cd ${TOPLEV}/stage3 $ CPATH=${TOPLEV}/stage2-prof-use-lto/install/bin/ $ cmake -G Ninja ${TOPLEV}/llvm -DLLVM_TARGETS_TO_BUILD=X86 -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_C_COMPILER=$CPATH/clang -DCMAKE_CXX_COMPILER=$CPATH/clang++ \ -DLLVM_ENABLE_PROJECTS="clang" \ -DLLVM_USE_LINKER=lld -DCMAKE_INSTALL_PREFIX=${TOPLEV}/stage3/install $ perf record -e cycles:u -j any,u -- ninja clang ``` Once the last command is finished, it will create a `perf.data` file larger than 10GiB. We will first convert this profile into a more compact aggregated form suitable to be consumed by BOLT: ```bash $ perf2bolt $CPATH/clang-7 -p perf.data -o clang-7.fdata -w clang-7.yaml ``` Notice that we are passing `clang-7` to `perf2bolt` which is the real binary that `clang` and `clang++` are symlinking to. The next step will optimize Clang using the generated profile: ```bash $ llvm-bolt $CPATH/clang-7 -o $CPATH/clang-7.bolt -b clang-7.yaml \ -reorder-blocks=ext-tsp -reorder-functions=hfsort+ -split-functions \ -split-all-cold -dyno-stats -icf=1 -use-gnu-stack ``` The output will look similar to the one below: ```text ... BOLT-INFO: enabling relocation mode BOLT-INFO: 11415 functions out of 104526 simple functions (10.9%) have non-empty execution profile. ... BOLT-INFO: ICF folded 29144 out of 105177 functions in 8 passes. 82 functions had jump tables. BOLT-INFO: Removing all identical functions will save 5466.69 KB of code space. Folded functions were called 2131985 times based on profile. BOLT-INFO: basic block reordering modified layout of 7848 (10.32%) functions ... 660155947 : executed forward branches (-2.3%) 48252553 : taken forward branches (-57.2%) 129897961 : executed backward branches (+13.8%) 52389551 : taken backward branches (-19.5%) 35650038 : executed unconditional branches (-33.2%) 128338874 : all function calls (=) 19010563 : indirect calls (=) 9918250 : PLT calls (=) 6113398840 : executed instructions (-0.6%) 1519537463 : executed load instructions (=) 943321306 : executed store instructions (=) 20467109 : taken jump table branches (=) 825703946 : total branches (-2.1%) 136292142 : taken branches (-41.1%) 689411804 : non-taken conditional branches (+12.6%) 100642104 : taken conditional branches (-43.4%) 790053908 : all conditional branches (=) ... ``` The statistics in the output is based on the brstack profile (LBR) collected with `perf`, and since we were using the `cycles` counter, its accuracy is affected. However, the relative improvement in `taken conditional branches` is a good indication that BOLT was able to straighten out the code even after PGO. ## Measuring Compile-time Improvement `clang-7.bolt` can be used as a replacement for *PGO+LTO* Clang: ```bash $ mv $CPATH/clang-7 $CPATH/clang-7.org $ ln -fs $CPATH/clang-7.bolt $CPATH/clang-7 ``` Doing a new build of Clang using the new binary shows a significant overall build time reduction on a 48-core Haswell system: ```bash $ ln -fs $CPATH/clang-7.org $CPATH/clang-7 $ ninja clean && /bin/time -f %e ninja clang -j48 202.72 $ ln -fs $CPATH/clang-7.bolt $CPATH/clang-7 $ ninja clean && /bin/time -f %e ninja clang -j48 180.11 ``` That's 22.61 seconds (or 12%) faster compared to the *PGO+LTO* build. Notice that we are measuring an improvement of the total build time, which includes the time spent in the linker. Compilation time improvements for individual files differ, and speedups over 15% are not uncommon. If we run BOLT on a Clang binary compiled without *PGO+LTO* (in which case the build is finished in 253.32 seconds), the gains we see are over 50 seconds (25%), but, as expected, the result is still slower than *PGO+LTO+BOLT* build. ## Source of the Wins We mentioned that Clang suffers from considerable instruction cache misses. This can be measured with `perf`: ```bash $ ln -fs $CPATH/clang-7.org $CPATH/clang-7 $ ninja clean && perf stat -e instructions,L1-icache-misses -- ninja clang -j48 ... 16,366,101,626,647 instructions 359,996,216,537 L1-icache-misses ``` That's about 22 instruction cache misses per thousand instructions. As a rule of thumb, if the application has over 10 misses per thousand instructions, it is a good indication that it will be improved by BOLT. Now let's see how many misses are in the BOLTed binary: ```bash $ ln -fs $CPATH/clang-7.bolt $CPATH/clang-7 $ ninja clean && perf stat -e instructions,L1-icache-misses -- ninja clang -j48 ... 16,319,818,488,769 instructions 244,888,677,972 L1-icache-misses ``` The number of misses per thousand instructions went down from 22 to 15, significantly reducing the number of stalls in the CPU front-end. Notice how the number of executed instructions stayed roughly the same. That's because we didn't run any optimizations beyond the ones affecting the code layout. Other than instruction cache misses, BOLT also improves branch mispredictions, iTLB misses, and misses in L2 and L3. ## Using Clang for Other Applications We have collected profile for Clang using its own source code. Would it be enough to speed up the compilation of other projects? We picked `mysqld`, an open-source database, to do the test. On our 48-core Haswell system using the *PGO+LTO* Clang, the build finished in 136.06 seconds, while using the *PGO+LTO+BOLT* Clang, 126.10 seconds. That's a noticeable improvement, but not as significant as the one we saw on Clang itself. This is partially because the number of instruction cache misses is slightly lower on this scenario : 19 vs 22. Another reason is that Clang is run with a different set of options while building `mysqld` compared to the training run. Different options exercise different code paths, and if we trained without a specific option, we may have misplaced parts of the code responsible for handling it. To test this theory, we have collected another `perf` profile while building `mysqld`, and merged it with an existing profile using the `merge-fdata` utility that comes with BOLT. Optimized with that profile, the *PGO+LTO+BOLT* Clang was able to perform the `mysqld` build in 124.74 seconds, i.e. 11 seconds or 9% faster compared to *PGO+LGO* Clang. The merged profile didn't make the original Clang compilation slower either, while the number of profiled functions in Clang increased from 11,415 to 14,025. Ideally, the profile run has to be done with a superset of all commonly used options. However, the main improvement is expected with just the basic set. ## Summary In this tutorial we demonstrated how to use BOLT to improve the performance of the Clang compiler. Similarly, BOLT could be used to improve the performance of GCC, or any other application suffering from a high number of instruction cache misses. ---- # Appendix ## Bootstrapping Clang-7 with PGO and LTO Below we describe detailed steps to build Clang, and make it ready for BOLT optimizations. If you already have the build setup, you can skip this section, except for the last step that adds `-Wl,-q` linker flag to the final build. ### Getting Clang-7 Sources Set `$TOPLEV` to the directory of your preference where you would like to do builds. E.g. `TOPLEV=~/clang-7/`. Follow with commands to clone the `release_70` branch of LLVM monorepo: ```bash $ mkdir ${TOPLEV} $ cd ${TOPLEV} $ git clone --branch=release/7.x https://github.com/llvm/llvm-project.git ``` ### Building Stage 1 Compiler Stage 1 will be the first build we are going to do, and we will be using the default system compiler to build Clang. If your system lacks a compiler, use your distribution package manager to install one that supports C++11. In this example we are going to use GCC. In addition to the compiler, you will need the `cmake` and `ninja` packages. Note that we disable the build of certain compiler-rt components that are known to cause build issues at release/7.x. ```bash $ mkdir ${TOPLEV}/stage1 $ cd ${TOPLEV}/stage1 $ cmake -G Ninja ${TOPLEV}/llvm-project/llvm -DLLVM_TARGETS_TO_BUILD=X86 \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_ASM_COMPILER=gcc \ -DLLVM_ENABLE_PROJECTS="clang;lld" \ -DLLVM_ENABLE_RUNTIMES="compiler-rt" \ -DCOMPILER_RT_BUILD_SANITIZERS=OFF -DCOMPILER_RT_BUILD_XRAY=OFF \ -DCOMPILER_RT_BUILD_LIBFUZZER=OFF \ -DCMAKE_INSTALL_PREFIX=${TOPLEV}/stage1/install $ ninja install ``` ### Building Stage 2 Compiler With Instrumentation Using the freshly-baked stage 1 Clang compiler, we are going to build Clang with profile generation capabilities: ```bash $ mkdir ${TOPLEV}/stage2-prof-gen $ cd ${TOPLEV}/stage2-prof-gen $ CPATH=${TOPLEV}/stage1/install/bin/ $ cmake -G Ninja ${TOPLEV}/llvm-project/llvm -DLLVM_TARGETS_TO_BUILD=X86 \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_C_COMPILER=$CPATH/clang -DCMAKE_CXX_COMPILER=$CPATH/clang++ \ -DLLVM_ENABLE_PROJECTS="clang;lld" \ -DLLVM_USE_LINKER=lld -DLLVM_BUILD_INSTRUMENTED=ON \ -DCMAKE_INSTALL_PREFIX=${TOPLEV}/stage2-prof-gen/install $ ninja install ``` ### Generating Profile for PGO While there are many ways to obtain the profile data, we are going to use the source code already at our disposal, i.e. we are going to collect the profile while building Clang itself: ```bash $ mkdir ${TOPLEV}/stage3-train $ cd ${TOPLEV}/stage3-train $ CPATH=${TOPLEV}/stage2-prof-gen/install/bin $ cmake -G Ninja ${TOPLEV}/llvm-project/llvm -DLLVM_TARGETS_TO_BUILD=X86 \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_C_COMPILER=$CPATH/clang -DCMAKE_CXX_COMPILER=$CPATH/clang++ \ -DLLVM_ENABLE_PROJECTS="clang" \ -DLLVM_USE_LINKER=lld -DCMAKE_INSTALL_PREFIX=${TOPLEV}/stage3-train/install $ ninja clang ``` Once the build is completed, the profile files will be saved under `${TOPLEV}/stage2-prof-gen/profiles`. We will merge them before they can be passed back into Clang: ```bash $ cd ${TOPLEV}/stage2-prof-gen/profiles $ ${TOPLEV}/stage1/install/bin/llvm-profdata merge -output=clang.profdata * ``` ### Building Clang with PGO and LTO Now the profile can be used to guide optimizations to produce better code for our scenario, i.e. building Clang. We will also enable link-time optimizations to allow cross-module inlining and other optimizations. Finally, we are going to add one extra step that is useful for BOLT: a linker flag instructing it to preserve relocations in the output binary. Note that this flag does not affect the generated code or data used at runtime, it only writes metadata to the file on disk: ```bash $ mkdir ${TOPLEV}/stage2-prof-use-lto $ cd ${TOPLEV}/stage2-prof-use-lto $ CPATH=${TOPLEV}/stage1/install/bin/ $ export LDFLAGS="-Wl,-q" $ cmake -G Ninja ${TOPLEV}/llvm-project/llvm -DLLVM_TARGETS_TO_BUILD=X86 \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_C_COMPILER=$CPATH/clang -DCMAKE_CXX_COMPILER=$CPATH/clang++ \ -DLLVM_ENABLE_PROJECTS="clang;lld" \ -DLLVM_ENABLE_LTO=Full \ -DLLVM_PROFDATA_FILE=${TOPLEV}/stage2-prof-gen/profiles/clang.profdata \ -DLLVM_USE_LINKER=lld \ -DCMAKE_INSTALL_PREFIX=${TOPLEV}/stage2-prof-use-lto/install $ ninja install ``` Now we have a Clang compiler that can build itself much faster. As we will see, it builds other applications faster as well, and, with BOLT, the compile time can be improved even further. --- ### Bolt Docs OptimizingLinux (`bolt/docs/OptimizingLinux.md`) # Optimizing Linux Kernel with BOLT ## Introduction Many Linux applications spend a significant amount of their execution time in the kernel. Thus, when we consider code optimization for system performance, it is essential to improve the CPU utilization not only in the user-space applications and libraries but also in the kernel. BOLT has demonstrated double-digit gains while being applied to user-space programs. This guide shows how to apply BOLT to the x86-64 Linux kernel and enhance your system's performance. In our experiments, BOLT boosted database TPS by 2 percent when applied to the kernel compiled with the highest level optimizations, including PGO and LTO. The database spent ~40% of the time in the kernel and was quite sensitive to kernel performance. BOLT optimizes code layout based on a low-level execution profile collected with the Linux `perf` tool. The best quality profile should include branch history (brstack), such as Intel's last branch records (LBR) or AArch64's Branch Record Buffer Extension (BRBE). BOLT runs on a linked binary and reorders the code while combining frequently executed blocks of instructions in a manner best suited for the hardware. Other than branch instructions, most of the code is left unchanged. Additionally, BOLT updates all metadata associated with the modified code, including DWARF debug information and Linux ORC unwind information. While BOLT optimizations are not specific to the Linux kernel, certain quirks distinguish the kernel from user-level applications. BOLT has been successfully applied to and tested with several flavors of the x86-64 Linux kernel. ## QuickStart Guide BOLT operates on a statically-linked kernel executable, a.k.a. `vmlinux` binary. However, most Linux distributions use a `vmlinuz` compressed image for system booting. To use BOLT on the kernel, you must either repackage `vmlinuz` after BOLT optimizations or add steps for running BOLT into the kernel build and rebuild `vmlinuz`. Uncompressing `vmlinuz` and repackaging it with a new `vmlinux` binary falls beyond the scope of this guide, and at some point, we may add the capability to run BOLT directly on `vmlinuz`. Meanwhile, this guide focuses on steps for integrating BOLT into the kernel build process. ### Building the Kernel After downloading the kernel sources and configuration for your distribution, you should be able to build `vmlinuz` using the `make bzImage` command. Ideally, the kernel should binary match the kernel on the system you are about to optimize (the target system). The binary matching part is critical as BOLT performs profile matching and optimizations at the binary level. We recommend installing a freshly built kernel on the target system to avoid any discrepancies. Note that the kernel build will produce several artifacts besides bzImage. The most important of them is the uncompressed `vmlinux` binary, which will be used in the next steps. Make sure to save this file. Build and target systems should have a `perf` tool installed for collecting and processing profiles. If your build system differs from the target, make sure `perf` versions are compatible. The build system should also have the latest BOLT binary and tools (`llvm-bolt`, `perf2bolt`). Once the target system boots with the freshly-built kernel, start your workload, such as a database benchmark. While the system is under load, collect the kernel profile using perf: ```bash $ sudo perf record -a -e cycles -j any,k -F 5000 -- sleep 600 ``` Convert `perf` profile into a format suitable for BOLT passing the `vmlinux` binary to `perf2bolt`: ```bash $ sudo chown $USER perf.data $ perf2bolt -p perf.data -o perf.fdata vmlinux ``` Under a high load, `perf.data` should be several gigabytes in size and you should expect the converted `perf.fdata` not to exceed 100 MB. Profiles collected from multiple workloads could be joined into a single profile using `merge-fdata` utility: ```bash $ merge-fdata perf.1.fdata perf.2.fdata ... perf..fdata > perf.merged.fdata ``` Two changes are required for the kernel build. The first one is optional but highly recommended. It introduces a BOLT-reserved space into `vmlinux` code section: ```diff --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -139,6 +139,11 @@ SECTIONS STATIC_CALL_TEXT *(.gnu.warning) + /* Allocate space for BOLT */ + __bolt_reserved_start = .; + . += 2048 * 1024; + __bolt_reserved_end = .; + #ifdef CONFIG_RETPOLINE __indirect_thunk_start = .; *(.text.__x86.*) ``` The second patch adds a step that runs BOLT on `vmlinux` binary: ```diff --- a/scripts/link-vmlinux.sh +++ b/scripts/link-vmlinux.sh @@ -340,5 +340,13 @@ if is_enabled CONFIG_KALLSYMS; then fi fi +# Apply BOLT +BOLT=llvm-bolt +BOLT_PROFILE=perf.fdata +BOLT_OPTS="--dyno-stats --eliminate-unreachable=0 --reorder-blocks=ext-tsp --simplify-conditional-tail-calls=0 --skip-funcs=__entry_text_start,irq_entries_start --split-functions" +mv vmlinux vmlinux.pre-bolt +echo BOLTing vmlinux +${BOLT} vmlinux.pre-bolt -o vmlinux --data ${BOLT_PROFILE} ${BOLT_OPTS} + # For fixdep echo "vmlinux: $0" > .vmlinux.d ``` If you skipped the first step or are running BOLT on a pre-built `vmlinux` binary, drop the `--split-functions` option. ## Performance Expectations By improving the code layout, BOLT can boost the kernel's performance by up to 5% by reducing instruction cache misses and branch mispredictions. When measuring total system performance, you should scale this number accordingly based on the time your application spends in the kernel (excluding I/O time). ## Profile Quality The timing and duration of the profiling may have a significant effect on the performance of the BOLTed kernel. If you don't know your workload well, it's recommended that you profile for the whole duration of the benchmark run. As longer times will result in larger `perf.data` files, you can lower the profiling frequency by providing a smaller value of `-F` flag. E.g., to record the kernel profile for half an hour, use the following command: ```bash $ sudo perf record -a -e cycles -j any,k -F 1000 -- sleep 1800 ``` ## BOLT Disassembly BOLT annotates the disassembly with control-flow information and attaches Linux-specific metadata to the code. To view annotated disassembly, run: ```bash $ llvm-bolt vmlinux -o /dev/null --print-cfg ``` If you want to limit the disassembly to a set of functions, add `--print-only=,,...`, where a function name is specified using regular expressions. --- ### Bolt Docs PointerAuthDesign (`bolt/docs/PointerAuthDesign.md`) # Optimizing binaries with pac-ret hardening This is a design document about processing the `DW_CFA_AARCH64_negate_ra_state` DWARF instruction in BOLT. As it describes internal design decisions, the intended audience is BOLT developers. The document is an updated version of the [RFC posted on the LLVM Discourse](https://discourse.llvm.org/t/rfc-bolt-aarch64-handle-opnegaterastate-to-enable-optimizing-binaries-with-pac-ret-hardening/86594). `DW_CFA_AARCH64_negate_ra_state` is also referred to as `.cfi_negate_ra_state` in assembly, or `OpNegateRAState` in BOLT sources. In this document, I will use **negate-ra-state** as a shorthand. Note: there are two resolutions for CFI: - Call Frame Instruction: individual DWARF instruction, e.g. negate-ra-state - Control Flow Integrity: a security mechanism, e.g. pointer authentication. ## Introduction ### Pointer Authentication For more information, see the [pac-ret section of the BOLT-binary-analysis document](BinaryAnalysis.md#return-address-protection-ptrauth-pac-ret). ### DW_CFA_AARCH64_negate_ra_state The negate-ra-state CFI is a vendor-specific Call Frame Instruction defined in the [Arm ABI](https://github.com/ARM-software/abi-aa/blob/main/aadwarf64/aadwarf64.rst#id1). ``` The DW_CFA_AARCH64_negate_ra_state operation negates bit[0] of the RA_SIGN_STATE pseudo-register. ``` This bit indicates to the unwinder whether the current return address is signed or not (hence the name). The unwinder uses this information to authenticate the pointer, and remove the Pointer Authentication Code (PAC) bits. Incorrect placement of negate-ra-state CFIs causes the unwinder to either attempt to authenticate an unsigned pointer (resulting in a segmentation fault), or skip authentication on a signed pointer, which can also cause a fault. Note: some unwinders use the `xpac` instruction to strip the PAC bits without authenticating the pointer. This is an incorrect (incomplete) implementation, as it allows control-flow modification in the case of unwinding. There are no DWARF instructions to directly set or clear the RA State. However, two other CFIs can also affect the RA state: - `DW_CFA_remember_state`: this CFI stores register rules onto an implicit stack. - `DW_CFA_restore_state`: this CFI pops rules from this stack. Example: | CFI | Effect on RA state | | ------------------------------ | ------------------------------ | | (default) | 0 | | DW_CFA_AARCH64_negate_ra_state | 0 -> 1 | | DW_CFA_remember_state | 1 pushed to the stack | | DW_CFA_AARCH64_negate_ra_state | 1 -> 0 | | DW_CFA_restore_state | 0 -> 1 (popped from the stack) | The Arm ABI also defines the DW_CFA_AARCH64_negate_ra_state_with_pc CFI, but it is not widely used, and is [likely to become deprecated](https://github.com/ARM-software/abi-aa/issues/327). ### Where are these CFIs needed? Whenever two consecutive instructions have different RA states, the unwinder must be informed of the change. This typically occurs during pointer signing or authentication. If adjacent instructions differ in RA state but neither signs nor authenticates the return address, they must belong to different control flow paths. One is part of an execution path with signed RA, the other is part of a path with an unsigned RA. In the example below, the first BasicBlock ends in a conditional branch, and jumps to two different BasicBlocks, each with their own authentication, and return. The instructions on the border of the second and third BasicBlock have different RA states. The `ret` at the end of the second BasicBlock is in unsigned state. The start of the third BasicBlock is after the `paciasp` in the control flow, but before the authentication. In this case, a negate-ra-state is needed at the end of the second BasicBlock. ``` +----------------+ | paciasp | | | | b.cc | +--------+-------+ | +----------------+ | | | +--------v-------+ | | | | | autiasp | | | ret | // RA: unsigned | +----------------+ +----------------+ | +--------v-------+ // RA: signed | | | autiasp | | ret | +----------------+ ``` > [!important] > The unwinder does not follow the control flow graph. It reads unwind > information in the layout order. Because these locations are dependent on how the function layout looks, negate-ra-state CFIs will become invalid during BasicBlock reordering. ## Solution design The implementation introduces two new passes: 1. `PointerAuthCFIAnalyzer`: assigns the RA state to each instruction based on the CFIs in the input binary 2. `PointerAuthCFIFixup`: reads those assigned instruction RA states after optimizations, and emits `DW_CFA_AARCH64_negate_ra_state` CFIs at the correct places: wherever there is a state change between two consecutive instructions in the layout order. To track metadata on individual instructions, the `MCAnnotation` class was extended. These also have helper functions in `MCPlusBuilder`. ### Saving annotations at CFI reading CFIs are read and added to BinaryFunctions in `CFIReaderWriter::FillCFIInfoFor`. At this point, we add MCAnnotations about negate-ra-state, remember-state and restore-state CFIs to the instructions they refer to. This is to not interfere with the CFI processing that already happens in BOLT (e.g. remember-state and restore-state CFIs are removed in `normalizeCFIState` for reasons unrelated to PAC). As we add the MCAnnotations *to instructions*, we have to account for the case where the function starts with a CFI altering the RA state. As CFIs modify the RA state of the instructions before them, we cannot add the annotation to the first instruction. This special case is handled by adding an `initialRAState` bool to each BinaryFunction. If the `Offset` the CFI refers to is zero, we don't store an annotation, but set the `initialRAState` in `FillCFIInfoFor`. This information is then used in `PointerAuthCFIAnalyzer`. ### Binaries without DWARF info In some cases, the DWARF tables are stripped from the binary. These programs usually have some other unwind-mechanism. These passes only run on functions that include at least one negate-ra-state CFI. This avoids processing functions that do not use Pointer Authentication, or on functions that use Pointer Authentication, but do not have DWARF info. In summary: - pointer auth is not used: no change, the new passes do not run. - pointer auth is used, but DWARF info is stripped: no change, the new passes do not run. - pointer auth is used, and we have DWARF CFIs: passes run, and rewrite the negate-ra-state CFI. ### PointerAuthCFIAnalyzer pass This pass runs before optimizations reorder anything. It processes MCAnnotations generated during the CFI reading stage to check if instructions have either of the three CFIs that can modify RA state: - negate-ra-state, - remember-state, - restore-state. Then it adds new MCAnnotations to each instruction, indicating their RA state. Those annotations are: - Signed, - Unsigned. Below is a simple example, that shows the two different type of annotations: what we have before the pass, and after it. | Instruction | Before | After | | ----------------------------- | --------------- | -------- | | paciasp | negate-ra-state | unsigned | | stp x29, x30, [sp, #-0x10]! | | signed | | mov x29, sp | | signed | | ldp x29, x30, [sp], #0x10 | | signed | | autiasp | negate-ra-state | signed | | ret | | unsigned | #### Error handling in PointerAuthCFIAnalyzer pass: Whenever the PointerAuthCFIAnalyzer pass finds inconsistencies in the current BinaryFunction, it marks the function as ignored using `BF.setIgnored()`. BOLT will not optimize this function but will emit it unchanged in the original section (`.bolt.org.text`). The inconsistencies are as follows: - finding a `pac*` instruction when already in signed state - finding an `aut*` instruction when already in unsigned state - finding `pac*` and `aut*` instructions without `.cfi_negate_ra_state`. Users will be informed about the number of ignored functions in the pass, the exact functions ignored, and the found inconsistency. ### PointerAuthCFIFixup This pass runs after optimizations. It performs the _inverse_ of PointerAuthCFIAnalyzer pass: 1. it reads the RA state annotations attached to the instructions, and 2. whenever the state changes, it adds a PseudoInstruction that holds an OpNegateRAState CFI. #### Covering newly generated instructions: Some BOLT passes can add new Instructions. In PointerAuthCFIFixup, we have to know what RA state these have. > [!important] > As issue #160989 explains, unwind info is missing from stubs. > For this same reason, we cannot generate correct pac-specific unwind info: the > signedness of the _incorrect_ return address is meaningless. Assignment of RAStates to newly generated instructions is done in `inferUnknownStates`. We have two different cases to cover: 1. If a BasicBlock has some instructions with known RA state, and some without, we can copy the RAState of known instructions to the unknown ones. As the control flow only changes between BasicBlocks, instructions in the same BasicBlock have the same return address. (The exception is noreturn calls, but these would only cause problems, if the newly inserted instruction is right after the call.) 2. If a BasicBlock has no instructions with known RAState, we have to copy the RAState of the previous BasicBlock in layout order. ### Optimizations requiring special attention Marking states before optimizations ensure that instructions can be moved around freely. The only special case is function splitting. When a function is split, the split part becomes a new function in the emitted binary. For unwinding to work, it needs to "replay" all CFIs that lead up to the split point. BOLT does this for other CFIs. As negate-ra-state is not read (only stored as an Annotation), we have to do this manually in PointerAuthCFIFixup. Here, if the split part starts with an instruction that has Signed RA state, we add a negate-ra-state CFI to indicate this. ## Option to disallow the feature The feature can be guarded with the `--update-branch-prediction` flag, which is on by default. If the flag is set to false, and a function `containedNegateRAState()` after `FillCFIInfoFor()`, BOLT exits with an error. --- ### Bolt Docs README (`bolt/docs/README.txt`) BOLT Documentation ==================== The BOLT documentation is written using the Sphinx documentation generator. It is currently tested with Sphinx 1.1.3. To build the documents into html configure BOLT with the following cmake options: * -DLLVM_ENABLE_SPHINX=ON * -DBOLT_INCLUDE_DOCS=ON After configuring BOLT with these options the make rule `docs-bolt-html` should be available. --- ### Bolt Docs RuntimeLibrary (`bolt/docs/RuntimeLibrary.md`) # BOLT ORC-based linker A high-level view on the simple linker used to insert auxiliary/library code into the final binary produced by BOLT. This is built on top of LLVM's ORC infra (the newest iteration on JITting for LLVM). ## Several levels of code injection When BOLT starts processing an input executable, its first task is to raise the binary to a low-level IR with CFG. After this is done, we are ready to change code in this binary. Throughout BOLT's pipeline of code transformations, there are plenty of situations when we need to insert new code or fix existing code. If operating with small code changes inside a basic block, we typically defer this work to MCPlusBuilder. This is our target-independent interface to create new instructions, but it also contains some functions that may create code spanning multiple basic blocks (for instance, when doing indirect call promotion and unrolling an indirect call into a ladder of comparisons/direct calls). The implementation here usually boils down to programmatically creating new MCInst instructions while setting their opcodes according to the target list (see X86GenInstOpcodes.inc generated by tablegen in an LLVM build). However, this approach quickly becomes awkward if we want to insert a lot of code, especially if this code is frozen and never changes. In these situations, it is more convenient to have a runtime library with all the code you need to insert. This library defines some symbols and can be linked into the final binary. In this case, all you need to do in a BOLT transformation is to insert a call to your library. ## The runtime library Currently, our runtime library is written in C++ and contains code that helps us instrument a binary. ### Limitations Our library is not written with regular C++ code as it is not linked against any other libraries (this means we cannot rely on anything defined on libstdc++, glibc, libgcc etc), but is self sufficient. In runtime/CMakeLists.txt, we can see it is built with -ffreestanding, which requires the compiler to avoid using a runtime library by itself. While this requires us to make our own syscalls, it does simplify our linker a lot, which is very limited and can only do basic function name resolving. However, this is a big improvement in comparison with programmatically generating the code in assembly language using MCInsts. A few more quirks: * No BSS section: don't use uninitialized globals * No dependencies on foreign code: self sufficient * You should closely watch the generated bolt_rt object files, anything requiring fancy linker features will break. We only support bare bones .text, .data and nothing else. Read instr.cpp opening comment for more details. ## Linking While RewriteInstance::emitAndLink() will perform an initial link step to resolve all references of the input program, it will not start linking the runtime library right away. The input program lives in its own module that may end up with unresolved references to the runtime library. RewriteInstance::linkRuntime() has the job of actually reading individual .o files and adding them to the binary. We currently have a single .o file, so after it is read, ORC can finally resolve references from the first module to the newly inserted .o objects. This sequence of steps is done by calls to addObject() and emitAndFinalize(). The latter will trigger symbol resolution, relying on the symbol resolver provided by us when calling createLegacyLookupResolver(). --- ### Bolt Docs Profiles (`bolt/docs/profiles.md`) # BOLT Profile Formats BOLT accepts profile data in several formats. This document describes each format, how to generate it, and how BOLT consumes it. The general recommended workflow is to convert unsymbolized profiles (perf.data or pre-aggregated) into symbolized (fdata or YAML): ``` $ perf2bolt executable \ # perf.data is consumed directly: -p perf.data # OR pre-aggregated requires `--pa` switch: -p preagg --pa # fdata is the default output format, YAML is optionally emitted using `-w` flag: -o perf.fdata [-w perf.yaml] # the output format for `-o` can be switched with `--profile-format`: -o perf.yaml --profile-format=yaml # perf.data can also be cached as pre-aggregated trace data: -o perf.preagg --profile-format=preagg ``` # Unsymbolized profiles Sample or trace profiles without symbol information accepted by perf2bolt, to be converted into symbolized profile formats, used by llvm-bolt. ## Linux perf data ### Collection Example with brstack: ```bash perf record -j any,u -e cycles:u -o perf.data -- ./binary ``` ### Consumption modes - **Branch samples (default)**: Branch stack samples from capable hardware (Intel LBR, AMD LBRv2/BRS, ARM BRBE). Used by default with `perf2bolt` and `llvm-bolt -p perf.data`. - **Basic aggregation (`-ba`)**: Sample-based profile without branch stacks. Lower quality but works on hardware/VMs without branch sampling support. - **Tracing (`--itrace`)**: Synthesizing branch stacks from trace profile (Intel PT, ARM ETM). Requires a value (e.g. `i10usl`), see [perf documentation](https://github.com/torvalds/linux/blob/35f5aa9ccc83f4a4171cdb6ba023e514e2b2ecff/tools/perf/Documentation/itrace.txt) for details. - **ARM SPE (`--spe`)**: Statistical Profiling Extension on supported ARM platforms providing short (1-deep) branch stacks. ### Build-id verification BOLT verifies that the build-id in `perf.data` matches the input binary. Use `--ignore-build-id` to skip this check. ## Pre-aggregated format Pre-aggregated profile for direct consumption by `perf2bolt --pa` or `llvm-bolt --pa`. Enables external tools to generate BOLT-compatible profiles without going through `perf.data`. ### Entry types ``` E S [TR] [] B [] [Ff] r ``` Where: - `E` — Name of the sampling event used for subsequent entries. - `S` — Aggregated basic sample at ``. - `T` — Aggregated trace: branch from `` to `` with a fall-through to ``. - `R` — Aggregated trace originating at a return. - `B` — Aggregated branch from `` to ``. - `F` — Aggregated fall-through from `` to ``. - `f` — Aggregated fall-through with external origin. - `r` — Aggregated fall-through with external return as origin. ### Trace mapping Internally, all branch/fall-through formats are represented as traces, with the following field mapping: - `B ` -> `T BR_ONLY` - `F ` -> `T FT_ONLY ` - `f ` -> `T FT_EXTERNAL_ORIGIN ` - `r ` -> `R FT_EXTERNAL_RETURN ` The constants have the following values and can be specified directly: - `BR_ONLY`/`FT_ONLY`: UINT64_MAX: ffffffffffffffff or -1 - `FT_EXTERNAL_ORIGIN`: UINT64_MAX-1: fffffffffffffffe or -2 - `FT_EXTERNAL_RETURN`: UINT64_MAX-2: fffffffffffffffd or -3 ### Call continuation fall-throughs Where applicable, call continuation fall-throughs are extended back to cover the call site, improving profile continuity: - `f`: check that the fall-through start is a block start, not an entry point or a landing pad. - `R`/`r`: unconditional. - `T`: local origin: extend only if it's a return. External origin: use `f`. Return hint is only needed for external returns (e.g. from a PLT call), where the address can't be disassembled to distinguish a return from a jump. ### Best practices Use T/R format. To capture branch type using Linux perf events, set the `PERF_SAMPLE_BRANCH_TYPE_SAVE` flag in `perf_event_attr::branch_sample_type`. To represent top of brstack entry with no fall-through, use `BR_ONLY` as `ft_end`. Such traces can be extended with an average fall-through length by passing `--impute-trace-fall-through`. ### Location format Locations have the format `[:]`: - `` — Hex vaddr (non-PIE) or offset from the object base load address (PIE). - `:` — Offset within the object identified by ``. - `X:` — External address (outside the profiled binary). Base load address is the address of the first `PT_LOAD` segment in the binary, which may not be the same as the segment containing code address. ### Examples Basic samples profile: ``` E cycles S 41be50 3 E br_inst_retired.near_taken S 41be60 6 ``` Trace profile combining branches and fall-throughs: ``` T 4b196f 4b19e0 4b19ef 2 ``` Trace with unknown fall-through: ``` T 4b196f 4b19e0 -1 2 ``` Legacy branch profile with separate branches and fall-throughs: ``` F 41be50 41be50 3 F 41be90 41be90 4 B 4b1942 39b57f0 3 0 B 4b196f 4b19e0 2 0 ``` ### Generation Pre-aggregated profiles can be generated by external tools. See [ebpf-bolt](https://github.com/aaupov/ebpf-bolt) for a reference implementation using eBPF-based collection. `perf2bolt` can generate a pre-aggregated profile from`perf.data`: ``` perf2bolt ./binary -p perf.data -o perf.preagg --profile-format=preagg ``` # Symbolized profiles The profiles accepted by llvm-bolt. fdata is the legacy format, YAML is the rich (metadata-enabled) format. As local symbol names are not guaranteed to be unique, BOLT disambiguates them with suffixes: - short form `/`, - long form `//`, where `` is the containing debug file symbol. (Debug file symbols can be preserved when stripping debug information using `--keep-file-symbols`). As such, symbolized profiles are not expected to be produced by external tools, except the **symbols** fdata format mode which accepts names without suffixes but matches all aliases. ## fdata format Plaintext, space-separated branch profile format written by `perf2bolt` and consumed by `llvm-bolt -data `. Also produced by BOLT instrumentation. ### LBR mode format Each line records a branch: ``` ``` Where: - ``, ``: `1` if the name is an ELF symbol, `0` if it is a DSO name. Special values: `2` for local symbols (includes filename), `3`/`4`/`5` for memory events. - ``, ``: Symbol name or DSO name. - ``, ``: Hex offset relative to the symbol/DSO. - ``: Number of branch mispredictions. - ``: Total number of branches. Example: ``` 1 main 3fb 0 /lib/ld-2.21.so 12 4 221 ``` ### No-LBR mode format Requires `no_lbr` header followed by an optional event name: ``` no_lbr ``` ### Symbols mode format Requires `symbols` header, followed by a list of (mangled) symbol names: ``` symbols main BZ2_compressBlock ``` Each line matches all aliases of the symbol, each matching function is assigned an execution count of `1`. ### Special headers - `boltedcollection`: Indicates profile collected on a BOLTed binary. Requires BAT (BOLT Address Translation) tables for remapping. ### Memory events format Memory event types use `` values 3, 4, 5 to record load address information alongside the instruction location. ## YAML format Structured profile format with block-level granularity. More resilient to binary changes and supports stale profile matching. ### Schema Defined in `ProfileYAMLMapping.h`: ```yaml header: profile-version: binary-name: binary-build-id: # optional profile-flags: [lbr|sample|memevent] profile-origin: # optional, how profile was obtained profile-events: # optional, event names dfs-order: # optional, default true hash-func: # optional, default std-hash functions: - name: fid: hash: exec: nblocks: blocks: - bid: insns: hash: # optional exec: # optional succ: [{bid, cnt, mis}] # optional calls: [{off, fid, cnt}] # optional inline_tree: [...] # optional, pseudo probe info ``` ### Hash functions - `std-hash`: Standard hash function (default for backward compatibility). - `xxh3`: XXH3 hash function (recommended, better distribution). ### Stale profile matching BOLT supports matching profiles to modified binaries using block hashes and call graph matching. When the binary changes between profile collection and optimization, BOLT uses the hash values to find corresponding blocks in the new binary. --- ### Clang Tools Extra Docs CMakeLists (`clang-tools-extra/docs/CMakeLists.txt`) if (DOXYGEN_FOUND) if (LLVM_ENABLE_DOXYGEN) set(abs_srcdir ${CMAKE_CURRENT_SOURCE_DIR}) set(abs_builddir ${CMAKE_CURRENT_BINARY_DIR}) if (HAVE_DOT) set(DOT ${LLVM_PATH_DOT}) endif() if (LLVM_DOXYGEN_EXTERNAL_SEARCH) set(enable_searchengine "YES") set(searchengine_url "${LLVM_DOXYGEN_SEARCHENGINE_URL}") set(enable_server_based_search "YES") set(enable_external_search "YES") set(extra_search_mappings "${LLVM_DOXYGEN_SEARCH_MAPPINGS}") else() set(enable_searchengine "NO") set(searchengine_url "") set(enable_server_based_search "NO") set(enable_external_search "NO") set(extra_search_mappings "") endif() # If asked, configure doxygen for the creation of a Qt Compressed Help file. if (LLVM_ENABLE_DOXYGEN_QT_HELP) set(CLANG_TOOLS_DOXYGEN_QCH_FILENAME "org.llvm.clang.qch" CACHE STRING "Filename of the Qt Compressed help file") set(CLANG_TOOLS_DOXYGEN_QHP_NAMESPACE "org.llvm.clang" CACHE STRING "Namespace under which the intermediate Qt Help Project file lives") set(CLANG_TOOLS_DOXYGEN_QHP_CUST_FILTER_NAME "Clang ${CLANG_VERSION}" CACHE STRING "See http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom-filters") set(CLANG_TOOLS_DOXYGEN_QHP_CUST_FILTER_ATTRS "Clang,${CLANG_VERSION}" CACHE STRING "See http://qt-project.org/doc/qt-4.8/qthelpproject.html#filter-attributes") set(clang_tools_doxygen_generate_qhp "YES") set(clang_tools_doxygen_qch_filename "${CLANG_DOXYGEN_QCH_FILENAME}") set(clang_tools_doxygen_qhp_namespace "${CLANG_DOXYGEN_QHP_NAMESPACE}") set(clang_tools_doxygen_qhelpgenerator_path "${LLVM_DOXYGEN_QHELPGENERATOR_PATH}") set(clang_tools_doxygen_qhp_cust_filter_name "${CLANG_DOXYGEN_QHP_CUST_FILTER_NAME}") set(clang_tools_doxygen_qhp_cust_filter_attrs "${CLANG_DOXYGEN_QHP_CUST_FILTER_ATTRS}") else() set(clang_tools_doxygen_generate_qhp "NO") set(clang_tools_doxygen_qch_filename "") set(clang_tools_doxygen_qhp_namespace "") set(clang_tools_doxygen_qhelpgenerator_path "") set(clang_tools_doxygen_qhp_cust_filter_name "") set(clang_tools_doxygen_qhp_cust_filter_attrs "") endif() llvm_configure_doxygen() set(abs_top_srcdir) set(abs_top_builddir) set(DOT) set(enable_searchengine) set(searchengine_url) set(enable_server_based_search) set(enable_external_search) set(extra_search_mappings) set(clang_tools_doxygen_generate_qhp) set(clang_tools_doxygen_qch_filename) set(clang_tools_doxygen_qhp_namespace) set(clang_tools_doxygen_qhelpgenerator_path) set(clang_tools_doxygen_qhp_cust_filter_name) set(clang_tools_doxygen_qhp_cust_filter_attrs) add_custom_target(doxygen-clang-tools COMMAND ${DOXYGEN_EXECUTABLE} ${CMAKE_CURRENT_BINARY_DIR}/doxygen.cfg WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} COMMENT "Generating clang doxygen documentation." VERBATIM) set_target_properties(doxygen-clang-tools PROPERTIES FOLDER "Clang Tools Extra/Docs") if (LLVM_BUILD_DOCS) add_dependencies(doxygen doxygen-clang-tools) endif() if (NOT LLVM_INSTALL_TOOLCHAIN_ONLY) install(DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/doxygen/html DESTINATION docs/html) endif() endif() endif() if (LLVM_ENABLE_SPHINX) include(AddSphinxTarget) if (SPHINX_FOUND) if (${SPHINX_OUTPUT_HTML}) add_sphinx_target(html clang-tools) endif() if (${SPHINX_OUTPUT_MAN}) add_sphinx_target(man clang-tools) endif() endif() endif() --- ### Clang Tools Extra Docs ModularizeUsage (`clang-tools-extra/docs/ModularizeUsage.md`) # Modularize Usage `modularize [] [|]* [...]` `` is a place-holder for options specific to modularize, which are described below in `Modularize Command Line Options`. `` specifies the path of a file name for an existing module map. The module map must be well-formed in terms of syntax. Modularize will extract the header file names from the map. Only normal headers are checked, assuming headers marked "private", "textual", or "exclude" are not to be checked as a top-level include, assuming they either are included by other headers which are checked, or they are not suitable for modules. `` specifies the path of a file name for a file containing the newline-separated list of headers to check with respect to each other. Lines beginning with '#' and empty lines are ignored. Header file names followed by a colon and other space-separated file names will include those extra files as dependencies. The file names can be relative or full paths, but must be on the same line. For example: ``` header1.h header2.h header3.h: header1.h header2.h ``` Note that unless a `-prefix (header path)` option is specified, non-absolute file paths in the header list file will be relative to the header list file directory. Use -prefix to specify a different directory. `` is a place-holder for regular Clang front-end arguments, which must follow the ``. Note that by default, modularize assumes .h files contain C++ source, so if you are using a different language, you might need to use a `-x` option to tell Clang that the header contains another language, i.e.: `-x c` Note also that because modularize does not use the clang driver, you will likely need to pass in additional compiler front-end arguments to match those passed in by default by the driver. ## Modularize Command Line Options :::{option} -prefix= Prepend the given path to non-absolute file paths in the header list file. By default, headers are assumed to be relative to the header list file directory. Use `-prefix` to specify a different directory. ::: :::{option} -module-map-path= Generate a module map and output it to the given file. See the description in {ref}`module-map-generation`. ::: :::{option} -problem-files-list= For use only with module map assistant. Input list of files that have problems with respect to modules. These will still be included in the generated module map, but will be marked as "excluded" headers. ::: :::{option} -root-module= Put modules generated by the -module-map-path option in an enclosing module with the given name. See the description in {ref}`module-map-generation`. ::: :::{option} -block-check-header-list-only Limit the #include-inside-extern-or-namespace-block check to only those headers explicitly listed in the header list. This is a work-around for avoiding error messages for private includes that purposefully get included inside blocks. ::: :::{option} -no-coverage-check Don't do the coverage check for a module map. ::: :::{option} -coverage-check-only Only do the coverage check for a module map. ::: :::{option} -display-file-lists Display lists of good files (no compile errors), problem files, and a combined list with problem files preceded by a '#'. This can be used to quickly determine which files have problems. The latter combined list might be useful in starting to modularize a set of headers. You can start with a full list of headers, use -display-file-lists option, and then use the combined list as your intermediate list, uncommenting-out headers as you fix them. ::: --- ### Clang Tools Extra Docs README (`clang-tools-extra/docs/README.txt`) ---------------------------------- Documentation in clang-tools-extra ---------------------------------- To generate documentation in HTML format from files in clang-tools-extra/docs, build the docs-clang-tools-html target. To generate documentation from the source code using Doxygen, build the doxygen-clang-tools target. --- ### Clang Tools Extra Docs ReleaseNotes (`clang-tools-extra/docs/ReleaseNotes.md`) --- myst: enable_extensions: - attrs_block - colon_fence - substitution --- % If you want to modify sections/contents permanently, you should modify both % ReleaseNotes.md and ReleaseNotesTemplate.txt. {#extra-clang-tools-release-releasenotestitle} # Extra Clang Tools {{env.config.release}} {{ (('(In-Progress) ' if env.app.tags.has('PreRelease') else '') ~ 'Release Notes') }} Written by the [LLVM Team](https://llvm.org/) ::::{only} PreRelease :::{warning} These are in-progress notes for the upcoming Extra Clang Tools {{env.config.version}} release. Release notes for previous releases can be found on [the Download Page](https://releases.llvm.org/download.html). ::: :::: ## Introduction This document contains the release notes for the Extra Clang Tools, part of the Clang release {{env.config.release}}. Here we describe the status of the Extra Clang Tools in some detail, including major improvements from the previous release and new feature work. All LLVM releases may be downloaded from the [LLVM releases web site](https://llvm.org/releases/). For more information about Clang or LLVM, including information about the latest release, please see the [Clang Web Site](https://clang.llvm.org) or the [LLVM Web Site](https://llvm.org). Note that if you are reading this file from a Git checkout or the main Clang web page, this document applies to the *next* release, not the current one. To see the release notes for a specific release, please see the [releases page](https://llvm.org/releases/). {#what-s-new-in-extra-clang-tools-release} ## What's New in Extra Clang Tools {{env.config.release}}? Some of the major new features and improvements to Extra Clang Tools are listed here. Generic improvements to Extra Clang Tools as a whole or to its underlying infrastructure are described first, followed by tool-specific sections. ### Major New Features ### Potentially Breaking Changes - The deprecated `zircon` clang-tidy module has been removed. Users of `zircon-temporary-objects` should migrate to {doc}`fuchsia-temporary-objects `. - In 22nd release, The `clang-tidy/ClangTidyModuleRegistry.h` header was deprecated. All of the symbols it used to define were moved into `clang-tidy/ClangTidyModule.h`. The deprecated header has been removed in this release. ### Improvements to clangd #### Inlay hints #### Diagnostics - Query-based custom clang-tidy checks can now be enabled with the `Diagnostics.ClangTidy.ExperimentalCustomChecks` clangd configuration option. Custom checks are subject to `FastCheckFilter`, and therefore require `FastCheckFilter: Loose` or `None` to run. #### Semantic Highlighting #### Compile flags #### Hover - The type a `decltype` resolves to is now also shown for composite types, e.g. `decltype(x)&` is displayed as `int&`. Qualifiers applied to a `decltype` are no longer dropped, so `const decltype(x)` is displayed as `const int` rather than `int`. #### Code completion - Parameters declared with a `decltype` are now displayed as the type the `decltype` resolves to, e.g. `set_x(int val)` rather than `set_x(decltype(x) val)`. #### Code actions - clangd now applies clang-tidy fix-it post-processing before exposing fixes. - The `Extract to function` tweak is now offered for selections consisting of a single expression-statement (e.g. a lone function call or an overloaded operator call such as `stream << 42;`), which it previously refused to extract. - The `Extract to function` tweak is now also available in C files, where it previously always refused to apply. Captured variables are passed by value when possible, and otherwise via a pointer parameter, since C has no references. #### Signature help - Parameters declared with a `decltype` are now displayed as the type the `decltype` resolves to, as for code completion. #### Cross-references #### Objective-C #### Miscellaneous - Fixed `clangd-indexer --format=yaml` emitting invalid YAML when indexing C++20 concepts. ([#206875](https://github.com/llvm/llvm-project/issues/206875)) - `clangd-indexer` now reads clangd configuration files (`.clangd` and the user config) by default. Pass `--enable-config=false` to restore the previous behavior. - Added support for loading dynamic plugins via the `-load` command-line option. ### Improvements to clang-doc ### Improvements to clang-query ### Improvements to clang-tidy - Improved {program}`check_clang_tidy.py` by adding support of `-std=cXX-or-earlier` values, mirroring the existing `-std=cXX-or-later`. New construct expands to the given standard and every earlier one. - On Windows, {program}`clang-tidy` will now better match mixes of forwards and backwards slashes when using the `--line-filter` option. This also improves piping from {program}`git` to {program}`clang-tidy-diff.py`, where slashes will now be automatically normalized. #### New checks - New {doc}`llvm-invalid-regex-pattern ` check. Detects malformed regex patterns defined in a single string literal. - New {doc}`modernize-use-to-underlying ` check. Finds casts from a scoped enumeration (`enum class`) to an integer type and replaces them with a call to `std::to_underlying` (introduced in C++23). - New {doc}`performance-expensive-value-or ` check. Finds calls to `value_or` (and alternative spellings `valueOr`, `ValueOr`) on optional types where the return type is expensive to copy. - New {doc}`portability-avoid-pragma-comment ` check. Finds uses of `#pragma comment` and, for `lib` or `linker` comments, suggests using the build system for improved portability. - New {doc}`readability-redundant-zero-initializer ` check. Finds explicit zero initializers of arrays that can be replaced with empty braces. #### New check aliases #### Changes in existing checks - Improved {doc}`bugprone-easily-swappable-parameters ` check by no longer emitting empty notes for type aliases when {option}`ModelImplicitConversions` is enabled. - Improved {doc}`bugprone-implicit-widening-of-multiplication-result ` check by suggesting a wider type of the same signedness as the original operands, instead of forcing a signed type, when a multiplication of two unsigned operands narrower than `int` is only signed due to integer promotion. - Improved {doc}`bugprone-macro-parentheses ` by fixing invalid fixes for macro arguments used as names in alias declarations. - Fixed a crash in {doc}`bugprone-misplaced-operator-in-strlen-in-alloc ` when checking an array new expression without a size expression. - Fixed a crash in {doc}`bugprone-misplaced-pointer-arithmetic-in-alloc ` when pointer arithmetic is applied to a non-array `new` expression whose constructor has no arguments. - Fixed a crash in {doc}`bugprone-pointer-arithmetic-on-polymorphic-object ` when the pointer points to an incomplete (forward-declared) type. - Improved {doc}`bugprone-redundant-branch-condition ` check by fixing false positives when the condition variable is changed later in a loop that encloses the inner `if`. - Fixed a crash in {doc}`bugprone-std-namespace-modification ` when checking lambda closure types used as template arguments. - Improved {doc}`bugprone-unchecked-optional-access ` by fixing false positives on `bsl::optional` and `bdlb::NullableValue` constructed from a value or returned by `bsl::make_optional`. - Improved {doc}`cppcoreguidelines-missing-std-forward ` check by diagnosing unforwarded `auto&&` parameters in C++20 abbreviated function templates. - Improved {doc}`cppcoreguidelines-pro-type-member-init ` check by treating `std::array` the same as built-in arrays when `IgnoreArrays` option is enabled. - Improved {doc}`cppcoreguidelines-use-enum-class ` check by omitting unnamed enums from the `enum class` requirement, as previously the check suggested users an ill-formed fix. - Improved {doc}`cppcoreguidelines-virtual-class-destructor ` check by emitting the diagnostic and its fix-it notes at the destructor's location instead of the class name, whenever the destructor is user-declared. The diagnostics are still emitted at the class name for implicitly declared destructors. - Improved {doc}`misc-const-correctness ` check: - Fixed false positives when the pointee is written through a pointer that is incremented, decremented or adjusted with `+=` or `-=`, such as `*p++ = 0`. - Fixed false positives when the pointee is written through a pointer assignment, such as `*(p = q) = 0`. - No longer diagnoses variables declared with `decltype(auto)`, where the suggested `const` does not compile. - No longer diagnoses parameters of `main`, whose signature is fixed by the standard. - Fixed an infinite loop in {doc}`misc-multiple-inheritance ` when checking a class that inherits from itself or has a circular inheritance graph. - Improved {doc}`misc-redundant-expression ` by fixing false positives in nested expressions involving different macros or a mix of macro and non-macro operands. - Fixed a crash in {doc}`modernize-raw-string-literal ` on synthetic string literals created for raw user-defined literal operators, such as `12_w`. - Improved {doc}`modernize-return-braced-init-list ` check to no longer rewrite the return value when the constructed type has a `std::initializer_list` constructor, as the braced form could select a different constructor. - Fixed a crash in {doc}`modernize-use-designated-initializers ` when analyzing malformed code with nested classes and ambiguous initializer. - Improved {doc}`modernize-use-equals-default ` check to also capture default constructor with explicit default-initialization of parents. - Fixed a crash in {doc}`modernize-use-noexcept ` when analyzing malformed template code with an unparsed exception specification. - Extend {doc}`modernize-use-nullptr ` to turn `decltype(nullptr)` into `std::nullptr_t` from ``. - Improved {doc}`modernize-use-nullptr ` check to avoid replacing `0` with `nullptr` in comparisons with ordering types such as `std::strong_ordering`. - Improved {doc}`modernize-use-ranges ` check by preserving used output iterator results when replacing output algorithms such as `std::copy`. - Preserved used callable results when replacing `std::for_each` and structured binding results when replacing algorithms such as `std::equal_range`. - Kept diagnostics but suppressed unsafe fix-its when no safe result-preserving rewrite is available. - Improved {doc}`performance-inefficient-algorithm ` check to no longer produce a fix with the container or the searched-for value missing, such as `.find(43)` or `s.find()`, when either comes from a macro. The value is copied as written rather than with its parentheses stripped, and no fix is offered when an argument covers only part of a macro expansion, as it then has no source text of its own. - Improved {doc}`performance-inefficient-vector-operation ` by adding the {option}`ForRangeLoopClasses` to configure container classes that can be used as sources in range-based `for` loops. - Improved {doc}`performance-no-automatic-move ` check by no longer warning when returning a variable captured by a lambda or a block, which is never implicitly moved from. - Improved {doc}`performance-prefer-single-char-overloads ` check to avoid offering fix-its for string literals originating from macro expansions. - Improved {doc}`readability-convert-member-functions-to-static ` check by fixing a crash when checking a const-qualified method declared with the `lifetimebound` attribute. - Improved {doc}`readability-enum-initial-value ` check by adding the {option}`AllowReferencedInitialValues` to support the `INT09-C-EX1` exception, allowing enumerators initialized by referencing another enumerator in the same enum (e.g., `last = first`). - Improved {doc}`readability-function-cognitive-complexity ` check by fixing a crash when checking a function declared with the `alias` attribute. - Improved {doc}`readability-identifier-naming ` check: - Fixed a crash when a class inherits from a forward-declared base class. - Fixed a crash when checking forward-declared classes with {option}`DefaultHungarianPrefix` enabled. - Fixed {option}`DefaultHungarianPrefix` being incorrectly diagnosed as an invalid option. - Fixed invalid fixes when renaming parameter packs used in `sizeof...` expressions. - Added the {option}`TypedefInheritAnonTagConfig`, which checks a typedef or type alias that provides the only name of an otherwise unnamed tag, such as `typedef enum {} MyEnum;`, against the style configured for that tag kind instead of the typedef or type alias style. - Added support for naming lambda init-captures (e.g. `[Captured = Var]`) via the new `LambdaCapture` options. Simple, non-init captures continue to follow the naming style of the variable they capture. - Added the {option}`AllowTrailingUnderscore`, which permits a single trailing underscore on any identifier. - Improved {doc}`readability-named-parameter ` check by ignoring standard tag types (e.g. `std::in_place_t`, `std::allocator_arg_t`, `std::nothrow_t`, iterator tags, lock tags, etc.) that are used exclusively for overload resolution. Added the {option}`IgnoredTypes` option to allow customizing the set of ignored types. - Improved {doc}`readability-non-const-parameter ` check by fixing false positives on pointers passed to atomic builtins, whose operands may be written to, such as the `expected` parameter of `atomic_compare_exchange_strong()`. - Improved {doc}`readability-redundant-parentheses ` check: - Fixed a false positive on the required parentheses of `typeof` and `typeof_unqual` operands. - Fixed false positives and incorrect fixes caused by synthetic parentheses in reference non-type template parameter uses, `__builtin_dump_struct` calls, and OpenMP `linear` clauses. - Fixed {doc}`readability-simplify-boolean-expr ` producing invalid fixes when applying De Morgan's theorem to overloaded comparison operators. - Improved {doc}`readability-suspicious-call-argument ` check by fixing the default `dist` and `dst` abbreviations of `distance` not being recognized. - Improved {doc}`readability-trailing-comma ` check: - Fixed false positives on designated initializers, where initializer lists synthesized for intermediate subobjects caused the trailing comma of the enclosing list to be incorrectly rewritten. - Ignored preprocessor directives such as `#endif` that appear immediately before an enum's closing brace, which previously produced a false positive and a fix-it that inserted a comma after the directive. - Fixed a false positive on empty brace initializers of types with default member initializers. - Improved {doc}`readability-use-std-min-max ` check by fixing spurious trailing semicolons and lost comments when the `if` body has no braces. #### Removed checks - Removed the deprecated `zircon-temporary-objects` check. Users should migrate to {doc}`fuchsia-temporary-objects `. #### Miscellaneous - Fixed {program}`clang-tidy` adding an unwanted blank line when automatically inserting a missing `#include`, which stopped {program}`clang-format` and the `llvm-include-order` check from being able to sort it correctly afterwards. ### Improvements to include-fixer ### Improvements to clang-include-fixer ### Improvements to modularize ### Improvements to pp-trace ### Clang-tidy Visual Studio plugin --- ### Clang Tools Extra Docs ReleaseNotesTemplate (`clang-tools-extra/docs/ReleaseNotesTemplate.txt`) --- myst: enable_extensions: - attrs_block - colon_fence - substitution --- % If you want to modify sections/contents permanently, you should modify both % ReleaseNotes.md and ReleaseNotesTemplate.txt. {#extra-clang-tools-release-releasenotestitle} # Extra Clang Tools {{env.config.release}} {{ (('(In-Progress) ' if env.app.tags.has('PreRelease') else '') ~ 'Release Notes') }} ```{contents} :depth: 3 :local: true ``` Written by the [LLVM Team](https://llvm.org/) ::::{only} PreRelease :::{warning} These are in-progress notes for the upcoming Extra Clang Tools {{env.config.version}} release. Release notes for previous releases can be found on [the Download Page](https://releases.llvm.org/download.html). ::: :::: ## Introduction This document contains the release notes for the Extra Clang Tools, part of the Clang release {{env.config.release}}. Here we describe the status of the Extra Clang Tools in some detail, including major improvements from the previous release and new feature work. All LLVM releases may be downloaded from the [LLVM releases web site](https://llvm.org/releases/). For more information about Clang or LLVM, including information about the latest release, please see the [Clang Web Site](https://clang.llvm.org) or the [LLVM Web Site](https://llvm.org). Note that if you are reading this file from a Git checkout or the main Clang web page, this document applies to the *next* release, not the current one. To see the release notes for a specific release, please see the [releases page](https://llvm.org/releases/). {#what-s-new-in-extra-clang-tools-release} ## What's New in Extra Clang Tools {{env.config.release}}? Some of the major new features and improvements to Extra Clang Tools are listed here. Generic improvements to Extra Clang Tools as a whole or to its underlying infrastructure are described first, followed by tool-specific sections. ### Major New Features ### Potentially Breaking Changes ### Improvements to clangd #### Inlay hints #### Diagnostics #### Semantic Highlighting #### Compile flags #### Hover #### Code completion #### Code actions #### Signature help #### Cross-references #### Objective-C #### Miscellaneous ### Improvements to clang-doc ### Improvements to clang-query ### Improvements to clang-tidy #### New checks #### New check aliases #### Changes in existing checks #### Removed checks #### Miscellaneous ### Improvements to include-fixer ### Improvements to clang-include-fixer ### Improvements to modularize ### Improvements to pp-trace ### Clang-tidy Visual Studio plugin --- ### Clang Tools Extra Docs Clang Change Namespace (`clang-tools-extra/docs/clang-change-namespace.md`) # Clang-Change-Namespace {program}`clang-change-namespace` can be used to change the surrounding namespaces of class/function definitions. Classes/functions in the moved namespace will have new namespaces while references to symbols (e.g. types, functions) which are not defined in the changed namespace will be correctly qualified by prepending namespace specifiers before them. This will try to add shortest namespace specifiers possible. When a symbol reference needs to be fully-qualified, this adds a `::` prefix to the namespace specifiers unless the new namespace is the global namespace. For classes, only classes that are declared/defined in the given namespace in specified files will be moved: forward declarations will remain in the old namespace. The will be demonstrated in the next example. ## Example usage For example, consider this `test.cc` example here with the forward declared class `FWD` and the defined class `A`, both in the namespace `a`. ```c++ namespace a { class FWD; class A { FWD *fwd; }; } // namespace a ``` And now let's change the namespace `a` to `x`. ```console clang-change-namespace \ --old_namespace "a" \ --new_namespace "x" \ --file_pattern "test.cc" \ --i \ test.cc ``` Note that in the code below there's still the forward decalred class `FWD` that stayed in the namespace `a`. It wasn't moved to the new namespace because it wasn't defined/declared here in `a` but only forward declared. ```c++ namespace a { class FWD; } // namespace a namespace x { class A { a::FWD *fwd; }; } // namespace x ``` ## Another example Consider this `test.cc` file: ```c++ namespace na { class X {}; namespace nb { class Y { X x; }; } // namespace nb } // namespace na ``` To move the definition of class `Y` from namespace `na::nb` to `x::y`, run: ```console clang-change-namespace \ --old_namespace "na::nb" \ --new_namespace "x::y" \ --file_pattern "test.cc" \ --i \ test.cc ``` This will overwrite `test.cc` to look like this: ```c++ namespace na { class X {}; } // namespace na namespace x { namespace y { class Y { na::X x; }; } // namespace y } // namespace x ``` Note, that we've successfully moved the class `Y` from namespace `na::nb` to namespace `x::y`. ### Caveats ## Content already exists in new namespace Consider this `test.cc` example that defines two `class A` one inside the namespace `a` and one in namespace `b`: ```c++ namespace a { class A { int classAFromWithinNamespace_a; }; } // namespace a namespace b { class A { int classAFromWithinNamespace_b; }; } //namespace b ``` Let's move everything from namespace `a` to namespace `b`: ```console clang-change-namespace \ --old_namespace "a" \ --new_namespace "b" \ --file_pattern test.cc \ test.cc ``` As expected we now have to definitions of `class A` inside the namespace `b`: ```c++ namespace b { class A { int classAFromWithinNamespace_a; }; } // namespace b namespace b { class A { int classAFromWithinNamespace_b; }; } //namespace b ``` The re-factoring looks correct but the code will not compile due to the name duplication. It is not up to the tool to ensure compilability in that sense. But one has to be aware of that. ## Inline namespace doesn't work Consider this usage of two versions of implementations for a `greet` function: ```c++ #include namespace Greeter { inline namespace Version1 { const char* greet() { return "Hello from version 1!"; } } // namespace Version1 namespace Version2 { const char* greet() { return "Hello from version 2!"; } } // namespace Version2 } // namespace Greeter int main(int argc, char* argv[]) { printf("%s\n", Greeter::greet()); return 0; } ``` Note, that currently `Greeter::greet()` will result in a call to `Greeter::Version1::greet()` because that's the inlined namespace. Let's say you want to move one and make `Version2` the default now and remove the `inline` from the `Version1`. First let's try to turn `namespace Version2` into `inline namespace Version2`: ```console clang-change-namespace \ --old_namespace "Greeter::Version2" \ --new_namespace "inline Version2" \ --file_pattern main.cc main.cc ``` But this will put the `inline` keyword in the wrong place resulting in: ```c++ #include namespace Greeter { inline namespace Version1 { const char* greet() { return "Hello from version 1!"; } } // namespace Version1 } // namespace Greeter namespace inline Greeter { namespace Version2 { const char *greet() { return "Hello from version 2!"; } } // namespace Version2 } // namespace inline Greeter int main(int argc, char* argv[]) { printf("%s\n", Greeter::greet()); return 0; } ``` One cannot use {program}`clang-change-namespace` to inline a namespace. ## Symbol references not updated Consider this `test.cc` file: ```c++ namespace old { struct foo {}; } // namespace old namespace b { old::foo g_foo; } // namespace b ``` Notice that namespace `b` defines a global variable of type `old::foo`. If we now change the name of the `old` namespace to `modern`, the reference will not be updated: ```console clang-change-namespace \ --old_namespace "old" \ --new_namespace "modern" \ --file_pattern test.cc \ test.cc ``` ```c++ namespace modern { struct foo {}; } // namespace modern namespace b { old::foo g_foo; } // namespace b ``` `g_foo` is still of the no longer existing type `old::foo` while instead it should use `modern::foo`. Only symbol references in the moved namespace are updated, not outside of it. ### {program}`clang-change-namespace` Command Line Options :::{option} --allowed_file= A file containing regexes of symbol names that are not expected to be updated when changing namespaces around them. ::: :::{option} --dump_result Dump new file contents in YAML, if specified. ::: :::{option} --extra-arg= Additional argument to append to the compiler command line ::: :::{option} --extra-arg-before= Additional argument to prepend to the compiler command line ::: :::{option} --file_pattern= Only rename namespaces in files that match the given regular expression pattern. ::: :::{option} -i Inplace edit ``s, if specified. ::: :::{option} --new_namespace= New namespace. Use `""` when you target the global namespace. ::: :::{option} --old_namespace= Old namespace. ::: :::{option} -p Build path ::: :::{option} --style= The style name used for reformatting. ::: --- ### Clang Tools Extra Docs Clang Doc (`clang-tools-extra/docs/clang-doc.md`) # Clang-Doc {program}`clang-doc` is a tool for generating C and C++ documentation from source code and comments. The tool is in a very early development stage, so you might encounter bugs and crashes. Submitting reports with information about how to reproduce the issue to [the LLVM bug tracker](https://github.com/llvm/llvm-project/issues/) will definitely help the project. If you have any ideas or suggestions, please to put a feature request there. ## Use {program}`clang-doc` is a [LibTooling](https://clang.llvm.org/docs/LibTooling.html)-based tool, and so requires a compile command database for your project (for an example of how to do this see [How To Setup Tooling For LLVM](https://clang.llvm.org/docs/HowToSetupToolingForLLVM.html)). The tool will process a list of files by default: ```console $ clang-doc File1.cpp File2.cpp ... FileN.cpp ``` The tool can be also used with a compile commands database: ```console $ clang-doc --executor=all-TUs compile_commands.json ``` To select only a subset of files from the database, use the `--filter` flag: ```console $ clang-doc --executor=all-TUs --filter=File[0-9]+.cpp compile_commands.json ``` ## Output {program}`clang-doc` produces a directory of documentation. One file is produced for each namespace and record in the project source code, containing all documentation (including contained functions, methods, and enums) for that item. The top-level directory is configurable through the `output` flag: ```console $ clang-doc --output=output/directory/ compile_commands.json ``` ## Configuration Configuration for {program}`clang-doc` is currently limited to command-line options. In the future, it may develop the ability to use a configuration file, but no such efforts are currently in progress. ### Options {program}`clang-doc` offers the following options: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` The following flags should only be used if `format` is set to `html`: - `repository` - `source-root` - `stylesheets` --- ### Clang Tools Extra Docs Clang Include Fixer (`clang-tools-extra/docs/clang-include-fixer.md`) # Clang-Include-Fixer One of the major nuisances of C++ compared to other languages is the manual management of `#include` directives in any file. {program}`clang-include-fixer` addresses one aspect of this problem by providing an automated way of adding `#include` directives for missing symbols in one translation unit. While inserting missing `#include`, {program}`clang-include-fixer` adds missing namespace qualifiers to all instances of an unidentified symbol if the symbol is missing some prefix namespace qualifiers. ## Setup To use {program}`clang-include-fixer` two databases are required. Both can be generated with existing tools. - Compilation database. Contains the compiler commands for any given file in a project and can be generated by CMake, see [How To Setup Tooling For LLVM][how to setup tooling for llvm]. - Symbol index. Contains all symbol information in a project to match a given identifier to a header file. Ideally both databases (`compile_commands.json` and `find_all_symbols_db.yaml`) are linked into the root of the source tree they correspond to. Then the {program}`clang-include-fixer` can automatically pick them up if called with a source file from that tree. Note that by default `compile_commands.json` as generated by CMake does not include header files, so only implementation files can be handled by tools. ### Creating a Symbol Index From a Compilation Database The include fixer contains {program}`find-all-symbols`, a tool to create a symbol database in YAML format from a compilation database by parsing all source files listed in it. The following list of commands shows how to set up a database for LLVM, any project built by CMake should follow similar steps. ```console $ cd path/to/llvm-build $ ninja find-all-symbols // build find-all-symbols tool. $ ninja clang-include-fixer // build clang-include-fixer tool. $ ls compile_commands.json # Make sure compile_commands.json exists. compile_commands.json $ path/to/llvm/source/clang-tools-extra/clang-include-fixer/find-all-symbols/tool/run-find-all-symbols.py ... wait as clang indexes the code base ... $ ln -s $PWD/find_all_symbols_db.yaml path/to/llvm/source/ # Link database into the source tree. $ ln -s $PWD/compile_commands.json path/to/llvm/source/ # Also link compilation database if it's not there already. $ cd path/to/llvm/source $ /path/to/clang-include-fixer -db=yaml path/to/file/with/missing/include.cpp Added #include "foo.h" ``` ### Integrate with Vim To run `clang-include-fixer` on a potentially unsaved buffer in Vim. Add the following key binding to your `.vimrc`: ```console noremap cf :pyf path/to/llvm/source/clang-tools-extra/clang-include-fixer/tool/clang-include-fixer.py ``` This enables `clang-include-fixer` for NORMAL and VISUAL mode. Change `cf` to another binding if you need clang-include-fixer on a different key. The [\ key]() is a reference to a specific key defined by the mapleader variable and is bound to backslash by default. Make sure vim can find {program}`clang-include-fixer`: - Add the path to {program}`clang-include-fixer` to the PATH environment variable. - Or set `g:clang_include_fixer_path` in vimrc: `let g:clang_include_fixer_path=path/to/clang-include-fixer` You can customize the number of headers being shown by setting `let g:clang_include_fixer_maximum_suggested_headers=5` Customized settings in `.vimrc`: - `let g:clang_include_fixer_path = "clang-include-fixer"` Set clang-include-fixer binary file path. - `let g:clang_include_fixer_maximum_suggested_headers = 3` Set the maximum number of `#includes` to show. Default is 3. - `let g:clang_include_fixer_increment_num = 5` Set the increment number of #includes to show every time when pressing `m`. Default is 5. - `let g:clang_include_fixer_jump_to_include = 0` Set to 1 if you want to jump to the new inserted `#include` line. Default is 0. - `let g:clang_include_fixer_query_mode = 0` Set to 1 if you want to insert `#include` for the symbol under the cursor. Default is 0. Compared to normal mode, this mode won't parse the source file and only search the symbol from database, which is faster than normal mode. See `clang-include-fixer.py` for more details. ### Integrate with Emacs To run `clang-include-fixer` on a potentially unsaved buffer in Emacs. Ensure that Emacs finds `clang-include-fixer.el` by adding the directory containing the file to the `load-path` and requiring the `clang-include-fixer` in your `.emacs`: ```console (add-to-list 'load-path "path/to/llvm/source/clang-tools-extra/clang-include-fixer/tool/" (require 'clang-include-fixer) ``` Within Emacs the tool can be invoked with the command `M-x clang-include-fixer`. This will insert the header that defines the first undefined symbol; if there is more than one header that would define the symbol, the user is prompted to select one. To include the header that defines the symbol at point, run `M-x clang-include-fixer-at-point`. Make sure Emacs can find {program}`clang-include-fixer`: - Either add the parent directory of {program}`clang-include-fixer` to the PATH environment variable, or customize the Emacs user option `clang-include-fixer-executable` to point to the file name of the program. ## How it Works To get the most information out of Clang at parse time, {program}`clang-include-fixer` runs in tandem with the parse and receives callbacks from Clang's semantic analysis. In particular it reuses the existing support for typo corrections. Whenever Clang tries to correct a potential typo it emits a callback to the include fixer which then looks for a corresponding file. At this point rich lookup information is still available, which is not available in the AST at a later stage. The identifier that should be typo corrected is then sent to the database, if a header file is returned it is added as an include directive at the top of the file. Currently {program}`clang-include-fixer` only inserts a single include at a time to avoid getting caught in follow-up errors. If multiple `#include` additions are desired the program can be rerun until a fix-point is reached. [how to setup tooling for llvm]: https://clang.llvm.org/docs/HowToSetupToolingForLLVM.html --- ### Clang Tools Extra Docs Clang Modernize (`clang-tools-extra/docs/clang-modernize.md`) --- orphan: true --- All {program}`clang-modernize` transforms have moved to {doc}`clang-tidy/index` (see the `modernize` module). --- ### Clang Tools Extra Docs Clang Reorder Fields (`clang-tools-extra/docs/clang-reorder-fields.md`) # Clang-Reorder-Fields {program}`clang-reorder-fields` is a refactoring tool to reorder fields in C/C++ structs and classes. This tool automatically updates: - Field declarations in the record definition - Constructor initializer lists in C++ classes - Aggregate initialization expressions (both C and C++) - Designated initializer lists (C++20) This can be useful for optimizing memory layout, improving cache performance, or conforming to coding standards that require specific field orderings. ## Example usage ### Basic struct reordering Consider this simple struct in `example.c`: ```c struct Foo { const int *x; int y; double z; int w; }; int main() { const int val = 42; struct Foo foo = { &val, 0, 1.5, 17 }; return 0; } ``` To reorder the fields to `z, w, y, x`, run: ```console clang-reorder-fields -record-name Foo -fields-order z,w,y,x example.c -- ``` This will reorder both the struct definition and the initialization: ```c struct Foo { double z; int w; int y; const int *x; }; int main() { const int val = 42; struct Foo foo = { 1.5, 17, 0, &val }; return 0; } ``` ### Namespaced structs For C++ code with namespaces, use the fully-qualified name: ```c++ namespace bar { struct Foo { const int *x; int y; double z; int w; }; } ``` ```console clang-reorder-fields -record-name ::bar::Foo -fields-order z,w,y,x example.cpp -- ``` For classes defined in the global namespace (without any namespace), you can use either the simple class name or prefix it with `::`: ```console clang-reorder-fields -record-name Foo -fields-order z,w,y,x example.cpp -- # or clang-reorder-fields -record-name ::Foo -fields-order z,w,y,x example.cpp -- ``` ### C++ constructor initializer lists The tool also reorders constructor initializer lists. Given: ```c++ class Foo { public: Foo(); private: int x; const char *s1; const char *s2; double z; }; Foo::Foo(): x(12), s1("abc"), s2("def"), z(3.14) {} ``` Running: ```console clang-reorder-fields -record-name Foo -fields-order s1,x,z,s2 example.cpp -- ``` Will reorder both the field declarations and the constructor initializers: ```c++ class Foo { public: Foo(); private: const char *s1; int x; double z; const char *s2; }; Foo::Foo(): s1("abc"), x(12), z(3.14), s2("def") {} ``` ### Designated initializers For C++20 code using designated initializers: ```c++ struct Bar { char a; int b; int c; }; int main() { Bar bar1 = { 'a', 0, 123 }; Bar bar2 = { .a = 'a', .b = 0, .c = 123 }; return 0; } ``` ```console clang-reorder-fields --extra-arg="-std=c++20" -record-name Bar \ -fields-order c,a,b example.cpp -- ``` Will produce: ```c++ struct Bar { int c; char a; int b; }; int main() { Bar bar1 = { 123, 'a', 0 }; Bar bar2 = { .c = 123, .a = 'a', .b = 0 }; return 0; } ``` ### In-place editing Use the `-i` flag to modify files in-place: ```console clang-reorder-fields -record-name Foo -fields-order z,w,y,x -i example.c -- ``` ## Limitations and Caveats ### Different access specifiers The tool cannot reorder fields with different access specifiers (`public/private/protected`). All fields being reordered must have the same access level. ```c++ class Example { private: int x; public: int y; // Cannot reorder x and y - different access levels }; ``` ### Multiple field declarations Declarations with multiple fields in one statement are not supported: ```c struct Example { int a, b; // Not supported - multiple fields in one declaration }; ``` ### Macro-expanded fields Macros that expand to multiple field declarations are not supported. However, macros that expand to a single field declaration work correctly: ```c #define INT_FIELD(NAME) int NAME // Supported - expands to one field #define TWO_FIELDS int a; int b; // Not supported - expands to two fields struct Supported { INT_FIELD(x); // OK - this is a single field int y; INT_FIELD(z); // OK - this is a single field }; struct NotSupported { TWO_FIELDS // Not OK - expands to multiple fields int c; }; ``` The tool can reorder fields declared via macros as long as each macro invocation expands to exactly one field declaration. ### Preprocessor directives Structs with preprocessor directives between fields cannot be reordered: ```c struct Example { int a; #ifdef FEATURE int b; #endif int c; // Not supported - preprocessor directives present }; ``` ### Flexible array members In C, a flexible array member is an incomplete array type that must be the last member of a struct (as specified by C99 and later standards). This allows the struct to have a variable-length array at the end. Since this is a language requirement, the tool enforces that flexible array members remain in the last position: ```c struct Example { int count; int data[]; // Flexible array member - must remain last }; ``` Attempting to reorder fields such that the flexible array member is no longer last will result in an error: ```console clang-reorder-fields -record-name Example -fields-order data,count example.c -- ``` Will produce: ```text Flexible array member must remain the last field in the struct ``` This ensures the generated code remains valid C. ### Field dependencies in initializers The tool will issue a warning if reordering causes a field to be used in an initializer before it's initialized. Consider this example: ```c++ class Foo { public: Foo(int x, char c); int x; char c; Dummy z; }; Foo::Foo(int x, char c) : x(x), c(c), z(this->x, c) // z's initializer uses x and c {} ``` If you reorder the fields to `z, c, x`: ```console clang-reorder-fields -record-name Foo -fields-order z,c,x example.cpp -- ``` The tool will produce warnings: ```text example.cpp:10:3: warning: reordering field x after z makes x uninitialized when used in init expression example.cpp:10:3: warning: reordering field c after z makes c uninitialized when used in init expression ``` This warns you that in C++, member initializers are executed in the order that fields are declared in the class, not the order they appear in the initializer list. After reordering, `z` would be initialized first, but its initializer tries to use `x` and `c` which haven't been initialized yet. The tool will still perform the reordering but warns about the potential issue. You should review these warnings and adjust your code accordingly. ## {program}`clang-reorder-fields` Command Line Options :::{option} --record-name= The fully-qualified name of the struct or class to reorder. Required. For C structs, use the struct name directly (e.g., `Foo`). For C++ classes/structs in namespaces, use the fully-qualified name including namespaces (e.g., `::namespace::ClassName`). For C++ classes/structs in the global namespace, you can use either the simple name (e.g., `Foo`) or prefix with `::` (e.g., `::Foo`). ::: :::{option} --fields-order= Comma-separated list of field names in the desired order. Required. All field names must exactly match the fields in the struct/class definition. The number of fields must match the number in the definition. ::: :::{option} -i Overwrite edited files in-place. If not specified, the rewritten code is printed to stdout. ::: :::{option} --extra-arg= Additional argument to append to the compiler command line. Useful for specifying language standards (e.g., `--extra-arg="-std=c++20"`). ::: :::{option} --extra-arg-before= Additional argument to prepend to the compiler command line. ::: :::{option} -p Build path. Specifies the directory containing `compile_commands.json` for compilation database support. ::: ## Use Cases ### Memory layout optimization Reorder fields to minimize padding and improve cache locality: ```c // Before: 24 bytes (with padding) struct Data { char a; // 1 byte + 7 padding double b; // 8 bytes char c; // 1 byte + 7 padding }; ``` ```console clang-reorder-fields -record-name Data -fields-order b,a,c data.c -- ``` ```c // After: 16 bytes (less padding) struct Data { double b; // 8 bytes char a; // 1 byte char c; // 1 byte + 6 padding }; ``` ### Coding standard compliance Ensure fields are ordered according to project conventions (e.g., alphabetically, by type, or by access pattern). ### Field grouping Group related fields together for better code organization and readability. --- ### Clang Tools Extra Docs Clang Tidy (`clang-tools-extra/docs/clang-tidy.md`) --- orphan: true --- ```{eval-rst} .. meta:: :http-equiv=refresh: 0;URL='clang-tidy/' ``` clang-tidy documentation has moved here: [clang-tidy](clang-tidy/index.rst) --- ### Clang Tools Extra Docs Clangd (`clang-tools-extra/docs/clangd.md`) --- orphan: true redirect_target: https://clangd.llvm.org/ template: clangd_redirect.html --- --- ### Clang Tools Extra Docs Cpp11 Migrate (`clang-tools-extra/docs/cpp11-migrate.md`) --- orphan: true --- All {program}`clang-modernize` transforms have moved to {doc}`clang-tidy/index` (see the `modernize` module). --- ### Clang Tools Extra Docs Modularize (`clang-tools-extra/docs/modularize.md`) ```{index} modularize ``` # Modularize User's Manual ```{toctree} :hidden: true ModularizeUsage ``` {program}`modularize` is a standalone tool that checks whether a set of headers provides the consistent definitions required to use modules. For example, it detects whether the same entity (say, a NULL macro or size_t typedef) is defined in multiple headers or whether a header produces different definitions under different circumstances. These conditions cause modules built from the headers to behave poorly, and should be fixed before introducing a module map. {program}`modularize` also has an assistant mode option for generating a module map file based on the provided header list. The generated file is a functional module map that can be used as a starting point for a module.modulemap file. ## Getting Started To build from source: 1. Read [Getting Started with the LLVM System][getting started with the llvm system] and [Clang Tools Documentation][clang tools documentation] for information on getting sources for LLVM, Clang, and Clang Extra Tools. 2. [Getting Started with the LLVM System][getting started with the llvm system] and [Building LLVM with CMake][building llvm with cmake] give directions for how to build. With sources all checked out into the right place the LLVM build will build Clang Extra Tools and their dependencies automatically. - If using CMake, you can also use the `modularize` target to build just the modularize tool and its dependencies. Before continuing, take a look at {doc}`ModularizeUsage` to see how to invoke modularize. ## What Modularize Checks Modularize will check for the following: - Duplicate global type and variable definitions - Duplicate macro definitions - Macro instances, 'defined(macro)', or #if, #elif, #ifdef, #ifndef conditions that evaluate differently in a header - #include directives inside 'extern "C/C++" {}' or 'namespace (name) {}' blocks - Module map header coverage completeness (in the case of a module map input only) Modularize will do normal C/C++ parsing, reporting normal errors and warnings, but will also report special error messages like the following: ``` error: '(symbol)' defined at multiple locations: (file):(row):(column) (file):(row):(column) error: header '(file)' has different contents depending on how it was included ``` The latter might be followed by messages like the following: ``` note: '(symbol)' in (file) at (row):(column) not always provided ``` Checks will also be performed for macro expansions, defined(macro) expressions, and preprocessor conditional directives that evaluate inconsistently, and can produce error messages like the following: ``` (...)/SubHeader.h:11:5: #if SYMBOL == 1 ^ error: Macro instance 'SYMBOL' has different values in this header, depending on how it was included. 'SYMBOL' expanded to: '1' with respect to these inclusion paths: (...)/Header1.h (...)/SubHeader.h (...)/SubHeader.h:3:9: #define SYMBOL 1 ^ Macro defined here. 'SYMBOL' expanded to: '2' with respect to these inclusion paths: (...)/Header2.h (...)/SubHeader.h (...)/SubHeader.h:7:9: #define SYMBOL 2 ^ Macro defined here. ``` Checks will also be performed for '#include' directives that are nested inside 'extern "C/C++" {}' or 'namespace (name) {}' blocks, and can produce error message like the following: ``` IncludeInExtern.h:2:3: #include "Empty.h" ^ error: Include directive within extern "C" {}. IncludeInExtern.h:1:1: extern "C" { ^ The "extern "C" {}" block is here. ``` (module-map-coverage)= ## Module Map Coverage Check The coverage check uses the Clang library to read and parse the module map file. Starting at the module map file directory, or just the include paths, if specified, it will collect the names of all the files it considers headers (no extension, .h, or .inc--if you need more, modify the isHeader function). It then compares the headers against those referenced in the module map, either explicitly named, or implicitly named via an umbrella directory or umbrella file, as parsed by the ModuleMap object. If headers are found which are not referenced or covered by an umbrella directory or file, warning messages will be produced, and this program will return an error code of 1. If no problems are found, an error code of 0 is returned. Note that in the case of umbrella headers, this tool invokes the compiler to preprocess the file, and uses a callback to collect the header files included by the umbrella header or any of its nested includes. If any front end options are needed for these compiler invocations, these can be included on the command line after the module map file argument. Warning message have the form: > warning: module.modulemap does not account for file: Level3A.h Note that for the case of the module map referencing a file that does not exist, the module map parser in Clang will (at the time of this writing) display an error message. To limit the checks {program}`modularize` does to just the module map coverage check, use the `-coverage-check-only option`. For example: ``` modularize -coverage-check-only module.modulemap ``` (module-map-generation)= ## Module Map Generation If you specify the `-module-map-path=`, {program}`modularize` will output a module map based on the input header list. A module will be created for each header. Also, if the header in the header list is a partial path, a nested module hierarchy will be created in which a module will be created for each subdirectory component in the header path, with the header itself represented by the innermost module. If other headers use the same subdirectories, they will be enclosed in these same modules also. For example, for the header list: ``` SomeTypes.h SomeDecls.h SubModule1/Header1.h SubModule1/Header2.h SubModule2/Header3.h SubModule2/Header4.h SubModule2.h ``` The following module map will be generated: ``` // Output/NoProblemsAssistant.txt // Generated by: modularize -module-map-path=Output/NoProblemsAssistant.txt \ -root-module=Root NoProblemsAssistant.modularize module SomeTypes { header "SomeTypes.h" export * } module SomeDecls { header "SomeDecls.h" export * } module SubModule1 { module Header1 { header "SubModule1/Header1.h" export * } module Header2 { header "SubModule1/Header2.h" export * } } module SubModule2 { module Header3 { header "SubModule2/Header3.h" export * } module Header4 { header "SubModule2/Header4.h" export * } header "SubModule2.h" export * } ``` An optional `-root-module=` option can be used to cause a root module to be created which encloses all the modules. An optional `-problem-files-list=` can be used to input a list of files to be excluded, perhaps as a temporary stop-gap measure until problem headers can be fixed. For example, with the same header list from above: ``` // Output/NoProblemsAssistant.txt // Generated by: modularize -module-map-path=Output/NoProblemsAssistant.txt \ -root-module=Root NoProblemsAssistant.modularize module Root { module SomeTypes { header "SomeTypes.h" export * } module SomeDecls { header "SomeDecls.h" export * } module SubModule1 { module Header1 { header "SubModule1/Header1.h" export * } module Header2 { header "SubModule1/Header2.h" export * } } module SubModule2 { module Header3 { header "SubModule2/Header3.h" export * } module Header4 { header "SubModule2/Header4.h" export * } header "SubModule2.h" export * } } ``` Note that headers with dependents will be ignored with a warning, as the Clang module mechanism doesn't support headers that rely on other headers being included first. The module map format defines some keywords which can't be used in module names. If a header has one of these names, an underscore ('\_') will be prepended to the name. For example, if the header name is `header.h`, because `header` is a keyword, the module name will be `_header`. For a list of the module map keywords, please see: [Lexical structure](https://clang.llvm.org/docs/Modules.html#lexical-structure) [building llvm with cmake]: https://llvm.org/docs/CMake.html [clang tools documentation]: https://clang.llvm.org/docs/ClangTools.html [getting started with the llvm system]: https://llvm.org/docs/GettingStarted.html --- ### Clang Tools Extra Docs Pp Trace (`clang-tools-extra/docs/pp-trace.md`) ```{index} pp-trace ``` # pp-trace User's Manual {program}`pp-trace` is a standalone tool that traces preprocessor activity. It's also used as a test of Clang's PPCallbacks interface. It runs a given source file through the Clang preprocessor, displaying selected information from callback functions overridden in a [PPCallbacks](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html) derivation. The output is in a high-level YAML format, described in {ref}`OutputFormat`. (usage)= ## pp-trace Usage ### Command Line Format `pp-trace [] [-- ]` `` is a place-holder for options specific to pp-trace, which are described below in {ref}`CommandLineOptions`. `` specifies the source file to run through the preprocessor. `` is a place-holder for regular [Clang Compiler Options](https://clang.llvm.org/docs/UsersManual.html#command-line-options), which must follow the \. (commandlineoptions)= ### Command Line Options :::{option} -callbacks This option specifies a comma-separated list of globs describing the list of callbacks that should be traced. Globs are processed in order of appearance. Positive globs add matched callbacks to the set, negative globs (those with the '-' prefix) remove callacks from the set. - FileChanged - FileSkipped - InclusionDirective - moduleImport - EndOfMainFile - Ident - PragmaDirective - PragmaComment - PragmaDetectMismatch - PragmaDebug - PragmaMessage - PragmaDiagnosticPush - PragmaDiagnosticPop - PragmaDiagnostic - PragmaOpenCLExtension - PragmaWarning - PragmaWarningPush - PragmaWarningPop - MacroExpands - MacroDefined - MacroUndefined - Defined - SourceRangeSkipped - If - Elif - Ifdef - Ifndef - Else - Endif ::: :::{option} -output By default, pp-trace outputs the trace information to stdout. Use this option to output the trace information to a file. ::: (outputformat)= ## pp-trace Output Format The pp-trace output is formatted as YAML. See for general YAML information. It's arranged as a sequence of information about the callback call, including the callback name and argument information, for example: ```yaml --- - Callback: Name Argument1: Value1 Argument2: Value2 (etc.) ... ``` With real data: ```yaml --- - Callback: FileChanged Loc: "c:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-include.cpp:1:1" Reason: EnterFile FileType: C_User PrevFID: (invalid) (etc.) - Callback: FileChanged Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-include.cpp:5:1" Reason: ExitFile FileType: C_User PrevFID: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/Input/Level1B.h" - Callback: EndOfMainFile ... ``` In all but one case (MacroDirective) the "Argument" scalars have the same name as the argument in the corresponding PPCallbacks callback function. ### Callback Details The following sections describe the purpose and output format for each callback. Click on the callback name in the section heading to see the Doxygen documentation for the callback. The argument descriptions table describes the callback argument information displayed. The Argument Name field in most (but not all) cases is the same name as the callback function parameter. The Argument Value Syntax field describes the values that will be displayed for the argument value. It uses an ad hoc representation that mixes literal and symbolic representations. Enumeration member symbols are shown as the actual enum member in a (member1|member2|...) form. A name in parentheses can either represent a place holder for the described value, or confusingly, it might be a literal, such as (null), for a null pointer. Locations are shown as quoted only to avoid confusing the documentation generator. The Clang C++ Type field is the type from the callback function declaration. The description describes the argument or what is displayed for it. Note that in some cases, such as when a structure pointer is an argument value, only some key member or members are shown to represent the value, instead of trying to display all members of the structure. #### [FileChanged](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#a7cc8cfaf34114fc65e92af621cd6464e) Callback FileChanged is called when the preprocessor enters or exits a file, both the top level file being compiled, as well as any #include directives. It will also be called as a result of a system header pragma or in internal renaming of a file. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | ----------------------------------------------------- | ----------------------------- | ------------------------------ | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | Reason | (EnterFile\|ExitFile\|SystemHeaderPragma\|RenameFile) | PPCallbacks::FileChangeReason | Reason for change. | | FileType | (C_User\|C_System\|C_ExternCSystem) | SrcMgr::CharacteristicKind | Include type. | | PrevFID | ((file)\|(invalid)) | FileID | Previous file, if any. | Example: ```yaml - Callback: FileChanged Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-include.cpp:1:1" Reason: EnterFile FileType: C_User PrevFID: (invalid) ``` #### [FileSkipped](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#ab5b338a0670188eb05fa7685bbfb5128) Callback FileSkipped is called when a source file is skipped as the result of header guard optimization. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | ----------------------------------- | -------------------------- | -------------------------------------------------------- | | ParentFile | ("(file)" or (null)) | const FileEntry | The file that #included the skipped file. | | FilenameTok | (token) | const Token | The token in ParentFile that indicates the skipped file. | | FileType | (C_User\|C_System\|C_ExternCSystem) | SrcMgr::CharacteristicKind | The file type. | Example: ```yaml - Callback: FileSkipped ParentFile: "/path/filename.h" FilenameTok: "filename.h" FileType: C_User ``` #### [InclusionDirective](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#a557d9738c329793513a6f57d6b60de52) Callback InclusionDirective is called when an inclusion directive of any kind (#include\, #import\, etc.) has been processed, regardless of whether the inclusion will actually result in an inclusion. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | ----------------------- | --------------- | ------------------------------------------------------------------------------------------------------------ | | HashLoc | "(file):(line):(col)" | SourceLocation | The location of the '#' that starts the inclusion directive. | | IncludeTok | (token) | const Token | The token that indicates the kind of inclusion directive, e.g., 'include' or 'import'. | | FileName | "(file)" | StringRef | The name of the file being included, as written in the source code. | | IsAngled | (true\|false) | bool | Whether the file name was enclosed in angle brackets; otherwise, it was enclosed in quotes. | | FilenameRange | "(file)" | CharSourceRange | The character range of the quotes or angle brackets for the written file name. | | File | "(file)" | const FileEntry | The actual file that may be included by this inclusion directive. | | SearchPath | "(path)" | StringRef | Contains the search path which was used to find the file in the file system. | | RelativePath | "(path)" | StringRef | The path relative to SearchPath, at which the include file was found. | | Imported | ((module name)\|(null)) | const Module | The module, whenever an inclusion directive was automatically turned into a module import or null otherwise. | Example: ```yaml - Callback: InclusionDirective HashLoc: "D:/Clang/llvmnewmod/clang-tools-extra/test/pp-trace/pp-trace-include.cpp:4:1" IncludeTok: include FileName: "Input/Level1B.h" IsAngled: false FilenameRange: "Input/Level1B.h" File: "D:/Clang/llvmnewmod/clang-tools-extra/test/pp-trace/Input/Level1B.h" SearchPath: "D:/Clang/llvmnewmod/clang-tools-extra/test/pp-trace" RelativePath: "Input/Level1B.h" Imported: (null) ``` #### [moduleImport](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#af32dcf1b8b7c179c7fcd3e24e89830fe) Callback moduleImport is called when there was an explicit module-import syntax. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | ----------------------- | -------------- | ----------------------------------------------------------- | | ImportLoc | "(file):(line):(col)" | SourceLocation | The location of import directive token. | | Path | "(path)" | ModuleIdPath | The identifiers (and their locations) of the module "path". | | Imported | ((module name)\|(null)) | const Module | The imported module; can be null if importing failed. | Example: ```yaml - Callback: moduleImport ImportLoc: "d:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-modules.cpp:4:2" Path: [{Name: Level1B, Loc: "d:/Clang/llvmnewmod/clang-tools-extra/test/pp-trace/pp-trace-modules.cpp:4:9"}, {Name: Level2B, Loc: "d:/Clang/llvmnewmod/clang-tools-extra/test/pp-trace/pp-trace-modules.cpp:4:17"}] Imported: Level2B ``` #### [EndOfMainFile](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#a63e170d069e99bc1c9c7ea0f3bed8bcc) Callback EndOfMainFile is called when the end of the main file is reached. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | -------------- | --------------------- | -------------- | ----------- | | (no arguments) | | | | Example: ```yaml - Callback: EndOfMainFile ``` #### [Ident](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#a3683f1d1fa513e9b6193d446a5cc2b66) Callback Ident is called when a #ident or #sccs directive is read. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | --------------------- | ----------------- | ------------------------------ | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | str | (name) | const std::string | The text of the directive. | Example: ```yaml - Callback: Ident Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-ident.cpp:3:1" str: "$Id$" ``` #### [PragmaDirective](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#a0a2d7a72c62184b3cbde31fb62c6f2f7) Callback PragmaDirective is called when start reading any pragma directive. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | ------------------------------------------------ | -------------------- | --------------------------------- | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | Introducer | (PIK_HashPragma\|PIK\_\_Pragma\|PIK\_\_\_pragma) | PragmaIntroducerKind | The type of the pragma directive. | Example: ```yaml - Callback: PragmaDirective Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-pragma.cpp:3:1" Introducer: PIK_HashPragma ``` #### [PragmaComment](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#ace0d940fc2c12ab76441466aab58dc37) Callback PragmaComment is called when a #pragma comment directive is read. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | --------------------- | -------------------- | ------------------------------ | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | Kind | ((name)\|(null)) | const IdentifierInfo | The comment kind symbol. | | Str | (message directive) | const std::string | The comment message directive. | Example: ```yaml - Callback: PragmaComment Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-pragma.cpp:3:1" Kind: library Str: kernel32.lib ``` #### [PragmaDetectMismatch](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#ab11158c9149fb8ad8af1903f4a6cd65d) Callback PragmaDetectMismatch is called when a #pragma detect_mismatch directive is read. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | --------------------- | ----------------- | ------------------------------ | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | Name | "(name)" | const std::string | The name. | | Value | (string) | const std::string | The value. | Example: ```yaml - Callback: PragmaDetectMismatch Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-pragma.cpp:3:1" Name: name Value: value ``` #### [PragmaDebug](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#a57cdccb6dcc07e926513ac3d5b121466) Callback PragmaDebug is called when a #pragma clang \_\_debug directive is read. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | --------------------- | -------------- | -------------------------------- | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | DebugType | (string) | StringRef | Indicates type of debug message. | Example: ```yaml - Callback: PragmaDebug Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-pragma.cpp:3:1" DebugType: warning ``` #### [PragmaMessage](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#abb42935d9a9fd8e2c4f51cfdc4ea2ae1) Callback PragmaMessage is called when a #pragma message directive is read. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | ------------------------------------- | ------------------------------ | --------------------------------------- | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | Namespace | (name) | StringRef | The namespace of the message directive. | | Kind | (PMK_Message\|PMK_Warning\|PMK_Error) | PPCallbacks::PragmaMessageKind | The type of the message directive. | | Str | (string) | StringRef | The text of the message directive. | Example: ```yaml - Callback: PragmaMessage Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-pragma.cpp:3:1" Namespace: "GCC" Kind: PMK_Message Str: The message text. ``` #### [PragmaDiagnosticPush](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#a0f3ff19762baa38fe6c5c58022d32979) Callback PragmaDiagnosticPush is called when a #pragma gcc diagnostic push directive is read. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | --------------------- | -------------- | ------------------------------ | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | Namespace | (name) | StringRef | Namespace name. | Example: ```yaml - Callback: PragmaDiagnosticPush Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-pragma.cpp:3:1" Namespace: "GCC" ``` #### [PragmaDiagnosticPop](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#ac94d789873122221fba8d76f6c5ea45e) Callback PragmaDiagnosticPop is called when a #pragma gcc diagnostic pop directive is read. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | --------------------- | -------------- | ------------------------------ | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | Namespace | (name) | StringRef | Namespace name. | Example: ```yaml - Callback: PragmaDiagnosticPop Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-pragma.cpp:3:1" Namespace: "GCC" ``` #### [PragmaDiagnostic](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#afe7938f38a83cb7b4b25a13edfdd7bdd) Callback PragmaDiagnostic is called when a #pragma gcc diagnostic directive is read. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | -------------------------------------------------- | -------------- | ------------------------------ | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | Namespace | (name) | StringRef | Namespace name. | | mapping | (0\|MAP_IGNORE\|MAP_WARNING\|MAP_ERROR\|MAP_FATAL) | diag::Severity | Mapping type. | | Str | (string) | StringRef | Warning/error name. | Example: ```yaml - Callback: PragmaDiagnostic Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-pragma.cpp:3:1" Namespace: "GCC" mapping: MAP_WARNING Str: WarningName ``` #### [PragmaOpenCLExtension](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#a92a20a21fadbab4e2c788f4e27fe07e7) Callback PragmaOpenCLExtension is called when OpenCL extension is either disabled or enabled with a pragma. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | --------------------- | -------------------- | -------------------------- | | NameLoc | "(file):(line):(col)" | SourceLocation | The location of the name. | | Name | (name) | const IdentifierInfo | Name symbol. | | StateLoc | "(file):(line):(col)" | SourceLocation | The location of the state. | | State | (1\|0) | unsigned | Enabled/disabled state. | Example: ```yaml - Callback: PragmaOpenCLExtension NameLoc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-pragma.cpp:3:10" Name: Name StateLoc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-pragma.cpp:3:18" State: 1 ``` #### [PragmaWarning](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#aa17169d25fa1cf0a6992fc944d1d8730) Callback PragmaWarning is called when a #pragma warning directive is read. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | --------------------- | -------------- | ------------------------------ | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | WarningSpec | (string) | StringRef | The warning specifier. | | Ids | \[(number)[, ...]\] | ArrayRef\ | The warning numbers. | Example: ```yaml - Callback: PragmaWarning Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-pragma.cpp:3:1" WarningSpec: disable Ids: 1,2,3 ``` #### [PragmaWarningPush](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#ae5626ef70502687a859f323a809ed0b6) Callback PragmaWarningPush is called when a #pragma warning(push) directive is read. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | --------------------- | -------------- | ------------------------------ | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | Level | (number) | int | Warning level. | Example: ```yaml - Callback: PragmaWarningPush Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-pragma.cpp:3:1" Level: 1 ``` #### [PragmaWarningPop](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#ac98d502af8811b8a6e7342d7cd2b3b95) Callback PragmaWarningPop is called when a #pragma warning(pop) directive is read. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | --------------------- | -------------- | ------------------------------ | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | Example: ```yaml - Callback: PragmaWarningPop Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-pragma.cpp:3:1" ``` #### [MacroExpands](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#a9bc725209d3a071ea649144ab996d515) Callback MacroExpands is called when ::HandleMacroExpandedIdentifier when a macro invocation is found. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | -------------- | ---------------------------------------------- | -------------------- | ------------------------------------------------------------------------------------------------------ | | MacroNameTok | (token) | const Token | The macro name token. | | MacroDirective | (MD_Define\|MD_Undefine\|MD_Visibility) | const MacroDirective | The kind of macro directive from the MacroDirective structure. | | Range | ["(file):(line):(col)", "(file):(line):(col)"] | SourceRange | The source range for the expansion. | | Args | \[(name)\|(number)\|\<(token name)>[, ...]\] | const MacroArgs | The argument tokens. Names and numbers are literal, everything else is of the form '\<' tokenName '>'. | Example: ```yaml - Callback: MacroExpands MacroNameTok: X_IMPL MacroDirective: MD_Define Range: [(nonfile), (nonfile)] Args: [a y, b] ``` #### [MacroDefined](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#a8448fc9f96f22ad1b93ff393cffc5a76) Callback MacroDefined is called when a macro definition is seen. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | -------------- | --------------------------------------- | -------------------- | -------------------------------------------------------------- | | MacroNameTok | (token) | const Token | The macro name token. | | MacroDirective | (MD_Define\|MD_Undefine\|MD_Visibility) | const MacroDirective | The kind of macro directive from the MacroDirective structure. | Example: ```yaml - Callback: MacroDefined MacroNameTok: X_IMPL MacroDirective: MD_Define ``` #### [MacroUndefined](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#acb80fc6171a839db8e290945bf2c9d7a) Callback MacroUndefined is called when a macro #undef is seen. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | -------------- | --------------------------------------- | -------------------- | -------------------------------------------------------------- | | MacroNameTok | (token) | const Token | The macro name token. | | MacroDirective | (MD_Define\|MD_Undefine\|MD_Visibility) | const MacroDirective | The kind of macro directive from the MacroDirective structure. | Example: ```yaml - Callback: MacroUndefined MacroNameTok: X_IMPL MacroDirective: MD_Define ``` #### [Defined](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#a3cc2a644533d0e4088a13d2baf90db94) Callback Defined is called when the 'defined' operator is seen. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | -------------- | ---------------------------------------------- | -------------------- | -------------------------------------------------------------- | | MacroNameTok | (token) | const Token | The macro name token. | | MacroDirective | (MD_Define\|MD_Undefine\|MD_Visibility) | const MacroDirective | The kind of macro directive from the MacroDirective structure. | | Range | ["(file):(line):(col)", "(file):(line):(col)"] | SourceRange | The source range for the directive. | Example: ```yaml - Callback: Defined MacroNameTok: MACRO MacroDirective: (null) Range: ["D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-macro.cpp:8:5", "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-macro.cpp:8:19"] ``` #### [SourceRangeSkipped](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#abdb4ebe11610f079ac33515965794b46) Callback SourceRangeSkipped is called when a source range is skipped. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | ---------------------------------------------- | -------------- | ------------------------- | | Range | ["(file):(line):(col)", "(file):(line):(col)"] | SourceRange | The source range skipped. | Example: ```yaml - Callback: SourceRangeSkipped Range: [":/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-macro.cpp:8:2", ":/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-macro.cpp:9:2"] ``` #### [If](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#a645edcb0d6becbc6f256f02fd1287778) Callback If is called when an #if is seen. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | -------------- | ---------------------------------------------- | -------------- | ----------------------------------- | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | ConditionRange | ["(file):(line):(col)", "(file):(line):(col)"] | SourceRange | The source range for the condition. | | ConditionValue | (true\|false) | bool | The condition value. | Example: ```yaml - Callback: If Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-macro.cpp:8:2" ConditionRange: ["D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-macro.cpp:8:4", "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-macro.cpp:9:1"] ConditionValue: false ``` #### [Elif](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#a180c9e106a28d60a6112e16b1bb8302a) Callback Elif is called when an #elif is seen. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | -------------- | ---------------------------------------------- | -------------- | ----------------------------------- | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | ConditionRange | ["(file):(line):(col)", "(file):(line):(col)"] | SourceRange | The source range for the condition. | | ConditionValue | (true\|false) | bool | The condition value. | | IfLoc | "(file):(line):(col)" | SourceLocation | The location of the directive. | Example: ```yaml - Callback: Elif Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-macro.cpp:10:2" ConditionRange: ["D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-macro.cpp:10:4", "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-macro.cpp:11:1"] ConditionValue: false IfLoc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-macro.cpp:8:2" ``` #### [Ifdef](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#a0ce79575dda307784fd51a6dd4eec33d) Callback Ifdef is called when an #ifdef is seen. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | -------------- | --------------------------------------- | -------------------- | -------------------------------------------------------------- | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | MacroNameTok | (token) | const Token | The macro name token. | | MacroDirective | (MD_Define\|MD_Undefine\|MD_Visibility) | const MacroDirective | The kind of macro directive from the MacroDirective structure. | Example: ```yaml - Callback: Ifdef Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-conditional.cpp:3:1" MacroNameTok: MACRO MacroDirective: MD_Define ``` #### [Ifndef](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#a767af69f1cdcc4cd880fa2ebf77ad3ad) Callback Ifndef is called when an #ifndef is seen. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | -------------- | --------------------------------------- | -------------------- | -------------------------------------------------------------- | | Loc | "(file):(line):(col)" | SourceLocation | The location of the directive. | | MacroNameTok | (token) | const Token | The macro name token. | | MacroDirective | (MD_Define\|MD_Undefine\|MD_Visibility) | const MacroDirective | The kind of macro directive from the MacroDirective structure. | Example: ```yaml - Callback: Ifndef Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-conditional.cpp:3:1" MacroNameTok: MACRO MacroDirective: MD_Define ``` #### [Else](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#ad57f91b6d9c3cbcca326a2bfb49e0314) Callback Else is called when an #else is seen. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | --------------------- | -------------- | ----------------------------------- | | Loc | "(file):(line):(col)" | SourceLocation | The location of the else directive. | | IfLoc | "(file):(line):(col)" | SourceLocation | The location of the if directive. | Example: ```yaml - Callback: Else Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-macro.cpp:10:2" IfLoc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-macro.cpp:8:2" ``` #### [Endif](https://clang.llvm.org/doxygen/classclang_1_1PPCallbacks.html#afc62ca1401125f516d58b1629a2093ce) Callback Endif is called when an #endif is seen. Argument descriptions: | Argument Name | Argument Value Syntax | Clang C++ Type | Description | | ------------- | --------------------- | -------------- | ------------------------------------ | | Loc | "(file):(line):(col)" | SourceLocation | The location of the endif directive. | | IfLoc | "(file):(line):(col)" | SourceLocation | The location of the if directive. | Example: ```yaml - Callback: Endif Loc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-macro.cpp:10:2" IfLoc: "D:/Clang/llvm/clang-tools-extra/test/pp-trace/pp-trace-macro.cpp:8:2" ``` ## Building pp-trace To build from source: 1. Read [Getting Started with the LLVM System][getting started with the llvm system] and [Clang Tools Documentation][clang tools documentation] for information on getting sources for LLVM, Clang, and Clang Extra Tools. 2. [Getting Started with the LLVM System][getting started with the llvm system] and [Building LLVM with CMake][building llvm with cmake] give directions for how to build. With sources all checked out into the right place the LLVM build will build Clang Extra Tools and their dependencies automatically. - If using CMake, you can also use the `pp-trace` target to build just the pp-trace tool and its dependencies. [building llvm with cmake]: https://llvm.org/docs/CMake.html [clang tools documentation]: https://clang.llvm.org/docs/ClangTools.html [getting started with the llvm system]: https://llvm.org/docs/GettingStarted.html --- ### Clang Docs AMDGPUSupport (`clang/docs/AMDGPUSupport.md`) # AMDGPU Support Clang supports OpenCL, HIP and OpenMP on AMD GPU targets. ## Predefined Macros | Macro | Description | | --- | --- | | `__AMDGPU__` | Indicates that the code is being compiled for an AMD GPU. | | `__AMDGCN__` | Defined if the GPU target is AMDGCN. | | `__R600__` | Defined if the GPU target is R600. | | `____` | Defined with the name of the architecture (e.g., `__gfx906__` for the gfx906 architecture). | | `____` | Defines the GFX family (e.g., for gfx906, this macro would be `__GFX9__`). | | `__amdgcn_processor__` | Defined with the processor name as a string (e.g., `"gfx906"`). | | `__amdgcn_target_id__` | Defined with the target ID as a string. | | `__amdgcn_feature___` | Defined for each supported target feature. The value is 1 if the feature is enabled and 0 if it is disabled. Allowed feature names are sramecc and xnack. | | `__AMDGCN_CUMODE__` | Defined as 1 if the CU mode is enabled and 0 if the WGP mode is enabled. | | `__AMDGCN_UNSAFE_FP_ATOMICS__` | Defined if unsafe floating-point atomics are allowed. | | `__HAS_FMAF__` | Defined if FMAF instruction is available (deprecated). | | `__HAS_LDEXPF__` | Defined if LDEXPF instruction is available (deprecated). | | `__HAS_FP64__` | Defined if FP64 instruction is available (deprecated). | Please note that the specific architecture and feature names will vary depending on the GPU. Also, some macros are deprecated and may be removed in future releases. ## Target-Specific Builtins Clang exposes AMDGPU hardware intrinsics as target-specific builtins with the `__builtin_amdgcn_` prefix. These are documented in {doc}`AMDGPUBuiltinReference`. ## Target-Specific Types ### Named Workgroup Barrier Type The `__amdgpu_named_workgroup_barrier_t` type is used to represent the GFX12.5 named barriers. This type is subject to certain restrictions when used to declare non-static fields of classes. See the {ref}`named barrier wrappers` section below for more information. Example usage: ```c __amdgpu_named_workgroup_barrier_t x; __amdgpu_named_workgroup_barrier_t arr[2]; // Arrays are also fine struct SimpleWrapper { __amdgpu_named_workgroup_barrier_t foo; }; void foo(int a) { __builtin_amdgcn_s_barrier_init(&x, a); } ``` (namedbarrierwrappers)= ### Named Barrier Wrappers A "named barrier wrapper" is a class that contains exactly one non-static field of one of the following types: * `__amdgpu_named_workgroup_barrier_t`, or an array thereof. * Another named barrier wrapper, or an array thereof. In C++, a class that inherits from a named barrier wrapper is also considered a named barrier wrapper, and all named barrier wrappers must be standard-layout types (see `std::is_standard_layout`). This means that named barrier wrappers: * May not have a virtual table: they cannot declare or inherit any virtual functions, or inherit from a virtual base class. * May not have any extra fields, either declared by the class or inherited from a base class. --- ### Clang Docs APINotes (`clang/docs/APINotes.md`) # API Notes: Annotations Without Modifying Headers **The Problem:** You have headers you want to use, but you also want to add extra information to the API. You don't want to put that information in the headers themselves --- perhaps because you want to keep them clean for other clients, or perhaps because they're from some open source project and you don't want to modify them at all. **Incomplete solution:** Redeclare all the interesting parts of the API in your own header and add the attributes you want. Unfortunately, this: - doesn't work with attributes that must be present on a definition - doesn't allow changing the definition in other ways - requires your header to be included in any client code to take effect **Better solution:** Provide a "sidecar" file with the information you want to add, and have that automatically get picked up by the module-building logic in the compiler. That's API notes. API notes use a YAML-based file format. YAML is a format best explained by example, so here is a [small example](https://github.com/llvm/llvm-project/blob/main/clang/test/APINotes/Inputs/Frameworks/SomeKit.framework/Headers/SomeKit.apinotes) from the compiler test suite of API notes for a hypothetical "SomeKit" framework. ## Usage API notes files are found relative to the module map that defines a module, under the name "SomeKit.apinotes" for a module named "SomeKit". Additionally, a file named "SomeKit_private.apinotes" will also be picked up to go with a private module map. For bare modules these two files will be in the same directory as the corresponding module map; for framework modules, they should be placed in the Headers and PrivateHeaders directories, respectively. The module map for a private top-level framework module should be placed in the PrivateHeaders directory as well, though it does not need an additional "_private" suffix on its name. Clang will search for API notes files next to module maps only when passed the `-fapinotes-modules` option. ## Limitations - Since they're identified by module name, API notes cannot be used to modify arbitrary textual headers. ## "Versioned" API Notes Many API notes affect how a C API is imported into Swift. In order to change that behavior while still remaining backwards-compatible, API notes can be selectively applied based on the Swift compatibility version provided to the compiler (e.g. `-fapi-notes-swift-version=5`). The rule is that an explicitly-versioned API note applies to that version *and all earlier versions,* and any applicable explicitly-versioned API note takes precedence over an unversioned API note. ## Reference An API notes file contains a YAML dictionary with the following top-level entries: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Clang Docs AddressSanitizer (`clang/docs/AddressSanitizer.md`) # AddressSanitizer ## Introduction AddressSanitizer is a fast memory error detector. It consists of a compiler instrumentation module and a run-time library. The tool can detect the following types of bugs: - Out-of-bounds accesses to heap, stack and globals - Use-after-free - Use-after-return (clang flag `-fsanitize-address-use-after-return=(never|runtime|always)` default: `runtime`) : - Enable with: `ASAN_OPTIONS=detect_stack_use_after_return=1` (already enabled on Linux). - Disable with: `ASAN_OPTIONS=detect_stack_use_after_return=0`. - Use-after-scope (clang flag `-fsanitize-address-use-after-scope`) - Double-free, invalid free - Memory leaks (experimental) Typical slowdown introduced by AddressSanitizer is **2x**. ## How to build Build LLVM/Clang with [CMake](https://llvm.org/docs/CMake.html) and enable the `compiler-rt` runtime. An example CMake configuration that will allow for the use/testing of AddressSanitizer: ```console $ cmake -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang" -DLLVM_ENABLE_RUNTIMES="compiler-rt" /llvm ``` ## Usage Simply compile and link your program with `-fsanitize=address` flag. The AddressSanitizer run-time library should be linked to the final executable, so make sure to use `clang` (not `ld`) for the final link step. When linking shared libraries, the AddressSanitizer run-time is not linked, so `-Wl,-z,defs` may cause link errors (don't use it with AddressSanitizer). To get a reasonable performance add `-O1` or higher. To get nicer stack traces in error messages add `-fno-omit-frame-pointer`. To get perfect stack traces you may need to disable inlining (just use `-O1`) and tail call elimination (`-fno-optimize-sibling-calls`). ```console % cat example_UseAfterFree.cc int main(int argc, char **argv) { int *array = new int[100]; delete [] array; return array[argc]; // BOOM } # Compile and link % clang++ -O1 -g -fsanitize=address -fno-omit-frame-pointer example_UseAfterFree.cc ``` or: ```console # Compile % clang++ -O1 -g -fsanitize=address -fno-omit-frame-pointer -c example_UseAfterFree.cc # Link % clang++ -g -fsanitize=address example_UseAfterFree.o ``` If a bug is detected, the program will print an error message to stderr and exit with a non-zero exit code. AddressSanitizer exits on the first detected error. This is by design: - This approach allows AddressSanitizer to produce faster and smaller generated code (both by ~5%). - Fixing bugs becomes unavoidable. AddressSanitizer does not produce false alarms. Once a memory corruption occurs, the program is in an inconsistent state, which could lead to confusing results and potentially misleading subsequent reports. If your process is sandboxed and you are running on OS X 10.10 or earlier, you will need to set `DYLD_INSERT_LIBRARIES` environment variable and point it to the ASan library that is packaged with the compiler used to build the executable. (You can find the library by searching for dynamic libraries with `asan` in their name.) If the environment variable is not set, the process will try to re-exec. Also keep in mind that when moving the executable to another machine, the ASan library will also need to be copied over. ## Symbolizing the Reports To make AddressSanitizer symbolize its output you need to set the `ASAN_SYMBOLIZER_PATH` environment variable to point to the `llvm-symbolizer` binary (or make sure `llvm-symbolizer` is in your `$PATH`): ```console % ASAN_SYMBOLIZER_PATH=/usr/local/bin/llvm-symbolizer ./a.out ==9442== ERROR: AddressSanitizer heap-use-after-free on address 0x7f7ddab8c084 at pc 0x403c8c bp 0x7fff87fb82d0 sp 0x7fff87fb82c8 READ of size 4 at 0x7f7ddab8c084 thread T0 #0 0x403c8c in main example_UseAfterFree.cc:4 #1 0x7f7ddabcac4d in __libc_start_main ??:0 0x7f7ddab8c084 is located 4 bytes inside of 400-byte region [0x7f7ddab8c080,0x7f7ddab8c210) freed by thread T0 here: #0 0x404704 in operator delete[](void*) ??:0 #1 0x403c53 in main example_UseAfterFree.cc:4 #2 0x7f7ddabcac4d in __libc_start_main ??:0 previously allocated by thread T0 here: #0 0x404544 in operator new[](unsigned long) ??:0 #1 0x403c43 in main example_UseAfterFree.cc:2 #2 0x7f7ddabcac4d in __libc_start_main ??:0 ==9442== ABORTING ``` If that does not work for you (e.g. your process is sandboxed), you can use a separate script to symbolize the result offline (online symbolization can be force disabled by setting `ASAN_OPTIONS=symbolize=0`): ```console % ASAN_OPTIONS=symbolize=0 ./a.out 2> log % projects/compiler-rt/lib/asan/scripts/asan_symbolize.py / < log | c++filt ==9442== ERROR: AddressSanitizer heap-use-after-free on address 0x7f7ddab8c084 at pc 0x403c8c bp 0x7fff87fb82d0 sp 0x7fff87fb82c8 READ of size 4 at 0x7f7ddab8c084 thread T0 #0 0x403c8c in main example_UseAfterFree.cc:4 #1 0x7f7ddabcac4d in __libc_start_main ??:0 ... ``` Note that on macOS you may need to run `dsymutil` on your binary to have the file:line info in the AddressSanitizer reports. ## Additional Checks ### Initialization order checking AddressSanitizer can optionally detect dynamic initialization order problems, when initialization of globals defined in one translation unit uses globals defined in another translation unit. To enable this check at runtime, you should set environment variable `ASAN_OPTIONS=check_initialization_order=1`. Note that this option is not supported on macOS. ### Stack Use After Return (UAR) AddressSanitizer can optionally detect stack use after return problems. This is available by default, or explicitly (`-fsanitize-address-use-after-return=runtime`). To disable this check at runtime, set the environment variable `ASAN_OPTIONS=detect_stack_use_after_return=0`. Enabling this check (`-fsanitize-address-use-after-return=always`) will reduce code size. The code size may be reduced further by completely eliminating this check (`-fsanitize-address-use-after-return=never`). To summarize: `-fsanitize-address-use-after-return=` : - `never`: Completely disables detection of UAR errors (reduces code size). - `runtime`: Adds the code for detection, but it can be disabled via the runtime environment (`ASAN_OPTIONS=detect_stack_use_after_return=0`). - `always`: Enables detection of UAR errors in all cases. (reduces code size, but not as much as `never`). ### Container Overflow Detection AddressSanitizer can detect overflows in containers with custom allocators (such as std::vector) where the library developers have added calls into the AddressSanitizer runtime to indicate which memory is poisoned etc. Note that this feature is prone to false positives: : - Partially poisoning objects on stack, e.g. for small string optimization, can cause both false positives and negatives. - If the binary is partially AddressSanitizer instrumented, these checks can cause false positives. See [Disabling container overflow checks](#disabling-container-overflow-checks) for details on suppressing checks. ### Memory leak detection For more information on leak detector in AddressSanitizer, see {doc}`LeakSanitizer`. The leak detection is turned on by default on Linux, and can be enabled using `ASAN_OPTIONS=detect_leaks=1` on macOS; however, it is not yet supported on other platforms. ## Flags and Options Runtime flags can be passed to AddressSanitizer via the `ASAN_OPTIONS` environment variable: ```console $ ASAN_OPTIONS="verbosity=1:detect_stack_use_after_return=1" ./a.out ``` To see the full list of available flags, run an instrumented binary with `ASAN_OPTIONS="help=1"`. Flags passed via the `ASAN_OPTIONS` environment variable take precedence over compile-time default options. ### Compile-time Default Options Default options can be specified at compile/link time by defining the `__asan_default_options` function in your source code: ```c++ #include extern "C" const char *__asan_default_options() { return "verbosity=1:detect_stack_use_after_return=1"; } ``` ### Options Evaluation with Integrated Sanitizers When running AddressSanitizer with integrated {doc}`LeakSanitizer` or {doc}`UndefinedBehaviorSanitizer`: - `__asan_default_options()`, `__lsan_default_options()`, and `__ubsan_default_options()` are all evaluated independently by the runtime. - The environment variables `ASAN_OPTIONS`, `LSAN_OPTIONS`, and `UBSAN_OPTIONS` are also parsed independently. LSan and UBSan flags should be passed via their own environment variables (`LSAN_OPTIONS`, `UBSAN_OPTIONS`) or default option hooks (`__lsan_default_options()`, `__ubsan_default_options()`) rather than packed into `ASAN_OPTIONS` or `__asan_default_options()`. ## Issue Suppression AddressSanitizer is not expected to produce false positives. If you see one, look again; most likely it is a true positive! ### Suppressing Reports in External Libraries Runtime interposition allows AddressSanitizer to find bugs in code that is not being recompiled. If you run into an issue in external libraries, we recommend immediately reporting it to the library maintainer so that it gets addressed. However, you can use the following suppression mechanism to unblock yourself and continue on with the testing. This suppression mechanism should only be used for suppressing issues in external code; it does not work on code recompiled with AddressSanitizer. To suppress errors in external libraries, set the `ASAN_OPTIONS` environment variable to point to a suppression file. You can either specify the full path to the file or the path of the file relative to the location of your executable. ```bash ASAN_OPTIONS=suppressions=MyASan.supp ``` Each non-empty line of the suppression file represents one suppression of the form `suppression_type:suppression_pattern`. Supported types are: - `interceptor_via_fun`: Suppress errors when the given function is in the caller stack trace. - `interceptor_via_lib`: Suppress errors when the call originates from the given library. - `interceptor_name`: Suppress an interceptor by function name directly (e.g., `interceptor_name:memcpy`). - `odr_violation`: Suppress One Definition Rule violation reports for global variables. - `alloc_dealloc_mismatch`: Suppress allocation/deallocation mismatch errors for specific functions in the stack trace. ```bash interceptor_via_fun:NameOfCFunctionToSuppress interceptor_via_fun:-[ClassName objCMethodToSuppress:] interceptor_via_lib:NameOfTheLibraryToSuppress interceptor_name:memcpy odr_violation:my_global_var alloc_dealloc_mismatch:my_allocator_fn ``` Alternatively, you can provide default suppressions at compile time by defining the `__asan_default_suppressions` function in your source code: ```c++ #include extern "C" const char *__asan_default_suppressions() { return "interceptor_via_lib:NameOfTheLibraryToSuppress\n" "interceptor_name:memcpy\n"; } ``` ### Conditional Compilation with `__has_feature(address_sanitizer)` In some cases one may need to execute different code depending on whether AddressSanitizer is enabled. {ref}`__has_feature ` can be used for this purpose. ```c #if defined(__has_feature) # if __has_feature(address_sanitizer) // code that builds only under AddressSanitizer # endif #endif ``` ### Disabling Instrumentation with `__attribute__((no_sanitize("address")))` Some code should not be instrumented by AddressSanitizer. One may use the attribute `__attribute__((no_sanitize("address")))` (which has deprecated synonyms `no_sanitize_address` and `no_address_safety_analysis`) to disable instrumentation of a particular function. This attribute may not be supported by other compilers, so we suggest to use it together with `__has_feature(address_sanitizer)`. The same attribute used on a global variable prevents AddressSanitizer from adding redzones around it and detecting out of bounds accesses. AddressSanitizer also supports `__attribute__((disable_sanitizer_instrumentation))`. This attribute works similarly to `__attribute__((no_sanitize("address")))`, but it also prevents instrumentation performed by other sanitizers. ### Interaction of Inlining with Disabling Sanitizer Instrumentation - A `no_sanitize` function will not be inlined heuristically by the compiler into a sanitized function. - An `always_inline` function will adopt the instrumentation status of the function it is inlined into. - Forcibly combining `no_sanitize` and `__attribute__((always_inline))` is not supported, and will often lead to unexpected results. To avoid mixing these attributes, use: ```c // Note, __has_feature test for sanitizers is deprecated, and Clang will support __SANITIZE___ similar to GCC. #if __has_feature(address_sanitizer) || defined(__SANITIZE_ADDRESS__) || ... #define ALWAYS_INLINE_IF_UNINSTRUMENTED #else #define ALWAYS_INLINE_IF_UNINSTRUMENTED __attribute__((always_inline)) #endif ``` ### Explicit Sanitizer Checks with `__builtin_allow_sanitize_check` The `__builtin_allow_sanitize_check("address")` builtin can be used to conditionally execute code depending on whether AddressSanitizer checks are enabled and permitted by the current policy (after inlining). This is particularly useful for inserting explicit, sanitizer-specific checks around operations like syscalls or inline assembly, which might otherwise be unchecked by the sanitizer. Example: ```c void __asan_load8(void *); inline __attribute__((always_inline)) void my_helper(void *addr) { if (__builtin_allow_sanitize_check("address")) __asan_load8(addr); // ... actual logic, e.g. inline assembly ... asm volatile ("..." : : "r" (addr) : "memory"); } void instrumented_function() { ... my_helper(buf); // checks are active ... } __attribute__((no_sanitize("address"))) void uninstrumented_function() { ... my_helper(buf); // checks are skipped ... } ``` ### Disabling container overflow checks #### Runtime suppression Container overflow checks can be disabled at runtime using the `ASAN_OPTIONS=detect_container_overflow=0` environment variable. #### Compile time suppression `-D__SANITIZER_DISABLE_CONTAINER_OVERFLOW__` can be used at compile time to disable container overflow checks if the container library has added support for this define. To support a standard way to disable container overflow checks at compile time, library developers should use this definition in conjunction with the AddressSanitizer feature test to conditionally include container overflow related code compiled into user code: The recommended form is ```c // include the sanitizer common interfaces #include #if __has_feature(address_sanitizer) // Container overflow detection enabled - include annotations __sanitizer_annotate_contiguous_container(beg, end, old_mid, new_mid); #endif ``` This pattern ensures that: - Container overflow annotations are only included when AddressSanitizer is enabled - Container overflow detection can be disabled by passing `-D__SANITIZER_DISABLE_CONTAINER_OVERFLOW__` to the compiler ### Suppressing Errors in Recompiled Code (Ignorelist) AddressSanitizer supports `src` and `fun` entity types in {doc}`SanitizerSpecialCaseList`, that can be used to suppress error reports in the specified source files or functions. Additionally, AddressSanitizer introduces `global` and `type` entity types that can be used to suppress error reports for out-of-bound access to globals with certain names and types (you may only specify class or struct types). You may use an `init` category to suppress reports about initialization-order problems happening in certain source files or with certain global variables. ```bash # Suppress error reports for code in a file or in a function: src:bad_file.cpp # Ignore all functions with names containing MyFooBar: fun:*MyFooBar* # Disable out-of-bound checks for global: global:bad_array # Disable out-of-bound checks for global instances of a given class ... type:Namespace::BadClassName # ... or a given struct. Use wildcard to deal with anonymous namespace. type:Namespace2::*::BadStructName # Disable initialization-order checks for globals: global:bad_init_global=init type:*BadInitClassSubstring*=init src:bad/init/files/*=init ``` ### Suppressing memory leaks Memory leak reports produced by {doc}`LeakSanitizer` (when run as part of AddressSanitizer) can be suppressed at runtime via `LSAN_OPTIONS=suppressions=...` or at compile time via `__lsan_default_suppressions()`. See {doc}`LeakSanitizer` for full details on suppression rules, default options, and programmatic APIs. ## Code generation control ### Instrumentation code outlining By default AddressSanitizer inlines the instrumentation code to improve the run-time performance, which leads to increased binary size. Using the (clang flag `-fsanitize-address-outline-instrumentation` default: `false`) flag forces all code instrumentation to be outlined, which reduces the size of the generated code, but also reduces the run-time performance. ## Limitations - AddressSanitizer uses more real memory than a native run. Exact overhead depends on the allocation sizes. The smaller the allocations you make the bigger the overhead is. - AddressSanitizer uses more stack memory. We have seen up to 3x increase. - On 64-bit platforms AddressSanitizer maps (but not reserves) 16+ Terabytes of virtual address space. This means that tools like `ulimit` may not work as usually expected. - Static linking of executables is not supported. ## Security Considerations AddressSanitizer is a bug detection tool and its runtime is not meant to be linked against production executables. While it may be useful for testing, AddressSanitizer's runtime was not developed with security-sensitive constraints in mind and may compromise the security of the resulting executable. ## Supported Platforms AddressSanitizer is supported on: - Linux - macOS - iOS Simulator - Android - NetBSD - FreeBSD - Windows 8.1+ ## Current Status AddressSanitizer is fully functional on supported platforms starting from LLVM 3.1. The test suite is integrated into CMake build and can be run with `make check-asan` command. The Windows port is functional and is used by Chrome and Firefox, but it is not as well supported as the other ports. ## More Information [https://github.com/google/sanitizers/wiki/AddressSanitizer](https://github.com/google/sanitizers/wiki/AddressSanitizer) --- ### Clang Docs AllocToken (`clang/docs/AllocToken.md`) # Allocation Tokens ## Introduction Clang provides support for allocation tokens to enable allocator-level heap organization strategies. Clang assigns mode-dependent token IDs to allocation calls; the runtime behavior depends entirely on the implementation of a compatible memory allocator. Possible allocator strategies include: - **Security Hardening**: Placing allocations into separate, isolated heap partitions. For example, separating pointer-containing types from raw data can mitigate exploits that rely on overflowing a primitive buffer to corrupt object metadata. - **Memory Layout Optimization**: Grouping related allocations to improve data locality and cache utilization. - **Custom Allocation Policies**: Applying different management strategies to different partitions. ## Token Assignment Mode The default mode to calculate tokens is: - `typehashpointersplit`: This mode assigns a token ID based on the hash of the allocated type's name, where the top half ID-space is reserved for types that contain pointers and the bottom half for types that do not contain pointers. Other token ID assignment modes are supported, but they may be subject to change or removal. These may (experimentally) be selected with `-Xclang -falloc-token-mode=`: - `typehash`: This mode assigns a token ID based on the hash of the allocated type's name. - `typefunchashpointersplit`: This mode assigns a token ID based on the hash of the allocated type's name and the hash of the name of the function containing the allocation. The upper half of the token ID bits is the type name hash, the lower half of the token ID bits is the function name hash, and the most significant bit is a flag indicating whether the type contains pointers. - `typefunchash`: Like `typefunchashpointersplit`, but without the pointer flag. - `random`: This mode assigns a statically-determined random token ID to each allocation site. - `increment`: This mode assigns a simple, incrementally increasing token ID to each allocation site. The following command-line options affect generated token IDs: - `-falloc-token-max=` : Configures the maximum number of token IDs. By default the number of tokens is bounded by `SIZE_MAX`. - In the `typefunchashpointersplit` and `typefunchash` modes, the number of token IDs is `N` rounded down to a power of two. As an exception, if `N` is of the form `2^k-1` (e.g. the default `SIZE_MAX`), all `k` bits are used and token IDs are in `[0, N]`, i.e. a token ID may be equal to `N`. ## Querying Token IDs with `__builtin_infer_alloc_token` For use cases where the token ID must be known at compile time, Clang provides a builtin function: ```c size_t __builtin_infer_alloc_token(, ...); ``` This builtin returns the token ID inferred from its argument expressions, which mirror arguments normally passed to any allocation function. The argument expressions are **unevaluated**, so it can be used with expressions that would have side effects without any runtime impact. In the `typefunchashpointersplit`, `typefunchash`, `random`, and `increment` modes, the builtin cannot be used in constant expressions. For example, it can be used as follows: ```c struct MyType { ... }; void *__partition_alloc(size_t size, size_t partition); #define partition_alloc(...) __partition_alloc(__VA_ARGS__, __builtin_infer_alloc_token(__VA_ARGS__)) void foo(void) { MyType *x = partition_alloc(sizeof(*x)); } ``` ## Allocation Token Instrumentation To enable instrumentation of allocation functions, code can be compiled with the `-fsanitize=alloc-token` flag: ```console % clang++ -fsanitize=alloc-token example.cc ``` The instrumentation transforms allocation calls to include a token ID. For example: ```c // Original: ptr = malloc(size); // Instrumented: ptr = __alloc_token_malloc(size, ); ``` ### Runtime Interface A compatible runtime must be provided that implements the token-enabled allocation functions. The instrumentation generates calls to functions that take a final `size_t token_id` argument. ```c // C standard library functions void *__alloc_token_malloc(size_t size, size_t token_id); void *__alloc_token_calloc(size_t count, size_t size, size_t token_id); void *__alloc_token_realloc(void *ptr, size_t size, size_t token_id); // ... // C++ operators (mangled names) // operator new(size_t, size_t) void *__alloc_token__Znwm(size_t size, size_t token_id); // operator new[](size_t, size_t) void *__alloc_token__Znam(size_t size, size_t token_id); // ... other variants like nothrow, etc., are also instrumented. ``` ### Fast ABI An alternative ABI can be enabled with `-fsanitize-alloc-token-fast-abi`, which encodes the token ID in the allocation function name. ```c void *__alloc_token_0_malloc(size_t size); void *__alloc_token_1_malloc(size_t size); void *__alloc_token_2_malloc(size_t size); ... void *__alloc_token_0_Znwm(size_t size); void *__alloc_token_1_Znwm(size_t size); void *__alloc_token_2_Znwm(size_t size); ... ``` This ABI provides a more efficient alternative where `-falloc-token-max` is small. ### Instrumenting Non-Standard Allocation Functions By default, AllocToken only instruments standard library allocation functions. This simplifies adoption, as a compatible allocator only needs to provide token-enabled variants for a well-defined set of standard functions. To extend instrumentation to custom allocation functions, enable broader coverage with `-fsanitize-alloc-token-extended`. Such functions require being marked with the [malloc][malloc-attribute] or [alloc_size][alloc-size-attribute] attributes (or a combination). For example: ```c void *custom_malloc(size_t size) __attribute__((malloc)); void *my_malloc(size_t size) __attribute__((alloc_size(1))); // Original: ptr1 = custom_malloc(size); ptr2 = my_malloc(size); // Instrumented: ptr1 = __alloc_token_custom_malloc(size, token_id); ptr2 = __alloc_token_my_malloc(size, token_id); ``` Note: Even in the default mode (without `-fsanitize-alloc-token-extended`), an *inline* allocation wrapper marked with the [malloc][malloc-attribute] or [alloc_size][alloc-size-attribute] attribute is supported if it is inlined into its caller: the inferred token is propagated to the allocation call the wrapper returns, which is then instrumented normally. Wrappers that are not inlined still require `-fsanitize-alloc-token-extended`. [malloc-attribute]: AttributeReference.md#malloc [alloc-size-attribute]: AttributeReference.md#alloc-size ### Disabling Instrumentation To exclude specific functions from instrumentation, you can use the `no_sanitize("alloc-token")` attribute: ```c __attribute__((no_sanitize("alloc-token"))) void* custom_allocator(size_t size) { return malloc(size); // Uses original malloc } ``` Note: Independent of any given allocator support, the instrumentation aims to remain performance neutral. As such, `no_sanitize("alloc-token")` functions may be inlined into instrumented functions and vice-versa. If correctness is affected, such functions should explicitly be marked `noinline`. The `__attribute__((disable_sanitizer_instrumentation))` is also supported to disable this and other sanitizer instrumentations. ### Suppressions File (Ignorelist) AllocToken respects the `src` and `fun` entity types in the {doc}`SanitizerSpecialCaseList`, which can be used to omit specified source files or functions from instrumentation. ```bash [alloc-token] # Exclude specific source files src:third_party/allocator.c # Exclude function name patterns fun:*custom_malloc* fun:LowLevel::* ``` ```console % clang++ -fsanitize=alloc-token -fsanitize-ignorelist=my_ignorelist.txt example.cc ``` ### Conditional Compilation with `__SANITIZE_ALLOC_TOKEN__` In some cases, one may need to execute different code depending on whether AllocToken instrumentation is enabled. The `__SANITIZE_ALLOC_TOKEN__` macro can be used for this purpose. ```c #ifdef __SANITIZE_ALLOC_TOKEN__ // Code specific to -fsanitize=alloc-token builds #endif ``` --- ### Clang Docs AutomaticReferenceCounting (`clang/docs/AutomaticReferenceCounting.md`) % FIXME: move to the stylesheet or Sphinx plugin ```{raw} html ``` ```{role} arc-term ``` ```{role} revision ``` ```{role} when-revised ``` # Objective-C Automatic Reference Counting (ARC) (arc.meta)= ## About this document (arc.meta.purpose)= ### Purpose The first and primary purpose of this document is to serve as a complete technical specification of Automatic Reference Counting. Given a core Objective-C compiler and runtime, it should be possible to write a compiler and runtime which implements these new semantics. The secondary purpose is to act as a rationale for why ARC was designed in this way. This should remain tightly focused on the technical design and should not stray into marketing speculation. (arc.meta.background)= ### Background This document assumes a basic familiarity with C. {arc-term}`Blocks` are a C language extension for creating anonymous functions. Users interact with and transfer block objects using {arc-term}`block pointers`, which are represented like a normal pointer. A block may capture values from local variables; when this occurs, memory must be dynamically allocated. The initial allocation is done on the stack, but the runtime provides a `Block_copy` function which, given a block pointer, either copies the underlying block object to the heap, setting its reference count to 1 and returning the new block pointer, or (if the block object is already on the heap) increases its reference count by 1. The paired function is `Block_release`, which decreases the reference count by 1 and destroys the object if the count reaches zero and is on the heap. Objective-C is a set of language extensions, significant enough to be considered a different language. It is a strict superset of C. The extensions can also be imposed on C++, producing a language called Objective-C++. The primary feature is a single-inheritance object system; we briefly describe the modern dialect. Objective-C defines a new type kind, collectively called the {arc-term}`object pointer types`. This kind has two notable builtin members, `id` and `Class`; `id` is the final supertype of all object pointers. The validity of conversions between object pointer types is not checked at runtime. Users may define {arc-term}`classes`; each class is a type, and the pointer to that type is an object pointer type. A class may have a superclass; its pointer type is a subtype of its superclass's pointer type. A class has a set of {arc-term}`ivars`, fields which appear on all instances of that class. For every class *T* there's an associated metaclass; it has no fields, its superclass is the metaclass of *T*'s superclass, and its metaclass is a global class. Every class has a global object whose class is the class's metaclass; metaclasses have no associated type, so pointers to this object have type `Class`. A class declaration (`@interface`) declares a set of {arc-term}`methods`. A method has a return type, a list of argument types, and a {arc-term}`selector`: a name like `foo:bar:baz:`, where the number of colons corresponds to the number of formal arguments. A method may be an instance method, in which case it can be invoked on objects of the class, or a class method, in which case it can be invoked on objects of the metaclass. A method may be invoked by providing an object (called the {arc-term}`receiver`) and a list of formal arguments interspersed with the selector, like so: ```objc [receiver foo: fooArg bar: barArg baz: bazArg] ``` This looks in the dynamic class of the receiver for a method with this name, then in that class's superclass, etc., until it finds something it can execute. The receiver "expression" may also be the name of a class, in which case the actual receiver is the class object for that class, or (within method definitions) it may be `super`, in which case the lookup algorithm starts with the static superclass instead of the dynamic class. The actual methods dynamically found in a class are not those declared in the `@interface`, but those defined in a separate `@implementation` declaration; however, when compiling a call, typechecking is done based on the methods declared in the `@interface`. Method declarations may also be grouped into {arc-term}`protocols`, which are not inherently associated with any class, but which classes may claim to follow. Object pointer types may be qualified with additional protocols that the object is known to support. {arc-term}`Class extensions` are collections of ivars and methods, designed to allow a class's `@interface` to be split across multiple files; however, there is still a primary implementation file which must see the `@interface`s of all class extensions. {arc-term}`Categories` allow methods (but not ivars) to be declared *post hoc* on an arbitrary class; the methods in the category's `@implementation` will be dynamically added to that class's method tables which the category is loaded at runtime, replacing those methods in case of a collision. In the standard environment, objects are allocated on the heap, and their lifetime is manually managed using a reference count. This is done using two instance methods which all classes are expected to implement: `retain` increases the object's reference count by 1, whereas `release` decreases it by 1 and calls the instance method `dealloc` if the count reaches 0. To simplify certain operations, there is also an {arc-term}`autorelease pool`, a thread-local list of objects to call `release` on later; an object can be added to this pool by calling `autorelease` on it. Block pointers may be converted to type `id`; block objects are laid out in a way that makes them compatible with Objective-C objects. There is a builtin class that all block objects are considered to be objects of; this class implements `retain` by adjusting the reference count, not by calling `Block_copy`. (arc.meta.evolution)= ### Evolution ARC is under continual evolution, and this document must be updated as the language progresses. If a change increases the expressiveness of the language, for example by lifting a restriction or by adding new syntax, the change will be annotated with a revision marker, like so: > ARC applies to Objective-C pointer types, block pointer types, and > {when-revised}`[beginning Apple 8.0, LLVM 3.8]` {revision}`BPTRs declared > within` `extern "BCPL"` blocks. For now, it is sensible to version this document by the releases of its sole implementation (and its host project), clang. "LLVM X.Y" refers to an open-source release of clang from the LLVM project. "Apple X.Y" refers to an Apple-provided release of the Apple LLVM Compiler. Other organizations that prepare their own, separately-versioned clang releases and wish to maintain similar information in this document should send requests to cfe-dev. If a change decreases the expressiveness of the language, for example by imposing a new restriction, this should be taken as an oversight in the original specification and something to be avoided in all versions. Such changes are generally to be avoided. (arc.general)= ## General Automatic Reference Counting implements automatic memory management for Objective-C objects and blocks, freeing the programmer from the need to explicitly insert retains and releases. It does not provide a cycle collector; users must explicitly manage the lifetime of their objects, breaking cycles manually or with weak or unsafe references. ARC may be explicitly enabled with the compiler flag `-fobjc-arc`. It may also be explicitly disabled with the compiler flag `-fno-objc-arc`. The last of these two flags appearing on the compile line "wins". If ARC is enabled, `__has_feature(objc_arc)` will expand to 1 in the preprocessor. For more information about `__has_feature`, see the {ref}`language extensions ` document. (arc.objects)= ## Retainable object pointers This section describes retainable object pointers, their basic operations, and the restrictions imposed on their use under ARC. Note in particular that it covers the rules for pointer *values* (patterns of bits indicating the location of a pointed-to object), not pointer *objects* (locations in memory which store pointer values). The rules for objects are covered in the next section. A {arc-term}`retainable object pointer` (or "retainable pointer") is a value of a {arc-term}`retainable object pointer type` ("retainable type"). There are three kinds of retainable object pointer types: - block pointers (formed by applying the caret (`^`) declarator sigil to a function type) - Objective-C object pointers (`id`, `Class`, `NSFoo*`, etc.) - typedefs marked with `__attribute__((NSObject))` Other pointer types, such as `int*` and `CFStringRef`, are not subject to ARC's semantics and restrictions. :::{admonition} Rationale We are not at liberty to require all code to be recompiled with ARC; therefore, ARC must interoperate with Objective-C code which manages retains and releases manually. In general, there are three requirements in order for a compiler-supported reference-count system to provide reliable interoperation: - The type system must reliably identify which objects are to be managed. An `int*` might be a pointer to a `malloc`'ed array, or it might be an interior pointer to such an array, or it might point to some field or local variable. In contrast, values of the retainable object pointer types are never interior. - The type system must reliably indicate how to manage objects of a type. This usually means that the type must imply a procedure for incrementing and decrementing retain counts. Supporting single-ownership objects requires a lot more explicit mediation in the language. - There must be reliable conventions for whether and when "ownership" is passed between caller and callee, for both arguments and return values. Objective-C methods follow such a convention very reliably, at least for system libraries on macOS, and functions always pass objects at +0. The C-based APIs for Core Foundation objects, on the other hand, have much more varied transfer semantics. ::: The use of `__attribute__((NSObject))` typedefs is not recommended. If it's absolutely necessary to use this attribute, be very explicit about using the typedef, and do not assume that it will be preserved by language features like `__typeof` and C++ template argument substitution. :::{admonition} Rationale Any compiler operation which incidentally strips type "sugar" from a type will yield a type without the attribute, which may result in unexpected behavior. ::: (arc.objects.retains)= ### Retain count semantics A retainable object pointer is either a {arc-term}`null pointer` or a pointer to a valid object. Furthermore, if it has block pointer type and is not `null` then it must actually be a pointer to a block object, and if it has `Class` type (possibly protocol-qualified) then it must actually be a pointer to a class object. Otherwise ARC does not enforce the Objective-C type system as long as the implementing methods follow the signature of the static type. It is undefined behavior if ARC is exposed to an invalid pointer. For ARC's purposes, a valid object is one with "well-behaved" retaining operations. Specifically, the object must be laid out such that the Objective-C message send machinery can successfully send it the following messages: - `retain`, taking no arguments and returning a pointer to the object. - `release`, taking no arguments and returning `void`. - `autorelease`, taking no arguments and returning a pointer to the object. The behavior of these methods is constrained in the following ways. The term {arc-term}`high-level semantics` is an intentionally vague term; the intent is that programmers must implement these methods in a way such that the compiler, modifying code in ways it deems safe according to these constraints, will not violate their requirements. For example, if the user puts logging statements in `retain`, they should not be surprised if those statements are executed more or less often depending on optimization settings. These constraints are not exhaustive of the optimization opportunities: values held in local variables are subject to additional restrictions, described later in this document. It is undefined behavior if a computation history featuring a send of `retain` followed by a send of `release` to the same object, with no intervening `release` on that object, is not equivalent under the high-level semantics to a computation history in which these sends are removed. Note that this implies that these methods may not raise exceptions. It is undefined behavior if a computation history features any use whatsoever of an object following the completion of a send of `release` that is not preceded by a send of `retain` to the same object. The behavior of `autorelease` must be equivalent to sending `release` when one of the autorelease pools currently in scope is popped. It may not throw an exception. When the semantics call for performing one of these operations on a retainable object pointer, if that pointer is `null` then the effect is a no-op. All of the semantics described in this document are subject to additional {ref}`optimization rules ` which permit the removal or optimization of operations based on local knowledge of data flow. The semantics describe the high-level behaviors that the compiler implements, not an exact sequence of operations that a program will be compiled into. (arc.objects.operands)= ### Retainable object pointers as operands and arguments In general, ARC does not perform retain or release operations when simply using a retainable object pointer as an operand within an expression. This includes: - loading a retainable pointer from an object with non-weak {ref}`ownership `, - passing a retainable pointer as an argument to a function or method, and - receiving a retainable pointer as the result of a function or method call. :::{admonition} Rationale While this might seem uncontroversial, it is actually unsafe when multiple expressions are evaluated in "parallel", as with binary operators and calls, because (for example) one expression might load from an object while another writes to it. However, C and C++ already call this undefined behavior because the evaluations are unsequenced, and ARC simply exploits that here to avoid needing to retain arguments across a large number of calls. ::: The remainder of this section describes exceptions to these rules, how those exceptions are detected, and what those exceptions imply semantically. (arc.objects.operands.consumed)= #### Consumed parameters A function or method parameter of retainable object pointer type may be marked as {arc-term}`consumed`, signifying that the callee expects to take ownership of a +1 retain count. This is done by adding the `ns_consumed` attribute to the parameter declaration, like so: ```objc void foo(__attribute((ns_consumed)) id x); - (void) foo: (id) __attribute((ns_consumed)) x; ``` This attribute is part of the type of the function or method, not the type of the parameter. It controls only how the argument is passed and received. When passing such an argument, ARC retains the argument prior to making the call. When receiving such an argument, ARC releases the argument at the end of the function, subject to the usual optimizations for local values. :::{admonition} Rationale This formalizes direct transfers of ownership from a caller to a callee. The most common scenario here is passing the `self` parameter to `init`, but it is useful to generalize. Typically, local optimization will remove any extra retains and releases: on the caller side the retain will be merged with a +1 source, and on the callee side the release will be rolled into the initialization of the parameter. ::: The implicit `self` parameter of a method may be marked as consumed by adding `__attribute__((ns_consumes_self))` to the method declaration. Methods in the `init` {ref}`family ` are treated as if they were implicitly marked with this attribute. It is undefined behavior if an Objective-C message send to a method with `ns_consumed` parameters (other than self) is made with a null receiver. It is undefined behavior if the method to which an Objective-C message send statically resolves to has a different set of `ns_consumed` parameters than the method it dynamically resolves to. It is undefined behavior if a block or function call is made through a static type with a different set of `ns_consumed` parameters than the implementation of the called block or function. :::{admonition} Rationale Consumed parameters with null receiver are a guaranteed leak. Mismatches with consumed parameters will cause over-retains or over-releases, depending on the direction. The rule about function calls is really just an application of the existing C/C++ rule about calling functions through an incompatible function type, but it's useful to state it explicitly. ::: (arc.object.operands.retained-return-values)= #### Retained return values A function or method which returns a retainable object pointer type may be marked as returning a retained value, signifying that the caller expects to take ownership of a +1 retain count. This is done by adding the `ns_returns_retained` attribute to the function or method declaration, like so: ```objc id foo(void) __attribute((ns_returns_retained)); - (id) foo __attribute((ns_returns_retained)); ``` This attribute is part of the type of the function or method. When returning from such a function or method, ARC retains the value at the point of evaluation of the return statement, before leaving all local scopes. When receiving a return result from such a function or method, ARC releases the value at the end of the full-expression it is contained within, subject to the usual optimizations for local values. :::{admonition} Rationale This formalizes direct transfers of ownership from a callee to a caller. The most common scenario this models is the retained return from `init`, `alloc`, `new`, and `copy` methods, but there are other cases in the frameworks. After optimization there are typically no extra retains and releases required. ::: Methods in the `alloc`, `copy`, `init`, `mutableCopy`, and `new` {ref}`families ` are implicitly marked `__attribute__((ns_returns_retained))`. This may be suppressed by explicitly marking the method `__attribute__((ns_returns_not_retained))`. It is undefined behavior if the method to which an Objective-C message send statically resolves has different retain semantics on its result from the method it dynamically resolves to. It is undefined behavior if a block or function call is made through a static type with different retain semantics on its result from the implementation of the called block or function. :::{admonition} Rationale Mismatches with returned results will cause over-retains or over-releases, depending on the direction. Again, the rule about function calls is really just an application of the existing C/C++ rule about calling functions through an incompatible function type. ::: (arc.objects.operands.unretained-returns)= #### Unretained return values A method or function which returns a retainable object type but does not return a retained value must ensure that the object is still valid across the return boundary. When returning from such a function or method, ARC retains the value at the point of evaluation of the return statement, then leaves all local scopes, and then balances out the retain while ensuring that the value lives across the call boundary. In the worst case, this may involve an `autorelease`, but callers must not assume that the value is actually in the autorelease pool. ARC performs no extra mandatory work on the caller side, although it may elect to do something to shorten the lifetime of the returned value. :::{admonition} Rationale It is common in non-ARC code to not return an autoreleased value; therefore the convention does not force either path. It is convenient to not be required to do unnecessary retains and autoreleases; this permits optimizations such as eliding retain/autoreleases when it can be shown that the original pointer will still be valid at the point of return. ::: A method or function may be marked with `__attribute__((ns_returns_autoreleased))` to indicate that it returns a pointer which is guaranteed to be valid at least as long as the innermost autorelease pool. There are no additional semantics enforced in the definition of such a method; it merely enables optimizations in callers. (arc.objects.operands.casts)= #### Bridged casts A {arc-term}`bridged cast` is a C-style cast annotated with one of three keywords: - `(__bridge T) op` casts the operand to the destination type `T`. If `T` is a retainable object pointer type, then `op` must have a non-retainable pointer type. If `T` is a non-retainable pointer type, then `op` must have a retainable object pointer type. Otherwise the cast is ill-formed. There is no transfer of ownership, and ARC inserts no retain operations. - `(__bridge_retained T) op` casts the operand, which must have retainable object pointer type, to the destination type, which must be a non-retainable pointer type. ARC retains the value, subject to the usual optimizations on local values, and the recipient is responsible for balancing that +1. - `(__bridge_transfer T) op` casts the operand, which must have non-retainable pointer type, to the destination type, which must be a retainable object pointer type. ARC will release the value at the end of the enclosing full-expression, subject to the usual optimizations on local values. These casts are required in order to transfer objects in and out of ARC control; see the rationale in the section on {ref}`conversion of retainable object pointers `. Using a `__bridge_retained` or `__bridge_transfer` cast purely to convince ARC to emit an unbalanced retain or release, respectively, is poor form. (arc.objects.restrictions)= ### Restrictions (arc.objects.restrictions.conversion)= #### Conversion of retainable object pointers In general, a program which attempts to implicitly or explicitly convert a value of retainable object pointer type to any non-retainable type, or vice-versa, is ill-formed. For example, an Objective-C object pointer shall not be converted to `void*`. As an exception, cast to `intptr_t` is allowed because such casts are not transferring ownership. The {ref}`bridged casts ` may be used to perform these conversions where necessary. :::{admonition} Rationale We cannot ensure the correct management of the lifetime of objects if they may be freely passed around as unmanaged types. The bridged casts are provided so that the programmer may explicitly describe whether the cast transfers control into or out of ARC. ::: However, the following exceptions apply. (arc.objects.restrictions.conversion.with.known.semantics)= #### Conversion to retainable object pointer type of expressions with known semantics {when-revised}`[beginning Apple 4.0, LLVM 3.1]` {revision}`These exceptions have been greatly expanded; they previously applied only to a much-reduced subset which is difficult to categorize but which included null pointers, message sends (under the given rules), and the various global constants.` An unbridged conversion to a retainable object pointer type from a type other than a retainable object pointer type is ill-formed, as discussed above, unless the operand of the cast has a syntactic form which is known retained, known unretained, or known retain-agnostic. An expression is {arc-term}`known retain-agnostic` if it is: - an Objective-C string literal, - a load from a `const` system global variable of {ref}`C retainable pointer type `, or - a null pointer constant. An expression is {arc-term}`known unretained` if it is an rvalue of {ref}`C retainable pointer type ` and it is: - a direct call to a function, and either that function has the `cf_returns_not_retained` attribute or it is an {ref}`audited ` function that does not have the `cf_returns_retained` attribute and does not follow the create/copy naming convention, - a message send, and the declared method either has the `cf_returns_not_retained` attribute or it has neither the `cf_returns_retained` attribute nor a {ref}`selector family ` that implies a retained result, or - {when-revised}`[beginning LLVM 3.6]` {revision}`a load from a` `const` {revision}`non-system global variable.` An expression is {arc-term}`known retained` if it is an rvalue of {ref}`C retainable pointer type ` and it is: - a message send, and the declared method either has the `cf_returns_retained` attribute, or it does not have the `cf_returns_not_retained` attribute but it does have a {ref}`selector family ` that implies a retained result. Furthermore: - a comma expression is classified according to its right-hand side, - a statement expression is classified according to its result expression, if it has one, - an lvalue-to-rvalue conversion applied to an Objective-C property lvalue is classified according to the underlying message send, and - a conditional operator is classified according to its second and third operands, if they agree in classification, or else the other if one is known retain-agnostic. If the cast operand is known retained, the conversion is treated as a `__bridge_transfer` cast. If the cast operand is known unretained or known retain-agnostic, the conversion is treated as a `__bridge` cast. :::{admonition} Rationale Bridging casts are annoying. Absent the ability to completely automate the management of CF objects, however, we are left with relatively poor attempts to reduce the need for a glut of explicit bridges. Hence these rules. We've so far consciously refrained from implicitly turning retained CF results from function calls into `__bridge_transfer` casts. The worry is that some code patterns --- for example, creating a CF value, assigning it to an ObjC-typed local, and then calling `CFRelease` when done --- are a bit too likely to be accidentally accepted, leading to mysterious behavior. For loads from `const` global variables of {ref}`C retainable pointer type `, it is reasonable to assume that global system constants were initialized with true constants (e.g. string literals), but user constants might have been initialized with something dynamically allocated, using a global initializer. ::: (arc.objects.restrictions.conversion-exception-contextual)= #### Conversion from retainable object pointer type in certain contexts {when-revised}`[beginning Apple 4.0, LLVM 3.1]` If an expression of retainable object pointer type is explicitly cast to a {ref}`C retainable pointer type `, the program is ill-formed as discussed above unless the result is immediately used: - to initialize a parameter in an Objective-C message send where the parameter is not marked with the `cf_consumed` attribute, or - to initialize a parameter in a direct call to an {ref}`audited ` function where the parameter is not marked with the `cf_consumed` attribute. :::{admonition} Rationale Consumed parameters are left out because ARC would naturally balance them with a retain, which was judged too treacherous. This is in part because several of the most common consuming functions are in the `Release` family, and it would be quite unfortunate for explicit releases to be silently balanced out in this way. ::: (arc.ownership)= ## Ownership qualification This section describes the behavior of *objects* of retainable object pointer type; that is, locations in memory which store retainable object pointers. A type is a {arc-term}`retainable object owner type` if it is a retainable object pointer type or an array type whose element type is a retainable object owner type. An {arc-term}`ownership qualifier` is a type qualifier which applies only to retainable object owner types. An array type is ownership-qualified according to its element type, and adding an ownership qualifier to an array type so qualifies its element type. A program is ill-formed if it attempts to apply an ownership qualifier to a type which is already ownership-qualified, even if it is the same qualifier. There is a single exception to this rule: an ownership qualifier may be applied to a substituted template type parameter, which overrides the ownership qualifier provided by the template argument. When forming a function type, the result type is adjusted so that any top-level ownership qualifier is deleted. Except as described under the {ref}`inference rules `, a program is ill-formed if it attempts to form a pointer or reference type to a retainable object owner type which lacks an ownership qualifier. :::{admonition} Rationale These rules, together with the inference rules, ensure that all objects and lvalues of retainable object pointer type have an ownership qualifier. The ability to override an ownership qualifier during template substitution is required to counteract the {ref}`inference of __strong for template type arguments `. Ownership qualifiers on return types are dropped because they serve no purpose there except to cause spurious problems with overloading and templates. ::: There are four ownership qualifiers: - `__autoreleasing` - `__strong` - `__unsafe_unretained` - `__weak` A type is {arc-term}`nontrivially ownership-qualified` if it is qualified with `__autoreleasing`, `__strong`, or `__weak`. (arc.ownership.spelling)= ### Spelling The names of the ownership qualifiers are reserved for the implementation. A program may not assume that they are or are not implemented with macros, or what those macros expand to. An ownership qualifier may be written anywhere that any other type qualifier may be written. If an ownership qualifier appears in the *declaration-specifiers*, the following rules apply: - if the type specifier is a retainable object owner type, the qualifier initially applies to that type; - otherwise, if the outermost non-array declarator is a pointer or block pointer declarator, the qualifier initially applies to that type; - otherwise the program is ill-formed. - If the qualifier is so applied at a position in the declaration where the next-innermost declarator is a function declarator, and there is a block declarator within that function declarator, then the qualifier applies instead to that block declarator and this rule is considered afresh beginning from the new position. If an ownership qualifier appears on the declarator name, or on the declared object, it is applied to the innermost pointer or block-pointer type. If an ownership qualifier appears anywhere else in a declarator, it applies to the type there. :::{admonition} Rationale Ownership qualifiers are like `const` and `volatile` in the sense that they may sensibly apply at multiple distinct positions within a declarator. However, unlike those qualifiers, there are many situations where they are not meaningful, and so we make an effort to "move" the qualifier to a place where it will be meaningful. The general goal is to allow the programmer to write, say, `__strong` before the entire declaration and have it apply in the leftmost sensible place. ::: (arc.ownership.spelling.property)= #### Property declarations A property of retainable object pointer type may have ownership. If the property's type is ownership-qualified, then the property has that ownership. If the property has one of the following modifiers, then the property has the corresponding ownership. A property is ill-formed if it has conflicting sources of ownership, or if it has redundant ownership modifiers, or if it has `__autoreleasing` ownership. - `assign` implies `__unsafe_unretained` ownership. - `copy` implies `__strong` ownership, as well as the usual behavior of copy semantics on the setter. - `retain` implies `__strong` ownership. - `strong` implies `__strong` ownership. - `unsafe_unretained` implies `__unsafe_unretained` ownership. - `weak` implies `__weak` ownership. With the exception of `weak`, these modifiers are available in non-ARC modes. A property's specified ownership is preserved in its metadata, but otherwise the meaning is purely conventional unless the property is synthesized. If a property is synthesized, then the {arc-term}`associated instance variable` is the instance variable which is named, possibly implicitly, by the `@synthesize` declaration. If the associated instance variable already exists, then its ownership qualification must equal the ownership of the property; otherwise, the instance variable is created with that ownership qualification. A property of retainable object pointer type which is synthesized without a source of ownership has the ownership of its associated instance variable, if it already exists; otherwise, {when-revised}`[beginning Apple 3.1, LLVM 3.1]` {revision}`its ownership is implicitly` `strong`. Prior to this revision, it was ill-formed to synthesize such a property. :::{admonition} Rationale Using `strong` by default is safe and consistent with the generic ARC rule about {ref}`inferring ownership `. It is, unfortunately, inconsistent with the non-ARC rule which states that such properties are implicitly `assign`. However, that rule is clearly untenable in ARC, since it leads to default-unsafe code. The main merit to banning the properties is to avoid confusion with non-ARC practice, which did not ultimately strike us as sufficient to justify requiring extra syntax and (more importantly) forcing novices to understand ownership rules just to declare a property when the default is so reasonable. Changing the rule away from non-ARC practice was acceptable because we had conservatively banned the synthesis in order to give ourselves exactly this leeway. ::: Applying `__attribute__((NSObject))` to a property not of retainable object pointer type has the same behavior it does outside of ARC: it requires the property type to be some sort of pointer and permits the use of modifiers other than `assign`. These modifiers only affect the synthesized getter and setter; direct accesses to the ivar (even if synthesized) still have primitive semantics, and the value in the ivar will not be automatically released during deallocation. (arc.ownership.semantics)= ### Semantics There are five {arc-term}`managed operations` which may be performed on an object of retainable object pointer type. Each qualifier specifies different semantics for each of these operations. It is still undefined behavior to access an object outside of its lifetime. A load or store with "primitive semantics" has the same semantics as the respective operation would have on an `void*` lvalue with the same alignment and non-ownership qualification. {arc-term}`Reading` occurs when performing a lvalue-to-rvalue conversion on an object lvalue. - For `__weak` objects, the current pointee is retained and then released at the end of the current full-expression. In particular, messaging a `__weak` object keeps the object retained until the end of the full expression. ```objc __weak MyObject *weakObj; void foo() { // weakObj is retained before the message send and released at the end of // the full expression. [weakObj m]; } ``` This must execute atomically with respect to assignments and to the final release of the pointee. - For all other objects, the lvalue is loaded with primitive semantics. {arc-term}`Assignment` occurs when evaluating an assignment operator. The semantics vary based on the qualification: - For `__strong` objects, the new pointee is first retained; second, the lvalue is loaded with primitive semantics; third, the new pointee is stored into the lvalue with primitive semantics; and finally, the old pointee is released. This is not performed atomically; external synchronization must be used to make this safe in the face of concurrent loads and stores. - For `__weak` objects, the lvalue is updated to point to the new pointee, unless the new pointee is an object currently undergoing deallocation, in which case the lvalue is updated to a null pointer. This must execute atomically with respect to other assignments to the object, to reads from the object, and to the final release of the new pointee. - For `__unsafe_unretained` objects, the new pointee is stored into the lvalue using primitive semantics. - For `__autoreleasing` objects, the new pointee is retained, autoreleased, and stored into the lvalue using primitive semantics. {arc-term}`Initialization` occurs when an object's lifetime begins, which depends on its storage duration. Initialization proceeds in two stages: 1. First, a null pointer is stored into the lvalue using primitive semantics. This step is skipped if the object is `__unsafe_unretained`. 2. Second, if the object has an initializer, that expression is evaluated and then assigned into the object using the usual assignment semantics. {arc-term}`Destruction` occurs when an object's lifetime ends. In all cases it is semantically equivalent to assigning a null pointer to the object, with the proviso that of course the object cannot be legally read after the object's lifetime ends. {arc-term}`Moving` occurs in specific situations where an lvalue is "moved from", meaning that its current pointee will be used but the object may be left in a different (but still valid) state. This arises with `__block` variables and rvalue references in C++. For `__strong` lvalues, moving is equivalent to loading the lvalue with primitive semantics, writing a null pointer to it with primitive semantics, and then releasing the result of the load at the end of the current full-expression. For all other lvalues, moving is equivalent to reading the object. (arc.ownership.restrictions)= ### Restrictions (arc.ownership.restrictions.weak)= #### Weak-unavailable types It is explicitly permitted for Objective-C classes to not support `__weak` references. It is undefined behavior to perform an operation with weak assignment semantics with a pointer to an Objective-C object whose class does not support `__weak` references. :::{admonition} Rationale Historically, it has been possible for a class to provide its own reference-count implementation by overriding `retain`, `release`, etc. However, weak references to an object require coordination with its class's reference-count implementation because, among other things, weak loads and stores must be atomic with respect to the final release. Therefore, existing custom reference-count implementations will generally not support weak references without additional effort. This is unavoidable without breaking binary compatibility. ::: A class may indicate that it does not support weak references by providing the `objc_arc_weak_reference_unavailable` attribute on the class's interface declaration. A retainable object pointer type is **weak-unavailable** if it is a pointer to an (optionally protocol-qualified) Objective-C class `T` where `T` or one of its superclasses has the `objc_arc_weak_reference_unavailable` attribute. A program is ill-formed if it applies the `__weak` ownership qualifier to a weak-unavailable type or if the value operand of a weak assignment operation has a weak-unavailable type. (arc.ownership.restrictions.autoreleasing)= #### Storage duration of `__autoreleasing` objects A program is ill-formed if it declares an `__autoreleasing` object of non-automatic storage duration. A program is ill-formed if it captures an `__autoreleasing` object in a block or, unless by reference, in a C++11 lambda. :::{admonition} Rationale Autorelease pools are tied to the current thread and scope by their nature. While it is possible to have temporary objects whose instance variables are filled with autoreleased objects, there is no way that ARC can provide any sort of safety guarantee there. ::: It is undefined behavior if a non-null pointer is assigned to an `__autoreleasing` object while an autorelease pool is in scope and then that object is read after the autorelease pool's scope is left. (arc.ownership.restrictions.conversion.indirect)= #### Conversion of pointers to ownership-qualified types A program is ill-formed if an expression of type `T*` is converted, explicitly or implicitly, to the type `U*`, where `T` and `U` have different ownership qualification, unless: - `T` is qualified with `__strong`, `__autoreleasing`, or `__unsafe_unretained`, and `U` is qualified with both `const` and `__unsafe_unretained`; or - either `T` or `U` is `cv void`, where `cv` is an optional sequence of non-ownership qualifiers; or - the conversion is requested with a `reinterpret_cast` in Objective-C++; or - the conversion is a well-formed {ref}`pass-by-writeback `. The analogous rule applies to `T&` and `U&` in Objective-C++. :::{admonition} Rationale These rules provide a reasonable level of type-safety for indirect pointers, as long as the underlying memory is not deallocated. The conversion to `const __unsafe_unretained` is permitted because the semantics of reads are equivalent across all these ownership semantics, and that's a very useful and common pattern. The interconversion with `void*` is useful for allocating memory or otherwise escaping the type system, but use it carefully. `reinterpret_cast` is considered to be an obvious enough sign of taking responsibility for any problems. ::: It is undefined behavior to access an ownership-qualified object through an lvalue of a differently-qualified type, except that any non-`__weak` object may be read through an `__unsafe_unretained` lvalue. It is undefined behavior if the storage of a `__strong` or `__weak` object is not properly initialized before the first managed operation is performed on the object, or if the storage of such an object is freed or reused before the object has been properly deinitialized. Storage for a `__strong` or `__weak` object may be properly initialized by filling it with the representation of a null pointer, e.g. by acquiring the memory with `calloc` or using `bzero` to zero it out. A `__strong` or `__weak` object may be properly deinitialized by assigning a null pointer into it. A `__strong` object may also be properly initialized by copying into it (e.g. with `memcpy`) the representation of a different `__strong` object whose storage has been properly initialized; doing this properly deinitializes the source object and causes its storage to no longer be properly initialized. A `__weak` object may not be representation-copied in this way. These requirements are followed automatically for objects whose initialization and deinitialization are under the control of ARC: - objects of static, automatic, and temporary storage duration - instance variables of Objective-C objects - elements of arrays where the array object's initialization and deinitialization are under the control of ARC - fields of Objective-C struct types where the struct object's initialization and deinitialization are under the control of ARC - non-static data members of Objective-C++ non-union class types - Objective-C++ objects and arrays of dynamic storage duration created with the `new` or `new[]` operators and destroyed with the corresponding `delete` or `delete[]` operator They are not followed automatically for these objects: - objects of dynamic storage duration created in other memory, such as that returned by `malloc` - union members :::{admonition} Rationale ARC must perform special operations when initializing an object and when destroying it. In many common situations, ARC knows when an object is created and when it is destroyed and can ensure that these operations are performed correctly. Otherwise, however, ARC requires programmer cooperation to establish its initialization invariants because it is infeasible for ARC to dynamically infer whether they are intact. For example, there is no syntactic difference in C between an assignment that is intended by the programmer to initialize a variable and one that is intended to replace the existing value stored there, but ARC must perform one operation or the other. ARC chooses to always assume that objects are initialized (except when it is in charge of initializing them) because the only workable alternative would be to ban all code patterns that could potentially be used to access uninitialized memory, and that would be too limiting. In practice, this is rarely a problem because programmers do not generally need to work with objects for which the requirements are not handled automatically. ::: Note that dynamically-allocated Objective-C++ arrays of nontrivially-ownership-qualified type are not ABI-compatible with non-ARC code because the non-ARC code will consider the element type to be POD. Such arrays that are `new[]`'d in ARC translation units cannot be `delete[]`'d in non-ARC translation units and vice-versa. (arc.ownership.restrictions.pass_by_writeback)= #### Passing to an out parameter by writeback If the argument passed to a parameter of type `T __autoreleasing *` has type `U oq *`, where `oq` is an ownership qualifier, then the argument is a candidate for {arc-term}`pass-by-writeback`` if: - `oq` is `__strong` or `__weak`, and - it would be legal to initialize a `T __strong *` with a `U __strong *`. For purposes of overload resolution, an implicit conversion sequence requiring a pass-by-writeback is always worse than an implicit conversion sequence not requiring a pass-by-writeback. The pass-by-writeback is ill-formed if the argument expression does not have a legal form: - `&var`, where `var` is a scalar variable of automatic storage duration with retainable object pointer type - a conditional expression where the second and third operands are both legal forms - a cast whose operand is a legal form - a null pointer constant :::{admonition} Rationale The restriction in the form of the argument serves two purposes. First, it makes it impossible to pass the address of an array to the argument, which serves to protect against an otherwise serious risk of mis-inferring an "array" argument as an out-parameter. Second, it makes it much less likely that the user will see confusing aliasing problems due to the implementation, below, where their store to the writeback temporary is not immediately seen in the original argument variable. ::: A pass-by-writeback is evaluated as follows: 1. The argument is evaluated to yield a pointer `p` of type `U oq *`. 2. If `p` is a null pointer, then a null pointer is passed as the argument, and no further work is required for the pass-by-writeback. 3. Otherwise, a temporary of type `T __autoreleasing` is created and initialized to a null pointer. 4. If the parameter is not an Objective-C method parameter marked `out`, then `*p` is read, and the result is written into the temporary with primitive semantics. 5. The address of the temporary is passed as the argument to the actual call. 6. After the call completes, the temporary is loaded with primitive semantics, and that value is assigned into `*p`. :::{admonition} Rationale This is all admittedly convoluted. In an ideal world, we would see that a local variable is being passed to an out-parameter and retroactively modify its type to be `__autoreleasing` rather than `__strong`. This would be remarkably difficult and not always well-founded under the C type system. However, it was judged unacceptably invasive to require programmers to write `__autoreleasing` on all the variables they intend to use for out-parameters. This was the least bad solution. ::: (arc.ownership.restrictions.records)= #### Ownership-qualified fields of structs and unions A member of a struct or union may be declared to have ownership-qualified type. If the type is qualified with `__unsafe_unretained`, the semantics of the containing aggregate are unchanged from the semantics of an unqualified type in a non-ARC mode. If the type is qualified with `__autoreleasing`, the program is ill-formed. Otherwise, if the type is nontrivially ownership-qualified, additional rules apply. Both Objective-C and Objective-C++ support nontrivially ownership-qualified fields. Due to formal differences between the standards, the formal treatment is different; however, the basic language model is intended to be the same for identical code. :::{admonition} Rationale Permitting `__strong` and `__weak` references in aggregate types allows programmers to take advantage of the normal language tools of C and C++ while still automatically managing memory. While it is usually simpler and more idiomatic to use Objective-C objects for secondary data structures, doing so can introduce extra allocation and message-send overhead, which can cause unacceptable performance. Using structs can resolve some of this tension. `__autoreleasing` is forbidden because it is treacherous to rely on autoreleases as an ownership tool outside of a function-local contexts. Earlier releases of Clang permitted `__strong` and `__weak` only references in Objective-C++ classes, not in Objective-C. This restriction was an undesirable short-term constraint arising from the complexity of adding support for non-trivial struct types to C. ::: In Objective-C++, nontrivially ownership-qualified types are treated for nearly all purposes as if they were class types with non-trivial default constructors, copy constructors, move constructors, copy assignment operators, move assignment operators, and destructors. This includes the determination of the triviality of special members of classes with a non-static data member of such a type. In Objective-C, the definition cannot be so succinct: because the C standard lacks rules for non-trivial types, those rules must first be developed. They are given in the next section. The intent is that these rules are largely consistent with the rules of C++ for code expressible in both languages. ##### Formal rules for non-trivial types in C The following are base rules which can be added to C to support implementation-defined non-trivial types. A type in C is said to be *non-trivial to copy*, *non-trivial to destroy*, or *non-trivial to default-initialize* if: - it is a struct or union containing a member whose type is non-trivial to (respectively) copy, destroy, or default-initialize; - it is a qualified type whose unqualified type is non-trivial to (respectively) copy, destroy, or default-initialize (for at least the standard C qualifiers); or - it is an array type whose element type is non-trivial to (respectively) copy, destroy, or default-initialize. A type in C is said to be *illegal to copy*, *illegal to destroy*, or *illegal to default-initialize* if: - it is a union which contains a member whose type is either illegal or non-trivial to (respectively) copy, destroy, or initialize; - it is a qualified type whose unqualified type is illegal to (respectively) copy, destroy, or default-initialize (for at least the standard C qualifiers); or - it is an array type whose element type is illegal to (respectively) copy, destroy, or default-initialize. No type describable under the rules of the C standard shall be either non-trivial or illegal to copy, destroy, or default-initialize. An implementation may provide additional types which have one or more of these properties. An expression calls for a type to be copied if it: - passes an argument of that type to a function call, - defines a function which declares a parameter of that type, - calls or defines a function which returns a value of that type, - assigns to an l-value of that type, or - converts an l-value of that type to an r-value. A program calls for a type to be destroyed if it: - passes an argument of that type to a function call, - defines a function which declares a parameter of that type, - calls or defines a function which returns a value of that type, - creates an object of automatic storage duration of that type, - assigns to an l-value of that type, or - converts an l-value of that type to an r-value. A program calls for a type to be default-initialized if it: - declares a variable of that type without an initializer. An expression is ill-formed if calls for a type to be copied, destroyed, or default-initialized and that type is illegal to (respectively) copy, destroy, or default-initialize. A program is ill-formed if it contains a function type specifier with a parameter or return type that is illegal to copy or destroy. If a function type specifier would be ill-formed for this reason except that the parameter or return type was incomplete at that point in the translation unit, the program is ill-formed but no diagnostic is required. A `goto` or `switch` is ill-formed if it jumps into the scope of an object of automatic storage duration whose type is non-trivial to destroy. C specifies that it is generally undefined behavior to access an l-value if there is no object of that type at that location. Implementations are often lenient about this, but non-trivial types generally require it to be enforced more strictly. The following rules apply: The *static subobjects* of a type `T` at a location `L` are: - an object of type `T` spanning from `L` to `L + sizeof(T)`; - if `T` is a struct type, then for each field `f` of that struct, the static subobjects of `T` at location `L + offsetof(T, .f)`; and - if `T` is the array type `E[N]`, then for each `i` satisfying `0 <= i < N`, the static subobjects of `E` at location `L + i * sizeof(E)`. If an l-value is converted to an r-value, then all static subobjects whose types are non-trivial to copy are accessed. If an l-value is assigned to, or if an object of automatic storage duration goes out of scope, then all static subobjects of types that are non-trivial to destroy are accessed. A dynamic object is created at a location if an initialization initializes an object of that type there. A dynamic object ceases to exist at a location if the memory is repurposed. Memory is repurposed if it is freed or if a different dynamic object is created there, for example by assigning into a different union member. An implementation may provide additional rules for what constitutes creating or destroying a dynamic object. If an object is accessed under these rules at a location where no such dynamic object exists, the program has undefined behavior. If memory for a location is repurposed while a dynamic object that is non-trivial to destroy exists at that location, the program has undefined behavior. :::{admonition} Rationale While these rules are far less fine-grained than C++, they are nonetheless sufficient to express a wide spectrum of types. Types that express some sort of ownership will generally be non-trivial to both copy and destroy and either non-trivial or illegal to default-initialize. Types that don't express ownership may still be non-trivial to copy because of some sort of address sensitivity; for example, a relative reference. Distinguishing default initialization allows types to impose policies about how they are created. These rules assume that assignment into an l-value is always a modification of an existing object rather than an initialization. Assignment is then a compound operation where the old value is read and destroyed, if necessary, and the new value is put into place. These are the natural semantics of value propagation, where all basic operations on the type come down to copies and destroys, and everything else is just an optimization on top of those. The most glaring weakness of programming with non-trivial types in C is that there are no language mechanisms (akin to C++'s placement `new` and explicit destructor calls) for explicitly creating and destroying objects. Clang should consider adding builtins for this purpose, as well as for common optimizations like destructive relocation. ::: ##### Application of the formal C rules to nontrivial ownership qualifiers Nontrivially ownership-qualified types are considered non-trivial to copy, destroy, and default-initialize. A dynamic object of nontrivially ownership-qualified type contingently exists at a location if the memory is filled with a zero pattern, e.g. by `calloc` or `bzero`. Such an object can be safely accessed in all of the cases above, but its memory can also be safely repurposed. Assigning a null pointer into an l-value of `__weak` or `__strong`-qualified type accesses the dynamic object there (and thus may have undefined behavior if no such object exists), but afterwards the object's memory is guaranteed to be filled with a zero pattern and thus may be either further accessed or repurposed as needed. The upshot is that programs may safely initialize dynamically-allocated memory for nontrivially ownership-qualified types by ensuring it is zero-initialized, and they may safely deinitialize memory before freeing it by storing `nil` into any `__strong` or `__weak` references previously created in that memory. ##### C/C++ compatibility for structs and unions with non-trivial members Structs and unions with non-trivial members are compatible in different language modes (e.g. between Objective-C and Objective-C++, or between ARC and non-ARC modes) under the following conditions: - The types must be compatible ignoring ownership qualifiers according to the baseline, non-ARC rules (e.g. C struct compatibility or C++'s ODR). This condition implies a pairwise correspondence between fields. Note that an Objective-C++ class with base classes, a user-provided copy or move constructor, or a user-provided destructor is never compatible with an Objective-C type. - If two fields correspond as above, and at least one of the fields is ownership-qualified, then: - the fields must be identically qualified, or else - one type must be unqualified (and thus declared in a non-ARC mode), and the other type must be qualified with `__unsafe_unretained` or `__strong`. Note that `__weak` fields must always be declared `__weak` because of the need to pin those fields in memory and keep them properly registered with the Objective-C runtime. Non-ARC modes may still declare fields `__weak` by enabling `-fobjc-weak`. These compatibility rules permit a function that takes a parameter of non-trivial struct type to be written in ARC and called from non-ARC or vice-versa. The convention for this always transfers ownership of objects stored in `__strong` fields from the caller to the callee, just as for an `ns_consumed` argument. Therefore, non-ARC callers must ensure that such fields are initialized to a +1 reference, and non-ARC callees must balance that +1 by releasing the reference or transferring it as appropriate. Likewise, a function returning a non-trivial struct may be written in ARC and called from non-ARC or vice-versa. The convention for this always transfers ownership of objects stored in `__strong` fields from the callee to the caller, and so callees must initialize such fields with +1 references, and callers must balance that +1 by releasing or transferring them. Similar transfers of responsibility occur for `__weak` fields, but since both sides must use native `__weak` support to ensure calling convention compatibility, this transfer is always handled automatically by the compiler. :::{admonition} Rationale In earlier releases, when non-trivial ownership was only permitted on fields in Objective-C++, the ABI used for such classes was the ordinary ABI for non-trivial C++ classes, which passes arguments and returns indirectly and does not transfer responsibility for arguments. When support for Objective-C structs was added, it was decided to change to the current ABI for three reasons: - It permits ARC / non-ARC compatibility for structs containing only `__strong` references, as long as the non-ARC side is careful about transferring ownership. - It avoids unnecessary indirection for sufficiently small types that the C ABI would prefer to pass in registers. - Given that struct arguments must be produced at +1 to satisfy C's semantics of initializing the local parameter variable, transferring ownership of that copy to the callee is generally better for ARC optimization, since otherwise there will be releases in the caller that are much harder to pair with transfers in the callee. Breaking compatibility with existing Objective-C++ structures was considered an acceptable cost, as most Objective-C++ code does not have binary-compatibility requirements. Any existing code which cannot accept this compatibility break, which is necessarily Objective-C++, should force the use of the standard C++ ABI by declaring an empty (but non-defaulted) destructor. ::: (arc.ownership.inference)= ### Ownership inference (arc.ownership.inference.variables)= #### Objects If an object is declared with retainable object owner type, but without an explicit ownership qualifier, its type is implicitly adjusted to have `__strong` qualification. As a special case, if the object's base type is `Class` (possibly protocol-qualified), the type is adjusted to have `__unsafe_unretained` qualification instead. (arc.ownership.inference.indirect_parameters)= #### Indirect parameters If a function or method parameter has type `T*`, where `T` is an ownership-unqualified retainable object pointer type, then: - if `T` is `const`-qualified or `Class`, then it is implicitly qualified with `__unsafe_unretained`; - otherwise, it is implicitly qualified with `__autoreleasing`. :::{admonition} Rationale `__autoreleasing` exists mostly for this case, the Cocoa convention for out-parameters. Since a pointer to `const` is obviously not an out-parameter, we instead use a type more useful for passing arrays. If the user instead intends to pass in a *mutable* array, inferring `__autoreleasing` is the wrong thing to do; this directs some of the caution in the following rules about writeback. ::: Such a type written anywhere else would be ill-formed by the general rule requiring ownership qualifiers. This rule does not apply in Objective-C++ if a parameter's type is dependent in a template pattern and is only *instantiated* to a type which would be a pointer to an unqualified retainable object pointer type. Such code is still ill-formed. :::{admonition} Rationale The convention is very unlikely to be intentional in template code. ::: (arc.ownership.inference.template.arguments)= #### Template arguments If a template argument for a template type parameter is a retainable object owner type that does not have an explicit ownership qualifier, it is adjusted to have `__strong` qualification. This adjustment occurs regardless of whether the template argument was deduced or explicitly specified. :::{admonition} Rationale `__strong` is a useful default for containers (e.g., `std::vector`), which would otherwise require explicit qualification. Moreover, unqualified retainable object pointer types are unlikely to be useful within templates, since they generally need to have a qualifier applied to the before being used. ::: (arc.method-families)= ## Method families An Objective-C method may fall into a {arc-term}`method family`, which is a conventional set of behaviors ascribed to it by the Cocoa conventions. A method is in a certain method family if: - it has a `objc_method_family` attribute placing it in that family; or if not that, - it does not have an `objc_method_family` attribute placing it in a different or no family, and - its selector falls into the corresponding selector family, and - its signature obeys the added restrictions of the method family. A selector is in a certain selector family if, ignoring any leading underscores, the first component of the selector either consists entirely of the name of the method family or it begins with that name followed by a character other than a lowercase letter. For example, `_perform:with:` and `performWith:` would fall into the `perform` family (if we recognized one), but `performing:with` would not. The families and their added restrictions are: - `alloc` methods must return a retainable object pointer type. - `copy` methods must return a retainable object pointer type. - `mutableCopy` methods must return a retainable object pointer type. - `new` methods must return a retainable object pointer type. - `init` methods must be instance methods and must return an Objective-C pointer type. Additionally, a program is ill-formed if it declares or contains a call to an `init` method whose return type is neither `id` nor a pointer to a super-class or sub-class of the declaring class (if the method was declared on a class) or the static receiver type of the call (if it was declared on a protocol). :::{admonition} Rationale There are a fair number of existing methods with `init`-like selectors which nonetheless don't follow the `init` conventions. Typically these are either accidental naming collisions or helper methods called during initialization. Because of the peculiar retain/release behavior of `init` methods, it's very important not to treat these methods as `init` methods if they aren't meant to be. It was felt that implicitly defining these methods out of the family based on the exact relationship between the return type and the declaring class would be much too subtle and fragile. Therefore we identify a small number of legitimate-seeming return types and call everything else an error. This serves the secondary purpose of encouraging programmers not to accidentally give methods names in the `init` family. Note that a method with an `init`-family selector which returns a non-Objective-C type (e.g. `void`) is perfectly well-formed; it simply isn't in the `init` family. ::: A program is ill-formed if a method's declarations, implementations, and overrides do not all have the same method family. (arc.family.attribute)= ### Explicit method family control A method may be annotated with the `objc_method_family` attribute to precisely control which method family it belongs to. If a method in an `@implementation` does not have this attribute, but there is a method declared in the corresponding `@interface` that does, then the attribute is copied to the declaration in the `@implementation`. The attribute is available outside of ARC, and may be tested for with the preprocessor query `__has_attribute(objc_method_family)`. The attribute is spelled `__attribute__((objc_method_family(` *family* `)))`. If *family* is `none`, the method has no family, even if it would otherwise be considered to have one based on its selector and type. Otherwise, *family* must be one of `alloc`, `copy`, `init`, `mutableCopy`, or `new`, in which case the method is considered to belong to the corresponding family regardless of its selector. It is an error if a method that is explicitly added to a family in this way does not meet the requirements of the family other than the selector naming convention. :::{admonition} Rationale The rules codified in this document describe the standard conventions of Objective-C. However, as these conventions have not heretofore been enforced by an unforgiving mechanical system, they are only imperfectly kept, especially as they haven't always even been precisely defined. While it is possible to define low-level ownership semantics with attributes like `ns_returns_retained`, this attribute allows the user to communicate semantic intent, which is of use both to ARC (which, e.g., treats calls to `init` specially) and the static analyzer. ::: (arc.family.semantics)= ### Semantics of method families A method's membership in a method family may imply non-standard semantics for its parameters and return type. Methods in the `alloc`, `copy`, `mutableCopy`, and `new` families --- that is, methods in all the currently-defined families except `init` --- implicitly {ref}`return a retained object ` as if they were annotated with the `ns_returns_retained` attribute. This can be overridden by annotating the method with either of the `ns_returns_autoreleased` or `ns_returns_not_retained` attributes. Properties also follow same naming rules as methods. This means that those in the `alloc`, `copy`, `mutableCopy`, and `new` families provide access to {ref}`retained objects `. This can be overridden by annotating the property with `ns_returns_not_retained` attribute. (arc.family.semantics.init)= #### Semantics of `init` Methods in the `init` family implicitly {ref}`consume ` their `self` parameter and {ref}`return a retained object `. Neither of these properties can be altered through attributes. A call to an `init` method with a receiver that is either `self` (possibly parenthesized or casted) or `super` is called a {arc-term}`delegate init call`. It is an error for a delegate init call to be made except from an `init` method, and excluding blocks within such methods. As an exception to the {ref}`usual rule