### Documentation/Website/Docs/Warning Codes/5001 --- id: code-5001-public title: 5001 - Code Injection sidebar_label: 5001 - Code Injection --- ## TL;DR This category indicates that user-controlled input flows into a sink that allows code or shell command execution. This directly leads to Remote Code Execution which can be assumed to mean complete compromise of the server. ## RCE via Code Injection (eval/exec) ### ISSUE The simplest kind of RCE involves user input flowing into a function such as eval or exec which are intended to interpret or run python code. ### EXAMPLE ```python def update_search_account_filtering(request: HttpRequest) -> HttpResponse: ... if action in ["delete", "add", "update"]: ... filter_by_username = eval(request.POST.get("filter_by_username", "True")) ``` ### RECOMMENDED SOLUTION There are few reasons to use these functions, and even fewer reasons to allow a user to control the content of these functions. Generally, we recommend not making calls to these functions with user input. If you only need to eval python datatypes you can use `ast.literal_eval`. Using it on arbitrary user input can still lead to DOS attack but can't be exploited for code execution ([details](https://docs.python.org/3/library/ast.html#ast.literal_eval)). ## RCE via Command Injection (os.system) ### ISSUE This kind of RCE involves user input flowing into a command executed in a system shell. If a user can control a portion of the command being executed in a shell, they can potentially add additional arbitrary commands to be executed. ### EXAMPLE The following code is intended to run the spellcheck binary on a user provided text: ```python def spellcheck(request: HttpRequest): command = f"/usr/bin/spellcheck -l {request.GET['text']}" return os.system(command) ``` An attacker, however, can supply a path such as `'test' && rm -rf /`, which would result in the following command being executed: `/usr/bin/spellcheck -l 'test' && rm -rf /`. Since this command is executed in a system shell the `rm -rf /` command will be executed after the spellcheck command. ### RECOMMENDED SOLUTION In general, we recommend avoiding creation of a subprocess and prefer using the API provided by the language. However, if you need to create a subprocess, we recommend using an API such as `subprocess.run`, which allows you to separate arguments from the executable being invoked. **DO NOT add the `shell=True` argument otherwise the code would still be vulnerable like the previous example** ```python def spellcheck(request: HttpRequest): command = ["/usr/bin/spellcheck", "-l", request.GET['text']] subprocess.run(command) ``` *NOTE: be conscious of the fact that arguments to an executable can still lead to code execution (e.g., the `-exec` argument of `find`).* --- ### Documentation/Website/Docs/Warning Codes/6065 --- id: code-6065-public title: 6065 - Commandline arguments injection sidebar_label: 6065 - Commandline arguments injection --- ## TL;DR This category indicates that user-controlled input flows into a command-line argument used to execute an external process. Unlike category 5001, this leads to a Remote Code Execution issue only in specific cases (e.g., `shell=True` parameter or when executing particular binaries). ### ISSUE `subprocess.Popen`, `subprocess.run`, `subprocess.call`, and other functions do a good job in preventing by default the command injection issues we described in category [5001](warning_codes/5001.md). The values supplied in the `args` parameter (excluding the first which represents the executable) are considered only as arguments and not as commands to be interpreted in a system shell (more details in the python [documentation](https://docs.python.org/3/library/subprocess.html#subprocess.Popen)). However, this safe behaviour can be manually bypassed by specifying the `shell=True` parameter, which reintroduces the command injection issue. ### EXAMPLE The following code is intended to run the spellcheck binary on a user provided text: ```python def spellcheck(request: HttpRequest): command = "/usr/bin/spellcheck -l {}".format(request.GET['text']) return subprocess.run(command, shell=True) ``` An attacker, however, can supply a path such as `'test' && rm -rf /`, which would result in the following command being executed: `/usr/bin/spellcheck -l 'test' && rm -rf /`. Since this command is executed in a system shell the `rm -rf /` command will be executed after the spellcheck command. ### RECOMMENDED SOLUTION In general, we recommend avoiding creation of a subprocess and prefer using the API provided by the language. However, if you need to create a subprocess, we recommend using a safe API such as `subprocess.run` and avoiding use of the `shell=True` argument. If this is not possible, we recommend ensuring that the user-controlled values are shell-escaped with `shlex.quote`. ```python def spellcheck(request: HttpRequest): command = ["/usr/bin/spellcheck", "-l", request.GET['text']] subprocess.run(command) ``` *NOTE: be conscious of the fact that arguments to an executable can still lead to code execution (e.g., the `-exec` argument of `find`).* --- ### Documentation/Website/Docs/Warning Codes/Overview --- id: overview-public title: Overview sidebar_label: Overview --- Pysa tracks data flows from a set of sources to sinks. Each set of sources we track to sinks is assigned a unique code. When Pysa finds a flow, it emits issues associated with the unique code. This section of the website contains documentation for the warning codes we emit. The source of truth for these warning codes is the `taint_config.json` files used in a given run. --- ### Documentation/Website/Docs/Configuration --- id: configuration title: Pyre Configuration sidebar_label: Configuration --- Pyre can be run without a configuration (see [Command Line Arguments](configuration.md#command-line-arguments)) but we do recommend that you create a configuration (see [Getting Started](getting_started.md)) and commit that to your version control system to make sure everyone working on your project is using the same settings. ## The Pyre Configuration The Pyre configuration is a `.pyre_configuration` file sitting at the root of your project. Running Pyre anywhere inside your project directory will use the settings in this configuration. You can generate an initial configuration in your project directory with ```bash $ pyre init ``` The configuration is a `JSON` file. For example, ```json { "source_directories": [ "." ], "search_path": [ "/external/library", {"site-package": "foo"} ] } ``` specifies that the code Pyre checks is in the directory of the configuration and that Pyre should look in an additional directory as well as the `foo` package installed in your environment for library code. You specify additional information to configure Pyre. The following fields are supported: - `source_directories`: List of paths to type check. Paths can be a glob, for example, `"./foo*"`. Note: Pyre assumes that all imports are relative to the given source directory. For example, if your source directory is `root/directory`, then an import statement `import module` will be looking to import `root.directory.module`. If you wish to set a different import root for your source directory, you can provide an object `{"import_root": "root", "source": "directory"}` instead of `"root/directory"`. In this case, `import module` will be looking to import `root.module`. - `search_path`: List of paths to Python modules to include in the typing environment. Relative paths are the best way to reference locations above the configuration directory. `search_path` elements take precendence over `source_directories`, and the order within the search path indicates precedence. Individual items in the list can take one of the following forms: + A plain string, representing a path to the directories from which Pyre will search for modules. The paths can be globs, for example, `"./foo*"`. + An object `{"import_root": "root", "source": "directory"}`, which can be used to control import root of the search path. See explaination for `source_directories`. + An object `{"site-package": "package_name"}`. It is equivalent to `{"import_root": "site_root", "source": "package_name"}`, where `site_root` is the first element in `site_roots` that has the site package named `package_name` installed. This can be useful when you want to manually specify which `pip` package you want the type checker to see as a dependency to your project (in which case it is recommended to set `site_package_search_strategy` to `"none"` to disable site package auto discovery). + An object `{"site-package": "package_name", "is_toplevel_module": true}`, to specify the name as a single-file module found in the site-root rather than as a package. - `binary`: Location of Pyre's native binary. - `exclude`: List of regular expressions such as `".*\/node_modules\/.*"` which specify files and directories that should be completely ignored by Pyre. The regular expression will be matched against the *full* path of files as opposed to their relative path. - `extensions`: Consider extensions in this list equivalent to `.py` for type checking. Empty string indicates extensionless files. - `ignore_all_errors`: A list of paths to omit from type-checking. This may be useful for generated files, virtualenv directories, etc. These should be paths relative to the location of the configuration file and support globs. **Note**: Files matching these paths will still be processed (i.e. type and module names in those files are still visible to Pyre). Please refer to the `exclude` configuration item if you have files that are intended to be hidden from Pyre. - `logger`: Pyre will invoke this exectuable on every run, passing it statistics in JSON format. - `site_package_search_strategy`: Configure how Pyre looks for type checking dependencies installed (e.g. by `pip`) on the local Python environment. Dependent libraries will not be type-checked, but they are consulted to determine the existence of globals/functions/classes. The value of this option can take one of the following forms: + `"none"`. This indicates that Pyre should not attempt to search for any additional dependencies. Use this option if you know exactly what packages you depend on, and want to manually specify them with the `search_path` option. + `"all"`. Pyre will pull in the entire site package roots (as specified in the `site_roots` option) as dependencies. Any libraries installed as site packages, regardless of whether they are typed or not, will be examined. Use this option if you do not know exactly which packages your code depend on, but want to make sure that no dependencies are missing. + `"pep561"`. Similar to `"all"` but instead of pull in everything, Pyre will only pull in typed packages as dependencies according to rules specified in [PEP 561](https://peps.python.org/pep-0561/). This is usually the recommended option, as the behavior is closer to what other type checkers would do by default. Note: If incremental check is used, and the search strategy is set to `"pep561"`, then a `pyre restart` is needed when new dependencies are installed -- Pyre will not automatically discover the new package by default. This is a limitation of the current implementation of Pyre and it may be lifted in the future. - `site_roots`: List of path to where packages are installed. If not specified, Pyre will consult the current Python interpreter using `site.getusersitepackages()` and `site.getsitepackages()`, which should work in most cases. But if your codebase uses a different Python interpreter, you may want to specify this option manually so Pyre knows the correct location to look for site packages. - `typeshed`: Path to the [Typeshed](https://github.com/python/typeshed) standard library, which provides typed stubs for library functions. - `workers`: Number of workers to spawn for multiprocessing. - `strict`: Setting this to `true` will make [strict mode](gradual_typing.md#strict-mode) the default in your project. - `python_version`: This is expected to be a string of the form "X.Y.Z" which specifies what version of Python the checked code is expected to be executed with. By default, the version is set to the same as the Python interpreter that runs `pyre` itself. - `system_platform`: This is expected to be a string that specifies the platform the checked code is expected to be executed on, and should be a legal value for [`sys.platform`](https://docs.python.org/3/library/sys.html#sys.platform). By default, the platform is `"linux"`. Note: Since Pyre does not run your code, the setting of Python version and platform tends to have relatively small impact on the type checker's behavior. Currently, Pyre only uses Python version and platform to syntactically resolve version conditions of [certain form](https://github.com/python/typeshed/blob/main/CONTRIBUTING.md#stub-versioning). ## Command Line Arguments You can get a full and current list of options to run Pyre by running `pyre --help`. The following is a list of commonly used commands and options. ### Commands Pyre comes with a couple commands that can be invoked with `pyre `. The first command you might come in contact with is - `initialize`, `init`: Initial setup of a configuration for a project. If [Watchman](https://facebook.github.io/watchman/docs/install/) is installed, running Pyre with no positional arguments defaults to `incremental`, otherwise defaults to check. - `check`: Run Pyre end-to-end, i.e. *not* incrementally. - `incremental`: Run Pyre incrementally. When invoked for the first time, the command will automatically start a server listening to changes to the filesystem. Subsequent invocations will be faster. When Pyre is run incrementally, you can control the Pyre's *server* working in the background with the following commands. - `start`: Start the Pyre server. - `stop`: Stop the Pyre server. - `restart`: Restart the Pyre server. - `servers`: List all currently running Pyre servers. - `kill`: In case something goes wrong and the server becomes unresponsive, `kill` will attempt to terminate any processes. - `rage`: Print server logs for debugging or for context when reporting server errors. ### Commonly Used Flags These flags can be passed in before any of the positional arguments above. For example: ```bash $ pyre --source-directory "." --noninteractive check $ pyre --source-directory "." restart ``` - `--noninteractive`: Disable interactive logging, which by default overwrites intermediate logging output and adds colors for a more streamlined user experience. Non-interactive mode ensures all terminal output remains visible. - `--output {text, json, sarif}`: Formatting for error return values. Defaults to text. - `--search-path SEARCH_PATH`: Provide additional stubs or modules external to the project being type-checked. Can also be set in `.pyre_configuration`. - `--source-directory SOURCE_DIRECTORY`: Provide a path to the source root to check. This can also be specified in `.pyre_configuration`. - `--typeshed TYPESHED`: Path to the [Typeshed](https://github.com/python/typeshed) standard library, which provides typed stubs for library functions. This can also be set in `.pyre_configuration`. - `--version`: Print the current version of Pyre. ### Helper Commands ## Working with Multi-Project Repositories If you have a single repository with multiple independent Python projects, we recommend you use a separate `.pyre_configuration` for each one. This allows each project to be type checked independently. If you use virtual environments to manage separate dependencies for each project, you can install `pyre` as a development dependency in each one; by default Pyre will detect system packages from the environment it is installed in, so this will cause each project to detect the correct dependencies (assuming you activate the virtual environment before running Pyre). --- ### Documentation/Website/Docs/Editor Integration --- id: editor-integration title: Editor Integration sidebar_label: Editor Integration --- import {OssOnly, FbInternalOnly} from 'docusaurus-plugin-internaldocs-fb/internal'; --- ### Documentation/Website/Docs/Errors --- id: errors title: Type Errors sidebar_label: Type Errors --- import Internal from './fb/errors.md'; ## Common Issues ### Covariance and Contravariance [Variance](https://en.wikipedia.org/wiki/Covariance_and_contravariance_(computer_science)) is tricky and a common source of confusion for people new to Python's type system. Pyre will error when, for instance, a `List[int]` is passed in when a `List[float]` is expected, as in the following example: ```python def to_seconds(milliseconds: List[float]) -> List[int]: return [int(x/1000.0) for x in milliseconds] my_list: List[int] = [1] my_list = to_seconds(my_list) # Pyre errors here! ``` ``` $ pyre Unbound name [10]: Name `List` is used but not defined in the current scope. ``` This code works perfectly fine at runtime, and we may think that since `int` is a subtype of `float` this should not be a problem for the type checker either. However, consider the following code: ```python def halve_first_element(list: List[float]) -> None: list[0] /= 2 def function_taking_int(int: int) -> None: return None my_list: List[int] = [1] halve_first_element(my_list) function_taking_int(my_list[0]) # Oh no, my_list[0] is 0.5! ``` ``` $ pyre Incompatible parameter type [6]: In call `list.__setitem__`, for 2nd positional argument, expected `int` but got `float`. ``` If we allowed passing in `my_list` to the `halve_first_element` function here, the above code would type check. It's perfectly valid from the perspective of the callee to modify the list's element to be a float, as it was annotated as taking a list of floats, but because this list escapes the scope of the callee, we can't allow this in the type checker. To work around this, we can signal to the type checker that the parameter can't be modified. Here's how you can tell the type checker that you won't change the container in your function: ```python from typing import * # I can't modify milliseconds here, so it's safe to pass a Iterable[int]. def to_seconds(milliseconds: Iterable[float]) -> List[int]: return [int(x/1000.0) for x in milliseconds] my_list: List[int] = [1] my_list = to_seconds(my_list) # Type checks! ``` Most commonly used generic containers have immutable variants, and I would encourage you to use them for function parameters whenever you don't need to modify a container in your function. Here are some immutable variants for commonly used containers: ```python typing.List → typing.Sequence (if you need random access via my_list[id]) typing.List → typing.Iterable (if you're just iterating over the list in a loop and want to support sets as well) typing.Dict → typing.Mapping typing.Set → typing.AbstractSet ``` Invariance, combined with type inference, comes with a few gotchas. When you write an expression, Pyre infers the most precise type possible. For instance, Pyre infers the `List[int]` type for `[1, 2]`, even though `List[float]` would be a perfectly valid type here. This can cause issues, as in the following example: ```python def zeroes(number_of_elements: int) -> List[float]: a = [0] * number_of_elements return a # Pyre errors here! ``` ``` $ pyre Incompatible return type [7]: Expected `List[float]` but got `List[int]`. ``` What happened above is that Pyre inferred a type of `List[int]` for a, and invariance kicked in. You can work around this by adding an explicit annotation when declaring a: ```python def zeroes(number_of_elements: int) -> List[float]: a: List[float] = [0.0] * number_of_elements return a # Type checks! ``` #### Contravariance `Callable`, on the other hand, is [contravariant](https://en.wikipedia.org/wiki/Covariance_and_contravariance_(computer_science)#Contravariant_method_parameter_type) in its parameter types. This means that, when checking if `Callable[[A], None]` is compatible with `Callable[[B], None]`, we check if `B` is compatible with `A`, not the other way around. This is because the former should be capable of accepting any arguments accepted by the latter. For example, a function of type `Callable[[Base], int]` may be given an argument of type `Child2`. But if we passed in a function of type `Callable[[Child1], int]`, this could fail at runtime: ```python class Base: pass class Child1(Base): size: int = 42 # No size field. class Child2(Base): pass def print_child2_size(get_size: Callable[[Base], int]) -> None: child2 = Child2() size = get_size(child2) print(size) def size_of_child1(child1: Child1) -> int: return child1.size print_child2_size(size_of_child1) # BAD! # At runtime: # AttributeError: 'Child2' object has no attribute 'size' ``` To prevent such errors, Pyre raises a type error when violating contravariance: ``` $ pyre Incompatible parameter type [6]: Expected `typing.Callable[[Base], int]` for 1st positional only parameter to call `print_child2_size` but got `typing.Callable(size_of_child1)[[Named(child1, Child1)], int]`. ``` ### Optional Attributes A common pattern in Python is to check whether an attribute is `None` before accessing its value. E.g. ```python from typing import Optional class Data: field: Optional[int] def process_field(input: int) -> None: ... def process_data(data: Data) -> None: if data.field is not None: # ... interleaving logic process_field(data.field) ``` ``` $ pyre Incompatible parameter type [6]: expected int but got Optional[int] ``` The above fails to type-check because Pyre cannot guarantee that `data.field` remains not `None` if the interleaving logic between the explicit check and the later reference contains anything that may have side effects, like function calls. An interleaving call could set `field` back to `None`, since it's a non local variable and is mutable. Therefore any calls between the None check and the access will invalidate the "not `None`" refinement. If `data.field` is defined as a class property or if the parent class has overridden `__getattr__`, then all bets are off even if there are no interleaving calls. The preferred way to make this code type-check is to either move the check closer to the access, or to mark the attribute `Final` if it is not meant to be reassigned to, and you can guarantee to the type checker that no interleaving side effects can modify this attribute. ```python from typing import Final, Optional class Data: # Needs to be assigned in the constructor and cannot be changed afterwards. field: Final[Optional[int]] = 1 ``` It is always safe to refine attributes when their types are `Final`. Alternatively, it is also safe to assign the attribute to a local variable before accessing its value: ```python def process_data(data: Data) -> None: field = data.field if field is not None: # ... interleaving logic process_field(field) ``` or using Python 3.8's assignment expressions: ```python def process_data(data: Data) -> None: if (field := data.field) is not None: # ... interleaving logic process_field(field) ``` ### Third-Party Libraries Not all third-party libraries come with Python code that Pyre can analyze (e.g. `Cython` modules), and some libraries contain source code without annotations. This will often show up in the form of undefined attribute errors: ``` Undefined attribute [16]: Module has no attribute . ``` Since it is not always possible to annotate code, PEP 484 specifies a format for [stub files](https://www.python.org/dev/peps/pep-0484/#stub-files) with a `.pyi` extension. Pyre will look for stub files in [typeshed](https://github.com/python/typeshed), or next to your source code. You can also provide additional paths to Pyre to look for stubs (see [Configuration](configuration.md)). ## Error Codes Different errors raised by Pyre have different error codes. E.g. in ```bash (venv) $ pyre ƛ Found 1 type error! test.py:1:0 Incompatible variable type [9]: a is declared to have type `int` but is used as type `str`. ``` The `9` in the brackets indicates that we raised an error with code 9. ### 0: Unused Ignore Pyre fixmes and ignores allow you to ignore specific type errors by their code until you are able to fix them. In order to avoid outdated fixme comments in your project, Pyre will also error when a fixme is no longer needed. Removing the fixme comment will resolve the error. ```python # pyre-fixme[7] # unused ignore def foo() -> int: return 1 ``` ``` $ pyre Unused ignore [0]: The `pyre-ignore[7]` or `pyre-fixme[7]` comment is not suppressing type errors, please remove it. ``` ### 2: Missing Parameter Annotation If strict mode is turned on, Pyre will error when a function parameter is either annotated with a type that contains `typing.Any` or not annotated with any type at all (in which case Pyre will treat it as `typing.Any` by default). It will also error when a method parameter is not annotated, unless that parameter is the first parameter of a bound or static method (i.e. `self`, whose type pyre can infer). We enforce typed argument because `typing.Any` can hide type errors that will happen at runtime: ```python from typing import Any def say_hello(name) -> None: print("Hello " + name) # This line will raise at runtime, but no type error since `say_hello`s `name` has type `Any`. say_hello(42) ``` You can silence this by adding a non-`Any` annotation to all parameters of functions and methods (other than `self` and `cls` for bound and class methods, which you may omit). ### 3: Missing Return Annotation If strict mode is turned on, Pyre will error when a function is either annotated with a return type that contains `typing.Any`, or is not annotated with any return type at all (in which case Pyre will treat it as returning `typing.Any` by default). This is bad because a return type of `typing.Any` may potentially hiding legitimate type errors that may happen at runtime: ```python from typing import Any def f(): return 42 print("a" + f()) ``` ``` $ pyre Missing return annotation [3]: Returning `int` but no return type is specified. ``` The best way to silence this error is to add non-`Any` return annotation to every function. ### 4: Missing Attribute Annotation In strict mode, Pyre will error when an attribute does not have an annotation. ```python def foo() -> str: return "Hello, World!" class A: b = foo() # Missing attribute annotation ``` ``` $ pyre Missing attribute annotation [4]: Attribute `b` of class `A` has no type specified. ``` Adding a type annotation will resolve this error: ```python def foo() -> str: return "Hello, World!" class A: b: str = foo() ``` This error can also occur when pyre is inferring attribute types from constructors. For example, here we know that `b` is an int based on the parameter annotation in `__init__`: ```python class A: def __init__(self, b: int) -> None: self.b = b ``` But here we need a annotations because we can't just propagate an argument annotation: ```python class A: def __init__(self, arg: int) -> None: self.a = arg + 5 self.b = arg + 5 ``` ``` $ pyre Missing attribute annotation [4]: Attribute `a` of class `A` has type `int` but no type is specified. Missing attribute annotation [4]: Attribute `b` of class `A` has type `int` but no type is specified. ``` We can fix this by making the annotation explicit, either in the class body or in `__init__`: ```python class A: a: int def __init__(self, arg: int) -> None: self.a = arg + 5 self.b: int = arg + 5 ``` ### 5: Missing Global Annotation If strict mode is turned on, Pyre will error when a globally accessible variable is not annotated. If pyre was able to infer a type for the variable, it will emit this type in the error message. The fix is usually to add an annotation to the variable. Note: This error has also arisen when there is some ambiguity of whether a declaration is a global expression or a type alias, in these cases pyre assumes it is an expression. Adding a `: TypeAlias` annotation lets pyre know that it is a type alias and solves the problem. ```python from typing_extensions import TypeAlias # This declaration would result in an error MyTypeAlias = Dict[str, "AnotherTypeAlias"] ``` ``` $ pyre Missing global annotation [5]: Globally accessible variable `MyTypeAlias` has no type specified. ``` ```python from typing_extensions import TypeAlias # This declaration ensures that pyre knows MyTypeAlias is a type alias MyTypeAlias: TypeAlias = Dict[str, "AnotherTypeAlias"] ``` ### 6: Incompatible Parameter Type Pyre will error if an argument passed into a function call does not match the expected parameter type of that function. ```python def takes_int(x: int) -> None: pass def f(x: Optional[int]) -> None: takes_int(x) # Incompatible parameter type error ``` ``` $ pyre Incompatible parameter type [6]: In call `takes_int`, for 1st positional argument, expected `int` but got `Optional[int]`. ``` If you are seeing errors with invariant containers where some `Container[T]` is expected but you are passing `Container[S]` where `S < T`, please see [Covariance and Contravariance](errors.md#covariance-and-contravariance). ### 7: Incompatible Return Type Pyre will error when the value returned from a function does not match the annotation. ```python def foo() -> int: return "" # incompatible return type ``` ``` $ pyre Incompatible return type [7]: Expected `int` but got `str`. ``` Updating the return annotation, or the value returned from the function will resolve this error. ```python def foo() -> str: return "" # compatible: No error ``` ### 8: Incompatible Attribute Type Pyre will error if a value is assigned to an attribute that does not match the annotated type of that attribute. ```python class Foo: x: int = 0 def f(foo: Foo) -> None: foo.x = "abc" # Incompatible attribute type error ``` ``` $ pyre Incompatible attribute type [8]: Attribute `x` declared in class `Foo` has type `int` but is used as type `str`. ``` If you are seeing errors with invariant containers where some `Container[T]` is expected but you are passing `Container[S]` where `S < T`, please see [Covariance and Contravariance](errors.md#covariance-and-contravariance). ### 9: Incompatible Variable Type Pyre will error when assigning incompatible types to local variables and parameters that were explicitly annotated. That is, the following will error: ```python def f(x: int) -> None: x = "" # Incompatible variable type error y: int = 1 y = "" # Incompatible variable type error ``` ``` $ pyre Incompatible variable type [9]: x is declared to have type `int` but is used as type `str`. Incompatible variable type [9]: y is declared to have type `int` but is used as type `str`. ``` The rationale here is that it's surprising for an explicitly annotated variable to have an incompatible type later on in the same function. If you are constructing an object that is generic over an invariant type, you may run into an error: ```python from typing import TypeVar _T = TypeVar('_T') class Foo(Generic[_T]): def __init__(self, x: _T) -> None: ... def f() -> None: foo: Foo[Optional[int]] = Foo(x=1) # Incompatible variable type error ``` ``` $ pyre Incompatible variable type [9]: foo is declared to have type `Foo[Optional[int]]` but is used as type `Foo[int]`. ``` This is due to the fact that `Foo[X]` is not less than `Foo[Y]` even if `X < Y` when the type variable is invariant. You can declare your intention to initialize the object with a wider type than is given to fix this error: ```python from typing import TypeVar _T = TypeVar('_T') class Foo(Generic[_T]): def __init__(self, x: _T) -> None: ... def f() -> None: foo: Foo[Optional[int]] = Foo[Optional[int]](x=1) ``` ### 10: Unbound Name Pyre produces an Unbound Name error when your code access a variable (local or global) that pyre believes is not defined. In most cases code that does this is invalid and will always fail. For example this code will always fail at runtime: ```python def f() -> int: return x # use of an unbound name x ``` ``` $ pyre Unbound name [10]: Name `x` is used but not defined in the current scope. ``` There are some cases where python code that works fine at runtime could produce this error, for example if a function implicitly sets a module-level global variable that is not declared in the toplevel. Pyre will not accept this because module-level globals require type annotations, and if they have no declaration there is nowhere to put the annotation: ```python def set_x() -> None: global x x = 42 def use_x() -> None: print(x) # this code will run fine, but pyre cannot analyze the type or use of the # implicitly-defined global x and will complain about an unbound name. set_x() use_x() ``` ``` $ pyre Unbound name [10]: Name `x` is used but not defined in the current scope. ``` You can fix this by explicitly adding a declaration of the top-level variable `x`, for example: ```python x : Optional[int] = None def set_x() -> None: global x x = 42 def use_x() -> None: print(x) # this code will run fine set_x() use_x() ``` ### 11, 31: Undefined or Invalid Type Pyre recognizes class names as valid annotations. Most basic types are imported from the `typing` module or are already available from builtins like `str`, `int`, `bool`, etc. You can also define your own type alias on the global scope, which can be used as annotations: ```python from typing_extensions import TypeAlias INT_OR_STR: TypeAlias = Union[int, str] ``` If you use a name as an annotation that is not a valid type or valid alias, you will see this error: ```python from typing import Callable, List from typing_extensions import Final, Literal GLOBAL_VALUE = "string" def f0() -> GLOBAL_VALUE: ... # Error! `GLOBAL_VALUE` is a value, not a type. def f1() -> type(GLOBAL_VALUE): ... # Error! Static type annotations cannot be dynamically computed. def f2() -> [int]: ... # Error! `[int]` is not a valid type. If you mean a list of int, use `typing.List[int]`. def f3() -> (int, str): ... # Error! `(int, str)` is not a valid type. If you mean a pair of int and str, use `typing.Tuple[int, str]`. def f4() -> Callable[[int]]: ... # Error! `Callable[[int]]` is not a valid type because the return type of the callable is missing. Good example: `Callable[[int], int]`. def f5() -> Callable[int, int]: ... # Error! `Callable[int, int]` is not a valid type. The parameter types of the callable must be enclosed in square brackets. Good example: `Callable[[int], int]`. def f6() -> List[Final[int]]: ... # Error! `Final` may only be used as the outermost type in annotations. See PEP 591. def f7() -> Literal[GLOBAL_VALUE]: ... # Error! Only literals are allowed as parameters for `Literal`. See PEP586. Good example: `Literal[42]` or `Literal["string"]`. ``` ``` $ pyre Undefined or invalid type [11]: Annotation `GLOBAL_VALUE` is not defined as a type. Invalid type [31]: Expression `type(GLOBAL_VALUE)` is not a valid type. Invalid type [31]: Expression `[int]` is not a valid type. Invalid type [31]: Expression `(int, str)` is not a valid type. Invalid type [31]: Expression `typing.Callable[[int]]` is not a valid type. Invalid type [31]: Expression `typing.Callable[(int, int)]` is not a valid type. Invalid type [31]: Expression `GLOBAL_VALUE` is not a literal value. ``` You can fix this error by verifying that your annotation is 1. statically determined. 2. properly imported from `typing` if applicable. 3. properly defined in the module you are importing from. If the module you are importing from has a [stub file](errors.md#third-party-libraries), you should check the definition there. 4. properly adhere to the additional rules of special types (e.g. `Callable`, `Final`, and `Literal`). #### Type Aliases For type aliases, check that your type alias is defined 1. with a valid type on the RHS. If you provide an annotation for the TypeAlias assignment, it must be `typing_extensions.TypeAlias`. 2. on the global scope, not nested inside a function or class. #### ParamSpec For `ParamSpec`, check that you have used both `*args: P.args` and `**kwargs: P.kwargs` in your function's parameters: ```python from typing import Callable from pyre_extensions import ParameterSpecification P = ParameterSpecification("P") # Error because `**kwargs: P.kwargs` is missing. def bad1(f: Callable[P, int], *args: P.args) -> int: return f(*args) # Error because `*args: P.args` is missing. def bad2(f: Callable[P, int], **kwargs: P.kwargs) -> int: return f(**kwargs) ``` ``` $ pyre Undefined or invalid type [11]: Annotation `P.args` is not defined as a type. Call error [29]: `typing.Callable[P, int]` cannot be safely called because the types and kinds of its parameters depend on a type variable. Undefined or invalid type [11]: Annotation `P.kwargs` is not defined as a type. ``` No type error if you have used both `*args: P.args` and `**kwargs: P.kwargs` ```python from typing import Callable from pyre_extensions import ParameterSpecification P = ParameterSpecification("P") # OK def good(f: Callable[P, int], *args: P.args, **kwargs: P.kwargs) -> int: return f(*args, **kwargs) ``` ### 12: Incompatible Awaitable Type In strict mode, pyre will verify that all calls to `await` are on awaitable values, to ensure that you cannot get a runtime error awaiting an object that is not a coroutine. A common situation where working code will produce this error is when pyre cannot statically verify that an awaitable is non-Null, for example: ```python import asyncio async def f(flag: bool) -> None: if flag: task = asyncio.create_task(asyncio.sleep(1)) else: task = None await task # would throw a ValueError if flag were false asyncio.run(f(True)) ``` In this example, `task` will always be a valid awaitable unless some other module overwrites the global `flag`, but pyre cannot prove that this does not happen. The error we get has the message ``` $ pyre Expected an awaitable but got `typing.Optional[asyncio.tasks.Task[None]]` ``` You can fix this error by ensuring that the awaited object has an awaitable type. In the case of optional values, you can use refinement to rule out `None`. The example above can be fixed by tweaking the definition of `main`: ```python async def f(flag: bool) -> None: if flag: task = asyncio.create_task(asyncio.sleep(1)) else: task = None if task is not None: await task ``` ### 13: Uninitialized Attribute In strict mode, pyre will throw an error for class attributes which are declared without default values if they are not initialized in a constructor, for example: ```python class A: x : int def __init__(self) -> None: pass ``` ``` $ pyre Uninitialized attribute [13]: Attribute `x` is declared in class `A` to have type `int` but is never initialized. ``` For a case like this, you can fix the error either by setting a default value like `x : int = 0` at the class level, or by setting `x` in the constructor e.g. `self.x = 0`. ```python class A: x: int = 0 def __init__(self) -> None: pass ``` ```python class A: x: int def __init__(self, x: int = 0) -> None: self.x = x ``` #### Dataclass-like classes One case where this can occur is when using a library providing a "dataclass-like" decorator that, for example, autogenerates a constructor setting attributes. ```python from dataclasses import dataclass @dataclass class A: x: int ``` Pyre currently knows that that uninitialized attributes of classes wrapped in `dataclass` and `attrs` decorators will generate constructors that set the attributes. But it does not understand many custom libraries that do similar things, for example test frameworks, or new decorators that wrap the dataclass decorator and add more logic. There is not currently a way to fix this other than via `pyre-ignore` or `pyre-fixme` directives. The python typing community is aware of this problem but has not yet settled on a solution, you can see discussion [here](https://mail.python.org/archives/list/typing-sig@python.org/thread/TXL5LEHYX5ZJAZPZ7YHZU7MVFXMVUVWL/). ### 14,15: Behavioral Subtyping Method overrides should follow [Liskov's substitution principle](https://en.wikipedia.org/wiki/Liskov_substitution_principle). In short, parameter types can't be more restrictive and return types can't be more permissive in overridden methods. To see why, consider the following example: ```python def width(image: Image) -> float: return image.width() ``` Say we now have different implementations of our `Image` class, one of which violates the substitution principle: ```python class Image: def width() -> float: pass class JpegImage(Image): def width() -> int: return 10 # this is fine class ComplexImage(Image): def width() -> complex: return 1j def foo() -> None: image: Image = ComplexImage() print(int(image.width())) ``` The above code fails at runtime with `TypeError: can't convert complex to int`. The case for parameters follows analogously. #### Common Reasons + `Could not find parameter y in overriding signature.`: Check if the overriding function can accept all arguments that the overridden function can. ```python class Base: def foo(self, x: int, y: str) -> None: pass class Child(Base): def foo(self, x: int) -> None: pass ``` + `Type Foo is not a subtype of the overridden attribute type Bar`: ```python class Base: a: int = 0 class Child(Base): a: str = "" def foo() -> None: base: Base = Child() base.a + 1 ``` This would fail at runtime with `TypeError: can only concatenate str (not "int") to str`. + `Returned type Foo is not a subtype of the overridden return Bar.`: Check for reasons like [invariance](errors.md#covariance-and-contravariance). ### 16: Missing Attributes Your code is most likely trying to access an attribute that Pyre does not know about. Pyre has various ways of inferring what is an attribute of an object: #### Explicitly Declare the Attribute ```python class Derp: my_attribute: int = 1 @property def my_property(self) -> int: ... ``` #### Implicitly Declare the Attribute ```python class Derp: def __init__(self, foo: str) -> None: self.my_attribute: int = 1 # The `foo` attribute is inferred to have type `str` because the # parameter `foo` has type `str`. self.foo = foo ``` Pyre does one level of inlining to infer implicit parameters We suggest you do not heavily rely on this feature as it is not sound and makes our code brittle. Support for this is temporary. #### Common Reasons + `Optional type has no attribute foo.`: See [Optional attributes](errors.md#optional-attributes). + `Foo has no attribute bar.`: Check if you have explicitly provided the type for `bar` either in the constructor or as a class attribute. + `Module foo has no attribute bar`: Check if the library has [stubs](errors.md#third-party-libraries). If so, you may need to add the function, class, or global variable to the stub. + A library class has an attribute but it is not recognized by Pyre: Check if the library has [stubs](errors.md#third-party-libraries). If so, you may need to add the attribute to the class in the stub. + Your class has dynamic attributes: Consider using `__getattr__` in a [stub](gradual_typing.md#when-source-code-is-not-available) so that Pyre doesn't complain about those attributes. ### 17: Incompatible Constructor Annotation [PEP 484](https://www.python.org/dev/peps/pep-0484/#the-meaning-of-annotations) specifies that `__init__` method of any class must be annotated to return `None`. Pyre will emit an error if the user's annotation does not conform to the specification. ```python # pyre-strict class A: def __init__(self) -> "A": # Error 17: Invalid return annotation of `__init__`. ... class B: def __init__(self) -> None: # OK ... ``` ``` $ pyre Incompatible constructor annotation [17]: `__init__` is annotated as returning `A`, but it should return `None`. ``` ### 19: Too Many Arguments Pyre verifies that you pass a legal number of arguments to functions. The most obvious way to encounter this error is to just pass too many arguments to a function: ```python def f(x: int) -> int: return x f(5, 6) # this would throw a TypeError at runtime, and pyre complains ``` ``` $ pyre Too many arguments [19]: Call `f` expects 1 positional argument, 2 were provided. ``` To fix this, make sure you pass the correct number of parameters. In some cases you may encounter this error if you intended to use a variadic argument (`*args`) or to set a default value. Pyre will also throw this error if you pass too many positional arguments to a function that uses python's ability restrict arguments to be keyword-only specified by [PEP 3102](https://www.python.org/dev/peps/pep-3102/): ```python def f(*, x: int) -> int: return x f(5) # As before, this throws a TypeError because x is positional-only f(x=5) # this line will typecheck and run without error ``` ``` $ pyre Too many arguments [19]: Call `f` expects 0 positional arguments, 1 was provided. ``` ### 20: Missing Argument Pyre verifies that function calls provide all the expected arguments, so it will complain about code like this: ```python def f(x: int) -> ing: return x f() ``` ``` $ pyre Missing argument [20]: Call `f` expects argument `x`. ``` To fix this, make sure all required arguments are provided. ### 21: Undefined Name, Undefined Import This is usually caused by failing to import the proper module. ```python from my_module import my_function my_function() ``` ``` $ pyre Undefined import [21]: Could not find a module corresponding to import `my_module`. ``` Pyre will raise error 21 instead ("Undefined import") when the import statement is present, but the module to be imported could not be found in the search path. If the module provides stub files, please provide their location via the `--search-path` commandline parameter. #### Namespace Package Modules One case where you may run into undefined imports on code that works at runtime is when importing namespace modules. The CPython runtime allows you to import a directory that is on your `PYTHONPATH`, even if it contains no `__init__.py`; this behavior is defined in [PEP 420](https://peps.python.org/pep-0420/) and the module is called a namespace package. In order to make Pyre both fast and consistent on incremental updates, in Pyre we only allow importing namespace packages that have at least one python file as a direct child. So, for example, if I have a directory tree with just `a/b/c.py` then Pyre will allow `import a.b.c` and `import a.b` but not `import a`. A namespace package module can never contain useful types or code so it is rare to directly import it, but in special cases it might be useful (for example to access the `__name__` attribute). In these cases, you'll need to suppress Pyre errors. ### 22: Redundant Cast Pyre will warn when you attempt to use `typing.cast` to cast a variable to a type that the type checker already knows that variable has. This is because `typing.cast` is purely a tool for communicating with the static type checker, and will not provide any runtime guarantees. Therefore a redundant cast provides no value and is likely a mistake. ```python from typing import cast def foo(x: int) -> None: y = cast(int, x) ``` ``` $ pyre Redundant cast [22]: The value being cast is already of type `int`. ``` If you are trying to document the type of the variable, you can provide an explicit annotation where it is declared. If you are trying to add a sanity check at runtime that the type of a variable is what you already believe it must be, use `isinstance`. ### 23: Unable to Unpack Pyre will warn you when trying to assign a value to a tuple with the wrong number of items. ```python def foo() -> None: a, b = (1, 2, 3) x, y = 42 ``` ``` $ pyre Unable to unpack [23]: Unable to unpack 3 values, 2 were expected. Unable to unpack [23]: Unable to unpack `int` into 2 values. ``` Common reasons: + Trying to assign an `Optional` value to a tuple: ```python def bar() -> None: x = None if 2 + 2 == 4: x = ("a", "b") a, b = x ``` ``` $ pyre Unable to unpack [23]: Unable to unpack `typing.Optional[typing.Tuple[str, str]]` into 2 values. ``` + Unpacking an incorrect number of elements when looping over a list: ```python for a, b in [1, 2, 3]: print(a, b) $ pyre Unable to unpack [23]: Unable to unpack `int` into 2 values. ``` ### 24: Invalid Type Parameters Pyre will error if a generic type annotation is given with unexpected type parameters. #### "Generic type expects X type parameters ..." Either too few or too many type parameters were provided for the container type. For example, ```python x: List[int, str] = [] # Invalid type parameters error ``` ``` $ pyre Invalid type parameters [24]: Generic type `list` expects 1 type parameter, received 2, use `typing.List[]` to avoid runtime subscripting errors. ``` In this case, `typing.List` is a generic type taking exactly one type parameter. If we pass a single parameter, this resolves the issue. ```python x: List[Union[int, str]] = [] # No error ``` If you do not know or do not want to specify the type parameters, use `typing.Any` but still ensure the arity is correct. ```python x: List = [] # Invalid type parameters error x: List[Any] = [] # No error ``` Note: You may see a suggestion to use `typing.List` instead of builtins `list` as the type annotation when providing type parameters. This is to avoid runtime errors, because the builtin `list` does not support subscripting and `list[int]` is therefore not runtime-friendly. #### "Non-generic type cannot take type parameters ..." Type parameters are only meaningful if the container type is generic. Passing in the type parameter binds the provided parameter type to the generic in the container class. For example, ```python class Container: def add(self,element: int) -> None: ... def get_element(self) -> int: ... x: Container[int] = Container() # Invalid type parameter error ``` ``` $ pyre Invalid type parameters [24]: Non-generic type `Container` cannot take parameters. ``` ```python from typing import TypeVar, Generic T = TypeVar('T') class Container(Generic[T]): def add(self,element: T) -> None: ... def get_element(self) -> T: ... x: Container[int] = Container() x.get_element() # returns int y: Container[str] = Container() y.get_element() # returns str ``` #### "Type parameter violates constraints ..." If a container class is generic over a type variable with given type bounds, any type parameter used must comply with those type bounds. For example, ```python from typing import TypeVar, Union, Generic T = TypeVar('T', bound=Union[int, bool]) class Container(Generic[T]): def add(self, element: T) -> None: ... def get_element(self) -> T: ... x: Container[int] = Container() # No error y: Container[str] = Container() # Invalid type parameter error ``` ``` $ pyre Invalid type parameters [24]: Type parameter `str` violates constraints on `Variable[T (bound to typing.Union[bool, int])]` in generic type `Container`. ``` ### 26: Typed Dictionary Access With Non-Literal In python, [typed dictionaries can only be accessed using literal strings](https://www.python.org/dev/peps/pep-0589/#id21) that can be statically verified as valid. As a result, code like this will not typecheck even though it works at runtime, because we cannot statically verify that `key` in `print_value` is a valid `Shape` key: ```python from typing import TypedDict class Shape(TypedDict): sides: int color: str shape: Shape = {"sides": 4, "color": "blue"} print(shape["sides"]) # this is fine because "sides" is a literal for key in ["sides", "color"]: print(key, shape[key]) # pyre will complain here because it can't prove `key` is valid ``` ``` $ pyre TypedDict accessed with a non-literal [26]: TypedDict key must be a string literal. Expected one of ('color', 'sides'). ``` The example above shows a situation where you might hit this error: when you want to iterate over the fields of a typed dict. A suggested fix is to use type-safe operations like `dictionary.items` instead. For example the following code produces the same results but type checks: ```python class Shape(TypedDict): sides: int color: str shape: Shape = {"sides": 4, "color": "blue"} print(shape["sides"]) # this is fine because "sides" is a literal for key, value in shape.items(): print(key, value) # no error ``` In other cases where you need to access a `TypedDict` using a variable as a key, you can use `dictionary.get(key)`. ```python from typing import TypedDict class Shape(TypedDict): sides: int color: str shape: Shape = {"sides": 4, "color": "blue"} print(shape["sides"]) # this is fine because "sides" is a literal for key in ["sides", "color"]: print(key, shape.get(key)) # no error ``` ### 27: Typed Dictionary Key Not Found If you try to access a typed dictionary with a string literal that pyre knows is not a valid key, pyre will emit an error: ```python from typing import TypedDict class Shape(TypedDict): sides: int color: str def f(shape: Shape) -> None: print(shape["location"]) # error here ``` ``` $ pyre TypedDict accessed with a missing key [27]: TypedDict `Shape` has no key `location`. ``` A possible fix: pyre considers instances of any `TypedDict` with additional fields to be a subtype of `Shape`, so in many cases you could handle the need for a `location` field in some `Shape` dicts by creating a new type as follows: ```python from typing import TypedDict class Shape(TypedDict): sides: int color: str class ShapeWithLocation(TypedDict): sides: int color: str location: str def f(shape: ShapeWithLocation) -> None: print(shape["location"]) # okay g(shape) # also okay: ShapeWithLocation is a subtype of Shape def g(shape: Shape) -> None: print(shape) ``` ### 28: Unexpected Keyword Pyre will error if attempting to pass an argument by name and there are no parameters with a matching name. For example, ```python def foo(integer: int, string: str) -> None: ... foo(1, "one") # no error foo(string="one", integer=1) # no error foo(integer=1, undefined="one") # type error ``` ``` $ pyre Unexpected keyword [28]: Unexpected keyword argument `undefined` to call `foo`. ``` ### 29: Call Error Pyre will emit an error on seeing a call of one of the following types: 1. The called object is not a function. This means that its inferred type is not Callable and it is not an instance of a class which implements a `__call__` method. This could happen due to user error (the object is indeed not a function) or due to an incorrect or incomplete type stub for the object's class causing pyre to infer the wrong type. 2. The call cannot be safely typed since the types and kinds of its parameters depend on a type variable. This is seen when the callable is typed using a ParameterSpecification type variable and the `*args` and `**kwargs` are not passed into the call correctly, i.e. together and in order. (For more details see [PEP 612](https://www.python.org/dev/peps/pep-0612/#the-components-of-a-paramspec)) ```python from pyre_extensions import ParameterSpecification from typing import Callable P = ParameterSpecification("P") def decorator(f: Callable[P, int]) -> Callable[P, None]: def foo(*args: P.args, **kwargs: P.kwargs) -> None: f(*args, **kwargs) # Accepted, should resolve to int f(*args) # Rejected : error here f(*kwargs, **args) # Rejected : error here f(1, *args, **kwargs) # Accepted return foo ``` ``` $ pyre Call error [29]: `typing.Callable[sandbox.P, int]` cannot be safely called because the types and kinds of its parameters depend on a type variable. Call error [29]: `typing.Callable[sandbox.P, int]` cannot be safely called because the types and kinds of its parameters depend on a type variable. ``` ### 30, 36: Terminating Analysis, Mutually Recursive Type Variables #### Overly-complex Functions In very rare cases where a function has a lot of `if` branches or `for`-loops, Pyre may raise an error saying that is unable to analyze the function fully. Analyzing extremely complex functions in depth can be costly, so Pyre only does so up to a limit. This means that it won't infer precise types for some variables and won't catch errors related to their usage. For example: ```python def my_function() -> None: u = 42 if foo(): x1 = bar() if x1: x2 = baz() if x2: # else: # if foo2(): # if foo3(): # # $ pyre Analysis failure [30]: Pyre gave up inferring types for some variables because function `foo` was too complex. Please simplify the function by factoring out some if-statements or for-loops. ``` To remedy this, factor out some of the branching code into separate functions out that each function has a limited amount of branching logic: ```python def do_stuff() -> None: if foo(): # if foo(): # # def bar() -> None: u = 42 if foo(): x1 = bar() if x1: x2 = baz() if x2: # else: # do_stuff() ``` #### Other Analysis Failures These errors usually indicates a bug in Pyre. Please open an issue on [Github](https://github.com/facebook/pyre/issues). ### 31: Invalid Type This indicates that you are using some expression that pyre does not understand as a type. Some situations where you might run into this: + Using a list of types rather than `List[type]`: ```python x: [str] = ["a string"] ``` You can fix this by using the `List` type: ```python x: List[str] = ["a string"] ``` + Using a constructor call rather than a bare class name: ```python class A: ... a: A() = A() ``` You can fix this by using a bare type name: ```python a: A = A() ``` ### 32: Invalid Argument This error usually means you are using a variable in a way that is incompatible with its structure, either as an argument to a function call or as part of a data structure. This could be from using an invalid variadic parameter (informally a "splat"): ```python x: int = 5 print(*x) # invalid use of x, which is not iterable ``` ``` $ pyre Invalid argument [32]: Unpacked argument `x` must have an unpackable type but has type `typing_extensions.Literal[5]`. ``` or using an invalid keyword parameter (informally a "double-splat"): ```python from typing import Dict x: int = 5 d: Dict[int, int] = {**x} # invalid use of x, which is not a mapping dict(**d) # invalid use of d; function kwargs must be a mapping with string keys ``` ``` $ pyre Invalid argument [32]: Keyword argument `x` has type `typing_extensions.Literal[5]` but must be a mapping. Invalid argument [32]: Keyword argument `d` has type `Dict[int, int]` but must be a mapping with string keys. ``` It's also possible to hit this error code on constraint mismatches when using tuple variadic variables as specified in [PEP 646](https://www.python.org/dev/peps/pep-0646/), which are an advanced feature of pyre. ### 33: Prohibited Any Pyre will warn on any usage of `typing.Any` when run in [strict mode](gradual_typing.md#strict-mode). `Any` is an escape hatch that hides type errors and introduces potential type inconsistencies which Pyre strict is designed to make explicit. ```python import typing from typing import Any, Dict def foo() -> None: x: typing.Any = 1 ``` ``` $ pyre Prohibited any [33]: Expression `x` has type `int`; given explicit type cannot be `Any`. ``` To resolve this error, replace `Any` with any other annotation. Using builtins `object` is acceptable if you are looking for a supertype of all classes. ### 34: Invalid Type Variable Type variables can only be used as types when they have already been placed "in scope". A type variable can be placed into scope via: * Generic class declarations * for example, `class C(Generic[T]):` puts `T` into scope for the body of the class * The **parameter** types of a generic function * for example, `def foo(x: T)` puts `T` into scope for the body and return type annotation of the function For example: ```python from typing import List class Base: foo: List[T] = [] $ pyre Invalid type variable [34]: The current class isn't generic with respect to the type variable `Variable[T]`. ``` ```python def foo(x: int) -> List[T]: return [x, x] $ pyre Invalid type variable [34]: The type variable `Variable[T]` isn't present in the function's parameters. ``` Suggested fix: ```python from typing import Generic, List class Base(Generic[T]): foo: List[T] = [] base: Base[int] def foo(x: T) -> List[T]: return [x, x] ``` #### Decorator Factories One common error is when defining a generic decorator factory. The Python type system doesn't currently place `T` into scope within a `Callable` type. So, it considers `T` to be a type variable from the outer scope. This can lead to errors for apparently valid code: ```python from typing import Callable, TypeVar T = TypeVar("T") R = TypeVar("R") def my_decorator_factory(message: str) -> Callable[[Callable[[T], R]], Callable[[T], R]]: def _decorator(f: Callable[[T], R]) -> Callable[[T], R]: def _inner(x: T) -> R: print(message) return f(x) return _inner return _decorator $ pyre Invalid type variable [34]: The type variable `Variable[R]` isn't present in the function's parameters. Invalid type variable [34]: The type variable `Variable[T]` isn't present in the function's parameters. ``` Suggested fix: Use a callback protocol to define the return type. ```python from typing import Callable, Protocol, TypeVar T = TypeVar("T") R = TypeVar("R") class MyCallableProtocol(Protocol): def __call__(self, f: Callable[[T], R]) -> Callable[[T], R]: ... def my_decorator_factory(message: str) -> MyCallableProtocol: def _decorator(f: Callable[[T], R]) -> Callable[[T], R]: def _inner(x: T) -> R: print(message) return f(x) return _inner return _decorator ``` If you are using a `ParamSpec` in your decorator, use the following: ```python from typing import Any, Callable, Coroutine, Protocol, TypeVar from pyre_extensions import ParameterSpecification import asyncio R = TypeVar("R") P = ParameterSpecification("P") class MyCallableProtocol(Protocol): def __call__(self, f: Callable[P, Coroutine[object, object, R]]) -> Callable[P, Coroutine[object, object, R]]: ... def my_decorator_factory(message: str) -> MyCallableProtocol: def _decorator(f: Callable[P, Coroutine[object, object, R]]) -> Callable[P, Coroutine[object, object, R]]: async def _inner(*args: P.args, **kwargs: P.kwargs) -> R: print(message) return await f(*args, **kwargs) return _inner return _decorator @my_decorator_factory("hello!") async def foo() -> int: return 1 asyncio.run(foo()) ``` Note: Support for such callables is currently **experimental** and varies from one typechecker to another. This behavior may change in the future. ### 35: Illegal Annotation Target Pyre will error when a type annotation is applied to something that can't be annotated. This could happen when: 1. A variable is re-annotated after first declaration or an explicity annotated function parameter is re-annotated within the function body. This is not allowed as re-annotating variables reduces readability and causes the annotation of a variable to depend on the position in control flow. ```python def transformation(p: int) -> str: return str(p + 1) def foo(x: int) -> None: y: int = x + 2 z = x + 3 # Each of the following will produce an error x: str = transformation(x) y: str = transformation(y) z: int = 4 ``` ``` $ pyre Illegal annotation target [35]: Target `x` cannot be annotated after it is first declared. Illegal annotation target [35]: Target `y` cannot be annotated after it is first declared. Illegal annotation target [35]: Target `z` cannot be annotated after it is first declared. ``` An easy fix for the first two errors is to use a new variable rather than re-annotating the old variable so it can hold a new type. For the third error, `z` should have been annotated at first declaration. 2. Trying to annotate non-self attributes, i.e annotating the attributes of a different class than the one whose scope you are in: ```python class Foo: attribute: int = 1 class Bar: def __init__(self) -> None: Foo.attribute: str = "hello" def some_method() -> None: Foo.attribute: int = 5 ``` ``` $ pyre Illegal annotation target [35]: Target `sandbox.Foo.attribute` cannot be annotated. Illegal annotation target [35]: Target `sandbox.Foo.attribute` cannot be annotated. ``` This is not allowed as Pyre needs to be able to statically determine the type of globally accessible values, including class attributes. Even if Pyre followed control flow across functions to determine class attribute annotations, such re-annotations imply very dynamic behavior that makes the code difficult to work with. The fix for this situation, similar to the case above, is to annotate the class attribute at its definition in the class that owns it and remove any annotations elsewhere. If this attribute is from a third party library, then you can add a [stub](errors.md#third-party-libraries) for the class and annotate the attribute there. ### 39: Invalid Inheritance When defining a new class, Pyre will error if the base class given is not a valid parent class. This may be caused by various conditions: 1. The parent class is marked as final which means it explicitly is annotated as not supporting child classes. ```python @final class Base: ... class Derived(Base): # Invalid inheritance error ... ``` ``` $ pyre Invalid inheritance [39]: Cannot inherit from final class `Base`. ``` 2. The expression given in the base class field is not a class at all. ```python MY_GLOBAL: str = "string" class Foo(MY_GLOBAL): # Invalid inheritance error ... ``` Pyre does not support dynamic expressions as base classes, even if they may evaluate to a valid class at runtime. This is because the type checker relies on building up a valid class hierarchy before it can resolve types in the Python it is analyzing. On the other hand, type aliases are equivalent to types and are acceptable as base classes. 3. You are defining a typed dictionary that does not inherit from another typed dictionary. ```python from typing import TypedDict class NonTypedDict: ... class Movie(TypedDict): name: str year: int class BookBasedMovie(Movie): # No error based_on: str class BookBasedMovie(NonTypedDict): # Invalid inheritance error based_on: str ``` If inheriting from another typed dictionary, fields need to have a consistent type between child and parent, in order for subclassing to be sound. Similarly, a required field in the child must also be required for the parent. ### 40: Invalid Override Pyre will error when methods in a child class override those in a parent class inconsistently. Static methods cannot be overwritten by non-static methods, and final methods cannot be overwritten. ```python class A: @staticmethod def foo() -> int: pass class B(A): @classmethod # Non-static method `B.foo` cannot override a static method defined in `A`. def foo(cls) -> int: pass ``` ```python from typing import final class Foo: @final def bar(self) -> None: pass class Bar(Foo): def bar(self) -> None: # Invalid override, because Foo.bar is final pass ``` ### 41: Invalid Assignment Pyre will error on assignments to final attributes, read-only properties, and class variables from a class instance. For example, ```python from typing import Final, Optional class Foo: field: Final[Optional[int]] = 1 def foo() -> None: self.field = 2 # Invalid assignment class Bar: _x = 1 @property def x(self) -> int: return self._x def bar(b: Bar) -> None: b.x = 1 # Invalid assignment ``` To fix this error, change the definition of this attribute to something that is mutable, if it is not intended to be read-only. ### 42: Missing Overload Implementation Pyre will throw this error if a source module specifies one or more overloads via [`typing.overload`](https://fburl.com/d7b8cd2h) but fails to provide an implementation, for example: ```python from typing import overload @overload def f(x: int) -> float: ... @overload def f(x: str) -> str: ... ``` Missing implementations are allowed in `.pyi` stub files. To fix it, provide exactly one implementation (a function of the same name without the `typing.overload` decorator). For example above we could implement `f` as follows: ```python from typing import overload, Union @overload def f(x: int) -> float: ... @overload def f(x: str) -> str: ... @overload def f(x: str) -> str: ... def f(x: Union[int, str]) -> Union[float, str]: if isinstance(x, int): return float(x) else: return x ``` ### 43: Incompatible Overload Implementation Pyre will error if you define one or more overloads using [`typing.overload`](https://fburl.com/d7b8cd2h), and your concrete implementation has an incompatible type signature. For example, this code will produce an incompatible overload implementation error ```python # pyre-strict from typing import overload, Union @overload def f(x: int) -> float: ... @overload def f(x: float) -> int: ... @overload def f(x: str) -> str: ... def f(x: Union[int, float, str]) -> Union[int, str]: if isinstance(x, float): return int(x) elif isinstance(x, int): return float(x) else: return x ``` The problem here is that the return type `Union[int, str]` is too narrow to permit `f` to return `float` when called on an `int` argument. You can fix this by either removing incorrect overload delcarations or making sure all parameters and return type annotations on the concrete implementation are general enough to be consistent with the overloads: ```python def f(x: Union[int, float, str]) -> Union[int, float, str]: ``` ### 45: Invalid Class Instantiation In typed Python, some classes that represent abstract interfaces may not be directly instantiated. Pyre considers a class `C` abstract, and will error on invalid instantiation if you try to construct an instance directly in either of the following cases: 1. `C` contains one or more abstract methods that are left not overridden. Abstract methods are defined as methods that are decorated with [`@abc.abstractmethod`](https://docs.python.org/3/library/abc.html#abc.abstractmethod). For example, here `Derived0` is abstract because it does not override `bar`, but `Derived1` may be instantiated: ```python import abc from typing import Protocol class Base(abc.ABC): @abc.abstractmethod def foo(self) -> None: raise NotImplementedError @abc.abstractmethod def bar(self) -> str: raise NotImplementedError class Derived0(Base): def foo(self) -> None: print(self.bar()) class Derived1(Derived0): def bar(self) -> str: return "bar" def test0() -> None: base = Base() # Error! Class `Base` contains 2 abstract methods and therefore cannot be instantiated. derived0 = Derived0() # Error! Class `Derived0` contains 1 abstract method `bar` and therefore cannot be instantiated. derived1 = Derived1() # OK ``` 2. `C` directly inherits from [`typing.Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol). For example, here `MyProtocol` is abstract because it inherits directly from `typing.Protocol`, but `MyClass` (which implements the protocol interface) may be instantiated: ```python class MyProtocol(Protocol): def baz(self, x: int) -> int: ... class MyClass: def baz(self, x: int) -> int: return x def test1() -> None: object0 = MyProtocol() # Error! Class `MyProtocol` cannot be instantiated. object1 = MyClass() # OK ``` ### 46: Invalid Type Variance In brief, read-only data types can be covariant, write-only data types can be contravariant, and data types that support both reads and writes must be invariant. If a data type implements any functions accepting parameters of that type, we cannot guarantee that writes are not happening. If a data type implements any functions returning values of that type, we cannot guarantee that reads are not happening. For example (note: int is a subclass of float in the type system and in these examples): Writes taking covariants: ```python from typing import TypeVar, Generic _T_co = TypeVar("_T_co", covariant=True) class MyList(Generic[_T_co]): def write(self, element: _T_co) -> None: ... # adds element to list def takes_float_list(float_list: MyList[float]) -> None: float_list.write(1.0) int_list: MyList[int] = ... takes_float_list(int_list) # this call is OK because MyList is covariant: MyList[int] < MyList[float] # int_list contains floats ``` Reads returning contravariants: ```python from typing import TypeVar, Generic _T_cont = TypeVar("_T_cont", contravariant=True) class MyList(Generic[_T_cont]): def read(self) -> _T_cont: ... # returns first element from list def takes_int_list(int_list: MyList[int]) -> int: return int_list.read() float_list: MyList[float] = ... takes_int_list(float_list) # this call is OK because MyList is contravariant: MyList[float] < MyList[int] # problem with return above is clear ``` ### 47: Invalid Method Signature Pyre will error if a non-static method fails to specify an expected implicit parameter like `self` for an instance method or `cls` for a class method, as this argument is always implicitly passed in a call and will cause a runtime crash if not specified. Additionally, Pyre will warn if this parameter is specified but typed as something incompatible with the type of the parent class. Often times, the method may not need a `self` or `cls` and should be decorated with `@staticmethod` to resolve this error. For example, ```python class Foo: def foo() -> None: ... # type error class Foo: @staticmethod def foo() -> None: ... # no type error class Foo: def foo(self) -> None: ... # no type error ``` Only type variables with compatible bounds can be used to annotate the `self` or `cls` parameter. For example, ```python from typing import TypeVar P = TypeVar("T", bound="Parent") A = TypeVar("S", bound="ChildA") B = TypeVar("S", bound="ChildB") class Parent: ... class ChildA(Parent): @classmethod def foo(cls: Type[A]) -> A: ... # no type error class ChildB(Parent): def foo(self: A) -> A: ... # type error def bar(self: B) -> B: ... # no type error def baz(self: P) -> P: ... # no type error ``` ### 48: Invalid Exception In python, you can only raise objects that derive from `BaseException` (it's more common to subtype `Exception` or one of the standard library-defined errors like `ValueError`). Attempting to raise another object such as a bare string will result in a `TypeError`. As a result, pyre will flag code like this: ```python def f(x: int) -> None: if x > 1: raise "x is too big" ``` To fix this, wrap the information you are trying to raise (usually an error message) in some exception type, for example: ```python def f(x: int) -> None: if x > 1: raise ValueError("x is too big") ``` ### 49: Unsafe Cast To allow "safe" casts that preserve type soundness, you can use `pyre_extensions.safe_cast`. This will verify that the type you are casting to is broader than the type of the expression. In cases where this is not the case, pyre will produce an Unsafe Cast error. For example: ```python from pyre_extensions import safe_cast def foo(x: int) -> str: y = safe_cast(str, x) # Unsafe cast error z = safe_cast(Union[int, str], x) # No error return z # Invalid return type error ``` Some context on this: `pyre_extensions.safe_cast` is a type-safe alternative to `typing.cast`. The `typing.cast` function forces type checkers to accept a type for an expression that otherwise would not be valid, which is sometimes useful but also can hide clear type errors, for example: ```python from typing import cast def foo(x: int) -> str: y = cast(str, x) return y # No type error, even though this is unsound. ``` ### 51: Unused Local Mode This error will be thrown if you specify more than one local mode, by having multiple line comments of the form `# pyre-strict` or `# pyre-unsafe` in the header. Pyre will ask you to remove all but one local mode declaration if you have more than one because the mode needs to be unambiguous. Context: Pyre supports two modes of type checking, [unsafe](gradual_typing.md#gradual-typing) and [strict](gradual_typing.md#strict-mode). - By default, every file runs in unsafe mode, but you can change this default to strict in your [configuration file](configuration.md#configuration-files). - In addition, you can set the type checking mode of a module to differ from the default for the project by adding a comment in the form `# pyre-strict` or `# pyre-unsafe` comment on its own line to the file header. ### 52: Private Protocol Property Python [Protocols](https://www.python.org/dev/peps/pep-0544/) provide a way to statically check "duck typing", what many languages would refer to as `interfaces`. Because protocols specify only an interface, they should not include [private fields and methods](https://docs.python.org/3/tutorial/classes.html#private-variables), which cannot not be accessed outside of the class where they are defined (including in subclasses). Pyre will complain about the following: ```python from typing import Protocol class Duck(Protocol): def __quack(self) -> str: ... class SomeDuck: def __quack(self) -> str: return "quack" ``` To signal a non-public part of an interface, use a protected field or method (single leading underscore), which is accessible by classes implementing the interface: ```python from typing import Protocol class Duck(Protocol): def _quack(self) -> str: ... class SomeDuck: def _quack(self) -> str: return "quack" ``` ### 53: Missing Annotation For Captured Variables Pyre makes no attempt at trying to infer the types across function boundaries. The statement holds for nested functions as well. From a nested function's perspective, a variable defined in an nesting function behaves similarly to a global variable. As with global variables, an explicit annotation is required if strict mode is turned on: ```python def outer_function0() -> int: x = foo() def inner_function() -> int: return x # Due to the lack of explicit annotation, Pyre will treat this variable as having type `Any`. return inner_function() def outer_function1() -> int: x: int = foo() def inner_function() -> int: return x # This is ok: the type of `x` is known to be `int`. return inner_function() def outer_function2() -> int: x = foo() def inner_function(x: int) -> int: return x # This is also ok: even though the outer `x` is not annotated, the `x` parameter of the inner function is. return inner_function(x) ``` ### 54: Invalid TypedDict Operation In accordance with [PEP 598](https://www.python.org/dev/peps/pep-0589/), code that tries to assign a value of the wrong type to a field of a `TypedDict` will not typecheck: ```python from typing import TypedDict class MyDict(TypedDict): value: str d: MyDict = {"value": "hello"} d["value"] = 5 # Invalid TypedDict operation ``` To fix this you may need to change your field type to a `Union`, if variable types are actually needed for a field. ### 55: TypedDict Initialization Error Pyre will warn you when initializing a TypedDict with: + Missing required fields ```python from typing import TypedDict class Movie(TypedDict): name: str year: int movie: Movie = {"name": "The Matrix"} $ pyre TypedDict initialization error [55]: Missing required field `year` for TypedDict `Movie`. ``` + Incorrect field type ```python movie: Movie = {"name": "The Matrix", "year": "1999"} $ pyre TypedDict initialization error [55]: Expected type `int` for `Movie` field `year` but got `str`. ``` + Undefined fields ```python movie: Movie = {"name": "The Matrix", "year": 1999, "extra_field": "hello"} $ pyre TypedDict initialization error [55]: TypedDict `Movie` has no field `extra_field`. ``` ### 56: Invalid Decoration This error code is a catch-all for a variety of problems that can arise in the course of resolving the type of a decorated function. In all of these cases, these decoration failures will lead to the function being registered with type `Any` to avoid any spurious downstream errors. #### "Pyre was not able to infer the type of the decorator ..." This should only happen when the decorator access itself is invalid, e.g. when you use a decorator which isn't declared in the stubs for a third-party library. #### "Pyre was not able to infer the type of argument ..." When using the "decorator factory" pattern, we need to resolve the type of both the decorator factory itself as well as the arguments passed to the decorator factory. This is because the types of these arguments can alter the behavior of the returned decorator via overloads or type variables. However, this resolution has to happen early in the environment-building pipeline, when we don't yet have all of the context we need in order to resolve the types of arbitrary expressions. We support resolving literals and simple globals as arguments, but using anything else will result in this error. To work around this, you can statically type your arguments to the decorator factory as separate globals, which can be validated later in the type-checking pipeline. ```python from typing import TypeVar T = TypeVar("T") def decorator_factory(x: T) -> Callable[[Callable[[int], str]], Callable[[str], T]]: ... # pyre-fixme[56]: Pyre was not able to infer the type of argument # `complex_expression()` to decorator factory `decorator_factory`. @decorator_factory(complex_expression()) def foo(x: int) -> str: ... argument: float = complex_expression() @decorator_factory(argument) # Accepted! bar resolves to Callable[[str], float] def bar(x: int) -> str: ... ``` #### "Decorator factory \`X\` could not be called" This corresponds to when the decorator factory access resolves to a type that is not callable (i.e. has no `__call__` method). ```python not_a_factory: int = 5 # pyre-fixme[56]: Decorator factory `not_a_factory` could not be called, because its # type `int` is not callable @not_a_factory(1) def bar() -> None: pass ``` #### "Decorator \`X\` could not be called" Similarly, these errors correspond to when the entire decorator expression (potentially including arguments to a decorator factory), resolves to a non-callable type. ```python def foo() -> int: return 42 # pyre-fixme[56]: Decorator `foo()` could not be called, because its # type `int` is not callable @foo() def bar() -> None: pass ``` #### "While applying decorator factory ..." These errors are emitted from attempting to pass the resolved factory arguments to the factory, as with any other function call. ```python from typing import Callable def factory(x: str) -> Callable[[object], object]: ... # pyre-fixme[56]: While applying decorator factory `factory`: # Expected `str` for 1st param but got `int`. @factory(1) def foo() -> None: pass ``` #### "While applying decorator ..." Correspondingly, these errors are emitted from trying to pass the decorated function as an argument to the resolved decorator type. ```python from typing import Callable def decorator(f: Callable[[int], str]) -> int: ... # pyre-fixme[56]: While applying decorator `decorator`: # Expected `Callable[[int], str]` for 1st param but got `Callable[[str], int]`. @decorator def foo(x: str) -> int: return 5 ``` ### 57: Incompatible Async Generator Return Type An async generator function is an `async` function that contains at least one `yield` statement. The Python runtime ensures that all async generator would return an async generator object. Therefore, the return type of async generator functions should always be `typing.AsyncGenerator` or one of its supertypes. ```python from typing import AsyncGenerator async def f() -> int: # Error yield 0 async def g() -> AsyncGenerator[int, None]: # OK if False: yield 1 ``` ### 58: Unsupported Operand Pyre will warn if an infix operator is not supported for the right or left operands provided. In Python, an infix operator is converted to a method call on either of the operands - for example, `a < b` is equivalent to `a.__lt__(b)`. Therefore, this type error can also be considered sugar for an error that method `a.__lt__` does not accept the type of `b` as an argument. For example, ```python from typing import Optional def foo(x: Optional[int]) -> bool: return x < 0 # type error: Optional[int] is not a supported operand def bar(x: Optional[int]) -> bool: if x: return x < 0 # no type error return False ``` ### 59: Duplicate Type Variables This occurs when the same type variable is provided more than once to a `Generic` or `Protocol`. A type variable needs to be bound to a single value. Thus, if one wants two independent type variables with perhaps the same bounds or same properties, they have to be different variables. ```python from typing import TypeVar, Generic T0 = TypeVar("T0") T1 = TypeVar("T1") T2 = TypeVar("T2") class A(Generic[T0, T1, T0]): # Error pass class B(Generic[T0, T1, T2]): # OK pass ``` ### 60: Unable to Concatenate Tuple #### "Expected to unpack an iterable ..." This can occur if during concatenation of a tuple one tries to unpack a non-iterable since non-iterables can't be unpacked. Either try to unpack an iterable, or concatenate without unpacking. ```python def foo(x: int, not_iterable: int, iterable: list[int]) -> None: y = (x, *not_iterable) # Error z = (x, not_iterable) # OK w = (x, *iterable) # OK ``` #### "Concatenation not yet supported for multiple variadic tuples ..." This can occur if during concatenation one tries to use multiple variadic tuples. This is due the limitations of the current type system and there is no workaround currently. One may use `# pyre-ignore[60]` to suppress. ```python from typing import Tuple from pyre_extensions import TypeVarTuple Ts = TypeVarTuple("Ts") def foo(xs: Tuple[*Ts]) -> None: y = (*xs, *xs) # Error ``` ### 61: Uninitialized Local This indicates that there are code paths along which a local variable may not be initialized. Below are some common code patterns that may cause this error: #### Not initialized in all branches of condition ```python def f(x: int) -> None: z = None if x > 5: y = 2 z = 2 print(y) # Error print(z) # OK ``` `y` is not defined when the `if` condition is not met. For instance, `f(4)` will result in a runtime error. Possible ways to address this: - initialize `y` to a default value, outside the conditional or in the `else` branch - refactor so that initialization and access are in the same conditional Pyre static analysis does not reason about runtime values or potential side effects of interleaving calls, so for instance, in the example below we cannot guarantee that the two if statements will always be consistent and, hence, throw the same error: ```python def f(x: int) -> None: if x > 5: y = 2 # ...some operations... if x > 5: print(y) # Error ``` #### Initialized only inside a `for` loop ```python def f(xs: List[int]) -> None for x in xs: y = "yes" print("Last element is: ", x) # Error print("Did we enter the loop?", y) # Error ``` Here, if one calls `f([])`, it will result in errors. One way to remediate is to initialize outside the loop. For instance, ```python def f(xs: List[int]) -> None: x = None y = "no" for x in xs: y = "yes" print("Last element is: ", x) # OK print("Did we enter the loop?", y) # OK ``` #### Initialized in `try` block ```python def f(divisor: int) -> None: answer_good = None try: answer_bad = 5 / divisor answer_good = 5 / divisor answer_also_good = 5 / divisor print(f"5 divided by {divisor} is {answer_also_good}") # OK except ZeroDivisionError: pass print(f"5 divided by {divisor} is {answer_bad}") # Error print(f"5 divided by {divisor} is {answer_good}") # OK ``` Here, `f(0)` leads to an error on access of `answer_bad`. Suggested approaches to address this: - Initialize any variables needed after the `try` block to a default value before entering the `try` block. - Keep the access to variables initialized inside the `try` block within the `try` block. - Consider if pulling the initialization as-is before the `try` block is possible. It is generally considered a good practice to minimize the code inside a try block, and keep it to exception throwing code. This also helps with Pyre, as it does not reason about which operations might throw exceptions. ```python def bad(divisor: int) -> Optional[int]: try: dividend = 5 return dividend // divisor except ZeroDivisionError: print(f"Cannot divide {dividend} by 0") # Error (according to Pyre) def good(divisor: int) -> Optional[int]: dividend = 5 try: return dividend // divisor except ZeroDivisionError: print(f"Cannot divide {dividend} by 0") # OK ``` ### 62: Non-literal string Pyre will error if you passs in a non-literal string into a function call that expects a LiteralString. ```python def query_user(conn: Connection, user_id: str) -> User: query = f"SELECT * FROM data WHERE user_id = {user_id}" conn.execute(query) # Error: Expected LiteralString, got str. ... ``` ```python def query_user(conn: Connection, user_id: str) -> User: query = "SELECT * FROM data WHERE user_id = ?" conn.execute(query, (user_id,)) # OK. ``` LiteralStrings are created either from an explicit string literal like "foo" or from combining multiple string literals or LiteralStrings. This is a security focused typing feature for safely calling powerful APIs. See [PEP 675](https://peps.python.org/pep-0675/#appendix-b-limitations) for more details. ### 63: Suppression Comment Without Error Code When running Pyre in strict mode, suppression comments like `pyre-ignore` and `pyre-fixme` need to be associated with a specific error code or set of error codes. OK: ```python def foo() -> int: # pyre-ignore[7] return "" ``` Not OK: ```python def foo() -> int: # pyre-ignore return "" ``` For more details on how to write suppression comments, refer to the section on [Suppression](https://www.internalfb.com/intern/staticdocs/pyre/docs/errors/#suppressing-individual-errors) later in this doc. ### 64: Inconsistent Method Resolution Order Python supports multiple inheritance via multiple base classes. In order to decide the order that methods from base classes are overridden, there needs to be a consistent order among the base classes of a particular class. When this order cannot be created, for example because there is a cycle, then Pyre raises this error. ```python class A(B): # Error pass class B(A): # Error pass ``` For more details and examples for Python's Method Resolution Order (MRO), please refer to [the official guide](https://www.python.org/download/releases/2.3/mro/). ### 65: Duplicate Parameter Functions may not have multiple named parameters that share the same name. ```python def add(a: int, a: int) -> None: # Error pass ``` ### 66: Invalid Exception Handler Exception types listed in an `except` clause must extend `BaseException`. ```python try: pass except (int, bool): # Error pass ``` ### 67: Invalid Exception Group Handler Exception types listed in an `except*` clause (Python 3.12 exception group handler) must extend `BaseException` and may not extend `BaseExceptionGroup`. ```python try: pass except* ExceptionGroup as e: # Error pass ``` ```python try: pass except* int as e: # Error pass ``` ### 68: Invalid Type Guard User defined type-guards are functions that return `TypeIs[X]` or `TypeGuard[X]`; at runtime these return a `bool`, but the static type indicates to the type checker that they narrow the type of the first positional argument, for example: ```python from typing import TypeIs, reveal_type def is_int_or_str(val: object) -> TypeIs[int | str]: ... def f(x: object): if is_int_or_str(x): reveal_type(x) # int | str ``` The difference between `TypeGuard` and `TypeIs` is that `TypeGuard` only narrows in the positive case and is covariant in its type parameter, whereas `TypeIs` also narrows in the negative case and is invariant in its type parameter; for example: ```python # (extending the previous example) def is_int_or_str_typeguard(val: object) -> TypeGuard[int | str]: ... def g(x: int | str | bytes): if is_int_or_str(x): reveal_type(x) # int | str else: reveal_type(x) # bytes if is_int_or_str_typeguard(x): reveal_type(x) # int | str else: reveal_type(x) # int | str | bytes (no narrowing here) ``` It is a type error if such a type guard function does not accept at least one positional argument; the type guard cannot actually be invoked correctly in this case. For example: ```python from typing import TypeIs, TypeGuard # Error: the argument is keyword-only, this is not a valid type guard. def bad_type_guard(/, val: objec) -> TypeGuard[float]: ... class CustomTypeGuard: # Error: this is a non-static method and `self` is bound, so it does not # accept a positional argument. def guard(self) -> TypeIs[str]: ... ``` In addition, for `TypeIs` it is an error if the narrowed type (the type inside the `TypeIs`) is not assignable to the type of the first positional parameter. ```python from typing import TypeIs # Error: `int` is not a subtype of `str` def inconsistent_type_is(x: str) -> TypeIs[int]: ... ``` You can read more about type guards and narrowing in the specification: https://typing.readthedocs.io/en/latest/spec/narrowing.html. ### 69: Invalid Positional-Only Parameter Python has positional-only parameters which may not be passed by name. Positional-only parameters cannot appear after parameters that may be passed by name. Typically, all the parameters that proceed `/` in the parameter list are considered positional-only, but in functions without this syntax we treat parameters that are prefixed but not suffixed with `__` as positional-only for backwards-compatibility purposes. Refer to https://typing.readthedocs.io/en/latest/spec/historical.html#positional-only-parameters for more details. In the following example, `__y` is a positional-only that follows the regular parameter `x`, which is an error. ```python def foo(x: int, __y: int) -> None: # Error pass ``` ### 70: Assert Type `assert_type` is a static assertion that has no effect at runtime. Pyre will emit an error if the inferred type for the first argument does not match the type provided as the second argument. ```python x: int = 1 assert_type(x, str) # Error ``` ### 71: Typed Dictionary Isinstance Typed dictionaries are structurally typed. This means that it is invalid to use `isinstance` to check whether something is an instance of a typed dictionary. ```python class Coord(TypedDict): x: int y: int isinstance({}, Coord) # Error ``` ### 72: Tuple Delete Tuples are immutable, so deleting elements is not allowed. ```python x = (1, 2, 3) del x[0] # Error ``` ### 73: Tuple Out of Bounds When a tuple with a concrete/known length is indexed with a literal, Pyre will emit an error if the index is out of bounds. ```python x = (1, 2, 3) y: int = ... x[0] # OK x[-1] # OK x[5] # Error x[y] # OK ``` ### 74: Named Tuple Default Fields Since named tuple constructors are generated based on the order fields are declared in, fields without defaults may not follow fields with defaults in a named tuple declaration. ```python # This is OK class MyTuple(NamedTuple): x: int y: int = 1 # This is not allowed class MyTuple2(NamedTuple): x: int = 1 y: int ``` ## Suppression It is not always possible to address all errors immediately – some code is too dynamic and should be refactored, other times it's *just not the right time* to deal with a type error. We do encourage people to keep their type check results clean at all times and provide mechanisms to suppress errors that cannot be immediately fixed. ### Suppressing Individual Errors Pyre supports error suppression of individual errors with comments that can be placed on the line of the error or on the line preceding the error. - `# pyre-fixme` indicates there is an issue in the code that will be revisited later. - `# pyre-ignore` indicates there's an issue with the type checker or the code is too dynamic and we have decided to not fix this. If this is a Pyre bug, make sure you [open an issue](https://github.com/facebook/pyre/issues) on our tracker. Both comment styles allow you to suppress individual error codes as well as adding additional context. ```python def foo() -> int: # pyre-fixme[7]: only suppresses return mismatches return "" ``` Pyre also supports `# type: ignore` comments for backwards-compatibility with *MyPy*. ### Suppressing Errors within Format Strings If you want to suppress an error within an f-string, you can add a fixme comment on the line before the string. This will suppress all errors within the f-string matching that fixme: ```python def print_profile(name: str, age: Optional[int]) -> None: # pyre-fixme[58]: `-` is not supported for operand types `Optional[int]` and `int`. s = f""" Your personal details!! Name: {name} Age: {age - 3} """ print(s) ``` ### Suppressing All Errors You can use the [Pyre upgrade tool](gradual_typing.md#upgrade) to add inline error suppressions for all errors in your project. ### Suppressing Errors Across Files You can suppress all errors in entire sections of your code by adding the path to the [`ignore_all_errors` section of your configuration](configuration.md#the-global-configuration). Furthermore Pyre supports suppressing all errors in an individual file if you add a `# pyre-ignore-all-errors` to your file. Like the other suppression comments, you can use square brackets to chose to only ignore one or more particular error types. For example, you can suppress all incompatible return type errors by adding: ```python # pyre-ignore-all-errors[7] def foo(x: int) -> str: return x # pyre will not error here ``` ## Debugging --- ### Documentation/Website/Docs/Features --- id: features title: Pyre-Exclusive Features sidebar_label: Pyre-Exclusive Features --- Pyre has custom support for Python idioms that would otherwise not be supported by the usual type annotations. ## Registering attributes using PyTorch's register_buffer PyTorch allows subclasses of [`nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=nn%20module#torch.nn.Module) to [register](https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=register_buffer#torch.nn.Module.register_buffer) a buffer in an object using `self.register_buffer("foo", initial_value)`. Pyre supports this pattern when used within the constructor. It simply treats the buffer as a Tensor attribute of the class: ```python import torch import torch.nn as nn class Foo(nn.Module): def __init__(self) -> None: super(Foo, self).__init__() self.register_buffer("foo", torch.zeros(10, 20)) self.register_buffer("foo_persistent", torch.zeros(10, 20), persistent=False) def bar(self) -> None: reveal_type(self.foo) # => torch.Tensor reveal_type(self.foo_persistent) # => torch.Tensor def baz() -> None: y = Foo().foo reveal_type(y) # => torch.Tensor ``` Note that Pyre will not recognize buffers registered in methods other than the constructor (just like it doesn't recognize [attributes](errors.md#16-missing-attributes) defined in methods other than the constructor). It will also not recognize buffers that are initialized with `None` since it cannot infer the exact type of the buffer. In such a case, you can tell Pyre about the attribute's type by explicitly defining it in the class: ```python import torch import torch.nn as nn from typing import Optional class Foo(nn.Module): my_none_buffer: Optional[torch.Tensor] def __init__(self) -> None: super(Foo, self).__init__() self.register_buffer("my_none_buffer", None) def bar(self) -> None: reveal_type(self.my_none_buffer) # => Optional[torch.Tensor] ``` --- ### Documentation/Website/Docs/Getting Started --- id: getting-started title: Getting Started with Pyre sidebar_label: Getting Started --- import {OssOnly, FbInternalOnly} from 'docusaurus-plugin-internaldocs-fb/internal'; import FbGettingStarted from './fb/getting_started.md'; import YouTube from 'react-youtube'; Welcome to the wonderful world of static typing! This guide will get you from zero to a simple project that is type checked with Pyre. ## Requirements To get started, you need [Python 3.9 or later](https://www.python.org/getit/) and [watchman](https://facebook.github.io/watchman/) working on your system. On *MacOS* you can get everything with [homebrew](https://brew.sh/): ```bash $ brew install python3 watchman ``` On *Ubuntu*, *Mint*, or *Debian*; use `apt-get` and [homebrew](https://brew.sh/): ```bash $ sudo apt-get install python3 python3-pip python3-venv watchman ``` We tested Pyre on *Ubuntu 18.04.5 LTS*, *CentOS 7*, as well as *OSX 10.11* and later. ## Setting up a Project We start by creating an empty project directory and setting up a virtual environment: ```bash $ mkdir my_project && cd my_project $ python3 -m venv ~/.venvs/venv $ source ~/.venvs/venv/bin/activate (venv) $ pip install pyre-check ``` Next, we teach Pyre about our new project: ```bash (venv) $ pyre init ``` This command will set up a configuration for Pyre (`.pyre_configuration`) as well as watchman (`.watchmanconfig`) in your project's directory. Accept the defaults for now – you can change them later if necessary. ## Running Pyre We are now ready to run Pyre: ```bash (venv) $ echo "i: int = 'string'" > test.py (venv) $ pyre ƛ Found 1 type error! test.py:1:0 Incompatible variable type [9]: i is declared to have type `int` but is used as type `str`. ``` This first invocation will start a daemon listening for filesystem changes – type checking your project incrementally as you make edits to the code. You will notice that subsequent invocations of `pyre` will be faster than the first one. ## Introductory Video ## Further Reading This page should contain all of the basic information you need to get started with type checking your own project. If you are new to the type system, the [introduction to types in Python](gradual_typing.md) is recommended reading to familiarize with the type system, gradual typing, and common type errors. If you are looking for more options to configure your type checking experience, the [configuration](fb/configuration.md) page explores command line and configuration file settings. If you are looking for more options to configure your type checking experience, the [configuration](configuration.md) page explores command line and configuration file settings. For questions and support, please use our [Pyre Q&A](https://fb.workplace.com/groups/pyreqa) workplace group. For more insight into our roadmap, new features, and progress, please read our [Pyre FYI](https://fb.workplace.com/groups/295311271085134) workplace group. --- ### Documentation/Website/Docs/Gradual Typing --- id: types-in-python title: Types in Python sidebar_label: Introduction --- Python's type system was specified in [PEP 484](https://www.python.org/dev/peps/pep-0484/). If you are new to Python's type system and want to learn the basics, we highly recommend you take a look at [mypy's cheatsheet](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html) as well as their [type system reference](https://mypy.readthedocs.io/en/stable/builtin_types.html). The following discussion focuses on Pyre's approach to "gradual typing" and how you can get from an untyped codebase to a fully typed codebase. ## Why Types? ## Gradual Typing Most Python code does not (yet) start out typed. PEP 484 specifies a [gradual type system](https://en.wikipedia.org/wiki/Gradual_typing), which is built to allow you to *gradually* add annotations over time. It does so by - only reporting errors on functions that have an explicit return or parameter type annotation, - introducing an escape hatch: a special type `Any` that has all possible attributes and is both sub- and super-type of any other type, - and assuming that all untyped fuctions implicitly return `Any`. For example, ```python from typing import List def unannotated(): # implictly returns `Any` return b"" + "" # function body is not checked def annotated() -> List: # explicit return annotation means we type check `annotated` any = unannotated() any.attribute # `Any` has all possible attributes return 1 # Error: returning `int` but expecting `List` ``` In combination, these rules allow you to slowly annotate code without getting overwhelmed by type errors in one sitting. Incrementally adding more annotations will give you stronger safety and consistency guarantees in your codebase. In the example above, if you changed `unannotated` to return `str`, you would get a type error when accessing the attribute `any.attribute` in `annotated`. ## Strict Mode While `Any` is a necessary escape hatch when annotating large codebases over time, it can hide legitimate type errors. We've introduced *strict mode* in Pyre to address this problem. Strict mode can be toggled at a module level by introducing a `# pyre-strict` comment to the file. In strict mode, Pyre will - run on all functions, whether they are annotated or not, - error on functions, globals, or attributes that are missing annotations, - and error on annotations containing `Any` (with some exceptions to accommodate for common patterns). In our previous example, ```python # pyre-strict from typing import List def unannotated(): # Error: missing return annotation return b"" + "" # Error: function body *is* checked def annotated() -> List: # Error: implicit `Any` for generic parameter to `List` any = unannotated() any.attribute # Note: the type of `any` is still any. return 1 # Error: returning `int` but expecting `List` ``` As you can see in the example, `Any` can still sneak into modules that are strict, but increasing strict coverage and fixing the surfaced errors will gradually eliminate them. ### Strict-By-Default Strict mode can also be set as the default in a [project configuration](configuration.md). To opt individual files out of strict mode, use `# pyre-unsafe` in place of `# pyre-strict`. ## How to move away from `Any` ### What if I want to use `Any`? It may be tempting to annotate a generic function parameter with `Any`. But while it is a convenient utility for quickly annotating untyped code, it has no place in a strict-mode codebase. The main problem with `Any` is that it unifies with every type - which effectively hides all potential type errors that could stem from incorrect usage of anything annotated with `Any`. It is not, therefore, a good idea to use it in generic code. ### What then? We have two main methods for annotating generic code: `object` and `TypeVar`. As it turns out, it is not at all obvious which of them should be used where. However, the general **TL;DR** is that `object` is an opaque superclass of all types, while `TypeVar` is for preserving a type across one or more function calls. ### About `object` The advantage of using `object` over `Any` is that while any type can be “put inside” it, it is an error to use it as any type other than `object`. This can be useful everywhere we need type erasure, like (de)serialization or generic heterogeneous containers where there is no obvious common supertype. Thanks to Python’s runtime reflection, the original type can be recovered, e.g. through `isinstance`. Note that such checks are valid only for a short time, see [here](errors.md#optional-attributes). ### About `TypeVar` `TypeVar`s are somewhat interesting beasts, as in a vacuum, they can behave both like `Any` and like `object`. Like `object`, they accept all types, but within a single typecheck (i.e. one line/function call/operation) they remember what type they were. This makes them useful for e.g. linking the parameters and return types of a function, or class attribute types with its method signatures. There are, however, a couple of caveats. It doesn’t make sense to use a TypeVar on a function or method when it’s only used in its parameters (just use the most general known supertype instead). Another story is the difference between `TypeVar`s’ invariance, variance and contravariance, which is covered in [here](errors.md#covariance-and-contravariance). ## When Source Code is not Available We do not always have access to all the source code that contributes type information to our project: e.g. `builtins` is compiled native code, and other libraries may be using *Cython*. Other times, we may be working with Python code that is just too dynamic to be reasonably typed. To address these cases, Pyre will give precedence to type *stub files* with a `*.pyi` extension over source files when these are specified in the search path in the [project configuration](configuration.md) or if they are located next to the implementation file. Stub files have the same structure as implementation files but only contain class and function signatures: ```python # my_dynamic_module.pyi def dynamic_function() -> int: ... # Function body is omitted ``` If a `__getattr__` function is defined in the stub file as follows, Pyre will take it as a signal that the stub file is partially complete: accessing attributes whose name is not defined in the stub file will result in `Any` instead of a type error. ```python # my_stub.pyi from typing import Any foo: int = 42 # Parameter needs to be typed as `str` and return type needs to be `Any` def __getattr__(name: str) -> Any: ... # my_source.py import my_stub reveal_type(my_stub.foo) # Reveals `int` reveal_type(my_stub.undefined) # Reveals `Any` ``` ### Typeshed ## Strategies for Increasing Coverage Pyre comes with tooling to make it easy to increase type coverage in your project. ### Upgrade When upgrading the type checker, new errors inevitably get surfaced. In order to keep a codebase clean through upgrades we've built `pyre-upgrade`, which automatically [suppresses](errors.md#suppression) newly surfaced type errors. It takes Pyre's output and adds supression comments to the code explaining what's wrong so that developers can easily address the issues individually. You can run `pyre-upgrade` with ```bash (venv) $ pyre --output=json | pyre-upgrade fixme ``` or if you are using a local configuration ```bash (venv) $ pyre --output=json -l | pyre-upgrade fixme ``` ### Automatic Type Inference We have found tools that automatically add type annotations to code useful to get started with a project. There are two general approaches to automatic type inference: static inference and dynamic inference from runtime information. Both approaches come with their own trade-offs and we have found a combination of the two to be useful. Pyre can do static type inference. You can run ```bash (venv) $ cd ; pyre infer -i ``` to automatically apply annotations. For dynamic inference we recommend you give [MonkeyType](https://github.com/Instagram/MonkeyType) a try. --- ### Documentation/Website/Docs/Installation --- id: installation title: Installation sidebar_label: Installation --- We recommend that you use our binary distribution through [pypi](https://pypi.org/) inside of a virtual environment, which supports both *MacOs* and *Linux*. On *Windows* we have successfully gotten `pyre` to work through [WSL](https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux), but do not officially support it. ## Binary Distribution You can get Pyre through [pypi](https://pypi.org/) by running: ```bash $ (venv) $ pip install pyre-check ``` See our [Getting Started](getting_started.md) section for a more detailed example, including setup for a virtual environment. ## IDE Integration Pyre supports the [Language Server Protocol](https://en.wikipedia.org/wiki/Language_Server_Protocol). We provide an [extension for *VSCode*](https://marketplace.visualstudio.com/items?itemName=fb-pyre-check.pyre-vscode) that will automatically try to connect to a running server. You can also directly interact with the LSP by piping the appropriate `JSON` into `pyre persistent`. ## Building from Source These instructions are known to work on *Mac OS X* (tested on *High Sierra* through *OSX 10.13* - even though binaries are compatible with versions as old as *10.11*) and *Linux* (tested on *Ubuntu 16.04 LTS* and *CentOS 7*). ### Requirements In addition to [Python and watchman](getting_started.md#requirements), we need a working *OCaml* compiler. We use [Opam](https://opam.ocaml.org/) to manage our compiler and libraries. You can get Opam via various package management systems. Please follow their instructions for your particular operating system. ### Building the OCaml binary First, clone the repository from [GitHub](https://github.com/facebook/pyre-check) using: ```bash $ git clone https://github.com/facebook/pyre-check ``` You can complete the setup of your development environment with: ```bash $ cd pyre-check $ ./scripts/setup.sh --local ``` This will compile Pyre and run all the unit tests. This is likely going to take some time on your system. You can now make changes to the code. Run the following commands to compile and test your changes: ```bash $ cd source $ make $ make test ``` ### Testing changes to the Python Client In a virtualenv, install dependencies with `requirements.txt` and run python tests to make sure everything is set up correctly ```bash $ cd /path/to/pyre-check $ pip install -r requirements.txt $ ./scripts/run-python-tests.sh ``` When installing and running `pyre` from PyPi, the entry point to the executable is actually `client/pyre.py`. To be able to run this file from anywhere, add the directory containing the `pyre-check` directory to the `PYTHONPATH` environment variable and subsequently assign `pyre` as an alias for `pyre-check.client.pyre`. For the `pyre` command to correctly point to the compiled binary, also set the environment variable `PYRE_BINARY` to `source/build/default/main.exe`. ```bash $ echo "alias pyre='PYTHONPATH=\"/path/to/pyre-check/..:\$PYTHONPATH\" python -m pyre-check.client.pyre'" >> ~/.bashrc $ echo "export PYRE_BINARY=/path/to/pyre-check/source/_build/default/main.exe" >> ~/.bashrc $ source ~/.bashrc ``` You should be able to open a new shell and run `pyre -h` now, confirming `pyre` was set-up correctly. Any changes made to the Pyre Python client code should be immediately observable the next time you invoke `pyre` #### Testing changes for Plugin Development VSCode will not pick up your shell aliases, so the alias step in the previous section will not work if you're doing VSCode Plugin development. To get around this, instead of creating an alias, we can create an executable script called `pyre` and place it in a directory in our `PATH`: ```bash #!/bin/bash PYTHONPATH="/path/to/pyre-check/..:$PYTHONPATH" python -m pyre-check.client.pyre "$@" ``` Add the `pyre-check/scripts` directory to `PATH` (assuming you placed the above script in that directory) and then use the command `pyre` to launch the client like before ```bash $ echo 'PATH="/path/to/pyre-check/scripts:$PATH"' >> ~/.bashrc $ source ~/.bashrc ``` ## Building from Docker If you're having issues setting up or your OS is not yet supported, you can also use a Docker image. It runs [Debian GNU/Linux 10 (buster)](https://www.debian.org/) and builds pyre-check from source. Before starting, ensure that [Docker](https://docs.docker.com/get-docker/) is installed on your computer. 1. Clone the pyre-check repository and navigate to the root directory. ```bash git clone https://github.com/facebook/pyre-check.git cd pyre-check ``` 2. Build the Docker image with the tag `pyre-check` (or another tag if you wish) ```bash docker build -t pyre-docker . ``` 3. Run the new image in a new container `pyre-container` (or another name if you wish) ```bash docker run \ --name pyre-container \ -v /path/to/your/directory:/src \ -t -i \ pyre-check ``` *Note: Launching the container will build and run all tests.* 4. Inside the container, run any Pyre command now with `pyre`! *Note: When initializing Pyre with `pyre init`, you may have to enter the following paths for the binary and typeshed:* ```bash ƛ No `pyre.bin` found, enter the path manually: /home/opam/pyre-check/source/_build/default/main.exe ƛ Unable to locate typeshed, please enter its root: /home/opam/pyre-check/stubs/typeshed/typeshed-master ``` **For contributors:** Inside the Docker container, the added pyre-check directory is only editable by the root user. To contribute to Pyre, make edits in your local filesystem and rebuild the Docker by running Step 2, then running a new Docker container in Steps 3-4. ## Windows Subsystem for Linux (WSL) Install On *x86_64* Windows `pyre` can run via *Linux* using [WSL](https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux). A brief summary to get this running on *Ubuntu* please follow: - [Install WSL](https://docs.microsoft.com/en-us/windows/wsl/install-win10) *(external Microsoft Documentation)* - Once you have a login to your Linux of choice: - Optionally: Install build environment (some dependencies of `pyre` might need to be built) - Use `pip` as described above or via a [Python Virtual Environment](https://docs.python.org/3/tutorial/venv.html) - Here is an example using [Ubuntu](https://www.ubuntu.com/) with a [venv](https://docs.python.org/3/tutorial/venv.html): ```bash $ sudo apt install python3-venv build-essential python3-dev libpython3-dev $ python3 -m venv /tmp/tp $ /tmp/tp/bin/pip install --upgrade pip setuptools wheel $ /tmp/tp/bin/pip install pyre-check $ source /tmp/tp/bin/activate $ cd /mnt/c/path/to/repo $ pyre --source-directory . check $ (tp) cooper@TESTFAC-1FMHLI2:/mnt/c/path/to/repo$ pyre --source-directory . check ƛ Setting up a `.pyre_configuration` with `pyre init` may reduce overhead. ƛ No type errors found ``` --- ### Documentation/Website/Docs/Overview --- id: overview title: Overview sidebar_label: Overview --- import {Redirect} from '@docusaurus/router'; const Home = () => { return ; }; --- ### Documentation/Website/Docs/Pyre Faq --- id: pyre-faq title: FAQ & Troubleshooting sidebar_label: FAQ --- import {OssOnly, FbInternalOnly} from 'docusaurus-plugin-internaldocs-fb/internal'; WIP ## Thrift Stubs WIP ## Pyre Rage ## Typeshed & Stubs --- ### Documentation/Website/Docs/Pysa Advanced --- id: pysa-advanced title: Advanced Topics sidebar_label: Advanced Topics --- This page documents more advanced bits of Pysa. ## Conditional models Pysa models support if conditions for version and platform checks. This allows for conditional parsing of models based on the Python version or the target platform. ### Conditional models based on Python version ```python if sys.version == (3,9,0): def module.foo(): ... else: def module.bar(): ... ``` In this example, the first model will only be parsed and honored if the python version in the system or virtual environment from which Pysa is run is equal to `3.9.0`. In all other conditions, the second model will be parsed and honored. sys.version is the only allowed left hand expression and the right hand expression has to be a tuple of integers of the form (major, minor, micro). Only the major version number is required and the other two are optional. The comparison operators supported include `==` (equal to), `!=` (not equal to), `<` (less than), `>` greater than, `<=` (less than or equal to), and `>=` (greater than or equal to). If conditions can also be nested inside one another and follow the same behavior as python if conditions. ### Conditional models based on platform Models can also be conditionally enabled based on the target platform using `sys.platform`: ```python if sys.platform == "linux": def module.foo(): ... else: def module.bar(): ... ``` In this example, the first model will only be parsed if the target platform is `"linux"`. Otherwise, the second model will be used. The supported platform strings include `"linux"`, `"win32"`, and `"darwin"`, matching Python's `sys.platform` values. Only `==` (equal to) and `!=` (not equal to) comparison operators are supported for platform checks. Platform and version conditions can be nested: ```python if sys.version >= (3, 10): if sys.platform == "linux": def module.foo(): ... ``` You can also combine conditions using `and`, `or`, and `not`: ```python if sys.version >= (3, 10) and sys.platform == "linux": def module.foo(): ... if sys.version < (3, 9) or sys.platform == "win32": def module.bar(): ... if not sys.platform == "darwin": def module.baz(): ... ``` ## Obscure models When Pysa does not have enough information about a function or method, it will make basic assumptions about its behavior. This is referred to as an **obscure model**. Most notably, it assumes that the function or method propagates the taint from its arguments to its return value. This usually happens when Pysa doesn't know about the callee of a function call: ```python def foo(f: Any): x = input() y = f(x) # no information about `f`, y will be considered tainted. eval(y) ``` Functions and methods defined in type stubs or in a different language (for instance, in C or C++ bindings) will also be treated as obscure models. To prevent a function or method from being marked as obscure, one can use the `@SkipObscure` taint annotation in a `.pysa` file: ```python @SkipObscure def module.foo(): ... ``` ## Parameter and return path When writing a model for a source, the `ReturnPath` annotation allows to specify which index or attribute of the returned value is tainted. For instance: ```python def only_attribute_foo_tainted() -> TaintSource[Test, ReturnPath[_.foo]]: ... ``` Similarly, the `ParameterPath` annotation allows to specify which index or attribute of an argument leads to a sink. For instance: ```python def only_arg_dot_bar_is_sink(arg: TaintSink[Test, ParameterPath[_.bar]]): ... ``` ### Access path definition The `ParameterPath` and `ReturnPath` annotation takes an **access path** as an argument. An access path starts with an underscore `_` which represents the whole argument or return value (depending on the context). The underscore can be followed by attribute accesses (e.g, `_.foo.bar`) and index accesses (e.g, `_["foo"][0]["bar"]`), or a combination of both (e.g, `_.foo[0]`). In addition to these, four special calls can be used: `.all()`, `.keys()`, `.parameter_name()` and `.all_static_fields()`. #### all() `.all()` is used to represent that any index might be tainted. This is usually when the index cannot be known statically. For instance: ```python def foo(i: int): i = random.randint(0, 100) return {i: source()} ``` This can be represented by the model: ```python def foo(): TaintSource[Test, ReturnPath[_.all()]]: ... ``` #### keys() `.keys()` is used to represent that any key of the dictionary might be tainted. For instance: ```python def foo(): return {source(): 0} ``` This can be represented by the model: ```python def foo(): TaintSource[Test, ReturnPath[_.keys()]]: ... ``` #### all_static_fields() `.all_static_fields()` is used to mark all statically-known attributes of the given parameter or return value as a source or sink. The set of attributes is determined using the type annotation of the parameter or return value. If it is not annotated or that we could not find any attributes, the whole parameter or return value will be marked as a source or sink instead. For instance: ```python class A: x: str y: str def foo(a: A) -> B: ... ``` Using the following model: ```python def foo(a: TaintSink[Test, ParameterPath[_.all_static_fields()]]): ... ``` This will add a sink on `a.x` and `a.y`. This can also be used on the return value: ```python def foo() -> TaintSource[Test, ReturnPath[_.all_static_fields()]]: ... ``` In general, we recommend to **mark the whole parameter or return value** as a source or sink. This feature is **only useful** for power users that post process the result of the analysis and extract leaf ports. It is also very **computationally expensive**. #### parameter_name() `.parameter_name()` will be replaced by the name of the parameter that is being modelled. This can only be used for `TaintInTaintOut` on parameters of functions or methods. This is usually useful to model constructors of dataclass-like classes. For instance: ``` class A: def __init__(self, x, y, z): # method too complicated pass ``` Using the following [model query](pysa_model_dsl.md): ```python ModelQuery( name="constructors", find="method", where=[fully_qualified_name.equals("A.__init__")], model=[ Parameters(TaintInTaintOut[LocalReturn, NoCollapse, ReturnPath[_.parameter_name()]]), Modes([SkipAnalysis]) ] ) ``` This will automatically propagate taint from parameters `x`, `y`, `z` to `self.x`, `self.y` and `self.z`. Note that if the code is available and the constructor is not too complex, Pysa will do that automatically without the need for a model. ### Taint In Taint Out `ParameterPath` and `ReturnPath` can also be used to give more information about a propagation. For instance: ```python def foo(arg): return {"a": arg["b"][42]} ``` This can be represented by the model: ```python def foo(arg: TaintInTaintOut[ParameterPath[_["b"][42]], ReturnPath[_["a"]]]): ... ``` Note that Pysa will automatically infer propagations if it has access to the body of the function. Writing taint-in-taint-out models should rarely be required. When using the `Updates` annotation, the annotation `UpdatePath` is used instead of `ReturnPath`. For instance: ```python def MyClass.updates_foo(self, x: TaintInTaintOut[Updates[self], UpdatePath[_.foo]]): ... ``` ## Taint propagation from arguments to self By default, Pysa infers taint propagation from arguments to self for all methods, including constructors, property setters and the special `__setitem__` method. For instance: ```python class Foo: def __init__(self, x): self.x = x def set_x(self, x): self.x = x def issue(): foo = Foo(source()) sink(foo) # Issue (1) found. foo = Foo("") foo.set_x(source()) sink(foo) # Issue (2) found. ``` To disable the inference of propagations from arguments to self for all methods - except constructors, property setters and `__setitem__` - one can provide the command line argument `--no-infer-self-tito`. This should reduce the analysis time, as well as reduce false positives. The counterpart is that it can also lead to false negatives. For instance, Pysa wouldn't find issue (2) anymore. When using `--no-infer-self-tito`, we can use the taint annotation `@InferSelfTito` in a `.pysa` file to enable inference for specific methods: ```python @InferSelfTito def my_module.Foo.set_x(): ... ``` Pysa would now find the second issue properly. ## Taint propagation between arguments By default, Pysa does NOT infer taint propagation between arguments. For short, it assumes that functions do not mutate their arguments. For instance, this flow will NOT be found: ```python def append_wrapper(l: List[str], v: str) -> None: l.append(v) def issue(): l = [] append_wrapper(l, source()) sink(l[0]) # Issue NOT found. ``` To enable the inference of propagations between arguments for all functions and methods, one can provide the command line argument `--infer-argument-tito` or use the taint annotation `@InferArgumentTito` in a `.pysa` file: ```python @InferSelfTito def my_module.append_wrapper(): ... ``` Pysa would now find the issue properly. Note that `--infer-argument-tito` can significantly increase the analysis time as well as the amount of false positives. ## Taint broadening **Taint broadening** is an over-approximation performed by the taint analysis for correctness or performance reasons. After applying broadening, Pysa considers that a whole object or variable is tainted when only some attributes or keys were initially tainted. This is also called **taint collapsing** or **tree collapsing** because the taint is internally represented as a tree structure where edges are attributes or keys. Collapsing means merging the taint on all children into the root of the tree. For instance, this happens when Pysa does not have access to the body of a function: ```python def obscure_function(arg): ... def foo(): # Only `x['a']` is tainted. x = {"a": source()} # Taint broadening happens, `y` and all its attributes are considered tainted. y = obscure_function(x) # This is considered an issue, even if only `x['a']` was initially tainted. sink(y['b']) # Also an issue, `y` is entirely tainted. sink(y) ``` Note that whenever broadening happens, Pysa will automatically add a broadening [feature](pysa_features.md) on the taint flow, which can help discard false positives in post processing. Fine grained features are used for each different scenario leading to broadening. The most common causes for taint broadening are the following: ### Broadening on obscure models Taint that flows through an [obscure model](#obscure-models) - for instance, when Pysa does not have access to the body of the callee - is collapsed, since we must assume anything could get tainted, for correctness. In this scenario, the `tito-broadening` and `via:obscure:model` features are added to the flow. ### Broadening on taint-in-taint-out (TITO) When specifying a [taint propagation](pysa_basics.md#taint-propagation) (also called **Taint In Taint Out** or **TITO**) in a `.pysa` file, the propagation will collapse the taint by default. For instance: ```python # models.pysa def tito(arg: TaintInTaintOut): ... ``` ```python def foo(): x = {"a": source()} y = tito(x) sink(y['b']) # Considered an issue because of taint broadening. ``` In this scenario, the `tito-broadening` feature is added to the flow. If the function is known to preserve the structure of the argument, the `NoCollapse` annotation can be used to disable collapsing. For instance: ```python def tito(arg: TaintInTaintOut[NoCollapse]): ... ``` This would remove the issue from the previous example. Note that this can be used in combination with [`ParameterPath` and `ReturnPath`](#parameter-and-return-path). ### Model broadening When the number of tainted attributes or keys hits a certain threshold, taint broadening is applied to prevent the analysis from blowing up by tracking too many values. This is referred as **Model broadening** since this happens when the model (or summary) of a function is computed. For instance, this can happen when the number of tainted key-value pairs of a dictionary hit a certain threshold. For scalability reasons, Pysa cannot track an infinite amount of indices, and thus makes the approximation that the whole object is tainted. ```python def foo(condition): d = {} if condition: d["a"] = source() d["b"] = source() # c, d, e, etc. else: d["1"] = source() d["2"] = source() # etc. return d # too many indexes, the whole return value is considered tainted. ``` In this scenario, the `model-broadening` feature is added to the flow. See [analysis thresholds](#analysis-thresholds) for documentation about the different scenarios of model broadening. Note that model broadening can be disabled for a given function or method using the `@SkipModelBroadening` annotation in a `.pysa` file: ```python @SkipModelBroadening def foo(): ... ``` This can also be used in a `ModelQuery` using the [`Modes`](pysa_model_dsl.md##models-for-setting-modes) clause. Note that this should be used sparingly since this can potentially lead to an increase in analysis time. ### Model shaping When a specific attribute or key is tainted when the whole object is tainted with the same taint kind (e.g, `UserControlled`), taint collapsing is applied as an optimization to save analysis time. This is called **model shaping** and is applied right before **model broadening**. For instance: ```python def my_sink(x): sink(x) sink(x.foo) ``` The sink on `x.foo` (represented as `formal(x)[foo]`) is merged into the sink on `x`. Note that this is sound since attributes of a tainted object are also considered tainted. Thus if `my_sink` is actually called with `x.foo` tainted, the flow will be found as expected. The downside is that this can lead to false positives in cases where `my_sink` is called with another attribute (say `x.bar`) tainted. We would find a flow from `x.bar` to `x.foo` in `my_sink`. In this scenario, the `model-shaping` feature is added to the flow. The features `model-source-shaping`, `model-sink-shaping` and `model-tito-shaping` are also added to differentiate whether the shaping was on sources, sinks or tito. ### Widen broadening When the number of tainted attributes or keys is potentially infinite because of a loop or recursion, taint broadening is applied to allow the termination of the analysis. The term "widen" or "widening" refers to an operator that is applied to ensure convergence. It commonly happens within loops. For instance: ```python def foo(n): d = {} for _ in range(n): d = { "a": source(), "b": d, } return d ``` Technically, `d['b']...['b']['a']` (with an infinite number of access to `b`) could be tainted. To allow the analysis to terminate, Pysa stops at a certain depth. See [analysis thresholds](#analysis-thresholds) for documentation about the different scenarios of widen broadening. Another example: ```python def foo(person): while person.parent is not None: person = person.parent # Infer sinks on person.name, person.parent.name, person.parent.parent.name, etc. sink(person.name) ``` In these scenarios, the `widen-broadening` feature is added to the flow. ### Issue broadening When an object with a tainted attribute or key reaches a sink, Pysa considers the flow as valid even if the whole object is not tainted. For instance: ```python d = {"a": source(), "b": "foo"} sink(d) # `d` itself is not tainted, but `d["a"]` is, thus we emit an issue. ``` In this scenario, the `issue-broadening` feature is added to the issue. ## Tainting Specific `kwargs` Sometimes, a function can have potential sinks mixed together with benign parameters in the keyword arguments (`kwargs`) that it accepts. In these cases, tainting the whole `kwargs` variable will result in false positives when tainted data flows into a benign `kwarg`. Instead, for a function like this: ```python def eval_and_log(**kwargs): eval(kwargs["eval"]) logging.debug(kwargs["log"]) ``` We can lie a bit in our `.pysa` file, and break out the dangerous argument for tainting: ```python def eval_and_log(*, eval: TaintSink[RemoteCodeExecution], **kwargs): ... ``` This allows us to catch flows only into the `eval` keyword argument. ## Instance attributes versus class attributes Models can specify sources and sinks on attributes, following the type annotation syntax: ```python django.http.request.HttpRequest.GET: TaintSource[UserControlled] ``` Any access to `request.GET` will be tainted when `request` is an instance of `HttpRequest` or any of its children. However, note that the access to the class attribute (i.e, `HttpRequest.GET`) won't be considered tainted. To specify sources and sinks on class attributes, use the `__class__` prefix: ```python django.http.request.HttpRequest.__class__.GET: TaintSource[UserControlled] ``` To specify a source on both the class attribute and instance attribute, simply use both lines. ## Literal String Sources And Sinks Some security vulnerabilities are best captured by modeling strings of a given form flowing to dangerous functions, or format strings that match a pattern getting tainted data passed in. To mark all literal strings matching a pattern as sources, you first need to add a regular expression corresponding to the pattern to your `taint.config`: ```json { "sources": [ { "name": "IPAddress" } ], "implicit_sources": { "literal_strings": [ { "regexp": "\\d{1,3}(\\.\\d{1,3})+", "kind": "IPAddress", "description": "String that looks like an IP address." } ] } } ``` With this regex in place, whenever Pysa sees a string such as `123.456.789.123`, it will flag it as a taint source with the kind `IPAddress`. ```python def test() -> None: ip_address = "123.456.789.123" dont_pass_an_ip_address(ip_address) # Pysa will now flag this. ``` The converse of supporting literal strings as sinks is also supported, for data flowing into a tainted string. The syntax allows you to model data being used to format strings, like f-strings, manual string formatting, the string `format()` method, and printf-style string formatting with `%`. Template strings and manual string formatting with more than two subexpressions are not yet supported. To add a literal sink, first add the literal_sink to your configuration ```json { "sinks": [ { "name": "MayBeRendered" }, { "name": "MayBeSQL" } ], "implicit_sinks": { "literal_strings": [ { "regexp": "^<.*>$", "kind": "MayBeRendered", "description": "Indicates a string whose contents may be rendered." }, { "regexp": "^SELECT *.", "kind": "MayBeSQL", "description": "Indicates a string whose contents may be a SQL query." } ] } ``` Now, Pysa will treat any values flowing into a each of the following as a regular sink: ```python def may_render(parameter: str) -> None: result = f"" result = "".format(parameter) result = "" % (parameter,) ``` As well as values flowing into each of these as a regular sink: ```python def build_sql_query(columns: str) -> None: result = f"SELECT {columns} FROM users;" result = "SELECT {} FROM users;".format(columns) result = "SELECT %s FROM users" % (columns,) result = "SELECT " + columns + " FROM users;" ``` Note that string literal sinks have some limitations. For instance, they cannot catch issues if the string literal is stored in a temporary variable (e.g., `x = "SELECT {}"; x.format(input)`). We recommend using string combine rules instead (see below). ## Combined Source Rules Some security vulnerabilities are better modeled as *two* sources reaching sinks at the same call site. For example, leaking credentials via `requests.get` could be modeled as user controlled data flowing into the `url` parameter and credentials flowing into the `params` parameter. These flows can be modeled by *combined source rules*. Sources for combined source rules are declared as normal in `taint.config`. Sinks, however, are declared inside the rule definition, and are referred to as *partial sinks*. The rule itself is declared in the `combined_source_rules` top level entry. The actual flows are defined under section `rule`, which must contain two flows, one for each source. Each flow specifies a pair of (potentially multiple) sources and a single partial sink: ```json { "sources": [ { "name": "UserControlled" }, { "name": "Credentials" } ], "combined_source_rules": [ { "name": "Credentials leaked through requests", "rule": [ { "sources": [ "UserControlled" ], "partial_sink": "UserControlledRequestSink" }, { "sources": [ "Credentials" ], "partial_sink": "CredentialsSink" } ], "code": 1, "message_format": "Credentials leaked through requests", "main_trace_source": "url", } ] } ``` Sources are declared as normal in `.pysa` files. Instead of specifying sinks with a `TaintSink` annotation, however, `PartialSink` annotations are used to specify where each source needs to flow for the combined source rule. These `PartialSink` must reference the ones that were declared by the rule above: ```python def requests.api.get( url: PartialSink[UserControlledRequestSink], params: PartialSink[CredentialsSink], **kwargs ): ... ``` With the above configuration, Pysa can detect cases where `UserControlled` flows into `url` and `Credentials` flow into `params` *at the same time* (or at the same call site). Note that the same partial sink can be used in different rules, which avoids duplicating a given model for each rule (in some cases). ## String Combine Rules It is sometimes useful to detect data tainted with a source (e.g., `UserControlled` data) that is incorporated into a suspicious looking string (e.g., a string that looks like a SQL query). Detecting such a pattern is useful, because it allows Pysa to detect dynamic creation of SQL queries which could lead to SQL injection, even if the code under analysis is using a SQL framework that Pysa does not have models for. To detect such flows, one can specify a variant of [combined source rules](#combined-source-rules), called string combine rules, to detect when the suspicious string (identified via regex match) and the other configured source both flow into string formatting call sites (such as calling `str.__add__`, `str.__mod__`, `str.format` or constructing f-strings). For example, to detect flows from source `UserControlled` to sink `StringMayBeSQL`, one should specify the following contents in the taint configuration file, where `UserControlled` and `StringMayBeSQL` are both declared as sources: ```json { "sources": [ { "name": "UserControlled" }, { "name": "StringMayBeSQL" } ], "implicit_sources": { "literal_strings": [ { "regexp": "SELECT.*", "kind": "StringMayBeSQL", "comment": "matches a SQL statement" } ] }, "string_combine_rules": [ { "name": "User controlled data flows into potential SQL strings", "rule": [ { "sources": [ "UserControlled" ], "partial_sink": "UserControlledDataSink" }, { "sources": [ "StringMayBeSQL" ], "partial_sink": "StringMayBeSQLSink" } ], "code": 4324, "message_format": "User controlled data flows into potential SQL strings" } ] } ``` As shown above, the syntax is similar to that of [combined source rules](#combined-source-rules), especially for section `rule`. The above rule enables catching the following flows: ```python def issue(): uc = user_controlled() f"SELECT {uc} FROM async_query" "SELECT " + uc + " FROM async_query" "SELECT %s FROM async_query" % uc "SELECT {} FROM async_query".format(uc) ``` Note that the string combine rules are strictly more powerful than the feature of implicit literal string sinks. That is, any flow that can be detected via implicit literal string sinks can also be detected via string combine rules, but not vice versa. Hence, we recommend using the string combine rules. ## Prevent Inferring Models with `SkipAnalysis` In addition to the models defined in `.pysa` files, Pysa will infer models for functions based what sources, sinks, etc. they call in their body. The `SkipAnalysis` annotation can be used to prevent Pysa from inferring models, and instead force it to use only the user defined models for determining taint flow: ```python @SkipAnalysis def qualifier.dont_generate_models(argument): ... ``` `SkipAnalysis` can be applied at the class level as a shorthand to prevent pysa from infering models for all functions in a class: ```python class skip_analysis.SkipMe(SkipAnalysis): ... ``` ## Ignoring overrides When a method is called on a base class, Pysa has to assume that that call could actually invoke any subclass methods that override the base class's method. For heavily overriden methods, this can lead to both performance impacts and false positives. When running Pysa, you may see messages such as this in the output: ``` 2020-09-02 09:25:50,677 WARNING `object.__init__` has 106 overrides, this might slow down the analysis considerably. ``` The above message indicates that 106 subclasses of `object` have overridden `__init__`. If Pysa sees taint flowing into `object.__init__`, then it will treat all 106 overrides of `object.__init__` as also receiving that taint. The `@SkipOverrides` decorator can be applied to deal with false positives or performance issues from having too many overrides on a given function: ```python @SkipOverrides def object.__init__(self): ... ``` This annotation will cause Pysa not to propagate taint into to and from overridden methods on subclasses, when analyzing functions that call the overriden method on the base class. `maximum_overrides_to_analyze` can be added the the `options` block of `taint.config` to limit the number of overrides that Pysa will analyze: ```json { "sources": [], "sinks": [], "features": [], "rules": [], "options": { "maximum_overrides_to_analyze": 60 } } ``` This option can also be provided in the command line, using `--maximum-overrides-to-analyze`. This can speed up the analysis, but it will lead to false negatives, because Pysa will only propagate taint to or from 60 (in the case of the above example) overriden methods on subclasses. The remaining overriding methods will be ignored and treated as if they weren't actually overriding the base class method. By default, Pysa skips overrides on some functions that are typically problematic. You can find the full list of default-skipped functions in [`stubs/taint/common/skipped_overrides.pysa`](https://github.com/facebook/pyre-check/blob/main/stubs/taint/common/skipped_overrides.pysa) ## Force to analyze all overrides We also allow the use of decorator `@AnalyzeAllOverrides` to force analyzing all overriding methods of a given method, regardless of the configured maximum number of overrides to analyze (e.g., via command line option `--maximum-overrides-to-analyze`), or if there simultaneously exists an `@SkipOverrides` on the given method. An example is: ```python @AnalyzeAllOverrides def BaseClass.method(self): ... ``` Decorator `@AnalyzeAllOverrides` is often used to reduce false negatives, by analyzing all overrides of some selected methods. This offers a more fine-grained option than tweaking `--maximum-overrides-to-analyze` for all methods. Adding decorator `@AnalyzeAllOverrides` to some selected methods is faster than using a large threshold of maximum overrides for all methods, but achieves better precision than using a small threshold. ## Force to treat a callable as being called when passed as parameters We allow the usage of decorator `@CalledWhenParameter` to force treating the callable as being called, when the callable is passed as a parameter. An example is: ```python @CalledWhenParameter def foo(): ... def bar(f): f() def main(): bar(foo) # We would treat `foo` as being called here. ``` This is useful for reducing false negatives. By default, the higher order call graph building would always create a parameterized callable `bar[f=foo]` (i.e., a version of `bar` where `f` is instantiated as `foo`) at the call site in `main`. Hence, `foo` would no longer get called if the higher order call graph building is limited in knowing that `foo` is eventually called in `bar`. This may lead to false negatives. ## Skip inlining higher order functions with `@SkipInliningHigherOrderFunctions` By default, when Pysa detects a higher order function (i.e., a function that takes another function as a parameter and calls it), it creates **parameterized targets** — specialized versions of the function for each concrete callable passed as an argument. This allows Pysa to inline the callee and track taint through the higher order call precisely. For example, given: ```python def apply(f, x): return f(x) def goes_to_sink(arg): sink(arg) apply(goes_to_sink, source()) # Pysa creates apply[f=goes_to_sink] and finds the issue. ``` In some cases, this specialization is undesirable — for instance, if a function is called with many different callables, creating parameterized targets for each one can increase analysis time without providing useful results. The `@SkipInliningHigherOrderFunctions` annotation prevents Pysa from creating parameterized targets for a given function: ```python @SkipInliningHigherOrderFunctions def module.apply(f, x): ... ``` With this annotation, Pysa will not inline the higher order parameter `f` at call sites of `module.apply`. Instead, at a call site such as `apply(goes_to_sink, source())`, Pysa will treat `goes_to_sink` as being called directly at that call site (i.e., as if `goes_to_sink(source())` was called), rather than creating a specialized version `apply[f=goes_to_sink]`. See [Force to treat a callable as being called when passed as parameters](#force-to-treat-a-callable-as-being-called-when-passed-as-parameters) for more details on this behavior. This is useful for reducing analysis cost on functions that are called with many different callables, when the resulting precision is not needed. ## Skip calling higher order parameters with `@SkipCallHigherOrderFunctions` The `@SkipCallHigherOrderFunctions` annotation prevents Pysa from recording the implicit call to higher order parameters at the call site. Unlike `@SkipInliningHigherOrderFunctions`, it does **not** prevent the creation of parameterized targets — the higher order call graph fixpoint still inlines the function normally when it can. For example, given: ```python @SkipCallHigherOrderFunctions def module.apply(f, x): ... ``` At a call site such as `apply(goes_to_sink, source())`, Pysa will: - Still create a specialized version `apply[f=goes_to_sink]` and analyze it, if the function has an available body (i.e, not a stub). - **Not** treat `goes_to_sink` as being called directly at the call site (i.e., the implicit call from the `higher_order_parameters` is removed). This is useful when the implicit call recording causes false positives — for instance, when the implicit call is redundant because the parameterized target analysis already provides the correct result. In most cases, you will want to use both annotations together: ```python @SkipInliningHigherOrderFunctions @SkipCallHigherOrderFunctions def module.apply(f, x): ... ``` This both prevents parameterized target creation and removes the implicit call, effectively disabling all higher order function handling for this function. ## Limit the trace length for better signal and performance By default, Pysa will find all flows from sources to sinks matching a rule. This can lead to very long traces which are hard to understand and tend to be false positives. This also brings down the performance a lot. Pysa provides a `--maximum-trace-length ` command line argument which limits the length of traces that it finds. In general, this will also make Pysa faster. This option can also be added in the `taint.config` as follows: ```json { "sources": [], "sinks": [], "features": [], "rules": [], "options": { "maximum_trace_length": 20 } } ``` Note that this is not a silver bullet and that this might hide security vulnerabilities. Use it with caution. ## Limit the trace length for a given rule Similarly to the option described above, one can limit the trace length for a given rule, using the `filters` option: ``` "rules": [ { "name": "SQL injection.", "code": 1, "sources": [ "UserControlled" ], "sinks": [ "SQL" ], "message_format": "Data from [{$sources}] source(s) may reach [{$sinks}] sink(s)", "filters": { "maximum_source_distance": 10, "maximum_sink_distance": 5 } } ] ``` This will limit the trace length from the root to the source by 10, and the trace length from the root to the sink by 5, only for that specific rule. **Note**: This is meant to be used to limit the number of issues written to the database. Prefer using [SAPP](https://github.com/facebook/sapp#readme) to filter out false positives. ## Limit the tito depth for better signal and performance Pysa automatically infers when a function propagate the taint from one argument to its return value. This is called tito, for "Taint In Taint Out". In practice, infering it can be very expensive since the taint can go through an arbitrary number of hops (i.e, depth). For instance: ```python def foo(x): return x def bar(x): return foo(x) def baz(x): return bar(x) ``` In this example, `baz` propagates the taint on its argument to the return value using 3 hops. Pysa provides a `--maximum-tito-depth ` command line argument which limints the depth of inferred propagations. In combination with the trace length limit, this usually makes Pysa faster. This option can also be added in the `taint.config` as follows: ```json { "sources": [], "sinks": [], "features": [], "rules": [], "options": { "maximum_tito_depth": 20 } } ``` ## Decorators By default, Pysa does not generally understand decorators, and will treat a call to a decorated function as an obscure call. This will usually lead to false negatives if the decorated functions has sources or sinks. For instance: ```python def identity(f: Callable[[str], None]) -> Callable[[str], None]: return f @identity def decorated_sink(x: str) -> None: sink(x) decorated_sink(source()) # False negative, issue will NOT be found! ``` This also leads to false negatives if the decorator adds a flow to a sink. For instance: ```python def with_sink(f: Callable[[str], None]) -> Callable[[str], None]: def inner(x: str) -> None: sink(x) f(x) return inner @with_sink def foo(x: str) -> None: print(x) foo(source()) # False negative, issue will NOT be found! ``` Since the call to a decorated function is treated as an obscure call, it will conservatively propagate taint through decorated function: ```python def identity(f: Callable[[str], str]) -> Callable[[str], str]: return f @identity def decorated(x: str) -> str: # Whatever happens here will not be considered at the call site. return 'hello %s' % x sink(decorated(source())) # Issue is properly found. ``` Pysa provides a few ways to deal with these limitations. ## Ignoring decorators Pysa can entirely ignore a decorator, as if it was not present in the source code. This can be done safely when the decorator does not change the signature of the decorated function (i.e, it does not add or remove parameters). To ignore a decorator, use the `@IgnoreDecorator` annotation in a `.pysa` file: ```python @IgnoreDecorator def module.decorator(): ... ``` ## Single trace sanitizers with `@SanitizeSingleTrace` Sanitizers, as described in the [Overview](pysa_basics.md), are applied in both the forward (i.e source) trace and backward (i.e sink) trace. For instance, with the given `.pysa` file: ```python @Sanitize(TaintInTaintOut[TaintSink[RemoteCodeExecution]]) def shlex.quote(x): ... ``` And the following Python code: ```python import subprocess from shlex import quote def quoted_input(): x = input() # source 'UserControlled' y = quote(x) return y def echo(argument): subprocess.run(f'/bin/echo {argument}', shell=True) # sink 'RemoteCodeExecution' def issue(): x = quoted_input() # source trace: input -> quoted_input -> issue echo(x) # sink trace: issue -> echo -> subprocess.run ``` Pysa will NOT find an issue here, as expected. This is because during the propagation of the 'UserControlled' source in the forward trace, pysa remembers that it was sanitized for the sink 'RemoteCodeExecution'. However, Pysa provides a simpler version of sanitizers, which only sanitizes in the forward trace or the backward trace: ```python @SanitizeSingleTrace(TaintSource) def f(): ... @SanitizeSingleTrace(TaintSource[UserControlled]) def g(): ... @SanitizeSingleTrace(TaintSink) def h(): ... @SanitizeSingleTrace(TaintSink[RemoteCodeExecution]) def i(): ... ``` These sanitizers are a lot cheaper and could save analysis time. However, these might introduce false positives, so we recommend to use the default sanitizers. ## Filtering the call graph with `@Entrypoint` By default, Pysa will analyze the entire call graph of your program. This can lead to longer analysis times for larger programs, especially when you'd only like to perform analysis on specific parts of the program. This decorator will mark a specified function and the functions it calls as the only functions to be analyzed. Note: the flag `--limit-entrypoints` must be passed to `pyre analyze` for call graph filtering to occur, even if the `@Entrypoint` decorator is present. This allows for call graph filtering to be easily enabled or disabled without editing your `.pysa` files. If you have the following Python file: ```python class MyClass: def class_entrypoint(): taint_sink(taint_source()) def my_bad_func_1(): taint_sink(taint_source()) def my_bad_func_2(): taint_sink(taint_source()) def func_entrypoint(): my_bad_func_1() def main(): func_entrypoint() my_bad_func_2() MyClass().class_entrypoint() main() ``` And the following `.pysa` file: ```python @Entrypoint def my_file.MyClass.class_entrypoint(): ... @Entrypoint def func_entrypoint(): ... ``` Then issues will be found for taint in calls to `class_entrypoint` and `my_bad_func_1`, but not `my_bad_func_2`, since it isn't called by a function marked by an `@Entrypoint`. ## Add a feature to all local variables on function call In some situations, one might want to filter out issues where a specific function was called, but that call is not directly involved in the data flow. For instance: ```python x = some_source() if security_check(): some_sink(x) ``` The problem here is that `security_check()` does not transfer taint. Sanitizing this function would have no effect, Pysa will still find the issue. A solution is to use `@AddFeatureToState`: ``` @AddFeatureToState(Via[security_check]) def my_module.security_check(): ... ``` This will add the feature `via:security_check` to all local variables when calling `security_check()`. In the example above, the issue from `some_source()` to `some_sink()` will therefore have that feature, and it can be filtered out in post processing (see [features](pysa_features.md)). Note that `AddFeatureToState` is not contagious (i.e it is not propagated up): ``` def security_check_wrapper(): return security_check() x = some_source() if security_check_wrapper(): some_sink(x) ``` The issue won't have the feature `via:security_check` in that case. Note also that `AddBreadcrumbToState` is an alias for `AddFeatureToState`. ## Taint In Taint Out Transforms Taint in taint out transforms can be used to capture more precise flows. As an example: ```python def read_file(path): with open(path, "r") as f: content = f.read() return content ``` Without taint in taint transforms we can write a rule that captures a `UserControlled` path is `read`. Such a rule can be made much higher signal if we can detect that `content` is also `ReturnedToUser`. We can use taint in taint out transforms to stitch the two flows together. We mark `read` with a taint in taint out transform `FileRead`, and the rule becomes `UserControlled -> FileRead -> ReturnedToUser`. To contrast with feature annotations, there are two differences: * The filtering is done during analysis itself, and limits the issues generated (as opposed to a post-processing step by the user) * Taint in taint out transforms can be used to reason about the order of events ### Syntax In `taint.config`, one can specify `transforms` to define new transforms. Each transform is defined by following fields: * `name`: name of the transform, this is used when defining rules, as well as writing models * `comment`: description of the transform ``` { ... "transforms": [ { "name": "MyTransform", "comment": "This is my transform" }, ... ], ... } ``` Then, one may use these transforms in `rules` as follows: ``` { ... "rules": [ { "name": ..., "code": ..., "sources": ["SourceA"], "transforms": ["MyTransform1", "MyTransform2"], "sinks": ["SinkB"], "message_format": "[{$sources}] transformed by [${transforms}] may reach [${sinks}]" }, ... ], ... } ``` Intuitively, one can think of the rule above as `SourceA -> MyTransform1 -> MyTransform2 -> SinkB`. The order is important. Finally, in `.pysa` model files a taint transform can be specified using a `TaintInTaintOut[Transform[...]]` annotation, where the parameter is the name of the transform. ``` def my_function(arg: TaintInTaintOut[Transform[MyTransform]]): ... ``` ### Semantics ``` y = my_function(x) ``` If `x` has source taint `SourceA`, the taint of `y` is `MyTransform:SourceA`. This will correspond to matching `SourceA -> MyTransform` in a rule. Likewise, if `y` has sink taint `SinkB`, then the taint of `x` is `MyTransorm:SinkB`. This will correspond to matching `MyTransform -> SinkB` in a rule. Note that a transform modifies the taint itself. Hence, if a flow passes through a transform, it will no longer match rules which do not contain the transform. ``` RuleX: SourceA -> SinkB RuleY: SourceA -> MyTransform -> SinkB Flow1: SourceA -> SinkB Flow2: SourceA -> MyTransform -> SinkB ``` `Flow1` matches `RuleX` but not `RuleY`. `Flow2` matches `RuleY` but not `RuleX`. Consider the scenario where we have an additional rule: ``` RuleZ: SourceC -> SinkD ``` If transform `MyTransform` is applied to taint `SourceC`, there is no possible rule it can possibly match. As an optimization, we check for this continuously in our analysis and filter out eagerly. Also note that the existing TaintInTaintOut annotation semantics of TITO being assumed (instead of inferred) on the argument are unchanged. ## Analysis thresholds Pysa provides many options to fine tune the taint analysis. The following options can be provided either via the command line or in the `taint.config` file, under the `options` section. For instance: ```json { "sources": [], "sinks": [], "features": [], "rules": [], "options": { "maximum_model_source_tree_width": 10, "maximum_model_sink_tree_width": 10, "maximum_model_tito_tree_width": 10 } } ``` When not provided, these are set to the following defaults: ```ocaml file=source/interprocedural_analyses/taint/taintConfiguration.ml start=DOCUMENTATION_CONFIGURATION_START end=DOCUMENTATION_CONFIGURATION_END ``` ### Maximum model source tree width * Command line option: `--maximum-model-source-tree-width` * taint.config option: `maximum_model_source_tree_width` See [taint broadening](#taint-broadening) and [model broadening](#model-broadening). This limits the width of the source tree in the model for a callable, i.e the number of output paths in the return value. For instance: ```python def foo(): return {"a": source(), "b": source(), "c": source()} ``` The source tree for `foo` has a width of 3. Above the provided threshold, pysa will collapse the taint and consider the whole dictionary tainted. When that happens, the features `model-broadening` and `model-source-broadening` will be added to the flow. ### Maximum model sink tree width * Command line option: `--maximum-model-sink-tree-width` * taint.config option: `maximum_model_sink_tree_width` See [taint broadening](#taint-broadening) and [model broadening](#model-broadening). This limits the width of the sink tree in the model for a callable, i.e the number of input paths leading to a sink for a given parameter. For instance: ```python def foo(arg): sink(arg[1]) sink(arg[2]) sink(arg[3]) ``` The sink tree for `foo` and parameter `arg` has a width of 3. Above the provided threshold, pysa will collapse the taint and consider that the whole argument leads to a sink. When that happens, the features `model-broadening` and `model-sink-broadening` will be added to the flow. ### Maximum model tito tree width * Command line option: `--maximum-model-tito-tree-width` * taint.config option: `maximum_model_tito_tree_width` See [taint broadening](#taint-broadening) and [model broadening](#model-broadening). This limits the width of the taint-in-taint-out tree in the model for a callable, i.e the number of input paths propagated to the return value, for a given parameter. For instance: ```python def foo(arg): return '%s:%s:%s' % (arg.a, arg.b, arg.c) ``` The taint-in-taint-out tree for `foo` and parameter `arg` has a width of 3. Above the provided threshold, pysa will collapse the taint and consider that the taint on the whole argument is propagated to the return value. When that happens, the features `model-broadening` and `model-tito-broadening` will be added to the flow. ### Maximum tree depth after widening * Command line option: `--maximum-tree-depth-after-widening` * taint.config option: `maximum_tree_depth_after_widening` See [taint broadening](#taint-broadening) and [widen broadening](#widen-broadening). This limits the depth of the source, sink and tito trees within loops, i.e the length of source, sink and tito paths for each variables. For instance: ```python def foo(): variable = MyClass() for x in generate(): variable.a.b.c = source() return result ``` The source tree for `variable` has a depth of 3 (i.e, `a` -> `b` -> `c`). Within a loop, pysa limits the depth to the provided threshold. For instance, if that threshold is 1, we would consider that `variable.a` is entirely tainted. When that happens, the feature `widen-broadening` will be added to the flow. ### Maximum return access path width * Command line option: `--maximum-return-access-path-width` * taint.config option: `maximum_return_access_path_width` See [taint broadening](#taint-broadening) and [model broadening](#model-broadening). This limits the width of the return access path tree in the model for a callable, i.e the number of output paths propagated to the return value, for a given parameter. For instance: ```python def foo(arg): return {'a': arg, 'b': arg, 'c': arg} ``` The return access path tree for `foo` and parameter `arg` has a width of 3. Above the provided threshold, pysa will collapse the taint and consider that the whole return value is tainted whenever `arg` is tainted. When that happens, the features `model-broadening` and `model-tito-broadering` will be added to the flow. ### Maximum return access path depth after widening * Command line option: `--maximum-return-access-path-depth-after-widening` * taint.config option: `maximum_return_access_path_depth_after_widening` See [taint broadening](#taint-broadening) and [widen broadening](#widen-broadening). This limits the depth of the return access path tree within loops, i.e the length of output paths propagated to the return value, for a given parameter. For instance: ```python def foo(arg): result = MyClass() for x in generate(): result.a.b.c = arg return result ``` The return access path tree for `foo` and parameter `arg` has a depth of 3 (i.e, `a` -> `b` -> `c`). Within a loop, pysa limits the depth to the provided threshold. For instance, if that threshold is 2, we would cut the output path to just `a.b`. When that happens, the feature `model-broadening` and `model-tito-broadening` will be added to the flow. ### Maximum tito collapse depth * Command line option: `--maximum-tito-collapse-depth` * taint.config option: `maximum_tito_collapse_depth` This limits the depth of the taint tree after applying taint-in-taint-out, i.e the length of paths for taint propagated from a parameter to the return value. For instance: ```python def identity(arg): return arg def foo(): input = {'a': {'b': {'c': source()}}} output = identity(input) ``` The taint tree for `input` has a depth of 3 (i.e, `a` -> `b` -> `c`). When the taint is propagated to the return value of `identity`, we limit the resulting taint tree to the given depth. For instance, if that threshold is 1, we would consider that `output['a']` is tainted. This is also applied for sinks in the backward analysis: ```python def foo(arg): output = identity(arg) sink(output['a']['b']['c']) ``` With a threshold of 1, we would consider that `output['a']` leads to a sink. ### Maximum tito positions * Command line option: `--maximum-tito-positions` * taint.config option: `maximum_tito_positions` This limits the number of positions to keep track of when propagating taint. When taint is propagated through a function and returned (i.e, taint-in-taint-out), pysa will keep track of the position of the argument, and display it in the trace. For instance: ```python def foo(): x = source() y = tito(x) ^ z = {"a": y} ^ sink(z) ``` In this example, we have 2 tito positions. Above the provided threshold, pysa simply discards all positions. Note that the taint is still propagated. ## Partial Flows SAPP supports the concept of partial flows, which helps with triaging issues from different rules with a common flow. Details are described [here](https://github.com/facebook/sapp#readme). To leverage this capability in an end-to-end run (i.e., first running Pysa and then SAPP), you can add the following section to your `taint.config` file, so that they can be passed into the Pysa output files and then be picked up by SAPP: ```json "partial_flows": [ { "full_issue_code": 4101, "partial_issue_code": 4100, "full_issue_transform": "TransformB", "is_prefix_flow": true, "feature": "some_feature" } ] ``` In other words, the above configuration is *not* used by Pysa at all. Pysa simply passes them into the output files so that SAPP can read them. --- ### Documentation/Website/Docs/Pysa Basics --- id: pysa-basics title: Pysa (Static Analysis for Security) sidebar_label: Overview --- import Internal from './fb/pysa_basics_internal.md'; import YouTube from 'react-youtube'; import useBaseUrl from '@docusaurus/useBaseUrl'; Pysa Logo Pyre has applications beyond type checking python code: it can also run static analysis, more specifically called **Taint Analysis**, to identify potential security issues. The Python Static Analyzer feature of Pyre is usually abbreviated to Pysa (pronounced like the Leaning Tower of Pisa). ## Taint Analysis **Tainted data** is data that must be treated carefully. Pysa works by tracking flows of data from where they originate (sources) to where they terminate in a dangerous location (sinks). For example, we might use it to track flows where user-controllable request data flows into an `eval` call, leading to a remote code execution vulnerability. This analysis is made possible by user-created models which provide annotations on source code, as well as rules that define which sources are dangerous for which sinks. Pysa comes with many pre-written models and rules for builtin and common python libraries. Pysa propagates taint as operations are performed on tainted data. For example, if we start with a tainted integer and perform a number of operations on it, the end results will still be tainted: ```python x = some_function_that_returns_a_tainted_value() # 'x' is marked as tainted y = x + 10 s = str(x) f = f"Value = {s}" # 'f' is marked with the same taint 'x' had ``` Pysa will only analyze the code in the repo that it runs on, as well as code in directories listed in the `search_path` of your [`.pyre_configuration`](configuration.md) file. It does not see the source of your dependencies. **Just because** ***you*** **can see code in your editor does not mean Pysa has access to that code during analysis.** Because of this limitation, Pysa makes some simplifying assumptions. If taint flows into a function Pysa doesn't have the source for, it will assume that the return type of that function has the same taint. This helps prevents false negatives, but can also lead to false positives. When an object is tainted, that means that all attributes of that object are also tainted. Note that this is another source of potential false positives, such as taint flows that include `some_obj.__class__`. This means that Pysa will detect all of the following flows: ```python x = some_source() # 'x' is marked as tainted some_sink(x) # This is detected some_sink(x.some_attribute) # This is also detected some_sink(x.__class__) # This is (unfortunately) also detected ``` ## Pysa Configuration Pysa uses two types of files for configuration: a single `taint.config` file, and an unlimited number of files with a `.pysa` extension. The `taint.config` file is a JSON document which stores definitions for *sources*, *sinks*, *features*, and *rules* (discussed below). The `.pysa` files are model files (also discussed below) which annotate your code with the *sources*, *sinks*, and *features* defined in your `taint.config` file. Examples of these files can be found in the [Pyre repository](https://github.com/facebook/pyre-check/tree/main/stubs/taint). These files live in the directory configured by `taint_models_path` in your `.pyre_configuration` file. Any `.pysa` file found in this folder will be parsed by Pysa and the models will be used during the analysis. ## Sources Sources are where tainted data originates. They are declared in your `taint.config` file like this: ```json "sources": [ { "name": "Cookies", "comment": "used to annotate cookie sources" } ] ``` Models that indicate what is a source are then defined in `.pysa` files. Sources are declared with the same syntax as [type annotations in Python 3](https://docs.python.org/3/library/typing.html). Function return types, class/model attributes, and even entire classes can be declared as sources by adding `TaintSource[SOURCE_NAME]` wherever you would add a python type: ```python # Function return source def django.http.request.HttpRequest.get_signed_cookie( self, key, default=..., salt=..., max_age=... ) -> TaintSource[Cookies]: ... # Class attribute source: django.http.request.HttpRequest.COOKIES: TaintSource[Cookies] ``` When tainting an entire class, any return from a method or access of an attribute of the class will count as a returning tainted data. The specifics of these model files are discussed further in the Models section. ```python # Class source: class BaseException(TaintSource[Exception]): ... ``` When tainting indexable return types such as `Dict`s, `List`s, and `Tuple`s, the `ReturnPath` syntax can be used to only mark a portion of the return type as tainted: ```python def applies_to_index.only_applies_to_nested() -> TaintSource[Test, ReturnPath[_[0][1]]]: ... def applies_to_index.only_applies_to_a_key() -> TaintSource[Test, ReturnPath[_["a"]]]: ... ``` Note that `ReturnPath` syntax can also be applied to fields of classes and globals, which can be particularly helpful when annotating dictionaries. ```python # Source file: a.py class C: dictionary_field = {"text": "will_be_tainted"} # Model file: models.pysa a.C.dictionary_field: TaintSource[Test, ReturnPath[_["text"]]] ``` See [Parameter and Return Path](pysa_advanced.md#parameter-and-return-path) for additional information. ## Sinks Sinks are where tainted data terminates. They are declared in your `taint.config` file like this: ```json "sinks": [ { "name": "SQL", "comment": "use to annotate places of SQL injection risk" } ] ``` Models that indicate what is a sink are then defined in `.pysa` files. Sinks can be added to the same files as sources. Like sources, sinks are declared with the same syntax as [type annotations in Python 3](https://docs.python.org/3/library/typing.html). Function parameters, class attributes, and even whole classes can be declared as sinks by adding `TaintSink[SINK_NAME]` where you would add a python type: ```python # Function parameter sink def sqlite3.dbapi2.Cursor.execute(self, sql: TaintSink[SQL], parameters): ... # Attribute sink file_name.ClassName.attribute_name: TaintSink[RemoteCodeExecution] ``` When tainting an entire class, any flow into a method or attribute of the class will count as a flow to a taint sink. The specifics of these model files are discussed further in the Models section. ```python # Entire class sink class BaseException(TaintSink[Logging]): ... ``` ### Implicit Sinks Implicit sinks are program expressions that we want to act as sinks, but that cannot be specified via taint signatures in `.pysa` files. Currently, only conditional tests are supported as implicit sinks. This allows writing rules that track whether a particular source is used in a conditional test expression. ```json "implicit_sinks": { "conditional_test": [ ] } ``` ## Rules Rules declare which flows from source to sink we are concerned about. They are declared in your `taint.config` file like this: ```json "rules": [ { "name": "SQL injection.", "code": 1, "sources": [ "UserControlled" ], "sinks": [ "SQL" ], "message_format": "Data from [{$sources}] source(s) may reach [{$sinks}] sink(s)" } ] ``` Each rule needs a brief `name` that explains its purpose and a *unique* `code`. The rule must define a list of one or more `sources`, which we are concerned about flowing into one or more `sinks`. `message_format` can further explain the issue. When a flow is detected the `{$sources}` and `{$sinks}` variables will be replaced with the name of the specific source(s) and sink(s) that were involved in the detected flow. ## Sanitizers Sanitizers break a taint flow by removing taint from data. Models that indicate sanitizing functions are defined in `.pysa` files. Sanitizers can be added to the same files as sources and sinks. Functions are declared as sanitizers by adding a special decorator: ```python # This will remove any taint passing through a function, regardless of whether # it is a taint source returned by this function, taint reaching sinks within # the function via 'text', or taint propagateing through 'text' to the # return value. @Sanitize def django.utils.html.escape(text): ... ``` This annotation is useful in the case of explicit sanitizers such as `escape`, which helps prevent cross site scripting (XSS) by escaping HTML characters. The annotation is also useful, however, in cases where a function is not intended to sanitize inputs, but is known to always return safe data despite touching tainted data. One such example could be `hmac.digest(key, msg, digest)`, which returns sufficiently unpredictable data that the output should no longer be considered attacker-controlled after passing through. Sanitizers can also be scoped to only remove taint returned by a function, passing through a specific argument, or passing through all arguments. ```python # This will remove any taint returned by this function, but allow taint # to be passed in to the function via 'argument'. It also prevents taint # from propagating from any argument to the return value. def module.sanitize_return(argument) -> Sanitize: ... # This prevents any taint which passes through 'argument' from reaching a sink within # the function, but allows taint which originates within the function to be returned. def module.sanitize_parameter(argument: Sanitize): ... # This prevents any taint which passes through any parameter from entering the function, # but allows taint which originates within the function to be returned. It also prevents # taint from propagating from any argument to the return value. @Sanitize(Parameters) def module.sanitize_all_parameters(): ... # This will remove any taint which propagates through any argument to the return # value, but allow taint sources to be returned from the function as well as # allow taint to reach sinks within the function via any argument. @Sanitize(TaintInTaintOut) def module.sanitize_tito(a, b, c): ... # Same as before, but only for parameter 'b' def module.sanitize_tito_b(a, b: Sanitize[TaintInTaintOut], c): ... ``` Pysa also supports only sanitizing specific sources or sinks to ensure that the sanitizers used for a rule don't have adverse effects on other rules. The syntax used is identical to how taint sources and sinks are specified normally: ```python # Sanitizes only the `UserControlled` source kind. def module.return_not_user_controlled() -> Sanitize[TaintSource[UserControlled]]: ... # Sanitizes both the `SQL` and `Logging` sinks. def module.sanitizes_sql_and_logging_sinks( flows_to_sql: Sanitize[TaintSink[SQL]], logged_parameter: Sanitize[TaintSink[Logging]], ): ... ``` For taint-in-taint-out (TITO) sanitizers, Pysa supports only sanitizing specific sources and sinks through TITO: ```python # With this annotation, whenever `escape(data)` is called, the UserControlled taint of `data` # will be sanitized, whereas other taint that might be present on `data` will be preserved. @Sanitize(TaintInTaintOut[TaintSource[UserControlled]]) def django.utils.html.escape(text): ... @Sanitize(TaintInTaintOut[TaintSink[SQL, Logging]]) def module.sanitize_for_logging_and_sql(): ... ``` Note that you can use any combination of annotations, i.e sanitizing specific sources or specific sinks, on the return value, a specific parameter or all parameters: ```python file=source/interprocedural_analyses/taint/test/integration/sanitize.py.pysa start=DOCUMENTATION_RETURN_SANITIZERS_START end=DOCUMENTATION_RETURN_SANITIZERS_END ``` ```python file=source/interprocedural_analyses/taint/test/integration/sanitize.py.pysa start=DOCUMENTATION_PARAMETER_SPECIFIC_SANITIZERS_START end=DOCUMENTATION_PARAMETER_SPECIFIC_SANITIZERS_END ``` ```python file=source/interprocedural_analyses/taint/test/integration/sanitize.py.pysa start=DOCUMENTATION_PARAMETERS_SANITIZERS_START end=DOCUMENTATION_PARAMETERS_SANITIZERS_END ``` Attributes can also be marked as sanitizers to remove all taint passing through them: ```python django.http.request.HttpRequest.GET: Sanitize ``` Sanitizing specific sources and sinks can also be used with attributes: ```python def module.Node.id: Sanitize[TaintSource[UserSecrets]] = ... def module.Node.id: Sanitize[TaintSink[Logging]] = ... ``` Note that sanitizers come with the risk of losing legitimate taint flows. They remove all taint and aren't restricted to a specific rule or individual source to sink flows. This means you need to ensure you aren't potentially affecting other flows when you add a sanitizer for a flow you care about. For this reason, some of the above sanitizer examples might not be a good idea to use. For example, if you are trying to track flows where SQL injection occurs, the `escape` sanitizer removing all taint kinds would prevent you from seeing any flows where data going into your SQL query happened to be HTML escaped. The best practice with sanitizers, then, is to make them as specific as possible. It's recommended to sanitize specific sources and sinks over using the general `@Sanitize`, `-> Sanitize` or `: Sanitize` annotations. ### TITO Sanitizers vs Source/Sink Sanitizers Source/Sink sanitizers are used to sanitize functions belonging to the source/sink trace. Example ```python def render_string_safe(string: str): safe_strings_list = ["safe", "string", "list"] if string in safe_strings_list: return render(string) def render_input_view(request: HttpRequest): user_input = request.GET["user_input"] return render_string_safe(user_input) ``` Without any sanitizer this code would raise a pysa issue since the UserControlled input is flowing into the `render` function (imagining that the `render` function is an XSS sink). To avoid this we can create a model: ```python def render_string_safe(string: Sanitize[TaintSink[XSS]]): ... ``` This will instruct pysa to remove the XSS taint on the string parameter in this way even if we have a XSS sink (`render`) inside the `render_string_safe` function we will not trigger an issue. TITO Sanitizers instead are used to remove the taint when tainted value is flowing into (TaintIn) a function as a parameter and then it is returned (TaintOut) by the same function. ```python def sanitize_string(string: str): return re.sub('[^0-9a-z]+', '*', string) def render_input_view(request: HttpRequest): user_input = request.GET["user_input"] safe_str = sanitize_string(user_input) return render(safe_str) ``` Like in the example before this code would generate a Pysa XSS issue. To avoid this we can create a model: ```python def sanitize_string(string: Sanitize[TaintInTaintOut[TaintSink[XSS]]]): ... ``` This will instruct pysa to remove the XSS taint from the value returned by the `sanitize_string` when a tainted value is passed as `string` parameter to the `sanitize_string` function. ## Taint Propagation Sometimes, Pysa is unable to infer that tainted data provided as an argument to a function will be returned by that function. In such cases, Pysa models can be annotated with `TaintInTaintOut[LocalReturn]` to encode this information for the analysis. This annotation can be applied to any parameter, including `self`, and is useful in scenarios such as when retrieving a value from a collection containting tainted data: ```python # This tells Pysa that if a 'dict' contains tainted data, the result # of calling 'get' on that dict will also contain tainted data def dict.get(self: TaintInTaintOut[LocalReturn], key, default): ... ``` Note that `TaintInTaintOut` (ie. without square brackets) is also accepted and can be used as a short hand for `TaintInTaintOut[LocalReturn]`. `LocalReturn` is only ever *required* when using the `Updates` syntax below and wanting to preserve the `LocalReturn` behaviour. For performance reasons, Pysa does not keep track of when functions place taint into their parameters, such as when a function adds a tainted entry to a list it received (with some notable exceptions for taint assigned to `self` in a constructor or property). The `TaintInTaintOut[Updates[PARAMETER]]` annotation can be used to work around Pysa's limitations by telling Pysa that taint will flow the the annotated parameter into the parameter named `PARAMETER`: ```python # This tells Pysa that if 'dict.update' is called with tainted data, # then the 'self' object (ie. the dictionary itself) should then be # considered tainted. def dict.update(self, __m: TaintInTaintOut[Updates[self]]): ... ``` Note that **constructors** and **property setters** are treated as if they were returning `self`. This means you should use `LocalReturn` instead of `Updates[self]` when writing models those. For instance: ```python def MyClass.__init__(self, argument: TaintInTaintOut[LocalReturn]): ... @foo.setter def MyClass.foo(self, value: TaintInTaintOut[LocalReturn]): ... ``` [Feature annotations](pysa_features.md) may also be placed inside the `[]` blocks of `TaintInTaintOut[...]` annotations. ## Features Feature annotations are also placed in your `taint.config` and `.pysa` files. This is a larger topic and will be covered in detail on [its own page](pysa_features.md). ## Model files ### Usage By default, Pysa computes an inferred model for each function and combines it with any declared models in `.pysa` files (of which there can be more than one). The union of these models and their annotations will be used. For example, cookies are both user controlled and potentially sensitive to log, and Pysa allows us apply two different annotations to them: ```python django.http.request.HttpRequest.COOKIES: TaintSource[UserControlled] django.http.request.HttpRequest.COOKIES: TaintSource[Cookies] ``` ### Requirements and Features #### Fully qualified names Any declarations in `.pysa` files must use the fully qualified name for the function/attribute they are attempting to annotate. You can usually find the fully qualified name for a type by looking at how it is imported, however, it's important to note that fully qualified names correspond to where something is declared, not necessarily where it is imported from. For example, you can import `HttpRequest` directly from the `django.http` module, even though it is defined in `django.http.request`. If you wanted to taint an attribute of `HttpRequest`, you would need to use the module in which it was defined: ```python django.http.request.HttpRequest.GET: TaintSource[UserControlled] ``` #### Matching signatures The signature of any modeled function needs to match the signature of the function, as seen by Pyre. Note that Pyre doesn't always see the definition of the functions directly. If [`.pyi` stub files](https://www.python.org/dev/peps/pep-0484/#stub-files) are present, Pyre will use the signatures from those files, rather than the actual signature from the function definition in your or your dependencies' code. See the [Gradual Typing page](gradual_typing.md) for more info about these `.pyi` stubs. This matching signature requirement means that all parameters being modelled must be named identically to the parameters in the corresponding code or `.pyi` file. Unmodelled parameters, `*args`, and `**kwargs` may be included, but are not required. When copying parameters to your model, all type information must be removed, and all default values must be elided (see below). If a function includes an `*` that indicates [keyword only parameters](https://www.python.org/dev/peps/pep-3102/), or a `/` that indicates [positional-only parameters](https://www.python.org/dev/peps/pep-0570/), then that may be included in your model. Note that unlike when modeling named parameters, you need to include all positional only parameters the model so that Pysa knows what position is being tainted. For example, `urllib.request.urlopen` has the following signature: ```python def urlopen(url, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT, *, cafile=None, capath=None, cadefault=False, context=None): ``` Given that signature, either of the following models are acceptable: ```python def urllib.request.urlopen(url: TaintSink[HTTPClientRequest], data, timeout, *, cafile, capath, cadefault, context): ... def urllib.request.urlopen(url: TaintSink[HTTPClientRequest]): ... ``` Pysa will complain if the signature of your model doesn't match the implementation. When working with functions defined outside your project, where you don't directly see the source, you can use [`pyre query`](querying_pyre.md) with the `signature` argument to have Pysa dump it's internal model of a function, so you know exactly how to write your model. #### Eliding As you can see from the above examples, unmodelled parameters and function bodies can both be elided with `...`. Additionally, type annotations *must* be entirely omitted (not replaced with `...`), even when present on the declaration of the function. This is done to make parsing taint annotations unambiguous. ## Next Steps Ready to start writing some models? Check out our docs on the [end-to-end process of shipping pysa models.](fb/pysa_shipping_rules_models_internal.md) --- ### Documentation/Website/Docs/Pysa Explore --- id: pysa-explore title: Exploring Taint Models Interactively sidebar_label: Exploring Taint Models Interactively --- ## Overview After Pysa's analysis is complete, the tool will output a detailed JSON with its final view of the taint of callables in addition to the issues it finds. We provide a script to explore these taint models called `explore_pysa_models.py`, which can give you insight into why Pysa thinks there might or might not be taint for a given callable. ## Basic Usage Before using the explore script, you should already have run Pysa on your codebase. For the purposes of this page, we will assume you stored it in `/tmp/output_dir`, e.g. ```bash $ pyre analyze --save-results-to /tmp/output_dir ``` After the analysis succeeds, Pysa will write one or multiple taint output files `/tmp/output_dir/taint-output.json`, containing the taint of each callable in addition to the issues found. Let's load this JSON into our explore script: ```bash $ python3 -i scripts/explore_pysa_models.py # Pysa Model Explorer Available commands: index('/path/to/results-directory') Index all available models in the given taint output directory. callables_containing('foo.bar') Find all callables containing the given string. callables_matching(r'foo\..*') Find all callables matching the given regular expression. get_model('foo.bar') Get the model for the given callable. print_model('foo.bar') Pretty print the model for the given callable. Optional parameters: kind='UserControlled' Filter by taint kind. caller_port='result' Filter by caller port. remove_sources=False remove_sinks=False remove_tito=False remove_tito_positions=True remove_features=True remove_leaf_names=True get_issues('foo.bar') Get all issues within the given callable. print_issues('foo.bar') Pretty print the issues within the given callable. print_json({'a': 'b'}) Pretty print json objects with syntax highlighting. ``` ```python >>> index('/tmp/output_dir') Indexing `/tmp/output_dir/taint-output.json` Indexed 307120 models ``` Internally at Meta, we can use Bento to run the model explorer: ```bash $ python3 ~/fbsource/fbcode/tools/pyre/facebook/scripts/in_path/pysa-explore-models ``` Bento can be installed with ```bash $ feature install bento ``` Note: if you are using the [shell integration](fb/getting_started.md#shell-integrations-for-managing-the-frontend-and-backend), you can simply run `pysa-explore-models`. Once we've indexed our taint JSON, we're good to go! Let's investigate what models Pysa finds for HttpRequest. First, we'll need to get the full name of the relevant callables: ```python >>> callables_containing('HttpRequest') ['django.http.request.HttpRequest.__init__', 'django.http.request.HttpRequest.body', ...] >>> get_model('django.http.request.HttpRequest.__init_') {'callable': 'django.http.request.HttpRequest.__init__', 'sources': [], 'sinks': [], 'tito': [{'port': 'formal(self)', 'taint': [{'decl': None, 'leaves': [{'kind': 'LocalReturn', 'name': ''}]}]}]} ``` This (hard-to-parse) JSON is all that Pysa knows about the `HttpRequest.__init__` function. If you squint, you'll see that the model doesn't introduce any sources or sinks (as expected), but has taint-in-taint-out for the `self` parameter. Let's take a look at `body`, a slightly more interesting function. We'll also swap to using the `print_model()` function which will pretty print the output: ```python >>> print_model('django.http.request.HttpRequest.body') { "callable": "django.http.request.HttpRequest.body", "sources": [ { "port": "result", "taint": [ { "decl": null, "kinds": [ { "kind": "UserControlled" } ] }, ] } ], "sinks": [], ... ``` Much easier to read! This model shows that the `body` property of HttpRequests returns a UserControlled source. You can also use the `get_issues`, and corresponding pretty-printing `print_issues` functions to see all issues in a given callable. Note that the `get_issues` and `get_models` functions return Python objects that you can manipulate: ```python >>> print_json(get_issues('foo.bar.log_errors')[0]) # This is valid, will print first issue! ... >>> print_json(get_model('django.http.request.HttpRequest.body')["sources"]) # Pretty print only the sources. [ { "port": "result", "taint": [ { "decl": null, "kinds": [ { "kind": "UserControlled" } ] }, ] } ] ``` ---