### Guides/Api Style =============== House API Style =============== Here are some guidelines for how to write APIs so that they "feel" like a Hypothesis API. This is particularly focused on writing new strategies, as that's the major place where we add APIs, but also applies more generally. Note that it is not a guide to *code* style, only API design. The Hypothesis style evolves over time, and earlier strategies in particular may not be consistent with this style, and we've tried some experiments that didn't work out, so this style guide is more normative than descriptive and existing APIs may not match it. Where relevant, backwards compatibility is much more important than conformance to the style. We also encourage `third-party extensions `_ to follow this style guide, for consistent and user-friendly testing APIs, or get in touch to discuss changing it if it doesn't fit their domain. ~~~~~~~~~~~~~~~~~~ General Guidelines ~~~~~~~~~~~~~~~~~~ * When writing extras modules, consistency with Hypothesis trumps consistency with the library you're integrating with. * *Absolutely no subclassing as part of the public API* * We should not strive too hard to be pythonic, but if an API seems weird to a normal Python user we should see if we can come up with an API we like as much but is less weird. * Code which adds a dependency on a third party package should be put in a hypothesis.extra module. * Complexity should not be pushed onto the user. An easy to use API is more important than a simple implementation. ~~~~~~~~~~~~~~~~~~~~~~~~~ Guidelines for strategies ~~~~~~~~~~~~~~~~~~~~~~~~~ * A strategy function should be somewhere between a recipe for how to build a value and a range of valid values. * It should not include distribution hints. The arguments should only specify how to produce a valid value, not statistical properties of values. * Strategies should try to paper over non-uniformity in the underlying types as much as possible (e.g. ``hypothesis.extra.numpy`` has a number of workarounds for numpy's odd behaviour around object arrays). * Strategies should usually default to allowing generation of any example they can support. The only exceptions should be cases where certain inputs would trigger test failures which are almost never of interest: currently just non-UTF8 characters in ``st.text()``, and Numpy array shapes with zero dimensions or sides of length zero. In each case opting in should be trivial. ~~~~~~~~~~~~~~~~~ Argument handling ~~~~~~~~~~~~~~~~~ We have a reasonably distinctive style when it comes to handling arguments: * Arguments must be validated to the greatest extent possible. Hypothesis should reject bad arguments with an InvalidArgument error, not fail with an internal exception. * We make extensive use of default arguments. If an argument could reasonably have a default, it should. * Exception to the above: strategies for collection types should *not* have a default argument for element strategies. * Arguments which have a default value should also be keyword-only, with the exception of ``min_value`` and ``max_value`` (see "Argument Names" below). * ``min_value`` and ``max_value`` should default to None for unbounded types such as integers, and the minimal or maximal values for bounded types such as datetimes. ``floats()`` is an explicit exception to this rule due to special handling for infinities and not-a-number. * Interacting arguments (e.g. arguments that must be in a particular order, or where at most one is valid, or where one argument restricts the valid range of the other) are fine, but when this happens the behaviour of defaults should automatically be adjusted. e.g. if the normal default of an argument would become invalid, the function should still do the right thing if that default is used. * Where the actual default used depends on other arguments, the default parameter should be None. * It's worth thinking about the order of arguments: the first one or two arguments are likely to be passed positionally, so try to put values there where this is useful and not too confusing. * When adding arguments to strategies, think carefully about whether the user is likely to want that value to vary often. If so, make it a strategy instead of a value. In particular if it's likely to be common that they would want to write ``some_strategy.flatmap(lambda x: my_new_strategy(argument=x))`` then it should be a strategy. * Arguments should not be "a value or a strategy for generating that value". If you find yourself inclined to write something like that, instead make it take a strategy. If a user wants to pass a value they can wrap it in a call to ``just``. * If a combination of arguments make it impossible to generate anything, ``raise InvalidArgument`` instead of ``return nothing()``. Returning the null strategy is conceptually nice, but can lead to silently dropping parts from composed strategies and thus unexpectedly weak tests. ~~~~~~~~~~~~~~ Function Names ~~~~~~~~~~~~~~ We don't have any real consistency here. The rough approach we follow is: * Names are `snake_case` as is standard in Python. * Strategies for a particular type are typically named as a plural name for that type. Where that type has some truncated form (e.g. int, str) we use a longer form name. * Other strategies have no particular common naming convention. ~~~~~~~~~~~~~~ Argument Names ~~~~~~~~~~~~~~ We should try to use the same argument names and orders across different strategies wherever possible. In particular: * For collection types, the element strategy (or strategies) should always be the first arguments. Where there is only one element strategy it should be called ``elements`` (but e.g. ``dictionaries`` has element strategies named ``keys`` and ``values`` and that's fine). * For ordered types, the first two arguments should be a lower and an upper bound. They should be called ``min_value`` and ``max_value``. * Collection types should have a ``min_size`` and a ``max_size`` parameter that controls the range of their size. ``min_size`` should default to zero and ``max_size`` to ``None`` (even if internally it is bounded). ~~~~~~~~~~~~~~~ Deferred Errors ~~~~~~~~~~~~~~~ As far as is reasonable, functions should raise errors when the test is run (typically by deferring them until you try to draw from the strategy), not when they are called. This mostly applies to strategy functions and some error conditions in ``@given`` itself. Generally speaking this should be taken care of automatically by use of the ``@defines_strategy`` decorator. We do not currently do this for the ``TypeError`` that you will get from calling the function incorrectly (e.g. with invalid keyword arguments or missing required arguments). In principle we could, but it would result in much harder to read function signatures, so we would be trading off one form of comprehensibility for another, and so far that hasn't seemed to be worth it. The main reasons for preferring this style are: * Errors at test import time tend to throw people and be correspondingly hard for them to debug. There's an expectation that errors in your test code result in failures in your tests, and the fact that that test code happens to be defined in a decorator doesn't seem to change that expectation for people. * Things like deprecation warnings etc. localize better when they happen inside the test - test runners will often swallow them or put them in silly places if they're at import time, but will attach any output that happens in the test to the test itself. * There are a lot of cases where raising an error, deprecation warning, etc. is *only* possible in a test - e.g. if you're using the inline style with `data `_, or if you're using `flatmap `_ or `@composite `_ then the strategy won't actually get evaluated until we run the test, so that's the only place they can happen. It's nice to be consistent, and it's weird if sometimes strategy errors result in definition time errors and sometimes they result in test errors. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Inferring strategies from specifications ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Functions which infer a strategy from some specification or schema are both convenient for users, and offer a single source of truth about what inputs are allegedly valid and actually tested for correctness. * Such functions should be named "``from_foo()``" and the first argument should be the thing from which a strategy is inferred - like ``st.from_type()``, ``st.from_regex()``, ``extra.lark.from_lark()``, ``extra.numpy.from_dtype()``, etc. Any other arguments should be optional keyword-only parameters. * There should be a smooth path to customise *parts* of an inferred strategy, i.e. not require the user to start from scratch if they need something a little more specific. ``from_dtype()`` does this well; ``from_type()`` supports it by `pointing users to builds() instead `_. * Where practical, ensure that the ``repr`` of the returned strategy shows how it was constructed - only using e.g. ``@st.composite`` if required. For example, ``repr(from_type(int)) == "integers()"``. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A catalogue of current violations ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The following are places where we currently deviate from this style. Some of these should be considered targets for deprecation and/or improvement. * ``hypothesis.extra.numpy`` has some arguments which can be either strategies or values. * ``hypothesis.extra.numpy`` assumes arrays are fixed size and doesn't have ``min_size`` and ``max_size`` arguments (but this is probably OK because of more complicated shapes of array). * ``hypothesis.stateful`` is a great big subclassing based train wreck. --- ### Guides/Documentation ===================================== The Hypothesis Documentation Handbook ===================================== Good documentation can make the difference between good code and useful code - and Hypothesis is written to be used, as widely as possible. This is a working document-in-progress with some tips for how we try to write our docs, with a little of the what and a bigger chunk of the how. If you have ideas about how to improve these suggestions, meta issues or pull requests are just as welcome as for docs or code :D ---------------------------- What docs should be written? ---------------------------- All public APIs should be comprehensively described. If the docs are confusing to new users, incorrect or out of date, or simply incomplete - we consider all of those to be bugs; if you see them please raise an issue and perhaps submit a pull request. That's not much advice, but it's what we have so far. ------------ Using Sphinx ------------ We use `the Sphinx documentation system `_ to convert the .rst files into html with formatting and cross-references. Without repeating the docs for Sphinx, here are some tips: - When documenting a Python object (function, class, module, etc.), you can use autodoc to insert and interpret the docstring. - When referencing a function, you can insert a reference to a function as (eg) ``:func:`hypothesis.given`\``, which will appear as ``hypothesis.given()`` with a hyperlink to the appropriate docs. You can show only the last part (unqualified name) by adding a tilde at the start, like ``:func:`~hypothesis.given`\ `` -> ``given()``. Finally, you can give it alternative link text in the usual way: ``:func:`other text `\ `` -> ``other text``. - For the formatting and also hyperlinks, all cross-references should use the Sphinx cross-referencing syntax rather than plain text. ----------------- Changelog Entries ----------------- `Hypothesis does continuous deployment `_, where every pull request that touches ``./src`` results in a new release. That means every contributor gets to write their changelog! A changelog entry should be written in a new ``RELEASE.rst`` file in the `hypothesis` directory. The first line of the file specifies the component of the version number that will be updated, according to our `semantic versioning `_ policy. - ``RELEASE_TYPE: major`` is for breaking changes, and will only be used by the core team after extensive discussion. - ``RELEASE_TYPE: minor`` is for anything that adds to the public (ie documented) API, changes an argument signature, or adds a new deprecation or health check. Minor (or patch) releases **must not** cause errors in any code that runs without errors on an earlier version of Hypothesis, using only the public API. Silent errors *may* be converted to noisy errors, but generally we prefer to issue a deprecation warning and use the new behaviour if possible. This stability policy only applies to use of Hypothesis itself, not the results of user-written tests that use Hypothesis. - ``RELEASE_TYPE: patch`` is for changes that are not visible in the public interface, from improving a docstring to backwards-compatible improvements in shrinking behaviour. This first line will be removed from the final change log entry. The remaining lines are the actual changelog text for this release, which should: - concisely describe what changed and why - use Sphinx cross-references to any functions or classes mentioned - if closing an issue, mention it with the ``:issue:`` role to generate a link - finish with a note of thanks from the maintainers: "Thanks to for this bug fix / feature / contribution" (depending on which it is). If this is your first contribution, don't forget to add yourself to AUTHORS.rst! --- ### Guides/Internals =================================== How to Work on Hypothesis Internals =================================== This is a guide to how to work on Hypothesis internals, with a particular focus on helping people who are new to it. Right now it is very rudimentary and is intended primarily for people who are looking to get started writing shrink passes as part of our `current outreach program to get more people doing that `_, but it will expand over time. ------------------------ Bird's Eye View Concepts ------------------------ The core engine of Hypothesis is called Conjecture. The "fundamental idea" of Conjecture is that you can represent an arbitrary randomized test case as the sequence of *choices* made while producing it. Whenever the test does something "random" it instead draws the next choice from the sequence and does what it says. Each choice is one of a small number of typed primitives - an integer, float, boolean, string, or bytestring - and corresponds to one of the ``draw_*`` methods on ``ConjectureData``. We call this typed sequence the *choice sequence* (older parts of the codebase and history may refer to it as the "IR", for intermediate representation). These choices don't *have* to come from a PRNG, and we can run the test given any choice sequence we like. By manipulating the choices, we can achieve more interesting effects than pure randomness would allow us to do, while retaining the power and ease of use of random testing. (Historically Conjecture worked directly on the underlying byte sequence read from the PRNG; it now works on the typed choice sequence instead, which shrinks far better because we no longer have to reason about the encoding of each value into bytes.) The greatest strength of this idea is that we have a single source of truth for what an example should look like: Every choice sequence is one that *could* have been produced by drawing from strategies, and thus is a valid thing to try for our test. The only ways it can fail to be a valid test input are for it to be too short or for it to not satisfy one of the test's preconditions, and both are easily detectable. The idea of shrinking in particular is that once we have this representation, we can shrink arbitrary test cases based on it. We try to produce a choice sequence that is *shortlex minimal*. What this means is that it has the shortest possible length and among those sequences of minimal length is the smallest by comparing choices one at a time from the left (see ``sort_key`` in ``shrinker.py``, which orders each choice by ``choice_to_index``). Ideally we could think of the shrinker as a generic function that takes a choice sequence satisfying some predicate and returns the shortlex minimal sequence that also satisfies it. We depart from this ideal in two ways: * we can only *approximate* such a minimal sequence. Finding the actual minimum is intractable in general. * we are only interested in minimizing things where the predicate goes through the Hypothesis API, which lets us track how the data is used and use that to guide the process. We then use a number of different transformations of the choice sequence to try and reduce our input. These vary from principled general transformations to shameless hacks that special case something we need to work well. One such example is the handling of floating point numbers. Floats are drawn as a single choice, but have their own dedicated shrinker (in ``shrinking/floats.py``) which knows about our lexicographic encoding of floats and tries shrinks - such as dropping fractional digits or moving to a nearby integer - that only make sense for that encoding. -------------------------- Useful Files to Know About -------------------------- The code associated with Conjecture lives in `src/hypothesis/internal/conjecture `_. There are a number of files in there, but the most important ones are ``engine.py`` and ``data.py``. ``data.py`` defines the core type that is used to represent test cases, and ``engine.py`` contains the main driver for deciding what test cases to run. ``shrinker.py`` contains the shrinker proper, described below. There is also the ``shrinking/`` package, which contains a collection of small, general purpose shrinkers for individual values - ``Integer``, ``Float``, ``String``, ``Bytes``, ``Collection``, and ``Ordering``. Each takes a single value and a predicate over values and tries to produce a simpler value satisfying it. Unlike the shrinker in general these don't know anything about the testing API; the shrinker drives them with a predicate that knows how to substitute a candidate value back into the full choice sequence and rerun the test. This is the part of the code that means we can do things like replacing an integer with a smaller one. ------- Testing ------- For general information about how to test Hypothesis, take a look at the `testing guide `_, but there are a couple of areas that it's worth specifically highlighting for making changes to the engine: The first is `tests/conjecture/ `_, which is a set of unit tests designed to put the engine into particular scenarios to exercise specific behaviours, with a goal of achieving 100% coverage on it in isolation (though it currently does not quite achieve that for some specific edge cases. We may fix and enforce this later). The other set of tests that are worth knowing about are the quality tests, in `tests/quality `_. These assert specific hard to satisfy properties about the examples that Hypothesis finds - either their existence, or something about the final shrunk result. ----------------------- Engine Design Specifics ----------------------- There are a couple of code patterns that are mostly peculiar to Conjecture that you may not have encountered before and are worth being aware of. ~~~~~~~~~~~~~~~~~~~~ Search State Objects ~~~~~~~~~~~~~~~~~~~~ There are a number of cases where we find ourself with a user-provided function (where the "user" might still be something that is entirely our code) and we want to pass a whole bunch of different examples to it in order to achieve some result. Currently this includes each of the main engine, the Shrinker (in ``engine.py``) and the minimizer, but there are likely to be more in future. We typically organise such things in terms of an object that you create with the function and possibly an initial argument that stores these on self and has some ``run`` or similar method. They then run for a while, repeatedly calling the function they were given. Generally speaking they do not call the function directly, but instead wrap calls to it. This allows them to implement a certain amount of decision caching, e.g. avoiding trying the same shrink twice, but also gives us a place where we can update metadata about the search process. For objects whose goal is some form of optimisation (Shrinker, Minimizer) one of the pieces of metadata they will typically track is a "current target". This is typically the best example they have seen so far. By wrapping every call to the predicate, we ensure that we never miss an example even when we're passing through other things. For objects whose goal is some broader form of search (currently only ``ConjectureRunner``) this also allows them to keep track of *other* examples of interest. For example, as part of our multiple bug discovery, ``ConjectureRunner`` keeps track of the smallest example of each distinct failure that it has seen, and updates this automatically each time the test function is called. This means that if during shrinking we "slip" and find a different bug than the one we started with, we will *not* shrink to that, but it will get remembered by the runner if it was either novel or better than our current example. ~~~~~~~~~~~ Weird Loops ~~~~~~~~~~~ The loops inside a lot of the engine look very strange and unidiomatic. For example: .. code-block:: python i = 0 while i < len(self.shrink_target.nodes): if not self.consider_new_nodes( self.shrink_target.nodes[:i] + self.shrink_target.nodes[i + 1 :] ): i += 1 The more natural way to write this in Python would be: .. code-block:: python for i in range(len(self.shrink_target.nodes)): self.consider_new_nodes( self.shrink_target.nodes[:i] + self.shrink_target.nodes[i + 1 :] ) This is not equivalent in this case, and would exhibit the wrong behaviour. Every time ``consider_new_nodes`` succeeds, it changes the shape of the current shrink target. This consequently changes the shape of the choice sequence, both its particular values and its current length - on each loop iteration the loop might stop either because ``i`` increases or because ``len(self.shrink_target.nodes)`` decreases. We do not reset ``i`` to zero on success, as this would cause us to retry deleting things that we have already tried. This *might* work, but is less likely to. In the event that none of the earlier deletions succeed, this causes us to do retry the entire prefix uselessly, which can result in a pass taking O(n^2) time to do O(n) deletions. An additional quirk is that we only increment ``i`` on failure. The reason for this is that if we successfully deleted the choice at position ``i`` then the node now in position ``i`` is whatever used to follow it, which is probably the next thing we would have tried deleting if we hadn't succeeded (or something like it), so we don't want to advance past it. This is specific to deletion: If we are just replacing the contents of something then we expect it to still be in the same place, so there we increment unconditionally. Examples of this include ``try_trivial_spans`` and ``minimize_individual_choices``. ------------ The Shrinker ------------ The shrinking part of Hypothesis is organised into a single class called ``Shrinker`` that lives in ``hypothesis/internal/conjecture/shrinker.py``. Its job is to take an initial ``ConjectureData`` object and some predicate that it satisfies, and to try to produce a simpler ``ConjectureData`` object that also satisfies that predicate. The search process mostly happens in the ``shrink`` method, which tries various shrink passes in the ``greedy_shrink`` method and then reports on the outcome. For details, you are strongly encouraged to read the source code. It is very well commented, and as the subject of active research often has newer techniques than are documented here. ~~~~~~~~~~~~~ Search Passes ~~~~~~~~~~~~~ Search passes are methods on the ``Shrinker`` class. They are designed to take the current shrink target and try a number of things that might be sensible shrinks of it. Typically the design of a search pass is that it should always try to run to completion rather than exiting as soon as it's found something good, but that it shouldn't retry things that are too like stuff it has already tried just because something worked. So for example in the above loop, we try deleting each interval (these roughly correspond to regions of the input that are responsible for some particular value or small number of adjacent values). When we succeed, we keep going and try deleting more intervals, but we don't try to delete any intervals before the current index. The reason for this is that retrying things from the beginning might work but probably won't. Thus if we restarted every time we made a change we would end up doing a lot of useless work. Additionally, they are *more* likely to work after other shrink passes have run because frequently other changes are likely to unlock changes in the current pass that were previously impossible. e.g. when we reorder some examples we might make a big region deletable that previously contained something critical to the relevant behaviour of the test but is now just noise. Because the shrinker runs in a big loop, if we've made progress the shrink pass will always be run again (assuming we don't hit some limit that terminates the shrink early, but by making the shrinker better we try to ensure that that never happens). This means that we will always get an opportunity to start again later if we made progress, and if we didn't make progress we've tried everything anyway. ~~~~~~~~~~~~~~~~~~~~~~~ Expensive Shrink Passes ~~~~~~~~~~~~~~~~~~~~~~~ We have a bunch of search passes that are considered "expensive". Typically this means "quadratic or worse complexity". When shrinking we initially don't run these, and the first time that we get to the end of our main passes and have failed to make the input any smaller, we then turn them on. This allows the shrinker to switch from a good but slightly timid mode while its input is large into a more aggressive DELETE ALL THE THINGS mode once that stops working. By that point we've usually made our input small enough that quadratic complexity is acceptable. We turn these on once and then they stay on. The reason for this is to avoid a "flip-flopping" scenario where an expensive pass unlocks one trivial change that the cheap passes can find and then they get stuck again and have to do an extra useless run through the passes to prove that. ~~~~~~~~~~~~~~~~~~~~~~ Adaptive Shrink Passes ~~~~~~~~~~~~~~~~~~~~~~ A useful trick that some of the shrink passes use is to try a thing and if it doesn't work take a look at what the test function did to guess *why* it didn't work and try to repair that. The main example is ``try_shrinking_nodes``, which is used by the passes that minimize individual choices. When it lowers one or more choices and the result doesn't satisfy the predicate, it inspects what the test function did: * If the attempt was *misaligned* - the test tried to draw a choice of a different type or size than the one we substituted, which commonly happens when an earlier choice controls the size of a later collection - it tries to realign the tree, for example by truncating a string to the size the test actually asked for. * Otherwise, if the attempt lost some nodes (e.g. lowering a value made a following list shorter), it tries deleting the corresponding region of the choice sequence immediately after the lowered choice to see if that repairs the shape. -------------- Playing Around -------------- I often find that it is informative to watch the shrink process in action using Hypothesis's verbosity settings. This can give you an idea of what the format of your data is, and how the shrink process transforms it. In particular, it is often useful to run a test with the flag ``-s`` to tell it not to hide output and the environment variable ``HYPOTHESIS_VERBOSITY_LEVEL=debug``. This will give you a very detailed log of what the testing process is running, along with information about what passes in the shrinker are running and how they transform it. --------------- Getting Started --------------- The best way of getting started on working on the engine is to work on the shrinker. This is because it has the most well defined problems, the best documented code among the engine, and it's generally fun to work on. If you have not already done so, check out `Issue #1093 `_, which collates a number of other issues about shrink quality that are good starting points for people. The best place to get started thus is to take a look at those linked issues and jump in and try things! Find one that you think sounds fun. Note that some of them suggest not doing these as your first foray into the shrinker, as some are harder than others. *Please* ask questions if you have any - either the main issue for general purpose questions or specific issues for questions about a particular problem - if you get stuck or if anything doesn't make sense. We're trying to make this process easier for everyone to work on, so asking us questions is actively helpful to us and we will be very grateful to you for doing so. --- ### Guides/README # Guides for Hypothesis Development This is a general collection of useful documentation for people working on Hypothesis. It is separate from the main documentation because it is not much use if you are merely *using* Hypothesis. It's purely for working on it, and aimed more at maintainers than casual contributors. --- ### Guides/Review =================================== The Hypothesis Code Review Handbook =================================== This document outlines the process for reviewing changes to Hypothesis. It's partly descriptive, partly prescriptive, and entirely prone to change in response to circumstance and need. We're still figuring this thing out! ----------------- What Needs Review ----------------- All changes must be signed off by at least one person with write access to the repo other than the author of the change. ---------------- How Review Works ---------------- Once the build is green and a reviewer has approved the change, anyone on the maintainer team may merge the request. More than one maintainer *may* review a change if they wish to, but it's not required. Any maintainer may block a pull request by requesting changes. Consensus on a review is best but not required. If some reviewers have approved a pull request and some have requested changes, ideally you would try to address all of the changes, but it is OK to dismiss dissenting reviews if you feel it appropriate. We've not tested the case of differing opinions much in practice yet, so we may grow firmer guidelines on what to do there over time. ------------ Review Goals ------------ At a high level, the two things we're looking for in review are answers to the following questions: 1. Is this change going to make users' lives worse? 2. Is this change going to make the maintainers' lives worse? Code review is a collaborative process between the author and the reviewer to try to ensure that the answer to both of those questions is no. Ideally of course the change should also make one or both of the users' and our lives *better*, but it's OK for changes to be mostly neutral. The author should be presumed to have a good reason for submitting the change in the first place, so neutral is good enough! -------------- Social Factors -------------- * Always thank external contributors. Thank maintainers too, ideally! * Remember that the `Code of Conduct `_ applies to pull requests and issues too. Feel free to throw your weight around to enforce this if necessary. * Anyone, maintainer or not, is welcome to do a code review. Only official maintainers have the ability to actually approve and merge a pull request, but outside review is also welcome. ------------ Requirements ------------ The rest of this document outlines specific things reviewers should focus on in aid of this, broken up by sections according to their area of applicability. All of these conditions must be satisfied for merge. Where the reviewer thinks this conflicts with the above higher level goals, they may make an exception if both the author and another maintainer agree. ~~~~~~~~~~~~~ Orthogonality ~~~~~~~~~~~~~ For all minor or patch releases, we enforce a hard and fast rule that they contain no more than one user-visible change. Major releases are allowed to bundle multiple changes together, but these should be structured as smaller pull requests into some tracking branch. We are currently very bad at this, so reviewers should feel empowered to be extra strict and provide a lot of push back on this. What counts as a user visible change is somewhat up to individual judgement, but you should err in the direction of assuming that if it might count then it does count. A good rule of thumb is that if the ``RELEASE.rst`` uses the words "additionally" or needs bullet points to be clear, it is likely too large. Ideally changes that are not user visible should also be self-contained into their own releases, but a certain amount of leniency is permitted - it's certainly OK to do a moderate amount of refactoring while you're in the area, and if a pull request involves no release at all then the same level of orthogonality is not required (but is still desirable). ~~~~~~~~~~~~~~~~~~~~~~ Clarity of Description ~~~~~~~~~~~~~~~~~~~~~~ The ``RELEASE.rst`` should contain a description of the change that makes clear: 1. The motivation for the change 2. The likely consequences of the change This doesn't have to be an essay. If you're following the orthogonality requirements a paragraph or two is likely sufficient. Any additional information that is useful to reviewers should be provided in the pull request comment. This can include e.g. background, why the particular approach was taken, references to internals that are unlikely to be of interest to users. ~~~~~~~~~~~~~~~~~~~~~ Functionality Changes ~~~~~~~~~~~~~~~~~~~~~ This section applies to any changes in Hypothesis's behaviour, regardless of their nature. A good rule of thumb is that if it touches a file in src then it counts. 1. The code should be clear in its intent and behaviour. 2. Behaviour changes should come with appropriate tests to demonstrate the new behaviour. 3. Hypothesis must never be *flaky*. Flakiness here is defined as anything where a test fails and this does not indicate a bug in Hypothesis or in the way the user wrote the code or the test. 4. The changelog (in ``RELEASE.rst``) should bump the minor or patch version (see guides/documentation.rst for details), accurately describe the changes, and shouldn't refer to internal-only APIs. For complicated markup, consider building the docs and manually checking the changelog for formatting errors that didn't result in a compilation error. ~~~~~~~~~~~ API Changes ~~~~~~~~~~~ Public API changes require the most careful scrutiny of all reviews, because they are the ones we are stuck with for the longest: Hypothesis follows semantic versioning, and we don't release new major versions very often. Public API changes must satisfy the following: 1. All public API changes must be well documented. If it's not documented, it doesn't count as public API! 2. Changes must be backwards compatible. Where this is not possible, they must first introduce a deprecation warning, then once the major version is bumped the deprecation warning and the functionality may be removed. 3. If an API is deprecated, the deprecation warning must make it clear how the user should modify their code to adapt to this change ( possibly by referring to documentation). If the required code change could be automated, the deprecation should have either `a codemod to fix it `__ or a tracking issue to write one (see "asking for more work" below). 4. If it is likely that we will want to make backwards incompatible changes to an API later, to whatever extent possible these should be made immediately when it is introduced instead. 5. APIs should give clear and helpful error messages in response to invalid inputs. In particular error messages should always display the value that triggered the error, and ideally be specific about the relevant feature of it that caused this failure (e.g. the type). 6. Incorrect usage should never "fail silently" - when a user accidentally misuses an API this should result in an explicit error. 7. Functionality should be limited to that which is easy to support in the long-term. In particular functionality which is very tied to the current Hypothesis internals should be avoided. 8. `DRMacIver `_ or `Zac-HD `_ must approve the changes though other maintainers are welcome and likely to chip in to review as well. 9. We have a separate guide for `house API style `_ which should be followed. ~~~~~~~~~ Bug Fixes ~~~~~~~~~ 1. All bug fixes must come with a test that demonstrates the bug on master and which is fixed in this branch. An exception *may* be made here if the submitter can convincingly argue that testing this would be prohibitively difficult. 2. Where possible, a fix that makes it impossible for similar bugs to occur is better. 3. Where possible, a test that will catch both this bug and a more general class of bug that contains it is better. ~~~~~~~~~~~~~~~~ Settings Changes ~~~~~~~~~~~~~~~~ Note: This section currently only applies to the Python version. It is tempting to use the Hypothesis settings object as a dumping ground for anything and everything that you can think of to control Hypothesis. This rapidly gets confusing for users and should be carefully avoided. New settings should: 1. Be something that the user can meaningfully have an opinion on. Many of the settings that have been added to Hypothesis are just cases where Hypothesis is abdicating responsibility to do the right thing to the user. 2. Make sense without reference to Hypothesis internals. 3. Correspond to behaviour which can meaningfully differ between tests - either between two different tests or between two different runs of the same test (e.g. one use case is the profile system, where you might want to run Hypothesis differently in CI and development). If you would never expect a test suite to have more than one value for a setting across any of its runs, it should be some sort of global configuration, not a setting. When deprecating a setting for later removal, we prefer to change the default value of the setting to a private singleton (``not_set``), and implement the future behaviour immediately. Passing any other value triggers a deprecation warning, but is otherwise a no-op (i.e. we still use the future behaviour). For settings where this would be especially disruptive, we have also prefixed that deprecation process with a process where we emit a warning, add a special value that can be passed to opt-in to the future behaviour, and then in the following major release we deprecate *that*, make it an no-op, and make it an error to pass any other value. ~~~~~~~~~~~~~~ Engine Changes ~~~~~~~~~~~~~~ Engine changes are anything that change a "fundamental" of how Hypothesis works. A good rule of thumb is that an engine change is anything that touches a file in ``hypothesis.internal.conjecture`` (Python version). All such changes should: 1. Be approved (or authored) by DRMacIver or Zac-HD. 2. Be approved (or authored) by someone who *isn't* DRMacIver (a major problem with this section of the code is that there is too much that only DRMacIver understands properly and we want to fix this). 3. If appropriate, come with a test in test_discovery_ability.py showing new examples that were previously hard to discover. 4. If appropriate, come with a test in test_shrink_quality.py showing how they improve the shrinker. ~~~~~~~~~~~~~~~~~~~~~~ Non-Blocking Questions ~~~~~~~~~~~~~~~~~~~~~~ These questions should *not* block merge, but may result in additional issues or changes being opened, either by the original author or by the reviewer. 1. Is this change well covered by the review items and is there anything that could usefully be added to the guidelines to improve that? 2. Were any of the review items confusing or annoying when reviewing this change? Could they be improved? 3. Are there any more general changes suggested by this, and do they have appropriate issues and/or pull requests associated with them? ~~~~~~~~~~~~~~~~~~~~ Asking for more work ~~~~~~~~~~~~~~~~~~~~ Reviewers should in general not request changes that expand the scope of a pull request beyond its original intended goal. The primary design philosophy of our work-flow is that making correct changes should be cheap, and scope creep on pull requests works against that - If you can't touch something without having to touch a number of related areas as well, changing things becomes expensive again. This of course doesn't cover things where additional work is required to ensure the change is actually correct - for example, if you change public functionality you certainly need to update its documentation. That isn't scope creep, that's just the normal scope. If a pull request suggests additional work then between the reviewer and the author people should ensure that there are relevant tracking issues for that work (as per question 3 in "Non-Blocking Questions" above), but there is no obligation for either of them to actually do any of the work on those issues. By default it is the reviewer who should open these issues, but the author is welcome to as well. That being said, it's legitimate to expand the scope of a pull request in some cases. For example: * If not doing so is likely to cause problems later. For example, because of backwards compatibility requirements it might make sense to ask for some additional functionality that is likely to be added later so that the arguments to a function are in a more sensible order. * Cases where the added functionality feels extremely incomplete in some way without an additional change. The litmus test here should be "this will almost never be useful because...". This is still fairly subjective, but at least one good use case where the change is a clear improvement over the status quo is enough to indicate that this doesn't apply. If it's unclear, the reviewer should feel free to suggest additional work (but if the author is someone new, please make sure that it's clear that this is a suggestion and not a requirement!), but the author of the pull request should feel equally free to decline the suggestion. --- ### Guides/Strategies That Shrink =================================== Designing strategies to shrink well =================================== Reducing test cases to a minimal example is a great feature of Hypothesis, the implementation of which depends on both the shrinking engine and the structure of the strategy (or combination of strategies) which created the example to reduce. This document is organised into three parts: 1. How to tell if you need to think about shrinking (you probably don't!) 2. Designing for shrinking 'above' the Hypothesis public API 3. Implementation tricks used in our internals, for interested contributors It is written for people implementing complex third-party strategies (such as `hypothesis-networkx `__), current or potential contributors to Hypothesis itself, and anyone interested in how this works under the hood. ------------------------------------ Do you need to design for shrinking? ------------------------------------ You should only attempt to tune custom strategies for better shrinking behaviour if more time would otherwise be spent reducing examples by hand or debugging more complex examples. It *may* be worthwhile if: - Your custom strategy will be used by many people, so that spending the same effort tuning the strategy has much larger benefits, or - You have personally spent time debugging failures which better example shrinking could have avoided and think this might happen again. If neither of these apply to you, relax! Hypothesis' test-case reduction is among the best in the world, and our built-in strategies are carefully designed to work well with it as discussed below. ------------------------------------ Shrinking for third-party strategies ------------------------------------ That is, strategies built out of other strategies until you get down to Hypothesis' public API. These often but not always use ``@composite``. Composition of shrinking ~~~~~~~~~~~~~~~~~~~~~~~~ The first and most important rule is that Hypothesis shrinks from the 'bottom up'. If any component of your strategy is replaced with a simpler example, the end result should also become simpler. We usually try to define "simpler" here to match a reasonable intuition about the strategy, and avoid weird edge cases when it's combined with another strategy or predicate. `Issue #1076 `_, where magnitude constraints were added to the ``complex_numbers`` strategy, makes a nice case study. We wanted to continue shrinking the real and imaginary parts like ``builds(complex, floats(), floats())``. In a worst-case scenario, the performance of filtering could be arbitrarily bad, while a 'generate and scale' approach would mean that simple inputs could lead to irrational outputs. Instead, we choose an imaginary part between +/- max_magnitude, then calculate the resulting bounds on the real part and draw it from a strategy that will always be valid. This ensures that the imaginary part shrinks to zero first, as we think real-valued complex numbers are simpler than imaginary-valued complex numbers. Let generation be lucky ~~~~~~~~~~~~~~~~~~~~~~~ Sometimes, it's worth searching for a particularly nasty value to try. This trick should be used sparingly, and always behind a branch that the shrinker can decide not to take such as ``if draw(booleans()):``, but might occasionally worth trying. Measure the results before you keep it! `Issue #69 `_ provides a nice case study: when generating tz-aware datetimes, we would like to generate instants that are skipped or repeated due to a daylight-savings transition more often than by chance. Of course, there may or may not be any such moments allowed by the bounds and tz strategy! Eliding much of the detail, a key part is to find such a moment between two endpoints, when we can only check whether one or more exists. The traditional approach would be to use a binary search, but this would be relatively expensive to shrink as we would pay the log-n cost on every attempted shrink. Instead of choosing the midpoint, we draw a *random* point between our known endpoints, and repeat this until we find a satisfactory moment. This allows the shrinker to delete all the intermediate draws - and appear lucky enough to find the moment we were looking for on the first guess! Keep things local ~~~~~~~~~~~~~~~~~ Hypothesis' shrinking engine sees every example as a labelled tree of choices, with possible reductions represented as operations on the tree. An attempted shrink succeeds if the new tree can be converted into an example, and the resulting example triggers the same bug in the test function. The most common way we see users breaking data locality is by drawing a size, then drawing a collection of that size. This is tempting because it's simple and it _works_, but it's often much slower than the alternatives. .. code:: python # Both of these strategies can generate exactly the same kind of examples, # but the second has better performance as well as style. integers(0, 10).flatmap(lambda n: st.lists(..., min_size=n, max_size=n)) st.lists(..., min_size=1, max_size=10) Another easy way to keep things local is to ensure that any ``.filter(...)`` or ``assume(...)`` calls you use are as close as possible to the relevant part of the strategy. That way, Hypothesis can retry just the part that failed instead of the entire strategy, which might be much slower. For efficient shrinking, local operations on the tree should correspond with valid (and preferably local) shrinks to the final example. For example: .. code:: python # This form of loop is hard to shrink, because we'd have to reduce `n` and # delete something in the loop simultaneously. It's equivalent to the # `.flatmap` example above. We _do_ shrink this, but much more slowly. n = draw(integers(0, 10)) for _ in range(n): ... draw(...) ... # In this form, the shrinker can see a repeated structure of labels # and delete one loop iteration without touching anything else. # We use a variant of this trick to generate collections internally! while draw(integers(0, x)) > threshold: ... draw(...) ... Similarly, it's better to draw all the attributes or inputs you need for an object at the same time, again so they can be modified or deleted together. The exact behaviour of the shrinking is a topic of active research and development, so if you are interested in the details we recommend reading the `internals guide `_ and the well-commented source code in ``hypothesis.internal.conjecture`` as well as David's ECOOP 2020 paper `Test-Case Reduction via Test-Case Generation: Insights From the Hypothesis Reducer `__. ------------------------------------- Shrinking in the Hypothesis internals ------------------------------------- The last section is for current or prospective Hypothesis contributors only. These tricks rely on implementation details that are not available to third-party libraries or users, **and can change in any patch release**. Occasionally they are also indispensable to get good performance in underlying primitives, so please contact us if the public API is not enough and we may be able to work something out. What do internals get you? ~~~~~~~~~~~~~~~~~~~~~~~~~~ Using the low-level, internal APIs complements, rather than changing, the principles above. The bytestream-level view has some important advantages: Because we operate at the level of bits, the relationship between a value and the corresponding buffer is much more obvious. If we're careful, that means we can calculate the value we want and then write the corresponding buffer to recreate it when the test case is shrunk or replayed. A small step up from bits, we can also see the spans that indicate a subset of the buffer to consider for various transformations such as transposition or deletion. Sometimes these features are the only way to maintain acceptable performance in very rare or even pathological cases - consider shrinking a complex number with a single allowed magnitude - but it's almost certain that someone will need the core strategies to do just that. However, using low-level APIs also comes at a cost - they are verbose and generally more difficult to use, and can violate key invariants of the engine if misused. Internally, our strategies mostly use the public API or something that looks a lot like ``@composite``, so it's fairly easy to follow along. There are just a few tricks enabled by those low-level advantages that we wanted to name and document, so we can recognise them discuss them and invent more... Make your own luck ~~~~~~~~~~~~~~~~~~ This is the simplest trick that uses our ability to write choices to the buffer. We use it for ``sampled_from(...).filter(...)``, after trying an initial draw with the usual rejection sampling technique, and added the ``SearchStrategy.do_filtered_draw`` method so other strategies can opt-in as we design similar tricks for their structure. It was originally designed for stateful testing, where "lucky generation" might be inefficient if there are many rules but only a few allowed by their preconditions. Here's how it works for stateful testing: 1. Draw an index into the unfiltered list of rules. Return the corresponding rule if it's allowed - we got lucky! (or someone set us up...) 2. Create a list of allowed rules, and choose one from that shortlist instead. 3. Find the index of the chosen rule *in the unfiltered list*, and write that index to the buffer. Finally, return the chosen rule. When the shrinker tries to delete the first two draws, the resulting buffer will lead to the same rule being chosen at step *one* instead. We've made our own luck! This trick is especially useful when we want to avoid rejection sampling (the ``.filter`` method, ``assume``) for performance reasons, but also need to give the shrinker the same low-level representation for each instance of a repeated choice. Flags "shrink open" ~~~~~~~~~~~~~~~~~~~ An important insight from `Swarm Testing (PDF) `__ is that randomly disabling some features can actually reduce the expected time before finding a bug, because some bugs may be suppressed by otherwise common features or attributes of the data. As discussed on `issue #1401 `__, there are a few points to keep in mind when implementing shrinkable swarm testing: - You need swarm flags to "shrink open" so that once the shrinker has run to completion, all flags are enabled. e.g. you could do this by generating a set of banned flags. - You need to use rejection sampling rather than anything more clever, or at least look like it to the shrinker. (see e.g. *Make your own luck*, above) Taking Unicode as an example, we'd like to use our knowledge of Unicode categories to generate more complex examples, but shrink the generated string without reference to categories. While we haven't actually implemented this yet - it's pretty hairy - the simple version of the idea goes like this: 1. Generate a set of banned categories. 2. Use ``characters().filter(category_is_not_banned)`` When shrinking, we start by removing categories from the banned set, after which characters in the string can be reduced as usual. In a serious version, the make-your-own-luck approach would be essential to make the filter reasonably efficient, but that's not a problem internally. In more complicated structures, it would be nice to generate the flags on first use rather than up front before we know if we need them. The trick there is to write each flag to the buffer every time we check it, in such a way that if we delete the first use the second turns into an initialisation. Explicit example boundaries ~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is almost always handled implicitly, e.g. by ``cu.many``, but *sometimes* it can be useful to explicitly insert boundaries around draws that should be deleted simultaneously using ``data.start_span``. This is used to group the value and sign of floating-point numbers, for example, which we split up in order to provide a more natural shrinking order. Explicit example management can also be useful to delineate variably-sized draws, such as our internal helper ``cu.biased_coin``, which makes eliminating dead bytes much cheaper. Finally, labelling otherwise indistinguishable draws means the shrinker can attempt to swap only the like values. --- ### Guides/Testing Hypothesis ================== Testing Hypothesis ================== Note: This guide is currently entirely specific to the Python version of Hypothesis. This is a guide to the process of testing Hypothesis itself, both how to run its tests and how to write new ones. -------------------------- General Testing Philosophy -------------------------- The test suite for Hypothesis is unusually powerful - as you might hope! - but the secret is actually more about attitude than technology. The key is that we treat any bug in Hypothesis as a bug in our test suite too - and think about the kinds of bugs that might not be caught, then write tests that would catch them. We also use a variety of tools to check our code automatically, including formatting, import order, linting, and typing our API with Mypy. All of this is checked in CI - which means that once the build is green, humans can all focus on meaningful review rather than nitpicking operator spacing. Similarly, we require all code to have tests with 100% branch coverage - as a starting point, not the final goal. - Requiring full coverage can't guarantee that we've written all the tests worth writing (for example, maybe we left off a useful assertion about the result), but less than full coverage guarantees that there's some code we're not testing at all. - Tests beyond full coverage generally aim to demonstrate that a particular feature works, or that some subtle failure case is not present - often because when it was found and fixed, someone wrote a test to make sure it couldn't come back! The ``hypothesis/tests/`` directory has some notes in the README file on where various kinds of tests can be found or added. Go there for the practical stuff, or just ask one of the maintainers for help on a pull request! Further reading: How `SQLite is tested `_, `how the Space Shuttle was tested `_, `how to misuse code coverage `_ (for inspiration, *not* implementation). Dan Luu writes about `fuzz testing `_ and `broken processes `_, among other things. ------------- Running Tests ------------- Tests are run via ``build.sh``. See ``CONTRIBUTING.rst`` for more details. --- ### README
# Hypothesis * [Website](https://hypothesis.works/) * [Documentation](https://hypothesis.readthedocs.io/en/latest/) * [Source code](https://github.com/hypothesisWorks/hypothesis/) * [Contributing](https://github.com/HypothesisWorks/hypothesis/blob/master/CONTRIBUTING.rst) * [Community](https://hypothesis.readthedocs.io/en/latest/community.html) Hypothesis is the property-based testing library for Python. With Hypothesis, you write tests which should pass for all inputs in whatever range you describe, and let Hypothesis randomly choose which of those inputs to check - including edge cases you might not have thought about. For example: ```python from hypothesis import given, strategies as st @given(st.lists(st.integers())) def test_matches_builtin(ls): assert sorted(ls) == my_sort(ls) ``` This randomized testing can catch bugs and edge cases that you didn't think of and wouldn't have found. In addition, when Hypothesis does find a bug, it doesn't just report any failing test case — it reports the simplest possible one. This makes property-based tests a powerful tool for debugging, as well as testing. For instance, ```python def my_sort(ls): return sorted(set(ls)) ``` fails with the simplest possible failing test case: ``` Failing test case: test_matches_builtin(ls=[0, 0]) ``` ### Installation To install Hypothesis: ``` pip install hypothesis ``` There are also [optional extras available](https://hypothesis.readthedocs.io/en/latest/extras.html). ---