Guides/Api Style
===============
House API Style
===============
Here are some guidelines for how to write APIs so that they "feel" like
a Hypothesis API. This is particularly focused on writing new strategies, as
that's the major place where we add APIs, but also applies more generally.
Note that it is not a guide to code style, only API design.
The Hypothesis style evolves over time, and earlier strategies in particular
may not be consistent with this style, and we've tried some experiments
that didn't work out, so this style guide is more normative than descriptive
and existing APIs may not match it. Where relevant, backwards compatibility is
much more important than conformance to the style.
We also encourage third-party extensions <https://hypothesis.readthedocs.io/en/latest/strategies.html>_
to follow this style guide, for consistent and user-friendly testing APIs,
or get in touch to discuss changing it if it doesn't fit their domain.
~~~~~~~~~~~~~~~~~~
General Guidelines
~~~~~~~~~~~~~~~~~~
* When writing extras modules, consistency with Hypothesis trumps consistency
with the library you're integrating with.
Absolutely no subclassing as part of the public API*
* We should not strive too hard to be pythonic, but if an API seems weird to a
normal Python user we should see if we can come up with an API we like as
much but is less weird.
* Code which adds a dependency on a third party package should be put in a
hypothesis.extra module.
* Complexity should not be pushed onto the user. An easy to use API is more
important than a simple implementation.
~~~~~~~~~~~~~~~~~~~~~~~~~
Guidelines for strategies
~~~~~~~~~~~~~~~~~~~~~~~~~
* A strategy function should be somewhere between a recipe for how to build a
value and a range of valid values.
* It should not include distribution hints. The arguments should only specify
how to produce a valid value, not statistical properties of values.
* Strategies should try to paper over non-uniformity in the underlying types
as much as possible (e.g. `hypothesis.extra.numpy has a number of
workarounds for numpy's odd behaviour around object arrays).
* Strategies should usually default to allowing generation of any example they
can support. The only exceptions should be cases where certain inputs would
trigger test failures which are almost never of interest: currently just
non-UTF8 characters in st.text(), and Numpy array shapes with zero
dimensions or sides of length zero. In each case opting in should be trivial.
~~~~~~~~~~~~~~~~~
Argument handling
~~~~~~~~~~~~~~~~~
We have a reasonably distinctive style when it comes to handling arguments:
* Arguments must be validated to the greatest extent possible. Hypothesis
should reject bad arguments with an InvalidArgument error, not fail with an
internal exception.
* We make extensive use of default arguments. If an argument could reasonably
have a default, it should.
Exception to the above: strategies for collection types should not* have a
default argument for element strategies.
* Arguments which have a default value should also be keyword-only, with the
exception of min_value and max_value (see "Argument Names" below).
* min_value and max_value should default to None for unbounded types
such as integers, and the minimal or maximal values for bounded types such
as datetimes. floats() is an explicit exception to this rule due to
special handling for infinities and not-a-number.
* Interacting arguments (e.g. arguments that must be in a particular order, or
where at most one is valid, or where one argument restricts the valid range
of the other) are fine, but when this happens the behaviour of defaults
should automatically be adjusted. e.g. if the normal default of an argument
would become invalid, the function should still do the right thing if that
default is used.
* Where the actual default used depends on other arguments, the default parameter
should be None.
* It's worth thinking about the order of arguments: the first one or two
arguments are likely to be passed positionally, so try to put values there
where this is useful and not too confusing.
* When adding arguments to strategies, think carefully about whether the user
is likely to want that value to vary often. If so, make it a strategy instead
of a value. In particular if it's likely to be common that they would want to
write some_strategy.flatmap(lambda x: my_new_strategy(argument=x)) then
it should be a strategy.
* Arguments should not be "a value or a strategy for generating that value".
If you find yourself inclined to write something like that, instead make it
take a strategy. If a user wants to pass a value they can wrap it in a call
to just.
* If a combination of arguments make it impossible to generate anything,
raise InvalidArgument instead of return nothing(). Returning the
null strategy is conceptually nice, but can lead to silently dropping parts
from composed strategies and thus unexpectedly weak tests.
~~~~~~~~~~~~~~
Function Names
~~~~~~~~~~~~~~
We don't have any real consistency here. The rough approach we follow is:
* Names are snake_case as is standard in Python.
* Strategies for a particular type are typically named as a plural name for
that type. Where that type has some truncated form (e.g. int, str) we use a
longer form name.
* Other strategies have no particular common naming convention.
~~~~~~~~~~~~~~
Argument Names
~~~~~~~~~~~~~~
We should try to use the same argument names and orders across different
strategies wherever possible. In particular:
* For collection types, the element strategy (or strategies) should always be
the first arguments. Where there is only one element strategy it should be
called elements (but e.g. dictionaries has element strategies named
keys and values and that's fine).
* For ordered types, the first two arguments should be a lower and an upper
bound. They should be called min_value and max_value.
* Collection types should have a min_size and a max_size parameter that
controls the range of their size. min_size should default to zero and
max_size to None (even if internally it is bounded).
~~~~~~~~~~~~~~~
Deferred Errors
~~~~~~~~~~~~~~~
As far as is reasonable, functions should raise errors when the test is run
(typically by deferring them until you try to draw from the strategy),
not when they are called.
This mostly applies to strategy functions and some error conditions in@given itself.
Generally speaking this should be taken care of automatically by use of the@defines_strategy decorator.
We do not currently do this for the TypeError that you will get from
calling the function incorrectly (e.g. with invalid keyword arguments or
missing required arguments).
In principle we could, but it would result in much harder to read function
signatures, so we would be trading off one form of comprehensibility for
another, and so far that hasn't seemed to be worth it.
The main reasons for preferring this style are:
* Errors at test import time tend to throw people and be correspondingly hard
for them to debug.
There's an expectation that errors in your test code result in failures in
your tests, and the fact that that test code happens to be defined in a
decorator doesn't seem to change that expectation for people.
* Things like deprecation warnings etc. localize better when they happen
inside the test - test runners will often swallow them or put them in silly
places if they're at import time, but will attach any output that happens
in the test to the test itself.
* There are a lot of cases where raising an error, deprecation warning, etc.
is only possible in a test - e.g. if you're using the inline style with
data <https://hypothesis.readthedocs.io/en/latest/data.html#drawing-interactively-in-tests>_,flatmap <https://hypothesis.readthedocs.io/en/latest/data.html#chaining-strategies-together>
or if you're using
_@composite <https://hypothesis.readthedocs.io/en/latest/data.html#composite-strategies>
or
_
then the strategy won't actually get evaluated until we run the test,
so that's the only place they can happen.
It's nice to be consistent, and it's weird if sometimes strategy errors result in
definition time errors and sometimes they result in test errors.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Inferring strategies from specifications
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Functions which infer a strategy from some specification or schema are both
convenient for users, and offer a single source of truth about what inputs
are allegedly valid and actually tested for correctness.
* Such functions should be named "from_foo()" and the first argument should
be the thing from which a strategy is inferred - like st.from_type(),
st.from_regex(), extra.lark.from_lark(), extra.numpy.from_dtype(),
etc. Any other arguments should be optional keyword-only parameters.
There should be a smooth path to customise parts* of an inferred strategy,
i.e. not require the user to start from scratch if they need something a
little more specific. from_dtype() does this well; from_type() supportspointing users to builds() instead <https://hypothesis.works/articles/types-and-properties/>
it by _.
* Where practical, ensure that the repr of the returned strategy shows
how it was constructed - only using e.g. @st.composite if required.
For example, repr(from_type(int)) == "integers()".
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A catalogue of current violations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The following are places where we currently deviate from this style. Some of
these should be considered targets for deprecation and/or improvement.
* hypothesis.extra.numpy has some arguments which can be either
strategies or values.
* hypothesis.extra.numpy assumes arrays are fixed size and doesn't have
min_size and max_size arguments (but this is probably OK because of
more complicated shapes of array).
* hypothesis.stateful is a great big subclassing based train wreck.
---
Guides/Documentation
=====================================
The Hypothesis Documentation Handbook
=====================================
Good documentation can make the difference between good code and useful code -
and Hypothesis is written to be used, as widely as possible.
This is a working document-in-progress with some tips for how we try to write
our docs, with a little of the what and a bigger chunk of the how.
If you have ideas about how to improve these suggestions, meta issues or pull
requests are just as welcome as for docs or code :D
----------------------------
What docs should be written?
----------------------------
All public APIs should be comprehensively described. If the docs are
confusing to new users, incorrect or out of date, or simply incomplete - we
consider all of those to be bugs; if you see them please raise an issue and
perhaps submit a pull request.
That's not much advice, but it's what we have so far.
------------
Using Sphinx
------------
We use the Sphinx documentation system <http://sphinx-doc.org>_ to
convert the .rst files into html with formatting and
cross-references. Without repeating the docs for Sphinx, here are some tips:
- When documenting a Python object (function, class, module, etc.), you can
use autodoc to insert and interpret the docstring.
- When referencing a function, you can insert a reference to a function as
(eg) :func:hypothesis.given\, which will appear as
hypothesis.given() with a hyperlink to the appropriate docs. You can
show only the last part (unqualified name) by adding a tilde at the start,
like :func:~hypothesis.given\ -> given(). Finally, you can give
it alternative link text in the usual way:
:func:other text <hypothesis.given>\ -> other text.
- For the formatting and also hyperlinks, all cross-references should use the
Sphinx cross-referencing syntax rather than plain text.
-----------------
Changelog Entries
-----------------
Hypothesis does continuous deployment <https://github.com/HypothesisWorks/hypothesis/issues/555>_,
where every pull request that touches ./src results in a new release.
That means every contributor gets to write their changelog!
A changelog entry should be written in a new RELEASE.rst file inhypothesis
the directory. The first line of the file specifies the componentsemantic versioning <https://semver.org/>
of the version number that will be updated, according to our_ policy.
- RELEASE_TYPE: major is for breaking changes, and will only be used by the
core team after extensive discussion.
- RELEASE_TYPE: minor is for anything that adds to the public (ie documented)
API, changes an argument signature, or adds a new deprecation or health check.
Minor (or patch) releases must not cause errors in any code that runs
without errors on an earlier version of Hypothesis, using only the public API.
Silent errors may be converted to noisy errors, but generally we prefer
to issue a deprecation warning and use the new behaviour if possible.
This stability policy only applies to use of Hypothesis itself, not the
results of user-written tests that use Hypothesis.
- RELEASE_TYPE: patch is for changes that are not visible in the public
interface, from improving a docstring to backwards-compatible improvements
in shrinking behaviour.
This first line will be removed from the final change log entry.
The remaining lines are the actual changelog text for this release,
which should:
- concisely describe what changed and why
- use Sphinx cross-references to any functions or classes mentioned
- if closing an issue, mention it with the :issue: role to generate a link
- finish with a note of thanks from the maintainers:
"Thanks to <your name> for this bug fix / feature / contribution"
(depending on which it is). If this is your first contribution,
don't forget to add yourself to AUTHORS.rst!
---
Guides/Internals
===================================
How to Work on Hypothesis Internals
===================================
This is a guide to how to work on Hypothesis internals,
with a particular focus on helping people who are new to it.
Right now it is very rudimentary and is intended primarily for people who are
looking to get started writing shrink passes as part of our current outreach
program to get more people doing that <https://github.com/HypothesisWorks/hypothesis/issues/1093>_,
but it will expand over time.
------------------------
Bird's Eye View Concepts
------------------------
The core engine of Hypothesis is called Conjecture.
The "fundamental idea" of Conjecture is that you can represent an arbitrary
randomized test case as the sequence of choices made while producing it.
Whenever the test does something "random" it instead draws the next choice from
the sequence and does what it says. Each choice is one of a small number of
typed primitives - an integer, float, boolean, string, or bytestring - and
corresponds to one of the draw_* methods on ConjectureData. We call this
typed sequence the choice sequence (older parts of the codebase and history
may refer to it as the "IR", for intermediate representation).
These choices don't have to come from a PRNG, and we can run the test given
any choice sequence we like. By manipulating the choices, we can achieve more
interesting effects than pure randomness would allow us to do, while retaining
the power and ease of use of random testing. (Historically Conjecture worked
directly on the underlying byte sequence read from the PRNG; it now works on the
typed choice sequence instead, which shrinks far better because we no longer
have to reason about the encoding of each value into bytes.)
The greatest strength of this idea is that we have a single source of truth
for what an example should look like: Every choice sequence is one that could
have been produced by drawing from strategies, and thus is a valid thing to try
for our test. The only ways it can fail to be a valid test input are for it to
be too short or for it to not satisfy one of the test's preconditions, and both
are easily detectable.
The idea of shrinking in particular is that once we have this representation,
we can shrink arbitrary test cases based on it. We try to produce a choice
sequence that is shortlex minimal. What this means is that it has the shortest
possible length and among those sequences of minimal length is the smallest by
comparing choices one at a time from the left (see sort_key inshrinker.py, which orders each choice by choice_to_index).
Ideally we could think of the shrinker as a generic function that takes a
choice sequence satisfying some predicate and returns the shortlex minimal
sequence that also satisfies it.
We depart from this ideal in two ways:
we can only approximate* such a minimal sequence. Finding the actual minimum
is intractable in general.
* we are only interested in minimizing things where the predicate goes through
the Hypothesis API, which lets us track how the data is used and use that to
guide the process.
We then use a number of different transformations of the choice sequence to try
and reduce our input. These vary from principled general transformations to
shameless hacks that special case something we need to work well.
One such example is the handling of floating point numbers. Floats are drawn as
a single choice, but have their own dedicated shrinker (inshrinking/floats.py) which knows about our lexicographic encoding of floats
and tries shrinks - such as dropping fractional digits or moving to a nearby
integer - that only make sense for that encoding.
--------------------------
Useful Files to Know About
--------------------------
The code associated with Conjecture lives in
src/hypothesis/internal/conjecture <https://github.com/HypothesisWorks/hypothesis/tree/master/hypothesis/src/hypothesis/internal/conjecture>_.
There are a number of files in there,
but the most important ones are engine.py and data.py.data.py defines the core type that is used to represent test cases,
and engine.py contains the main driver for deciding what test cases to run.shrinker.py contains the shrinker proper, described below.
There is also the shrinking/ package, which contains a collection of
small, general purpose shrinkers for individual values - Integer,Float, String, Bytes, Collection, and Ordering. Each takes
a single value and a predicate over values and tries to produce a simpler value
satisfying it. Unlike the shrinker in general these don't know anything about
the testing API; the shrinker drives them with a predicate that knows how to
substitute a candidate value back into the full choice sequence and rerun the
test. This is the part of the code that means we can do things like replacing an
integer with a smaller one.
-------
Testing
-------
For general information about how to test Hypothesis, take a look at
the testing guide <testing-hypothesis.rst>_, but there are a couple
of areas that it's worth specifically highlighting for making changes
to the engine:
The first is tests/conjecture/ <https://github.com/HypothesisWorks/hypothesis/tree/master/hypothesis/tests/conjecture>_,
which is a set of unit tests designed to put the engine into particular scenarios to exercise specific behaviours,
with a goal of achieving 100% coverage on it in isolation (though it currently does not quite achieve that for some specific edge cases.
We may fix and enforce this later).
The other set of tests that are worth knowing about are the quality tests,
in tests/quality <https://github.com/HypothesisWorks/hypothesis/tree/master/hypothesis/tests/quality>_.
These assert specific hard to satisfy properties about the examples that Hypothesis finds -
either their existence, or something about the final shrunk result.
-----------------------
Engine Design Specifics
-----------------------
There are a couple of code patterns that are mostly peculiar to Conjecture that
you may not have encountered before and are worth being aware of.
~~~~~~~~~~~~~~~~~~~~
Search State Objects
~~~~~~~~~~~~~~~~~~~~
There are a number of cases where we find ourself with a user-provided function
(where the "user" might still be something that is entirely our code) and we
want to pass a whole bunch of different examples to it in order to achieve some
result. Currently this includes each of the main engine, the Shrinker (inengine.py) and the minimizer, but there are likely to be more in future.
We typically organise such things in terms of an object that you create with
the function and possibly an initial argument that stores these on self and
has some run or similar method. They then run for a while, repeatedly
calling the function they were given.
Generally speaking they do not call the function directly, but instead wrap
calls to it. This allows them to implement a certain amount of decision caching,
e.g. avoiding trying the same shrink twice, but also gives us a place where we
can update metadata about the search process.
For objects whose goal is some form of optimisation (Shrinker, Minimizer) one
of the pieces of metadata they will typically track is a "current target". This
is typically the best example they have seen so far. By wrapping every call to
the predicate, we ensure that we never miss an example even when we're passing
through other things.
For objects whose goal is some broader form of search (currently onlyConjectureRunner) this also allows them to keep track of other examples
of interest. For example, as part of our multiple bug discovery,ConjectureRunner keeps track of the smallest example of each distinct
failure that it has seen, and updates this automatically each time the test
function is called. This means that if during shrinking we "slip" and find a
different bug than the one we started with, we will not shrink to that, but
it will get remembered by the runner if it was either novel or better than our
current example.
~~~~~~~~~~~
Weird Loops
~~~~~~~~~~~
The loops inside a lot of the engine look very strange and unidiomatic. For
example:
.. code-block:: python
i = 0
while i < len(self.shrink_target.nodes):
if not self.consider_new_nodes(
self.shrink_target.nodes[:i] + self.shrink_target.nodes[i + 1 :]
):
i += 1
The more natural way to write this in Python would be:
.. code-block:: python
for i in range(len(self.shrink_target.nodes)):
self.consider_new_nodes(
self.shrink_target.nodes[:i] + self.shrink_target.nodes[i + 1 :]
)
This is not equivalent in this case, and would exhibit the wrong behaviour.
Every time consider_new_nodes succeeds, it changes the shape of the
current shrink target. This consequently changes the shape of the choice
sequence, both its particular values and its current length - on each loop
iteration the loop might stop either because i increases or becauselen(self.shrink_target.nodes) decreases.
We do not reset i to zero on success, as this would cause us to retry deleting
things that we have already tried. This might work, but is less likely to.
In the event that none of the earlier deletions succeed, this causes us to do
retry the entire prefix uselessly, which can result in a pass taking O(n^2) time
to do O(n) deletions.
An additional quirk is that we only increment i on failure. The reason for
this is that if we successfully deleted the choice at position i then the
node now in position i is whatever used to follow it, which is probably the
next thing we would have tried deleting if we hadn't succeeded (or something
like it), so we don't want to advance past it.
This is specific to deletion: If we are just replacing the contents of
something then we expect it to still be in the same place, so there we increment
unconditionally.
Examples of this include try_trivial_spans and minimize_individual_choices.
------------
The Shrinker
------------
The shrinking part of Hypothesis is organised into a single class called Shrinker
that lives in hypothesis/internal/conjecture/shrinker.py.
Its job is to take an initial ConjectureData object and some predicate that
it satisfies, and to try to produce a simpler ConjectureData object that
also satisfies that predicate.
The search process mostly happens in the shrink method, which tries various
shrink passes in the greedy_shrink method and then reports on the outcome.
For details, you are strongly encouraged to read the source code. It is very
well commented, and as the subject of active research often has newer techniques
than are documented here.
~~~~~~~~~~~~~
Search Passes
~~~~~~~~~~~~~
Search passes are methods on the Shrinker class. They are
designed to take the current shrink target and try a number of things that might
be sensible shrinks of it.
Typically the design of a search pass is that it should always try to run to
completion rather than exiting as soon as it's found something good, but that
it shouldn't retry things that are too like stuff it has already tried just
because something worked. So for example in the above loop, we try deleting
each interval (these roughly correspond to regions of the input that are
responsible for some particular value or small number of adjacent values).
When we succeed, we keep going and try deleting more intervals, but we don't
try to delete any intervals before the current index.
The reason for this is that retrying things from the beginning might work but
probably won't. Thus if we restarted every time we made a change we would end
up doing a lot of useless work. Additionally, they are more likely to work
after other shrink passes have run because frequently other changes are likely
to unlock changes in the current pass that were previously impossible. e.g.
when we reorder some examples we might make a big region deletable that
previously contained something critical to the relevant behaviour of the test
but is now just noise.
Because the shrinker runs in a big loop, if we've made progress the shrink pass
will always be run again (assuming we don't hit some limit that terminates the
shrink early, but by making the shrinker better we try to ensure that that
never happens).
This means that we will always get an opportunity to start again later if we
made progress, and if we didn't make progress we've tried everything anyway.
~~~~~~~~~~~~~~~~~~~~~~~
Expensive Shrink Passes
~~~~~~~~~~~~~~~~~~~~~~~
We have a bunch of search passes that are considered "expensive". Typically
this means "quadratic or worse complexity". When shrinking we initially don't
run these, and the first time that we get to the end of our main passes and
have failed to make the input any smaller, we then turn them on.
This allows the shrinker to switch from a good but slightly timid mode while its
input is large into a more aggressive DELETE ALL THE THINGS mode once that stops
working. By that point we've usually made our input small enough that quadratic
complexity is acceptable.
We turn these on once and then they stay on. The reason for this is to avoid a
"flip-flopping" scenario where an expensive pass unlocks one trivial change that
the cheap passes can find and then they get stuck again and have to do an extra
useless run through the passes to prove that.
~~~~~~~~~~~~~~~~~~~~~~
Adaptive Shrink Passes
~~~~~~~~~~~~~~~~~~~~~~
A useful trick that some of the shrink passes use is to try a thing and if it
doesn't work take a look at what the test function did to guess why it didn't
work and try to repair that.
The main example is try_shrinking_nodes, which is used by the passes that
minimize individual choices. When it lowers one or more choices and the result
doesn't satisfy the predicate, it inspects what the test function did:
If the attempt was misaligned* - the test tried to draw a choice of a
different type or size than the one we substituted, which commonly happens
when an earlier choice controls the size of a later collection - it tries to
realign the tree, for example by truncating a string to the size the test
actually asked for.
* Otherwise, if the attempt lost some nodes (e.g. lowering a value made a
following list shorter), it tries deleting the corresponding region of the
choice sequence immediately after the lowered choice to see if that repairs
the shape.
--------------
Playing Around
--------------
I often find that it is informative to watch the shrink process in action using
Hypothesis's verbosity settings. This can give you an idea of what the format
of your data is, and how the shrink process transforms it.
In particular, it is often useful to run a test with the flag -s to tell it
not to hide output and the environment variable HYPOTHESIS_VERBOSITY_LEVEL=debug.
This will give you a very detailed log of what the testing process is running,
along with information about what passes in the shrinker are running and how
they transform it.
---------------
Getting Started
---------------
The best way of getting started on working on the engine is to work on the
shrinker. This is because it has the most well defined problems, the best
documented code among the engine, and it's generally fun to work on.
If you have not already done so, check out Issue #1093 <https://github.com/HypothesisWorks/hypothesis/issues/1093>_,
which collates a number of other issues about shrink quality that are good starting
points for people.
The best place to get started thus is to take a look at those linked issues and
jump in and try things! Find one that you think sounds fun. Note that some
of them suggest not doing these as your first foray into the shrinker, as some
are harder than others.
Please ask questions if you have any - either the main issue for general
purpose questions or specific issues for questions about a particular problem -
if you get stuck or if anything doesn't make sense. We're trying to make this
process easier for everyone to work on, so asking us questions is actively
helpful to us and we will be very grateful to you for doing so.
---
Guides/README
Guides for Hypothesis Development
This is a general collection of useful documentation for people working on Hypothesis.
It is separate from the main documentation because it is not much use if you are merely using Hypothesis. It's purely for working on it, and aimed more at maintainers than casual contributors.
---
Guides/Review
===================================
The Hypothesis Code Review Handbook
===================================
This document outlines the process for reviewing changes to Hypothesis. It's
partly descriptive, partly prescriptive, and entirely prone to change in
response to circumstance and need. We're still figuring this thing out!
-----------------
What Needs Review
-----------------
All changes must be signed off by at least one person
with write access to the repo other than the author of the change.
----------------
How Review Works
----------------
Once the build is green and a reviewer has approved the change, anyone on the
maintainer team may merge the request.
More than one maintainer may review a change if they wish to, but it's
not required. Any maintainer may block a pull request by requesting changes.
Consensus on a review is best but not required. If some reviewers have
approved a pull request and some have requested changes, ideally you
would try to address all of the changes, but it is OK to dismiss dissenting
reviews if you feel it appropriate.
We've not tested the case of differing opinions much in practice yet, so
we may grow firmer guidelines on what to do there over time.
------------
Review Goals
------------
At a high level, the two things we're looking for in review are answers
to the following questions:
1. Is this change going to make users' lives worse?
2. Is this change going to make the maintainers' lives worse?
Code review is a collaborative process between the author and the
reviewer to try to ensure that the answer to both of those questions
is no.
Ideally of course the change should also make one or both of the users'
and our lives better, but it's OK for changes to be mostly neutral.
The author should be presumed to have a good reason for submitting the
change in the first place, so neutral is good enough!
--------------
Social Factors
--------------
* Always thank external contributors. Thank maintainers too, ideally!
* Remember that the Code of Conduct <https://hypothesis.readthedocs.io/en/latest/community.html#code-of-conduct>_
applies to pull requests and issues too. Feel free to throw your weight
around to enforce this if necessary.
* Anyone, maintainer or not, is welcome to do a code review. Only official
maintainers have the ability to actually approve and merge a pull
request, but outside review is also welcome.
------------
Requirements
------------
The rest of this document outlines specific things reviewers should
focus on in aid of this, broken up by sections according to their area
of applicability.
All of these conditions must be satisfied for merge. Where the reviewer
thinks this conflicts with the above higher level goals, they may make
an exception if both the author and another maintainer agree.
~~~~~~~~~~~~~
Orthogonality
~~~~~~~~~~~~~
For all minor or patch releases, we enforce a hard and fast rule that they
contain no more than one user-visible change. Major releases are allowed
to bundle multiple changes together, but these should be structured as
smaller pull requests into some tracking branch.
We are currently very bad at this, so reviewers should feel empowered
to be extra strict and provide a lot of push back on this.
What counts as a user visible change is somewhat up to individual
judgement, but you should err in the direction of assuming that
if it might count then it does count.
A good rule of thumb is that if the RELEASE.rst uses the words "additionally"
or needs bullet points to be clear, it is likely too large.
Ideally changes that are not user visible should also be self-contained
into their own releases, but a certain amount of leniency is permitted -
it's certainly OK to do a moderate amount of refactoring while you're
in the area, and if a pull request involves no release at all then the same
level of orthogonality is not required (but is still desirable).
~~~~~~~~~~~~~~~~~~~~~~
Clarity of Description
~~~~~~~~~~~~~~~~~~~~~~
The RELEASE.rst should contain a description of the change that
makes clear:
1. The motivation for the change
2. The likely consequences of the change
This doesn't have to be an essay. If you're following the orthogonality
requirements a paragraph or two is likely sufficient.
Any additional information that is useful to reviewers should be provided
in the pull request comment. This can include e.g. background, why the
particular approach was taken, references to internals that are unlikely
to be of interest to users.
~~~~~~~~~~~~~~~~~~~~~
Functionality Changes
~~~~~~~~~~~~~~~~~~~~~
This section applies to any changes in Hypothesis's behaviour, regardless
of their nature. A good rule of thumb is that if it touches a file in
src then it counts.
1. The code should be clear in its intent and behaviour.
2. Behaviour changes should come with appropriate tests to demonstrate
the new behaviour.
3. Hypothesis must never be flaky. Flakiness here is
defined as anything where a test fails and this does not indicate
a bug in Hypothesis or in the way the user wrote the code or the test.
4. The changelog (in RELEASE.rst) should bump the minor or patch version
(see guides/documentation.rst for details), accurately describe the
changes, and shouldn't refer to internal-only APIs. For complicated
markup, consider building the docs and manually checking the changelog
for formatting errors that didn't result in a compilation error.
~~~~~~~~~~~
API Changes
~~~~~~~~~~~
Public API changes require the most careful scrutiny of all reviews,
because they are the ones we are stuck with for the longest: Hypothesis
follows semantic versioning, and we don't release new major versions
very often.
Public API changes must satisfy the following:
1. All public API changes must be well documented. If it's not documented,
it doesn't count as public API!
2. Changes must be backwards compatible. Where this is not possible, they
must first introduce a deprecation warning, then once the major version
is bumped the deprecation warning and the functionality may be removed.
3. If an API is deprecated, the deprecation warning must make it clear
how the user should modify their code to adapt to this change (
possibly by referring to documentation).
If the required code change could be automated, the deprecation should have either
a codemod to fix it <https://github.com/HypothesisWorks/hypothesis/issues/2705>__DRMacIver <https://github.com/DRMacIver>
or a tracking issue to write one (see "asking for more work" below).
4. If it is likely that we will want to make backwards incompatible changes
to an API later, to whatever extent possible these should be made immediately
when it is introduced instead.
5. APIs should give clear and helpful error messages in response to invalid inputs.
In particular error messages should always display
the value that triggered the error, and ideally be specific about the
relevant feature of it that caused this failure (e.g. the type).
6. Incorrect usage should never "fail silently" - when a user accidentally
misuses an API this should result in an explicit error.
7. Functionality should be limited to that which is easy to support in the
long-term. In particular functionality which is very tied to the
current Hypothesis internals should be avoided.
8. _ orZac-HD <https://github.com/Zac-HD>
_ must approve the changeshouse API style <api-style.rst>
though other maintainers are welcome and likely to chip in to review as
well.
9. We have a separate guide for _ which should
be followed.
~~~~~~~~~
Bug Fixes
~~~~~~~~~
1. All bug fixes must come with a test that demonstrates the bug on master and
which is fixed in this branch. An exception may be made here if the submitter
can convincingly argue that testing this would be prohibitively difficult.
2. Where possible, a fix that makes it impossible for similar bugs to occur is
better.
3. Where possible, a test that will catch both this bug and a more general class
of bug that contains it is better.
~~~~~~~~~~~~~~~~
Settings Changes
~~~~~~~~~~~~~~~~
Note: This section currently only applies to the Python version.
It is tempting to use the Hypothesis settings object as a dumping ground for
anything and everything that you can think of to control Hypothesis. This
rapidly gets confusing for users and should be carefully avoided.
New settings should:
1. Be something that the user can meaningfully have an opinion on. Many of the
settings that have been added to Hypothesis are just cases where Hypothesis
is abdicating responsibility to do the right thing to the user.
2. Make sense without reference to Hypothesis internals.
3. Correspond to behaviour which can meaningfully differ between tests - either
between two different tests or between two different runs of the same test
(e.g. one use case is the profile system, where you might want to run Hypothesis
differently in CI and development). If you would never expect a test suite to
have more than one value for a setting across any of its runs, it should be
some sort of global configuration, not a setting.
When deprecating a setting for later removal, we prefer to change the default
value of the setting to a private singleton (not_set), and implement the
future behaviour immediately. Passing any other value triggers a deprecation
warning, but is otherwise a no-op (i.e. we still use the future behaviour).
For settings where this would be especially disruptive, we have also prefixed
that deprecation process with a process where we emit a warning, add a special
value that can be passed to opt-in to the future behaviour, and then in the
following major release we deprecate that, make it an no-op, and make it an
error to pass any other value.
~~~~~~~~~~~~~~
Engine Changes
~~~~~~~~~~~~~~
Engine changes are anything that change a "fundamental" of how Hypothesis
works. A good rule of thumb is that an engine change is anything that touches
a file in hypothesis.internal.conjecture (Python version).
All such changes should:
1. Be approved (or authored) by DRMacIver or Zac-HD.
2. Be approved (or authored) by someone who isn't DRMacIver (a major problem
with this section of the code is that there is too much that only DRMacIver
understands properly and we want to fix this).
3. If appropriate, come with a test in test_discovery_ability.py showing new
examples that were previously hard to discover.
4. If appropriate, come with a test in test_shrink_quality.py showing how they
improve the shrinker.
~~~~~~~~~~~~~~~~~~~~~~
Non-Blocking Questions
~~~~~~~~~~~~~~~~~~~~~~
These questions should not block merge, but may result in additional
issues or changes being opened, either by the original author or by the
reviewer.
1. Is this change well covered by the review items and is there
anything that could usefully be added to the guidelines to improve
that?
2. Were any of the review items confusing or annoying when reviewing this
change? Could they be improved?
3. Are there any more general changes suggested by this, and do they have
appropriate issues and/or pull requests associated with them?
~~~~~~~~~~~~~~~~~~~~
Asking for more work
~~~~~~~~~~~~~~~~~~~~
Reviewers should in general not request changes that expand the scope of
a pull request beyond its original intended goal. The primary design
philosophy of our work-flow is that making correct changes should be cheap,
and scope creep on pull requests works against that - If you can't touch
something without having to touch a number of related areas as well,
changing things becomes expensive again.
This of course doesn't cover things where additional work is required to
ensure the change is actually correct - for example, if you change public
functionality you certainly need to update its documentation. That isn't
scope creep, that's just the normal scope.
If a pull request suggests additional work then between the reviewer and the
author people should ensure that there are relevant tracking issues for that
work (as per question 3 in "Non-Blocking Questions" above), but there is no
obligation for either of them to actually do any of the work on those issues.
By default it is the reviewer who should open these issues, but the author
is welcome to as well.
That being said, it's legitimate to expand the scope of a pull request in
some cases. For example:
* If not doing so is likely to cause problems later. For example, because
of backwards compatibility requirements it might make sense to ask for some
additional functionality that is likely to be added later so that the arguments
to a function are in a more sensible order.
* Cases where the added functionality feels extremely incomplete in some
way without an additional change. The litmus test here should be "this will
almost never be useful because...". This is still fairly subjective, but at
least one good use case where the change is a clear improvement over the status
quo is enough to indicate that this doesn't apply.
If it's unclear, the reviewer should feel free to suggest additional work
(but if the author is someone new, please make sure that it's clear that this
is a suggestion and not a requirement!), but the author of the pull request should
feel equally free to decline the suggestion.
---
Guides/Strategies That Shrink
===================================
Designing strategies to shrink well
===================================
Reducing test cases to a minimal example is a great feature of Hypothesis,
the implementation of which depends on both the shrinking engine and the
structure of the strategy (or combination of strategies) which created the
example to reduce.
This document is organised into three parts:
1. How to tell if you need to think about shrinking (you probably don't!)
2. Designing for shrinking 'above' the Hypothesis public API
3. Implementation tricks used in our internals, for interested contributors
It is written for people implementing complex third-party strategies (such
as hypothesis-networkx <https://pypi.org/project/hypothesis-networkx/>__),
current or potential contributors to Hypothesis itself, and anyone interested
in how this works under the hood.
------------------------------------
Do you need to design for shrinking?
------------------------------------
You should only attempt to tune custom strategies for better shrinking
behaviour if more time would otherwise be spent reducing examples by hand
or debugging more complex examples. It may be worthwhile if:
- Your custom strategy will be used by many people, so that spending
the same effort tuning the strategy has much larger benefits, or
- You have personally spent time debugging failures which better example
shrinking could have avoided and think this might happen again.
If neither of these apply to you, relax! Hypothesis' test-case reduction
is among the best in the world, and our built-in strategies are carefully
designed to work well with it as discussed below.
------------------------------------
Shrinking for third-party strategies
------------------------------------
That is, strategies built out of other strategies until you get down to
Hypothesis' public API. These often but not always use @composite.
Composition of shrinking
~~~~~~~~~~~~~~~~~~~~~~~~
The first and most important rule is that Hypothesis shrinks from the
'bottom up'. If any component of your strategy is replaced with a simpler
example, the end result should also become simpler. We usually try to define
"simpler" here to match a reasonable intuition about the strategy, and avoid
weird edge cases when it's combined with another strategy or predicate.
Issue #1076 <https://github.com/HypothesisWorks/hypothesis/issues/1076>_,
where magnitude constraints were added to the complex_numbers strategy,
makes a nice case study. We wanted to continue shrinking the real and
imaginary parts like builds(complex, floats(), floats()).
In a worst-case scenario, the performance of filtering could be arbitrarily
bad, while a 'generate and scale' approach would mean that simple inputs
could lead to irrational outputs. Instead, we choose an imaginary part
between +/- max_magnitude, then calculate the resulting bounds on the real
part and draw it from a strategy that will always be valid. This ensures
that the imaginary part shrinks to zero first, as we think real-valued
complex numbers are simpler than imaginary-valued complex numbers.
Let generation be lucky
~~~~~~~~~~~~~~~~~~~~~~~
Sometimes, it's worth searching for a particularly nasty value to try.
This trick should be used sparingly, and always behind a branch that the
shrinker can decide not to take such as if draw(booleans()):, but might
occasionally worth trying. Measure the results before you keep it!
Issue #69 <https://github.com/HypothesisWorks/hypothesis/issues/69>_ provides
a nice case study: when generating tz-aware datetimes, we would like to generate
instants that are skipped or repeated due to a daylight-savings transition more
often than by chance. Of course, there may or may not be any such moments
allowed by the bounds and tz strategy!
Eliding much of the detail, a key part is to find such a moment between two
endpoints, when we can only check whether one or more exists. The traditional
approach would be to use a binary search, but this would be relatively expensive
to shrink as we would pay the log-n cost on every attempted shrink.
Instead of choosing the midpoint, we draw a random point between our known
endpoints, and repeat this until we find a satisfactory moment. This allows
the shrinker to delete all the intermediate draws - and appear lucky enough
to find the moment we were looking for on the first guess!
Keep things local
~~~~~~~~~~~~~~~~~
Hypothesis' shrinking engine sees every example as a labelled tree of choices,
with possible reductions represented as operations on the tree. An attempted
shrink succeeds if the new tree can be converted into an example, and the
resulting example triggers the same bug in the test function.
The most common way we see users breaking data locality is by drawing a size,
then drawing a collection of that size. This is tempting because it's simple
and it _works_, but it's often much slower than the alternatives.
.. code:: python
# Both of these strategies can generate exactly the same kind of examples,
# but the second has better performance as well as style.
integers(0, 10).flatmap(lambda n: st.lists(..., min_size=n, max_size=n))
st.lists(..., min_size=1, max_size=10)
Another easy way to keep things local is to ensure that any .filter(...)
or assume(...) calls you use are as close as possible to the relevant part
of the strategy. That way, Hypothesis can retry just the part that failed
instead of the entire strategy, which might be much slower.
For efficient shrinking, local operations on the tree should correspond with
valid (and preferably local) shrinks to the final example. For example:
.. code:: python
# This form of loop is hard to shrink, because we'd have to reduce n and.flatmap
# delete something in the loop simultaneously. It's equivalent to the
# example above. We _do_ shrink this, but much more slowly.
n = draw(integers(0, 10))
for _ in range(n):
...
draw(...)
...
# In this form, the shrinker can see a repeated structure of labels
# and delete one loop iteration without touching anything else.
# We use a variant of this trick to generate collections internally!
while draw(integers(0, x)) > threshold:
...
draw(...)
...
Similarly, it's better to draw all the attributes or inputs you need for an
object at the same time, again so they can be modified or deleted together.
The exact behaviour of the shrinking is a topic of active research and
development, so if you are interested in the details we recommend reading
the internals guide <https://github.com/HypothesisWorks/hypothesis/blob/master/guides/internals.rst>_
and the well-commented source code inhypothesis.internal.conjecture as well as David's ECOOP 2020 paperTest-Case Reduction via Test-Case Generation: Insights From the Hypothesis Reducer
<https://2020.ecoop.org/details/ecoop-2020-papers/13/Test-Case-Reduction-via-Test-Case-Generation-Insights-From-the-Hypothesis-Reducer>__.
-------------------------------------
Shrinking in the Hypothesis internals
-------------------------------------
The last section is for current or prospective Hypothesis contributors only.
These tricks rely on implementation details that are not available to
third-party libraries or users, and can change in any patch release.
Occasionally they are also indispensable to get good performance in underlying
primitives, so please contact us if the public API is not enough and we may
be able to work something out.
What do internals get you?
~~~~~~~~~~~~~~~~~~~~~~~~~~
Using the low-level, internal APIs complements, rather than changing, the
principles above. The bytestream-level view has some important advantages:
Because we operate at the level of bits, the relationship between a value and
the corresponding buffer is much more obvious. If we're careful, that means
we can calculate the value we want and then write the corresponding buffer
to recreate it when the test case is shrunk or replayed.
A small step up from bits, we can also see the spans that indicate a subset
of the buffer to consider for various transformations such as transposition
or deletion.
Sometimes these features are the only way to maintain acceptable performance
in very rare or even pathological cases - consider shrinking a complex number
with a single allowed magnitude - but it's almost certain that someone will
need the core strategies to do just that.
However, using low-level APIs also comes at a cost - they are verbose and
generally more difficult to use, and can violate key invariants of the engine
if misused.
Internally, our strategies mostly use the public API or something that looks
a lot like @composite, so it's fairly easy to follow along. There are
just a few tricks enabled by those low-level advantages that we wanted to
name and document, so we can recognise them discuss them and invent more...
Make your own luck
~~~~~~~~~~~~~~~~~~
This is the simplest trick that uses our ability to write choices to the
buffer. We use it for sampled_from(...).filter(...), after trying an
initial draw with the usual rejection sampling technique, and added theSearchStrategy.do_filtered_draw method so other strategies can opt-in
as we design similar tricks for their structure.
It was originally designed for stateful testing, where "lucky generation"
might be inefficient if there are many rules but only a few allowed by their
preconditions. Here's how it works for stateful testing:
1. Draw an index into the unfiltered list of rules. Return the corresponding
rule if it's allowed - we got lucky! (or someone set us up...)
2. Create a list of allowed rules, and choose one from that shortlist instead.
3. Find the index of the chosen rule in the unfiltered list, and write that
index to the buffer. Finally, return the chosen rule.
When the shrinker tries to delete the first two draws, the resulting buffer
will lead to the same rule being chosen at step one instead. We've made
our own luck!
This trick is especially useful when we want to avoid rejection sampling
(the .filter method, assume) for performance reasons, but also
need to give the shrinker the same low-level representation for each instance
of a repeated choice.
Flags "shrink open"
~~~~~~~~~~~~~~~~~~~
An important insight from Swarm Testing (PDF) <https://www.cs.utah.edu/~regehr/papers/swarm12.pdf>__
is that randomly disabling some features can actually reduce the expected time
before finding a bug, because some bugs may be suppressed by otherwise common
features or attributes of the data.
As discussed on issue #1401 <https://github.com/HypothesisWorks/hypothesis/issues/1401>__,
there are a few points to keep in mind when implementing shrinkable swarm testing:
- You need swarm flags to "shrink open" so that once the shrinker has run to
completion, all flags are enabled. e.g. you could do this by generating a
set of banned flags.
- You need to use rejection sampling rather than anything more clever, or at
least look like it to the shrinker. (see e.g. Make your own luck, above)
Taking Unicode as an example, we'd like to use our knowledge of Unicode
categories to generate more complex examples, but shrink the generated string
without reference to categories. While we haven't actually implemented this
yet - it's pretty hairy - the simple version of the idea goes like this:
1. Generate a set of banned categories.
2. Use characters().filter(category_is_not_banned)
When shrinking, we start by removing categories from the banned set, after
which characters in the string can be reduced as usual. In a serious version,
the make-your-own-luck approach would be essential to make the filter
reasonably efficient, but that's not a problem internally.
In more complicated structures, it would be nice to generate the flags on first
use rather than up front before we know if we need them. The trick there is
to write each flag to the buffer every time we check it, in such a way that if
we delete the first use the second turns into an initialisation.
Explicit example boundaries
~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is almost always handled implicitly, e.g. by cu.many, but sometimes
it can be useful to explicitly insert boundaries around draws that should be
deleted simultaneously using data.start_span. This is used to group
the value and sign of floating-point numbers, for example, which we split up
in order to provide a more natural shrinking order.
Explicit example management can also be useful to delineate variably-sized
draws, such as our internal helper cu.biased_coin, which makes eliminating
dead bytes much cheaper. Finally, labelling otherwise indistinguishable draws
means the shrinker can attempt to swap only the like values.
---
Guides/Testing Hypothesis
==================
Testing Hypothesis
==================
Note: This guide is currently entirely specific to the Python version of
Hypothesis.
This is a guide to the process of testing Hypothesis itself, both how to
run its tests and how to write new ones.
--------------------------
General Testing Philosophy
--------------------------
The test suite for Hypothesis is unusually powerful - as you might hope! -
but the secret is actually more about attitude than technology.
The key is that we treat any bug in Hypothesis as a bug in our test suite
too - and think about the kinds of bugs that might not be caught, then write
tests that would catch them.
We also use a variety of tools to check our code automatically, including
formatting, import order, linting, and typing our API with Mypy.
All of this is checked in CI - which means that once the build is
green, humans can all focus on meaningful review rather than nitpicking
operator spacing.
Similarly, we require all code to have tests with 100% branch coverage - as
a starting point, not the final goal.
- Requiring full coverage can't guarantee that we've written all the tests
worth writing (for example, maybe we left off a useful assertion about the
result), but less than full coverage guarantees that there's some code we're
not testing at all.
- Tests beyond full coverage generally aim to demonstrate that a particular
feature works, or that some subtle failure case is not present - often
because when it was found and fixed, someone wrote a test to make sure it
couldn't come back!
The hypothesis/tests/ directory has some notes in the README file on where various
kinds of tests can be found or added. Go there for the practical stuff, or
just ask one of the maintainers for help on a pull request!
Further reading: How SQLite is tested <https://sqlite.org/testing.html>_,how the Space Shuttle was tested <https://www.fastcompany.com/28121/they-write-right-stuff>_,how to misuse code coverage <http://www.exampler.com/testing-com/writings/coverage.pdf>_fuzz testing <https://danluu.com/testing/>
(for inspiration, not implementation).
Dan Luu writes about _ andbroken processes <https://danluu.com/wat/>_, among other things.
-------------
Running Tests
-------------
Tests are run via build.sh. See CONTRIBUTING.rst` for more details.
---
README
<div align="center">
<img src="https://raw.githubusercontent.com/HypothesisWorks/hypothesis/master/brand/dragonfly-rainbow.svg" width="300">
</div>
Hypothesis
* Website
* Documentation
* Source code
* Contributing
* Community
Hypothesis is the property-based testing library for Python. With Hypothesis, you write tests which should pass for all inputs in whatever range you describe, and let Hypothesis randomly choose which of those inputs to check - including edge cases you might not have thought about. For example:
from hypothesis import given, strategies as st
@given(st.lists(st.integers()))
def test_matches_builtin(ls):
assert sorted(ls) == my_sort(ls)
This randomized testing can catch bugs and edge cases that you didn't think of and wouldn't have found. In addition, when Hypothesis does find a bug, it doesn't just report any failing test case — it reports the simplest possible one. This makes property-based tests a powerful tool for debugging, as well as testing.
For instance,
def my_sort(ls):
return sorted(set(ls))fails with the simplest possible failing test case:
Failing test case: test_matches_builtin(ls=[0, 0])Installation
To install Hypothesis:
pip install hypothesisThere are also optional extras available.
---