What is property testing?

Property testing is a style of testing where you state various properties about your component under test and write a test for it. Sounds like a normal test right? Not quite, the big thing about property testing is that you test it with a whole bunch of randomly generated cases based on the inputs you specify. For example, if your function took an array of integers, a property test would test your function using many different arrays of different sizes and contents, like fuzz testing but on smaller scale. Lets make this more concrete:

The canonical example of property testing is checking that addition is commutative, that is, x + y == y + x. That’s the property. Our test would gen generate many, many different integers for x and y to test this property. Rust has a great crate for this called proptest. In proptest, we’d write this test as:

use proptest::prelude::*;

proptest! {
        #[test]
        fn addition_is_commutative(x in any::<i32>(), y in any::<i32>()) {
                assert_eq!(x + y, y + x);
        }
}

This says to run the test with x and y as anything of type i32, a signed integer stored in 32 bits. An interesting feature of the proptest crate is shrinking. This feature is in many property testing libraries and what it does is try to narrow down a failing case to the minimal example.

The usual benefits

Property testing is talked about all over the place, notably it has a section in The Pragmatic Programmer¹. The main touted beneifits are that it lets you write tests thinking about the invariants the unit under test must uphold. In the example above, we’re testing that addition is commutative, we don’t need to think about the inputs and instead generalise it across the whole sample space. By itself this is a good enough reason to write property tests, but there is another reason that I haven’t heard talked about much but I think is also very beneficial: No invalid states in types!

No invalid states in types!

What I mean by this is that property testing encourages you to create types that cannot be constructed in an invalid state. For example, the quick easy way might be to have a single structure with optional fields where you enforce the fact they’re only optional in certain contexts in code. In property testing you’re programmatically generating these structs as inputs so if there are invalid constructions it’s a bit of a pain because you have to filter them out.

As a concrete example of this, in one of my projects unce², a DNS resolver, there is a struct representing a resource record³. Initially I had stored the rdlength which the number of byes in the data section of the resource record. This is useful in the DNS wire format to specify how long the field you’re about to read will be but isn’t useful to store in the struct I was using to represent the resource record because we can derive this information from the data field - the data is denormalised. When I was writing property tests for the DNSResourceRecord struct (what I used to represent the resource record) I initially had to write the code that generates it in such a way that the rdlength field was based off the generated rdata field. Removing the field all together simplified the struct generation for the tests and meant that it was impossible to construct an invalid DNSResourceRecord. I no longer had to make sure the two fields, rdata and rlength, were in sync. By parsing the data into the struct I knew it was valid - parse don’t validate⁴.

In some circumstance you might want to allow the construction of invalid types, perhaps denormalising your structs is necessary for performance reasons or it would make the interface difficult to use (sometimes the case with the builder pattern). If you do have to do this, think about how you can at only expose an interface that allows the construction of valid types, even if, underneath, it’s possible for the type to be invalid.