How to split test like a 17th Century Natural Philosopher

I’ve read everything Neal Stephenson has written up to but not including Seveneves. Yes, I even read The Baroque Cycle cover to cover. In fact, there’s a part of Quicksilver that I constantly refer back to.

It's about the early days of the Royal Society, featuring such legendary characters as Hooke, Boyle and Newton. It’s the history part of his historical fiction, based on the actual annals of the Royal Society.

And it shows how very different real science was from how we were taught it in school.

In school, it seems as if each discovery, each law is built cleanly and logically on the last. Hooke's Law, Boyle's Law, Newton's Laws of Motion...

I remember trying cack-handed versions of key experiments with springs, dubious weights and battered wooden rulers. But it sort of never worked properly and our results were all over the place. So we calculated what our graph of results should have been based on the formula, then added in a couple of anomalous results to make it look truthy.

That’s what we did at school. That’s not science.

When Neal Stephenson brings the early scientific process to life, it becomes clear that there was no such thing as a simple, logical progression.  

Setting off a cannon next to a deaf man’s head to clear the blockage in his ears... Drinking mercury by the tumbler to fortify their health... Dissecting a live dog to try to spot the precise moment when the “vital force” left its body... Newton was even mad keen on alchemy. Alchemy!

These scientists were batshit crazy. They tried anything and everything. I wouldn’t be surprised if they actually ate the batshit.

It was through all the messy, wild experimentation and play that they found the important results that we still revere today. 

I didn’t realise any of this when I started applying science to design through tools like split testing or user testing. I found myself shocked by how messy it can be; dismayed by how difficult it is to tease out important insights.

But all the split testing reports I read sounded so clean and simple! Take a look at some – they strip away all the digging through dirt leaving only the seam of gold. There are three reasons for this:

  1. Human memory is crap – when we’ve got to the insights, our brain re-evaluates all the data and makes it fit or bins it.
  2. Case studies are about telling an elegant story. All the messy stuff would be confusing and create Chekov’s revolvers that would never get fired.
  3. Most case studies are marketing documents. That makes it hard to say, “We didn’t really know what we were doing, but after a load of barking up the wrong tree we got there in the end!”

So take heart when you find yourself digging through dirt with no gold in sight. It’s probably right underneath your nose.

Try being a bit less like a clinical case study and a bit more like the Royal Society.