Innovate, but still evaluate and keep it randomised!

By Professor Shaun Treweek - on 26/01/2016

Ahead of the launch of the Standards of Evidence, Professor Shaun Treweek talks about the benefits of Randomised Controlled Trials and how they can be used in the housing sector.

The British Navy’s greatest enemy for hundreds of years was not a foreign power but scurvy.  Commander George Anson’s four year expedition around the globe in search of, among other things, Spanish loot, set off with 2,000 men in 1740 and returned with 600 in 1744.  Four of the dead were killed in action; all but a few of the rest were felled by scurvy and vitamin deficiencies. 

Suggestions for cures by the medical great and good were plentiful: try mirth and having a cheerful nature, or how about sweating, bloodletting, purging, smoking, burning tar, drinking vinegar, cider or sulphuric acid, bathing in the blood of beasts, or consuming lots of sea water.  At least one man was buried up to his neck in sand to cure his scurvy.  There was certainly innovation.

Evaluation, however, was in short supply.  It seems hard to believe but until 1747 there had been no attempt to rigorously test any of the suggested cures for scurvy despite it being the single biggest killer of sailors for centuries.  When a Scottish naval surgeon called James Lind finally did a test in 1747, he found that oranges and lemons were spectacularly effective, cider was somewhat effective and the remaining four treatments he evaluated were completely useless.  The rest, as they say, is history and nicely described in David Harvie’s book ‘Limeys’  

Well, not quite.  Lind was testing treatments where one, oranges and lemons, had a huge beneficial effect and the others had little or no effect.  He was lucky because the treatment effect was much greater than the effect other differences between the men (e.g. how sick or old they were) might have had on the outcome.  Such huge treatment effects are rare, most effects are small and are easily lost in the noise.  What Lind didn’t do was allocate men to treatments in a way that evenly spread out all the things that might affect the outcome, he just picked men for each treatment. 

The best way of fair allocation is randomisation, effectively tossing a coin.  The great advantage of randomisation is that it not only spreads around things you think might affect the outcome, such as how sick people are, but the things you don’t know about.  As Donald Rumsfeld might put it, randomisation evenly distributes everything including the unknown unknowns.    This is why healthcare makes such extensive use of randomised trials to evaluate new treatments and therapies; it’s the best design for creating groups of people who are the same apart from the treatments being tested.  Not doing trials can lead to tragedy: think of birth defects due to thalidomide, or increased cot deaths because of advice to avoid putting babies to sleep on their backs. 

So are randomised trials relevant to housing?  Absolutely and for the same reasons as healthcare: they are the best way of creating groups of people who are the same apart from the innovation being tested.  And unless the effect of that innovation is as dramatic and as clear as Lind’s oranges and lemons, those equal groups are crucial to being able to pick out an effect from the noise.        

So innovation is good but it needs to be paired with rigorous evaluation.  A recent trial done in Scottish football clubs (see also FFIT) found that a 12-week in-club program aimed at middle-aged men led to weight loss and lots of other beneficial changes. The clubs now routinely run the program; it was also recently introduced at Southampton FC.

Investing resources in the program is a sound decision because a large, randomised trial done in real football club settings showed that the program was effective.  But would the clubs be investing time and money without that trial evidence?  I doubt it and they would be right not to.  As James Lind said in 1747 – ‘I shall propose nothing dictated merely from theory; but shall confirm all by experience and facts, the surest and most unerring guides.‘  Good advice for all would-be innovators.   

Tags: 
RCT

Add new comment