Complex policy problems can prompt many different interventions, often from many different organisations, all at once. Uncontrolled, uncoordinated, and often unstructured trials make it hard to unpick the effects of interventions, or to make informed decisions about which trials to scale up and which to wind up. Even after extensive trials, big decisions are made based on hypotheses rather than evidence because the trials were random and uncontrolled.
Many substantial public investments are based on limited, weak, or mixed evidence. Some longstanding programs are just assumed to be effective, especially when vocal stakeholders believe in their value. It can be difficult for leaders to admit that the programs they invest in may not be effective, especially if better alternatives have not yet been found. Doing something is usually assumed to be better than doing nothing, even if the benefits are modest, or unknown.
Engaged and motivated policy workers and practitioners can generate or adapt many ideas quickly, but testing those ideas is often time and resource intensive. Robust real-world experiments can be seen as cold, unethical, or inequitable, especially if people are randomised to different treatment groups.
When the relative value of investments cannot be determined, programs that are inefficient, ineffective, or both continue to be funded at the expense of scaling up better programs, or trialling promising new alternatives. Poorly managed trials run on and on, regardless of their merits, becoming more entrenched and harder to exit over time as habit and familiarity are accepted as substitutes for robust evidence.
By trying to avoid accusations of being cold, unfeeling, or unfair, publicly funded organisations that shy away from robust trial designs risk locking in worse experiences and outcomes for everyone. Without reliable information about their real-world effects, programs can do worse than just being ineffective or inefficient; they can harm the people they should help.
Policy makers can learn from robust approaches to experimentation in other domains. Clinical trials are expected in the development of drug and medical treatment protocols. Some private sector investment models, like venture capitalism, also make controlled short-term investments expecting that not every experiment will succeed, and some will be abandoned. Adopting a similar approach to policy responses requires visibility and coordination of investment decisions across different programs and organisations.
Appropriate ethical frameworks balance risks for individuals against aggregate outcomes, whether an experiment is relatively low risk and simple, like randomising how information is presented to test which is most effective, or complex, like randomly assigning vulnerable citizens to different programs.
Rigorous experimentation helps to distinguish high value investments from those that are less effective, or even potentially harmful. This, in turn, give decision makers reliable evidence to continue or scale up investment in programs that work, while discontinuing programs that do not to reinvest in more promising alternatives. Objective appraisals of relative cost-benefit and outcomes offer more robust evidence than post-hoc evaluations of uncontrolled programs, which tend to focus on implementation and how to improve a program, rather than whether it should end.
Coordinated investments across organisations, and robust ethical frameworks for controlled trials, improve aggregate outcomes while managing risks to individual citizens. Randomisation should be a tool used within controlled trials, not a policy investment approach.