Predictiv - Observatory of Public Sector Innovation

Predictiv is a web platform that enables governments to test whether new policies and interventions work with an online population before they are deployed in the real world. The tests take 1 to 2 weeks to complete, enabling policymakers and Ministers to get answers to questions that would have taken months (or years) to answer in the past.

Innovation Summary

Innovation Overview

Policymakers want rapid answers to questions about the best way of putting in place a new intervention or policy before it goes live. In the past, this might have involved months if not years of research. With Predictiv, we are able to answer critical policy questions in days.

Predictiv is an online platform that enables governments to run randomised controlled trials with an online population of participants. This enables policymakers to test different versions of interventions, policies, or campaigns that they are about to run, to see which one is most effective at achieving the desired outcome.

Predictiv has been used by government departments and agencies to test what the best way of encouraging people to make financially sound choices might be; which letters they are about to send out are easiest to understand; and which food labels are most likely to encourage people to make low carbon choices. Predictiv has also been used to support the UK Government Equality Office encourage more fathers to take up Shared Parental Leave. They were finding that the current information provided was too complex for people to understand, so they used Predictiv to test two new versions of the information being provided; one which simplified the messages, and one which expressed the Shared Parental leave as an entitlement.

Predictiv enabled the Government Equality Office to run a trial involving people who were in a relationship and were planning to have children in the next four years and the results were available within two weeks of starting the trial. The trial showed that both of the new messages were much more effective than the current information being provided, and that the best performing message was the version expressed as an entitlement. The government Equality Office and the Department for Work and Pensions are now using the results to change the way that the communicate Shared Parental leave to parents. In each of these cases and more, policymakers wanted to be sure that the version of the policy, letter or campaign that they were about to roll out was likely to be the most effective. But they had no means, in the time available, of answering that question without Predictiv. Predictiv enables policymakers to upload different versions of the letter, policy or campaign, and these are then randomly allocated to individuals through online panels – enabling us to reach hundreds of thousands of individuals.

What sets Predictiv apart from other forms of research is the speed at which the tests can be done. If, for example, a Minister asks a question about whether it might not be better to change a policy, these suggestions can be tested in a matter of days to see whether they do indeed have the desired effect.

Innovation Description

What Makes Your Project Innovative?

Predictiv enables organisations to run high-quality research projects, in a fraction of the time that it normally takes. There are alternative ways of answering the kind of questions that Predictiv can answer, but they take much longer (often months or years, rather than one to two weeks), cost much more, and are often not as rigorous. One option, for example, would be to run a trial ‘in the field’ (i.e. in the real world policy environment), something which the Behavioural Insights Team does regularly. The main drawback with field experiments, however, is that they take time to complete and require a lot of upfront work with the organisations that are implementing them. They also cost a lot more to run – a result of the time and resources required to run them. Predictiv enables policymakers to get answers quickly, in the kind of time frames that are often critical to make changes to policy as it is being developed.

What is the current status of your innovation?

Predictiv is now a fully functioning web platform that government departments, or any institution wishing to pursue social purpose research, can use to test what works with an online panel of participants. Over the coming months, our aim is to make the platform increasingly automated, by building out a set of standard ‘templates’ that cover the full range of questions that policymakers might want to answer. We are also now able to make Predictiv available internationally, by plugging the platform into international panels. In theory, wherever there is a sufficiently large online panel, we can run Predictiv trials.

Innovation Development

Collaborations & Partnerships

Every Predictiv trial results in new partnerships, covering all kinds of different areas. Our most involved partner has been Money Advice Service, with whom the Behavioural Insights Team has established a ‘Financial Capability Lab’. The aim of the Financial Capability Lab is to generate new ideas to address some of the most complex and difficult financial challenges facing people across the UK. When these ideas are developed, they are tested with Predictiv.

Users, Stakeholders & Beneficiaries

Our partners are typically involved by helping to generate the ideas that will be tested on the Predictiv platform. Experts within the Behavioural Insights Team advise on what kinds of tests are likely to work. Very often, our partners ask questions that we ourselves have not thought of, which means that we have to design new features for the Predictiv platform. This means that we are constantly innovating and improving the capabilities of the platform, in response to departments’ new interests.

Innovation Reflections

Results, Outcomes & Impacts

Of the roughly 30 trials run through Predictiv so far, the results have shaped government policy. For example, the Government Equality Office and the Department for Work and Pensions are using the results of one Predictiv trial to change the their communications around shared parental leave, and Greater Manchester Combined Authorities are currently rolling out a version of the simplified Privacy Notices trialed through Predictiv across the Working Well back-to-work programme.

Another example is our trials with the Financial Capability Lab, which will be coming to a close towards the end of this year, at which point we look forward to the most successful innovations from the Predictiv trials being taken forward to field trials and subsequent policy innovation. In addition, as we develop the platform to enable greater use of templates and capability to be used in international contexts the functionality of Predictiv will be available to a wider range of policymakers and organisations.

Challenges and Failures

The main challenge has been on the technical development of the online platform, which required us to build an interface between the experiment and the online platforms that we use to test ideas.

Conditions for Success

You need to have the ability to back ideas, to give them space to grow, and not to be afraid to confront challenges that you will (inevitably) meet along the way. You also need to be able to accept that not all new innovations are going to work, which means supporting the energy that goes into their development as much as the results that are delivered from them. There is also some merit in supporting a portfolio of innovations – in acknowledgement of the fact that some will fail.

Replication

We built Predictiv in order to hugely scale our ability to run rigorous evaluations. The whole point of the platform is to enable government departments to run many, many more tests of this kind, without having to have PhDs in experimental economics themselves to do so.