Decreasing corruption in the Colombian School Meals Programme PAE - A bottom-up approach

The Colombian Public Innovation Team ran a trial aimed to improve the quality and quantity of food served in the Colombian government’s school meals programme (PAE). It combined SMS messages to encourage parental engagement in the programme and lighter audits by a third party (local university students). Behavioural insights were applied to inform message design. The project included a learning phase to test and adapt elements of the intervention and an experimental evaluation (RCT) in the territories of Nariño and Cesar.

Innovation Summary

Innovation Overview

Tackling the corruption that affects the provision of school meals is a priority for the Colombian government. Since a video in 2016 revealed the extent of fraud and deception, the issue has become a national scandal. The video showed a provider taking photos of children with one same plate full of food, as supposed evidence that the food was being provided, but also the reality that , immediately after the photos were taken, children were actually receiving much smaller portions of food, directly into their hands.

The project involved a collaboration between several units in the Colombian government (National Department of Planning, Secretariat for Transparency, Ministry of Education, and Ministry for ICTs), the UK's Behavioural Insights Team and the Inter-American Development Bank, with the support of local government entities and universities. We followed the TEST (target, explore, solution, trial) methodology proposed by the Behavioural Insights Team (BIT). In the target and explore phases, we conducted field work (direct observations, focus groups, semi-structured interviews) with members of local school communities in Colombia (government officials, principals, teachers, students, parents, food providers, cooks) and found that (a) parents usually had little knowledge about the food programme and the spaces availaible for their participation, and that (b) the vast majority of parents, even in remote rural areas, had access to cell-phones with SMS capabilities.

Text messages have proved to be an effective way of informing and engaging individuals in a variety of contexts, including parents of school children. This inspired us to test an SMS component in the intervention.

The implementation of the intervention was divided into two phases. In the first, or "learning" phase, we focused on understanding how to communicate with parents to increase their engagement with PAE. The second phtase tested the approach at scale with an RCT. The intervention combined SMS messages, based on results from the first stage, with spot-checks conducted at the schools. We tested whether the intervention improved the quantity and the quality of the food.

In the learning phase, we gained knowledge about logistical needs and tested messages sent via a two-way SMS platform, i.e. parents were able to respond to the messages we sent, and receive back an automated response. We used the Ministry of Technology’s SMS platform Urna de Cristal. We texted parents information on the entitlements of their children within PAE e.g. "Hola Manuel, Hoy Daniel debio haber comido pollo, arroz y brocoli. Si comio eso?" For 5 weeks approximately, 3000 parents in 8 schools in Nariño (Southwest Colombia) and Cesar (Northern Colombia) received our messages. Parents were randomly allocated each week to see a variation in the content of the message, or to receive it at a different time of day. This allocation allowed us to test the impact such variations on the likelihood of getting a response from parents.

The messages followed an adaptive or “winner stays on” design. Each week we adapted the messages based on what we found to be the most successful variation the week before. Specifically, we tested the impact of the following on parents’ level of engagement: 1) to what extent personalisation of the message matters; 2) to what extent timing of the message during the day matters; 3) whether response rates are higher if we provide parents with more or less information about what their child should have received; 4) whether parents are more likely to respond to a closed- or open-form question.

After the learning phase, we deployed an RCT with a sample of 208 schools, randomizing at the school level. The intervention showed no effect on the primary outcome measures: quantity and quality of food supplied by providers (as assessed during the data collection audits) and parents’ satisfaction with the food served (data collected through a household survey).

We knew before the launch that, given the experimental design, the minimum detectable effect size was relatively ambitious. Considering the high cost of the intervention, expecting or requiring such an effect size for the intervention judged to be a ‘success’ seemed reasonable. However, the confidence intervals are not very wide, suggesting that the actual impact of the treatment was close to 0. Apart from the intervention genuinely not having an impact on provider behaviour, we think there might be 2 reasons leading to spillover effects and thus a smaller effect size:
-Baseline survey as treatment conducted by the university students in both Treatment and Control schools (providers adjusted their behaviour in response if they interpreted these as a sign of increased oversight).
-Providers adjust behaviour across the board, since they served both Treatment and Control schools, they improved food delivery in both.

Innovation Description

What Makes Your Project Innovative?

We used a behavioural-insights-inspired approach to tackle a complex policy problem going "beyond nudges". We divided the project into 2 phases: The first phase focused on understanding how to communicate with parents to increase their engagement with PAE. We used the Ministry of Communication and Technologies two-way SMS platform "Urna de Cristal." The second phase tested the approach at scale. The intervention combined SMS messages, based on results from the first stage, with spot-checks conducted at the schools.

Using our adaptive approach, we learned a lot about how to engage parents in monitoring the school meals provision. Each week we found a statistically significant difference between the two variations of the content of the messages.

What is the current status of your innovation?

We are currently striving to take this project to a new iteration that builds upon the results and insights gains in the first. We believe that there are good reasons to persuade government officials in Colombia to continue experimenting with approaches to both, increase parental engagement in the PAE as well as a redesigning the audit mechanisms for the program to make them cheaper and smarter. We have also found that there is a lot of learning value involve in the project despite the fact that we did not see significant differences for the primary results variables of the experiment. We are looking forward to transform this learning value into different communication vehicles and formats in order to support policy and decision making for the school study program.

Innovation Development

Collaborations & Partnerships

The Programa de Alimentación Escolar (PAE) is a Ministry of Education policy that aims to encourage school attendance and retention. The trial was designed and implemented in collaboration with the Inter-American Development Bank (IDB) as well as the President’s Office Transparency Secretariat (Secretaría de Transparencia, ST), the National Planning Department (Departamento de Planeación Nacional, DNP) and the Ministry of Education in Colombia.

Users, Stakeholders & Beneficiaries

Children: are the main and only users of the PAE. Understanding their behavior was fundamental for the project because in the field trips the interaction with them allowed us to understand the program failures. Professors: indirect viewers of what happens at school. We realized the need to empower them to inform parents about the PAE. Parents: are the main allies in making social control of the PAE. We understood the importance of being informed about the menus that their children received.

Innovation Reflections

Results, Outcomes & Impacts

This trial aimed to improve the quality and the quantity of the food served under PAE through a combination of encouraging parental engagement in the monitoring of the programme and through highly visible audits.

We find a statistically significant impact on response rates of:

1) asking closed questions instead of open-ended questions - a 163% increase,

2)naming 3 food items their child should have eaten instead of 1 - a 59% increase,

3) sending messages in the morning as opposed to the afternoon– a 45% increase,

4) sending messages about the specific child as opposed to messages talking about thousands of similar children –32% increase.

We did not find any statistically significant impact on the primary outcome measures - quantity and quality of the food served and parents’ satisfaction. We did not find any statistically significant impact on the other secondary outcome measures (parents’ knowledge about PAE and their report).

Challenges and Failures

During fieldwork we observed some of the barriers that might explain the lack of engagement from parents:

1) Lack of knowledge: parents do not know what the delivery mechanisms of PAE are, what their child is entitled to and what their child should receive on a given day. Additionally, parents seem unaware of channels they could use to effectively raise concerns and lack information on the performance of providers.

2) Collective action problem: parents might be scared to take action individually, fearing repercussions by providers or other stakeholders involved in sub-provision of PAE. Some parents perceive PAE as a favour and fear that, if they were to complain, their child would stop receiving food or would be kicked out of the school. Parents may also fear that individual actions and requests will not be listened to because they do not carry sufficient weight to change the operator’s behaviour.

Conditions for Success

There has been a lot of interest in this work from very high political level. First, the former Director of the DNP mentioned it as one of their core innovation projects both at the UN General Assembly and at the World Economic Forum in Davos. Second, the Colombian President – Juan Manuel Santos - also mentioned the project at a presidential broadcast as one of the government’s main innovation policies to fight corruption. On September 19th, President Iván Duque launched the new PAE whose three main areas are: greater financing, coverage and transparency. This restructuring takes up several of the findings found in this project, which indicates that it will continue to be a key input when the new program begins to be implemented.

Replication

There were two main implications from this project. First, we found that we can engage parents in the school food provision by using two-way SMS messages. These trials show that SMS messages can elicit responses from parents and so act as a cost–effective mechanism to collect valuable information from citizens. Second, we found that small variations in the content of text messages significantly change how effectively they engage their recipients.
In that sense, we consider that what we discovered in terms of citizen control is fundamental for any type of public policy. It is necessary to not underestimate the auditor character of a citizen, in this case of a parent, teacher or school rector, throughout the whole procurement process. From the public tender to the food delivery.

Lessons Learned

We confirmed findings from other studies, that suggest that personalisation of messages can be a powerful way of getting individuals to respond to communication addressed to them. Additionally, we showed that parents are more likely to respond if they receive a closed rather than an open-ended question. Parents were more likely to respond when we named three rather than one food item that their child should have received on the day in question. One possible interpretation of this result is that their was more likely to be a discrepancy between the food served and the food on the menu for the former case and that parents are more likely to respond if their child did not receive the food they should have received. There is some (weak) indication that this might be the case: among the parents who provided a ‘correct’ response, 54% in T 1 indicated that their children did not receive the food they should have compared to 46%. We also found that parents were more likely to respond to text messages sent in the morning compared to the afternoon and if asked to help to improve the food for their own child (rather than other children in the same ETC). Furthermore, the trial has provided valuable lessons on how to run a large-scale RCT trial in general and more specifically using Urna de Cristal’s messaging platform.

Anything Else?

During the first phase we decided to randomly allocate a sample of roughly 2000 parents to receive a variation in the content of the messages. We called the model a winner stays on design because each week we would adapt the messages based on what had one the previous week. We tested the impact on response rates from parents and we sent them the information on what their child should have received on a given day. In the second phase we randomly allocated 220 schools to and control. Ideally we would have allocated providers to T & C, because one provider covers several schools but there were only 4 providers in the two states where we were running the trial. One of the main limitations of this study.