This project demonstrates the successful use of mixed-methods that combined Design and BI, a first in the Government of Canada. Specifically, we gained a deeper understanding of the needs and experiences of Canadians living in low income using ethnography, and then integrated those insights into behaviourally informed communications. These were then tested using rigorous experimental methods (Randomized Control Trial). Given the success of the trial in that the letters based the insights derived from the ethnography outperformed the standard letter used in regular communications with eligible families, this project shows the complementarity of both methods and benefits of being able to test lessons learned from the design project (qualitative) at a large scale (quantitative) in a BI trial. The findings from this trial have been adopted by the CLB program for broader use.

With the successful completion of the first BI trial directly informed by a design project, the Government of Canada is now implementing principles from the highest performing letter into regular communications of the CLB program. Provincial and municipal governments, and community organizations are also looking to integrate these findings into their communications. We are now involved in a number of new trials to test and validate other insights generated from the design project, thus building knowledge rigorously and incrementally. Other insights generated by the design project remain to be integrated to BI interventions beyond letters and we plan to test and validate them in new trials. Additionally, we seek to extend the use of mixed-methods to other sticky problems faced by the department, given the success of this innovation. We are also disseminating these findings across the Government of Canada and hopefully globally in the near future.

The initial BI project involved academics contributing expertise in the field of behavioural science. The design project collaborated with provincial and municipal governments to better understand the inner workings and challenges of the program. We interviewed 126 Canadian citizens to gather their stories, concerns and aspirations regarding post-secondary education. Community organizations and teachers informed us about the best practices regarding working with Canadians living in low income.

The main beneficiary of the innovation is Canadian families living in low income. Because of this innovation, more of them are now benefiting from the CLB. Community organizations as well as other levels of government are also benefiting from our lessons learned as we continue to share our findings so that they may better promote the CLB to eligible families. Having such a project in government has also stirred interest in using mixed methods across other areas.

Outcomes included improved program take-up (i.e., increased education savings for Canadians living in low income), a greater understanding of the users (e.g., their perceptions of education savings, post-secondary education), and improved ability of the department to serve this population. A randomized controlled trial was used to conduct mailing trials comparing the effectiveness of different letters. Receiving the standard (no nudge, original) letter increased CLB take-up (5.7%) compared to a group that did not receive a letter (2.2%). Based on design project insights, presenting the exact amount of money they would receive when opening up an account increased take-up over and above the standard letter (8.0% take-up), and reduced emphasis on “saving” (living with low income, many participants felt this term did not resonate with them) further improved take-up (8.8%). Combining BI with design insights resulted in a 55% increase in take-up above sending the standard letter.

This project may not have taken place were it not for the failure of the first BI mailing trial conducted in 2016. This trial showed that the best nudges developed performed only as well as the program’s standard letter, and the other nudges significantly reduced CLB take-up. These failures revealed our lack of understanding of our clients and spurred the development of the design project. Getting the design project up and running required substantial investments in time, including the lengthy process of obtaining clearance from the privacy division to be able to speak with Canadians. Though we have had successes integrating BI into the letters and increasing take-up, it is clear that letter communications will not fulfill the program’s enrolment goals. Other lessons learned from the design project that are more transformational in nature for the program have been suggested. However, these larger scale recommendations have been met with a degree of resistance by the program.

The Innovation Lab brought together design and BI, two complementary approaches, which was critical to the project’s realization. Another crucial component to this projects success is our access to eligibility and enrolment data for the CLB. This allowed us to conduct a randomized controlled trial and analyze its results all in house. The Innovation Lab has a high level of engagement and support from senior management. Without their support of experimentation and innovation in government, this project would not have happened. We were able to hire appropriate skills sets, including design, ethnography, behavioural insights, and data analytics. Funding covered travel costs to go meet Canadians in their homes and community organizations. Regarding values, this project could not have happened without the openness of the CLB program owners to be sometimes uncomfortable with innovation, and without the Canadians’ interviewed trust in government and belief in the value of education.

A project of this scale is a first in the government of Canada. Given its success, here at ESDC we are currently considering using this mixed-method approach to other program areas, such as employment insurance and student loans. There are also other similar efforts, though smaller in scale and complexity to conduct mixed-method projects in other federal departments. This approach does have the potential to be replicable in government and other types of organizations (e.g., private and non-profit). However, the Innovation Lab provides a unique environment conducive to innovation and experimentation and skills set that are unfortunately uncommon across other government organizations. Nonetheless, it does emit a strong signal of the value in building internal capacity for innovation and experimentation in government.

Much was gained by combining Design and BI to better understand Canadians living in low income. We achieved a better understanding of how living in low income interplays with programs that incent savings. For example, framing such programs as benefits instead of savings incentives, and removing uncertainties such as “could benefit up to”, can have positive impacts on program uptake. Finally, Canadians have expectations of government letter communications. We learned that “Bland is the Brand” (flashy, modern letters with images sometimes reduced take-up) and we should respect this for our communications to appear legitimate (not mistaken for spam).

Experimental methods can shed light onto the effectiveness of our programs and services. When management realizes the effectiveness of experimentation they are willing to act to learn more and innovate.

The success of this project depended on skills atypical and rare to government including ethnography, design, and behavioral science. From a broader perspective, the successes and failures experienced in our trials revealed the limits of using a one-size-fits-all approach with BI. Although BI principles are supported by decades of research, the reality is that interventions informed BI principles do not readily generalize across contexts, behaviours, and populations. Our research suggests that this seems to be especially true with vulnerable populations (low income families) who face unique challenges and barriers. As these populations have seldom been represented in the research supporting BI interventions, there is a need to adopt a tailored approach to their specific needs and experiences. The mixed-method approach proved to be a powerful tool to help us craft more tailored messages. Specifically, the qualitative research helped us better understand the program’s users, and in turn generated insights instrumental for designing more effective messages.

Here are a few quotes from citizens living in low income illustrating their complex situations which inspired some of our BI interventions.

“It’s hard to even picture something in the future now, because it’s hard to know what it will bring.”
“It seems really big – it’s adult – to talk about Registered Education Savings Programs (RESP) – It’s such an adult word. It scares me a little bit and it’s a bit intimidating.”
“I have heard about RESPs. The issue with that, it’s hard enough to save for the future for retirement – I don’t even have any retirement savings, let alone savings for education."
“It’s difficult as a low income person to take advantage of the programs that they do have.”
“Our kids, they need to have education, the same as the other kids. It doesn’t matter if you have low income, or high class.”
“Education is important for me for my kids, because I don’t want them to have to walk in my shoes.”
“I can see why people wouldn’t do it, because it is a process.”