Impact Canada Challenges have incentivized innovative solutions in the areas of climate action, housing, food, and health. A quasi-experimental impact assessment approach uses tax data (available in most countries) to assess the effectiveness of Challenges by profiling Challenge participants and comparing business performance indicators of Challenge participants vs. non-participants. Requiring only the participant’s business number, this is a no burden, comparative, long-term measurement plan.
Innovation Summary
Innovation Overview
Established in 2017, Impact Canada uses novel public policy methods to address identified gaps in effectively translating policy objectives into meaningful and measurable outcomes for Canadians. It delivers on its mandate through the deployment of two main lines of business: Challenges and applied behavioural science.
Challenges are an open innovation approach that incentivize innovators to tackle problems where solutions are not obvious. Multiple lines of inquiry have been explored by Impact Canada to assess the impact of Challenges on business performance, including surveys, social network analysis and quasi-experimental work using tax data. This work is guided by a central logic model for Challenges, which moves from process outcomes to sustainable business models to greater public value. The work described in this case study primarily corresponds to the sustainable business model outcomes of increased investment, enhanced skill and capacity and innovative products and services.
In 2019, Impact Canada partnered with Statistics Canada’s Centre for Special Business Projects to explore the possibility of using tax data to respond to questions about the performance of Challenge participants and evaluate the economic impact of Challenges. The approach employed provides information on what types of organizations participate in Challenges by profiling businesses by business characteristics and financial information. This approach also provides insights into whether Challenges help to stimulate and grow innovative market entrants (e.g., new firm-types, new products) compared to if Challenges were not used as a funding instrument. A comparison group of similar non-participant organizations were created using nearest-neighbour-propensity score matching technique to compare the business performance growth rates between the two groups on the following indicators: revenue, R&D expense, salaries and wages, employment, and labour productivity.
A registered business number (BN) is required to track participants using tax data. This is obtained when a participant applies for a Challenge. Where a BN is not available, applicants are matched, using available information (name and address of business) and various record-linkage methods, to a Business Register to obtain the correct BN. The BN can be linked to various data sources, including tax data, which are updated annually. This impact assessment approach requires no further reporting from applicants and allows key business indicators to be monitored both before and after the Challenge is launched.
Governments, around the world, know that collecting data to help understand the impacts of funds being dispersed is incredibly important; and yet, attention must be paid to the reporting burden placed on fund recipients. Challenges are meant to attract new players, focus on outcomes, and ensure low barriers to entry. Innovators are asked to focus their data and evidence efforts toward demonstrating the viability of their solution. This impact assessment approach allows for a high rigour but low touch method to understanding the economic impacts of our challenges.
There are multiple benefits to this innovative approach of measuring impact. At a macro level, the relevancy of using tax data to understand the impacts public facing investments programs is examined. Typically, these types of funding programs require quarterly or annual reporting from recipients, who may conduct their own evaluation or work with third-party evaluators. Evaluations may be more focused on process and outputs than outcomes and impacts, and generally do not track participants over time. The Government of Canada spends billions of dollars each year on public facing programs. This presents a significant opportunity to use real-world data to estimate economic impacts of program funding, which ensures low response burden.
The seven-year partnership between Impact Canada and Statistics Canada allows multiple rounds of analysis to be undertaken. A second round of analysis is underway which takes a sectoral focus, based on Impact Canada’s portfolio of work, and will generate valuable information on performance of Challenge innovators in the key sectors of agriculture, housing, climate, and public health for the Government. This data allows for a better case for scaling the use of Challenges in more traditional funding programs to be considered. As more tax data becomes available, richer methodological approaches will be explored including dynamic difference-in-differences and other causal inference methods. Given that Impact Canada Challenges are scheduled to run through 2028, a significant opportunity exists to create a rich evidence-base not only on Challenge participants but extending the use of administrative data to assess the impact of what was traditionally thought of as “community-based programming”.
Innovation Description
What Makes Your Project Innovative?
Unlike surveys and other methods of gathering data, using administrative data, such as tax data, for purposes other than it’s intended use is a resourceful and inventive approach for conducting impact analysis.
Tax data is comprehensive, and available for multiple time periods, allowing for longitudinal analysis. In Canada, with appropriate permissions, administrative datasets are able to be linked to other datasets for research purposes, which increases the scope and capabilities of possible research to be conducted. For this project, Challenge applicant information collected by Impact Canada was linked with business and tax data that is accessible and maintained at Statistics Canada.
Using tax data, in the context of Challenges, is innovative as it opens up more possibilities for proxy variables to assess the effects of innovation, and also reduces response burden of participants involved in Challenges from having to answer questionnaires.
What is the current status of your innovation?
Impact Canada began exploring quasi-experimental lines of inquiry, using tax data for impact assessment of Challenges, with Statistics Canada in 2019. The first round of analysis assessed the feasibility of evaluating the success of 7 Challenges in achieving their goals. This was done by profiling participants by business characteristics and financial information, and comparing growth rates of business performance indicators of participants to a control group.
A second round of analysis is underway for 14 Challenges, which expands on the previous work by profiling applicants against semi-finalists and finalists to assess the impact of the financial and non-financial support provided to applicants along the Challenges process who progress at each stage. The analysis will include additional years of data, a sectoral analysis of priority portfolios, opportunities to explore new proxy measures for innovation and the potential to assess dynamic treatment effects of Challenges over time.
Innovation Development
Collaborations & Partnerships
An integral part of this program is the ongoing collaboration between Impact Canada and Statistics Canada, Canada’s national statistical agency. With Impact Canada’s innovation program advice and experience, Statistics Canada’s Centre for Special Business Projects provided access to data, their statistical infrastructure, statistical expertise, and analytical capabilities to conduct the analysis.
Users, Stakeholders & Beneficiaries
Beneficiaries include stakeholders, governments and organizations interested in measuring the impact of innovation tools, such as Challenges. For Impact Canada, this work facilitated a method of evaluating the impact of Challenges as a public sector instrument, in order to deliver evidence-based products and services to clients and support outcome-based funding approaches. This project provided Statistics Canada learning opportunities to test new methods of analysis on Impact Canada programs.
Innovation Reflections
Results, Outcomes & Impacts
The results from the first round of quasi-experimental analysis using tax data indicated that 50% of Challenge applicants were businesses and they were diverse varying in sex and age of the primary decision-maker, size of the business, industry, and the number of years in operation. The growth rate of revenue, R&D expense, salaries and wages, employment, and labour productivity also varied for participants compared to non-participants. Methodological enhancements were also a result of this innovative approach and helps to inform on which experimental approaches could be employed in the second round of analysis. For example, in the first round of analysis 72% of business Challenge participants were matched to a business number and tax data and a comparative analysis was conducted. In the second round, additional Challenges will be added, and methods will be explored to increase the match rate and assess the impact of the Challenges using analytical methods like difference-in-difference.
Challenges and Failures
- Firstly, using corporate tax data poses a challenge as only business participants whose information is able to be linked to tax data is able to be assessed. This approach omits academics, individuals, and non-business participants, limiting innovation to only a business perspective. This decreases the sample size and has implications on what data can be disclosed given data confidentiality requirements. Furthermore, some tax information is not provided by all businesses, such as R&D expenditures, which limits metrics available to measure innovation.
- Secondly, Impact Canada’s Challenges follow different timelines, as some are completed, while others are in progress at different stages. This coupled with varying available data prohibits the ability to compare the impact of a Challenges against another in a given year. A point-in-time measure is, however, can be obtained for Challenges both completed and in-progress, to get a sense of the impacts of Challenges.
Conditions for Success
- The Impact Canada Challenge logic model is foundational in highlighting measurable outcomes that Challenges are designed to achieve and inform on business performance metrics to include in the analysis.
- This quasi-experimental approach was able to be executed due to a 7-year commitment between Impact Canada and Statistics Canada, along with dedicated resources to conduct the analysis and provide advice.
- Having an iterative co-design process between both organizations contributed to the success of the project.
- While the use of administrative data for research purposes is supported in many countries, it is under-utilized due to government and technical barriers. Using tax data to assess innovation, requires the appropriate access to data, and necessary infrastructure to link and manipulate data.
- A strong collaborative partnership between an organization with access to data, and statistical and analytical expertise coupled with innovation leads is needed to conduct a similar approach.
Replication
This method is already being replicated in the second round of analysis, which is underway in Canada, that builds on the first round by including more Challenges and digs deeper into comparing Challenge participants against non-participants, semi-finalists, and finalists. The second round will also explore new methods of measuring impact dynamically.
There is potential for this method to be replicated by other countries where tax data is accessible and there is an ability to link administrative datasets, in order to assess the impact of innovation on business performance if tax data or other administrative data sources contain proxy variables to measure the intended outcomes.
Similar work has already been conducted at Statistics Canada in terms of program impact assessment using tax data linked to the Business Innovation and Growth Support (BIGS) database, which was developed specifically for impact evaluation of government program streams (Demers, 2021).
Lessons Learned
In the context of Challenges, economic improvements are longer-term outcomes that require a number of Challenges to be completed and a sufficient amount of time and data pre and post Challenge to evaluate the impact of a given Challenge. As a result, this provides many opportunities to explore other quasi-experimental approaches, as well as increase the robustness of analyses conducted in the upcoming years, as more data becomes available on participants, and as more research questions are explored to inform on outcome-based funding.
This project provided an opportunity for Impact Canada to explore quasi-experimental analyses for the first time to assess how Challenges affects business performance. This would not have been possible without a strong and collaborative partnership with Statistics Canada. Working with a statistical agency with data analysis expertise has helped Impact Canada to test and refine the subsequent quasi-experimental approaches to be conducted.
Status:
- Evaluation - understanding whether the innovative initiative has delivered what was needed
- Diffusing Lessons - using what was learnt to inform other projects and understanding how the innovation can be applied in other ways
Innovation provided by:
Date Published:
18 January 2023