Toolkit Navigator

Measuring impact by design: A guide to methods for impact measurement

The Privy Council Office's Impact and Innovation Unit’s (IIU) developed this guidance for impact measurement, in support of its work under Impact Canada. This resource provides an introduction to the topic, as well as a reference for those involved in the design, delivery, procurement or appraisal of impact measurement strategies in Canada and beyond. It provides descriptions and considerations of different ways of measuring impact versus traditional methods.
It includes the reasoning for measuring impact, threats to validity of impact evaluation, a matrix for evidence, descriptions and guidance on methods, including Randomized Control Trials and quasi-experimental approaches.
The resource is available in English and French.

Basic BASIC:
Allows reuse

About this resource

Features
Country/Territory

Canada

Date Published

2019

License

Copyrighted-All rights reserved

Formats

Web-based resource

PDF publication

See cases from others doing this in government

Go to case studies

Find experts and advisers who can assist me with this

Go to advice

Other toolkits related to Public Policy

2 reviews for "Measuring impact by design: A guide to methods for impact measurement"

  1. Inès Lr says:

    The ‘Measuring Impact by Design’ toolkit consists of a set of guidelines meant to support anyone involved in impact measurement. In particular, it provides a clear differentiation between the notion of ‘impact’ and that of ‘outcome’. The Impact Canada Evidence Matrix offers a rigorous approach to measurement that reveals to be valuable for anyone who is not familiar with impact assessment. At the same time, the toolkit is also useful for those that might be more proficient with impact measurement methods. In that sense, the ‘Measuring Impact by Design’ toolkit can serve as an introductory or complementary source depending on one’s knowledge of measurement methods. The methods detailed in the toolkit are varied (factorial designs, instrumental design, propensity score matching…). This surely constitutes an advantage as it provides a comprehensive overview of measurement design. However, those that are totally unfamiliar with impact measurement might be confused when faced with such an amount of information.

  2. This toolkit intends to provide an introduction to the various existing approaches to “impact measurement”. By offering this guide, “Impact Canada” aims to understand impact measurement more effectively. Judging there is a need for good impact measurement practices, they created this toolkit presenting the different existing approaches, as well as their advantages, disadvantages, and the conditions necessary for their implementation. Intended for novices, this toolkit, by its completeness, is also intended for professionals responsible for supervising the creation and then the implementation of public policies.
    If it is very rich (RCT, experimental, quasi-experimental, and non-experimental methods, etc.), and well illustrated (a graphic design is proposed for each theory), it does not however provide concrete propositions to the user. In fact, the high density of information present in this toolkit present an important risk of losing the novice who, at the end of its reading, will not necessarily know which approach to use and how. Likewise, the experienced user will not come out of this reading with a clear protocol for impact measurement; on the contrary, he will rather come out with a complete panorama of the various existing approaches, which should be explored further according to his needs. In the end, an explanation is missing to move from theory to practice. Also, this toolkit, being a very dense webpage to scroll down, is not very intuitive and navigatable.
    In short, this toolkit is distinguished by its more than complete and rigorous overview of existing approaches in terms of impact measurement, but its biggest strength is also its biggest weakness.

Add your review

Have you used this toolkit? If so, what did you use it for and how did it work? What are its strengths and limitations? Share your experience so others can learn from you.

Your email address will not be published. Required fields are marked *