Measuring Public Sector Innovation: Why, when, how, for whom and where to? (Innovation Lifecycle 5b/6)

Overview

The current discussion paper accompanies the fifth public sector innovation lifecycle study on innovation evaluation funded by the European Commission under the Horizon 2020 grant number 671526. While measurement is often mixed up with evaluation, the two have distinct goals. While one tries to assign numerical measures to inputs, processes, outputs and impacts, the other takes the form of a systemic assessment of causal relationships between intervention and change. Thus, measurement and evaluation should be mutually reinforcing, but they should not be seen as interchangeable entities.

Measurement systems, indicators and metrics in the public sector are becoming increasingly complex, yet, they do not a priori say that public sector’s activities lead to change; they serve as descriptive tools, feedback measures and alarms for the need of further control and assessment. Yet, in many cases, measurement becomes the proxy of evaluation. This in context of innovation (and probably also elsewhere) should be avoided or at least critically examined – quantitative measures by design simplify and thus, they serve some goals better than others. Generalisability, comparability and uniformity seem to rule over usefulness of measurement towards different types of innovation and user needs. Hence, the paper argues that there is a need to go back to basics and start to link public sector innovation measurement better with its different purposes and user needs; so, that ‘why’ measure public sector innovation does not get lost in ‘what’ can be measured.

In developing the argument, the study reviews the experiences of previous public sector innovation measurement efforts and the influence of economics, management theories and private sector innovation on this work. Here the Oslo Manual has had a widespread influence, both positive and negative. The lack of a market or a ‘price measure’ of public sector innovation has meant that effectiveness, efficiency and impacts of innovations in the public sector are rarely if ever studied and mostly reliant on self-reported data and individuals’ perceptions. Most measurement efforts have relied on survey methodologies and been good at testing and describing a variety of factors connected to the ‘success of innovations’. Yet, how these factors are linked to each other, how they change over time and the size of their influence is not examined. Nevertheless, public sector innovation measurement is far from a blank slate – there are plenty of learnings to be gleaned in prior efforts. Yet, the fundamental contradictions of technology, change and productivity dynamics of the sector will not make any measurement effort easy. Far from it.

What is, however, needed now for a qualitative jump in public sector innovation measurement, is a public sector specific conceptual framework that differentiates between different types of public sector innovation based on its directionality and uncertainty. Different types of innovations or innovation facets – adaption, missions, enhancement, and anticipation – need varying support and conditions and thus, uniformity of measurement systems may be actually hurting specific innovation efforts.

In addition, the ‘positive’ prism of innovation is still prevalent in the public sector – innovation is seen as something that is always beneficial. This assumption should be refuted, because it stands in the way of meaningful measurement systems that act in a sentinel manner. Innovation does not always make things better, it can make things worse for different people and introduce considerable value shifts and public sector measurement systems should capture the former.

Last, but not least, digitalisation is introducing fundamentally new ways of measuring different phenomenon in the public sector. More experimentation with big data, machine learning, social media scraping and crowdsourcing/ coproducing information should become the norm in public sector innovation measurement. One cannot study innovation, but negating change and new ways of learning. What is certain is that public sector innovation measurement should always be in flux – simply put, the past is not the predictor of present of the future, especially when it comes to innovation.