Steps towards a New Data Deal

Steps towards a New Data Deal: Prototyping an evidence-based innovation panel for the Portuguese Public Administration

A rigorous, realistic, and actionable knowledge of current innovation practices is indispensable in order to meet the present challenges and proactively prepare the future of Public Administration. From the outset, innovation practices in Public Administration must understand the specificities of public service guaranteed by the State. For the Experimentation Lab for Public Administration (LabX), a team from the Portuguese Administrative Modernization Agency, the experimental project entitled ‘Innovation Panel for the Public Sector’ was aimed at developing a diagnosis and a prototype of an instrument for monitoring and adapting innovation strategies.

The goal was to apply such instrument to the context of the Portuguese public sector as an evidence-based decision-making navigation panel. There are other ongoing initiatives to develop valuable methodologies for public sector innovation measurements, such as the co-creative process promoted around the Nordic Innovation Barometer. For our experimental project, we were engaged in the re-contextualization of innovation measurement in the Portuguese Public Administration. We wished to prevent both the extrapolation of private sector innovation rationales, and to avoid mimicking innovation measurements from other varieties of nation-states, which are designed for their own national history and contexts. For all those reasons, we are here dealing with the idea of a New Data Deal: an experimental approach that can test before implement a solution to endow public organizations with usable, responsible and relevant evidence to inform their innovation strategies.

Networking: a collaborative approach to design

The development of an experimental project of this nature benefits from an active interconnection with partners in the innovation ecosystem. On the one hand, it makes it possible to recover and integrate the experiences and knowledge accumulated by them in similar exercises; this means we did not need to start from nothing and did not waste the wealth of their knowledge reserves. On the other hand, it is crucial to have diverse views on the public sector, avoiding consensus around the self-image that the Public Administration might create of itself when it develops instruments to measure its initiatives. For this mode of collaborative work, we have relied on the liaison with experts from a research centre with comprehensive evidence in this field (CIPES) and with the Observatory of Public Sector Innovation (OPSI). In its final stage, this experimental project also counted on the contributions of the team formed for this project within the Collaborative Work Plan, developed in the Portuguese Public Administration, to validate its outputs and improve its design and features.

Four stages of experimentation

The development process of this experimental project took place over four major iteration cycles, each of which included its own mechanisms of methodological self-control. This iterative method guaranteed that planned objectives critical to continuation to the next stage were fulfilled. At the same time, regular stress tests identified new opportunities or risks, and the introduction of incremental improvements brought us closer to a better solution to the ultimate objectives of this experimental project. Starting in September last year, the plan was to attain the final results before the end of June, as it happens to occur. We don’t want to unnecessarily extend the discussions or get stuck in dilemmas, but grab the opportunity to obtain a demonstrative example and an accelerated accumulation of ‘errors’, i.e. gain useful lessons through a controlled experiment.

Stage 1: Design the analysis scheme and research instruments

The design of the heuristic model and construction of the research instruments were the objectives for the first stage. The structuring of the research protocols and the construction of the research instruments was carried out in a collaborative manner between the tripartite team. The entire action plan could be validated in good time by a consequence scanning exercise that brought greater clarity, sought to anticipate setbacks and defined clear priorities for all members of the project team.

The vision presented by the Observatory of Public Sector Innovation (OPSI, 2016) about the ‘innovation lifecycle’ was retained as an analytical scheme that helps to overcome the usual obstacles in the treatment of public sector innovation. Such vision prevents dilettantish discussions about the starting-point of innovation, seeing it instead as a cumulative process open to disruptions; it also prevents the selective and partial perception of innovation stages, isolating them and reducing the understanding of its holistic and multidimensional nature; it highlights the rough and tangible side of innovation through its implementation, ahead of the inspirational discourses and the ideological consensus. The heuristic model that was developed (see Figure 2) assumes a multidimensional, dynamic and holistic nature of public innovation.

• Multidimensional: this heuristic model accepts the multidimensionality of public innovation, integrating indicators from four areas associated with the innovation cycle: dissemination and ecosystem (environmental context); incentive management (organizational context); innovation processes (application and implementation of innovation); and results management (innovation impacts);
• Dynamic: the existence of an innovation cycle makes it possible to accept the transitions, conversions and transmissions between the moments of the innovation production processes, assuming that the resources available for innovation are dependent on the ecosystem and that, in turn, the incentives and processes are dependent on the resources available. On the other hand, the results of innovation are dependent on the incentives and processes of innovation and the ecosystem is dependent on the results of innovation obtained, at which point the cycle of innovation implicit in this proposal converges. In this way, we clarify problematic and unprofitable discussions about precedence and move away from unilinear and teleological processes.
• Holistic: the existence of the horizontal and vertical axes, which structure the four moments of the innovation cycle, makes it possible to highlight the mutual belonging between these dimensions of innovation and the ‘lines of force’ that pull by their procedural dynamics, exposing their movements (for example, between the ‘interior’ and the ‘exterior’ of Public Administration). The horizontal axis of the model defines the background and the consequences of the innovation process. Two underlying characteristics are presented by its vertical axis: on the one hand, indicators associated to the context (external to the institution) and to public sector entities (surrounding society and innovation ecosystem); on the other hand, indicators associated to the monitoring and dissemination and the search for novelty and development.

Stage 2: Pre-testing & Testing

According to the experimental nature of this project, the research instruments were subjected to a series of tests with the purpose of understanding and improving the experience of interaction with potential respondents. For the survey, cognitive interviews were made with potential respondents; afterwards, pre-tests conducted on the online survey platform also indicated barriers and indicated potential improvements. A robust sample selection procedure was performed to include participants from all levels of administration and all areas of government.

An electronic platform was used to send the surveys to the participating public entities and to manage the application of the instrument. Throughout the period in which the survey was available, the participants had access to support through e-mail or telephone for any questions or doubts. The management of the relationship with the participants was very time-intensive: although it is possible to automate the sending of reminders to those slow to respond or messages of thanks to those who fill out the questionnaire, there were direct questions sent by public entities that needed personalised answers. At the end of the period of application of the survey by questionnaire, a total of 92 replies were collected.

Stage 3: Analysing and evaluating data

For the third cycle of iteration, the data collected in the responses of the participating public entities were processed and consolidated, being afterwards analysed in depth. The mass of data gathered allowed a wide range of more specific analyses to be conducted. After the collection of replies, the consolidation of the database – automatically generated by the survey platform – and the exploitation of the data began. In this way, it was not only possible to implement the research protocol that was initially planned, but also to broaden the range of analytical options once the potential of the data had been verified, and to develop a series of proposals for analysis and interpretation (e.g. aggregated or composite indicators). This first level of data analysis resulted in an up-to-date, robust and integrated knowledge of the areas of innovation in the Portuguese public sector.

The areas of innovation included:

• The consideration of the strategic objectives and the structural features of the organisation;
• The description of the innovation ecosystem and the organizational environment;
• The solutions with higher impact and tangible results;
• The mapping of the networks established.

There was also the opportunity to connect with the evaluation carried out by the public entities regarding the research instrument, aided by the feedback section included in the survey. A preliminary presentation format benefited from the comments of a group of colleagues created under the aegis of the above mentioned Collaborative Work Plan.

Stage 4: Disseminating and activating results

From the outset, it had been assumed that the objective of the project went beyond the compilation of statistical information or obtaining abstract measures on innovation in the Public Administration. The intention was to make data accessible and usable for the real priorities, needs and expectations of its users, namely: public leaders or managers, as well as specialists or policy makers. In this sense, this fourth cycle served to build data communication channels, structuring them in an accessible and actionable way for their main stakeholders.

We sent out individual templates capturing the innovation practices for each of the public entities participating in this experimental project – and we organized a collective webinar for them to give us back their opinion and share their questions with us. The production of those individual templates gives the participating public entities a characterization report with a consolidated presentation of data. We manage to ensure that it was issued automatically from the data processing platform used to manage the date collected. Through those templates, participating public entities have access, at the end of the analysis process, to a concise, self-explanatory, and informative report that not only provided up-to-date knowledge on their innovation practices, but also placed them in relative terms within the body of Public Administration institutions as a whole.

Furthermore, an innovation dashboard prototype was developed. It adopted an evidence-based decision making approach to support public entities in defining their best strategic options through the presentation of ‘usable data’ (useful, impactful, insightful, intuitive) rather than massive or intricate data. This dashboard presents structural properties in response to needs or challenges detected among its potential users:

• Visual: the presentation format communicates complex information in a structured, synthetic and visually organised way in order to eliminate or at least lower data literacy barriers and present only the most relevant content to the recipients;
• Interactive: The use of an automatic data extractor allows the creation of an interactive dashboard, which the user can set or customize (to choose, for example, the reference value for comparing the results of their organisation);
• Actionable: the definition of the components and indicators presented by this dashboard included the priorities for concrete action to be taken by the addressees, i.e. the managers and directors of public entities, which expresses the objective of supporting decision making in empirical data;
• Up-to-date diagnosis: it gives a panoramic picture of the Public Administration and an individualised characterization of each public entity in this universe, which allows to add to the potential users of this instrument all the public decision makers with the intention of a vision that is, at the same time, sensitive to different contexts and to the coordination of the whole.

Lessons learned

We have learnt a vast amount, doing a sort of ‘primitive accumulation’ of knowledge for the implementation of future initiatives. Besides simple observations (such as the need to replace the vernacular language used when discussing innovation, as it discourages ‘lay’ persons from engaging and taking part in those debates, or establishing benchmarks to enable measurements over time), we choose to highlight:

• Relevance of user-centricity: rather than feeding a ‘pure’ research or making the diagnosis an end in itself (by obtaining an index, for example), our concern was to focus this measurement and monitoring tool on the needs and priorities of users, ensuring that the results obtained were accessible and actionable by public authorities for their daily activity or by supporting them in defining their strategies for innovation.
• The need to combine innovation practices and discourses: it has proved important to weigh the opinions and perceptions expressed by the respondents’ answers about innovation with the objective demonstration of the actual presence of innovative practices and products in their organizations (use of innovative tools, implementation impact, budgeting, etc.). The adoption of innovation as an ideology or as a normative narrative among public entities, needs to be complemented with an attention to its achievements, at least as a procedure for detecting any gap between knowledge and action.
• Making available instruments of action to the public sector: from the outset, the purpose of this experimental project was to make an instrument of action available to Portuguese public entities for the definition of their strategies, rather than carrying out a ranking or suggesting a sophisticated interpretation of data. This instrument had to be oriented to respond to the needs, doubts and priorities of public entities in this field; to ensure that it was intelligible and accessible by its users, both in terms of data literacy and data delivery support; and to be able to leverage decisions from robust empirical data.
• Advantages of the experimentation process: more than just betting on the accumulation of theoretical knowledge, this project bet on the importance of taking an experimental approach: with a preliminary exploration, iteration mechanisms and a controlled environment to mitigate potential risks and allow the assimilation of learning from ‘mistakes’. This experimental project gives some direction for future projects and enables continuous learning. Although on a controlled scale, the experimental nature of this project meant that calibrated responses to the context were developed to control and circumvent the difficulties encountered in this first attempt and a more realistic and rigorous management of the efforts required for subsequent applications of this innovation tool was fine-tuned.
• Promoting emulation: rather than comparison with the private sector, stimulating blind competition among individual organisations, or levelling innovation in a one-dimensional hierarchy (for example, as a ranking), the formats adopted for data presentation aim to promote emulation. They place the public entity in question among the whole set of Public Administration organisations, allowing readings in multiple dimensions of innovation.
• Enabling the contextualisation of innovation: public entities can identify strengths and weaknesses in their organisation, detect gaps and deviations from their declared objectives, or map opportunities hitherto latent in their innovation activities. In this way, the presentation of a first diagnostic report can be continued by an interactive use of a data platform, inviting closer inspection of the support mechanisms, development processes, and assessed impacts of implemented innovation strategies and, on the other side, highlighting actual practices of innovations through the effectively applied innovation tools, practices and products.

Coda: try again, try better

The results of the experimental project allow us to envisage the development of future initiatives for the implementation of the project, in particular to overcome the obstacles identified or to take advantage of latent opportunities. Looking at the short-term feasibility, we can immediately think about exploring applications of the knowledge and instruments created in the meantime. For instance, changing the scale of the analysis, capturing variations among departments of the same public organization. For the future, there is the opportunity to engage in an in-depth approach to the monitoring of innovation in the Portuguese public sector innovation through the establishment of a sustainable initiative for applying iteratively improved interventions.

There are latent possibilities open to be equated and proved through experiment:

• The need to stabilise the measurement process, making it an integral part of normality. More important than episodic and insulated interventions, it is essential to promote a culture of measurement and evaluation in a sustained manner by the Public Administration.
• Advantage of using existing administrative data, wavering unnecessary effort and boosting data cross-checking. The ideal would be to be able to use existing administrative data, which would reduce the effort required from public entities in providing answers and, at the same time, would enhance the cross-referencing of data using reliable sources.
• Providing an inducer of change by itself: the results presented by this experimental project may be a nudge, as it is not anodyne in its consequences. As soon as certain data are requested or certain variables of innovation are highlighted in the eyes of public entities, this instrument has the capacity to act as an instigator of change. In fact, this model of innovation is not intended to be a neutral version of public innovation; on the contrary, it seeks to promote a notion of innovation that is focused on the needs of citizens, open, collaborative and with an experimental and iterative nature.

If you got interested on knowing more about this experimental project, you can take a look on its executive report. We welcome any comments and queries you may have. Please, reach us using our address (geral@labx.gov.pt).