Prototyping the Toolkit Navigator

Over the last few months, the OPSI team did a lot of assumption testing, prototyping and usability testing of the “meta-toolkit” concept (which we are now calling Toolkit Navigator). This web resource will offer self-service guidance to public sector staff and policymakers on existing methods and toolkits, as well as related case studies, peer connections, and capacity building resources based on their specific needs and contexts.

As we launch the beta site for the purpose of testing it, we want to know what you think. Take a look at it here and let us know what you like and what is confusing or missing. As it is a proof-of-concept, we are also looking for partners as we work on future phases of the Toolkit Navigator. Reach out if you want to know more.

Our journey until now

In my previous blogs, I suggested that we are beyond peak toolkit (here) and explored the classification of toolkits—the difference between a manual, guidance, a playbook, etc. (here). Toolkits and published content change relatively slowly, so classifying and critisicing them were the easy parts. The most difficult part of this project has been finding the right level of detail to characterise the contexts in which toolkits are used. In this blog, I am sharing that journey with you.

From our initial exploratory research in 2016-2017, we learned a few things about the role of innovation toolkits in the daily lives of public sector staff:

Using these findings as design criteria, we developed and tested initial navigation concepts with a broad range of users earlier this year. This included the OPSI National Contact Points, who are our delegation of national government innovation leaders.

We also tested the concepts with a select group of United States mayors from the US Conference of Mayors Annual Meeting, and with participants of a hands-on workshop at the European Commission’s Future in the Making conference. In addition, anyone venturing into my orbit during the last few months became inadvertent and mostly willing research subjects and usability testers.

We received 349 pieces of feedback that are shaping everything from the terminology we use to the product development roadmap. For instance, at a granular level, we learned that many people really dislike the word “unpack” to describe the exploration of a problem. At a product level, we heard that we need to invest more in our capacity for ongoing content curation lest our resource become a toolkit graveyard.

Transferability, not replicability

While classifying objects such as toolkits is achievable, classifying all the ways they are used in context is impossible. We faced this critical challenge in building a single resource for a wide range of users. This challenge is especially true for innovation toolkits since even established methods will be novel when used in a different situation.

We are cataloguing and classifying a growing compendium of toolkits based on the things our users said help them navigate to helpful toolkits (and if we missed a good toolkit, please add it here). We noticed that many toolkits had a discipline or practice “baked in” (i.e. public policy, service design, open government, etc.) so we associated each toolkit accordingly. But classifying a toolkit’s utility is very different than classifying situations that people actually face when innovating in government. When making sense of the contexts in which toolkits are applied, we need enough detail and granularity to allow methodology transfer without being overly prescriptive.

This transferability challenge lead me to look into pattern languages, pioneered by Christopher Alexander in the field of architecture in the 1970s but now used to design collective action by social innovation practitioners. Many of the toolkits are patterns that facilitate a particular methodology. For instance, the Business Model Canvas, one of the toolkits in our curation, is a pattern for developing offerings around value propositions. So, what is a useful pattern for selecting the right methodology and toolkits?

Typologies and patterns may provide us with just enough structure to allow for transfer across contexts. For instance, with regard to the “baked in” discipline/practice, a toolkit developed by service design practitioners in an international development organisation is likely more relevant to peers than a toolkit developed by managers of a software development startup company. However, the more complex work of diagnosing the universe of situations, problem types, individual motivations, group/team identities, organisational capacity, and other factors is still a human-based process; for instance, we cannot translate all of the tacit knowledge and experience of a team like OPSI into a self-service algorithm-driven wizard—at least not yet.

As we have played with different ways of sorting, and classifying different aspects of the many toolkits in the compendium, we used a cloud-based relational database to help us experiment with different ways of classifying the content. Once we had something we liked, the prototyping team even figured out a way to import the relational data to our CMS.

We are focusing on patterns of tasks or “jobs to be done” by public sector innovators and will suggest a selection of toolkits to meet those needs. This selection needs to be less overwhelming than the Google search results for “innovation toolkit” but not presume that there is only one toolkit soulmate per seeker—or even whether a toolkit is what is needed.

Strategy design, service architecture and enterprise design offer some interesting human-centred models and task patterns that connect interactions, operations, governance and policy at an organisational level. These practices are still emerging and expertise is relatively rare and expensive; and, needless to say, few toolkits exist. Nevertheless, dozens of advisors have donated their time and expertise and continue to shape our thinking about linking context to methods and tools.

Transferability means flexible information architecture

In developing our web resource, we faced a classic dilemma in information management—whether to use ontologies (absolute descriptions and a strict hierarchy) or attributes (relativist links and tags).

Clay Shirky wrote a seminal article about information classification in the digital age and gave an interesting example about attempting to classify the world through information hierarchies: the Dewey Decimal System, a widespread content hierarchy used in libraries. He gives the following example and asks, “what is problematic about this hierarchy”?

There is an obvious worldview bias built in, of course. This privileges the world view of the information architect and does not necessarily represent users’ various realities. For a constantly evolving context of public sector innovation, we think a looser hierarchy will provide more flexibility.

But, we faced another challenge in taking this approach inside of an international policy organisation, one which routinely acts as a standard setter for language. Our OECD collaborators each have official policy recommendation language, such as here and here, which they are obligated to use when communicating to governments. For instance, while OECD recommendation language uses the term “stakeholder engagement” a civil servant might search for the term “citizen participation” and both may be referring to the same exact practice. If we build the Toolkit Navigator to filter toolkits based on specific terminology, its relevance will be limited to those who know the lingo, which excludes a wide range of civil servants who do not start their search using the “correct” term.

While designing the Toolkit Navigator, we are walking the line between structure and flexibility since we still need to maintain the OECD’s standard-setting role while being inclusive and user-driven. As a result, we are attempting to balance this challenge by relying on sets of categories and tags—some of which will be emergent and community-defined—to link toolkits, people, case studies, and skills/capacity knowledge base.

Some abstract tags will be predefined and constrained as categories and others will be emergent and community-defined. The goal is to provide automated linking of these elements to augment our team’s capacity to convene and assist governments. For instance, when reviewing a page about a certain behavioural science toolkit, invisible tags will offer suggestions to the user:

We think that by starting with user-contributed data and information first and then making sense of the patterns that emerge, we will be able to explore and make sense of novel and innovative practices. This approach, in and of itself, will be a bit of an experiment within the OECD, even though it is commonplace elsewhere.

Inclusivity and accessibility beyond the “innovation class”

Adding to the challenge of this work has been the need to design for such a wide range of users, those beyond what my colleague Piret Tõnurist calls “the innovator class,” those officially charged with being innovative (with matching job titles and cool project names) and afforded the flexibility to work outside of rules and norms of an organisation. This class’ innovation method expertise and practice is often well-developed but organisationally isolated (and eventually at risk), an issue for organisational innovation capacity building and sustainability. This topic merits its own blog, but I mention it now as something on our radar for the final Toolkit Navigator design. Although we are sticking with the word “innovation,” we would like to signal to those outside of the innovator class that different ways of working are possible.

Our presumed users are diverse. Here are some examples:

• a civil servant in a service delivery role with a limited margin of action but curious to try a better way
• a civil servant with deep practice expertise in a particular methodology (i.e. lean process improvement) but knows a new methodology is needed for a different type of work
• a senior policy maker or civil service manager with a comprehensive but shallow view of their sector who wants to create more space for innovation
• a Chief Innovation Officer in a national government who rarely practises innovative methods but is a key convener, promoter and awareness-builder for new practises

We do not want to bore an experienced practitioner with basic knowledge or presume we know the best toolkits for them (after all, some innovators may reject being told what to do). However, we also want to encourage curious civil servants to get started—suggesting a curated and limited “starter kit” is less daunting. To balance the interests of these different users, we are building the navigation to accommodate a diversity of tasks and pathways to the toolkit content.

Based on our initial research, we know policy staff will want to review guidance and cases on innovative practices, a Chief Innovation Officer will want to easily share bits of information, and practitioners will want interactive tools for putting methods into action. Everyone wanted to connect with their peers for different reasons. Curiously, tasks related to designing a new strategy or connecting with others were five times more interesting to users than those related to exploring or solving problems. Because of these varying needs, our top level navigation includes several pathways to “Explore Topics,” “Take Action,” and “Connect” with many options to loop back to other types of tasks.

Toolkits as a public service

Using and adapting a toolkit can be a great way to build capacity and practice expertise within an organisation or team. We decided to only include toolkits that could be accessed freely, including downloads in exchange for an email address or free login. In fact, because many of the toolkits are licensed for adaptation (under Creative Commons 4.0 or similar) and because of the OECD’s unique position as a longstanding intergovernmental institution, we decided to take access a step further. In the spirit of public sector innovation dissemination, the Toolkit Navigator will be a living library, including the editable source files where the license allows and the publisher agrees. Many innovation toolkits serve as marketing tools for consultants—however, consultants’ real value-add is associated expertise, so we found that many publishers are willing to offer their source files as a public service or because it increases exposure to their methodologies and brands. We will highlight the publishers that license and provide their source material for adaptation and remixing. As part of the longer term product roadmap, we will enable toolkit versioning to encourage sharing adapted toolkits.

Showing social proof

OPSI is in a key position to convene and connect public sector innovators across the globe. Because innovation is a social learning process, we were sure to include social features in the Toolkit Navigator without creating a login barrier for those who just want to browse. Our platform supports user profiles and many social features but we recognise how much work proper online community management requires as well as how many well-developed communities already exist (for example, on Slack teams and in LinkedIn groups). While requiring user logins would provide us with better information on user activity and help us make better toolkit recommendations, we know that logging in is a huge barrier, so most of the Toolkit Navigator features will be available without a login.

Additionally, the Toolkit Navigator will include a review feature for each toolkit—similar to online product reviews. We want to enable users to share information about their use of the toolkits—including how it worked and its limitations. We have our own ideas about this, but we would like for the public sector innovation community to speak for itself.

Want to help?

We are grateful to the dozens of advisors who have shaped the direction of the Toolkit Navigator so far. We have come a long way but we still would like feedback as we test the beta site.  If you have 5 minutes, you can test the Toolkit Navigator and let us know what is helpful or confusing.

We are also seeking institutional content curation partners to help curate new toolkits, produce content defining the state of the topic as well as emerging trends, write and solicit toolkit reviews from the topic community, do some light community moderation within the Toolkit Navigator. Ideal partners are established thought leaders in their topic area. We are looking for content curation partners on the following topics:

• Design
• Social innovation
• Open government
• Public policy
• Service design
• Digital transformation
• Strategic design
• Open government
• Organisational design
• Behavioural insights
• Systems change
• International development
• Process facilitation & co-design
• Product design
• Futures & foresight

Would you like to partner with us or share feedback on our process to date? Please reach out at opsi@oecd.org