Ukraine is transforming its public procurement system to bring public procurement into the open, so citizens get a better deal and government runs smoothly. EBRD has supported the Ukrainian government to develop a new legal framework, and to develop cutting edge tools which can process vast amounts of procurement data in real time. State auditors can now quickly & pre-emptively spot risks or inefficiencies in the system and address them. It is the first innovation of this kind in the world.
Innovation Summary
Innovation Overview
Ukraine’s annual public procurement budget is worth up to US$20 billion. How it is spent has a massive effect on its people – it determines what medicines they can access, what schoolbooks they read, and how easy it is to travel.
Following decades of secrecy and corruption, the revolution of 2014 created the conditions to overhaul the public procurement system for goods, works and services. The intention was to give taxpayers a better deal and make government more effective and trusted.
To achieve this a new legal framework and cutting-edge tools were needed, built on the principles of open government and open data. EBRD has worked with the Ukrainian government, civil society and businesses to develop both. This has allowed the country to leap ahead of the rest of the world in how it manages procurement.
First, EBRD led a collaboration of government, civil society and businesses in building a cutting-edge digital system called Prozorro, which means “transparency” in Ukrainian. Built on the Open Contracting Data Standard (OCDS), this system can manage the entire public procurement process systematically, efficiently and transparently. Key to its potential is the ability of auditors to use the data to detect risks and inefficiencies early before they become problems.
During a successful pilot phase, the system saw excellent uptake and generated vast amounts of data. When processed effectively in real time, these data enable auditors to proactively anticipate risks, react quickly when abuses or abnormalities are detected, and flag gaps and inefficiencies in the system for future improvement.
Initially, it was assumed that civil society would process this data, but it was soon obvious they lacked the resources, skills or capacity. Data that could be fed back into the system to flag risks and make it more efficient and transparent was not being processed, which significantly hampered its effectiveness.
So, the EBRD team behind Prozorro again worked with the Ukrainian government to first develop the legal basis for a system of indicators to flag these risks early and mitigate them. They then developed tools that could deliver this, combining cutting edge business intelligence technologies with real-time risk analysis techniques.
The Prozorro system is unique in drawing information from existing commercial platforms into a single central repository. So, the auditing system needs to be able to plug data from lots of different platforms into a singular system and detect and flag risks across a constantly changing data set.
A critical part of the reform was to create an independent body responsible for picking up previously unknown risks identified by the new analytical tools, flagging them to auditors and identifying ways the system could be improved.
These innovations will have benefits right the way through Ukrainian society. It is designed for use by government officials who use it to ensure the government tendering service is fair and efficient. It allows them to do things in minutes that would take their peers in other countries days, and to ensure they are enforcing the principles of open contracting now enshrined in Ukraine’s law.
These changes make it much harder for the procurement process to be abused, and easier for civil society to monitor and support it. It allows policymakers to show they are building a healthy, inclusive economy which rewards innovation and quality of service effectively. And it brings new companies and workers into the economy by reducing biases and other barriers to entry. The ultimate beneficiary is the Ukrainian citizen, who gets better products and services from companies equipped to the do the job, at the lowest possible price.
The process has been built with the future firmly in mind. Still in its pilot phase, the platform has been built using data-driven analytical tools which highlight instances of risk in procurement, and feed back into these principles and inform them for future.
All data on suppliers and previous tenders will be online and made easily accessible to the authorities. Any risks or issues with a specific company will be logged in the system for future reference. Data can be analysed to detect common problems across the whole procurement system, and any changes needed for future. This means it can be constantly improved and refined to deliver the best possible service for the people of Ukraine.
The innovation’s ground-breaking approach and early success has implications for state procurement all over the world, and the open government movement in general. It conforms with the OCDS, meaning the same principles and approach can be applied anywhere in the world.
The project is currently under review by the OECD Auditors Alliance, and a version of the project will be piloted in the Kyrgyz Republic from June – December 2019.
Innovation Description
What Makes Your Project Innovative?
Monitoring of public procurement is usually manual, making it hugely susceptible to manipulation, corruption and inefficiency. Because it works with open data, Prozorro can help consign such problems to the past. EBRD developed a set of algorithms that analyse open data from procurement processes submitted online and automatically flag risks to the enforcement agencies as soon as they occur. Because these risks are processed by an automated platform, rather than by humans the system is incorruptible.
This makes it much more effective in preventing risks and provides a solid knowledge base with which to further improve the system. Auditors can do their jobs more effectively on a day to day basis, and can also step back to look across the whole public procurement system and identify patterns in how and where risks emerge. This data can then be fed back into the system to further improve the automated risk indicators. This is the first innovation of this kind in the world.
Innovation Development
Collaborations & Partnerships
The State Audit Service of Ukraine helped design the ex-ante monitoring methodology and now use it for monitoring procurement.
The State Procurement Agency developed automated monitoring tools which flag risks in procurement processes to auditors. The Ministry of Economic Development & Trade is ultimately responsible for implementing procurement reform. They set the rules of the system. CSOs including Open Contracting Partnership and Transparency International gave insights into the problem.
Users, Stakeholders & Beneficiaries
The SAS uses the tool to monitor public procurement, to flag risks it should be addressing in its efforts to ensure Ukraine’s new laws on transparency in procurement are enforced
The Ministry of Economic Development and Trade sees the benefit of the tool is as part of its drive to transform how taxpayer money is spent on goods and services.
Civil society organisations analyse the data to improve the accountability of government officials for public procurement.
Innovation Reflections
Results, Outcomes & Impacts
The ex-ante monitoring system has 27 unique algorithms and 35 automated risk indicators, which allows officials to track and monitor procurement transactions in real time e.g. if tender documentation has been uploaded incorrectly, or deadlines have been missed, the system flags this up for potential investigation.
A second type points to the likelihood of a given risk so that auditors know what to look out for. Examples include disqualifying all but one bidder from a process or awarding it to someone with no experience. When the system picks this up, auditors can then follow up to understand why.
15% of transactions are flagged as risky each week, while 9% are reported to officials as requiring immediate attention. This makes the monitoring process quicker and more effective in stopping abuses. EBRD is also helping policy development and proposed monitoring methodology informed the development of Ukraine government’s new law on public procurement monitoring passed in 2017.
Challenges and Failures
The project involved designing something that has not been done before, anywhere in the world. Furthermore, the whole Prozorro system which this tool is designed to monitor is still very new in Ukraine. This meant challenges appeared and evolved fast.
Conditions for Success
The integrity of the system relies on auditors having a shared understanding and approach to risk management. It is therefore essential that all officials are trained to the same standard and are attuned to risk. At the beginning the system flagged credible risks which had no precedent under the old manual system. So patterns identified in the data were used to show that these were genuine risks and push for them to be included in the development of new laws.
The effectiveness of the Prozorro system as a whole needs to be regularly assessed, to enable the monitoring tool to detect new and emerging risks. The government should introduce policy to make this happen.
The quality of the open data that the tool uses is critical to the functioning of the whole system. Ukraine needs to keep investing in supporting officials and companies to ensure they know how to enter quality data into the system.
Replication
The tool is designed to work for any public procurement system that supports the open contracting data standard (OCDS) format.
All the risk parameters are adjustable and can be tailored to suit the auditing needs of any public procurement system.
The system is constantly being improved as more data is gathered on its effectiveness, problems are ironed out and new opportunities for reform spotted.
Providing they are implemented properly, the automated indicators are incorruptible and provide results of the highest integrity, because there is no room for human interference.
The tool can be developed further, as it is easy to add new algorithms and integrate additional data sources.
A pilot of the tool will be tested to audit online public procurement in the Kyrgyz Republic.
Lessons Learned
People: A sophisticated risk assessment framework drawn from state-of-the-art business intelligence techniques underpins the system. Principal stakeholders were not familiar with these principles and initially struggled to apply them to their work.
During the implementation phase we worked to introduce the principal stakeholders to basic risk management concepts. This has seen some success, but further capacity building is required.
Users need to be trained in risk management concepts before using them. Time is needed to develop a shared understanding of likely risks within a specific public procurement process.
Process: Embedding an automated risk-based monitoring approach to an entire procurement system is a complex task.
The procurement process faces a range of risks including errors, irregularities, fraud, corruption and inefficiency. The sources of risk also vary widely, from the type of procedure, contracting authority, region, market or supplier.
We took pragmatic decisions to group risks into a representative set of indicators for the pilot. This was reductive, so it was critical to keep feeding data back into the system to refine these categories.
It is important to account for the fluid nature of risks in public procurement monitoring, and ensure data is analysed to detect those risks.
Technologies: Data quality is vital to the effectiveness of any automated tool for monitoring risk.
The lack of shared understanding among users of the type of risks encountered and the ways in which the data should be entered limited the range of automated indicators which could be developed for this procurement process.
The data produced was also uneven in quality because the underlying systems were not set up to deliver systematic data – much of the old system relied on free text fields, for example.
Data quality management and training should be provided for all system users and sources open data generated by open governance projects.
Innovation provided by:
Media:
Date Published:
12 April 2019