Forecasting GDP with Explainable AI - Observatory of Public Sector Innovation

We have developed an innovative GDP forecasting application based on Explainable Machine Learning (XML). It allows users to generate accurate and explicable economic forecasts from data sets with multivariate time-series. The application displays novel prediction changes for temporally ordered variable values, which largely increases the ability to explain predictions. It also includes a hybrid machine learning (ML) model that seamlessly combines all algorithms which outperform Sweden's National Financial Management Authority (ESV) own pre-pandemic GDP forecasts.

Innovation Summary

Innovation Overview

The International Monetary Fund (IMF) recently published a study of Machine Learning (ML) for multivariate time series forecasting of the GDP for a number of countries. It found that ML models not only outperform traditional statistical techniques, but also IMF’s own World Economic Outlook forecasts. However, the model with the strongest predictive performance consisted of a so-called black-box model, with low explainable power. In the discussion of future work, the IMF study points out the need for methods that are able “to unbox and interpret machine learning models to provide explanation for their outputs, and help understand the differences in forecast performance across a wide-range of model and expert-based forecasts” (1).

The innovation project at the Swedish National Financial Management Authority (ESV) was set out to find a solution to these needs of Explainable AI (XML) as well as building grounds for integrating AI in the financial management of Swedish government. XML is a set of tools and frameworks to help humans understand and interpret predictions made by ML models. In collaboration with a highly distinguished ML scholar at the KTH Royal Institute of Technology, we developed a cutting edge web application for analyzing the impact that each data variable has on the prediction of the underlying black-box model (2). It both displays temporal variation of the variable impact and the aggregated effect over longer periods of time. Such analysis is pivotal for officials working with economic forecasting in the improvement of existing forecasting models.

Traditionally, macroeconomic forecasting is based on a number a theoretical assumptions. In a continuously changing world, this form of theory driven analysis has proven its weaknesses. As a complement, the data driven approach is based on rather few assumptions and instead lets the algorithms find the best fit between input data and economic output. The main challenge with the data driven approach is nevertheless low levels of explainability. This means that ML algorithms provide us with accurate predictions but without theoretical explanation. In the data driven approach there is therefore a need to constantly build and revise the explanatory accounts. Our application helps solving this issue.

The application provides any user – politicians, researchers, journalists and citizens – a tool to perform accurate forecasts that continuously learn from historical changes in variable impacts. After closer evaluation, the application could potentially be integrated into any ordinary economic forecasts performed by different bodies of government and used as basis of governmental policy-making. It gives evidence on how to use existing ML algorithms without jeopardizing the level of explainability of predictions. It also shows the benefits of adopting a strict data driven approach to economic forecasting. For citizens, companies, and NGO stakeholders the application is driving for more open government, exposing all included data and methods in forecasting.

This application was developed in our Data lab at ESV. The purpose of the Data lab is to create a national arena for data on financial management to further the development of a data driven performance culture in the government administration and advance decision support at various levels. To fulfil the purpose, we develop several applications to conduct queries, computer processing, analysis and visualization with the support of machine learning and other AI methods. The work is carried out together with need owners, AI researchers, IT consultants, and remunerated students.

References
(1) Jung, J.K., Patnam, M., Ter-Martirosyan, A.: An Algorithmic Crystal Ball: Forecasts-based on Machine Learning. International Monetary Fund (2018), p. 27.
(2) Boström, H., Höglund, P., Junker, S. O., Öberg, A. S., & Sparr, M. (2020). Explaining Multivariate Time Series Forecasts: An Application to Predicting the Swedish GDP. In XI-ML@ KI.

Innovation Description

What Makes Your Project Innovative?

It is a unique forecasting model based on Explainable AI (XML). As far as we know, there is currently no other example of XML forecasting models developed and used by governments. It has the possibility to set new standards for economic forecasting in the modern AI era. Our XML application helps finding a rational balance between a theory driven and a data driven approach. In addition to calculating the aggregated variable effect, measured with or without a time constraint, our application presents an approach to visualize variables’ effect on individual predictions, and showing variations over time. Anyone can use our application to forecast Swedish GDP. Currently, outcome data is available up to and including the third quarter of 2021, and forecasts can be made up to and including the third quarter of 2024.

What is the current status of your innovation?

The XML application is in a stable and testable format and we are gladly sharing the code to anyone interested in XML prediction modelling. The latest development of our innovation project is to let users configure and model the input features/variables, in order to test new ideas and develop more rigid prediction theories. In the short run, our aim is to spread the developed XML forecasting model to other macroeconomic projection areas such as inflation, wages, unemployment and international trade. In the mid-term, we expect to run a new project that includes other types of data, e.g. daily statistics from digital trading behavior or text data from economic reporting. Overall, we believe that XML algorithms will most probably fundamentally change the processes of making forecast and prediction within the coming decades.

Innovation Development

Collaborations & Partnerships

The XML application was designed and programmed in collaboration with a highly distinguished ML scholar at the KTH Royal Institute of Technology. In the testing of the application, we have used a broad network of officials in government. In two separate workshops the applications were advanced to meet user needs.

Users, Stakeholders & Beneficiaries

Governments use forecasts to project the consequences of their policies for reaching specific targets and goals. Moreover, economic forecasting at both the national and international set the agenda for governments and function as rationales for new initiatives and policies. Government officials are the main target group and beneficiaries for our XML forecasting application. The target group can potentially use the application as a tool for developing more efficient policies.

Innovation Reflections

Results, Outcomes & Impacts

Currently, the application is available for all users on ESV’s website and we use it in different settings to lab and test ideas to target groups.

In addition to this, the innovation Forecasting GDP with Explainable AI has generated three main results. Firstly, we confirm the core proposition of the referred IMF article (see above), i.e. that ML algorithms are likely to outperform humans in making economic forecasts on the short and mid-term. This ensures good conditions for the continuing of this service development. Secondly, our new forecasting model is in line with the OECD’s appeal for data driven innovation and that countries should “act to seize these benefits, by training more and better data scientists, reducing barriers to cross-border data flows, and encouraging investment in business processes to incorporate data analytics” (1). Thirdly, it encourages the furthering of Explainable ML (XML) and gives a concrete example of its benefits.

References
(1) OECD (2015), Data-Driven Innovation – Big Data for Growth and Well-Being, https://www.oecd.org/sti/data-driven-innovation-9789264229358-en.htm

Challenges and Failures

The integration of ML in governments’ economic forecasting raises novel challenges for ensuring fairness, accountability, and transparency (FAT) in policy-making. This normally holds back any attempt of ML development despite a wide knowledge of the regular “errors” such as bias and noise in established human forecasting methods. Accountability, i.e. the need for someone to be held accountable if the XML forecasting goes wrong, is the main challenge we have encountered in our innovation project. As a response, we are presently working on different paths to establish a solid measurement of the application’s impact. Our intention is to collaborate with external researchers of economics and machine learning to develop numeric results of the innovation.

Conditions for Success

The innovation is a result of a fruitful collaboration between government officials and academic researchers in the ESV Data lab. The partners have been involved in running one or more related sub-projects with feedback loops. The work is characterized as iterative and exploratory. On several occasions, we have participated in both public seminars and workshops with external experts and government representatives. Results have been shared through primarily open workshops regarding several of the applications and two public knowledge meetings.

Replication

Within the organization we have taken first steps to integrate the innovation in ordinary forecasting procedures. A decision has been made to launch the hybrid XML model on our agency’s new interactive website, which continuously is launched during 2023. Up to this point in time we have shared the developed XML code with two other external developers. However, there are still no replications to address similar problems.

Lessons Learned

The government organization of making economic forecasting is a highly institutionalized field. Few steps have been taken to adopt to Big Data and data driven analysis. In contrary, most conducted economic predictions still use small amounts of data with quarterly based annual frequency. It is important to work across boundaries with academic researchers, companies, and civil society interest for furthering the integration of AI in government processes. The main lesson we would like to share to other government officials is this: Have the courage to work iteratively across organizational borders to develop testable applications and to promote the digital transformation.

🇸🇪