The reason for creation was to improve inspection models of external control, which were based mainly on legality and conformity. IEG-M index is an instrument for measuring public area effectiveness, as it analyses the inputs, resources and processes of the local administration, evaluating its policies and activities. There are many finalistic results indexes, but none of them deals with the evaluation of processes to analyse the application of public money that impacts local communities.
The index creation (IEG-M) established a new observation framework in public external control processes. It presents relevant elements in public management activities in 7 major areas:
- Tax Management
- Protected Cities
- Governance in Information Technology.
These dimensions were selected based on their strategic position in the context of public finances and the weights were distributed in the historical budgetary proportion of these themes.
IEG-M is innovative because it establishes annual measurement of processes in the budget execution of the local public administration with results presented in bands. The Brazilian federation has more than 5700 municipalities (local administration), whose managers essentially follow the constitutional principles for the application of public resources to achieve the common good, however, there were no technical parameters to analyse the expenses incurred as a result of the collection of taxes.
One of the main index objectives was to create technical control points in the management of the local administration to bring to the communities in need more quality in the public services of the local government such as Education, Health, Safety and Fiscal Management. Real benefits can be found in different target audiences:
- Public Audit: for the first time in the State of São Paulo, later expanded to Brazil, public audits and budget execution started to have analysis parameters, in addition to the possibility of historical analysis;
- Public Account Judges: It delivered technical elements found in the audits, so that the analysis processes were not based solely on the letter of the law, but on the execution of processes that could impact society and local communities, whose judgments could take into account the common good;
- Public Managers: they find new ways in public policy processes to meet the general interest by observing the indicators in the 07 dimensions (Health, Education, Planning, Environment, Security and use of Technology), without this contradicting the rules dictated by the political party you are a part of or your administrative thoughts.
The index is composed of the combination of the following items:
- Government data and other official sources of information
- Data from automated inspection support systems CTAAT - Computer Assisted Auditing Techniques)
- Information collected from questionnaires filled in by Municipalities.
- IEG-M is a dynamic and flexible index, which allows its adaptation from year to year according to the consolidation of a certain policy by all or most municipalities or with the addition of new topics of social interest.
Each dimension is calculated individually and the numerical score results in the classification of municipal entities in 5 ranges:
- Range A - Highly Effective
- Range B+ - Very Effective
- Range B - Effective
- Range C+ - In the adaptation phase
- Range C - Low level of adequacy
In addition, the questions answered go through control algorithms, so that the points outside the expected curve are presented to the audit teams, optimizing the time spent in the planning of control actions, given the diagnosis of the sensitive points of the management.
Therefore, the simple methodology used (WEB questionnaire), adjusted to statistical mathematical models, allowed the IEG-M project to be used by a wide spectrum of actors: public account judges, public managers, audits, the press, public institutions, academics and researchers, interested companies, social control and population.
What Makes Your Project Innovative?
There was no means index in Brazil that tried to measure inputs and processes of public management in this range. Existing indexes until then were essentially results.
IEG-M observes how the local public manager executes its policy and if there was an alignment with the federal, regional plans and with local interests.
Innovation was necessary, as the environment at the time (2014/2015) was one of increasing linkages between assessment of municipal accounts and formal and conformity aspects, and reduced verification of public spending effective benefits delivered to communities.
São Paulo Court of Auditors has always sought to be at forefront of public accounts auditing processes, however the vision has always been one of legality and conformity: the challenge was to improve audit reports so that more technical judgments would find, in the communities, the meaning of the expression quality public service.
What is the current status of your innovation?
IEG-M is in its eighth edition (execution year). Annually, maintenance and scores calculation are carried out by a small group of servers, updating the tool so that other sectors can plan and evaluate local public management nuances, both for specific inspection operations (specific subject audit) and for annual local management evaluation.
Initially, local public administration participation was voluntary. In 2016, the questionnaire was included in Annual Accountability Calendar. In 2018, the index results monitoring was included in Annual Accounts Reports. In 2020, there were index mentions in some account’s judgment.
In 2021 and 2022, the low performance for consecutive years will be considered as a reason for an unfavourable opinion on local public manager accounts. Lessons learned from the innovation were replicated in other management indexes creation and in methodology application by all Brazilian federation Audit Courts.
Collaborations & Partnerships
The creation and maintenance of the IEG-M was only possible with interaction between auditors from São Paulo Court of Auditors and specialists from other areas of regional and local governments, such as State Secretariats of Education, Health, Environment, Military House/Civil Defense; as well as institutions such as São Paulo State Regional Medicine Council – CREMESP; CETESB; TODOS PELA EDUCAÇÃO Foundation; Brava Institute.
Users, Stakeholders & Beneficiaries
The beneficiaries of the IEG-M are:
- TCESP: tool used to analyze public accounts and assist in advisers' opinions
- Local public manager: policy measurement mechanism, course correction, reassessment of priorities and consolidation of planning
- Local legislative council: database for consultation and supervision of the local public manager
- Researchers and students: annual data source on local public management
- Citizen: consultation indicator for social audit
Results, Outcomes & Impacts
Regarding the impacts, IEG-M became one of main criteria for risk matrix elaboration of inspection type by local government, it was included in inspection scripts for answers validation "in loco", notes generated by the index are analysed and corroborate reporting counsellor vote, with many cases of accounts rejection due to index poor performance.
Index made it possible to detect sectors that deserve greater surveillance of inspection action. It is observed, with each new year of indicator use, that public manager seeks to achieve social interests and search for common good in points that were once little explored.
What is expected is that public manager uses this policy measurement instrument to evaluate positive points and attention points of its management every mandate year and to implement other municipalities good practices of similar size or neighbouring cities. And that society in general can effectively carry out social control.
Challenges and Failures
The biggest index challenge was public methodology recognition so that it can be used to further improve people conditions in situations that require public intervention.
Another challenge was constant legislative updating. Brazil is in maturing process of its laws and regulations, there are national, regional and local targets that are regularly reviewed and emergency needs arise that public manager needs to meet and the index has this flexibility to meet these changes.
Some flaws, over the 8 years of use, were (and are) observed, such as low technological investment in public management analysis models, circumvented by internal solutions adapted to demand. There is still a need to improve processing algorithms of innovative tool to produce reports and alerts that can be used in real time.
Conditions for Success
IEG-M tool was created by external control technicians to be used as an inspection tool. We understand that the innovation was successful because of engaged and technically knowledgeable work team; and for institution's top management support to facilitate norms approval and reinforce guidelines for agents involved, allocating human, technological and financial resources to the project. The involvement of all affected actors, the participation and guidance of those under jurisdiction through lectures, manuals, a direct service channel, the availability of an intuitive technological tool that is easy to understand and the dissemination of results in a broad and transparent manner allowed the index exposure and its purposes generally and continuously; these are conditions for its success.
The same methodology was replicated by all Brazil’s Courts of Auditors through IRB –Rui Barbosa Institution, which consolidates the results of all municipalities in the IEGM Brazil.
The relevance of the project was recognized by the UN, which, through the United Nations Development Program (UNDP), signed a partnership with TCESP for the use of the IEG-M as the organization's official tool for monitoring municipal actions in the State of São Paulo towards achieving the goals of the SDGs – Sustainable Development Goals.
Given this success, countries such as Argentina, Angola, Mozambique and Cape Verde have also shown interest in adopting the methodology.
The index was also the inspiration for the creation of 2 other indicators by TCESP: the IEG-E (State Management Effectiveness Index) and the Municipal IEG-Prev (Municipal Social Security Management Effectiveness Index), with the IEG-E already was replicated in some courts of auditors in other states.
- Be bold: team brainstorm because everyday conversations, without pretense, can turn into real project
- Perseverance: lot of criticism will come and go, absorb what you can, solve what is plausible and discard the rest, pleasing everyone is impractical
- Lot of study: technical knowledge is essential when it comes to an objective tool, so dedication is necessary to understand topics relevant to society, legislative updating, judicial decisions, doctrinal currents and good market practices
- Creativity: creating small solutions without losing global concept, small tasks are the ones that require the most commitment
- Technological knowledge: “homemade” technological solutions can be a kick-start until the project gains volume
- Listening to the people involved: understanding the needs of the team and peers, other internal areas of the institution, people outside the institution, such as: jurisdictions, other institutions in the field, specialists in the area, educational institutions and society.
TCESP signed a partnership with the UN, through the United Nations Development Program (UNDP), for the use of the IEG-M as the organization's official tool for monitoring municipal actions in the State of São Paulo in relation to the goals of the SDGs – Sustainable Development Goals.
The Ministry of Economy introduced the IEG-M/TCESP as an evaluation criterion for voluntary transfers from the federal government to municipalities.
In 2018, IEG-M/TCESP was a finalist for the Innovare Award, one of the largest and most important awards in the legal field in Brazil, which encourages and disseminates practices that collaborate to improve the quality of services delivered to Brazilian citizens.
- Implementation - making the innovation happen
- Evaluation - understanding whether the innovative initiative has delivered what was needed
- Diffusing Lessons - using what was learnt to inform other projects and understanding how the innovation can be applied in other ways
26 January 2023