The São Paulo State Court of Auditors built a new analysis methodology, the School-Effect Indicator (IEE), which includes factors such as the socioeconomic status of students in the evaluation of school effectiveness. Through Artificial Intelligence (AI), it is calculated how much the school has contributed to the competence of its students, highlighting the principle of equity, recognizing the needs of each school group and working to reduce the impact of differences.
Innovation Summary
Innovation Overview
The São Paulo State Court of Auditors (TCESP), in the context of improving its main actions, identifies that the external control environment is being influenced by three main trends, addressed at the XXIII International Organization of Supreme Audit Institutions Congress (INCOSAI), which developed the document “The Moscow Declaration” and presented the guidelines for the triennium 2019-2022 of Supreme Audit Institutions (SAIs) of the Courts of Auditors Associated with INTOSAI (International Organization of Supreme Audit Institutions):
- Increase in social demands for Audit Institutions to evaluate the results of public policies, including those related to the Sustainable Development Goals (SDGs) of the 2030 Agenda;
- Need for Audit Institutions to deepen data analysis and;
- Need for SAI actions will have a greater impact on public management.
Within this alignment with INTOSAI guidelines, TCESP built a new analysis methodology, the School-Effect Indicator (IEE), which evaluates school effectiveness based on the analysis of factors related to both school and extra-school factors, such as student socioeconomic status. The work carried out was inspired by an article published in 2007 called School-Effect Indicator: a methodology for identifying school success based on data from the Prova Brasil, authored by teachers Amaury Patrick Gremaud, Fabiana de Felicio and Roberta Loboda Biondi, from the National Institute of Educational Studies and Research Anísio Teixeira (INEP.)
By evaluating schools according to the principle of equity, TCESP recognizes the needs of each school group and begins to work to reduce the impact of these differences. The identification of schools that provide their students with a better result than, in theory, is expected of them, brings to light a new inspection methodology, which evaluates education policies for being inclusive, of quality and equitable, aligning up to SDG 4 of the 2030 Agenda. The methodology hypothesis predicts that schools located in neighborhoods/districts whose population has greater purchasing power achieve a higher average in the result of their students in the National Basic Education Assessment System (SAEB). On the other hand, schools whose students belong to the most precarious socioeconomic strata of the population tend to present lower results in the final exam grade.
The project uses Artificial Intelligence to calculate the expected grade of students from each school and compares it with the grade observed in the SAEB exam. The difference between the observed and expected value becomes the added value of the school, that is, how much it has added to the proficiency of its students. Using the average grade of schools and socioeconomic data, we were able to estimate the portion of the grade that belongs to socioeconomic factors and, residually, the value added by schools to the performance of their students.
The School-Effect Indicator (IEE) is created from the residual of the model, that is, the difference between the actual grade of the schoolchild in the SAEB exam and the grade estimated by the model that only considers the socioeconomic factors of the students. It is an estimate of the portion of the grade that is dependent on school factors, and therefore a measure of school effectiveness. Example: A school with a score of 260 points in the SAEB exam, whose score estimated by the model is 250 points, has an IEE of 10 points. We can interpret it as follows: that school achieved 10 points more than what would be expected of schools with a similar student socioeconomic profile.
The IEE evaluation methodology allows the auditor to select the schools that stood out the most (highest IEE) to identify the good practices that possibly influenced the above-expected performance and, thus, such actions can be disseminated to other schools. On the other hand, schools with the worst IEE scores are also selected for evaluation and possible identification of the factors that determined a lower-than-expected performance. As the suggestions are included in the audit report, the public manager becomes aware of the notes and thus can work to suppress the factors causing the low scores and disseminate the good practices recognized by the TCESP assessment. The IEE brings numerous opportunities for conducting research on the learning environment, including good practices and effective processes, that explain why certain schools differ from others operating in similar conditions.
The current School-Effect Indicator analyzes state public schools. At this moment, TCESP and the University of São Paulo, through a cross-sector partnership in Artificial Intelligence, have been working together to improve the model and expand the analysis to municipal public schools, thus covering the entire universe of public schools in the state of São Paulo, whether they are linked to state or municipal managers.
Innovation Description
What Makes Your Project Innovative?
The School-Effect Indicator methodology is innovative because:
- It brings to light the analysis, done by the external control, of public policies in education to be inclusive and equitable.
- Allows TCESP to achieve greater impact on public management, reinforcing the value of public auditing by expanding the forecast of recommendations based on the analysis of strategic public policies in education, promoting good governance through the dissemination of knowledge and good practices.
- Use of techniques in Artificial Intelligence allows TCESP to synthesize a significant mass of data and find solutions to critical problems, such as the evaluation of school performance. The partnership with the University of São Paulo, through its Institute of Mathematics and Computer Science, provides a new perspective for improving the methodology and dissemination of the IEE, for managers across the country and for the academic environment.
What is the current status of your innovation?
After applying the methodology of the School-Effect Indicator, in the audit carried out in 2021 in schools selected as more or less effective, according to the IEE score, 13 recommendations were made for the management of the government of the state of São Paulo, in order to improve school performance based on findings and good practices identified by the TCESP audit. Based on these recommendations, issued in July of this year, TCESP monitors and controls eventual corrections and adoption of measures by government management. The IEE methodology will be extended to municipal schools, under the management of the 644 municipalities in São Paulo (except for the capital of SP, which is governed by the Court of Auditors of the Municipality of SP). To this end, TCESP has a cross-sector partnership with USP to improve the model and hierarchical analysis to include the comparison between different education networks.
Innovation Development
Collaborations & Partnerships
- Amaury Patrick Gremaud, Fabiana de Felicio and Roberta Loboda Biondi, from INEP – Authors of the article that inspired the creation of the TCESP methodology.
- Livia Kobayashi Hiratsuka – Author of the IEE at TCESP.
- Mariana Castro Pires – Co-author of the IEE at TCESP.
- Carmen Leite Vanin, Marina Ferreira Pacini and Tiago Romani Variz – Validation of the model for auditing at TCESP.
- Mariana Cury, Gabriel Tabak, Roseli Aparecida, from University of São Paulo (USP ICMC) – Improvement of the AI model.
Users, Stakeholders & Beneficiaries
- Governor's Accounts Board (DCG) – TCESP's audit unit that used the indicator to evaluate schools.
- Secretary of Education of the State of São Paulo (SEE) – Took knowledge of the evaluation and recommendations of TCESP on public schools of the state government and can work towards the adoption of measures to improve school performance.
- University of São Paulo (USP) - Incorporates the methodology for the academic environment and opens research opportunities in the field of education.
Innovation Reflections
Results, Outcomes & Impacts
The education system in the state of São Paulo has 5,351 teaching units, 5,044 of which are regular, 38 of indigenous education, 4 “quilombolas”, 16 in rural settlements, 153 penitentiary classes and 96 socio-educational service centers. This amounts to a total of 3,466,547 students served in 2021, according to data from the Department of Education. The School-Effect Indicator (IEE) was used for the first time in 2021, at TCESP, as a guiding element for the evaluation of public policies in education. The IEE was able to help the TCESP audit to select schools considered effective by the proposed methodology, identifying educational policies with greater evidence of impact on school performance. These audit findings were compiled in a report that came to the attention of state public managers in the form of 13 recommendations that can contribute to better student performance.
Challenges and Failures
The audit took place throughout 2021 and health restrictions due to the pandemic prevented on site inspections. Further, the atypical operation of schools made observation tests impossible. To overcome this challenge, the solution chosen was to intensify data analysis and videoconference interviews. The investigation regarding the execution of the Education Department's policies was also impacted by the disruption of the outlined plans. Remote learning showed how large socioeconomic conditions can affect students, where not everyone has access to adequate infrastructure and these factors had to be considered in the analyses. Finally, the theme has a close relationship with academic research, requiring specialized knowledge in the area. For this reason, the categorization of the schools studied, although carried out, will not be disclosed, so as not to cause harm to school communities. The analyses, therefore, were carried out for the schools together, avoiding stigmas, inaccuracies, or arbitrariness.
Conditions for Success
The methodology encompasses new ways of acting by the control bodies, focusing on the analysis of public policies, the use of artificial intelligence and the search for more efficient results with greater impact on public management. The School-Effect Indicator is a new element that needs to be disseminated to auditors, senior leadership, and later, to municipal and state managers. The involvement of the academic community is essential, considering that the topic needs education specialists to validate the analyzes carried out. Universities can also act in the construction of algorithms. Because of this, the cross-sector partnership between TCESP and USP is fundamental for the construction of a more assertive mathematical model. The IEE demonstrates the importance of information transparency and how it is essential for the diagnosis and construction of more effective public policies, even though data from the educational area (INEP) is no longer available as an open source.
Replication
The School-Effect Indicator (IEE) can be replicated throughout Brazil, considering that school management is based on national guidelines (Law 9394/96 and Law 14113/20) and the SAEB exam is national. The theme of equity, when raised by the IEE, can be expanded to the Municipal Management Effectiveness Index (IEGM), used throughout the country and winner of an honorable mention of the Innovare Award 2018. The IEGM evaluates the effectiveness of public policies in health, education, planning, fiscal management, civil defense, environment, and governance in information technology. By involving universities in the construction of the IEE methodology, opportunities are created for the elaboration and analysis of the effectiveness of public policies with the participation of education specialists. This increases the robustness of the IEE and facilitates its dissemination to other sectors of society, such as government representatives, civil society organizations and companies.
Lessons Learned
The idea of the School-Effect Indicator project came to meet a new demand inherent to the process of modernization of control actions, which demands from the Audit Institutions evaluations of the results of public policies, including those related to the SDGs of the 2030 Agenda. The Audit Institutions have been seeking to increase their impact on improving public management. An innovation, in this case, a new methodology for analyzing public policies, according to the principle of equity, can face resistance if the institution's people are not involved, whether they are part of the top leadership or the executors of the actions. The involvement of public workers helps to mitigate resistance, create sympathy, goodwill, and engagement necessary for the project to be successful and welcomed by the institution. Finally, the communication process is fundamental and good feedback to the participants provides a suitable environment for the emergence of innovations.
Anything Else?
The School-Effect Indicator project was selected for presentation at the 7th International Seminar on Public Administration Data Analysis, organized by the Federal Court of Auditors (TCU), the Federal Controllership General (CGU) and the National School of Public Administration (ENAP), in Brasilia, in the year 2021. The project is being perfected together with researchers and professors from the University of São Paulo (USP), because of the cross-sector partnership TCESP-USP for projects that use Artificial Intelligence.
Project Pitch
Supporting Videos
Status:
- Evaluation - understanding whether the innovative initiative has delivered what was needed
- Diffusing Lessons - using what was learnt to inform other projects and understanding how the innovation can be applied in other ways
Innovation provided by:
Media:
Files:
- Data Analysis Seminar - School-Effect Presentation slides at the 7th International Seminar on Public Administration Data Analysis
- Audit Report The document is part of the audit report of the accounts of the government of the state of São Paulo, fiscal year 2021. In it, the notes, inspection findings and recommendations made to improve school performance are reported.
- Technical Documentation of the School-Effect Project It reports the objectives, methodology, data used and results of the School-Effect model.
- Assessment of School Effectiveness by Hierarchical Models Presentation slides of the new proposal to improve the School-Effect Indicator, carried out in partnership with USP.
- Agreement signed TCESP and USP Document that formalizes the inter-institutional partnership between USP and TCESP for works using artificial intelligence. In the work plan there is a description of the 3 chosen projects. The School-Effect Indicator itself, a predictive model of public expenditure and a benchmark price database.
Date Published:
10 July 2023