A data led approach to managing harms and supporting vulnerable users

AFSA regulates the bankruptcy system in Australia. The system is designed to support those in vulnerable situations due to circumstances beyond their control. However, there are bad actors who exploit the system by hiding assets/income. Using our risk-based regulatory approach, we have designed a system that uses an applicant's data and validates it against external data sets to detect potential misconduct. We aspire to build a data model to predict non-compliance at the system entry point.

Innovation Summary

Innovation Overview

In the early 2000's as government service digitisation started to take shape, we began using a system automated checklist-based approach to select bankruptcy applicants for further investigation based on the disclosures they made on their application and some monetary thresholds and ratios. Those who met certain defined criteria were then subject to further scrutiny and investigations so that assets or income could be identified and recovered.

While this method did identify cases where an applicant had made honest disclosures about their assets/income which could be recovered, it did not necessarily identify all malicious and sophisticated actors who either did not disclose their true asset/income positions or disclosed information in a way that avoided detection. To cater for the large variations in debtor behaviours the system was designed so that it cast a wide net which unfortunately also caught a large cohort of honest debtors without recoverable assets who were then subject to the same level of scrutiny at a significant cost to taxpayers. This also created regulatory burdens on honest debtors at a time when they were under significant financial stress and seeking relief.

Importantly, this method also did not identify debtors who were particularly vulnerable due to their individual circumstances (e.g., family violence or physical/mental health conditions) so that appropriate government and/or NGO support systems could be identified and connected to them.

With the increasing understanding of the potential of data as a key enabler of targeted public service delivery, and with the global shift towards risk based and harms focused regulation that is in the public interest, a small team of service delivery, data, regulatory, and policy experts was formed to fundamentally rethink how AFSA could use its limited resources effectively and efficiently to:

detect and manage system level harms caused by bad actors,
identify and support vulnerable users who enter the system, and
maintain community confidence that the system is fair, and misuse is not tolerated

We used the 'Double Diamond Design Thinking' principles to (a) explore the problem, (b) define it, (c) look at a range of potential solutions, and (d) select a solution which we believe will work best for us. This approach was particularly useful as the problem exploration phase allowed us to look at the varied drivers of non-compliance, which in turn helped us to select the appropriate solution.

The solution design has three elements:

Randomized sampling of new cases: A statistically valid sample of new bankruptcy cases are selected for compliance assessments. The information disclosed by these debtors in their applications is validated against external data and where non-compliance is detected, debtors are interviewed to better understand the drivers of non-compliance and whether it was intentional or unintentional.
Harnessing Community Intelligence: We have begun to track all tip-offs and complaints against debtors and creditors to better understand community expectations and the drivers that cause erosion of trust in the system. The outcomes of these investigations (especially those that result in regulatory actions) are helping us inform the data model we are developing to detect non-compliance.
Deep Diving into Harms Hypotheses: We are periodically (typically every 3 months) doing deep dives into a sample of cases with unique characteristics to better understand the types of harms that manifest in the system and how they can be detected early so that preventative measures can be designed. An example of a harm is applicants obtaining advice from untrustworthy sources to either their own detriment or that of their creditors.

All three elements have generated a valuable new data asset which is yielding insights into the nature and magnitude of non-compliance and harms. This dataset is designed to enable statistically reliable estimates (SE ~ 10%, confidence level 5%), is subject to ongoing methodological review and refinement based on sound statistical principles (reducing and measuring uncertainty/defining risk) and engineered for continual improvement.

Preliminary work has commenced on leveraging the data from these activities to create a training dataset and build a machine learning model to auto detect harms and non-compliance at the point of a person's entry into the insolvency system.

From a user perspective, it will allow applicants to be alerted to potential issues with the information in the application they are submitting and pinpoint areas they may need to review. This capability will be enabled by a continually learning data model.

From a regulatory perspective, it will not only enable applicants to better comply with the regulatory framework, but also inform future policy and legislative changes to minimise harms.

Innovation Description

What Makes Your Project Innovative?

The three elements of the program have generated a valuable new data asset which is yielding insights into the nature, magnitude, and prevalence of harms in the bankruptcy system. This dataset is designed to enable statistically reliable estimates, thereby improving confidence in future service design, strategic planning, governance oversight, and data informed policy development and legislative design.

The innovative approach adopted by AFSA is based on harnessing:

Data,
Technology, and
Human intellect.

It demonstrates that digitisation of public services can deliver better value by leveraging data, technology, and human intellect to deliver solutions that are quick, seamless and user focussed, but which have appropriate guardrails to detect and prevent abuse that can quickly lead to loss of trust in the community.

What is the current status of your innovation?

Work has commenced to build a data model to detect non-compliance based on the training dataset that is being generated with the three activities that we are undertaking. This is iterative process that will be refined as the size of the training dataset increases and new insights are generated through the human led interview processes.

Preliminary work has also commenced on leveraging this dataset to inform user focused service design that provides for a streamlined experience for compliant users while ensuring appropriate frictions are created within the service flow which allows for harms and non-compliance to be identified quickly.

We have also commenced exploring automated external data validation at the application stage and are in preliminary discussions with commercial data aggregators.

The current activities have already identified significant areas of regulatory concern for us and also helped identify cases where the system could better support vulnerable users.

Innovation Development

Collaborations & Partnerships

The operational team's collaboration across AFSA was essential for success, including:

Business Applications - who found and developed innovative system solutions in our existing suite that would deliver what was needed for our staff conducting the compliance work; data that can be used for machine learning, and useful reporting
Data and Statistics - who assisted with developing the required samples and survey questions, making sure there is no bias and it is statistically significant.

Users, Stakeholders & Beneficiaries

Citizens will generally have more confidence in Australia's bankruptcy system. It sends a clear message to bad actors that the system can't be "gamed" now - it doesn't matter what is disclosed in an application - your application may be selected for a compliance assessment. The insights from this work also feeds into improved service delivery so citizens using our services will have an improved experience - make compliance simple.

Innovation Reflections

Results, Outcomes & Impacts

While it is early days, preliminary data suggests there is material non-compliance in 12% new cases.

Around 37.5% of this non-compliant behaviour was subsequently assessed as unintentional user errors which provides us with valuable insights to improve the design of our services to reduce unintentional user errors.

The remainder 62.5% was assessed as intentional/malicious non-compliance. These were typically bad actors seeking to hide assets and income from creditors by either not disclosing them or by transferring them out to friends or relatives.

All identified cases of material and malicious non-compliance were subject to regulatory actions which could include formal warnings, sanctions and/or referrals for criminal prosecution.

This re-imagined approach to managing compliance has significantly refined our understanding of harms. It has enabled us to better appreciate the experiences of our vulnerable users, as well as examine the intentions and approaches of bad actors.

Challenges and Failures

Regulatory impost on third parties - We have had to manage the expectations of third parties (other government agencies, banks, vehicle registries, etc.) as there has been a spike in the number of information requests being made to these organisations to validate the data disclosed on the debtor's application. We expect to commence discussions with a few organisations to move towards an automated electronic exchange of data to minimise the manual effort currently being applied by these organisations.

Internal downstream impacts within AFSA - When we commenced this program, we had failed to appreciate the high volume of incoming correspondence that would be generated because of the many information validation requests that we made to external organisations. The sheer volume of responses to our information requests created a significant backlog in our service centre, which in turn delayed compliance assessments. The proposed automated data exchange is likely to ease this challenge.

Conditions for Success

Culture that is conducive to continuous learning and experimentation
Resilience
Data ethics framework
A clear risk appetite statement
Leadership that enables people
Leaders communicating information what is changing and why, how these changes affect day to day activities, align to business goals
Change culture

Replication

The types of government services that could adopt this innovative approach to managing compliance include: Services where citizens require permission from government to perform an activity (e.g. licencing, permits, etc.) and which typically require the citizen to satisfy certain conditions and/or prove their eligibility. Applying the same level of scrutiny to every application can be costly and with little commensurate benefit. A smarter approach might be to only subject those applications to detailed scrutiny where there are sufficient triggers to suggest non-compliance or fraud. Further, a similar approach could also be taken in services for claiming government benefits or grants.

Lessons Learned

Stop thinking about it and just do it! It will never be a perfect time to implement a change of this type. You need to have a go, accept the first iteration may not be perfect, and continue to refine and improve. We could have waited until until we had external validation on some or all data points, however waiting for perfection will have continued our practice that we had determined was not fair for clients, and also not useful for our regulatory purposes.

🇦🇺