Skip to content
An official website of the OECD. Find out more
Created by the Public Governance Directorate

This website was created by the OECD Observatory of Public Sector Innovation (OPSI), part of the OECD Public Governance Directorate (GOV).

How to validate authenticity

Validation that this is an official OECD website can be found on the Innovative Government page of the corporate OECD website.

Increasing participation in the Adult Migrant English Program (AMEP)

General Information

Project description

The Adult Migrant English Program (AMEP) is a government-funded program for all permanent visa holders and some temporary visa holders who have less than a vocational level of English. Not all people who are eligible for the AMEP participate in it. The Department of Home Affairs commissioned BETA to evaluate the effectiveness of communication in participants’ home languages on increasing participation in the AMEP.

Trial 1: We will send text messages to speakers of Mandarin, Cantonese, Vietnamese and Arabic who have previously enrolled in the AMEP and have disengaged. Text messages will be in either participants’ home languages or English and come from their service provider or the department.

Trial 2: We will send letters and emails to speakers of Mandarin, Cantonese, Vietnamese and Arabic who are eligible for the AMEP and have never enrolled. The letter and emails will be in either participants’ home languages or English.

Analysis Plan

Pre-analysis plan: Is there a pre-analysis plan associated with this registration?

Yes

Hypothesis

Trial 1
H1: There will be a higher proportion of re-engagement among people who received a message in their home language as compared with those who received a message in English (A1 > A0).
H2: There will be differences in re-engagement between the Department of Home Affairs messenger group and the service provider messenger group (B1 ≠ B0)

Trial 2
H1: There will be a higher proportion of enrolments among people who receive the letter/email in their home language as compared with those who receive the letter/email in English (A1 > A0).

How hypothesis will be tested

Trial 1: A 2x2 factorial design, students will be randomised at the address level so people at the same addresses are in the same treatment group (clustered). We will stratify by service provider region to ensure balance in treatment group assignment across the four regions.

Trial 2: Potential students will be randomised at the address level so people at the same addresses are in the same treatment group (clustered). We will stratify by service provider and language region to ensure balance in treatment group assignment across the four regions.

Dependent variables

Primary outcome
Trial 1: The primary outcome measure is re-engagement in the AMEP, indicated by a scheduled activity date (when the participant contacts the service provider to schedule returning to class).
Trial 2: The primary outcome measure is enrolments in the AMEP, as measured by registration date.

Secondary outcome
Trial 1: The secondary outcome will be the proportion of participants who click on the link embedded in the text message.
Trial 2: The secondary outcome will be the proportion of participants who visit the link embedded in the email and letter.

Analyses

Trial 1
For both primary outcome measures, we will estimate treatment effects using Ordinary Least Squares (OLS) regression with standard errors (CR2) clustered using the following specification:
Y= α+ β_1 A+ β_2 B+ γ_1 X+ γ_2 X*A+γ_3 X*B+v+w
Where Y is either a binary variable indicating whether the student re-engaged. A indicates whether the cluster is allocated to the home language or English language group, B indicates whether the cluster was allocated to the Home Affairs or provider messenger group, X indicates the set of de-meaned baseline covariates (years of education) and vector of block indicators, X * A , X * B are the interaction of these covariates with the treatment indicators, v is a cluster-level error term and ω is the individual-level error term.
We will not adjust p-values for multiple comparisons. We will conduct logistic regression as a robustness check for our primary OLS specification and report average marginal effects.

Trial 2
For the primary outcome measure, we will estimate treatment effects using Ordinary Least Squares (OLS) regression with standard errors (CR2) clustered using the following specification:
Y= α+ β_1 A+ γ_1 X+ γ_2 X*A+v+w
Where Y is a binary variable indicating whether the student enrolled. A indicates whether the cluster is allocated to the home language or English language group, X indicates a vector of block indicators, X * A is the interaction of this vector with the treatment indicator, v is a cluster-level error term and ω is the individual-level error term.
We will not adjust p-values for multiple comparisons. We will conduct logistic regression as a robustness check for our primary OLS specification and report average marginal effects.

Sample Size. How many observations will be collected or what will determine sample size?

Expected sample size:
Trial 1: 26,000
Trial 2: 16,990

External link

Who is behind the project?

Institution: Prime Minister and Cabinet (PMC)
Team: Behavioural Economics Team of the Australian Government (BETA)

Project status:

Pre-registration

Methods

Methodology: Field Experiment, Mixed Methods
Could you self-grade the strength of the evidence generated by this study?: 1
Data collection: Have any data been collected for this project already?: No data collection has occurred
Start date: 12/27/2021

What is the project about?

Policy area(s): Social Welfare
Topic(s): Services Uptake
Behavioural tool(s): Messenger effect

Date published:

14 December 2021

Comments are closed.