General Information
Project description
The Ontario Treasury Board Secretariat (TBS) is responsible for paying Ontario Public Service (OPS) employees on a bi-weekly basis. The Pay and Benefits Service Centre inboxes receive a large volume of documents. Pay and Benefits transactions are processed according to Pay Processing Deadlines and Pay and Benefits Service Priorities. Each month, OPS employees are required to confirm their attendance in the OPS Human Resources system (WIN System) for the previous month, and managers are encouraged to approve these attendance records within the first 15 days of the month. When attendance approvals are not completed on time, this can result in delays and additional manual work for staff.
The Ontario Behavioural Insights Unit (BIU) has partnered with TBS to apply a behavioural science lens to help increase the rate of timely attendance approvals from its current baseline of 65%. Informed by a behavioural analysis that identified key friction points in the attendance approvals process, the BIU plans to conduct a randomized controlled trial among a sample of ~7,300 OPS managers to test the effectiveness of behaviourally informed reminders in increasing timely attendance approvals.
Analysis Plan
Pre-analysis plan: Is there a pre-analysis plan associated with this registration?
Hypothesis
Behaviourally informed reminders will increase on-time attendance approval rates relative to the baseline (65%).
The effect of reminders on timely attendance approvals will vary as a function of reminder condition.
How hypothesis will be tested
Three conditions in a randomized controlled field experiment. Participants will be randomly assigned to one of three conditions:
(1) social norm reminder;
(2) calendar invite reminder;
(3) passive control (no reminder).
Reminders will be sent out as emails. Data will be collected on an individual level for seven weeks and then analyzed to quantify the effect size of reminders and identify any statistically significant differences between conditions.
Dependent variables
Primary outcome
On-time approval of attendance (yes/no) based on administrative data.
Secondary outcome
Email engagement (i.e., email open and click-through rates) as recorded by the email platform.
Analyses
The effects of behaviourally informed reminders on attendance approvals will be analyzed by means of multilevel logistic regression models that assess managers’ probability of approving their employees’ attendance on time (primary outcome) and engaging with the email content (secondary outcome), respectively. The models will specify simple fixed effects for the reminder version managers received and the employment status of their employees. To account for nesting within the data (e.g., in cases where a manager has multiple employees reporting to them; observations from different ministries), the models will further specify random effects for manager and ministry as determined by model fit. The random-effects structure (i.e., random intercepts vs. random slopes) will be selected based on the maximal complexity supported by the data.
Sample Size. How many observations will be collected or what will determine sample size?
Approximately 7,300 participants as determined by the total number of managers in the OPS (~2,430 per condition)
Who is behind the project?
Project status:
Pre-registration
Methods
What is the project about?
Date published:
24 April 2025