General Information
Project description
The Ontario Public Service (OPS) is committed to promoting employment equity. Collecting sociodemographic data from employees will help the OPS better understand how different demographic groups are represented in the workforce and progress through their careers. However, recent initiatives to promote the voluntary submission of sociodemographic data through the OPS Human Resources portal (known as WIN) has had lower uptake than desired, reducing the quality of this data for decision-making purposes. The Ontario Behavioural Insights Unit has partnered with the People and Culture Division of the Ontario Treasury Board Secretariat to run an experimental pilot project testing whether behaviourally-informed messaging can influence the movable middle of employees for whom minor friction costs and forgetting may keep them from taking action to volunteer this data. Later workstreams will build on these findings to better understand and address additional motivational barriers to sharing this information.
Analysis Plan
Hypothesis
1. Behaviourally-informed emails prompting employees to provide their sociodemographic data will increase the rate of sociodemographic data provision relative to the status quo email and a passive control group receiving no emails.
2. Relative to the status quo email, behaviourally-informed emails will increase email engagement, as defined by email open and click-through rates.
3. Both status quo and behaviourally-informed emails will be more likely to increase sociodemographic data provision when sent closer to the dates on which employees confirm their monthly attendance within the same HR platform (usually at the end of a month or the beginning of the next month). A parabolic relation between the email sent date and sociodemographic data provision is hypothesized, whereby sent dates closer to the transition between months should result in higher levels of sociodemographic data provision (inverted ‘U’ shape).
4. No a priori hypotheses are defined regarding the effects of sent date on email engagement, as defined by email open and click-through rates. Potential effects will be explored as part of the data analysis.
How hypothesis will be tested
Five conditions in a randomized controlled trial. Participants will be randomly assigned to either the passive control condition (receiving no email during the trial) or one of four email conditions: (1) status quo (text adapted from initial promotional material for the launch of the sociodemographic initiative), (2) simplified information, (3) implementation intention, and (4) active choice. Emails will be sent out evenly across conditions over a 15-day trial period, starting approximately a week before the end of the month and ending approximately a week after the beginning of the following month. Sociodemographic data provision and email engagement will be compared across conditions and sent dates.
Dependent variables
Primary outcome
• Entry of sociodemographic data into the HR platform. To preserve the privacy of employees, the data will be reported to the research team as aggregate counts per condition and sent date.
Secondary outcomes:
• Open rate of email message as recorded by the email platform.
• Click-through rate of email message as recorded by the email platform.
Analyses
Logistic regression analyses will be conducted to analyze differences in the effects of conditions and sent dates on sociodemographic data provision, email open rates, and click-through rates. Note that only aggregate data will be reported to the research team to preserve employees’ privacy. The aggregate data will be in the form of frequency tables showing rates of sociodemographic data provision by experimental condition and sent date. However, because only categorical and ordinal variables are being examined in the analyses, it will be possible to reconstitute anonymized individual-level data by weighting the variable levels by observed frequencies. As such, both chi-square and logistic regression models may be used to analyze the data.
To compare the effects of different email conditions on sociodemographic data provision and email engagement, the following planned orthogonal contrasts will be used:
Contrast 1: Passive control vs. all 4 email conditions.
Contrast 2: Status quo email condition vs. all 3 behaviourally-informed email conditions.
Contrast 3: Active choice email condition vs. simplified information & implementation intention email conditions.
Contrast 4: Simplified information email condition vs. implementation intention email condition.
To test for an effect of email sent date, a logistic regression will be run using sent date as an ordinal variable to predict the likelihood of sociodemographic data provision. The hypothesized parabolic relationship between sent date and outcome likelihood will be tested by determining whether a logistic regression model including both x_date and x_date squared as predictors explains significantly more variance than a model with only x_date as a predictor.
Sample Size. How many observations will be collected or what will determine sample size?
The total sample size for this RCT is approximately 70,000 with 15,000 employees in the email conditions and 55,000 in the passive control condition.
Data Exclusion
Data of employees who leave the OPS during the trial period will be excluded from the analyses. Similarly, data of new employees joining the OPS during the trial period will be excluded.
Who is behind the project?
Project status:
Pre-registration
Methods
What is the project about?
Date published:
21 July 2022