How can response rates to questionnaires be increased? In the first quarter of 2023, some of the participants in a CBS survey received a letter in which the call-to-action appeared as far as possible at the top of the letter. The other participants received standard letters. The analyses showed that the standard letters produced a significantly higher response rate than the letters in which the call-to-action was placed at the top.

Why this experiment was conducted: high response rate is preferable
Statistics Netherlands (CBS) collects information for its statistics partly by sending questionnaires to a section of the population of the Netherlands. These questionnaires can be about multiple topics, such as work situation, health or living environment. A sample of the population will receive a letter asking them to take part in the survey. For results to be as representative as possible, it is important that as many Dutch citizens as possible actually complete the questionnaire. CBS conducted an experiment to test whether the order of certain paragraphs in letters has an effect on respondent behaviour.

Type of intervention: call-to-action moved to the top of the letter
Experiments with invitation and reminder letters were carried out in the Dutch Labour Force Survey (LFS). The purpose of this survey, which is conducted by CBS, is to examine labour market developments in the Netherlands. Information is sought on a variety of topics, including the number of jobseekers, the reasons why individuals opt to work less or the proportion of young people holding down a part-time job alongside studying. Two types of invitation and reminder letters were sent:
• Standard letters: these started with two paragraphs about CBS and the specific research being conducted. This was followed by a paragraph on the incentives that could be won, such as an iPad or a VVV gif card. Next, there was a paragraph explaining how people could take part in the survey, with website and login details.
• Experimental letters: here, the paragraph on how to take part was moved right to the top of the letter. Specific information on the research being conducted appeared later in the letter.

Method used: field experiment
Between January and March 2023, 10% of the sample group received the experimental letter (N=5,031) and 90% received the standard letter (N=45,292). Via the website on which individuals filled in the questionnaire, CBS was recording for each invited person whether they opened the questionnaire (login), whether they broke of filling in the questionnaire before completing it (survey dropout), or whether they completed the entire questionnaire (response).

Result obtained: standard letter position of call-to-action is most effective
The standard letters produce a significantly higher response rate (37.6% of recipients) than the letters where the call-to-action is positioned earlier in the text (36.1%). This is a significant difference. For a sample size of roughly 50,000 people, a 1.5% increased response rate results in 750 more completed questionnaires. It was also found that in the experimental group, fewer people made a login attempt compared to the control group (39.6% vs. 40.7%) and that they are more likely to drop out (i.e. discontinue before reaching the end) once logged in (3.5% vs. 3.0%). However, these differences are not significant. In certain age groups, the difference in response rates between the letters is relatively large. In the age group up to 40 years old, 33.2% of those invited to take part in the survey completed the entire questionnaire when given the standard letter, compared to 30.7% of respondents shown the experimental letter − a significant difference. By contrast, the age group over 65 years old seems to respond better to the experimental letter (53.9% vs. 52.4% for the standard letter), although this difference was not found to be significant.

Impact: improved statistics
A more effective invitation letter leads to significantly more questionnaire responses, and hence more reliable results. This enables CBS to enhance the accuracy of its statistical overview of the Dutch population. Moreover, this intervention is low-cost: one letter is no more expensive than the other.

Source: https://www.binnl.nl/home+-+en/knowledge/publications/bin+nl+publications/HandlerDownloadFiles.ashx?idnv=2719979