1- Whether particular modes of presenting evidence (Impact Calculator and Side-by-Side) can improve sensitivities
2- How does sensitivity and the impact of our treatments depend on the particular impact-relevant feature? For instance, are people differentially sensitive to a change in scope compared to a change in the outcome?
3- Is higher certainty correlated with increased sensitivity to impact?
4- Do the impact calculator and/or side-by-side presentations increase certainty in responses?
5- Does measured sensitivity look comparable across incentivized and unincentivized responses?
6- Is more experience with evidence and evaluation correlated with both increased sensitivity as well as higher certainty?
7- Is the time spent on the program valuation screens predictive of increased sensitivity?

The survey will be broken down into three sections, which each respondent sees in random order:
1. Baseline - 2 valuations, 2 unique programs: Document baseline insensitivities in “valuations”; to what extent are policymakers attending to features like scope or outcome type when assessing a program?
2. Treatment 1 - Impact Calculator - 2 valuations, 2 unique programs: Introduce an “Impact calculator” to increase sensitivity to impact-relevant features of a program. If respondents are presented with a calculation of impact (based only on the program description on the page), does sensitivity increase?
3. Treatment 2 - Side-by-Side - 2 valuations, 1 unique program: Show program descriptions side-by-side rather than sequentially. Does the direct comparison increase sensitivity to impact-relevant features?

After respondents see each baseline question, they will then be asked to assess their certainty in their valuation. Specifically, they will answer the following question: How certain are you in your answer ?
After responding to these six core questions, respondents will again be shown one of the baseline program descriptions. This time, they will be asked to predict the modal cost selected by other respondents. Because this beliefs-based question has a clear true or false answer, we are able to incentivize responses.
The survey will conclude with questions to understand the additional barriers policymakers confront when translating evidence into policies and programs, as well as what this process looks like at different agencies. The survey will also ask for information about respondents’ work in government as well as demographic characteristics.

Each hypothetical program description will include randomly-assigned variations corresponding to different hypothesized impact-relevant features of a program
These features can be multiplied together to estimate the impact of a program. Specifically, multiplying the number of people a program affects, the rate at which a “final” outcome is achieved, and the persistence of effects gives us an estimate for the people affected per year, who wouldn’t achieve the outcome in question in the absence of the program. Each respondent will see one of four combinations of these variations for each program:
1. High scope, Final outcome, Long persistence
2. Low scope, Final outcome, Long persistence
3. High scope, Intermediate outcome, Long persistence
4. High scope, Final outcome, Short persistence

1- The primary outcome for this analysis is the “valuation” or perceived program value, which is defined as the maximum cost at which the respondent is willing to fund the program, as identified in the survey. We log-transform this outcome given that the costs are presented on a semi-logarithmic scale. We then normalize the program valuations around the mean perceived value ascribed to the lowest-impact combination for the program.

The only new outcome measure the questions 2 to 7 introduce is that of certainty when exploring heterogeneities. The baseline certainty measure will be coded as a continuous variable ranging from 0% to 100% certainty (or certain). The certainty measure relevant to treatment effects will be coded as -1 if respondents report less certainty in the face of a treatment, 0 if respondents are equally certain, and 1 if respondents report more certainty.

1- Regression of valuation over the condition, the program's impact and controlling for program and participant fixed effects.
2- Relative sensitivity of the impact features
3- Heterogeneous treatment effects by our three impact features
4- The relationship between certainty and sensitivity to impact
5- The impact of our treatments on certainty
6- The relationship between incentivized and unincentivized responses
7- The relationship between respondent characteristics, notably experience with evidence
and evaluation, and sensitivity as well as certainty
8- The relationship between time spent on the program valuations and sensitivity

We will exclude respondents who didn’t complete the main portion of the survey. We will also only
keep the first complete response received via a single personalized survey link. That is, if the
survey was taken multiple times using the same survey ID, we will not look at additional complete
responses. We will also run robustness checks where we look only at respondents who spent at
least 4 minutes on the survey in total

Treatment of Missing Data:
We will require responses to all core program valuation and certainty questions. Any skipped
end-of-survey questions will be dropped, without imputing the data.