In their Reply (LGO15) Lewandowsky et al. criticise us for checking the sensitivity of the fit to the removal of 35 respondents to the LGO13-panel survey who responded ‘neutral’ on all CY and CLIM items. Without those respondents (~3% of the dataset), we showed that the linear fit was not statistically significant, highlighting the marginal nature of the CY-CLIM correlation. Our main analysis, however, included all the responses.
In testing the effect of removing the 35 respondents who selected option ‘3’ (‘neutral’) on all of the 6 CY and 5 CLIM questions (of which 23 chose ‘3’ for every one of the 39 questions in the survey) we were simply doing the housekeeping step that the authors should have done before analysing the data (just as they should have removed the apparently 32,757-year-old respondent before running correlations with age). Respondents who are not fully engaged in a survey or who know little about the topic tend to minimise the effort they put into completing the survey by selecting non-responsive answers (this effect is known as satisficing (Krosnick 1991)). Satisficing can be a problem among panel participants, such as those in the LGO13-panel who ‘were compensated by Qualtrics with cash-equivalent points’ (p.9) and so had an incentive to participate even if they had little interest in the topics surveyed. Such (non-)responses should be (i) discouraged by the survey design, and (ii) if possible, removed from the dataset before analysis.
If respondents use the mid-point on the 5-point Likert scale from ‘strongly agree’ to ‘strongly disagree’ to signify an opinion that lies outside the scale altogether (such as ‘don’t know’ or ‘don’t care’) including such responses will distort the dataset (Kulas et al. 2008). In the LGO13-panel survey, selecting ‘neutral’ for each conspiracy theory put the respondent at the high end of conspiracy endorsement. Selecting ‘neutral’ for CLIM questions was less noticeable as this was near to the average response (but can still be seen as a spike at CLIM=3).
One way of minimising such behaviour is to include questions to test the respondents’ engagement with the survey. Although the LGO13-panel survey included an ‘attention-filter’ question, that question instructed the respondent to select option ‘3’ (‘neutral’), the option that would be likely to be ‘picked’ by satisficers. That choice of attention-filter displays a staggering lack of insight on the part of LGO into the type of behaviour that such filters aim to prevent.
Krosnick, J. A. (1991), Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys. Applied Cognitive Psychology, 5, 213–236.
Kulas, J. T., Stachowski, A. A., and Haynes, B. A. (2008), “Middle Response Functioning in Likert-Responses to Personality Items,” Journal of Business and Psychology, 22(3), 251–259.