Response to Lewandowsky, Part II: Satisficing

In their Reply (LGO15) Lewandowsky et al. criticise us for checking the sensitivity of the fit to the removal of 35 respondents to the LGO13-panel survey who responded ‘neutral’ on all CY and CLIM items. Without those respondents (~3% of the dataset), we showed that the linear fit was not statistically significant, highlighting the marginal nature of the CY-CLIM correlation. Our main analysis, however, included all the responses.

In testing the effect of removing the 35 respondents who selected option ‘3’ (‘neutral’) on all of the 6 CY and 5 CLIM questions (of which 23 chose ‘3’ for every one of the 39 questions in the survey) we were simply doing the housekeeping step that the authors should have done before analysing the data (just as they should have removed the apparently 32,757-year-old respondent before running correlations with age). Respondents who are not fully engaged in a survey or who know little about the topic tend to minimise the effort they put into completing the survey by selecting non-responsive answers (this effect is known as satisficing (Krosnick 1991)). Satisficing can be a problem among panel participants, such as those in the LGO13-panel who ‘were compensated by Qualtrics with cash-equivalent points’ (p.9) and so had an incentive to participate even if they had little interest in the topics surveyed. Such (non-)responses should be (i) discouraged by the survey design, and (ii) if possible, removed from the dataset before analysis.

If respondents use the mid-point on the 5-point Likert scale from ‘strongly agree’ to ‘strongly disagree’ to signify an opinion that lies outside the scale altogether (such as ‘don’t know’ or ‘don’t care’) including such responses will distort the dataset (Kulas et al. 2008). In the LGO13-panel survey, selecting ‘neutral’ for each conspiracy theory put the respondent at the high end of conspiracy endorsement. Selecting ‘neutral’ for CLIM questions was less noticeable as this was near to the average response (but can still be seen as a spike at CLIM=3).

Figure II-1. Number of Responses at each Level of CY and CLIM. Light grey bar represents respondents who had chosen ‘neutral’ for all 11 questions relating to CLIM and CY

Figure II-1. Number of Responses at each Level of CY and CLIM. Light grey bar represents respondents who had chosen ‘neutral’ for all 11 questions relating to CLIM and CY

One way of minimising such behaviour is to include questions to test the respondents’ engagement with the survey. Although the LGO13-panel survey included an ‘attention-filter’ question, that question instructed the respondent to select option ‘3’ (‘neutral’), the option that would be likely to be ‘picked’ by satisficers. That choice of attention-filter displays a staggering lack of insight on the part of LGO into the type of behaviour that such filters aim to prevent.

Krosnick, J. A. (1991), Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys. Applied Cognitive Psychology, 5, 213–236.

Kulas, J. T., Stachowski, A. A., and Haynes, B. A. (2008), “Middle Response Functioning in Likert-Responses to Personality Items,” Journal of Business and Psychology, 22(3), 251–259.

1 thought on “Response to Lewandowsky, Part II: Satisficing

  1. I worked on designing surveys for 30 years. Eliminating respondents who appeared not to read the survey questions is an essential part in ensuring that your data is meaningful. This is one of the reasons why I despise Likert type questions. We developed a technique for measuring engagement by posing open ended questions prior to each set of Likert type items. The effect was to increase the variance in the responses and made it clear which respondents were simply going through the motions. We also had a program that calculated the runs of the same answers to questions, i.e. a run of 10 5s or 10 1s. Both approaches were very important when we did voluntary surveys with a reward for survey completion. For what it is worth, I fully support your efforts to quality control respondents. It is a pity Lewandowsky does not understand the basics of psychometrics.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s