Response to Lewandowsky, Part I: Reversing relationships

In their reply (LGO15) to our Commentary, Lewandowsky et al. (LGO) are highly critical of our decision to investigate the relationship between CY and CLIM in both directions, that is not only using CY to predict CLIM (their preferred direction) but also using CLIM to predict CY. They claim that our analysis ‘reaches its main conclusion only by reversing the role of criterion and predictor without any theoretical justification.’

There are many things wrong with this claim. Superficially it seems attractive: LGO argue that CY represents ‘conspiracist Ideation’, a personality attribute, while CLIM represents ‘acceptance of climate science’, an opinion, and that personality attributes can drive opinions but not vice versa. However these identifications, and the corresponding supposed direction of causation, are simply asserted. In reality CY and CLIM are simply averaged answers to a series of survey questions, and the design of the study does not permit the direction of causality to be inferred. Describing this choice as ‘theoretically motivated’ is just a fancy way of saying ‘in accord with our preconceptions’.

If the relationship between CY and CLIM was indeed linear, as LGO assume, then the direction of association would not be particularly important: while best fit straight lines do depend on the direction of prediction, the correlation coefficient is identical in both directions, and this is also true for structural equation modelling (SEM) as usually implemented. With non-linear relationships, however, one must be more careful. In particular if a relationship is non-monotonic in one direction, so that the same value of y occurs for multiple values of x, then the transposed relationship will be multi-valued, with multiple values of y occurring for the same value of x. Fitting to such multi-valued functions is challenging, and the vast majority of fitting techniques assume that the fitting function is single valued, and so it is vital to investigate non-linear relationships in both directions.

LGO15’s response to this is two-fold.

First they claim that ‘When CLIM is regressed on CY, there is no evidence for a non-linear relationship, … which nullifies the purported statistical justification for the reversal’. This comment is wholly bizarre: if the relationship is described by a bell-shaped curve then the transposed relationship will be essentially a flat line, as is indeed seen. To misinterpret an absence of apparent structure in one direction as evidence against non-linearity in the other direction indicates a complete lack of understanding of the underlying issue.

Secondly they claim that the bell-shaped curve is not in fact present in the data, which they say are ‘simply spread out more at the lower end of the CY scale, which introduces nothing but heteroscedasticity in the regression of CLIM on CY’. But this is simply another way of saying that high values of CY are associated with medium levels of CLIM, while low values of CY are associated with both high and low values of CLIM, which is exactly the point we made in our Commentary. Both descriptions of the data lead to the same conclusion. And heteroscedasticity (when the variance in variable Y depends on variable X) is by no means a trivial problem to overcome. As we show in a separate post, the combination of heteroscedastic noise and skewed sampling will lead to apparent linear trends which have no reality.

Beyond these questions of data-processing, LGO’s claim that CY predicts CLIM is also exposed by the data collection process for LOG13-blogs. If LGO really believe that CY must be the independent variable and CLIM the dependent variable, then their decision to post their survey at climate-related blogs constitutes selection on an endogenous (dependent) variable. If they wished to find out the views of conspiracy theorists, they would have surveyed different types of blogs.

In his classic paper, Richard Berk (1983) demonstrated that non-random selection on an endogenous variable (a variable whose value depends on other variables in a causal model) compromises not only external validity (the ability to draw inferences about a wider population) but also internal validity (the ability to draw inferences even about the sample itself). This arises because the ‘problems caused by nonrandom exclusion of certain observations are manifested in the expected values of the endogenous variable [Y]. When the usual linear form is fit to the data, the expected values of the disturbances [in Y] for each value of X [the exogenous variable] are no longer zero [… and ] the disturbances are then correlated with the exogenous variable’ (Berk 1983). Unless steps are taken to ameliorate this problem, the relationship between the variables will be incorrectly inferred, even for the sample itself.

Finally we consider issues arising from the reporting of these two studies. It is far from clear that the authors have actually worried much (until now) about the direction of the CY-CLIM relationship, as reported in the press and other media coverage. In 2010, before starting the data collection for either study, Lewandowsky wrote a blog post portraying climate sceptics as conspiracy theorists, comparing them to 9/11 ‘truthers’. The original paper LOG13-blogs (p.624) noted the need for ‘empirical evidence about how widespread [conspiracist] ideations are among people who reject scientific evidence, in particular as it relates to climate change’. Both imply that CY is viewed as the dependent variable.

Many press articles focused on the characteristics of climate sceptics (or ‘deniers’) rather than on those of conspiracy theorists. For example: ‘Climate sceptics more likely to be conspiracy theorists and free market advocates, study claims’ (headline, Redfearn, Guardian 2013) and ‘new research finds that sceptics also tend to support conspiracy theories’ (Corner, sub-head, Guardian blog); ‘An Australian study says avid climate change deniers tend to be either extreme free marketeers or conspiracy theorists.’ (Telegraph). This claim was not only made in headlines, but also in the body of the text of a Scientific American blog: ‘climate change denialists also seem to display two other characteristics; a belief in laissez-faire capitalism and more troublingly, a tendency to espouse conspiracy theories’.

Even the Psychological Science ‘Observer’ magazine website published an article by Scott Sleek (American Psychological Society’s director of news) which implies that CLIM predicts CY: ‘Lewandowsky detailed his research suggesting that people who reject climate science also tend to believe in assorted conspiracy theories’.

All of these examples imply that CY is the dependent variable. If the authors were strongly concerned with the direction of causality they should have asked the Guardian, the Telegraph, Scientific American and Psychological Science to correct their misconceptions.

[Update, May 2015] There are examples of similar variables being used in both directions in the academic literature. For instance, a paper was published in Jan 2015 in which belief in conspiracy theories was treated as the dependent variable throughout (this study, interestingly, detected a curved relationship): Political Extremism Predicts Belief in Conspiracy Theories.

And in a book chapter published in 2009, Lewandowsky and colleagues describe a scale that they developed to measure scepticism (in this case towards information from the media and politicians) which they used as a predictor variable, contrary to their treatment of climate scepticism in LOG13 and LGO13. A ‘manuscript in preparation’ about the scepticism scale was cited in the chapter but I can find no sign that it was ever published.

One thought on “Response to Lewandowsky, Part I: Reversing relationships

  1. As an academic layman who barely passed a fully robust statistics course in pursuit of a Poly Sci degree in the late 60’s I find these arguments amazingly similar to prevailing discussions amongst a diverse faculty at Ohio State where the old guard was trying to stem the tide of social activism. Interestingly enough, the notion that Social Change as a raison-de-etra for engaging in social science seems to be an ongoing battle for the human mind. Even after 40 year of absence from that arena as an I/T professional, I find these arguments made above to be almost as compelling as the ones made against the young Turks by a wise old professor whose core knowledge came from surviving a Nazi death camp. It really is all in the details.

    [Ruth: Thank you! Very interesting points.]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s