This is a joint post by Ruth Dixon and Jonathan Jones about our Commentary entitled ‘Conspiracist Ideation as a Predictor of Climate Science Rejection: An Alternative Analysis.’. [The link is now to the version of record, published in May 2015].
After nearly a year, two journals, and four rounds of review, our Commentary on two studies by Stephan Lewandowsky was published in Psychological Science on 26 March 2015. This post describes our findings in more detail than the tight word-limit in Psychological Science allowed.
In two papers published in 2013, Stephan Lewandowsky and his colleagues Gilles Gignac and Klaus Oberauer suggested that ‘conspiracist ideation’ (the tendency to believe in conspiracy theories) predicted scepticism about anthropogenic climate change. In our reanalyses of the data from both studies, we found that there was a curved relationship between these variables. Both climate-change sceptics and the ‘climate-convinced’ tended to disbelieve in conspiracy theories. The linear models used by Lewandowsky and colleagues were therefore not appropriate descriptions of the data. Both datasets show this effect, although they resulted from very different survey types (the first surveyed readers of ‘climate blogs’ (LOG13-blogs, published in Psychological Science) and the second surveyed a panel representative of the US population (LGO13-panel, published in PLoS)), so we are confident that our findings are robust.
As we describe in more detail later in this post, our main finding was that there is a curved relationship between belief in anthropogenic climate change (CLIM) and belief in conspiracy theories (CY). This curvilinear relationship is most clearly seen in the LGO13-panel dataset (Figure 1).
As we argue below, all this really shows is that people who are undecided about one fairly technical matter (conspiracy theories) also have no firm opinion about another (climate change). The complex statistical models used by Lewandowsky et al. mask this rather obvious and uninteresting finding.
A note on the publication of our Commentary
Although Lewandowsky and colleagues were sent our Commentary to review in late 2014, Eric Eich, the editor of Psychological Science, did not agree with their opinion that our paper should be rejected. Their (non-anonymous) review made much the same points as their Reply which is published alongside our Commentary.
When accepting our paper, Eric Eich placed an embargo on all discussion until the Commentary and Reply were published.
Lewandowsky and his colleagues used Structural Equation Modelling (SEM) to analyse the datasets. SEM can be a powerful technique which allows complex relationships to be teased out, but it has two disadvantages compared with the Exploratory Data Analysis approach (EDA) that we used. First, SEM requires an underlying theoretical model to describe the presumed relationship between variables, and then estimates the parameters for this model, which renders SEM studies prone to confirmation bias. By contrast EDA allows the data to speak for itself, with the minimum of presuppositions. Second, SEM results are usually presented as diagrams that schematically describe the theoretical model and list the parameters, giving artificial prominence to detected relationships which are not substantial or important. The simpler approach of EDA clearly reveals the strength and importance of any relationship to visual inspection.
The rest of this post describes our findings in more detail.
The LGO13-panel data
LGO13-panel surveyed US citizens drawn from an internet panel. Composite variables for beliefs in climate science (CLIM) and conspiracy theories (CY) were calculated by simple averaging of the relevant question scores on a 5-point scale from ‘strongly disagree’ (1) to ‘strongly agree’ (5).
In Figure 2, we summarise the relationships between the CY and CLIM variables in both datasets using loess local regression, which seeks to find a reasonably smooth curve which summarises any substantial structure in the data. Lewandowsky and colleagues’ choice of using CY to predict CLIM arises from a theoretical model in which conspiracist ideation is supposed to cause climate scepticism, although their experimental design does not test the direction of causality. We investigated both directions to avoid any presuppositions, and show the plot which reveals the structure most clearly. This reversal of direction is not important for straight line fits, but is very important when using loess.
These plots use ‘jittering’, which moves each data point by a small random amount, to give some idea of the number of repeated occurrences of each data point, and the grey bands show the plausible region in which the best fit lines could lie.
It is immediately obvious from the scatter plots that there is no strong relationship between CY and CLIM, with points fairly evenly scattered over the graph. The best fit straight lines (see Supplemental Table and Figures (pdf) Figure S1) show that CLIM and CY are weakly anti-correlated, that is, agreement with the IPCC consensus view on climate change falls slightly as belief in conspiracy theories rises, as reported by Lewandowsky and colleagues. This relationship is, however, extremely weak, being barely distinguishable from a flat line: a statistical analysis of this straight line fit shows that CY predicts less than 1% of the variation in CLIM.
The loess plots in Figure 2 reveal the underlying structure of the data. Using CY to predict CLIM (left-hand plot) still shows an essentially flat line, but using CLIM to predict CY (right-hand plot) gives a clear bell shaped curve. The highest values of CY (the greatest ‘endorsement’ of conspiracy theories) occur not for climate sceptics (low values of CLIM), but for people at the middle of the CLIM spectrum, those with no strong opinion either way.
The LOG13-blogs data
The LOG13-blogs survey invitation appeared on climate-related blogs in 2010; responses were on a 4-point scale from ‘strongly disagree’ (1) to ‘strongly agree’ (4). The manner in which this data was collected has been strongly criticised by many commentators, but we do not address that point here, beyond noting that the respondents to this survey were far more strongly convinced of the IPCC consensus position on climate change than the more representative panel survey. By far the largest group of respondents (41%) expressed maximum agreement with all five propositions relating to climate change (CLIM=4.0), while only 15% were ‘more sceptical than not’ having CLIM in the range 1.0–2.4. The stark contrast of the LOG13-blogs and LGO13-panel datasets is shown in Figure 3.
This extreme skew in the dataset makes analysis more challenging, but loess fits to scatter plots are still useful. Figure 4 indicates that the LOG13-blogs respondents with high values of CLIM gave lower credence to conspiracy theories than those at moderate values of CLIM. The behaviour at low values of CLIM (climate sceptics) is much less clear, but there is no sign that the line continues upwards as CLIM decreases, as implied by the conclusion of the LOG13-blogs paper.
The clarity of this plot can be greatly increased by excluding the 18 data points from the blogs dataset which were identified as ‘outliers’ by Lewandowsky and his colleagues. These points correspond to particularly high beliefs in conspiracy theories (CY>2.5, across the whole range of CLIM). Excluding data points from analyses must be done with care, but Lewandowsky and colleagues specifically noted in their paper that their analysis was robust to the removal of these 18 data points (less than 2% of the whole). The effect on the loess fit, however, is substantial, as shown in Figure 5.
The LOG13-blogs loess plot now shows a clear fall in CY at low values of CLIM, in agreement with the LGO13-panel data.
What does it mean?
In their reply (LGO15), Lewandowsky et al. are highly critical of the lack of ‘theory’ in our commentary, saying ‘Alternative models should reflect alternative theoretically motivated hypotheses, any mention of which is conspicuously lacking in in Dixon and Jones’s Commentary’ and ‘The reader is left in the dark as to what any of this means, which is ironic in light of Dixon and Jones’s admonition against use of SEM as a “black box”.’
Lewandowsky and colleagues argue that since any correlation matrix can be fit by more than one model the greatest consideration should be given to ‘theoretically motivated’ models. This appears to us to be a recipe for confirmation bias. Any model can be used to justify a pre-existing prejudice if it happens to meet (in this case minimal) statistical tests.
If the authors of LGO15 want to know ‘what does it mean?’ they should look at the data. Our graphs show clearly that the relationship between CY (the average of ‘agreement with conspiracy theories’) and CLIM (the average of ‘belief in anthropogenic climate change’) is bell-shaped. That is, both ‘climate sceptics’ (low values of CLIM) and those convinced of the dangers of anthropogenic climate change (high values of CLIM) were less likely than the ‘climate-neutral’ to agree with conspiracy theories. A very straightforward explanation is that people who have firm opinions (and are willing to express them) are sceptical of (most) conspiracy theories and have polarised opinions on climate change. People who are less sure (or less willing to express an opinion) are ‘neutral’ about both CLIM and CY. Such people tend to occupy the middle of the range on CLIM, but being ‘neutral’ about conspiracy theories places them near the top of the range of responses for CY.
All the data really shows is that people who have no opinion about one fairly technical matter (conspiracy theories) also have no opinion about another fairly technical matter (climate change). Complex models mask this obvious (and trivial) finding.
Our Commentary (open access)
Our Supplemental Table and Figures (open access). This document lists the climate and conspiracy questions included in CLIM and CY, and gives brief methodological details of Lewandowsky’s two surveys. It also contains some additional graphs (our Commentary was restricted to one figure with 2 panels).
We will address other points made in Lewandowsky’s Reply in future posts