Dear Theos, Please learn to handle survey data

Cross SunsetTheos, a Christian think tank, have been in the news today with a survey on the subject of belief and spirituality. They claim that their report, “The Spirit of Things Unseen: belief in post-religious Britain”, which has been released to promote and coincide with a new podcast (telling you all you need to know about its objectivity), challenges a belief that Britain has become “more secular, or more sceptical, or more rational”. The very first paragraph of the executive summary reads in full:

For all that formalised religious belief and institutionalised religious belonging has declined over recent decades, the British have not become a nation of atheists or materialists. On the contrary, a spiritual current runs as, if not more, powerfully through the nation than it once did

This is where the first gaping chasm between responses and interpretation arises – despite claiming to challenge this idea of a trend, nowhere do they present any baseline data to compare these figures with earlier surveys to discern a direction of travel. Yes, seriously – maybe if anyone suggests my hair might be starting to go grey, I should hold up a couple of non-grey hairs to “disprove” the claim. That would work, right?

You might have thought that it would be a problem for anyone claiming to discern national trends to have no previous figures to hand for the purpose of comparison, but maybe the truth is that these figures would be deeply uncomfortable for Theos and undermine the conclusion they wished to draw. The only time they mention a very rough comparison, it’s to compare the 68% who identified with any religion in the 2011 census with the 60% in this survey. Immediately, you might be wondering where they got this idea of a secular rationalist nation, but worse is to come.

Despite this figure, only 30% of respondents believed in God in the broadest possible terms, as a universal life force – just 44% of these “religious” people accept the most basic element of the belief they identify with. If you narrow it down to a personal God, just 12% agree, 21% of self-identified believers. In other hands, this would clearly be the headline finding from the survey, but Theos’s attention lies elsewhere.

One key aim of the survey is clearly to be able to point to the spiritual beliefs of non-religious respondents, casting doubt on their lack of religiosity. It’s notable that several questions offer a wide range of possible positive responses, which can then be aggregated to create statistics like “X% of non-religious people believe in some form of spiritual being”. I can’t help noticing a similarity with Richard Dawkins’ survey of Census Christians, but the crucial difference here is that the group in question hasn’t identified with any group, merely responding that they don’t belong to a religion.

Dream GraphTheos seem to make the mistake of fallaciously assuming all non-religious respondents to be atheists, expressing surprise any time a significant number admit to anything slightly spiritual, while conveniently neglecting  to apply the equivalent reasoning to the more obvious religious groups. If it’s news that 34% of non-religious people believe in some form of spiritual being (from a pretty long list), what about the 25% of religious people who don’t, the 36% of believers who think spiritual forces have no influence or don’t know whether they do, or the 20% who think humans are no different from any other animal?

The only answer I can give is that they don’t like those statistics. From the first question, it seems that the questions are strongly angled towards specific answers:

Q1. To what extent, if at all, do you agree or disagree that there are things in life we simply cannot explain through science or any other means?

There are two problems with this question. First, it’s not specified what these “other means” are, leaving it open to interpretation. But more importantly, it isn’t clear from the question whether this refers to things we are currently unable to explain, or things that cannot be explained in principle. As stated, it would take a very strong-minded, even dogmatic person to express absolute disagreement, because there are so many unknowns. There’s also a missing step (or several) in Theos’s implied conclusion that agreement indicates anything like belief in supernatural beings.

Because of this, to say that the responses to this question indicate that “not being religious does not necessarily mean being rationalistic or sceptical in your outlook” (As Theos do) is a conclusion that hugely overreaches the data and neglects to even consider the many reasons why people might answer positively while being distinctly rationalist and sceptical, such as an acceptance that our knowledge is necessarily imperfect and caution about making excessive claims based on it.

Another glaring distortion appears in section 2.2 of the report, on spiritual beliefs. Given another long pick list of beliefs, they found that 33% of non-religious respondents identified with at least one. So 67% didn’t, right? But that isn’t good enough for Theos, who decided to compare this gerrymandered 33% with the 25% who were prepared to confidently state that “humans are purely material beings with no spiritual element” (another question that could do with further definition) and draw the conclusion that:

A sense of the spiritual is not only not limited to the religious, but it appears to be a majority position even among the non-religious

No, no and a thousand times no! This is terribly shoddy, cherry-picking data from very different questions in different forms and attempting to compare them as if they were analogous. They’re not. It isn’t. Fail.

Candles 2Despite all this, I think the most egregious example of abusing the survey data comes from Q5, which asks respondents which of various procedures (such as tarot readings, reiki and crystal healing) they had ever participated in. Immediately, looking at that list, I suspected that Christians would score low, as the church tends to frown upon them. This is supported by the results, which show 37% of self-identified Christians undergoing any of these activities at some point, lower than the average.

This huge Christian bloc significantly distorts the figures. In the report, Theos attempt to equate the 38% of non-religious with the 40% of religious respondents who have ever participated in any of the named activities, but as over half the total sample belong to a religious group which positively discourages and opposes such things, this is questionable at best. There are some notable omissions from the list, and there is no control for levels of involvement (a single giggly tarot reading at university scores the same as expensive daily sessions), but the biggest flaw is the most basic – these are past activities.

The report claims to be examining trends, but here it relies on past activities in order to draw conclusions about the present. Who thought that made any sense? To make it even worse, these past activities – which may now be viewed in a very different light by the respondents – are subdivided by group based on current self-identification of religious labels. No doubt they’d think it news that plenty of atheists used to go to church. As a way of determining how particular religious groups behave (especially the fast-growing non-religious group), it’s utterly worthless.

My general response to this survey is not to wonder at the high level of spiritual beliefs among non-religious people (which, as already discussed, is a much bigger and more diverse group than simply atheists), but the low levels among the rapidly dwindling number of believers. Strip away the spin, and some of these figures on the most basic features of religious belief should be greatly troubling to anyone who has an interest in the continued survival of religion.

An interesting project for the future (and the best use of this survey) would be to do what Theos have failed to do – repeat it in ten years so that the results can be directly compared, and see how beliefs have changed. I’m willing to bet that it would show a further decline of religion.

The report can be found here and the survey data here

Images courtesy of svilen001, iamwahid and davidvives, used with permission


Tags: , , , , , , , , , ,

About Recovering Agnostic

I'm Christian by upbringing, agnostic by belief, cynical by temperament, broadly scientific in approach, and looking for answers. My main interest at the moment is in turning my current disengaged shrug into at least a working hypothesis.

3 responses to “Dear Theos, Please learn to handle survey data”

  1. markedfrombirth says :

    Great post! People can spin statistics to fit their thesis so it is always important to look deeper. Especially when statistics are being used to support an idea someone already believes!

Trackbacks / Pingbacks

  1. What Theos Didn’t Report | Recovering Agnostic - October 18, 2013
  2. Animism | - November 1, 2013

Love it? Hate it? Leave a comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: