Who was the biggest loser in the UK General Election?

If resignations are recriminations are anything to go by, it was neck and neck between the Lib Dems, Labour and UKIP. But in terms of credibility, the biggest losers were the opinion pollsters. Unanimous in their prediction of a hung parliament, they all missed the true outcome by a mile. The only forecast that was remotely on target was the virtually clean sweep of Scotland by the SNP.

The fallout from failed predictions is enormous. When results are so far adrift of expectations, heads roll, careers are shattered and history changes course. It is frustrating - but weirdly heart-warming - that despite all the weapons of Big Data, advanced analytics and sophisticated modelling, we still can’t penetrate the mysteries of human behaviour.

How come so many experts got it wrong?

There were lots of floating voters, but it is unlikely that they all jumped in the same direction at the last minute. Some pollsters talk about the ‘reticent right’: big-C Conservatives who are backward in coming forward with their opinions. This might explain the late lurch to the right in both the UK and Israeli elections: http://www.ibtimes.co.uk/sack-pollsters-how-did-experts-get-it-so-wrong-britain-israel-1500267

There are also flaws in the methodology of opinion polls. When canvassed for their opinion, people often say what they think the questioner wants to hear, or what they believe someone in their position ought to say.

Often, polling errors will cancel each other out. The huge disparity in the General Election suggests that a systematic bias was at play. One possible source of bias is the conservative (small c) instinct. People may demand change, but when faced with something new and unfamiliar they often revert to what they know. For example, they may oppose cuts, but accept austerity as the price of economic stability. This instinct may also explain why polls before the Scottish Independence vote suggested the result would be close, but when push came to shove – faced with a leap into the unknown – people clung to the status quo.

How does that explain the shift to the SNP in Scotland during the General Election? With independence off the menu, perhaps voters felt they could be radical without taking such a big risk.

Voting in a General Election ought to be the purest form of decision making: a single moment in time, clearly defined alternatives, and one choice. The fact that we can’t predict the outcome even when voters tell us their intentions shows how little we understand human motivation.

Why is there such a difference between what people say and what they do? Well for a start, a normal English conversation has at least 6 layers of meaning. Here’s an example:

 As this shows, there may be little correspondence between the different layers of communication. And even if you know exactly what another person is thinking, you don’t know what they are going to do, because usually they don’t either. It’s no wonder that a simple question about voting intentions can yield inaccurate results.

In healthcare marketing, we are accustomed to the shortcomings of market research. If all doctors prescribed new medicines with the enthusiasm they display in detail follow up (DFU) testing, there would be no need to market new drugs.

So we take research findings with an unhealthy pinch of salt, and like politicians, we don’t count our chickens unless we have a sizeable majority. If market research delivers a recommendation that seems counterintuitive, we don’t hesitate to challenge it.

Today, smart agencies don’t just think about what to communicate to customers. They focus on closing the gap between belief and behaviour. Because what people say and the decisions they take may be polls apart.