Should we use polls or magic 8 balls? Part 1 of 2

In a recent broadcast, the delightful Stephen Colbert, host of the Late Show, after lamenting and mourning the outcome of the U.S. presidential election, remarked: “And what’s with the inaccuracy of the polls? How can we possibly trust them when they were so wrong! One might as well use magic 8 balls and Ouija boards to make the predictions!”

Just after the election, we were on the phone with the Chief Content Officer of the American Marketing Association (AMA) in Chicago, who told us that for the association (which comprises of tens of thousands of marketers in the U.S. from brand managers all the way to CMOs), the biggest marketing story of the year was the election. It was his assessment that the Trump campaign simply out-marketed the Clinton team, and that’s hard to argue with. He too noted that the polls were not the correct way to guide strategic decisions in the campaigns, as they were all distorted or just plain wrong.

With the recent Canadian federal election, the polls again painted a picture (3 parties in a dead heat) that was far different from the reality: a decisive victory for Trudeau and the Liberals.

So how is it that the polls got it so wrong? There are several issues, one of which is of course the risk in asking people to answer relatively private or sensitive questions in a non-private manner. Being surrounded by total strangers in a focus group, or filling out a poll where you know someone is going to read your answers – is quite different from the total privacy of a voting booth. All manner of lies and evasions can ensue. But the other problem is a more intractable one, one that, unbeknownst to many, was identified over four decades ago by a young American economist named Kenneth Arrow.

In 1972, Arrow won the Nobel Prize in Economics by making a rather startling discovery in his paper entitled “Social Choice and Individual Values”. In it he states:

“Aggregating individual preference data to ascertain societal preference – forfeits logical consistency.”

To most, this statement may seem opaque or academic, but it’s rather earth shattering. It means that no matter what statistical method you use, you cannot reliably ascertain “societal preference” (which we sometimes call in politics “the will of the people”) by asking a group of individuals to state their preference. Polling also neglects the “silent majority” – a large group of people who don’t express their opinions publicly –, and no matter how many thousands of people you ask, and no matter how you ask them, you cannot avoid these flaws. Whether we’re asking them to tell us which detergents, car brands, shampoos or political candidates they prefer, one cannot accurately determine a collective preference through methods such as polling, because there’s a fundamental logical flaw in the math. Or, to put it bluntly, the machine came broken from the factory.

Arrow called this “The Impossibility Theorem”, and after he won the Nobel Prize, it came to be called “Arrow’s Paradox”.

In next week’s post, we’ll explore what this paradox really means for today’s marketers and advertising strategists, and what are some effective ways to avoid it. After all, we are all in the business of trying to ascertain – and sway – societal preference.