Ship of Fools: Polling data based on lies

Thread: Polling data based on lies Board: Oblivion / Ship of Fools.

To visit this thread, use this URL:
http://forum.ship-of-fools.com/cgi-bin/ultimatebb.cgi?ubb=get_topic;f=70;t=027772

Posted by deano (# 12063) on :

Following on from the recent threads on the Scottish referendum and the fall out from it, here is an interesting article on the BBC's Magazine site...

Small Data: Do people lie in surveys?

It reinforces my view that polls are pretty worthless in extracting the truth. Nobody wants to admit voting for someone who is perceived as unpopular, but when in the voting booth, with nobody to see them, they put their cross against the unpopular option.

There are few people who admit to voting for Mrs Thatcher in the 80's (I did), but she won three terms in office so somebody (other than me) must have done.

Again, nobody wanted to admit to voting for John Major in the early 90's, but he won.

The polls for the Scottish referendum showed the result should have been much closer than it was.

Would shipmates lie about this on polls? Given the left is perceived as "nice", and therefore popular, I suspect most shippies will say they don't lie, and they do actually vote the left-wing ticket.

But does the above article shake your confidence that the electorate is apt to be swayed towards actually voting left, as opposed to saying they will to appear "nice" but not doing so in the privacy of the voting booth?

The sort of thing highlighted in the article is why we on the right don't take much notice of polls. Better to have the ability that Mrs Thatcher had of being able to "sniff the wind" and instinctively know what the middle-classes want.

Posted by LeRoc (# 3216) on :

quote:
deano: Following on from the recent threads on the Scottish referendum and the fall out from it, here is an interesting article on the BBC's Magazine site...

IIRC, the polls predicted a small 'No' victory. What we got was ... a small 'No' victory. I'd say they were pretty much spot on.

quote:
deano: It reinforces my view that polls are pretty worthless in extracting the truth. Nobody wants to admit voting for someone who is perceived as unpopular, but when in the voting booth, with nobody to see them, they put their cross against the unpopular option.

Good polls normally correct for this.

quote:
deano: There are few people who admit to voting for Mrs Thatcher in the 80's (I did), but she won three terms in office so somebody (other than me) must have done.

I'm sure that polls in the eighties predicted a Thatcher victory. Which is exactly what happened.

Posted by Sioni Sais (# 5713) on :

I don't think it has much if anything to do with lying, but that more people are willing to give an opinion to someone who comes to them, whether in the street, at the door or over the phone, than will leave home to vote or even send off a postal vote. They may also feel that a quick word to a pollster doesn't matter very much.

Most political parties know that they lose elections because of erstwhile supporters who don't vote, rather than those who switch to 'the other lot'.

Posted by M. (# 3291) on :

OF COURSE I lie in opinion polls, surveys* etc. Doesn't everybody? I suppose I'd sort of assumed, if I thought anything, that people's lies cancel each other out.

M.

*though I don't do many face-to-face surveys these days, as I always end up rowing with the surveyor.

Posted by Alan Cresswell (# 31) on :

Polling organisations also have to adjust for selection bias, the people who on the phone or the street refuse to answer the questions. Plus the awkward gits who find good reasons not to be able to answer the questions and argue with pollster. "What do you mean? I want to say I'm concerned about immigration because current regulations are too restrictive and you don't have a box to tick for that?" I might be like M.

Posted by deano (# 12063) on :

Some of you say pollsters correct for the bias. Here is a direct quite form the article...

quote:
Anyway, dishonest respondents are a serious problem for pollsters, especially in a situation such as an election, in which the quality of the sampling will be tested shortly afterwards by the actual result.

There is almost nothing that polling organisations can do about this. Online polling may give some weight to how people voted in previous elections, although they may also be lying or misremembering about that.

So it doesn't look like it can be solved easily.

Also the assumption in the article is that the bias is toward the left, which means it won't self-correct by an equal amount of false answers cancelling each other out.

Posted by LeRoc (# 3216) on :

quote:
deano: So it doesn't look like it can be solved easily.

Of course it can. Pollsters need only to look at the polls for previous elections and compare them with the actual results, not at what people remember about them.

If the effect you describe has led to a 3% bias to the left in the past, then all pollsters need to do is correct their predictions 3% to the right. Which is what they do.

Posted by Enoch (# 14322) on :

quote:
Originally posted by deano:
... So it doesn't look like it can be solved easily. ...

Yes there is. Ban polls, either all the time or within X weeks of an election. I think some countries do that.

As for the more ethical question, to what extent do we think a person is under a moral obligation to answer a pollster truthfully?

I don't think I've ever been polled - which says something for the degree of credibility I attribute to them. I can only remember one person ever saying they had been. So I don't know what my reaction would be. I suspect I would politely say 'it's none of your business'. But the pollster has butted into the victim's life, uninvited, with a question the victim might regard as personal and intrusive. I don't think I have it in my heart to condemn someone who lies in that situation.

Posted by Sioni Sais (# 5713) on :

deano:

Apart from citing one election, the 1992 one in which John Major's Conservative Party defeated Neil Kinnock's Labour Party, where do you have evidence that there is an assumed bias to the Left? There's no mention of Mrs Thatcher in the article for a start and the ORB survey does not ask about political allegience either.

My feeling is that the winning side picks up more of the late undecided votes than does the losers. A glance through wikipedia and elsewhere suggests tht while the core voters are quite willing to trumpet their allegience, those who are on the edge and a little unsure hedge their bets and are less demonstrative.

[ 22. September 2014, 11:33: Message edited by: Sioni Sais ]

Posted by Barnabas62 (# 9110) on :

I guess the real issue is methodological. Surveys by questionnaire require well designed questions and properly controlled sampling techniques. Their interpretation requires a pretty high level of "smart".

This guy is smart. His 538 blog became essential reading in the 2012 Presidential Election. Not because he was a pollster, but because he developed an effective interpretative method of making sense of the wide and prolific range of polling information available. [He did a good job, for example, in demonstrating the "out of line" nature of Gallop polls and providing possible methodological reasons for this.]

I guess we may find it difficult to accept that samples based on a very small percentage of a population may actually be rather good at forecasting outcomes. It seems counter-intuitive that this should be so.

But the truth appears to be that with good sampling and questionnaire methods designed to check on consistency, sample polls are pretty good at doing this. The Scottish Referendum poll of polls was predicting about 53/47 on the eve of the referendum. The actual result, 55/45, looks to be within the normal range of sampling error. In this case, the polls got the result right, but underestimated the margin.

There is a "snake oil" element to all of this of course. A famous Yes Prime Minister episode pointed out that it was possible to design surveys to produce any result you wanted to achieve, simply by rigging the way the questions were asked. Also, I think professional pollsters do try to evaluate the lie and the secrecy factor, try to identify the extent to which it might bias results and make allowances for that.

A healthy scepticism, coupled with a critical approach to both methods and claims based on sample data seems pretty wise to me. But sample polling is here to stay, so it's a good idea to learn how to weigh the results critically, rather than just dismiss the whole enterprise out of hand as all "snake oil".

Posted by Schroedinger's cat (# 64) on :

If you are doing polling on questionnaires for academic purposes, then it is crucial to take into account the possibility of people lying, misremembering, exaggerating etc. There are a range of techniques for this, especially when the sample size if large, however these do require time and effort, and these are why most academic results give "indications" and often discuss the "significance" of results - just because a survey shows a bias towards one answer does not mean that this is the true answer - the result may not be statistically significant.

The problem is that political pollsters do not have the time to do this sort of analysis. their questions are much simpler, and they do, in some cases, take into account the significance of results, but this level of analysis is not possible when the results are needed for the late night news the same day as the poll is taken.

Do I lie? I try not to. If I was not going to give accurate information, I would refuse to do the poll. However often, a poll will polarise my position more than is accurate. I may say, for example, that I will vote Labour, if the choices are Labour or Conservative. However, that takes no account of the difference between a dyed-in-the-wool Labour supporter who would only not vote for them if they proposed wholesale murder of women and children, and the person who is just edging towards them, and may be switched by the smallest local thing.

The other problem is that they are often reported in a far more black-and-white way than is justified. They might show a "small victory" for one party, that is not statistically justified. But the news reporters don't like having to say "we have done surveys, but they have told us nothing significant".

Posted by Horseman Bree (# 5290) on :

There's an element of uncertainty that has been introduced by the use of cellphones. A lot of people don't have landlines, preferring to use cellphones (or Skype and its equivalents) and therefore don't get direct calls. In North America, cellphone numbers are not listed, and there are "Do Not Call" registries, which are necessary because the phone user gets billed for airtime and doesn't want unwanted calls.

Also, even if they do get the calls, their location is not known to the caller, so they can't be included in analysis by seat/town/whatever. This all means that the uncertainty rate is higher than it used to be, and significant swaths, particularly of younger respondents, are simply not available.

Calls of "lies" are almost always from people who don't like the result, rarely from people who actually have any evidence.

[ 22. September 2014, 12:10: Message edited by: Horseman Bree ]

Posted by tclune (# 7959) on :

quote:
Originally posted by LeRoc:
IIRC, the polls predicted a small 'No' victory. What we got was ... a small 'No' victory. I'd say they were pretty much spot on.

[Tangent}In the US, a 10% margin of victory is considered a landslide. Is that only here? Are other countries generally more uniform in their views than we are?[/Tangent]

--Tom Clune

Posted by Boogie (# 13538) on :

quote:
Originally posted by M.:
OF COURSE I lie in opinion polls, surveys* etc. Doesn't everybody?

So do I - especially to canvassers. I say 'of course I will vote for you/your party' to all of them. It gets rid of them far quicker than arguing.

Posted by LeRoc (# 3216) on :

quote:
Horseman Bree: There's an element of uncertainty that has been introduced by the use of cellphones.

Normally, pollsters correct for that too.

quote:
tclune: In the US, a 10% margin of victory is considered a landslide. Is that only here?

Yes, that seems to be a US thing. It's quite remarkable actually, how close the two parties often are.

Posted by Enoch (# 14322) on :

In the UK a 10% difference may not be a landslide but is significant enough to be decisive, not a small margin or a close run thing.

With a referendum, it's a simple headcount. With a general election, it's what translates into elected members. The first past the post system distorts that heavily. Depending on where the votes are and for whom, a small overall difference in headcount can translate into anything from a landslide to no overall control. In 1951, the Conservatives evicted Labour from government and got a small working majority despite getting a headcount that was c200,000 votes less than Labour.

Posted by Heavenly Anarchist (# 13313) on :

Online polls are generally taken from a panel of signed up members so that weighting is easier. This has it's own issues of bias but does mean that polling companies have a record of how you have previously answered questions and I'm sure they find the 'misremembering' fascinating. I'm on several online polling panels, the payment is very small for this type of panel (around 50p each one as opposed to a detailed consumer research company which pays £3 each) but I'm guessing a lot of correspondents are genuinely interested in polls and that helps with honestly. Interest in politics is what motivated me to join the panels.
The most exasperating question for me is the YouGov one asking me which political party I will vote for in the next general election. I answer according to the party the person I will vote for is in but I do not have an allegiance to his party so the question is inaccurate for me.

Posted by North East Quine (# 13049) on :

I'm signed up to YouGov too. I do try to answer questions honestly, and I do always answer questions about voting intentions scrupulously.

The recent YouGov ones had a flaw regarding voting history as they asked "which party". We can vote for more than one party for Holyrood, and as I do split my vote, I have a choice of truthful answers. [Biased]

Posted by Beeswax Altar (# 11644) on :

Political opinion polls should be outlawed. I'm convinced politicians would run on what they truly believed and voters would vote for the candidate they truly preferred. Political coverage is too much about the horse race.

Posted by Barnabas62 (# 9110) on :

I've got some sympathy for the view that published polling may have some influence on results. The "bandwagon effect" is real, but I guess that the "uncertains" who jump on bandwagons need evidence to make them jump.

"Bandwagon effects" seem to be based more on what you might call "instant appeal" factors, rather than serious policy considerations.

But in a free society, that freedom includes the right to make a vote choice for non-serious or ephemeral reasons. My mum and dad nearly always voted differently. He was staunch Labour, would have voted Labour if the Conservative candidate was the Archangel Gabriel and the Labour Candidate was a donkey. He was a loyalist. My mum did different as an exercise of personal freedom. She was a classic nonconformist! Sure, they would argue politically, but in the end I think what influenced their votes most didn't have a lot to do with policies. I don't think they are that unusual in that.

Virtually all politics in the democratic West these days is strongly influenced by the concept of the floating voter. How are they to be influenced. And polling is the means of checking, dynamically, whether the electoral strategies in play are having any effect, and if so how much. In long election campaigns, like the Scottish referendum and the US Presidential campaigns, there is time to adapt the strategies of appeal.

My guess is that even if, by some (pretty unconstitutional) process, the publication of polls results was outlawed during election campaigns, the parties would still find ways of measuring bandwagon effects. Only we wouldn't get to know the results. And there may still be claims made that "the tide is turning in our favour".

So I don't think in practice there is much to be done to change the political desire to measure what's going on. Once that is accepted, the outlawing of polls looks more like suppression of information which a free electorate is entitled to know and make use of, if it suits particular electors' books.

Posted by Heavenly Anarchist (# 13313) on :

One of my favourite bits of UK polling trivia is on referendums. YouGov's Anthony Wells (owner of UK Polling Report) says that if you poll the British public about whether they want a referendum on any subject they will always want one. Any subject, regardless of constitutional importance. Except for whether the monarchy should be abolished, they definitely don't want a referendum on that.

Posted by Beeswax Altar (# 11644) on :

quote:
"Bandwagon effects" seem to be based more on what you might call "instant appeal" factors, rather than serious policy considerations.

But in a free society, that freedom includes the right to make a vote choice for non-serious or ephemeral reasons.

Which is why I favor amending the constitution to provide for election of the president via reality television show entitled the American President.

Other than meeting the constitutional requirements for eligibility, interested candidates must collect a certain number of signatures in each state and pay a fee. Then, let the game begin! Each candidate will get the opportunity to make a stump speech on television. Then, the voters vote and one or more candidates are eliminated. After that, the speeches will be over issues determined by the voters. Candidates give speech. Voters vote. Another campaign is suspended. Eventually, after the issues are addressed, the voters will have eliminated enough of the candidates to make a debate possible. Moderators and formats will be determined by the polls. The candidates debate. The voters vote. Another campaign is suspended. Eventually, only four will remain. Those four pick running mates. Running mates give speeches. Voters vote. Another campaign is suspended. You get the idea.

Given the US love of reality shows, voter participation should be very high. Obviously, some games will have to be introduced to mix things up. Identifying foreign leaders with the country they lead might be interesting. Maybe a presidential simulator of some sort could be invented.

Posted by Barnabas62 (# 9110) on :

Thanks BA; that made this Host smile!

(As did Marlee Matlin's rather good playing of the deaf pollster Joey Lucas in "The West Wing", which I've been re-watching recently).

A strong sense of the absurd is very helpful when discussing politics.

UBB.classic^TM 6.5.0