Category: politics

One good thing: a better electoral forecasting model

So it looks like the local elections didn’t go so badly. While we’re in the intermediate phase between Corbyn’s Labour doing something reasonably well, and them throwing it all away through some sort of terrible cake-and-arse juggle, I’d like to take note of something.

John Curtice reckoned we should expect a net-loss of about 170 seats. Jeremy Corbyn said very publicly that he didn’t believe Labour would lose any seats at all. I’m not sure whether he intended this as a prediction, or just felt he should put a brave face on things and cheer up the troops, but it amounts to predicting a net-loss of zero seats.

One thing pretty much everyone agrees on is that the bad polling had a profound, if subtle, effect on the 2015 election campaign. Right up until the last minute, it was possible to believe – even necessary, on the grounds that you should base beliefs on the best available evidence – that we were going to win.

People who worried about a coalition with the SNP were worried because the polls looked like that might happen. People who worried that Ed Miliband wasn’t keen enough on coalition with the SNP and decided to go Green or not bother voting did so because the polls looked like a Labour/SNP alliance would be necessary. I know at least some of these people existed because they used to shout at me on Twitter. Tories who thought the public was scared of the SNP acted as they did because the polls looked that way. Sizable chunks of the Tories’ policy agenda that just look weird in the post-2015 context only existed because the polls looked like another coalition was the only way they had a hope of getting back in. So much of the ambiguity and chaos of the 2015 election is down to the fact that politics is largely perceived through polling, and the polling was crap.

Blaming the pollsters is futile. For a remarkably policy- and data-oriented Labour team, Ed Miliband’s staff seem to have comprehensively failed to be intelligent consumers of polling. As a result, it wasn’t until final canvass returns from the West Midlands were analysed with two days to go that anyone realised the polls might be wrong, and the result was the Edstone. For those of us who weren’t privy to those data sets, we just had to wait for the epic punch in the guts that was the 10pm exit poll on the night.

If Corbyn meant it about not losing any seats, that forecast was off by 23 net seats out of 866 contested or 2.6%. That’s pretty good. I recently reviewed a market share forecast I prepared in 2012, and was more than pleased – in fact, ecstatic – to be within 4%. The nature of forecasting is that it’s very hard to tell quality (the technical term is skill) from good luck. The nature of forecasting is also that when I started drafting this post, the score was -2 out of 866 or 0.23%, and my point was stronger by a factor of 10 or thereabouts. Either way, it’s not 170.

But Labour would have been vastly better off in 2012-2015 had it been able to derive quality forecasts from polling or canvass data. If party HQ has a much better analytical capability, or the quality and coverage of canvassing is much better, this is an important fact about the practicalities of politics.

It’s just a pity we keep going on about Hitler. Also, the e-mail hasn’t been quite as bad as the leadership campaign, but as of yesterday 21 out of the first 50 messages in my inbox came from different bits of Labour. I’m having words with Tom Watson about it. Seriously:

Lancashire Labour PLUMMETS to miniscule 50% GROWTH

I usually like Jim Pickard’s work for the FT, so I was a bit disappointed by this piece. We certainly do need some data journalism on the surge of Labour Party membership, an underreported fact of British politics and one that has been going on for some time. This mobilisation predates Jeremy Corbyn’s leadership campaign – my own CLP started growing dramatically during 2014 and kept going through the election, until the point where my ward has more Labour members than Conservative voters.

But I’m afraid this won’t do. The FT says:

Take Burnley, for example, which Labour seized back from the Lib Dems in May with a narrow majority. There the membership has risen from 319 to just 484 full members. Or the Rhondda, a deprived constituency in the Welsh valleys, where full membership has risen from 355 to 485.

To put it another way, Burnley CLP’s membership has grown by a puny 45% and Rhondda’s by a miniscule 36%.

Sure, it’s not as dramatic as Holborn & St Pancras, which tripled. But your local Tory membership secretary would fuck pigs for 45% more members…whoops, they do that for fun. I’d rather not think what they might do for 45% more members. The “average” Scottish CLP is up 26% – quite a lot less, but nonetheless it’s substantial growth, given how many of the potential pool of members have gone to the SNP.

A tell in the piece, by the way, is that there are no percentages, nor tables, nor charts, nor links to the data set they used. Another tell is that it quotes Simon Danczuk:

“We have probably had about 100 new members on a total of 600,” says Simon Danczuk, MP for Rochdale and a fierce critic of Mr Corbyn. “I wouldn’t say it’s a massive change.”

That’s an increase of 16%, if you believe Danczuk’s suspiciously round numbers. (Also, what does “probably” mean – doesn’t Rochdale CLP count them?) The interesting bit here, though, is that we can compare two very similar Lancashire mill town constituencies, Burnley and Rochdale. It’s a natural experiment. Burnley CLP has grown by 45%, Rochdale by 16%. One of these numbers is big. The other is small. Also, Burnley is much more marginal than Rochdale, with a Labour majority of 3,244 compared to 12,442. One of these numbers is small. The other is big.

Maybe Burnley’s become a hipster ghetto, a Kreuzberg of the north-west since I was last there…or perhaps it’s just that Simon Danczuk is nobody’s idea of an inspiring leader.

Also, there’s this quote from Hugh Pemberton of Bristol University:

Scottish constituencies were already withering. “It was dire in Scotland; it was clear they had a major problem back then,” said Mr Pemberton.

OK, so the collapse in membership in Scotland was predictive of the election disasters and the rise of the SNP. It follows that increasing membership is probably good, especially in places like Burnley. Right? But somehow that’s not the conclusion. Apparently there aren’t any new members in marginal seats, but the only marginal actually quoted is Burnley, and as we have seen, it’s up by 45%.

Call me a cynic, but I wonder if there actually was a link to the data in an earlier version, but it got spiked? After all, there’s this quote:

The FT has collated data from eight Labour seats that are among the richest 50 constituencies in the country, and eight Labour seats in the poorest 50.

But where are they? The comparison, like the data set, is missing. In the end, the impression I get is that there are two articles here that have been edited into one.

The first one was an honest effort to quantify the mobilisation of Labour members, probably written by Jim Pickard. It concluded that there has been a surge of membership across the board, which was strongest by far in London, but very substantial in Burnley or the Rhondda and far from negligible in Scotland. The second was a hit piece briefed out to Michael Lindsay by one of the “senior party sources” mentioned or possibly Simon Danczuk himself, aiming to deny that there are any new members.

In theory, you line up thesis and antithesis, and get synthesis. In journalism, you line up thesis and antithesis, and get fired.

Why so many Republicans are still running for president

People occasionally wonder why there are still so many Republicans running for president. We can make a simple model of the situation to understand this.

Any candidate who decides to drop out of the race will probably drop out in favour of some other candidate, throwing their support to that candidate. They can expect some kind of reward from that candidate in the event of success. A candidate X will decide to drop out if the expected value of staying in the race falls below the best offer they could get from another candidate.

The value of staying in the race is interesting; it consists of a fairly predictable, risk- or debt-like term which represents their chance of winning based on the current polls, plus an uncertainty or equity-like term which represents the residual value of just being in the race.

While X stays in the race, it’s possible that some other candidate will drop out in X’s favour. It’s also possible that some event will knock out a major candidate and transform the others’ fortunes. This being America, the classic example is a wild-eyed loser sneaking into a Trump rally with a pink AR-15. Public opinion drives the first term, while the other one is basically exogenous, with the important proviso that its value declines as we approach polling day – there is less and less time available for our wild-eyed loser to intervene. This implies that the chance of dropout is asymptotically increasing with elapsed time. It also, interestingly, implies that greater generalised uncertainty – more wild-eyed losers – predicts more candidates, and I think you can actually see this effect in world politics (compare, say, Italian and British political parties).

The potential offer is also interesting. Candidates with less support will be willing to bid higher, because they need additional votes more intensely. We would therefore expect to get the highest offer from the next candidate ahead of us in the race. This would suggest that we’d see a cascade of exits, as the weakest candidates exited first, until only Donald Trump and one non-Donald are left, providing only that non-Donalds all prefer non-Donald candidates to the Donald, and that they can make credible promises of reward.

This last point is where it gets complicated. The offer consists of two components – the reward candidate Y offers candidate X, and the probability that Y will deliver. This itself consists of two components, the probability that Y will be President and therefore in a position to make X Ambassador to the Netherlands or whatever, and the probability that Y will betray X.

It’s possible that social trust is different between the parties, even probable, but this risks letting prejudice cloud our thinking, so let’s assume that political cynicism is evenly distributed for the moment. Therefore, the second component is a constant representing the average level of political deceit.

Now, the risk-adjusted value of the reward will be higher the more likely Y is to be elected President. The face value of the reward will be higher the less likely Y is to become the candidate. If the gap between the leading non-Donald and the rest is large, the credibility effect will dominate; if it is small, the reward effect will dominate. But in the aggregate, the value of any reward offered will be greater the more likely the Party of X & Y is to win the general election, because the winning candidate is more likely to be in a position to deliver.

(Y could also drop out in their turn, and transfer the promise to X along with their vote to some candidate Z, but we can deal with this by pointing out that such promises are only likely to be weakly transitive, and further by thinking about the party’s chances rather than Y or Z’s.)

Compare the Democrats. Most people expect Hillary Clinton to be the candidate and to win. Not surprisingly, all the other potential candidates skipped, accepting either an implicit or explicit offer of office or support in the knowledge Clinton is likely to be able to deliver on it. (Bernie Sanders chose to run in order to make a point, rather like John McDonnell in 2007 or Jeremy Corbyn this year.)

Upshot: the less likely your party is to win, the more candidates will stay in the race to be the candidate, and more generalised chaos tends to cause more candidates. Paradoxical! Further, if there is differential social trust between parties, the more trustful party will tend to have fewer candidates in their race.

Polar Behr wanders the frozen North

This Rafael Behr piece about the Oldham by-election has been getting the bird, not surprisingly given the central prediction was hopelessly wrong. The really interesting bit, in my view, is that if we take a maximally charitable view and assume that there is nothing outside the text, so everyone he spoke to in Oldham who expressed an opinion is mentioned and is quoted accurately…well, he had all the information he needed to call it correctly.

We hear from “Rob”, who apparently voted UKIP at the general election but is going to “lend” Labour his vote this time. That is to say, he’s going to vote Labour. “Jo” is also going to vote Labour. We also hear from “Warren”, described as a “UKIP supporter”. We don’t actually hear whether he intends to vote or not, so we can’t weight him by likelihood to vote, but let’s be conservative and score him a strong K-for-Kipper. We also speak to someone who describes Jeremy Corbyn as “just another liar” and refuses to vote, and a ‘kipper who says he’s not going to vote.

Someone else says Jeremy Corbyn is an idiot and needs to get his act together, but doesn’t say which way they might vote or if they will vote. You might say it probably won’t be Labour on the strength of their remarks, but if you asked me I’d say much the same, and I would have voted Labour.

Either way, we’ve got 2 Labour votes with a full turnout weighting, 1 UKIP vote with a full turnout weighting, two people who aren’t going to vote and whose opinions are therefore zero-weighted, and someone who might do pretty much anything. On the night, 2.69 Labour votes were cast for each UKIP vote, so this micro-poll was actually pretty good. Apparently, Rafael Behr is a fairly effective device for generating randomised population samples!

Or maybe not, as none of his respondents has a name implying South Asian ancestry. But that in itself is interesting; it tells us that the UKIP fantasies of vast numbers of fake postal votes from Those People are just that, fantasies, or rather, excuses. If the white population broke 2.x to 1 against UKIP, they were always on a hiding to nothing, because they just didn’t have the votes, and too many of their supporters were popping off general crankiness rather than seriously proposing to do anything.

This would have been a great story, of course. But such was the power of Behr’s preconceptions, the mere fact that he met twice as many people who actually intended to vote Labour as he did ‘kippers doesn’t seem to have passed through his mind en route from memory to keyboard.

Daniel Davies has formed the opinion that Behr is the worst opinion former currently practicing in the UK. I’m not so sure. This piece on Tories taking an interest in Ed Miliband’s ideas about the economy is not only good, it also appeared in the paper on the morning of George Osborne’s autumn statement, seriously prefiguring the Ozzer’s U-turn so big it was more a Gefechtskehrtwendung. Either that was a very good prediction, or else Behr has a high-quality Treasury source.

the problem with a dead cat strategy is that you end up being that guy with the dead cat

So I was saying to Dan Hardie that every couple of days, I feel relatively optimistic about Labour. Thousands more members sign up. And then either Jeremy Corbyn or John McDonnell does something incredibly strange. Actually, I was literally interrupted at this moment by a notification on my phone, because Ken Livingstone had just done something incredibly strange, by repeatedly insulting Kevan Jones while referring to him as “Jeremy” and futilely trying to outprole the ex-coal miner.

And then we won the tax credits row…until…

Yes, I know he was trying to make a point about wanting to privatise everything by selling it to some other nation’s nationalised industries. But as they say, if you’re explaining you’re losing. And if you’re explaining why you chucked a copy of the thoughts of Mao Zedong across the despatch box…well. I mean, it’s the sort of thing I’d do.

On this occasion, thanks to @simonk133’s Twitter feed, I was able to time the bipolar cycle from victory, to doing something incredibly strange, at exactly 23 minutes. We’ve managed to get this from 24 hours or so in October, down to the same time in minutes. A 60x improvement. It’s like Toyota production, for pratfalls.

and if you do…will anything happen?

The British Election Study is back and it’s very interesting indeed. Here’s the key chart.


The pink zone is essentially people who were convinced by Labour but not motivated to go and vote. Either mobilisation fell dramatically in 2010-2015, or else we managed to turn a lot of heads, but not get them to turn out. The BES commentary says:

The evidence in the BES suggests that the reason for the increased impact of differential turnout is not due to a change in the relative enthusiasm between Labour and Conservative supporters since 2010. 84% of Labour supporters in 2015 said that it was “very likely” that they would vote, compared to 86% of Conservative supporters, while in 2010 the figures were 87% and 90% respectively. Rather the data suggest that the increase in the turnout gap between Labour and the Conservatives can be explained by shifts in party support amongst those who are actually less likely to turnout to vote, even if they say they will. This evidence strongly suggests that differential turnout was a major factor in the polling miss.

The people we successfully addressed agreed, but they didn’t believe anything would happen. Emerging low-trust society, how are ya.

Also, perhaps that database of Shapps’s worked better than we (or they) thought. Mobilisation was a big, big part of our problem, so it is not surprising the mobilisation-first strategy failed. That said, good luck with the option of trying to get as few people as possible to vote!


UKIP as cashpoint

The previous post was about chancers, among other people. Here’s another post about chancers.

What’s interesting in this one is the multiple levels of chancership. You have the two blogger/staffer/whatever types, classic chancers, who would love to hire on with UKIP, but mostly because they could sell that experience to someone with more money. You have the Americans, who have more money but are still only Breitbart, who would love to get involved with UKIP so they could have a business in Britain.

And you have Nigel Farage himself, who after all the toing and froing the post describes ended up by adding a sinecure with Breitbart to the one he has with Russia Today. Cute, in the Irish sense. On the way, Farage did manage to meet Rupert Murdoch, which is serious, but also Senator Jeff Sessions of Alabama, which is…somewhat less helpful, unless you’re Breitbart of course and you have a radio show to fill.

As they say in Nigeria, when thief thief thief, the world laughs. Especially as the dense, intertwined chancership seems to have rendered UKIP much less effective than it might have been…except from the point of view of its proprietor.

Chancers, respectability politics, and IDS: A4e, Kids Co, and Trussell

What to make of this story? Camila Batmanghelidjh has been basically forced out of Kids’ Company after the Cabinet Office wanted to know what it’s been doing with the money. Much detail is available in a Buzzfeed piece of the sort you’re meant to think Buzzfeed doesn’t do.

She, of course, blames austerity, cuts, etc. This is pretty rich coming from someone who repeatedly endorsed David Cameron in the run-in to the 2010 elections and even afterwards, appearing at the 2006 Conservative conference and taking part in the Big Society founding meeting at Downing Street in May, 2010.


This may have been the brief interlude between the coalition being formed and the first Osborne Budget, but they had already agreed on the extra £6bn in-year cuts, so it’s not as if we weren’t warned.


One way to look at this is the photo above. The Tories had a go at being nice, that’s how she fell in with them, and then they turned on her. Because they’re Tories. By December 2010, this narrative was already falling into place. Even the Lib Dems had their own creepily sexist and racist version of the same pattern.

But there was always another side to the Big Society project. The May 2010 kickoff meeting also included Ray “Robocop” Mallon, mayor of Middlesbrough and professional populist, for example. That doesn’t sound like the nice straightforward story about the Tories letting down a nice charity for children, does it? Also, this blog post was justly much read for what it said about working for Kids Co – pretty much the management from hell – but it also mentioned them “feeding in” to the DWP’s policy agenda. It also brought out was the degree to which their astonishingly high staff-pupil ratios were achieved by ruthlessly strip-mining volunteers.

The post and indeed the whole site has now been scrubbed, to the point of getting rid of it from the Internet Archive. But let’s take a look at what they might have been feeding in to policy. Here’s a document from July 2007.

Ms Batmanghelidjh told the MPs: “I actually think the mothers [specifically black – ed] are hugely responsible because they have created a culture where they can get rid of the adolescent boy. They can get rid of the male partner, they can survive on their own.

Often people think it’s the males who are the culprits, the irresponsible people who actually come along and make these girls pregnant and walk off. And they underestimate the level of rejection and cruelty from the females towards the males.

I actually think the males are really vulnerable and it starts in adolescence. The minute the adolescent boy begins to look slightly like a male and behave like a male, often the mother wants that young male banished from the house. A hate relationship often develops. I really think we underestimate the vulnerabilities of young black men.”

I don’t know about you but I see a hell of a lot of NOT OK in there. The “males”? That’s not a bit…veterinary? Also, I read the first paragraph as carrying an anti-welfare subtext, especially as the target audience was a bunch of MPs.

What we’re talking about here is US-style respectability politics, really. For some fine snark, which would do for a quick start tutorial, try Adam Serwer. This is very important in understanding Iain Duncan Smith and specifically his interest in the works of Lawrence Mead. It’s a double-edged sword; one swing blames the poor for their problems, while the backswing has at the unglamorous poor-bloody-infantry in schools and social services with their unions and budgets.

As I’ve said before, I see A4e, Kids’ Company, and the Trussell Trust as key institutions in understanding how we got into this mess.

A4e actually did start off as a grassroots charity in post-steel Sheffield, but it drifted – in terms of its leaders’ aspirations, in terms of the increasing irrelevance of the approach both it and the DWP had learned, and in terms of its ability to manage a rapidly growing team of people who were working on commission, essentially a sales force.

Kids’ Company represents something different. In its constant drive to recruit celebrity donors, it came to offer legitimacy to the politicians. Iain Duncan Smith, more than David Cameron, locked onto this as political cover for his respectability politics agenda. Because it didn’t really have an articulated agenda of its own, it also leaked unexamined prejudices into the policy market (see above). And its commitment to an interventionist, treatment-ist world view fit right in with the IDS agenda.

Trussell will need another post, or a PhD, but I’m deeply suspicious of it because I can’t think of anywhere that has foodbanks that has ever managed to get rid of them and return to a normal society. I suspect they know this and worry about it, which is why they’re really quite critical of the DWP, rather than just hollering for more budget, and why Iain Duncan Smith keeps falling out with them.

And of course there’s the great overarching IDS theme, the role of the chancer. It is beautifully ironic that the great chancer of the times himself accuses Trussell of chancerism, when they are the least chancerish of the satellite NGOs spinning around DWP. Chancers are important; have you ever wondered why everyone started carrying bottles of water around in the 1990s? Wonder no more. Yes. He’s her dad.

Another Labour-SNP post.

Circling back on this post and also this one, I think the EVEL fiasco has been a really important political event, because it has decided a really important issue.

Coming out of the elections, there was a possibility at least that the SNP and the Government would come to an arrangement. The Government would give them what they wanted regarding devomax, and in return, the SNP would take a strict interpretation of the idea that they stayed out of non-Scottish issues. This could have been formalised by EVEL or just left as a political agreement. This would have suited the Conservatives rather well, as it would have given them a free hand on anything that could be coloured as an England-and-Wales issue.

The alternative was that the SNP would stay in protest mode and assert Scottish interest in almost anything controversial. As we’ve seen this week, this wouldn’t suit the Conservatives at all. The first option would basically replace the Lib Dems with a sort of negative coalition partner – rather than adding to the government bench, it would subtract from the opposition bench. The second would do the opposite.

For the SNP this amounts to a choice between holding onto their precious capital of authenticity, and cashing it in for influence. I discussed this issue in a previous post. Arguably their authenticity is so precious they would be well advised always to pick opposition, but as with a lot of political decisions, leaving it uncertain up to the last minute has value. Also, they have to balance cooperating at Westminster with competing at Holyrood.

So the ideal situation for the SNP is one where Labour decides to go quiet on some issue, because they can’t count on the SNP to vote against, and the SNP then flips, making them look like the real opposition. We’ve seen this play out over the Welfare bill this week – although the tax credit elements are clearly UK-wide, housing is a devolved issue and as Joe Halewood points out, the SNP might want to implement some of the Bill themselves in order to give social tenants a rate cut and ram the bill to Labour-run local councils.

Going back to my authenticity post, I think the idea of an In-party in Holyrood that’s an Out-party in Westminster fits the facts pretty well and therefore Labour should work on the presumption they will show up. So, yes, joining the bandwagon against Harman a bit late.

The SNP is in. Let’s find out which Tories are really up for it.

So Labour are voting against the relevant clauses in the Finance Bill tomorrow and the Welfare bill is up to the new leader. The SNP will be voting against fox hunting on Wednesday.

These facts are connected. The Westminster SNP is apparently pissed off enough about the attempt to pass EVEL earlier in the week that it doesn’t feel bound by the idea that it should abstain on anything in Holyrood’s area of responsibility.

As a result, trying to pick off the centremost 6 Tories on the welfare bill is most definitely back on. A few days ago it looked like doing anything effective about it meant trying to find 62 votes, as there was no assurance of the SNP having a vote or using it if they did. Now, though, they’re going to be available, so the target is down to 6. What was really important this week was getting the cooperation between the Labour and SNP whips going again and seeing off the EVEL gerrymander.

Having got that done, a lot of possibilities open up. On the tax credits/housing issue, who are the six weakest Tories? Let’s get the chin music going. They tell me it’s hell out there.