if you can’t spell this you might be a troll

So I got round to reading the original paper about automatically predicting who’s likely to be a troll. This was always likely to be fun:


Defining trolls as those who get banned for trolling, a pragmatic solution if nothing else, they obtained a large corpus of comments from three high-volume sources, CNN, a gamer news site, and Breitbart. (Clearly they weren’t about to risk not finding enough trolls.) They paid people to classify the comments on various metrics, and also derived a lot of algorithmic metrics, and used this to train a machine learning model to guess which users were likely to be banned down the line.

The results are pretty fascinating. For a start, there are two kinds of troll – ones who troll-out fast, explode, and get banned, and ones whose trollness develops gradually. But it always develops, getting worse over time.

In general, we can conclude that trolls of all kinds post too much, they obsess about relatively few topics, they are often off topic, and their prose is unreadable as measured by an automated index of readability. Readability was one of the strongest predictors they found. They also generate lots of replies and monopolise attention.


Not surprisingly, predictions are harder the further the moment of the ban is into the future. However, the classifier was most effective looking at the last 5 to 10 posts – it actually lost forecasting skill if you gave it more data. Fortunately, because trolling is a progressive condition that tends to get worse, scoring the last 10 comments on a rolling basis is a valid strategy.

Their algorithm, in the end, identified trolls with about 80% reliability. Very interestingly indeed, its performance didn’t suffer much if it was trained against normal below-the-line noise and then used on gamergate, or if it was trained against gamers and then used on libertarians (perhaps less of a surprise), or whatever. The authors argue that this is an indication that it’s picking up some kind of pondlife tao, an invariant essence of disruptive windbag.


The really interesting bit, though, was when they got to the feedback-loop between potential trolls, moderators, and the civilian population. You might think that being able to identify potential trolls within the first 5-10 comments presents the possibility of an early-intervention strategy. My own experience with Fistful of Euros back when it had 150-comment threads about the Middle East was that explicit early warnings – yellow cards – often worked. They found, however, that earlier and more aggressive intervention from moderators and other users was correlated with faster escalation. Specifically, those who had posts deleted early saw their readability index scores worsen rapidly, one of the strongest markers of trollness.

Now, you might say this doesn’t matter. Just stick the OtoMerala Super-Rapid 76 in automatic close-in defence mode and let the machines do the work!

But there’s a serious issue here and it’s our old pal, the Terroriser algorithm. They make the excellent point that 80% is pretty good but it’s a lot of false-positive results. Given that the principal components we mentioned above are basically conventional norms of discursive civility, there’s also the problem that our filter might be both racist and snobbish. The fact it worked well across dissimilar communities, though, is encouraging.

The distinction between fast and slow trolls – Hi-FBUs and Lo-FBUs in the paper – also suggests that there’s something going on here about different strategies of anti-social behaviour. Perhaps trolls with more cultural capital adopt strategies of disruption that allow them to persist longer and do more damage? More research, as they say, is needed. That said, I wouldn’t write off early intervention completely, and neither do the authors – the question may just be an optimisation.

Obviously what it needs now is an implementation.

why Ed Miliband won’t commit in advance to the SNP

Why won’t Ed Miliband commit to a deal with the SNP in advance? The explanation is incredibly simple. Here are the last three Scottish polls – Survation for the Daily Record, TNS Global, and Ipsos MORI. As expected they both show a monster LAB>SNP swing. But the interesting bit is this: there are a lot of undecided voters.

TNS, for example, shows 29% of their sample as Undecided. That’s twice as many as all the other parties other than Labour or the SNP. That’s as many Don’t Knows as there were Scottish Nationalists. That’s more than the gap between the SNP and Labour. The biggest groups of DKs are the young (34% of 18-34s, 38% of 18-24s). Socioeconomic groups C1 and C2 and women are the other likely DKs, but the difference from the national result is much smaller.

Survation formulated its questions rather differently. As a result, you might think TNS got a dodgy sample, as they only saw 11.5% Undecided. However, they also asked those respondents who picked a party in the voting intention question if they might change their minds between now and polling day – i.e. if they might actually be undecided. They found 19.2% were still, in this sense, undecided.

You can’t add the two percentages, because the second only includes those who picked a party. But we have the full tables, so we can tot up the 163 undecided here and the 111 earlier and divide by the weighted n=968, and conclude that 28.3% of Survation’s respondents are undecided.

Ipsos MORI, on the same basis, gets 26% undecided/DK. Pretty close!

If you’re Scottish, you are much more likely to prefer a Labour government to a Conservative one. TNS makes the split 35% to 16%. Therefore, if you’re undecided you are most likely wavering between a Labour-first route to this or a Labour-SNP route. The most immediate reason to prefer Labour is obviously that voting SNP denies Labour seats it needs to form a government and therefore risks a Tory-[something] coalition.

If Ed Miliband was to pre-commit to an alliance with the SNP, this possibility would vanish and with it, any chance to save seats in Scotland. Also, you know if he thought there was any advantage in stabbing Scottish Labour in the neck he’d already have done it.

With almost a third of the voters still in play, there is absolutely no point in giving up. This ain’t the movies, so it’s not as if all 29% of them will break for Labour, but there is a possibility that the SNP will disappoint significantly on the night. Rather like they did in the referendum, in fact. Also, Survation provides the interesting detail that the biggest group of people who might change their minds, out of those who named a party, are ex-Lib Dems, or in other words, the group of people in the UK most likely to support Ed Miliband.

That’s why Miliband won’t pre-commit to a deal with the SNP.

that was meant to be satire

So I was all sarcastic about Iain Martin pretending Peter Lilley foresaw the great financial crisis back in 1997. And then I found out from a comment on Simon Wren-Lewis’s blog that Lilley actually claims this himself.

” …..I was shadow Chancellor when the Bill that became the Bank of England Act 1998 was introduced. He pointed out that I then warned the House that: –

“With the removal of banking control to the Financial Services Authority…it is difficult to see how…the Bank remains, as it surely must, responsible for ensuring the liquidity of the banking system and preventing systemic collapse.”

And so it turned out. I added:

]“setting up the FSA may cause regulators to take their eye off the ball, while spivs and crooks have a field day.”-[Official Report, 11 November 1997; Vol. 300, c. 731-32.]

So that turned out, too. I could foresee that, because the problem was not deregulation, but the regulatory confusion and the proliferation of regulation introduced by the former Chancellor, which resulted from a failure to focus on the banking system’s inherent instability, and to provide for its stability.”

The line about regulators “taking their eye off the ball” is the one Martin relied in his book on RBS. It was nonsense then and it’s still nonsense now. The plain English meaning of Lilley’s remark is that the process of setting up the FSA might distract the regulators temporarily. Does anyone imagine the phrase as he uses it might stretch to events ten years in the future, long after the FSA was operational?

The remark about the Bank of England “remaining…responsible for ensuring the liquidity of the banking system” is even worse. “Ensuring the liquidity of the banking system” is what a central bank does all day, every day. In 2007, it ensured the liquidity of the banking system by lending the banks a lot of money, and then later by implementing QE. It did so until it thought Northern Rock was insolvent, and then, as its principles required, cut it off.

Lilley might have had a point had he said “solvency” rather than “liquidity”. The problem, in the end, was that the banks ran out of capital, not out of cash. If it had only been a liquidity problem, the Bank of England would have been able to fix it with much less drama. But he didn’t say that.

And he couldn’t possibly have done so. After all, he is still in denial about financial deregulation. He claims the problem was too much regulation. Surely, if he believes that, distracting the regulator would have been a good thing? And if the problem is too much regulation, what would the Bank have done to restrain the banks? Wouldn’t that be awfully like regulation?

The main conclusion I draw from this is that Lilley is just as weaselly as he was when he was a minister.

An interesting chart on Ashcroft polls

Someone wanted an evaluation of Ashcroft polling (possibly Dan Hardie). LSE‘s election forecasting project has tried to characterise the difference between their model and Ashcroft observations. They are looking at this in the opposite sense, because they have a forecasting model and Ashcroft polls offer more observations to constrain it with, but you could also look at it as being how much they diverge from an adjusted, blended national VI uniform swing model. Going straight to a chart:


A higher result for LSE’s model than Ashcroft’s poll is to the right, and vice versa. Also, this chart shows us the degree of variance in these results through their spread.

So it looks like LSE reads high/Ashcroft low on UKIP and Ashcroft reads low/LSE high on Tories. There is quite a bit of variation, but the distinction is clear. LSE is high/Ashcroft low on the Greens, but not by much, although the difference is very consistent. On Labour and the Lib Dems, there doesn’t seem to be a systematic difference and there is a lot of variability, i.e. randomness.

I think I said earlier that the big difference between pollsters at the moment seems to be how they split Tories and ‘kippers.

No, the Treasury did not “check it was FCA-compliant”

Remember that time the Conservative Party wrote to pensioners spruiking the 4% codger bonds and pretending to be an official communication from the Government? Sure ya do because I blogged it.

Back then, either Rowena Mason or Patrick Wintour got the following response from a nameless “Treasury aide”, i.e. one of Osborne’s spads aka @ToryTreasury, either Rupert Harrison, Poppy Mitchell-Rose, Eleanor Shawcross, or Ramesh Chhabra.

A Treasury aide said: “HM Treasury officials checked and confirmed this is FCA-compliant.”

Got that? HMT, he claims, checked that this was compliant with the Financial Conduct Authority’s rules. I can tell you now, thanks to the Freedom of Information Act request I filed, that has just been answered, that they did no such thing or at least the FCA denies it has any record of it. I’ve caught and rethrown it to NS&I.

The FCA also says that if you asked it, it wouldn’t answer because it doesn’t “pre-approve” financial promotions. It also says that National Savings & Investments is outside their authority because it belongs entirely to the Treasury, oh miracle. However, they also specifically deny that anyone checked.

Presumably the check consisted of Osborne asking himself if he thought it was OK, but in that case, why bother saying that they checked that it was OK with the FCA when the FCA doesn’t have anything to do with it?

so who is meant to be getting the extra £3.9bn in LHA?

OK so, we’ve had the Tories’ big idea, Right to Buy in clown shoes, denounced at the same time by the Southwark Renters’ Maoist reading group and the Confederation of British Industry. Truly, the coalition was the golden era of the harebrained scheme and the half-baked thinktank.

But let’s try to keep a straight face. What if the idea isn’t to “move” social housing, but rather to replace it in-situ? Travelling without moving; wasn’t that a Jamiroquai song? The Tories apparently want to sell councils’ most “expensive” homes.

Expensive here is a term of art; as everyone already pointed out, they are already built and don’t cost that much to run. What is expensive here is the land they sit on, which should point us clearly towards London. The idea is that as long as this land is not sold, the local authority is foregoing the money it could get from selling it, and this is a kind of cost. However, the local authority cannot just sell because it has a legal obligation to house people. What to do?

A surface reading of the proposal tells us that the homes sold would be replaced by “cheaper” ones. Most comment on this assumes that this means building new somewhere else, on cheaper land. But as we are now talking about building new, it might also mean building more cheaply, or perhaps more cheaply from the point of view of DCLG, in other words, getting someone else to pay.

Now, the proposal is that 12,000 “expensive” flats would be sold. This, of course, doesn’t clear the land they are built on – right-to-buy property is always leasehold on the council’s land. At the same time, 12,000 more “cheaper” units are meant to be built. These are described as “affordable”, a word that has a specific legal definition – it means that the rent is set at the 80th percentile of market rents in the area, usually about three times the social rent charged by local authorities. In effect, this means that more tenants end up claiming more Local Housing Allowance, a straight transfer from taxpayers to landlords. The NAO has scored the cost of the Tory proposal in LHA at £3.9bn.

The policy statement says that councils will “oversee” the process. Obviously this does not state that they will own the replacements, nor that they will be the client for the job. One way in which it might indeed be “cheaper” from a council or DCLG point of view would be if the job was done by private developers. The money from sales would be used to get the ball rolling, but they would borrow the rest, in the knowledge they could count on a substantial revenue stream from LHA. This might explain the Natalie Bennett-esque weirdness of the numbers involved.

I don’t know about you, but this makes me think about the emerging lobby for “estate regeneration” and the curious entity, “Create Streets”, run by intimate of Peter Lilley and ex-Tory MP, Nicholas Boys Smith, his mate Heneage the Dublin boom-era developer who dumped his losses on the Irish taxpayer, and George Osborne’s old speechwriter.

The idea is that either major London estates get clear-felled, and then rebuilt at much higher density and 80% rents, or else that a lot of new, 80% rent, property gets built as in-fill in the existing ones. Either way, a lot of people lose their existing homes in exchange for promises, and the 80% rents get billed to the central government budget for Local Housing Allowance. A hint that we’re going this way might be the mention of a £1bn fund being set aside for “brownfield”, i.e. urban, projects.

This outfit is seriously under-scrutinised and the reason is that it is hugely well-connected and is paying for a lot of lobby, with God knows who’s money. Here’s Faisal Islam uncritically nodding along. Here’s mysteriously influential unelected pol Andrew Adonis rebranding it “city villages” alongside Gary Yardley of Earl’s Court Project infamy and his ex-Blairite PR man who happens to be all over Tessa Jowell’s campaign for mayor like a nasty rash, in a piece by the Guardian‘s Dave Hill that takes the opportunity to complain about the dirty hippies in fine Iraq War-era style:

On the plus side, City Villages takes on some of the head-bending housing supply issues that much of London’s current, high-profile, oppositional housing activism hasn’t much to say about…Activists are against that too…

He’s sort of aware of the problems, but he just wants to mourn this terrible lack of civility from the internet juicebox mafia. I remember that. Well, Dave, if it’s ideas you’re after you could try the London Labour Housing Group blog. But the last time we did this the government minister responsible’s son ended up owning them all. You can see how some people might be a little tetchy.

The other take-away from this is that clearly the day after the election will be time to pull, roll, and get on the Jowell campaign’s tail. The thing about being oppositional is that you have to learn to like it.

After the latest update the problems will be solved…

Remember that time Mitt Romney bought his campaign a massive IT system that completely broke down and failed utterly, because they gave all the money to their mates’ outsourcing company and did everything on the cheap, and nobody would take any responsibility for it? Now consider the possibilities of getting Grant Shapps to procure a major computer system. He’s a digital entrepreneur, after all. Both of him.

The Tory Diary:

Thirdly, there’s the question of campaign technology. VoteSource has arrived, Merlin is mercifully on its way out, and I gather that after its latest update many of the new system’s early teething problems have been solved. However, it’s far from ideal to receive and have to adapt to a whole new platform so shortly before polling day. Grant Shapps would, I’m sure, have introduced it earlier had he been appointed Party Chairman earlier, and previous Party Chairmen should have acted on it years ago, but the practical impact remains the same regardless of intentions. Most associations still only have a small number of people trained in using VoteSource, while recent weeks have been spent battling with and then ironing out problems (either of the system or of its users’ abilities or both).

O HAI. It looks like Flipper’s trying to tell us something. Shapps decided to cut over to a completely new platform? After the campaign started? This could be fun. You’ve got to love the “if only the Tsar knew!” bit where they say the solution would obviously have been even more Shapps.

That said I did have an exciting technology moment on the doorstep the other weekend when our voter ID database claimed I was about to canvass probably the only black, cockney Plaid Cymru voter in London. Well, as the board runner said, anything’s possible, but the returns use a one-letter code for most things and then get OCRd, so it’s quite possible that a slightly sloppy character going through the slightly chancy OCR process gives you a weird data point.

She’s going to vote Labour.