Junk science – the oil of the new web

There’s a case to made that James Surowecki’s The Wisdom of Crowds is the most influential book of the decade – The Selfish Gene for the noughties. Both have something else in common: the title of each book is profoundly misleading. Crowds aren’t wise, nor can genes be selfish – as one critic famously wrote, any more than atoms can be jealous.

Just as the young polemicist Dawkins paved the way for the social darwinism of the Reagan and Thatcher years, Surowecki’s discussion of futures markets and “collective intelligence” provides the flimsy premise for a spending splurge on junk technology. It’s the common thread that unites several of the disparate “Web 2.0″ start-ups we wrote about yesterday, in our must-read roundup.

Both authors were the catalyst for entire schools of junk science – yet both can justifiably claim to have been misrepresented to some degree. While Surowecki is clearly as bewitched by “collective intelligence” as Dawkins was by a gene-eyed view of evolution, he also warns that the crowd only picks winners in very specific circumstances, where the collective guess work acts as a kind of risk hedging. If these factors aren’t present, then the market falls victim to the inevitable: gaming.

But even when this appears to work, so what? Seth Finkelstein notes that in some situations, throwing darts at a dartboard produces excellent results. Citing the Wall Street Journal Dartboard Contest, he writes,

“People are fascinated by ways in which data-mining seems to represent some sort of over-mind. But sometimes there’s no deep meaning at all. Dartboards are competitive with individual money managers – but nobody talks about the ‘wisdom of darts'”

And today, Canadian hockey fans are rejoicing in the return of Maggie the Macaque. The simian (on the right) out-performed the experts in predicting the results of key games during the 2003 season. Could it be Maggie’s diet of crabs, or could it be – “The Wisdom of Monkeys”?

One need only look at the composition of the internet to understand why the “Wisdom of Crowds” will never apply: the internet isn’t representative of society, and even amongst this whiter-than-white sample, only a self-selecting few have any interest in participating in a given pseudo-market.

While Wisdom of Crowds was self-consciously written with the purpose of restoring the public’s faith in the market, after the dot.com bubble burst – it was titled after Charles Mackay’s Extraordinary Popular Decisions and The Madness of Crowds – it’s had the opposite effect.

The self-selecting nature of participation in computer networks simply amplifies groupthink. Facts that don’t fit the belief are discarded. The consequences abound, wherever you look.

The great Wikipedia experiment is already over, says Nick Carr, the inevitable result of an open editing policy.

He cites what may prove to be the 21st Century’s equivalent of the 1948 newspaper headline, “DEWEY WON”, Time magazine’s declaration that,

“everyone predicted that [Wikipedia’s] mob rule would lead to chaos. Instead it has led to what may prove to be the most powerful industrial model of the 21st century: peer production. Wikipedia is proof that it works, and Jimmy Wales is its prophet.”

Praise be!

But to buy into this world view, one must disregard all evidence to the contrary. Veteran Wikipedia administrator ‘Skippy’ of Wikitruth.info – a site strangely absent from Wikipedia’s “sum of all human knowledge” – mailed us his summary yesterday:

“Wikipedia is proof that an encyclopedia that ‘anyone can edit’ doesn’t mesh with the reality of human nature.”

A harsher summary from the Village Voice recently declared:

“No true believer in the democratic promise of the Web can fail to gladden at the very mention of this grand experiment – the universal encyclopedia ‘anyone can edit’!—or fail to have noticed, by now, what a fucked-up little mockery of that promise it can sometimes be.”

It’s no surprise to discover that Time magazine’s puff piece was written by WiReD magazine editor Chris “Long Tail” Anderson. Three years ago, Anderson bet your reporter that by today Wi-Fi chipsets would outsell GSM or CDMA chipsets. This was on the occasion of an Intel-sponsored edition of his publication, and Anderson was in the grip of the religious mania about Wi-Fi. His prediction has fallen short by around a billion units.

(If you want faith-based economic theory, Anderson’s your man.)

We’ve written about groupthink on so many occasions – particularly after the collapse of the Howard Dean presidential run – we won’t bore you with repetition. But a golden rule of internet companies is that the more faith they place on the “new wisdom of the web”, the more inevitable their demise.

For Google, which buys into the junk science more than any other Silicon Valley company, this is very bad news indeed. The “democracy of the web” was short-lived, and the company devotes most of its brainpower resources not to developing new products, but trying to rescue its search engine from “Grey Goo”. Faith-based junk science can be a real handicap.

Where does all this affect us? Wherever their advocated bad ideas waste money and resources. For those of us who want better technology, the mini splurge of capital investment in fatuous companies is more than troubling. A dollar spent on a doomed web site is a dollar that could have been spent on solving some real, overdue infrastructural problems.

Seth Finkelstein points out an immediate consequence which is already taking place. Wisdom… gained such traction on the net, because of its cultural distrust of expertise. This stops where the net stops, however – it’s hard to envisage even the most militant Wikipedia fan choosing to be operated upon by amateur heart surgeon. But it’s accelerated the process of deskilling, and the new flood of cheap (but wise!) amateur labor promises to depress wages even further.

The media, and Time is a great example, espouses the rosy view that our public networks are in rude health. I’m confident that this utopian view carries little weight with a public frustrated with pop-ups, viruses and spam.

So to return to our original question. If the public so wilfully buys into sloppy thinking, are the authors themselves responsible? In the case of both Dawkins and Surowecki, who mistitled their books, they may protest too much