Junk science – the oil of the new web

There’s a case to made that James Surowecki’s The Wisdom of Crowds is the most influential book of the decade – The Selfish Gene for the noughties. Both have something else in common: the title of each book is profoundly misleading. Crowds aren’t wise, nor can genes be selfish – as one critic famously wrote, any more than atoms can be jealous.

Just as the young polemicist Dawkins paved the way for the social darwinism of the Reagan and Thatcher years, Surowecki’s discussion of futures markets and “collective intelligence” provides the flimsy premise for a spending splurge on junk technology. It’s the common thread that unites several of the disparate “Web 2.0” start-ups we wrote about yesterday, in our must-read roundup.

Both authors were the catalyst for entire schools of junk science – yet both can justifiably claim to have been misrepresented to some degree. While Surowecki is clearly as bewitched by “collective intelligence” as Dawkins was by a gene-eyed view of evolution, he also warns that the crowd only picks winners in very specific circumstances, where the collective guess work acts as a kind of risk hedging. If these factors aren’t present, then the market falls victim to the inevitable: gaming.

But even when this appears to work, so what? Seth Finkelstein notes that in some situations, throwing darts at a dartboard produces excellent results. Citing the Wall Street Journal Dartboard Contest, he writes,

“People are fascinated by ways in which data-mining seems to represent some sort of over-mind. But sometimes there’s no deep meaning at all. Dartboards are competitive with individual money managers – but nobody talks about the ‘wisdom of darts'”

And today, Canadian hockey fans are rejoicing in the return of Maggie the Macaque. The simian (on the right) out-performed the experts in predicting the results of key games during the 2003 season. Could it be Maggie’s diet of crabs, or could it be – “The Wisdom of Monkeys”?

One need only look at the composition of the internet to understand why the “Wisdom of Crowds” will never apply: the internet isn’t representative of society, and even amongst this whiter-than-white sample, only a self-selecting few have any interest in participating in a given pseudo-market.

While Wisdom of Crowds was self-consciously written with the purpose of restoring the public’s faith in the market, after the dot.com bubble burst – it was titled after Charles Mackay’s Extraordinary Popular Decisions and The Madness of Crowds – it’s had the opposite effect.

The self-selecting nature of participation in computer networks simply amplifies groupthink. Facts that don’t fit the belief are discarded. The consequences abound, wherever you look.

The great Wikipedia experiment is already over, says Nick Carr, the inevitable result of an open editing policy.

He cites what may prove to be the 21st Century’s equivalent of the 1948 newspaper headline, “DEWEY WON”, Time magazine’s declaration that,

“everyone predicted that [Wikipedia’s] mob rule would lead to chaos. Instead it has led to what may prove to be the most powerful industrial model of the 21st century: peer production. Wikipedia is proof that it works, and Jimmy Wales is its prophet.”

Praise be!

But to buy into this world view, one must disregard all evidence to the contrary. Veteran Wikipedia administrator ‘Skippy’ of Wikitruth.info – a site strangely absent from Wikipedia’s “sum of all human knowledge” – mailed us his summary yesterday:

“Wikipedia is proof that an encyclopedia that ‘anyone can edit’ doesn’t mesh with the reality of human nature.”

A harsher summary from the Village Voice recently declared:

“No true believer in the democratic promise of the Web can fail to gladden at the very mention of this grand experiment – the universal encyclopedia ‘anyone can edit’!—or fail to have noticed, by now, what a fucked-up little mockery of that promise it can sometimes be.”

It’s no surprise to discover that Time magazine’s puff piece was written by WiReD magazine editor Chris “Long Tail” Anderson. Three years ago, Anderson bet your reporter that by today Wi-Fi chipsets would outsell GSM or CDMA chipsets. This was on the occasion of an Intel-sponsored edition of his publication, and Anderson was in the grip of the religious mania about Wi-Fi. His prediction has fallen short by around a billion units.

(If you want faith-based economic theory, Anderson’s your man.)

We’ve written about groupthink on so many occasions – particularly after the collapse of the Howard Dean presidential run – we won’t bore you with repetition. But a golden rule of internet companies is that the more faith they place on the “new wisdom of the web”, the more inevitable their demise.

For Google, which buys into the junk science more than any other Silicon Valley company, this is very bad news indeed. The “democracy of the web” was short-lived, and the company devotes most of its brainpower resources not to developing new products, but trying to rescue its search engine from “Grey Goo”. Faith-based junk science can be a real handicap.

Where does all this affect us? Wherever their advocated bad ideas waste money and resources. For those of us who want better technology, the mini splurge of capital investment in fatuous companies is more than troubling. A dollar spent on a doomed web site is a dollar that could have been spent on solving some real, overdue infrastructural problems.

Seth Finkelstein points out an immediate consequence which is already taking place. Wisdom… gained such traction on the net, because of its cultural distrust of expertise. This stops where the net stops, however – it’s hard to envisage even the most militant Wikipedia fan choosing to be operated upon by amateur heart surgeon. But it’s accelerated the process of deskilling, and the new flood of cheap (but wise!) amateur labor promises to depress wages even further.

The media, and Time is a great example, espouses the rosy view that our public networks are in rude health. I’m confident that this utopian view carries little weight with a public frustrated with pop-ups, viruses and spam.

So to return to our original question. If the public so wilfully buys into sloppy thinking, are the authors themselves responsible? In the case of both Dawkins and Surowecki, who mistitled their books, they may protest too much

Justice Dept slams 'Machiavellian' Microsoft

Vista’s fate foretold… by Machiavelli. A prophetic quote.

trust settlement, and quoted Machiavelli to support its case for an extension to the monitoring program.

As of 1 February, over 700 issues remained outstanding out of over 1,000 submitted to the monitoring committee, which was set up to ensure Microsoft keeps to its word in the settlement to the long running anti-trust lawsuit. Microsoft was found guilty in 2000 of abusing its monopoly position, and a final decree issued in 2002.

The decree set up a monitoring program that’s due to expire next year. Now the DoJ wants to extend the compliance monitoring program for at least two years to 2009, and ideally to 2012 – by which time Windows Vista may or may not have been released.

The monitoring committee says Microsoft’s compliance is so inadequate that even Microsoft agrees it needs to be restarted. The software giant is keeping contractors in Bangalore busy as it races to complete protocol documentation which almost everyone agrees is useless, in time for a June deadline.

The DoJ quoted Machiavelli to describe Microsoft’s chaotic development procedures, which if you’re being charitable, explains its difficulties in explaining how its software works.

“He who has not first laid his foundations may be able with great ability to lay them afterwards, but they will be laid with trouble to the architect and danger to the building,”

That’s a quote from Chapter eight of The Prince, titled “Concerning New Principalities Which Are Acquired Either By The Arms of Others, Or By Good Fortune”. That’s where Machiavelli cautions on how to avoid “inconstant and unstable things.”

Perhaps he was giving the Medici family advanced warning of the Windows USB stack.

The worse Google gets, the more money it makes?

Microsoft today is barely acquainted with how its software is produced. Now Google’s search results look similarly out of whack.

It’s hard to imagine now, but there was a time when the mainstream press was barely acquainted with the genius and foresight of today’s technology leaders.

Fifteen years ago Bill Gates appeared on the BBC’s Wogan show – which the Beeb thought of as a nightly Johnny Carson, but which was really like watching Regis Philbin on cough syrup – to show off his WinPad PC. The wooden Gates made a joke about making his money disappear, with only a couple of clicks, using only a stylus. As Gates blinked, a nation which had never heard of Microsoft, and couldn’t quite figure out why the guy in glasses wasn’t singing or dancing, looked on in sympathetic embarrassment.

But Gates’s prime time TV appearance underscored one point, popular in the public prints at the time, which was that a nerdish, upstart technology was changing the very foundations of the world as we know it. Microsoft was simply smarter, more agile, more cunning, and far more darkly mysterious than the fusty incumbents, like IBM, could ever realize. To stand in the way of Microsoft was to stand in the way of youth, innovation and progress itself.

Now, it may puzzle you as much as it puzzles us that this idea ever gained popular currency – let’s save that discussion for another day. But it can’t have escaped your notice that this mythical struggle has been reprised by the inkies several times – in the mid-1990s with Netscape – and today with the phoney war between Microsoft and Google.

If you’re of the view that history repeats itself the second time round as farce, then the parallels are even more uncomfortable.

Continue reading “The worse Google gets, the more money it makes?”

'Lightweight, high-velocity and very connected'

At ZDNet, it’s Microsoft’s “Pearl Harbor”! Forbes screams, “Google’s office invasion is on!”

Only it isn’t – and we have the founder’s word for it.

As we reported yesterday, Google has paid an undisclosed sum for a web-based document editor, Writely. It’s a product that seems as mature as the company which produced it, Upstartle.

Explaining why she decided to sell the company, whose only product has been in a limited, closed beta for just six months, co-founder Claudia Carpenter wrote –

“We like lava lamps and they’re pretty much standard decor at Google.”

Moving onto the vision thing, Carpenter explained –

“Writely is like a caterpillar that we hope to make into a beautiful butterfly at Google!”

(No blonde jokes, please.)

A measure of how mature the software is can also be gleaned from this blog post. Writely gained the feature “delete from trash” five weeks ago, a lower priority for the team than “new toolbar”. When the ability to remove your own work from a hosted web service is considered less important than cosmetics, you have a fair idea of the software designers’ values.

So far, so very “Web 2.0”.

That’s because of the kind of work people are doing now, which co-founder Sam Schillace explained to NPR recently, is –

“Lightweight, high-velocity and very connected.”

Or did he mean the people behind it are lightweight, high-velocity and very connected?

To be fair, Schillace is an experienced developer who created what later became Claris Home Page, before going on to lead teams at Intuit and Macromedia. And Schillace correctly denies what the headlines writers want to believe today – that Writely is a replacement for ‘fat client’ word processors.

But these are bubble days, and it’s discordant to hear a rational explanation – but one comes from Joe Wilcox at Jupiter Research. The Writely feature set is so poor, he points out, that Google bought the software solely to beef up its editing facilities in Gmail and Blogger.

Read more at El Reg.