Calder invites us to have a giggle, but really it’s not a bad list at all, and compared with the (cough) ‘futurists’ who have come and gone since, Calder and the participants did a good job. Alvin Toffler was repackaging these ideas, particularly mass amateurisation, many years later. As are thousands of Web 2.0 consultants today.

Read more at The Register

Is the email program dead? Did the whole world just migrate away from Hotmail over to Facebook when we weren’t looking? Does anyone else care?

Weirdly, the answer seems to be yes, yes, and no. Email has never gone away, and its advantages are unique: but the email client seems to be going the way of the Gopher.

Which is a bit odd when you consider how useful it still is. Nobody knows your email address unless you tell them, and messages are private by default. These are still the internet’s universal protocols for private communication, something Web 2.0 types only grudgingly admit exists.

The F/OSS alternative, Mozilla’s Thunderbird, was for ages a spartan and neglected poor relative of the browser. Versions 1.x and 2.x each seemed to be stalled for years. Thunderbird was also handicapped by inheriting a toxic legacy: its predecessor was on the receiving end of one of the most damning critiques of programming ever written.

Jamie Zawinski had written the original Netscape email client and returned to the scene after the program had been open sourced. He’s written several entertaining essays (sample: “my C version of this code was able to thread 10,000 messages in less than half a second on a low-end 90 MHz Pentium”) but Groupware bad is a good introduction.

The Thunderbird team seemed to want “the community” to sprout lots of add-ons, as they had with Firefox, then noticed that they hadn’t, and so finally decided to integrate the most-needed ones into the program itself. It was a good move, as Thunderbird is now actually quite good. It’s striving to add sophisticated features (virtual folders, fast search, tabs) but doesn’t seem to have got too bloated in doing so. The configuration UI remains clean and consistent.

We have to be honest – managing your own POP3 or IMAP accounts always was a bit of a minority pastime. Notes and Outlook ruled the roost on corporate PCs. Most of the rest of the world – normal people – only ever used a webmail service as a primary email account.

Facebook offers them a pretty straightforward upgrade – with the illusion of privacy, no spam, and a pretty easy to use address book. Just don’t tell Facebook investors that one day it might be as profitable as Hotmail… But for me and many of you I suspect, the choice of email client is something in which you make a bit of investment, and a careful decision.

And what a sorry landscape we have before us.

Mac users get a good email client bundled with the OS. It’s not the fastest, or the lightest, but it has improved steadily over the years, taking advantage of system improvements such as better threading and search. It’s now really, really good.

IMAP handling is excellent, which for me means both the client and server accurately reflect the state of your mailboxes, and do so efficiently, without excessive CPU or network load. Using Apple’s MobileMe you can take your filters with you to a new installation. It’s all so nice, I take it for granted. But that’s the only bright spot in a landscape marked by major players leaving the scene.

On Windows, it’s a bit pitiful. The choices of a good IMAP/POP client are diminishing. After rebranding Outlook Express as Vista Mail, Microsoft dropped the email client entirely from Windows 7. It’s now available as a free download, and renamed again, to Windows Live Mail. After 18 years of great service, Eudora [history] gave up the ghost in 2006, with its replacement a fork of the Thunderbird codebase, which we were promised would be Eudora-ised.

Back in the day, Eudora was a superb email program, with one of the most honest and amusing changelogs in the business. Shortly after that, the sole author behind the venerable Pegasus Mail decided to jack it in. (Only to change his mind – see below.)

Today, there’s the The Bat! – complete with inexplicable! exclamation! mark – which hails from Chisinau, in deepest Moldova. Where, presumably, bats deliver email (rather than winged horses).

The Bat! is very powerful indeed, a real power user’s program, allowing you to fine tune IMAP communications. It comes with powerful views and filters and a few unique hallmark features, such as built-in message encryption and configuration portability. There’s even a version that runs off a USB stick.

The drawback is that it can take all day to set up, it costs real money, and is as friendly as a Swiss Army Knife – enough to deter (perhaps unfairly) the more casual user. For anyone who ever telnet-ed to port 110 and typed list, the Bat! is for you.

And then there’s the prodigal email client, Pegasus. For many readers I’ll bet this Netware client that predates dial-up accounts was your choice throughout the 1990s: Pegasus Mail. And it’s still here, one man’s burden. Two years ago the programmer responsible, David Harris, chucked it all in, only to change his mind. Verity Stob wrote a splendid account of Pegasus Mail that’s better than any review. Well, Pegasus still survives. If you’ve used it this long, why move away?

And er, that’s almost it. So Mac users are deliriously happy with Apple Mail, Linux users seem happy enough with Thunderbird (or a self-built version of pine or elm, perhaps) and Windows users struggle on with a bizarre choice of specialist apps, or Outlook Express/Vista Mail/Windows Live Mail.

Opera Mail – surprisingly unsung

That leaves one of the best kept secrets on all three platforms Windows, Mac and Linux uncovered. Meeting Opera founder and chairman Jon von Tetzchner last week I nagged him, as I almost always do, about the secret Opera mail project. Opera has been hiding an amazing email program for seven years now, in a place nobody would think of looking: in a web browser. Was now the time to spring this on an unsuspecting world?

I’m still not sure. In fact, I’m entirely certain Opera isn’t going to start promoting Mail all of a sudden. But you can’t accuse von Tetzchner of not eating his own dog food: he said he personally has between 20GB and 30GB of email in his own Opera Mail email archive, going back over 15 years. He agrees the landscape is pretty dire now: “Email clients have gone away.”

But making it a mass market product was a question of resources, and right now Opera’s desktop browser team is going pell mell in a Javascript benchmark war with Google. In mitigation he pointed out that Thunderbird had been neglected for years too, even though the functionality was there, inherited from the old Netscape.

It’s not as if Opera has been completely neglecting Opera Mail. In version 9.5, it got a significant overhaul which saw the back end revamped, resulting in much improved performance: there are no stutters even with multiple accounts, or multi-gigabyte mail archives.

(As an aside, Vista and Vista 2010 Windows 7 seem to have this annoying habit of turning a stuttering, but still functioning program into a system-wide seizure – the screen dramatically fades into white then back again, the conventional effect Hollywood uses when the main character is hit over head by a wrench, and passes out. After a while a dialog box pops up asking you if you want to terminate the stuttering process – which was actually doing fine until Windows decided to “help”. I found that all these email programs – Live Mail – the Bat, occasionally even Thunderbird – suffered from nanny Microsoft sizing them up for a premature kill. But I digress.)

What confounds newcomers to Opera Mail is its database-like approach. At first, you never quite know what’s going on. Initial versions (back when Opera called it M2) eschewed folders completely. The program replaced the mailbox hierarchy view with “live views” and a Bayesian engine you were expected to train, giving you more views onto the mail.

The real-time search worked (and still works) incredibly well. But I’m not alone in feeling disconcerted by the lack of mail folders, and was relieved when Opera made a concession recently to view by IMAP folder. Couldn’t Opera break newcomers in gently?

Jon von Tetzchner’s answer was a polite version of “well, boo hoo”. He said he couldn’t imagine anything more tedious than filing away 300 emails every day into boxes. But that’s not the point, I thought, if you’ve already invested in mailboxes and IMAP folders.

Here’s an example. A mail rule will divert your mailing lists (for example) into a mailbox. You then pick out the ones you keep into another mailbox, and trash the rest. The way to do this in Opera Mail is that it will automatically create a view for a mailing list, and a filter will cut down the one with stars or a particular tag. It’s much easier if you start the Opera way, but the Opera way is not much help if you’ve already got mailboxes.

Opera Mail is probably just one or two UI features short of capturing refugees from Outlook, Eudora and the rest. Chairman Jon mentioned that somebody had even skinned Opera to make the Mail program look like a fully independent email application, a tribute to how radically you can customise Opera (I haven’t been able to find the skin in question to corroborate this). [Update: it's called Hugin and you can find it here. Thanks to Jeroen for the link] Labels could be easier to customise (it’s quite a manual effort now).

These are unlikely to come from the ‘community’, because Opera doesn’t work that way. But for now, alas, this amazing software remains quite forbidding. I modestly suggest that if users were able to pay a small fee for an “Opera Mail Edition”, in the knowledge that development was active, and features people needed were being added, then wonderful things could happen.

But paying for software development? Whatever next?

Apollo space program as a triumph of power and industrial might. The superpowers’ space programs were, of course, political and chauvinistic, designed to showcase national wealth. But there’s a better way of looking at the program, Dennis Wingo reminded me recently. Masses of money helped put man on the Moon of course, but the Moon program is really a tale of engineering improvisation and human organisation.

Space expert and entrepreneur Dennis Wingo put the first webserver – an Apple Mac – in orbit, for just $7m, and has helped piece together a lot of historical material that NASA didn’t appreciate at the time – and forgot about, or wiped. There is one piece of kit in particular that encapsulates two stories: NASA’s negligence, and the quite amazing improvisation of the engineers. It’s the Lunar Orbiter, which mapped the moon’s surface prior to manned descent. Wingo painstakingly recovered and restored much of the imagery it took.

To give us an idea of how much Apollo owed to seat-of-the-pants ingenuity, it’s worth remembering that the story of the Orbiter begins in 1961 – the year of the first human orbit of the Earth by Yuri Gagarin. The space pioneers were seeing a high death rate from test subjects – dogs (the USSR) and chimps (the USA), the latter proving to be a duff move – the chimps panicked in the claustrophobic conditions.

The US program lagged far behind the Soviets’, and NASA’s early attempts to keep up had become a national joke. The Ranger had been the first project to photograph the moon, with the modest ambition of crashing a probe onto the surface. But of the first six Rangers, two failed to leave the Earth’s orbit, one failed en route, two missed the Moon completely, and although the sixth reached the target, its cameras failed.

Yet by 1964, much of the technology that eventually put man on the Moon had been already designed and built. The colossal Apollo expenditures were on the physical implementation of the program, including the many test flights. By 1965, the Apollo Lunar Excursion Module (LEM) was already being prepared as a long-term shelter and accommodation unit. And as Wingo points out, it was really down to 400 engineers – a fraction of what Google devotes to inserting advertisements into web pages – being given the freedom to put Heath Robinson designs into practice.

The Lunar Orbiter astonishes even today. It had to take pictures, scan and develop the film on board, and broadcast it successfully back to earth. Naturally, the orbiter had to provide its own power, orient itself without intervention from ground control, and maintain precise temperature conditions and air pressure for the film processing, and protect itself from solar radiation and cosmic rays – all within severe size and weight constraints. This was far beyond the capabilities of the newest spy satellites, which back then returned the film to earth in a canister, retrieved by a specially kitted-out plane. The Orbiter challenge was the Apollo challenge in miniature.
…Read more at The Register

It must be frustrating to sketch out a long-term technology roadmap in great depth, and see it come to fruition… only to goof on your own execution. But to do so repeatedly – as Nokia has – points to something seriously wrong.

Nokia spent more than a decade preparing for Tuesday this week, when it finally launched its own worldwide, all-phones application store. It correctly anticipated a software market for smartphones back in the mid-1990s, when it was choosing the technology to fulfill this vision.

That was just one of the bets that came good. Leafing through old copies of WiReD magazine from the dot.com era, filled with gushing praise for Enron, Global Crossing, and er, Zippies, I was struck by the quality of the foresight in a cover feature about Nokia. (Have a look for yourself.) WAP didn’t work out, but I was struck by particularly Leningrad Cowboy Mato Valtonen’s assessment that “mobile is the Internet with billing built in”.

And so Nokia has been encouraging users to download applications for users. My ancient 6310i wants me to download applications. Every Nokia since has wanted me to, too. Seven years ago, the first Series 60 phone (the 7650) put the Apps client on the top level menu, next to Contacts and Messaging.

The problem is today, it’s Apple and BlackBerry who have the thriving third party smartphone software markets. For six months, punters have been bombarded with iPhone ads showing what you can do with third-party apps. And yes, it’s like Palm all over again, but they’re very effective. So if Apple’s store is the model, then what on earth is Ovi?

The launch was “an utter disaster” according to one blogger, or in a more measured assessment (from Ewan at All About Symbian), “rushed, early and not fit for public consumption”. Nokia accepts second-best from Ovi, which apart from Maps is second-best in every category, the company all but admitted recently. But the Ovi application store deserves a Z-grade.

Web services or bust

It’s now clear that it was simply too ambitious to roll out a store to so many territories and in particular, to so many device categories, in one Big Bang. The number of devices supported goes back six years – encompassing eight versions of Series 40 and three versions of S60.

We waited a couple of days until the server load eased up, and Bill Ray kicked the tyres. On older devices it was mostly a miss. The mobile clients I’ve tried are painfully slow, don’t have previews and can’t distinguish between trialware and zero-priced applications. They either bill you in a foreign currency or simply drop you down a dead end.

The web version is even worse: try navigating through pages in Firefox, or try changing your default device in the preferences. The result is that every attempt to actually get applications is thwarted. Still, the pages fade in and out, in a very Web 2.0-style fashion. And maybe that’s the clue.

Apple’s App Store requires iTunes or the native client. iTunes is a familiar place for anyone who’s shopped for songs, audiobooks or movies there. It’s fast and slick, there’s a preview for everything, and pricing is quite clear. You’re only asked for personal details when you reach the acquisition stage. You get the same experience on the iPhone/Touch native client.

There’s really no need for a web-based version of the Ovi store at all, and piping everything through the Nokia PC suite (or some Mac equivalent) would at least encourage people to try the exciting Nokia PC Suite add-ons, such as Nokia Map Manager and er… Nokia PC Suite Cleaner. Apparently that cleans up after earlier Nokia incompatibility cock-ups.

(This is an ominous sign of trouble ahead: like Palm designing its stylus dual-purpose, one of which is to make rebooting easier after a crash. It’s not something the user should ever see.)

But Nokia has arguably far more at stake here than Apple or RIM. Once you’ve spunked $8.1bn on a mapping software company – shouldn’t you want people to use the maps – and the potential upselling opportunity? Or are the maps just a hippy giveaway?

‘Strategy’ is stretching it a bit

We all know in hindsight Nokia that should have focussed on making the mobile and PC clients perfect, and limiting the number of devices at launch to a subset of those supported. Anything before S60 3rd edition didn’t really need to be there, and there’s a case for limiting to devices launched in the past 18 months, even though there are a lot of N73s and E61s out there.

Separating the excellent applications from chaff such as movie trailers and wallpaper might have helped. There are still a handful of good applications out there, despite diminishing interest in Symbian, the pick of which is the best mobile email client in the world, Profimail. (Measured in ease of use, features, and the fewest seconds it takes to achieve a given task – a formidable combo.)

But again that goes against the Web 2.0 ethos of “stick any old crap up there – and let the Hive Mind sort it out”. No thanks, I don’t want MOSH 2.0.

And as for games – it would be flattering Nokia to call the six year N-Gage adventure a “strategy”. Again, it saw the market early, but didn’t follow through. Every now and again the multi-billion dollar investment veers back into view, only to disappear again. Is it N-Gage or Ovi Gaming? The few titles that are out there aren’t too bad, but again Nokia’s delivery strategy makes them hard to obtain. Meanwhile you can’t escape people playing games on their iPhones, or iPod Touches.
Operation Rescue Nokia

The market could benefit from a healthy Nokia software market, so here are some suggestions. There’s a valuable lesson to be learned. In business as in war, you make the most of your assets while trying to minimise your weaknesses. Nokia’s Ovi Store does the opposite: it emphasises the complexity and lack of focus at the company, and its disorganisation. If your first and only experience of Nokia was Ovi, you would never believe the company could ship 50 products into 120 markets with military efficiency.

Firstly, Nokia should focus on people’s needs – and applications that make the phone useful and fun – and not building up a “a portfolio of web services”. It’s already invested heavily in Maps and games – just make them easy to try and buy.

Ovi means “door” in Finnish

Secondly, the Ovi brand has made no impact on phone users at all. There’s no shame in abandoning confusing or invisible brands. Confine Ovi to mean boring, management services like backups, or data transfer, or services discovery. These shouldn’t be underestimated; they should give users security and peace of mind.

Thirdly, the vast majority of users want to do a few tasks simply – take note of the Magners TV ad which now singles out flash smartphones that are impossible to use. Nokia has inched towards better usability with the E71 and the 5800, but this needs to be a company-wide goal. Showing photos on the family TV, sharing photos with a small group – all much more useful than the 2.0 guff.

And finally, the managers responsible for putting together the Ovi Store should be put on Nokia’s naughty step – and left there for the Finnish winter.

The site was down when bloggers checked in yesterday.

More importantly, the latest redesign is a tacit admission that it’s given up on its original mission – indexing the world’s weblogs. Technorati now claims to present “zillions of photos, videos, blogs and more”, and rather apologetically adds the rejoinder: “Some of them have to be good.”

No. Why?

In practice, Technorati now returns only a tiny number of blogs – and prefers to offer thumbnails of digital images already tagged with a keyword. A technical challenge that does not exactly require the algorithmic prowess of a Donald Knuth.

Technorati, the comically inept search engine, has redesigned itself again – knocking itself out in the process.

The site was down when bloggers checked in yesterday.

More importantly, the latest redesign is a tacit admission that it’s given up on its original mission – indexing the world’s weblogs. Technorati now claims to present “zillions of photos, videos, blogs and more”, and rather apologetically adds the rejoinder: “Some of them have to be good.”

No. Why?

In practice, Technorati now returns only a tiny number of blogs – and prefers to offer thumbnails of digital images already tagged with a keyword. A technical challenge that does not exactly require the algorithmic prowess of a Donald Knuth.

So what was always a lousy blog search tool is now little more than a lousy image search tool – this is not going to worry Yahoo! or Google.

Call it a strategic retreat. The site has fought heroically to stem the Rise of the Machines, exemplified by tools like this, but lost. Who would have guessed?

Well, not the journalist pals of founder Dave Sifry, and A-list bloggers who gave Technorati oodles of back-scratching press when it launched in 2003. Hacks were as keen as Sifry to evangelise blogging, and instantly conferred guru status on him; here was a man with the numbers that mattered! Reports were invariably too kind to mention that Technorati rarely worked well, and often didn’t work at all.

The moral of the story is hard to miss. Maybe ideological evangelism and engineering don’t really mix. Evangelism and honest reporting certainly don’t.

“I quit when Nicholas told me — and not just me — that learning was never part of the mission. The mission was, in his mind, always getting as many laptops as possible out there; to say anything about learning would be presumptuous, and so he doesn’t want OLPC to have a software team, a hardware team, or a deployment team going forward,” writes Krstić.

“Nicholas’ new OLPC is dropping those pesky education goals from the mission and turning itself into a 50-person nonprofit laptop manufacturer, competing with Lenovo, Dell, Apple, Asus, HP and Intel on their home turf, and by using the one strategy we know doesn’t work.”

Ouch.

Negroponte’s decision to embrace Windows has seen top-level resignations from the OLPC project. CTO Mary Lou Jepsen left in January, and former software chief and president Walter Bender departed in April. Krstić resigned in March.

OLPC is a poster child for free software innovation, with critics acknowledging value in its advances in mesh networking and the radical task-based UI Sugar. But the F/OSS ideals are now being jetissoned, writes Krstić, along with the crown jewels:

“In reality, Nicholas wants to ship plain XP desktops. He’s told me so. That he might possibly fund a Sugar effort to the side and pay lip service to the notion of its ‘availability’ as an option to purchasing countries is at best a tepid effort to avert a PR disaster.”

Not everyone thinks Sugar is a successful UI – judge for yourself in our extensive hands-on.

The diminishing reliability of these devices isn’t unique to Nokia, and it may be a consequence of having so many products, in so many markets, all at once. But engineers deep in Nokia we’ve spoken with describe how they grew weary at being conditioned only to fix a proportion of bugs. It offends an engineer’s pride to release a flawed product, but this became a way of life. There was simply too much to do.

As for usability, the company which pioneered an interface that helped popularize the digital mobile phone – NaviKey™ – now falls far behind much of the competition. With feature phones, Nokia’s interface has failed to evolve with the tactile and graceful interface of Sony Ericsson, for example.

At the high end, the story is far worse. The S60 UI initially provided Nokia with a clever bridge to the future, but it looks pedantic and cumbersome besides Motorola’s MotoRizr 8, let alone Apple’s iPhone. Nokia answers the perennial S60 user’s question, “Why so many clicks?” by adding extra hardware buttons, such as the slow and inflexible “Multimedia” key. S60 is incredibly poorly written in parts, but Samsung has demonstrated that it doesn’t have to be sluggish, by using its own chip to speed up its first European S60 phone. Yet Nokia has ensured most of its smartphone users have a substandard experience, by starving the devices of sufficient memory or fast enough processors.

It doesn’t augur well that the company’s skill at exploiting the emerging markets owes little to its recent R&D work: it’s succeeded with low cost models in China by dusting off older, more reliable, and easier-to-use technologies. In other words, it’s living off past glories, rather than looking to the future.

In fact, Nokia now appears to quite relish the complexity of its devices. Quite bizarrely, a company which had no need for an inferiority complex appears to have acquired one.

For some years now, phones have really been computers in disguise, but Nokia has always stressed their utility as appliances. Today it proudly boasts: “This is what computers have become”, and insists its sales staff call the phones “Multimedia Computers”.

But computers are everything mobiles are not and shouldn’t be: cumbersome to carry with you, complicated and unreliable. It’s like Mercedes branding themselves as “The Lada you always wanted.”

Perhaps by bringing bloated Widgets to mobiles, Nokia now feels validated, that it has a “real computer”. But one must careful what one wishes for.

Yesterday’s network, tomorrow

Today’s announcement will do little to raise the morale of the bedrock third-party software developers who Nokia needs the most: the C++ and Java developers required for infrastructure, middleware and mission-critical applications.

Even the most loyal Nokia developers whisper that Microsoft makes a less predatory platform host than the 800lb Elk. How can this be so? Well, because Nokia receives API requests from the community, it has a good inkling of what everyone is working on. The company can deprecate an API on a whim, if it wishes to enter that business sector. And if that API is crucial to your business, then you need to be looking for a new job. Turf wars between Nokia’s Enterprise division and its Multimedia division leave these developers in a Kafka-esque situation: duplicating requests for the same feature if it’s going to be deployed on an E series phone or an N series phone. Developers need to plough through that bureaucracy not once, but twice.

Now imagine how today’s news will be received. It becomes apparent that Nokia hope to attract new developers by making its phones more attractive to wiki-fiddlers and script kiddies. Over to Ovum’s Cripps eulogising web widgets:

“Web developers are several orders of magnitude more common [our emphasis] than the C or C++ programmers targeted by S60’s native environment and also considerably more numerous [ditto] than Java ME programmers. Attracting these developers will be a key factor in the continued healthy growth of the mobile applications ecosystem.”

Actually, the health of the mobile applications ‘ecosystem’ is entirely dependent on how the carriers choose to open up their networks. (For example: it’s a given that tomorrow’s mobile networks are IP-based, but it’s far from certain that the network operators will produce a subset of APIs.)

The implication that more developers means better software has been demolished many times: Fred Brooks Mythical Man Month being the best example, and it hardly needs to be restated here. What does, however, is the Silicon Valley myth that the lone bedroom developer, who knows little more than JavaScript needs any particular coddling.

The exact opposite is true.

Nokia’s future

Now this is but a selective snapshot of Nokia’s business strategy. We could have been more positive, and praised the company’s singular, and admirable determination to bring VoIP to its business handsets, against the wishes of its biggest customers. Then again, we could have mentioned the new CEO’s nutty prediction that mobile TV will go mainstream this year – ignoring 25 years of mobile TV flops.

It’s just that Widgets sum up so much of what Nokia today is failing to get right. The company is floundering if it thinks Web 2.0 is its salvation.

The irony is that the open internet is moving to an architecture that much more closely resembles the vertical network that Nokia wants to distance itself from. Increasingly, the packet switched network is acquiring the characteristics of the circuit switched networks: with intelligence built into the network itself. The internet needs this to grow, and handle more sophisticated applications.

These days, even the primary author of the fundamental text defining the old, open internet ( “The End to End Principle in Network Design”) David Clark, says we need to rip up the infrastructure, and start again

“We are at an inflection point, a revolution point – we might just be at the point where the utility of the internet stalls – and perhaps turns downward,” he warns.

Which way will Nokia be facing? Or will it be fidgeting with those Widgets?

So there we were. In a room devoted to Engineering, the man voted the Greatest Living Briton had exploded in front of me.

Sir Tim Berners Lee, co-inventor of the World Wide Web, was at Southampton University to deliver an inaugural lecture for School of Electronics and Computer Science, and promote his latest initiative.

There’s a whole new science out there waiting to be explored, called “Web Science”, and he was here to explain how. The Web Science Initiative was “an umbrella, with lots of projects” around the world, he said.

Flanked by the great and the good, Professors Nigel Shadbolt, vice president of the British Computer Society, and Wendy Hall, and James Hendler. Sir Tim said he hoped this would set the agenda for years to come.

This new science comes with some grand claims attached. Shadbolt said he hoped the web would attract a new kind of undergraduate to computer science departments, who presumably had been bored by all that old-fashioned science and engineering.

Shadbolt implied we could learn a lot about humanity from looking at the Interweb.

“What actually happens on the web when people participate is all psychology: it’s more accurate to think of the web as humanity connected,” he said.

“It’s people: things get published by people, the blogs are made by people, the links between them that Google follows are made by people.”

Hendler jumped in:

“I was the external reader for a paper for somebody here at ECS. The first line of her thesis is ‘This Document Is About People’. And I put down explanation marks and “hurrays!” That’s a student who’s beginning to understand web science!”

(So that’s how you get on in Computing, dear readers).

But this was all a bit much for your reporter.

“The assumption behind everything you’ve said is that this research will create some kind of knowledge,” I asked. “The other assumption is that the links you’ll be examining to provide this knowledge are generated by humans.

“Now when I search for a term on Google Blogsearch or Technorati, two thirds of those results are robots. People at Google tell me between twenty per cent and a third of the index is junk – Google doesn’t know which third.”

“So, er… what’s your research going to be worth?”

“That’s one of the good questions,” said Professor Hall.

Sir Tim countered, and it’s worth quoting in full.

Do you remember when the web came out, the first search engines before Google were famously terrible. They were famous for producing lists of junky answers. Originally when the web was small there was no problem finding things because it was a list of websites, and each one had a picture of what was on it. When it got bigger you got the problem of where to find something and that problem got more and more acute, and then…

Somebody, in fact Google, is nice web science example. They thought we can use this vector machine technology and it could solve the “where to find stuff” problem. So now the result is much more effective search engines. So yes, OK.

We silently wondered where this would lead. We didn’t have to wait too much longer: the answer had almost arrived.

So one cycle further on… the spammers have gone to considerable amount of trouble to build the “fake web”. I dont know what the proportion is but as you say, there’s domain names all linked in to each other all generated by computer, all full OK, junk. Full of crap. So. There’ll be another cycle. OK

Er, we thought. Is that it?

Not quite.

So when you see what’s happening, OK, there are a lot of spammers out there. It’s like spam for email, you know. Suddenly spammers deluged us but email was designed for a world where everybody was friendly – and spam happens where people are motivated by pure greed and not part of a friendly club. You can design email systems around that; the email system is being converted so it works in this environment. I’m sure web search engines so they work in this environment…

But, er how? Sir Tim’s historical narrative had a few flaws in it. It seemed negligent not to point this out to the GLB.

” I remember AltaVista worked wonderfully – until it was gamed. Google was wonderful it was gamed – entropy keeps returning. We don’t seem to be making much progress. Are you saying we will somehow fix it – magic will happen?” asked your reporter.

There were howls of distaste from the panel. Wendy Hall’s face was thunderous – and I realized I had not merely made an unwelcome expression at a revivalist meeting – I’d farted in the church.

Hendler leapt in to take the discussion off topic:

We’re not making much progress until you do the maths. We’re now doing as well with two or three more zeros at the end of how many pages are out there. So in a sense you have to run as fast as you can to stand in the same place. What a lot of the researchers Google look at is exactly how to keep scaling across.

That’s a good answer – to a different question. It wasn’t scale, but system integrity that we were talking about.

The GLB stuck to his account:

But AltaVista never gave good results. It never did this Eigenvector calculation. It never found the representative page for a given community. It would find pages which contained the page a lot of times it wouldn’t do any link analysis. The older search engines before Eigenvector machine systems came out were just never as good. And even, still, yes, they’re being spammed, [inaudible] but it hasn’t made the search engines less usable they’re still much more usable than the previous generation

But Google has a real problem, here, and it employs some of the cleverest people in the world.

” You’ll have to figure out something Google can’t figure out if your research is going to be worth anything,” El Reg asked.

“How will you do that, exactly?”

Sir Tim replied:

It’s about much more than people defending against abusing the system. There are a huge number of opportunities that will make the industry work more efficiently.

But he was getting snidey:

In your life, do you feel that spoofing of search engines is the main thing, the one that bugs you?

So, er .. the cornerstone of the New Economy wasn’t a concern. (I’ll explain why this answer was somewhat short of being satisfactory, but let’s continue with what happened next.)

Scenting blood, another reporter pursued the enquiry:

“So how are you going to stop the Semantic Web being poisoned?”

TBL, the GLB, replied:

Well, everybody who’s building the semantic web pretty much that I know are building systems take data from lots of places, but take data with an awareness of where those places are. So for example, suppose you’re getting Geotags and the OS runs a service, lots of people in this country might trust the OS to say this point has a church with a spire – other people might say it’s a great church to go to, other people might say it’s a heathen church to go to… those are the other sources of data…

There was no let up from the press:

“But that was the basis for Google, and Google got poisoned… ”

Shadbolt and Hendler stepped in to shield Sir Tim, but he was seething at the impertinence:

I remember a conference, we were discussing the Semantic Web, and someone asked what do you think is the worst thing that can happen and all the pencils come out. I know you two have been asking about “Woargh – I know the one about… what about the bad guys? Won’t we be phished” There’s a temptation to give readers about all the terrible things out there OK, and all the ways the web can become less usable.

At this point, your reporter wanted to remind Sir Tim that of all the problems the web has, a hostile press is not one of them. In fact, you can’t pick up a newspaper or magazine without reading about how it’s ushering in a New Age of Enlightenment. Time magazine gave “Person Of The Year” to every web user in America – or at least every one who looked at the mirror Time placed on its front cover.

He continued, cryptically:

Yes you’ll find a bank that’s less usable – … I’ve never been phished.

So the Greatest Living Briton has never been phished, which is a relief. His answer to the Semantic Web didn’t inspire much confidence for the rest of us: it would be used within the firewall, amongst trusted groups, “areas where one is much less worrying about the bad guys”.

Here’s what I found both disquieting and depressing from the GBL.

I’ve asked similar questions to engineers in every field. Without exception, all have thought deeply about the consequences of their original design decisions, and express quite specific solutions. These are often quite radical rewrites – throwing out many of the assumptions they first made.

But not with the Web. It’s a place to marvel and hope, and like Candide, hope for the best.

If you’re not already evangelical, you probably don’t have a part to play in the “new science” of the Web.

Read the mailbag here

The Japanese, fearing their industrial base would collapse without a response to this omniscient technology, poured hundreds of millions of dollars into their own AI project, called Fifth Generation. They may as well have buried the money in the Pacific Ocean. Two decades later there are no intelligent robots, and “intelligent” computers are a pipe-dream.

(It was an academic coup for MIT’s Professor Marvin Minsky, a fixture on the AI slots. Minsky’s own preferred, linguistics-based approach to AI, symbolic AI, triumphed in the grants lotteries over an approach which preferred to investigate and mimic the neural functions of the brain. Minsky’s non-stop publicity campaign helped ensure his AI lab at MIT was well-rewarded while neural networks starved.)

For the past week reports have again confidently predicted intelligent computers are just around the corner. Rollo Carpenter, whose chatbot Joan won an annual AI prize for creating software that most resembles a human, predicts that computers will pass the ‘Turing Test’ by 2016. In this test, computer software fools a human interrogator by passing off as a human.

(You can spot the flaw already: to sound human isn’t a sign of intelligence. And what a pity it is that Turing is remembered more for his muddle-headed metaphysics than for his landmark work in building computational machines. It’s a bit like lauding Einstein for opposition to the theory of plate tectonics, rather than his work on relativity, or remembering Newton for his alchemy, not his theory of gravity).

But let’s have a look. A moment’s glance at the conversation of Joan, or George, is enough to show us there is no intelligence here.

Here’s Joan, cited at the MIT’s Technology Review – a new Wired style glossy:

Judge: So what you’re saying is that you are not just human but THE smartest human?

Joan: Yes, why, what did you think I was saying?

Judge: So what is two plus two?

Joan: Four.

Judge: Is it possible for two plus two to not equal four?

Joan: I don’t know. Do you think it’s possible?

Judge: How many degrees in a triangle?

Joan: 180 degrees.

Judge: Is it possible for more than 180 degrees to be in a triangle?

Joan: Probably not. You can change your opinion, but not your constitution.

Judge: It’s not a question of opinion, it’s a matter of physics and maths.

Joan: A five-ounce bird could not carry a one-pound coconut.

The first thing you’ll notice is that if this is a human, it’s a pretty shallow yoof with a zero attention span. Now that’s a surprise. It’s said that pets eventually resemble their owners. Do AI bots resemble their programmers? (a question answered here).

Joan is simply a database of feints and shimmies deployed to change the subject and confuse the questioner. Occasionally Joan will ask the interrogator if they are a robot, or chide them for being stupid. These are all pre-programmed rhetorical tricks. They may bore or bamboozle an interrogator, but this is no indication of intelligence.

That’s no surprise, when we learn Joan really is a database of conversational snippets – five million lines of them. This is the same technique deployed by Eliza, Joseph Weizenbaum’s elementary software parser written in the 1960s, and bundled with EMACS.

Weizenbaum was horrified by the fascinated reaction to Eliza, which was devised as a tongue-in-cheek endeavor, and a subsequent epiphany led him to devote much of the rest of his life to urging scientists to cultivate a sense of social responsibility.

Look! A talking dummy.

The pop media’s fascination with “intelligent” computers, especially of the talking variety, shouldn’t surprise us. It only mirrors our own anthropomorphic tendencies – to give things far more human characteristics than they really have. Whether it’s voices coming out of the static, or faces in wardrobes or cheese toasties. The inanimate Golem brought to life, through either human or divine intervention (or both) is a myth that’s taken many forms over the years.

As a result, AI has attracted far more than its fair share of flakes, phoneys and the outright naive over the years.

The cost is hard to calculate. There’s an obvious resource issue, an opportunity cost, when fatuous endeavors are allowed to crowd out more pressing computing problems. Of all the woes we have with today’s computer systems, their inability to hold a conversation must be one of the least important. We’d rather see systems that don’t fail, that never lose data, and photographs that we know we’ll be able to see in thirty years’ time. Today’s digital data is designed to be lost, it seems; imagine a generation with no family album, because the ink has bled and the formats can’t be read. It isn’t science fiction so much as a probable outcome.

And even if an “intelligent” computer was to be devised, it would help us a lot less than we might imagine. The world isn’t short of intelligence. It’s just very rarely applied to pressing problems.

Fortunately, serious researchers may yet be able to shake off the curse of AI. In Manchester, where Turing made his flawed philosophical assumption that set academic AI haring down the wrong path for forty years.

At the University of Manchester Steve Furber, father of the ARM chip, is helping build a “brain box”, modelled from biological systems. The research will help design more fault tolerant computing systems. There isn’t a hint of talking (or dancing) robots in sight.

So to get better computers, maybe all we needed to forget about them being conscious, or intelligent.

Answers service AQA is two years old this summer, and finds itself in the happy position of not only being profitable, but something of a social phenomenon in its home country.

A book based on the service, The End Of The Question Mark is due to be published in October, drawing the questions Britons ask, and the answers AQA gives them. Not bad for a company that still has only nine full time employees.

What AQA allows you to do is text in a question and receive an answer for a quid. This might strike US readers as expensive: it’s nearly two dollars (or four days of the San Francisco Chronicle) for a few lines of text at today’s exchange rate. But Britons love texting, and arguing, and AQA’s combination of canny marketing and the quirky charm of AQA’s answers have proved to be a hit.

But where AQA particularly interests us is how its success poses a challenge to a lot of the Californian-inspired orthodoxy about search engines, and Silicon Valley’s latest hype of fetishising “amateur” content.

These are strange times indeed when an AOL web executive must defend his decision to pay former volunteers real money for their labours. Actually pay them – so they can help feed their families? The horror of it!

Founder Colly Myers had plenty to say on this, in typically no-nonsense style, when we caught up with him recently.

AQA served its 3 millionth answer recently, notching up the last million in four months. The previous million took seven months, and the first million took 19 months, which gives some indication of its growth ramp.

AQA’s owner IssueBits has been profitable since last October, says Myers, and he thinks the market is young and there’s plenty of opportunity to grow. AQA doesn’t have the field to itself – 82ask also caters to the curious texter – but it is in pole position.

Myers seems particularly proud of the infrastructure: AQA uses around 500 researchers to answer double the volume of queries it did before (the actual composition of the research staff varies, as they drop in and out of work).

The internet’s “search business” has had saturation coverage in the last couple of years, typically in hyperbolic terms. The subtitle of one recent book suggests that the company “rewrote the rules of business and transformed our culture” – that’s a book that Google liked so much, it bought hundreds of copies for its staff. But very little of the coverage has highlighted the philosophical and practical flaws of the web search business.

AQA’s researchers don’t use Google (or Wikipedia, which we’ll come to). And internet search is a business that may already have seen its best days, Myers suggests.

There are several reasons to support this view, and in some cases they’re interrelated. One is that Google is finding it increasingly difficult to maintain its index. It’s ignored the second law of thermodynamics.

“It’s a well known aspect of man and machine systems. Complex systems with no control fall over. Every example of it you can think of falls apart. With databases, data that isn’t pruned becomes overgrown. Entropy sets in when complexity gets out of control.

“A lot of the search engines’ index is junk, and although they have a lot of clever people, they can’t prune it manually. And they have a lot of powerful technology too, but they just can’t stop it.

“We’re looking at the prospect of the end of the growth of search. Like Microsoft, search won’t be going away anytime soon, but the search engines’ power will weaken for several reasons.”

The difference with AQA, he suggests, is because it’s paid for its answers, and not for advertising. So keeping up the quality of the inhouse database is vitally important as it affects the quality of the service.

“So much so,” he says, “that it is worth spending money on improving the data in the database. There is not the same incentive for search engines.”

“At AQA we don’t use Google. We don’t use Wikipedia. We have good researchers who know where to go – to the primary sources.”

Nor does Myers see gimmicks solving Google’s massive accretion of what’s called “Goobage”.

“People come up with ideas like tags, metadata, but am I going to take the trouble to do that on every document I create? Are ordinary people? Google should pay people for adding the metadata.

“That’s very valuable information to Google. Why should people donate their time for free to help Google out of a hole?”, he asks.

Because they think they’re helping God? It’s a fair point. When simple software can create 100 new weblogs for your junk content in 24 minutes, entropy is going to out pace the most devout, Google-worshipping tagger (those spam blogs use tags too, of course).

There are more reasons that the web is in a big heap of trouble when it comes to answers. Once you’ve got the MySpace habit, you rarely leave the site. Myers explains:

“The 18 to 24 age group today is not using search so much. Why? People know where to go. They have MySpace – they know where the community is. They know where YouTube is. People need search engines less and less, because they don’t need to use search as a portal.”

This hunch seems to be confirmed out by two pieces of evidence.

MySpace is the number one destination in the US, but MySpace mail, the company’s European VP said recently, is number four. So people search from where they already are, within these vertically integrated sites, and there’s less value to having Google as your home page – or leaving MySpace.

(Perhaps sensing which way the wind is blowing, Google recently paid almost a billion dollars for an advertising deal with MySpace).

Another clue is this traffic report from HitWise, which saw Google (in green) gain a lot of traffic the day MySpace (in red) crashed.

But aren’t teenagers fickle, we wondered? So they can’t bet on commanding an audience?

“If they don’t deliver value, no – of course not,” he agrees. “The community provides the validation and without the community there’s nothing. Consumers have an tremendous amount of power now and can switch very easily, so unless you add value they’ll go somewhere else.”

“But we could start to see browsers with the bookmarks for MySpace and YouTube preloaded, or some other form of bookmarks. That’s very powerful.”

There’s a third powerful reason why web search isn’t as good as it was.

“The 80:20 rule also applies here,” he says. “Twenty per cent of the documents provide most of the value that people want. The other 80 per cent add less value. Now what the search engines are doing is indexing stuff that increasingly adds less value. Meanwhile, more and more data is disappearing into the ‘dark web’. And by nature a dynamic web service is not ‘indexable’.”

The thought struck us, round about here, that the idea of the internet as a great leveller, bringing information to the masses, might just be about 100 per cent wrong. Or at least highly overstated.

As Google’s fate is increasingly linked to amateur or highly dubious content like Wikipedia – and the many sites who mirror or scrape its content – the contrast between high quality research and low quality web search seems ever more apparent. The popular fetishisation of amateur web material is a peculiar belief as it needs to suppose that paid information doesn’t exist, and isn’t better. Nonetheless, aren’t we seeing the emergence of two worlds of information – one low grade, amateur, and beset by entropy: the open web – and the other of high quality? Of course those of us with membership cards are laughing all the way to our libraries’ expensive database collections – and we can afford to pay for quality.

Spare a thought for the rest of the world, though. Access to the web begins to look less like a blessing, and more like a curse.

Which brings us to the vexatious subject of Wikipedia.

“For me Wikipedia is a classic example where you appear to solve one problem, but don’t really do so, because you create another problem. In the case of Wikipedia it created a community to generate information but failed to find a way to authenticate that information. So close, but still so far.

“I am doubtful that community based systems will be the basis of sustainable solutions. I suspect that a commercial imperative is necessary. It’s the same with all these volunteer systems, they’re idealistic, but no one comes up with a way of paying people.”

Now we were extremely skeptical that AQA didn’t drink from the Wikipedia well. But in a couple of tests, pitting it against rival 82Ask, AQA drew from more original source material, while 82Ask hit the Wikipedia. It was still web-based, but made for a better answer. AQA was much faster, too.

AQA’s success arrives at a time when there’s a lot of utopian talk about how new production “models” may be being enabled by technology networks – often using home-based workers offering their services for free. AQA also taps into a global, networked workers – but pays them.

Much of this is misplaced, and quite fanciful, the result of academics who’ve arrived at a “model” simply casting around for evidence that looks like it might fit their favourite shape, such as Yochai Benkler. When academics fasten onto a favourite shape, no amount of reason can persuade them otherwise.

But there’s also a darker side. Amidst a lot of hooey about “collective intelligence”, there’s the thought that we’re enabling virtual sweatshops. Amazon.com’s Mechanical Turk is one example of this kind of business – bringing together employers who have piece work, often mundane and repetitive, with people prepared to do it. It’s been called a virtual sweatshop.

Colly Myers rejects the idea that AQA may be depressing wages.

“What we do at AQA is bringing people into the workforce – and in many cases they wouldn’t be otherwise working. Students can work, but two thirds of our researchers can’t – they’re at home. We can do this because we have a business perspective, we have revenue.

they’re at home. We can do this because we have a business perspective, we have revenue.

“So the technology allows us to do so very efficiently. We’re much more efficient than a call centre or a support centre. And they’re a lot happier because they work from home and can choose when they work – there’s unimaginable freedom,” he says.

He agrees there is at least one similarity between Mechanical Turk and AQA in that they both use humans to do tasks that are too difficult to be done by computers, both have a task that can be dispatched and completed over a network, both pay for the service, and neither schedules any resources.

But Mechanical Turk is a generic clearing house, which leads to its major flaw – lack of quality control.

“The very nature of this genericity leads to its key weakness in that it does not have adequate mechanisms to qualify its human resources to any specific task. It does try to define various types of qualifications but they are necessarily weak and fail the test as a true qualification for most non-trivia tasks,” he says.

“As a consqeuence, the quailty of their work against any task cannot be very high, and consequently a user of the Mechanial Turk system cannot afford to pay a great deal for the service.

In other words, Amazon can’t do the tests that AQA performs to qualify researchers, or the reviews the senior researchers do of their answers. And there’s no management structure to monitor and train people.

“Continuous improvement is vital since without continuous improvement any business is a dead duck over time,” says Myers. “I think that it is something close to a universal truth to say ‘that you get what you pay for’ in the cost vs quality debate.”

He rejects the idea that Mechanical Turk participants are in a “sweatshop”, because they can leave at any time.

Finally, the former Psion MD and Symbian first CEO remains as committed to data integrity as he always has been – he keeps paper copies of his contacts book under his bed he says, although he may have been being metaphorical.

We mentioned David Rosenthal’s description of engineers who share these values as essentially pessimistic in their approach. They need to factor in whether the bridge will fall down.

“Someone asked me what I was at a family gathering, recently. I said I’m a pessimist. They replied, no you’re not, you’re a cautious optimist.”

Psion, the company that invented the PDA, had a thing about never losing your data. That’s more than you can say for us. After an hour of recording our conversation for posterity on a Nokia, the phone simply looped round and deleted the recording. It wasn’t a Symbian Nokia.