Germany’s Federal Commissioner for Data Protection Peter Schaar says he’s "horrified" by the discovery.

"I am appalled… I call upon Google to delete previously unlawfully collected personal data on the wireless network immediately and stop the rides for Street View," according to German broadcaster ARD.

Spooks have long desired the ability to cross reference the Mac address of a user’s connection with their real identity and virtual identity, such as their Gmail or Facebook account.

Read more at The Register

Of course, out of the ten subject areas, the one labelled ‘online copyright infringement’ has attracted the most attention from their Lordships. Lord Mandelson made a number of modifications acknowledging these concerns this week – including some substantial changes to the processes. It’s the procedure rather than the principle that is vexing the Lords.

Nobody – not even those who support the Bill – is entirely happy with the procedures. Yet there is no great grassroots outpouring of opposition. While 500,000 people may have paid 79p in one week to register a protest vote for the Christmas Number One single, fewer than 500 have signed up to the Open Rights Group’s “Message to Mandelson” campaign – and some of those are supportive. We spotted one ‘Go Mandy’ from a major record label staffer and another urging his Lordship to bash the ‘freetards’.

Read more at The Register

“The Internet is becoming more and more widespread and will increasingly represent a scientific random sample of the population”
– Joi Ito

“Igor, to the machines – we have a sample”

One thing seems to have been forgotten following AOL’s careless, but quite magnificent data dump of the internet’s “hive mind” at play this week.

AOL’s assiduous documentation of the private thoughts of over 600,000 web searchers has certainly added some much needed sparkle to a public internet that of late, has been in dire need of a tonic. Now, internet users’ most private thoughts are revealed, in all their banality and creepiness, and we must count ourselves fortunate.

“AOL’s data sketch sometimes scary picture of personalities searching Net,” was the headline USA Today newspaper chose, but this barely conveyed the voyeuristic frisson, or glee we felt as the AOL database made its way across the net.

Nothing in recent months has made the net come alive quite like these queries, and it’s not hard to see why. Recently, the net has been drowning in banality. Billions of identical blogs – some human generated, some machine generated – spring up every day, with identical opinions to match the identical templates each blog hoster seems to provide. This outpouring of new recorded writing has been trumpeted as a new era in human expression. But the truth is, in practice, the consequence of all this is that it’s getting increasingly difficult to tell which is which. Human, or machine?

But let’s focus on an aspect lost in the “scandal”. The thing that everyone has overlooked is that this wasn’t an accidental or negligent data loss by AOL. The search query data was sincerely released in the name of science.

Boffins at AOL Labs published the data for boffins at similar “labs” to peruse.

That’s strange enough in itself, and it should make you yearn for white-coated frontiersmen of yore. Things have changed a bit since then and now.

Behold: the Mighty Atom
Fifty years ago, scientists did things like, oh… split the atom, and deduce the shape of the DNA double helix. Today, working off the hottest and freshest evidence available, scientists proclaim breakthroughs such as “People get more drunk at weekends”.

Once upon a time scientists set out to describe the unknown, and make it understood in
mechanical terms. But now, like a group of well meaning, but slightly simple lifelong in-patients making their first tentative steps into the real world, they venture out to find what’s on their doorsteps.

Now, if science is to have any useful purpose in society, it’s in describing the unknown, not the bleeding obvious. No wonder it has gotten such a bad name recently.

How much easier it would have been, we suggest, if internet companies had from the outset sought what they would eventually publish anyway: the boring and creepy things that people type into their computers.

If good jokes can tell us truths that are otherwise unmentionable, then perhaps satire can offer us a glimpse of the future that futurologists dare not mention.

In fact, it just has.

AOL’s privacy fiasco was foretold by a splendid prank some Register readers will recall from a few years ago, which looks even better this week than it did at the time.

Brian Del Vecchio created a spoof site called AIMSearch, announcing it with the following:

“In November of 2001 AOL Time Warner, responding to a subpoena from Attorney General John Ashcroft, made available to the Justice Department a complete archive of all private conversations held over AOL Instant Messenger (AIM). Through the power of the Freedom of Information Act (FOIA), Google was able to obtain a copy of this entire logfile, totaling over 2 terabytes of conversations previously thought to be private.”

Then came the kicker:

“This unique resource provides insight into the minds of potential anti-American terrorists, cheating spouses, and countless computer neophytes.”

And so we have it. Within a fortnight, Google had objected to the misuse of its trademark, and the prank ceased.

But how much easier it would have been, for this new generation of “scientific” sociologists, who thanks to data sets like AOL’s query database claim to know so much more than we do about ourselves, and who place so much value on the internet’s “hive mind”, (cf. technology utopian Joi Ito) if it had been clear at the time who was speaking to whom.

The author of the prank, Del Vecchio told us, today -

“Back in 2002, just a few months after 9/11, we wondered how people would react if AOL were to cave to demands from the government and massively betray user privacy. We wanted them to feel that betrayal like a kick in the gut, even if just for a brief second,” he wrote.

“Four years later, there is still a huge gap between the privacy that users imagine they have, and the laxity with which service providers like AOL guard that privacy. I think users like 711391 may be feeling that kick in the gut right now.”

Indeed. We all are.

On the other hand, user 711391 (“christian women caught in extramarital affairs”) really needs help.

And spare a thought for the fellow who typed in “how to murder your wife” dozens and dozens of times. Surely, by now, he’ll be feeling some disappointment that the much vaunted internet, this fabled electronic communications medium, hasn’t yet conjured forth an elite squad of Ninja Assassins to finish her off.

People will always type dark and dirty thoughts into computers – we guess that’s why public, open computer networks were invented, as a kind of public sinkhole. But to turn these private writings into a basis for a new sociology seems to be a little presumptive.

In fact, taking anything that’s typed into the public sinkhole seriously ought to worry us. The “murder my wife” chap is a staple of Northern folklore – he may well one day die a peaceful death having done nothing more harmful than forget to feed his cat. Meanwhile, there are law enforcement agencies, who following the same scientific principles of guesswork and presumption as the “AOL scientists”, who may be keen to argue otherwise.

So if science is to devote itself to this collection of data, may we suggest it be careful. Or preferably, find more pressing issues with which to concern itself.

Facebook’s privacy-busting referral scheme called Beacon is to be modified. If you buy something elsewhere on the web, this information is piped back into your Facebook profile, so your social network can see what you’ve just bought.

Facebook already offered something similar, but with an opt-in model. This opted everyone in by default. People don’t mind telling friends they’ve gone to see Led Zepp – they don’t necessarily want them to see they’ve just bought a blow-up doll.

Who would have guessed?

It’s damaged Facebook and participating advertisers far more than anyone has realized. Facebook’s notoriously weaselly approach to privacy was well in evident, even as it begun to roll out the “fix”.

“Facebook already has made changes to ensure that no information is shared unless a user receives notifications … ” the company explained. Note, not “permissions”, but “notifications”.

In the reader comments, Darren Coleman asks,

“I can’t really see how Facebook can make any money outside of the traditional model of invasive banner ads and Adwords. As sites go it’s a victim of its own success – you can’t monetise the userbase because they’d sooner just jump ship to the next Web 2.0 darling, and if you’re seen to be doing anything that could be construed as towing the corporate line (e.g. ads, tracking, etc) then suddenly you’re no longer the plucky young upstart website – you’re the corporate mouthpiece bought and paid for by the kind of people that talk earnestly about monetisation, incentivising, growing brands, etc. Urgh.”

“It’s the ultimate self-defeating paradigm.”

Good point – is that it, then? Well, not quite, because there are three ways of making money here, and Facebook is trying them all.

Mark “I’m the CEO … bitch” Zuckenberg called the referral program the “holy grail” of advertising when he announced it, and it remains a pipe dream.

The other two programs are safer bets: giving advertisers even more slightly accurate demographic information is sure to be welcomed: advertisers currently get nothing at all.

And getting a cut of transactions through Facebook remains an obvious strategy. As I pointed out at the time, however, this may be smaller than many people suppose. A store that shares the transaction revenue with Facebook is only going to be prepared to do so as long as it considers Facebook a part of that transaction. Is Amazon going to be prepared to pay every referrer for a transaction? You can bet not.

Facebook’s Beacon experience simply demonstrates that it’s been too clever by half: thinking it can do “permission marketing” without your permission.

And the company’s impatience and greed also explain why it faces a long drawn out battle with regulators in Europe. Like a Roach Motel, you can join Facebook – but you’ll never leave.

Fortunately, that company is supremely well-equipped to deal with problems of a technical nature, employing some of the best scientific brains in the world. Unfortunately, neither of these two potential company-crushers has a technical solution: and the answers the brains come up with are only likely to make the problem worse.

The company in question is Google, of course, and here are the two problems.

The first is that most of Google’s wealth – and with it the earnings of businesses both large and small who depend on the advertising broker for the majority of their income – is generated from a system Google controls.

The self-service contextual classified advertising operation is a black box. It looks like a “market” – with buyers and sellers negotiating a price – but it’s a market that Google dominates. Google ultimately sets the price, and when it comes to disputes it’s hanging judge and jury too.

This doesn’t particularly appeal to Wall Street. Not because capital has suddenly been overcome with a dose of ethics – there’s nothing it loves more than a sure monopoly – but markets needs arbitrage. When they’re presented with an opaque model, there’s no way to measure the risk, let alone hedge it.

Google has been most fortunate in that its technological prowess – which we’ve been reminded of constantly over three years of near-hysterical Googlemania – has proved a successful distraction. The deep questions aren’t being asked about its business (and that of Yahoo! of course, too) because the people who usually ask the questions hadn’t realised it yet. But that’s changing.

(When we raised this issue in June, making the Enron comparison, the mailbag drew two very different responses. An injured contingent of fans wrongly assumed we were suggesting Google’s profits are a fiction – of course, they’re very real. But a second set of respondents – from financial institutions – confirmed that this is a make-or-break issue.)

Internet advertising is ripe for fraud, and Google – let’s assume – is doing everything it can to prevent it. But everything might not be enough. From the Church of the Algorithm, there isn’t an algorithm that can help, so long as Google remains judge and jury.

Google’s cultural legacy of secrecy, and it’s sheer geeky pleasure in cranking the handle of its black box, is also evident in this tale Nick Carr relates. Like a casino owner, Google bets with the house’s money to promote its own products, rigging the market. When attention is drawn to its behaviour, it first protests its purity and innocence – then overturns the roulette table. A picture is worth a thousand words, so here’s an illustration of how Google tips the playing field in the direction of its choosing.:

In a decade defined by the hedge fund, how curious it is to see what’s called the “future” of online business given a pass by the risk business.

We can remember it for you wholesale

The second issue is so unmentionable it’s rarely raised in polite company. Again, it’s an area in which we find Google not to be the unique company facing the problem – but uniquely ill-equipped to deal with it.

It’s the question of how much we are prepared to disclose to an anonymous, friendly looking computer system.

Earlier this year, AOL Research quite deliberately and not without some pride, released the search queries of more than half a million users. It wasn’t long before the anonymised queries were matched to their authors. AOL then expressed its horror, and dismissed the staff who it had encouraged to disclose the information (a fine example in its own right of corporate responsibility).

But the cat was out of the bag. The story confirmed that people today choose to disclose information to Google that they wouldn’t tell their husband or wife – and a search engine never forgets this intimate knowledge. They would almost certainly not disclose it if they thought it would be released – or available for casual perusal by cops (which it is).

Similarly, bloggers who blurt away only to be “discovered” are often shocked to learn their writing was visible. Did they think they had some super-selective invisibility cloak?

Technology evangelists of a utopian bent – people who believe this great detritus of disclosure now being collected by information systems such as Google will prove to be of great importance to us – argue that the future is safe. We’ll adapt to the machines, they say. But history tells us that the opposite is true: computer systems that fail to be trusted, fail to be used.

Google’s own experience corroborates this.

Usenet is a medium not too dissimilar to many blogs today, where people wrote informally. The half-life of a Usenet posting was several weeks – it depended on the popularity of the newsgroup – but in most cases archives were not maintained, or easily accessible. And Usenet continued in rude good health long after its demise was predicted. Then, Google took over the creaking archive from Deja, and made searching the entire historical record trivially easy. People simply stopped using it. Usenet died the day Google turned it into a database.

Technical assurances are little use when the trust is lost – or was never there in the first place.

In another example, few countries think a nationally-accessible medical records database is a good idea, and the revolt in the UK against the proposal to create one has led to an opt-out campaign and some (cosmetic) concessions from the government.

People expect records to be computerised – but they don’t want them to be confidential on a need-to-know basis – and not available to any browsing employer, insurance company, bureaucrat, or cop, which they effectively would be given the current state of security.

The saga briefly resurrected HR.4731, Rep Markey’s bill to “Eliminate Warehousing of Consumer Internet Data”. Then Markey went back to lobbying on behalf of Google for “net neutrality” – and the bill remained stalled.

Google’s response to this issue has been a public relations offensive and some muscular lobbying – to prevent more bills like Markey’s.

“We are reasonably satisfied…that this sort of thing would not happen at Google, although you can never say never,” was Google CEO Eric Schmidt’s response to the AOL scandal.

Obsessively secretive, and determined to hoard every piece of data it mines from us, Google appears ill-equipped to restore confidence in the relationship between surfers and the systems we use.

The company initially fended off attention by pointing to its inate goodness. These days it points to its own cleverness. Neither virtue nor engineering talent can solve either problem, however.

Regulation looms in both areas.

Another AOLer moves swiftly to quell his concerns.

“As a whole, no,” responds ‘Matthew’, with the confidence of a veterinary surgeon approaching a rabbit with a chloroform-soaked rag in his hand.

“It can only help make more complete human beings when our minds have been soothed with the information or images they want.”

If you say so.

AOL’s decision to release the search queries from more than half a million users made over a three month period, a decision it now says it regrets, has certainly added enormously to the gaiety of the nations.

And it’s a story, you may have noticed, that’s exciting non-technical internet users far more than those of us who prognosticate upon it, or who write XML parsers for a living, or who feel moved to celebrate “One Web Day”.

(Your reporter has written to “One Web Day” with the suggestion that Harold Pinter could be employed to read out some of the more memorable AOL search queries, but I fear it may not be in keeping with the upbeat tone the organizers are seeking.)

The horrors, you’re already familiar with. But readers have been nominating the strangest and most banal queries AOLers type into their computers, for example:

“nicole richie is so funny”

(While a searcher typing in “vacation and honeymoon ideas in maine and new hampshire” on March 19 was, by May 20, asking “how to overcome trust and jealousy issues”).

In the aftermath of the scandal, privacy groups demanded the only sensible answer to the problem. Search engines can only regain the trust of the public if they delete the search queries as soon as they get them. After all, we’ve established that these questions were never intended to be made public – and in many cases, would only have been made with the assumption that no one would ever see them. So what business do these commercial organizations have hoarding this information?

But alas, said Google CEO Eric Schmidt with a sigh, this is just something they couldn’t do.

Google recently successfully fought a government subpoena to hand over such information to the Department of Justice, while AOL, MSN and Yahoo! meekly complied. But Google didn’t do so in the name of “civil rights”, with which it has a fairly casual relationship, but rather because it feared its “proprietary” algorithms might be compromised. Google explicitly said so in its legal briefs. In other words, a cynic might conclude, it did so for entirely selfish reasons, but saw a low-cost publicity bonus as a consequence.

For reporters, Schmidt had this Churchillian assurance:

“We are reasonably satisfied … that this sort of thing would not happen at Google, although you can never say never.”

Not exactly inspiring, and Google, along with every other commercial search engine, may be underestimating the visceral impact of this episode.

As Seth Finkelstein notes, it’s been something of a watershed week for the internet:

“There’s a teachable moment happening right before our eyes, where conventional wisdom is being changed. Concerns about the implications of data retention, search logs, privacy invasion, etc, are suddenly moving from the outer reaches (ie. civil-libertarians) of polite society, to be respectable issues-of-the-day.”

These concerns have undoubtedly resonated with the public on a far greater scale than the technorati have begun to realize. If you doubt it, find some people for whom “Google” is a recently-acquired verb. They’ll be able to tell you.

There’s nothing like The New York Times ringing you up out of the blue to confirm your porn tastes, or diet fears, that so concentrates the mind.

Seth Finkelstein also has some good advice, for a techie: in the short-term use an anonymizing search engine such as Scroogle. We may also add, use an anonymizing search engine, or a meta search engine, and reject Google cookies.

In the longer term, however, only commonsense legislation can prevent these large commercial organizations from exploiting data that shouldn’t exist in the first place.

The “Zero Retention” campaign may have just kicked into life.

AOL Labs prompted a weekend of hyperventilation in the ‘blogosphere’ by publishing the search queries from 650,000 users. This mini-scandal may yet prove valuable, however, as it reveals an intriguing psychological study of the boundaries of what is considered acceptable privacy.

In his turgid book on Google – one so obsequious and unchallenging that Google bought thousands of copies to give away to its staff – former dot.bust publisher John Battelle enthused about something he called the “database of intentions”. The information collected by search engines, he trumpeted, would be a marketer’s dream, and tell us more about ourselves than we ever realized we could know. AOL’s publication is the first general release of such a database to the public.

But hold on a minute. Is it, really?

AOL’s data was anonymized, with user identification removed. The search logs contained 10.8m normalized queries from 658,086 unique users, collected between March 1 and 31 May this year, amounting to around a third of all queries made by its US users. The data has since been removed, but an AOL research paper which was built on the data can still be found, here [PDF, 228kb]. You may find it about as enlightening as similar studies we’ve covered before (for example, see People more drunk at weekends, researchers discover).

Although the user’s IDs were hidden, and didn’t contain information on what the user actually clicked on, some argued that the data permitted personally identifiable information to be inferred from the query logs.

Something called TechCrunch, a weblog devoted to hyping its publisher’s personal investments and companies created by his friends, explained how:

“The most serious problem is the fact that many people often search on their own name, or those of their friends and family, to see what information is available about them on the net. Combine these ego searches with porn queries and you have a serious embarrassment. Combine them with ‘buy ecstasy’ and you have evidence of a crime. Combine it with an address, social security number, etc., and you have an identity theft waiting to happen. The possibilities are endless.”

Now, you may be thinking – that only serves people right for conducting vanity searches. But more seriously, there are dangers in following this line of reasoning.

It’s not only individuals who “ego surf”, it could be the individual’s spouse, a member of their family, a colleague, or even their web stalker. (I’ve had a few).

Similarly, is the query “buy ecstasy” necessarily the intention of a raver, or tweaker? It might be a parent, a neighborhood watch scheme, or a promoter, keen to stamp out drug dealing at his venue before an event.

So the “database of intentions”, then, turns out to be more more of “a database of inferences” – as reflective as it is of the inferrer as the web surfer.

And if, as TechCrunch weakly suggests, the act of typing “buy ecstasy” into a search is itself “evidence of a crime”, then there will be a lot of happy policeman out there this evening, for whom the business of catching criminals has just been made a lot easier.

The” precogs” of Phillip K Dick’s story Minority Report – who are able to predict crimes before they take place, thus allowing them to be prevented – will no longer be necessary. Plod will simply be able issue a pre-emptive warrant for a crime that never took place, on the basis of a user’s Google results, no?

So that’s one line of sloppy thinking dealt with. It ignores another, however.

Leave No Trace

Law enforcement agencies, particularly in the US, tend to receive more strict oversight than corporations. The immediate harm for ordinary citizens comes not from paranoid SF fantasies, but from the “database of inferences” being exploited for commercial gain.

More lives are affected every day by the actions of banks, insurance companies and HMOs, than they are by data-mining cops. If your LiveJournal blog contains more frowns than smileys, you may well need to be prescribed a course of an anti-depressant. If your lifestyle involves risky situations in night spots, you may well need to pay a higher insurance premium. Yet such invasive data mining is the inexorable conclusion of overestimating the value of this harvest of so-called “machine wisdom”.

People would be rightly be outraged if Big Pharma, banks and the insurance business created “inference profiles” based on one’s data trail. But, wait! That’s what they already do. Human decision-making is playing an increasingly smaller role in whether credit applications are approved, or what kind of health care is permissible. When corporations do this, they are making implicit moral choices – that one person is more or less than deserving than another – but obscuring the decision behind a smokescreen of technology babble.

The addition of an internet clickstream to the mass of data they already possess about you is but a small, incremental step.

So why not tackle this problem at source?

The only solution to the problem of data abuse – and it’s only an inadequate, and very partial answer – is to ensure the data isn’t there to abuse in the first place. If search engines were required to delete their users’ queries as soon as they were made, and to leave no trace, this would greatly diminish the dangers of false inference by law enforcement officials, health companies, banks, HMOs, and anyone else seduced by the lure of a faulty algorithm.

Data that doesn’t exist is also less vulnerable to being stolen.

This would disappoint law enforcement officials, many corporations, and most of all the search engines themselves – Google CEO Eric Schmidt has boasted of building a “Google that knows more about you.”

If that takes a regulatory agency, to ensure search engines “Leave No Trace”, so be it. And meanwhile, drooling over such bad metaphors as “Database of Intentions”, or “Collective Intelligence”, is going to make data abuse more, and not less likely.

The file accompanied part of Google’s analyst day last week. Google has since withdrawn the file, telling the BBC that the information was not intended for publication.

The justification for this enormous data grab is that Google would be able to restore your data after a catastrophic system failure.

The notes reveal a plan to -

Store 100% of User Data

… With infinite storage, we can house all user files, including: emails, web history, pictures, bookmarks, etc and make it accessible from anywhere (any device, any platform, etc).

We already have efforts in this direction in terms of GDrive, GDS, Lighthouse, but all of them face bandwidth and storage constraints today. (…) This theme will help us make the client less important (thin client, thick server model) which suits our strength vis-a-vis Microsoft and is also of great value to the user.

As we move toward the ‘Store 100%’ reality, the online copy of your data will become your Golden Copy and your local-machine copy serves more like a cache.”

Perhaps it’s Google’s gift to the US government. In August 2003, Admiral John M Poindexter was forced to resign after his ‘Total Information Awareness’ data mining program was revealed to be indexing “everyday transactions as credit card purchases, travel reservations and e-mail.”

Exactly what Google will have if its ‘GDrive’ ever materializes.

And here’s a coincidence.

What tipped Poindexter’s resignation was his specific plan to operate “terror casino”. The scheme porported to tap “collective wisdom” of the public in predicting world events such as assassinations.

This hokum New Age idea, beloved by autistic technophiliacs, was rapidly shot down. But it has its fans in Silicon Valley, as this slide from Google’s analyst presentation shows.

Google has outspooked the spooks.

The phone poll, which sampled over 1000 internet users, was conducted by the Ponemon Institute following the DoJ subpoenas last week.

This suggests that the battle for internet privacy is far from over.

Google maintains a lifetime cookie that expires in 2038, and records the user’s IP address. But more recently it has begun to integrate services which record the user’s personal search history, email, shopping habits, and social contacts. After first promising not to tie its email service to its search service, Google went ahead and opted its users in anyway. It’s all part of CEO Eric Schmidt’s promise to create a “Google that knows more about you”.

The conundrum for Google now is does it come clean with the data it stores about users, or does it simply hope that the majority of users don’t care?

In the survey, 56 per cent of users said Google should not turn over information to the Government, and only 14 per cent were happy for Google to turn over information even in criminal cases.

The journalist in question, Edmonton newspaper columnist Kerry Diotte, wasn’t suspected of involvement in any crime. But Diotte had written a column criticizing the police force’s radar and camera technology as being more of a cash cow for the force than an effective measure against road fatalities – and the story enraged the local constabulary.

Diotte has been a consistent critic of the police’s technology dependency habit.

Police illegally tapped into the confidential vehicle licensing database to obtain Diotte’s details of car and home residence, then staked out the Overtime Bar in Edmonton with three officers.

According to one Edmonton officer, the journalist posed a real threat to the city’s burghers.

“It was true that we found that vehicle in a bar lot and the potential for serious harm or death was there,” claimed Sergeant Glen Hayden, testifying before a disciplinary hearing.

According to witnesses the journalist took a cab home and was reported to be sober. What kind of journalist is he?

The hearing continues.