Fifteen years ago, the world wide web was the playground of boffins. Its design reflected the open ethos of those users: it had no central managers, no main menu and no investment in content – indeed, no business plan whatsoever.

Instead, its framers assumed that people would put their own material online, and users would then surf from one site to another, following links on the pages.

Then the first search engines sprang up, which sent digital robots crawling from one link to the next, copying everything they found. The idea was to index the entire web in one place by obsessively following every path from several starting points.

Soon you could search the web by entering a search term and finding all the pages that contained it. This shortcut rankled with some webmasters. Even though they’d chosen to put their data on the web for all to see, they felt far more exposed once any words they used within their pages could turn up as search results.

But the webmasters didn’t turn to lawyers. They didn’t claim a right to be excluded from the robots’ indexing, even though they might well have had grounds to do so. Instead, an informal discussion produced an obscure standard called “robots.txt”. What this meant was that a webmaster could place very basic requests in a corner of his website to tell visiting robots which pages to ignore.

And it worked: every big search engine, no matter how data-hungry, passes over pages marked by webmasters as not to be searched. (Try visiting your favourite website while adding /robots.txt to its name, such as www.timesonline.co.uk/robots.txt, and you’ll see what I mean.)

Since then, robots.txt has become a crucial part of the scaffolding of the web, creating semi-private spaces rather than entirely private or entirely public ones. And, at a time when we ourselves are feeling exposed and vulnerable about the amount of personal information that is stored on the web, this nerdy concordat offers an important lesson.

By now, everyone has heard horror stories about invasions of privacy, cyber-bullying, photos taken out of context, embarrassing videos posted online and the mob mentality of some commenters.

Let’s take just one example: in 2002, a Canadian boy filmed himself swinging a golf ball retriever as if he were a Jedi knight. For a while, the tape lay forgotten. Then some of his friends saw it, and without his permission, placed it online.

Within two weeks, the video had been posted in many places and viewed millions of times. Spin-off videos were produced, adding soundtracks and extra graphics. People who encountered the video of the Star Wars Kid online happily forwarded it to friends – without considering the ethical implications.

The boy himself, however, was mortified. At school, he was taunted with chants of “Star Wars Kid”. Eventually, he was diagnosed with depression. And, to this day, he deeply regrets the airing of the film.

The first instinct in such cases is to look to the law. Indeed, the boy sued, and a settlement was reached with the families of the boys who initially put the video online.

But the law is always a last resort, and there are many more such stories that show how the openness of the web can lead to collateral damage. Consequently, some people place as little about themselves online as they can, hoping to remain a grain of sand on the digital beach. Which is why it is now time to recognise that the personal data flooding the web desperately need to include cues on privacy.

What is today a disembodied photo on a Google image search could be tagged by both its photographer and subjects to indicate just how far they’d like it to spread. In the tag, they could also request that certain uses of the picture be cleared with them first, and provide a way to reach them for consultation without divulging their identities.

That photo would then become anchored to the people who made it and are in it. If such a system had been in use when the Star Wars Kid made his video, he could have tagged it as private at the time he made it – just in case it ever escaped its videocassette home. His friends might not have posted it. And, even if they had posted it, others might not have helped it go viral.

Take the Star Wars Kid test yourself: imagine someone has forwarded you a link to the video and you think it’s funny. You’re about to share it with friends. But then you see that it’s been marked by the kid himself as private, with a desperate plea not to fan the flames along with an explanation of what happened to get it online in the first place. Would you forward the link?

If enough people – not everyone, of course, just enough – decided to respect the person’s wishes, the video might never reach the critical mass needed to take it viral. We know, of course, that there are bad apples online. There are people who won’t respect reasonable requests, made nicely. But it’s the rest of us who transform run-of-the-mill privacy violations online into the truly awful phenomena that they can become.

I’m not blaming anyone because, at the moment, there’s no easy way to convey those requests and cues that make civilisation breathe. But if we build these technologies and embed them in the web, we’ll be able to make true moral choices when we remix a video or redistribute information that might be just a bit of fun – or might be personally devastating.

Interestingly, the “kid” is not named in the extensive entry about his video on Wikipedia, the online encyclopedia that anyone can edit. On the article’s corresponding discussion page, a debate has raged about whether or not he should be identified.

“He’s already been very prominently featured in a USA Today story, as well as being mentioned in many, many places on the web,” writes one Wikipedian.

Another says: “I read that his parents requested his last name to be kept confidential in future reproductions. Therefore, I think it would be wise if we deleted the name, not in the fear that Wiki’ll be sued otherwise, just out of courtesy to the poor kid.”

Courtesy prevailed. Online masses 1, USA Today 0.

Jonathan Zittrain is the professor of internet governance and regulation at Oxford University, and author of The Future of the Internet, published by Allen Lane on May 1, priced £20