How to Block Google From Indexing Your Site

by Brian Blum on July 19, 2009

As newspaper chains and agencies like the Associated Press complain that Google News has been aggregating their content thereby hurting monetization opportunities, a Google exec says: “grow up.”

Josh Cohen, senior business product manager for Google, writing on Google’s public policy blog, explained that if you don’t want your content to show in search results or on Google News, you can put a simple piece of code on the site that will block Google from indexing the article.

For all you techies, just add <meta name=”googlebot” content=”noindex”> to a page and that’s it.  It’s called the Robots Exclusion Protocol and it’s been in use for more than 10 years.

(It’s something by the way many good blogging systems like WordPress have been able to do with free plug-in’s for years.)

Publishers can even require that the material Google indexes can be removed after a certain date (say, when it gets archived and goes behind a pay wall). You simply add a specification to the page reading “unavailable after,” Cohen went on to explain.

Cohen started off his post by quoting a declaration from a group of European newspaper and magazine publishers stating that they “no longer wish to be forced to give away property without having granted permission.”

“We agree,” he responded, “and that’s how things stand today. The truth is that news publishers, like all other content owners, are in complete control when it comes not only to what content they make available on the web, but also who can access it and at what price. This is the very backbone of the Web.”

Frankly, we don’t understand what all the fuss is about. Having content show up on Google News and point back to the original story seems like a way to increase traffic and generate revenue. Bill Grueskin, dean of academic affairs at Columbia University Journalism School and former deputy managing editor for The Wall Street Journal, said the same in an interview at PaidContent.

“Yes, it’s hard on the ego to watch another site get credit for your hard work, but is it really hurting the bottom line?” Aggregators are possibly “saviors” for drawing eyes to news sites, Grueskin said.

For more articles on newspapers and classified advertising, visit the industry experts:

Leave a Comment

Next post: