Do search engines delay indexing of blog posts they don’t like?

Posted By: James Rowles June 5, 2013

In recent months, the Observer has had the extraordinary experience of writing articles and posting them to this blog, on highly topical and timely issues, only to see the bots and indexing of search engines come suddenly to a halt, if effect censoring the content from distribution on the web.

When most posts are indexed within hours on a regular basis, and we now have posted some 390 articles, it is quite extraordinary to have your content blocked by the failure of search engines, in particular those who others cue off of, simply to index your new page.

It gives one an idea of the enormous power that lies in the hands of companies that control the search engines that millions of people throughout the world use to access information. A particular article from a particular source may simply be left out, not indexed, and remain invisible to the world.

Moreover, we have no government oversight of the processes and individuals which may be responsible for making such decisions. We know that search engines greatly distort our view of search results, filtering by country, language, location, and previous search results. But it is still something of a surprise to find a highly topical article, intimately tied to the news cycle, simply left out, simply omitted from the bots and indexing activities upon which we all depend to publish our views and opinions.

This is a power that authoritarian states use to control what their citizens can see on the internet. But in the United States?

There may well be an urgent need for government oversight of the uses of their powers by leading search companies, which have a direct impact on the exercise of first amendment rights to freedom of speech, and international human rights obligations protecting the right to impart and disseminate information.

If indexing is only delayed for two or three days, the effect may still be pernicious, depriving the world of the possibility of finding the opinions expressed in the article or blog post at the very moment when it is of interest and individuals are searching for it. In a real sense, indexing delayed is indexing denied, achieving the same effect as a court order blocking dissemination of certain ideas for a determined period of time.

It would be most interesting to hear from others who may have had similar experiences. Please write the Observer at “observer@trenchantobserver.com”.

The Trenchant Observer

About the Author

James Rowles

"The Trenchant Observer" is edited and published by James Rowles (aka "The Observer"), an author and international lawyer who has taught International Law, Human Rights, and Comparative Law at major U.S. universities, including Harvard, Brandeis, the University of Pittsburgh, and the University of Kansas. Dr. Rowles is a former staff attorney at the Inter-American Commission on Human Rights (IACHR) of the Organization of American States OAS), in Wasington, D.C., , where he was in charge of Brazil, Haiti, Mexico and the United States, and also worked on complaints from and reports on other countries including Argentina, Chile, Uruguay, El Salvador, Nicaragua, and Guatemala. As an international development expert, he has worked on Rule of Law, Human Rights, and Judicial Reform in a number of countries in Latin America, the Caribbean, Africa, the Middle East, South Asia, and the Russian Federation. In the private sector, Dr. Rowles has worked as an international attorney for a leading national law firm and major global companies, on joint ventures and other matters in a number of countries in Europe (including Russia and the Ukraine), throughout Latin America and the Caribbean, and in Australia, Indonesia, Vietnam, China and Japan. The Trenchant Observer blog provides an unfiltered international perspective for news and opinion on current events, in their historical context, drawing on a daily review of leading German, French, Spanish and English newspapers as well as the New York Times, the Wall Street Journal, the Washington Post, and other American newspapers, and on sources in other countries relevant to issues being analyzed. Dr. Rowles speaks fluent English, French, German, Portuguese and Spanish, and also knows other languages. He holds an S.J.D. or Doctor of Juridical Science in International Law from Harvard University, and a Doctor of Law (J.D.) and a Master of the Science of Law (J.S.M.=LL.M.), from Stanford University. As an undergraduate, he received a Bachelor of Arts degree, also from Stanford, where he graduated “With Great Distinction” (summa cum laude) and received the James Birdsall Weter Prize for the best Senior Honors Thesis in History. In addition to having taught as a Lecturer on Law at Harvard Law School, Dr. Rowles has been a Visiting Scholar at Harvard University's Center for International Affairs (CFIA). His fellowships include a Stanford Postdoctoral Fellowship in Law and Development, the Rómulo Gallegos Fellowship in International Human Rights awarded by the Inter-American Commission on Human Rights, and a Harvard MacArthur Fellowship in International Peace and Security. Beyond his articles in The Trenchant Observer, he is the author of two books and numerous scholarly articles on subjects of international and comparative law. Currently he is working on a manuscript drawing on some the best articles that have appeared in the blog.