In recent months, the Observer has had the extraordinary experience of writing articles and posting them to this blog, on highly topical and timely issues, only to see the bots and indexing of search engines come suddenly to a halt, if effect censoring the content from distribution on the web.
When most posts are indexed within hours on a regular basis, and we now have posted some 390 articles, it is quite extraordinary to have your content blocked by the failure of search engines, in particular those who others cue off of, simply to index your new page.
It gives one an idea of the enormous power that lies in the hands of companies that control the search engines that millions of people throughout the world use to access information. A particular article from a particular source may simply be left out, not indexed, and remain invisible to the world.
Moreover, we have no government oversight of the processes and individuals which may be responsible for making such decisions. We know that search engines greatly distort our view of search results, filtering by country, language, location, and previous search results. But it is still something of a surprise to find a highly topical article, intimately tied to the news cycle, simply left out, simply omitted from the bots and indexing activities upon which we all depend to publish our views and opinions.
This is a power that authoritarian states use to control what their citizens can see on the internet. But in the United States?
There may well be an urgent need for government oversight of the uses of their powers by leading search companies, which have a direct impact on the exercise of first amendment rights to freedom of speech, and international human rights obligations protecting the right to impart and disseminate information.
If indexing is only delayed for two or three days, the effect may still be pernicious, depriving the world of the possibility of finding the opinions expressed in the article or blog post at the very moment when it is of interest and individuals are searching for it. In a real sense, indexing delayed is indexing denied, achieving the same effect as a court order blocking dissemination of certain ideas for a determined period of time.
It would be most interesting to hear from others who may have had similar experiences. Please write the Observer at “email@example.com”.
The Trenchant Observer