Remember that Dennis Crouch initially focused attention on the Patent and Trademark Office’s use of a ‘robots.txt’ file to restrict search engines from crawling and indexing decisions of the Board of Patent Appeals and Interferences. I explained the technical cause and effect of the robots file in this post, and even offered a proposed solution. While researching that post, I discovered a bit of technical history that made me wonder if someone at USPTO made a conscious policy decision to restrict search engine access in order to force users to use its own flawed search function:
“As an aside, the presence of this robots.txt file on the BPAI server is particularly troublesome when you consider a bit of history. As recently as a few years ago, PTO apparently did not use such bot-blocking tactics – users were able to search BPAI decisions using Google by limiting your search to the old decisions directory. Now, this hack provides an extremely limited results set – today, an unlimited search on that directory returns only 137 results. It appears, therefore, that a decision was made to actively remove Board decisions from the internet search space in favor of the flawed USPTO search page.”
The USPTO search page for BPAI decisions is so flawed in this modern age of quick and effective browser-based search technologies that no one can seriously argue that it provides an acceptable degree of transparency. Indeed, the technical switch made a couple years ago seems to have placed Board decisions behind an opaque shield.
Now we have an interesting twist: President Obama, acting swiftly on his stated commitment to transparency in government, adopted a wide-open robots.txt file on the whitehouse.gov servers on his first day in office. The 2-line permissive Obama file replaced the 2400-line restrictive robots.txt file that was on the servers, thanks to the Bush administration, up to the day before.
The PTO cleary can’t keep its restrictive approach under the new administration. And, thanks to President Obama’s swift action, we now see that, indeed, it is something that can be changed overnight.



