Web Search
Source: GCN
“White House: Web site doesn�t steer clear of Iraq”
What many thought was the perfect web conspiracy theory was really an example of how a lack of knowledge about how search engines work can cause the conspiracy theorists to have a field day. It also illustrates how quickly incorrect info can spread over the web. The robots.txt files that has been in place on WhiteHouse.Gov has been there for many months. However, it’s worth pointing out that other sites block spiders from crawling because they DO NOT want the material accessible via a general engine.
See Also: Learn About How Robots.Txt Works

This entry was posted on Wednesday, October 29th, 2003 and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.