Resources, Reports, Tools, Lists, and Full Text
U.S. Government–Web Content–Archives–Databases
Source: IA
NARA/Internet Archive Collection of U.S. Government Web Material NOW Keyword Searchable
Word from the Internet Archive (IA) that a special collection (about 75 million pages) of web material that they collected/”harvested”/captured for the National Archives (NARA) IS NOW keyword searchable. The collection is titled, “2004 Presidential Term Web Harvest.” Look for the new search box on the right side of the page. This collection first became available to the public in January 2005.

From the web site:
“The National Archives and Records Administration (NARA) conducted a harvest (i.e., capture) of Federal Agency public web sites as they existed prior to January 20, 2005. This harvest was intended to document Federal agencies’ presence on the World Wide Web at the time that the Presidential Administration term ended in early 2005.”

“The 2004 Presidential Term Web Harvest is a National Archives and Records Administration (NARA) project that produced a collection of federal web sites copied, or harvested, from the world wide web between 10/14/04 and 11/19/04. The Heritrix web harvester (http://crawler.archive.org/) and a list of 982 active and unrestricted second level URLs were used to capture all linked federal sites down to the fourth level. Those initial 982 ‘.gov’ and ‘.mil’ URLs were provided by U.S. General Services Administration’s (GSA) ‘.GOV’ Internet Domain Registry and the Defense Information Systems Agency (DOD/DISA).”

Today’s posting from the Internet Archive:
“This is our [IA] largest public single text searchable collection to date. The index was created using the NUTCH and NUTCHWAX extensions open source software.”

Kudos to the IA. I wonder if they have any plans to bring back keyword search capabilities to The Wayback Machine (or at least a portion of it). I hope so. A few years ago the IA offered Recall. It allowed users to keyword search a portion of TWM. More here about this gone-but-not-forgotten service here. The search technology was developed by Anna Patterson who now works at Google.

This entry was posted on Friday, March 3rd, 2006 at 12:55 am and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.