From the Post:

I’m on the board of CommonCrawl.Org, a nonprofit corporation that is attempting to provide a web crawl for use by all. An interesting report just got sent to us about the use of robots.txt files within the .Gov Top Level Domain, a standard known as the Robots Exclusion Standard.

In examining about 32,000 subdomains in .gov, it turns at least 1,188 of these have a robots.txt file with a “global disallow,” meaning robots ar>e excluded from indexing this content. Even more curious, on 175 of these sites, while there is a global disallow, there is a specific bypass that allows the Googlebot to index the data. You can look at the raw data on Factual.

At Public.Resource.Org, we’ve always felt that the use of a robots.txt file by the government should only be used for purposes of security and integrity of the site, not because some webmaster arbitrarily decides they don’t want to be indexed. Indeed, on several occasions we have deliberately ignored government imposed robots.txt files because we felt this was an arbitrary and illegal attempt to keep the public out.

Access the Complete Post

Source: O’Reilly Radar

See Also: This is not a new issue, we posted about it about 2.5 years ago.

See Also: Study shows Google favored over other search engines by webmasters (via Penn St. Live; )

See Also: BotSeeer

See Also: Intro to Robots.txt from SearchTools.com

Broken links will soon be a thing of the past for UK Government websites, as The National Archives launches its Web Continuity project.

The first of its kind anywhere in the world, the project has already enabled millions of people using government websites to find information which would previously have been lost through broken web links.

Officially launching at the House of Lords next month, the initiative links seamlessly with The National Archives’ UK Government Web Archive, which regularly captures and preserves 1500 government websites for posterity.

If someone clicks on a link which is no longer live, redirection software being installed by government departments will automatically take them to where the information they need is held in the web archive. Currently, the service is leading to more than six million redirected hits a month.

Dr Amanda Spencer, Head of Web Continuity at The National Archives, said: “When it was first created, the internet was often regarded as ephemeral. Websites weren’t viewed as records which needed to be preserved.

“However, as the internet has developed to be the predominate source of government information for most people, this has changed. Today, some information only ever exists online and as the experts in preserving the future of history, we have had to adapt.

“Our web archive holds more than 340 million documents from government websites dating back to 1997, and we now archive all central government websites three times a year. The Web Continuity project is a natural progression of this.”

Much More After Clicking

To date six central government departments together with The National Archives have installed the necessary redirection software.

“While broken web links are a problem across the whole of the internet, as pages move, information is taken down and websites change, The National Archives is the first to offer a solution,” said Dr Spencer.

The Web Continuity project is due to be formally launched at the House of Lords on December 2, 2009.

From the Article

Starting Wednesday in Sweden, the European Union is holding a conference of ministers of technology from across Europe that will be looking at lessons learned throughout the EU.

In conjunction with that, a new book is out: State of the eUnion: Government 2.0 and Onwards. It’s available online for free, and will eventually come to a store near you.

It pulls from some of the Web 2.0 thought leaders, many of whom you have heard here on Federal News Radio, including Tim O’Reilly, Stanford law professor Lawrence Lessig, and Mark Drapeau from George Washington University.

Source: Federal News Radio
Hat Tip: Pete W.

Access the Full Text Book: State of the eUnion: Government 2.0 and Onwards (321 pages; PDF)

This Article offers a new way of thinking and talking about government secrecy. In the vast literature on the topic, little attention has been paid to the structure of government secrets, as distinct from their substance or function. Yet these secrets differ systematically depending on how many people know of their existence, what sorts of people know, how much they know, and how soon they know. When a small group of similarly situated officials conceals from outsiders the fact that it is concealing something, the result is a deep secret. When members of the general public understand they are being denied particular items of information, the result is a shallow secret. Every act of state secrecy can be located on a continuum ranging between these two poles.

Attending to the depth of state secrets, the Article shows, can make a variety of conceptual and practical contributions to the debate on their usage. The deep/shallow distinction provides a vocabulary and an analytic framework with which to describe, assess, and compare secrets, without having to judge what they conceal. It sheds light on how secrecy is employed and experienced, which types are likely to do the most damage, and where to focus reform efforts. And it gives more rigorous content to criticisms of Bush administration practices. Elaborating these claims, the Article also mines new constitutional territory – providing an original account of the role of state secrecy generally, as well as deep secrecy specifically, in our constitutional order.

Several options available for retrieval of full text.

Source: Stanford Law Review, Forthcoming (David Pozen)

Hat tip: Secrecy News

The U.S. Food and Drug Administration (FDA) and Everyday Health today announced a collaboration that will expand the delivery of the agency’s vital consumer health information to the 30 million unique users who visit Everyday Health each month. This joint effort reflects FDA’s emphasis on using innovative, technology-based strategies to carry out its mission of protecting and promoting the public health.
…
The partnership will initially include:

• A new online resource at www.EverydayHealth.com/FDA: The new co-branded Web site will feature a variety of health content from FDA. The latest information on food and medical product safety as well as prevention and wellness topics will be featured. In the event of breaking public health information, Everyday Health will also feature special “FDA Alert” modules in select locations throughout the site and network, and in e-mail newsletters.
• A new FDA/Everyday Health co-branded weekly newsletter: The latest FDA consumer health information will be sent to subscribers in a weekly Everyday Health newsletter (www.EverydayHealth.com/FDA). The FDA/Everyday Health newsletter will contain “FDA Alerts” as well as up-to-date information on drug safety, cosmetics and skin care products and children’s health products, to name a few.

Source: U.S. Food and Drug Administration

The last great archives of Franklin D. Roosevelt’s presidency may soon be available to researchers and the public – 14 boxes of handwritten notes, gifts and correspondence, including a letter from Italian dictator Benito Mussolini congratulating him on his 1933 inauguration.

The House on Monday approved a bill to clear the way for the memorabilia to be donated to Roosevelt’s presidential library and museum in Hyde Park, N.Y.

While the House bill is identical to legislation the Senate passed in October, it will still have to return to the Senate for one more vote before it goes to the president

Read and Track the Legislation
House Bill ||| Senate Bill
Source: GovTrack.us

Source: Washington Post

“For America to obtain an optimal return on our investment in science, publicly funded research must be shared as broadly as possible,” is the message that forty one Nobel Prize-winning scientists in medicine, physics, and chemistry gave to Congress in an open letter delivered yesterday. The letter marks the fourth time in five years that leading scientists have called on Congress to ensure free, timely access to the results of federally funded research – this time asking leaders to support the Federal Research Public Access Act of 2009 (S.1373).
…
The bi-partisan Federal Research Public Access Act (FRPAA), introduced by Senators Lieberman (I-CT) and Cornyn (R-TX), would deliver online public access to the published results of research funded through eleven U.S. agencies and departments, requiring that peer-reviewed journal articles stemming from publicly funded research be made available in an online repository no later than six months after publication.

+ An Open Letter to the U.S. Congress Signed by 41 Nobel Prize Winners (November 2009)

Source: Alliance for Taxpayer Access

Department of Homeland Security (DHS) Secretary Janet Napolitano today unveiled a new website designed to provide a one-stop location for veterans and veterans organizations to learn about DHS’ many veteran outreach initiatives and hiring and contracting opportunities.

“This new website reflects the shared commitment across the Department to hiring American veterans,” said Secretary Napolitano. “Veterans play a vital role in the Department of Homeland Security’s mission to protect the nation, and this website will help us build our veteran workforce to more than 50,000 Department-wide by 2012.”
…
The new website, available at www.dhs.gov/veterans, features information for veterans about how to find employment opportunities at DHS, ways to get involved in community-based efforts like Citizen Corps, and special veteran programs such as Operation Warfighter and Wounded Warrior, which provide employment opportunities for severely wounded or recovering service members to assist their transition back to the military or civilian workforce.

Confused by all the proposed changes to the Patriot Act ricocheting through the Capitol? The Center for Democracy and Technology (CDT) has put together a handy chart comparing the current law with the various amendments in the House and Senate.

The chart compares proposed amendments (.pdf) to National Security Letters (NSLs) and the so-called “lone wolf” provisions of the Patriot Act. The proposals have only been passed by the judiciary committees, and face further amendments before they hit the full House and Senate for votes.

+ Direct to the Amendments Chart (7 pages; PDF)

+ Access the Complete Wired Story
Much more on the proposed amendments.

Source: Threat Level (Wired)

Source: Wired

The database can be searched by keyword (the way to go) or browsed a page at a time (there are 169 pages). You can also ask a question and there is even an RSS feed so you can be alerted to new database entries.

Questions and answers cover a wide variety of topics. From the Census Bureau’s use of GPS to converting NAICS-based data into SIC categories.

A resource to add to your ready reference resources.

Source: U.S. Census

See Also: USA.gov Also Has a Searchable Database Containing More than 2500 Asked and Answered Questions
This database also allows you to browse by topic and subtopic.

The first text messages were sent during 1992 and 1993, although commercially, text messaging was not widely offered or used until 2000. Even then, messages could only be sent between users subscribed to the same wireless carrier, e.g., Sprint customers could only exchange messages with other Sprint customers. In November 2001, however, wireless service providers began to connect their networks for text messaging, allowing subscribers on different networks to exchange text messages. Since then, the number of text messages in the United States has grown to over 48 billion messages every month. Additionally, text messages are no longer only sent as “point-to- point” communications between two mobile device users.

More specifically, messages are also commonly sent from Web-based applications within a Web browser (e.g., from an Internet e-mail address) and from instant messaging clients like AIM or MSN. For Congressional policymakers, two major categories of issues have arisen: (1) “same problem, different platform” and (2) issues stemming from the difficulty in applying existing technical definitions to a new service, such as whether a text message is sent “phone-to-phone” or using the phone’s associated email address. There are numerous examples of each. An example of the first category would be consumer fraud and children’s accessing inappropriate content, which have existed previously in the “wired world,” but have now found their way to the “wireless world.” An example of the second category would be that spam sent between two phones or from one phone to many phones does not fall under the definition of spam in the CAN-SPAM Act of 2003 (Controlling the Assault of Non-Solicited Pornography and Marketing Act, P.L. 108-187); however, if that same message were to be sent from a phone or computer using the phone’s associated e-mail address, it would.

The increasing use of text and multimedia messaging has raised several policy issues: applicability of CAN-SPAM Act to unwanted wireless messages; refusal of some carriers to allow users to disable text messaging; carrier blocking of Common Short Code messages; deceptive and misleading Common Short Code programs; protecting children from inappropriate content on wireless devices; “sexting”; mobile cyberbullying; and balancing user privacy with “Sunshine,” Open Government, and Freedom of Information Laws.

Access the Full Text (19 pages; PDF; October 22, 2009 Version)

Links to Previous Versions

Source: Congressional Research Service (via OpenCRS)

BillMaps is very easy to use but potentially very useful for those who like to “see” how something looks. Sometimes you can see something on a map (e.g. a trend) that would be difficult to detect just by looking at the text.

Simply enter a bill number (the database goes back to 101st Congress) and then select what you would one of the two mapping options. You can either map where the sponsors of a bill are from or what a vote looks like on a map. In other words, Google Map “pins” are placed inside the state where the congressperson is from and colored either green for “aye” or red for “nay.”

Each pin can be clicked and you’ll find the name of the voter and direct links to info about that person from the OpenCongress database and the Govtrack.us database (a ResourceShelf fave).

On the home page you can find links to:

+ Most Tracked Bills this Week
+ Most Supported Bills this Week
+ Most Opposed Bills this Week
+ Hot Bills
+ Most Blogged Bills this Week

Btw, on any list page you can access a brief bill summary by moving your cursor over the title of the bill.

So here’s an example. First, we selected Most Tracked Bills this Week
We’re finding out where the “most tracked” number is coming from. Our guess, GovTrack.us.

Next, we selected #H.197: National Right-to-Carry Reciprocity Act of 2009, click and we see where the 162 sponsors are from.
In addition to the map you’ll find a brief summary and related bills.

Here’s another example, we went to the top of the home page and entered H.1 from the 110th Congress and then vote. The bill was titled, Implementing the 9/11 Commission Recommendations Act of 2007. Here’s the map.

Below the map you’ll see 435 votes. Next to that you’ll spot an “S.” Click and you’ll go to the sponsor map. On a sponsor map, look for a “V.” When clicked you’ll go to a vote map for that piece of legislation.

A nice use of several databases and API’s (Application Programming Interfaces) to create something that can provide a view not visible without the use of a map.

Access BillMaps

A new video features Minnesota Speaker of the House Margaret Anderson Kelliher talking about new methods to preserve and provide access to digital records of state legislatures. The production describes the work of A Model Technological and Social Architecture for the Preservation of State Government Digital Information Project, which is supported by the Library of Congress National Digital Information Infrastructure and Preservation Program.

Direct to Video (via Minnesota Historical Society)
It runs about six minutes.

Source: National Digital Information Infrastructure and Preservation Program

Direct to DoD Social Media

Look for Material from these Services
+ Delicious
+ Facebook
+ Flickr
+ iReport
+ MySpace
+ nowpublic
+ Twitter
+ Vimeo
+ YouTube
+ and a good helping of blogs.

This is also a compilation where you need to read the disclaimer. The guide sits on a Department of Defense server but DoD doesn’t endorse the content on any of these social media sites/services or exercise any editorial control over the information.

So, awareness of the situation (read the disclaimer) and some good critical info skills are required for effective use of these resources which have the potential for being both interesting and useful.

Here’s the Disclaimer:

The appearance of external hyperlinks does not constitute endorsement by the United States Department of Defense of the linked web sites, or the information, products or services contained therein. For other than authorized activities as military exchanges and Morale, Welfare and Recreation (MWR) sites, the United States Department of Defense does not exersise any editorial control over the information you may find at these locations. All links are provided consistent with the mission of Defense.gov. Please let us know about existing external links which you believe are inappropriate.

Source: U.S. Department of Defense
Hat Tip: Free Government Information (Good Post to Check For More Info and Some Interesting Links)

From the Announcement:

The Electronic Frontier Foundation (EFF), Electronic Information for Libraries (eIFL.net), and other international copyright experts joined together today to launch Copyright Watch — a public website created to centralize resources on national copyright laws at www.copyright-watch.org.

[Snip]

Copyright Watch is the first comprehensive and up-to-date online repository of national copyright laws. To find links to national and regional copyright laws, users can choose a continent or search using a country name. The site will be updated over time to include proposed amendments to laws, as well as commentary and context from national copyright experts. Copyright Watch will help document how legislators around the world are coping with the challenges of new technology and new business models.

Access Copyright Watch

See Also: Keeping a Global Eye on Copyright Law

Source: Electronic Frontier Foundation
Hat Tip: Library Stuff

Three NARA blogs

+ Collaborate (Blog and Forum for Teachers)
+ NARAtions (A Blog about Online Public Access to the Records of the U.S. National Archives)
+ RACO 2009 Blog (Records Administration Conference)

Eight Facebook Pages

+ US National Archives
+ Research at the US National Archives
+ Federal Register
+ Eisenhower Presidential Library and Museum
+ John F. Kennedy Presidential Library and Museum
+ Jimmy Carter Presidential Library
+ George Bush Presidential Library Foundation
+ Clinton Presidential Center

One Flickr Account

Four RSS Feeds

+ National Archives News Subscribe
+ Prologue Magazine Subscribe
+ The Federal Register Public Inspection List Subscribe
+ Today’s Document from the National Archives Subscribe

Three Twitter Feeds

+ National Archives News
+ Federal Register
+ RACO 2009 on Twitter (Records Administration Conference)

Four YouTube Channels

+ National Archives on YouTube
+ John F. Kennedy Presidential Library and Museum
+ Franklin D. Roosevelt Presidential Library and Museum
+ Harry S. Truman Library and Museum

Access the Complete List (with Links)

Source: NARA

1) At the end of April, 2009, Google announced that they would start adding “public” data to search results “when available.” They launched with a couple of datasets from the U.S. Census and the U.S. Department of Labor. Here’s a Census result and one with labor data. Trigger words that would show the data have to be precise. For example, unemployment in California does not return a result with labor stats but unemployment Rate in California does.

It would be very useful for info pros and researchers if Google would provide a list of what data sets are available and what trigger words have to be used to get results that include “public data.”

Since the initial launch we’ve heard almost nothing about the program until today.

As of today, Google’s main database includes content (17 development indicators to be precise) from the World Bank’s Development Indicators (WDI) (2009 edition that looks as if it will be updated regularly). The complete WDI has over 800 indicators. Although the full text book is fee-based they do make some indicators available in this free “Quick Query” database and this VERY COOL WDI visualization tool.

The Google blog post points out seven indicators and the terms to trigger them:
1) [gdp of indonesia]
2 [life expectancy brazil]
3) [rwanda's population growth]
4) [energy use of iceland]
5) [co2 emissions of iceland]
6) [gdp growth rate argentina].
7) [internet users in the united states]

Here’s a search for GDP of Canada and as promised, you see the stat and a graph at the top of the page. If you click on the graph, you’ll be taken to a page where you can select a country or countries and see a graph comparing the statistic. Here’s an example. Links to email, IM, etc. as a well as code to embed the graph are located top right on the graph, “labeled ” link.

The indicators are:
+ CO2 emissions per capita
+ Energy use per capita
+ Electricity consumption per capita
+ Exports as percentage of GDP
+ Fertility rate
+ GNI per capita in PPP dollars
+ Gross Domestic Product
+ Gross National Income in PPP dollars
+ GDP deflator change
+ GDP growth rate
+ Imports as percentage of GDP
+ Internet users as percent of population
+ Life expectancy
+ Military expenditure as percentage of GDP
+ Mortality rate, under 5
+ Population
+ Population growth rate

Almost forgot. The two datasets that Google began the program with from the U.S. Census and Dept. of Labor are still available. According to this page, no new content from these organizations have been added to the database.

2) In other Google news, the company released a new feature allowing their SafeSearch filter to be “locked down in the “Strict Search” mode, the highest level of filtering. A password is required to change the setting. Additionally, pages will have drawings of colored balls (in Google’s colors) on them to indicate that Strict Search is on and locked.

From a Blog Post:

Even from across the room, the colored balls give parents and teachers a clear visual cue that SafeSearch is still locked. And if you don’t see them, it’s quick and easy to verify and re-lock SafeSearch.

To change settings, head to the “search settings” page. If you’re going to use the new “search lock” feature, it would also be a good idea to take a look at this 95 second video.

Although it’s unlikely that Google would release the numbers, we wonder what the adoption rate of this service will be in 4-6 months. Will schools use it in addition to any third party filtering they use? Will Google continually market the service (perhaps even with paid ads) or will it only get major attention in the search industry press for a few days?

Source: Official Google Blog

For those of you who have never visited and/or used the massive amount of data OpenSecrets provides. Here’s how the site describes itself:

OpenSecrets.org is your nonpartisan guide to money’s influence on U.S. elections and public policy. Whether you’re a voter, journalist, activist, student or interested citizen, use our free site to shine light on your government. Count cash and make change.

A blog post from OpenSecrets.org says this first version of the mobile site has profiles of members of congress (it’s searchable) and also provides access to the always interesting Capital Eye Blog. The post is clear in saying that more sections and data will be added to the mobile site. We will be waiting and watching.

The site should auto-detect your mobile web browser or you can go directly to: http;//m.opensecrets.org

Source: Capital Eye Blog / OpenSecrets.org

From the Summary

As a result of requirements for both electronic and paper-based versions of congressional documents, GPO oversees an information distribution process that produces and distributes most of the congressional information for which it is responsible in both electronic and printed forms. This process provides the necessary information and appropriate formats for Congress to carry out and document its activities, but it may also result in some unwanted printed copies of congressional documents being delivered to congressional users who prefer to access those resources electronically. More broadly, the transition to electronic distribution of materials may raise questions about the capacity of current law and congressional practices to effectively oversee GPO’s management and distribution responsibilities regarding congressional information.

This report, which will be updated as events warrant, provides an overview and analysis of issues related to the processing and distribution of congressional information by the Government Printing Office. Subsequent sections address several issues, including funding congressional printing, printing authorizations, current printing practices, and options for Congress. Finally, the report provides congressional printing appropriations, production, and distribution data in a
number of tables.

Source: Congressional Research Service (via Secrecy News)
Hat Tip: FGI Blog

From the Article:

Handling congressional papers is no easy task. While presidential libraries maintain a relatively static collection, the Center for Legislative Archives’ holdings increase every time Congress passes a bill, discusses proposed legislation, confirms a presidential appointee, or does anything at all.

When the House closes its congressional sessions every two years, all the House’s related documents—paper or otherwise—are organized, held on site for four years, and then shipped to the National Archives Building for storage.

The Senate delivers the boxes in accessions sporadically. “Accessions can vary from a box to 300 boxes” explains Matt Fulghum, the Center’s assistant director, adding that the Center receives “four or five hundred accessions in one year.”

More than 13 million pages will arrive this year, says Richard Hunt, the Center’s director. “From the 1980s up to the present, our holdings of House and Senate records have been doubling every 10 to 15 years,” a nearly incomprehensible amount considering the Center currently has a half-billion documents to track, enough to circle the globe three times, if one laid the pages end-to-end.

As if organizing and storing these documents weren’t difficult enough, the Center’s team has 24 hours to fill requests from committees. They made 187 of these loans—totaling more than 1 million pages—in fiscal year 2008 alone. And that’s not all.

Much More in the Full Text Article

Source: Prologue (National Archives and Records Administration)

See Also: Center for Legislative Archives Web Page