Mar 2

Mar 2 Will NOAA’s “Big Data Partnership” be a Model for Other Government Agencies?

Dennis D. McDonald

Acquisitions, Agency Data, Big Data, Governance, NOAA, Open Data, Procurement, Project Management

By Dennis D. McDonald

Click or tap here to download a .pdf of this article.

Introduction

Last week’s Open Data Leaders meetup in Washington DC, sponsored by the Center for Data Innovation, was devoted to a presentation about NOAA’s “big data partnership” program by NOAA’s Dave McClure and Maia Hansen; see Challenges of Public-Private Interfaces in Open Data and Big Data Partnerships for an earlier discussion of this program.

The idea behind the program is fundamentally simple. The details of how it will operate are taking a while to work out:

NOAA has a lot of data that it knows are valuable.
NOAA is not funded to make all its data available.
NOAA is initially enlisting a small number of vendors who will work to (a) make raw data available to the public at cost and (b) engage with third parties to support development of data-based products and services that can be resold.

Working out the details

According to the presenters NOAA will be experimenting with ways to extend its data infrastructure via public-private partnership arrangements that simultaneously protect public access via a “level playing field” while promoting innovation and entrepreneurship.

Working out the legal and policy details of this is, based on what was discussed during the presentation, has, however, proven to be complicated, starting with a government procurement system that is more supportive of “buying stuff” than to working out 21st century style partnership relationships involving large quantities of weather and environmental data.

NOAA gathers more data to run its official programs than it is able to process and support for public access. It’s one thing if you are an academic or government research facility scientist with the tools and expertise to process raw data in support of a long-term academic research project. It’s quite another if you want to develop a “marketplace” incorporating all the metadata, standards, and customer support services that a near real time data commercial business might require.

NOAA can – and does — support the former on a case by case basis. The latter requires authority and resources it doesn’t currently have, hence its interest in partnering with cloud vendors and entrepreneurs.

The demand exists

NOAA has learned over the years that potential commercial demand exists for these data were they more accessible. The solution NOAA is pursuing is to structure a program whereby its data assets are pushed out to a series of anchor partners who will then – at no net cost to the government — take responsibility for providing access to entrepreneurs and resellers with an interest in developing new and innovative commercial products. Revenue from the sale of such products, as part of their agreement with NOAA, will support a “level playing field” by providing public access to the raw data.

If the system works it’s a win-win-win situation:

The cloud based anchor partners win since they get paid by the alliance partners they work with to develop and sell products and services based on NOAA data.
Alliance partners win through development and selling of new products and services based on arrangements they make with the anchor partners.
The public wins since it can access raw data directly from the anchor partners or can buy enhanced data products or services from alliance members serviced and supported by the anchor partners.

How will we know if the program is a success? According to the NOAA presenters, one measure will be when alliance partners are making money and costs associated with free public access are being covered as well.

Important assumptions

When I first heard about the program I was skeptical and felt it might not appeal to risk-averse companies. I found this view reinforced while talking with some of the companies that had expressed an interest in the initial Request for Information that NOAA published last year when it launched discussions with potential partners.

The more I’ve researched, though, the more I’ve come to believe that this is a valuable experiment that needs to be tried. NOAA is attempting something new here. Assuming that a commercial demand actually exists for the types of data that will be pumped through the cloud vendor anchor company that NOAA eventually commits to working with, and further assuming that potential vendors and resellers are in a position to develop marketable products, this program could end up stimulating demand for and access to useful – and profitable — products and services that the government, by itself, is not in a position to support.

Data program governance questions

My interest in open data program governance, based on past involvement in data management and publishing projects, suggests that the following types of questions:

What happens to cloud based public access to NOAA data if efforts to commercialize it fail or take longer than expected to evolve?
How will data governance and metadata standardization issues be addressed if no direct connection exists between the data source and the customers of data services provided by third-party partners?
Can NOAA program managers be assured they will obtain feedback from the users of the data they’re generating?
Assuming that intellectual-property “ownership” issues around “government data” are resolved, will commercial vendors still be able to maintain secrecy around their development and testing of competitive products?
Will small businesses and startups be able to compete in product development if their focus is more on public access then on commercial products?

How is this program important?

NOAA is a data intensive government agency. According to the presenters it only publishes about 10% of the 20 TB of data generated every day. While we can’t assume that all data are necessarily marketable or useful beyond the originating programs, NOAA management is convinced that a much larger market exists than it can currently serve. Other government agencies will be watching, including agencies that are not as data-intensive as NOAA. That is a good sign; NOAA is actually trying something new in how works with the private sector and the public.

Of interest is how the management and governance of this program evolves. Eventually there could be many “moving parts,” many types of data and systems, and many communities of data users and intermediaries that need to communicate and collaborate. They may not be operating under a completely centralized authority. They may instead operate under an outsourcing of authority that requires cloud vendors themselves to adhere to — and to enforce — policies and rules governing pricing and access.

How this will differ from the established program management model currently used by many government agencies remains to be seen. Details of the program have not yet been released. In the traditional contracting model agencies already outsource many day-to-day operational and oversight responsibilities to large prime contractors. These primes in turn screen and select subcontractors to carry out much government work under the direction of the prime acting on behalf of the government. The new feature of the emerging NOAA big data partnership program is that the source of funding for data access services will come not from direct taxpayer dollars that have been legislatively authorized by Congress but from the sale and marketing of new data-based services.

The federal government is certainly not new to using contractors. Also, commercialization of products and services based on work done based on public funding has a long history. For example, Federal R&D funding has long flowed to academic, nonprofit, and private sector organizations that can generate marketable products and/or intellectual property.

The open data movement is changing things

One thing that’s different with the NOAA data partnership program is not just its focus on making data available for access and exploitation via the cloud but the changing market environment in which it will be operating. One expression of this changing environment is the “open data” movement and what it says about the changing relationship between governments and citizens.

Increasingly, citizens are in a position to see into how government agencies operate as government agencies at all levels gradually “open up” data about their operations to public scrutiny and re-use. While the original basis for the open data movement may have been improved transparency and accountability, citizens (and entrepreneurs) now have the improving ability to use public data to do new and innovative things, things that may not have been anticipated by the legislators or agencies responsible for the use of public funds to generate the data.

That’s why the questions I listed earlier have so much to do with governance. The growing marketplace for open government data may also be causing us to rethink our traditional definitions of what’s inherently governmental and what isn’t.