ICTworks

USAID published their Open Data policy, ADS 579, and have begun including language in all contracts and cooperative agreements after October 1, 2014 to require partners to abide by the policy.

There have been many more questions than answers at this stage, which is to be expected when any new policy is rolled out. For those of us in the USAID contractor community, especially small businesses, unexpected change and lots of questions do not make us rest easy. Before you fret, we at Sonjara are experienced with open data evaluations and compliance, and have developed a quick overview for what the new Open Data policy means for you.

Who do USAID’s Open Data rules apply to?

All contracts and cooperative agreements awarded on or after October 1, 2014. Many contracts are having these modifications also rolled out retroactively via the IPN. To quote the ADS 579 page 10:

USAID staff, as well as contractors and recipients of USAID assistance awards (e.g. grants and cooperative agreements) must submit any Dataset created or collected with USAID funding to the DDL in accordance with the terms and conditions of their awards.

This is in keeping with Executive Order 13642 and the OMB Open Data Policy (M-13-13) which states that an agency’s “public data listing may also include, to the extent permitted by law and existing terms and conditions, Datasets that were produced through agency-funded grants, contracts, and cooperative agreements.

What specific data do they want?

USAID is looking for organized collection of structured data, including data contained in spreadsheets, whether presented in tabular or non-tabular form (e.g. single spreadsheet, an extensible mark-up language (XML) file, a geospatial data file, or an organized collection of these).

In general, they want data supporting “intellectual work” that documents the implementation, monitoring, evaluation, and results of international development assistance activities. Examples of this include:

• Baseline Household Surveys
• School Attendance Data
• Facility Surveys
• Cluster Samples
• Monitoring Data
• Economic Assessments

Now that doesn’t mean they want everything, as there are a few things not included:

• Unstructured data (PDF, Word, graphics, emails)
• Award administration data (financials, operations, management)
• Data already submitted to USAID via a different data repository (such as the Feed the Future Management System).

What do I need to do for my organization to be compliant?

If your contract or cooperative agreement includes language on open data, you will need to create an open data plan, created with the COR/AOR and the Data Steward for the funding Bureau or Mission.

Any datasets that are identified by this plan as needing to be published in the Development Data Library will need to be published in open data formats (i.e. machine readable) using common structures when available.

You may want to perform an open data audit of your data capture systems and processes you will use before starting the conversation with USAID, to make sure that you are not committing to publication of sensitive data or an expensive data publication method, as well making sure you are already using common data structures such as IATI.

Think about how you can use this data for internal and external purposes. Open data offers opportunities for data sharing among offices and organizations that did not exist before. There may be major benefits to your organization with having this sort of data easily available between teams.

What is Open Data?

Open data has two parts – open and data.

1. Open: means publishing data in machine-readable formats so that anyone can automatically pull that data into their software applications without a human being involved.

For an example, think weather data – we all have it on our phones, right? Where does that data come from? Well, the raw data comes originally from NOAA, in most cases as an open data feed, and companies like the Weather Channel analyze it and come up with predictions. They then publish their own open data feeds, allowing app developers like me to write the code which displays today’s weather on your phone.

2. Data: means data that is structured and in machine-readable formats (XML or JSon are the two most popular). This could be weather data as listed above, demographic information such as from the census. It could be project data, such as published on the foreign assistance dashboard. It could be M&E data from a project. The key is that the data is structured and in machine-readable formats.

You also have to consider two other key components that are included in open data definitions when you think through your open data plan.

The data has an API to allow a developer to write code that will pull specifically parts of the data they need without needing to talk to anyone. For example, a developer should be able to extract only data on Bangladesh, or for 2013 from your data set just by looking at how its structured and writing code that matches your machine-readable format.

The data must be free to be used. This means removing any data that might violate someone’s privacy or cause a security incident. Remember that open data may be a subset of data that is used internally, but not shared externally, using the rest of the open data format. This can be beneficial as it can allow internal systems to interoperate more easily. Also, data must be free in the commercial sense – developers should be able to extract the data without having to pay for it.

Big Picture: Open Data is Here to Stay

The Open Data policy is part of a larger US Government commitment to Open Data, as well as the global commitment to open data for development. Open Data is also part of the digital “big data” revolution we have been experiencing in all parts of our lives. As more and more organizations look to data-driven performance management, open data will be part of these solutions.

Open Data is cheap when designed up front and expensive to do in retrospect, so it will be built into systems from now on. Just like Y2K, once everyone gets their systems “open data compliant” by default, open data will no longer be a huge burden on data gatherers.

Open data will be a huge communal resource for data analysts and researchers all around the world. Just like we cannot imagine working in a world before Google, we won’t remember what limitations we had without all this data at our fingertips.

Open Data is one key element in improving data access, but it is really only the start. There will still need to be translators, analysts, vizualizers, and archivists needed to manage the data. And we hope to see you exploring the bold new world of Open Data with us.