Open Government Data: The Book

By Joshua Tauberer. Second Edition: 2014.
Also available as a Paperback and for Kindle. Tweet me at @JoshData.

Civic Hacking

Civic hacking is a creative and often technological approach to solving civic problems.1 These civic problems run the gamut from voter registration and public education to helping consumers buy homes and choose financial advisors.2 Often civic hacking involves the use of government data to make governments more accountable, but the goals of civic hacking are as diverse as those who might call themselves hackers. Civic hackers can be programmers, designers, data scientists, good communicators, civic organizers, entrepreneurs, government employees and anyone willing to get his or her hands dirty solving problems. Some civic hackers are employed by nonprofits, such as Code for America. Some work for innovative for-profit companies, such as the geospatial software provider Azavea in Philadelphia. Others are civic hackers only by night.

This use of “hacking” might surprise you. The word “hacking” is a homonym. It has many meanings, in the same way that other words such as “mouse” and “fluke” have multiple meanings. The most familiar meaning of “hacking” means a computer crime, but that’s not the meaning of “hacking” in the phrase civic hacking. Here “hacking” means perverting something’s original purpose to solve a problem. Rube Goldberg machines are made up of hacks. The first computer games were hacks (computers were not meant for games). The website IKEAHackers.net shows how one can turn a pillow into a child’s costume, and ParentHacks.com offers counter-intuitive cooking tips. The best hacks are simple, creative solutions to interesting problems.

This meaning is a source of pride among programmers and geeks and actually was the original meaning of the word3 before it also became used to mean cybercrime. (Not convinced that words can have unrelated meanings? Hacking can also mean to cough, and there is also a party hack or a hack journalist, among several other meanings.)

Here are some problems that civic hackers work on:

In Uganda, improved public access to information about government funding of public schools increased school enrollment and test scores.4 A random audit of Brazilian mayors was found to affect elections most in municipalities where the audit was disseminated widely.5 Data on pedestrian injuries in San Francisco showed that “5 percent of the city streets account for 55 percent of the injuries weighted for severity.”6 And in Boston, a Code for America fellowship team built a public school eligibility website to help parents through school selection that, said a school official, “changed the way [the School Department] relates to parents.”7 The dissemination of government information leads to a wide range of improved policy outcomes.8 Local data helps neighborhood organizations make practical decisions (for more see Informing Policy Decisions).

Since so much of our lives is now about interfacing with our government, much of civic hacking is about improving what Cyd Harrell calls the “citizen experience.” Hacking the civic experience means re-imagining how our interactions with government could be improved if governments were better at respecting “citizen’s dignity and time” and citizens’ willingness to participate in governance.9 Civic hacking isn’t just coding: Hacking the citizen experience is a design problem requiring thinking about processes and interactions. Let’s rethink the process of registering to vote: TurboVote will send you text message reminders before elections. Let’s rethink how our government investigates predatory financial services: The U.S. government’s Consumer Financial Protection Bureau built an all-new complaint system that smoothly and painlessly walks users through the process of having their complaint investigated.

Civic hacking using government data in particular has implications far beyond our experience with government. It contributes to the national economy, helps consumers be more informed, and makes our own government more efficient. By empowering citizens to perform their own market oversight, for instance, we reduce the need for regulations and the bureaucracy that regulations create.10 The business world relies on XML corporate disclosures from the Securities and Exchange Commission that keep investors informed. The majority of Freedom of Information Act requests is in fact made for commercial purposes, such as competitive research.11

Government data can empower consumer choice. The real estate websites Zillow, Realtor.com, Trulia, Estately, and Redfin draw on government data. Writing on the White House blog in 2013, U.S. Chief Technology Officer Todd Park and U.S. Chief Information Officer Steven VanRoekel explained,

Zillow is powered, in part, by open government data – including freely available data from the Bureau of Labor Statistics, Federal Housing Finance Agency, and the Census Bureau. Zillow uses these data sets to do things like help home buyers in a given region understand the point in years at which buying a home is more financially advantageous than renting the same home.

The private sector has long known the value of government-produced information, though treating government as a platform was only coined “Gov 2.0” recently.12 Environmental, weather, occupational safety13, and health-related data have obvious practical consequences for public health. Healthcare IT News reported in early 2012 about two recent winners of a contest14 to develop health-related applications:

The winning apps . . . were each awarded $20,000 by [the Department of Health and Human Services’s] Office for the National Coordinator for Health Information Technology (ONC). They are:

Ask Dory! Submitted by Chintan Patel, Sharib Khan, MD, and Aamir Hussain of Applied Informatics, LLC, the app helps patients find information about clinical trials for cancer and other diseases, integrating data from ClinicalTrials.gov and making use of an entropy-based, decision-tree algorithm. . . .

My Cancer Genome. Submitted by Mia Levy, MD, of the Vanderbilt University Medical Center, the app provides therapeutic options based on the individual patient’s tumor gene mutations, making use of the NCI’s physician data query clinical trial registry data set and information on genes being evaluated in therapeutic clinical trials. The app is in operation at MyCancerGenome.org.15

Information is a crucial driving force in innovation and is a unique kind of resource because it is a so-called “public good.” Consumption of information by one individual does not reduce the availability of information for others. This is why the benefit of information can extend far beyond its initial purpose. Governments as major producers of information are in a strong position to spur innovation by promoting open government data. Open data is a public good and a sort of civic capital that leads to these extraordinary ideas, which are often unpredictable from the data they chose to use.

App development has lately been spurred by contests or “challenges.” The first I was aware of, back in 2008, was Apps for Democracy in Washington, D.C. Apps for Democracy put up $20,000 in prizes for applications built using the city government’s newly opened data. iStrategyLabs, which worked with the D.C. government to create the contest, said that the contest entries — including one mobile app to submit GPS-tagged photos of potholes and other city problems to the city’s 311 service — would have cost the government $2 million to build, 40 times the amount of money the D.C. government actually spent on encouraging the apps to be created.16

Solving real-world problems often takes years of deliberate effort, though. iStrategyLabs’s estimate of a 40-times return on investment for the DC government was a public relations exaggeration. Most app contests do not produce apps that actually solve problems. The same is true of “hackathons,” one- or two-day community-run problem-solving events typically held around a particular theme. At least, it is true when you look at a single contest or hackathon in isolation. But each is a small part of an incredibly broad movement.

At a hackathon I ran in Philadelphia in 2009, two analysts from the New Jersey State Police worked over the course of two days with five volunteer programmers to develop a visualization tool for gang activity tracked by the police department. On that same day, nearly 200 developers across the country were participating in the Sunlight Foundation’s call for a Great American Hackathon. Hackathons are growing in popularity. On in February 2014, hackathons were held in about 100 cities. Ours in Washington, D.C.17 had more than 300 participants working on projects to address government corruption, improving the public education system, cataloging open data, and teaching others data skills.

These short events strengthen the connections among local hackers and help orient them to the complexities of civic problems. That is especially true when subject matter experts, especially those in government, participate, as in the case of our gang crime hackathon in 2009. A weekend hackathon may only be a tiny portion of the time needed to make progress on a real problem, but a hackathon may not be the beginning and the end of the time spent on it — often it is a pit stop for participants that have long been invested in a problem. And while technology may be only a small part of solving most real world problems, the use of information technology is becoming more and more important to finding solutions as our world becomes more and more complex. There are few problems left that can be understood without data and information processing tools.18

  1. Alex Howard defined civic innovation as “a new idea, technology or methodology that challenges and improves upon existing processes and systems, thereby improving the lives of citizens or the function of the society that they live within.” For more discussion: http://gov20.govfresh.com/defining-civic-innovation-definition-open-government/, March 16, 2012.

  2. BrightScope.com uses public regulatory filings to compute metrics such as experience and conduct for financial advisors. Also see: Ryan Alfred and Mike Alfred. 2013. From Entrepreneurs to Civic Entrepreneurs, in Beyond Transparency, http://beyondtransparency.org.

  3. Eric Steven Raymond. 2000. A Brief History of Hackerdom.

  4. draft of Ritva Reinikka and Jakob Svensson. 2011. The Power of Information in Public Services: Evidence from Education in Uganda. In Journal of Public Economics. http://people.su.se/~jsven/PInformation2010.pdf

  5. Claudio Ferraz and Frederico Finan. 2008. Exposing Corrupt Politicians: The Effects of Brazil’s Publicly Released Audits on Electoral Outcomes. In The Quarterly Journal of Economics. http://eml.berkeley.edu/{\textasciitilde}ffinan/Finan_Audit.pdf

  6. Emily Badger. The Best Open Data Releases of 2012. In The Atlantic Cities. http://www.theatlanticcities.com/technology/2012/12/best-open-data-releases-2012/4200/

  7. Joel Mahoney. 2013. Open Data and Open Discourse at Boston Public Schools, in Beyond Transparency. http://beyondtransparency.org.

  8. Guo Xu. 2012. The Benefits of Open Data — Evidence From Economic Research. In Open Knowledge Foundation Blog. http://blog.okfn.org/2012/10/05/the-benefits-of-open-data-evidence-from-economic-research/

  9. Cyd Harrell. 2011. The Citizen Experience Needs Us: Why UX practitioners should join the Government 2.0 movement, in UX Magazine.

  10. an idea I first saw in a paper by Ed Mayo, chief executive of the U.K. National Consumer Council, and Tom Steinberg, the director of mySociety, the U.K.’s digital-government-transparency non-profit. http://www.opsi.gov.uk/advice/poi/power-of-information-review.pdf, p39

  11. http://www.cjog.net/documents/Who_Uses_FOIA.pdf

  12. For more on Gov 2.0, see Alex Howard’s blog post Defining Gov 2.0 and Open Government, January 5, 2011, http://gov20.govfresh.com/social-media-fastfwd-defining-gov-2-0-and-open-government-in-2011/.

  13. Records of mine safety inspections came to focus in 2010 after the explosion in the Massey Energy mine in West Virginia, the largest coal-mining disaster in 40 years in the country. Because there was an active mine-safety beat reporter covering the area, it stands to reason that the disaster might have been averted had the safety inspection documents submitted by Massey to the Mine Safety and Health Administration been made thoroughly available to the public as they were produced. See Dan Froomkin. June 14, 2010. Was Lack Of Government Transparency A Factor In Mine Deaths?, The Huffington Post; Tim Huber. June 29, 2011. Massey Kept Fake Safety Records To Throw Off Inspectors, Mine Disaster Victim’s Fiancee Says Of Latest MSHA Findings. The Huffington Post.

  14. Todd Park, mentioned above as the federal chief technology officer, was prior to that the chief technology officer at the Department of Health and Human Services where he spearheaded open data efforts including this contest.

  15. Mike Millard. January 5, 2012. App challenge winners harness public data for cancer treatment. Healthcare IT News.

  16. The D.C. government’s chief technology officer, who led the city government’s side of the contest, was Vivek Kundra. Kundra had created the D.C. Data Catalog the year before and became the federal chief information officer, Steven VanRoekel’s predecessor, the year after.

  17. which I co-organized with Eric Mill, Sam Lee, Kat Townsend, and Julia Bezgacheva and held at The World Bank, see http://dc.opendataday.org.

  18. For more about this, see Mark Headd’s post The Year That Civic Hacking Changed Everything.