There’s a new report out on civic tech from the Knight Foundation making the rounds. It’s worth taking a read through the findings and the data that Knight gathered in their attempt to map the “civic tech” landscape. What I find most important — and perhaps troubling — about the report is its assumptions and approach rather than anything specific in the actual material.

What’s There

Kudos to Knight for taking on such a difficult but potentially useful exercise in attempting to map the civic tech landscape. The meat of the report — who the players are and how much donors and investors are spending on civic tech — is based on a set of spreadsheets that attempt to standardize, structure, and aggregate publicly-available data around projects and organizations operating in the civic tech space. This allows the report’s authors to come up with some round number estimates for how big the space is (nearly half a billion dollars), who the most active donors and investors are, and where the most action is (“P2P Local Sharing,” which we’ll get to shortly). Also: a big thanks to the authors for avoiding a giant PDF as their vehicle of choice for presenting the final report…this is a godsend for people like me who don’t read anything.

Airbnb is Not Civic Tech

My major worry with the report’s methodology is the assumption that “peer-to-peer sharing of resident goods and services” belongs in the bucket known as “civic tech.” This is the first time, ever, that I have heard anyone associate commercial platforms like Airbnb or Lyft with civic tech or civic innovation. Other smart folks in the civic tech space with whom I’ve consulted are similarly baffled. Why? Because none of us can identify the “civic” nature of these services and platforms. I’m not sure the companies themselves could either.

On its About page on its website, Airbnb describes itself as, “a trusted community marketplace for people to list discover, and book unique accommodations around the world — online or from a mobile phone.” Period. Airbnb is in business to make money off of people renting out their spare rooms, not to strengthen the social fabric of local communities, enhance voter engagement, or provide tools for public monitoring of government. The “collaborative consumption space” is one of for-profit ventures, not traditional co-ops. While citizen-to-citizen sharing is indeed involved, the mission of these companies is focused on maximizing profit for their investors, not any sort of experiment in building social capital.

This head scratching inclusion calls into question a number of the report’s findings and insights. First, P2P companies comprise roughly half of the overall dollars tallied by the report’s authors. So if you agree that P2P doesn’t belong in civic tech, the big overall number of $431 million shrinks by half instantly. You also lose some famous names and brands from the report, including Marissa Mayer and Jeff Bezos, both of whom have invested in P2P startups. (A cynical observer might wonder if the inclusion of P2P companies in the report’s methodology was a convenient way to sprinkle some pop tech sugar on top of an otherwise dry subject.)

Second, one of the report’s four primary “strategic implications” is a call for philanthropists to exert pressure in “addressing outdated regulations inhibiting the growth of this [P2P] sector…” Given the rising worry over P2P’s impact on labor law and health and safety regulations, this strikes me as rash. We don’t yet fully understand whether or how P2P services should fit into traditional models of public regulation; racing to undo existing safety nets serves no one except the investors and executives of these firms.

Where’s the Impact?

Setting aside the intellectual debate over P2P’s germaneness, the big (missing) elephant in the room is whether any of the other civic tech interventions are having an impact. More than any mapping exercise, these are the really important questions we need to be asking. Are voter and citizen engagement tools leading to better public policy outcomes and trust in government? Are open data transparency tools curbing the influence of special interests on the political process? Are platforms for crowdsourcing cause-related interests saving lives on the ground? The answers to those questions are what should be guiding our decisions on whether to invest more or less in civic tech, not investment trend lines or cluster analysis. If I were advising other foundations considering the leap into the civic tech space, I’d suggest to them that while the Knight mapping exercise is an interesting data point, it doesn’t give us any sense of potential return on investment or the trade-offs and opportunity costs inherent in these investments. We need much more work on the impact front, and we need it yesterday. In Knight’s defense, the scope of this particular report was never intended to answer those questions, but if they aren’t next on the list, they should be.

(Note for transparency’s sake: Global Integrity pitched a similar idea for researching the ROI and impact of open government interventions to the Knight Foundation as part of their News Challenge on open government earlier this year. That idea did not make it past the first round; another idea of ours did but was not ultimately funded.)