Data breaches, fake news, surveillance capitalism… In 2020, data science finds itself in the crosshairs of a massive techlash. The intensity of feelings may be attributable, at least in part, to a sense that data scientists are no more accountable for the impact of their work than the inscrutable algorithms themselves.
What might an ethical data culture look like in the civic technology ecosystem? At Code for America, we rely on data to ensure that all of our decisions are informed by evidence. This is an opportunity for us to show what’s possible.
Our data team set out to create a statement of ethics for Code for America that can serve as a model for our colleagues in civic tech and government. Our audacious goal is not just to spark the kinds of ethical conversations and commitments the digital age requires, but to promote a vision of data professionalism with accountability as a core value. We need a “Hippocratic oath ‘with teeth,’” (to borrow a phrase from Cathy O’Neil, author of Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy): a framework such that a data scientist’s professional reputation depends on their ethical behavior as well as their technical and statistical skills.
Our statement of Ethical Data Use at Code for America, which you can read here, outlines the principles of ethical data use. We categorize these principles as relating to:
- Privacy, security, and informed consent
- Data collection and the risk of bias
- Analysis/modeling and the risk of harm to certain groups
These principles are our attempt to identify the points in the pipeline of data work where ethical concerns may arise.
We presume that these principles are not controversial in our organization, and we trust that our colleagues’ intentions are good. The greatest risk for causing harm lies in failing to take the time to thoughtfully weigh potential risks against potential benefits. Therefore, the most important component of the statement is the mechanisms of accountability it lays out.
The metaphorical Andon cord
Andon, a Japanese manufacturing practice, is a system by which workers on a production line can alert their colleagues and management to a quality problem by pulling a cord that stops production until the issue can be addressed.
In the context of data ethics, we want everyone on a project team to be empowered to pause a process or project because of an ethical concern, without fear of retaliation. The purpose of this metaphorical “Andon cord” is to create the space for debate about the best course of action in complex situations. Our intention is to grow an organizational culture in which wrestling with such questions is as much a part of our working process as any other risk/opportunity discussion. The expectation is that the vast majority of ethical questions will be addressed and resolved in team meetings and other public forums.
The ethical veto
Rarely, a project team may not be able to resolve an ethical question. In this situation, the question goes to the data team for review and discussion. If two or more members of our four-person data team believe that a project is unethical, they use their “ethical veto” to pause the project. They write up their reasons in a memo to our executive team. The executive team meets with the data team within the week to discuss the recommendation. If the executive team agrees, the project is discontinued. If the executive team disagrees, the question is brought to an outside expert for review and guidance. The executive team will document their final decision and reasons in writing.
The “ethical veto” is the critical lever of professional accountability. The data team can be held accountable for the impact of the organization’s data work only because they are explicitly empowered to stop a project that they believe is unethical. The ethical veto provides the “teeth” to our Hippocratic oath.
Hiring checks
When making hiring decisions for the data team, we explicitly seek out evidence of a candidate’s attention to the ethical use of data, and when providing references for our colleagues, we commit to including assessments of their commitment to ethical data practices.
Our intention with this policy is to signal the potentially important role of professional reputation in shaping the behavior of organizations. We believe that, like medical and scientific researchers, data professionals have a particular ethical responsibility in their organizations. A data practitioner’s professional reputation should reflect their ethical standards as well as their analytical work.
In conclusion
All of us in civic technology and government have a special obligation to attend thoughtfully to the impact of our work, including our work with data, on individuals and on society. We believe that, in an organizational culture that is open to and practiced in discussing ethical issues, the metaphorical “Andon cord,” the “ethical veto”, and hiring checks for data professionals provide mechanisms of accountability that help us all meet that obligation.