Carnegie Mellon University

Anupam Datta

Anupam Datta

Adjunct Professor, Electrical and Computer Engineering

  • Building 23 Silicon Valley
  • 412-268-4254
Address 5000 Forbes Avenue
Pittsburgh, PA 15213

Bio

Anupam Datta is an Adjunct Professor of Electrical and Computer Engineering at Carnegie Mellon University. His research focuses on enabling real-world complex systems to be accountable for their behavior, especially as they pertain to privacy, fairness, and security. His work has helped create foundations and tools for accountable data-driven systems. Specific examples include an accountability tool chain for privacy compliance deployed in industry, automated discovery of gender bias in the targeting of job-related online ads,  and principled tools for explaining decisions of artificial intelligence systems. Datta serves as lead PI of a large NSF project on Accountable Decision Systems, on the Steering Committee of the Conference on Fairness, Accountability, and Transparency in socio-technical systems, and as an Editor-in-Chief of Foundations and Trends of Privacy and Security. He obtained Ph.D. and M.S. degrees from Stanford University and a B.Tech. from IIT Kharagpur, all in Computer Science.

Education

Ph.D., 2005 
Computer Science 
Stanford University

M.S., 2002 
Computer Science 
Stanford University

BTech, 2000 
Computer Science and Engineering 
Indian Institute of Technology, Kharagpur

Research

My research focuses on the scientific foundations of security and privacy. I formalize security and privacy properties, design mechanisms for enforcing these properties, and principled analysis techniques for rigorously demonstrating that proposed mechanisms achieve desired properties. Much of my work has focused on reducing reasoning about security to reasoning about programs with suitable abstractions of security mechanisms (e.g., cryptographic primitives, hardware-based security features) and adversary capabilities (e.g., see my work on cryptographic protocols and trustworthy systems). More recently, I have started formally investigating the role of people in security and privacy (e.g., see my work on privacy, audit, and accountability).

Privacy, Audit and Accountability

Privacy has become a significant concern in modern society as personal information about individuals is increasingly collected, used, and shared, often using digital technologies, by a wide range of organizations. One goal of this project is to precisely articulate what privacy means in various settings, and whether and how it can be achieved. In other words, we seek to develop conceptual and technical frameworks in which privacy notions (policies) are given precise semantics, algorithms for enforcing such policies, and characterizations of classes of policies that can or cannot be enforced. In addition to general results of this form, another goal of the project is to study specific application domains that raise significant privacy concerns in modern society and to apply these results (or specialized versions thereof) to these domains. Our current focus is on the healthcare domain. We are also thinking about privacy issues on the web and in online social media.

Specifically, to mitigate privacy concerns, organizations are required to respect privacy laws in regulated sectors (e.g., HIPAA in healthcare, GLBA in financial sector) and to adhere to self-declared privacy policies in self-regulated sectors (e.g., privacy policies of companies such as Google and Facebook in Web services). We investigate the possibility of formalizing and enforcing such practical privacy policies using computational techniques. We formalize privacy policies that prescribe and proscribe flows of personal information as well as those that place restrictions on the purposes for which a governed entity may use personal information. Recognizing that traditional preventive access control and information flow control mechanisms are inadequate for enforcing such privacy policies, we develop principled audit and accountability mechanisms with provable properties that seek to encourage policy-compliant behavior by detecting policy violations, assigning blame and punishing violators. We apply these techniques to several US privacy laws and organizational privacy policies, in particular, producing the first complete logical specification and audit of all disclosure-related clauses of the HIPAA Privacy Rule.

Trustworthy Systems

The security universe includes a large class of computer systems (cryptographic protocols, trusted computing systems, hypervisors, virtual machine monitors, and Web browsers, to name a few) that are designed to provide security properties in the presence of actively interfering adversaries. A unifying theme of this work is to develop theories of security that include formal models of systems, adversaries, and properties, and support rigorous analyses indicating that the system satisfies the intended security property or identifying attacks on it. Given the complexity of these systems, two central classes of techniques that we have developed to achieve scalability are (a) composition techniques that enable us to conduct security analysis of complex systems by analyzing the smaller components from which they are built; and (b) abstraction techniques that enable us to reduce the security analysis of a complex system to that of a simpler system. The techniques are provably sound, i.e. no attacks are missed by applying them. We have applied these techniques to several classes of systems: (a) trusted computing systems - proving attestation properties and discovering a composition attack on two standard protocols; (b) hypervisors -discovering attacks that violate address separation properties and proving absence of attacks on the fixed designs; (c) network protocols - proving authentication and confidentiality properties of the OpenSSL handshake implementation and rediscovering a version rollback attack on it.

Cryptographic Protocols

Protocols that enable secure communication over an untrusted network constitute an important part of the current computing infrastructure. Common examples of such protocols are SSL, TLS, Kerberos, and the IPSec and IEEE 802.11i protocol suites. SSL and TLS are used by internet browsers and web servers to allow secure transactions in applications like online banking. The IPSec protocol suite provides confidentiality and integrity at the IP layer and is widely used to secure corporate VPNs. IEEE 802.11i provides data protection and integrity in wireless local area networks, while Kerberos is used for network authentication. The design and security analysis of such network protocols presents a difficult problem. In several instances, serious security vulnerabilities were uncovered in protocols many years after they were first published or deployed.

We have developed Protocol Composition Logic (PCL), a formal logic for proving security properties of such network protocols. Two central results for PCL are a set of composition theorems and a computational soundness theorem. In contrast to traditional folk wisdom in computer security, the composition theorems allow proofs of complex protocols to be built up from proofs of their constituent sub-protocols. The computational soundness theorem guarantees that, for a class of security properties and protocols, axiomatic proofs in PCL carry the same meaning as reduction-style cryptographic proofs. Tool implementation efforts are also underway. PCL and a complementary model-checking method have been successfully applied to a number of internet, wireless and mobile network security protocols developed by the IEEE and IETF Working Groups. This work identified serious security vulnerabilities in the IEEE 802.11i wireless security standard and the IETF GDOI standard. The suggested fixes have been adopted by the respective standards bodies.

Keywords

  • Scientific foundations of security and privacy
  • Accountable machine learning

Related news

Thursday, February 25, 2021

Datta to Chair National Academies Workshop on Trustworthy AI

National Academies will convene “Assessing and Improving AI Trustworthiness: Current Contexts, Potential Paths,” a public workshop sponsored by the National Institute of Standards and Technology and chaired by Datta, to help think through the challenges associated with building trustworthy AI.
Friday, August 14, 2020

Datta Featured on AI

Anapam Datta was featured in VentureBeat on his AI company, Truera.
Monday, December 09, 2019

Six things you should know about AI from experts in the field

Researchers from Carnegie Mellon University’s College of Engineering share what they have learned about artificial intelligence while working in the field—from what led to the explosion of AI applications, to where it could have the biggest impact in the future, to areas still ripe for discovery.
Monday, February 04, 2019

Thwarting bias in AI systems

As machine learning systems are used more and more to make decisions about insurance, criminal justice, credit, and child welfare, we need to ensure that they are fair.
Monday, September 24, 2018

Providing secure and resilient naval software

CMU Team Receives $7.5M Office of Naval Research Grant in Collaboration with Penn and Stanford on Software Complexity Reduction.
Monday, August 27, 2018

Datta on discriminatory algorithms

Techfestival reported on ECE’s Anupam Datta’s research illuminating discriminatory biases in algorithms.
Wednesday, June 06, 2018

Datta quoted on privacy and encryption

Scientific American quoted Anupam Datta on privacy and encryption issues with AI assistants.
Wednesday, May 09, 2018

Datta cited by Fortune on the fairness of AI

An article in Fortune cited Anupam Datta and discussed the power of AI to create fair and unfair virtual and real-world experiences.
Wednesday, May 09, 2018

Datta cited by New Scientist on biased algorithms

A study by Datta demonstrated that Google’s online advertising algorithm would show ads for high-income jobs to more male than female users.
Tuesday, February 27, 2018

Datta stresses internal processes of AI

Anupam Datta was featured in a story in The Economist discussing the push to understand why artificial intelligence (AI) agents make the decisions they do.
Tuesday, February 13, 2018

Datta's study cited in New York Times book review

In his study, Datta found that, when an equal number of men and women visited 100 recruitment sites, men were shown ads for the highest-paying jobs six times more often than women.
Monday, December 18, 2017

Datta’s 2015 algorithm bias study referenced by HuffPost UK

Embrace an android. That’s the rallying cry of the UK’s deputy Labour leader, Tom Watson. His party’s Future of Work Commission recently released a study finding, in part, that the “increasing use of hi-tech machines could create as many jobs as it destroys,” but only if the government invests in training low-skilled workers to fill those jobs. With the rise of automation in the workplace, the Commission also recommended the establishment of ethics and algorithm training to prohibit discrimination by algorithm.
Friday, December 15, 2017

Datta’s research referenced in story addressing AI liability

In 2015, Anupam Datta co-authored a study that revealed digital ads for high-paying jobs were shown to significantly more men than women.
Tuesday, November 28, 2017

Datta study cited in article about fairness in AI

Recent advancements in artificial intelligence have revealed the presence of bias within the learning processes of neural networks. Anupam Datta conducted a study in 2015 that offers proof: in certain settings, Google ads that promised help for applicants in getting jobs with salaries greater than $200,000 were shown to significantly fewer women than men.
Friday, September 29, 2017

Datta quoted on machine bias

CyLab/ECE’s Anupam Datta was quoted in Science News on machine bias. With increased dependence on machine-learning, algorithms also pick up biases along the way. But is it possible to get a completely unbiased algorithm?
Monday, August 14, 2017

Datta leads NSF project on accountable decision systems

Increasingly, decisions and actions affecting people's lives are determined by automated systems processing personal data. Excitement about these systems has been accompanied by serious concerns about their opacity and threats they pose to privacy, fairness, and other values.
Thursday, March 16, 2017

Datta publishes article on automated decision-making tasks

Anupam Datta recently published an article in The Conversation on automated decision-making tasks. Specifically, Datta explored the issue of using machine learning algorithms for credit decisions.
Monday, August 15, 2016

Datta and Datta’s study featured in the Guardian

The study found that Google is more likely to advertise executive-level salaried positions to search engine users if it thinks the user is male.
Wednesday, June 08, 2016

Datta’s system to test bias in algorithms featured in Engadget

ECE’s Anupam Datta’s Quantitative Input Influence system was featured in Engadget. The system, detailed in a paper he presented at the IEEE Symposium on Security and Privacy, detects bias by testing algorithms for the influence of a given variable over the outcome.
Wednesday, June 01, 2016

Transparency reports make AI decision-making accountable

Machine-learning algorithms increasingly make decisions about credit, medical diagnoses, personalized recommendations, advertising and job opportunities, among other things, but exactly how usually remains a mystery.
Tuesday, May 17, 2016

Datta featured in Communications of the ACM for creating AdFisher

Anupam Datta, associate professor of computer science and of electrical and computer engineering, is featured in Communications of the ACM for creating AdFisher, a program that simulates browsing behavior and collects information about the ads returned after Google searches.
Wednesday, March 16, 2016

CMU, Stanford researchers devise method to share password data safely

An unfortunate reality for cybersecurity researchers is that real-world data for their research too often comes via a security breach. Now computer scientists have devised a way to let organizations share statistics about their users’ passwords without putting those same customers at risk of being hacked.