Invited Talk
in
Workshop: Debugging Machine Learning Models

Safe and Reliable Machine Learning: Preventing and Identifying Failures

Suchi Saria

2019 Invited Talk
in
Workshop: Debugging Machine Learning Models

Abstract

Machine Learning driven decision-making systems are being increasingly used to decide bank loans, make hiring decisions,perform clinical decision-making, and more. As we march towards a future in which these systems underpin most of society’s decision-making infrastructure, it is critical for us to understand the principles that will help us engineer for reliability. Drawing from reliability engineering, we will briefly outline three principles to group and guide technical solutions for addressing and ensuring reliability in machine learning systems: 1) Failure Prevention, 2) Failure Identification, and 3) Maintenance. In particular, we will discuss a framework (https://arxiv.org/abs/1904.07204) for preventing failures due to differences between the training and deployment environments that proactively addresses the problem of dataset shift. We will contrast this with typical reactive solutions which require deployment environment data and discuss relations with similar problems such as robustness to adversarial examples.

Speaker

Suchi Saria

Suchi Saria is the John C. Malone Assistant Professor of computer science, statistics and health policy at Johns Hopkins University. She is also the founding Research Director of the Malone Center for Engineering in Healthcare at Hopkins. Saria also serves on the editorial board of the Journal of Machine Learning Research. Her research focuses on developing next-generation diagnostic and treatment planning tools that leverage statistical methods to individualize care. Towards this, her methodological work focuses on questions such as: How can we combine different sources of information with prior knowledge to derive actionable inferences? How can we characterize and improve the reliability of the resulting inferences in challenging real-world settings? How can we support decision-making in safety-critical domains? Saria joined Hopkins in 2012. Prior to that, she received her PhD from Stanford University working with Prof. Daphne Koller. Her work has received recognition in the form of two cover articles in Science Translational Medicine (2010, 2015), paper awards by the Association for Uncertainty in Artificial Intelligence (2007) and the American Medical Informatics Association (2011), an Annual Scientific Award by the Society of Critical Care Medicine (2014), a Rambus Fellowship (2004-2010), an NSF Computing Innovation Fellowship (2011), selection by IEEE Intelligent Systems to Artificial Intelligence’s “10 to Watch” (2015), the DARPA Young Faculty Award (2016), MIT Technology Review’s ‘35 Innovators under 35’ (2017), and the Sloan Research Fellowship (2018). She has given over 80 invited talks including presentations at the National Academy of Sciences, National Academy of Engineering, and the White House Frontiers Meeting. In 2017, her work was among four research contributions presented by Dr. France Córdova, Director of the National Science Foundation to Congress’ Commerce, Justice Science Appropriations Committee.

Chat is not available.