Machine Learning driven decision-making systems are being increasingly used to decide bank loans, make hiring decisions,perform clinical decision-making, and more. As we march towards a future in which these systems underpin most of society’s decision-making infrastructure, it is critical for us to understand the principles that will help us engineer for reliability. Drawing from reliability engineering, we will briefly outline three principles to group and guide technical solutions for addressing and ensuring reliability in machine learning systems: 1) Failure Prevention, 2) Failure Identification, and 3) Maintenance. In particular, we will discuss a framework (https://arxiv.org/abs/1904.07204) for preventing failures due to differences between the training and deployment environments that proactively addresses the problem of dataset shift. We will contrast this with typical reactive solutions which require deployment environment data and discuss relations with similar problems such as robustness to adversarial examples.