Intelligent safety-critical systems such as autonomous cars operate in the complex open world; as such, they must not only deliver excellent performance in their operational design domain, but also be robust to all unexpected inputs caused by extreme weather and lighting conditions, changes of operation domains, and rare but potentially catastrophic situations. In this talk, I will present novel approaches covering the full spectrum of settings for this challenge, ranging from domain synthesis to unsupervised domain adaptation, to test-time domain adaptation, and domain generalization. Our approaches have achieved state-of-the-art performance for a wide variety of domain changes, e.g. cross-datasets, normal-to-adverse, and synthetic-to-real. At the end of the talk, I will briefly present two benchmarks that we have developed for robust visual perception.

A simulator with diverse photorealistic 3D assets is essential for autonomous driving. While reconstructing real-world scenes is promising for photorealistic simulation, generative models further enable the effortless creation of novel objects and scenes. In this talk, I will present our recent progress in 3D-aware image synthesis of objects, humans, and scenes toward generative simulation. I will introduce several 3D-aware generative models, including VoxGRAF for efficient object generation, VeRi3D for controllable human generation, and UrbanGIRAFFE for compositional and controllable urban scene generation.

Dense visual perception of the geometry and semantics of the surrounding scene differs from basic global tasks such as classification in that it requires to distinguish fine spatial details besides aggregating context across the extent of the input. These two conflicting goals are typically pursued through encoder-decoder architectures, which involve an information bottleneck at the interface of the encoder and the decoder. Motivated by the fact that this internal bottleneck marks the limit between context aggregation and fine-grained parsing, we operate on the respective internal representations and optimize them in conjunction with the output representations by introducing dedicated modules and losses in various geometric and semantic perception settings. The talk will include a thorough review of a geometric case and two semantic cases of the above internal representation optimization paradigm. The former case consists in our internal discretization method for dense regression, which has general application on geometric tasks such as supervised monocular depth estimation and surface normal estimation. The latter cases consist in our condition-invariant methods for unsupervised domain adaptation of semantic segmentation models, which introduce feature invariance and cross-domain contrastive losses on the internal network representations.

Advances in machine learning have led to the rapid and widespread deployment of ML algorithms in safety-critical applications, such as autonomous driving and medical healthcare. Standard machine learning systems, however, assume that training and test data follow the same, or similar, distributions, without explicitly considering active adversaries manipulating either distribution. For instance, our recent work has demonstrated that motivated adversaries can circumvent anomaly detection or other machine learning models at test-time through evasion attacks, or can inject well-crafted malicious instances into training data to induce errors during inference through poisoning attacks. In this talk, I will describe different perspectives of security and safety in machine learning, such as robustness, privacy, generalization, and their underlying interconnections. I will focus on a certifiably robust learning approach based on statistical learning with logical reasoning as an example, and then discuss the principles towards designing and developing practical trustworthy machine learning systems with guarantees, by considering these trustworthiness perspectives in a holistic view. I will also introduce our unified platform SefeBench which generates diverse safety-critical autonomous driving scenarios for safety tests for autonomous vehicles.