ML Models for Safety-Critical Systems with Lucas García - #705

Update: 2024-10-14

Digest

This episode of the podcast delves into the complex world of integrating AI models into safety-critical systems, a topic of growing importance in various industries. The guest, Lucas Garcia, a Principal Product Manager for Deep Learning at MathWorks, shares his insights based on his presentation at the NURPS conference. The discussion begins by clarifying the distinction between verification and validation in software development, emphasizing their crucial role in safety-critical systems. Lucas introduces the traditional V model and the W cycle, a specialized development cycle tailored for AI-based systems. He then provides a concrete example of a battery state of charge estimator, illustrating how verification and validation steps are applied in practice. The importance of data management and learning process verification is highlighted. The episode addresses the question of why AI models are preferred over traditional methods in safety-critical systems. Lucas discusses the limitations of traditional models and the advantages of AI in terms of generalization and data-driven approaches. The W cycle workflow is broken down in detail, outlining the specific steps involved in data management, learning process management, model training, learning process verification, implementation, and integration. Lucas further discusses the role of external specifications and regulations in safety-critical systems, highlighting the need for certification and the evolving standards for AI-based systems. The use of formal methods in AI is explored, emphasizing their importance in proving the robustness of AI models. The Deep Learning Toolbox Verification Library, which uses formal verification methods to test the stability of deep learning networks, is introduced. The episode clarifies the role of abstract transformers in the verification process, explaining that they are modeling tools used for verification, not production models. The limitations of formal methods and the need for alternative approaches like scenario-based testing and runtime monitoring are discussed. Lucas discusses neuron coverage, a technique for assessing the coverage of neurons in a neural network. He acknowledges the challenges of connecting neuron coverage to specific objectives and requirements. The use of benchmarks in industry, highlighting the ML Leap project and the importance of real-world use cases, is explored. The use of transformers in safety-critical systems is discussed, highlighting their potential for time series modeling and embedded AI. The challenges of applying large transformers and LLMs in safety-critical environments are acknowledged. The challenges of benchmarking in safety-critical AI are discussed, highlighting the role of certification bodies like ASA and the ML Leap project. A real-world example of a runway sign classifier is provided, illustrating the incremental approach to certification. The concept of architectural mitigation, a technique for achieving higher criticality levels by combining dissimilar components, is explained. This is illustrated with the example of two independent runway sign classifiers, achieving a DALC certification. Lucas introduces constrained deep learning, an approach to training neural networks by incorporating domain-specific constraints. The benefits of this approach in terms of guaranteeing desirable properties like monotonicity and convexity are discussed. The trade-offs and challenges of using convex neural networks are discussed. The potential benefits in terms of reliability and robustness are emphasized, but the challenges of convergence speed and the need for more complex network architectures are acknowledged. The question of why convex neural networks are not more widely used is addressed. The challenges of convergence speed and the lack of a comprehensive framework for constrained deep learning are discussed. The need for more research and development in this area is highlighted. The potential for developing a library of convex models, similar to existing libraries for non-convex models, is discussed. The need for further research and development to make this a reality is emphasized. Future advancements could simplify the verification process by baking desired properties into the network architecture.

Outlines