Report Details Assurance for AI in Aerospace
A technical report by SRI International and the City University of London reviews how classical assurance principles can be applied to AI in safety-critical systems. The paper stresses the need to build explicit, structured assurance cases that link safety claims to evidence. It also highlights the importance of separating arguments about system correctness from those about dependability, which addresses behavior in the presence of faults or uncertainty.
- Current aviation standards like DO-178C, used for certifying safety-critical software, were not designed for AI and machine learning, creating significant challenges for validating data-driven systems. Industry groups such as RTCA and SAE International are actively working to develop new standards, like those from EUROCAE WG-114 and SAE G-34, to address these gaps. - Formal methods, which are mathematical techniques for specifying and verifying systems, are being explored to provide the rigorous verification required for AI in safety-critical applications. These techniques include model checking and theorem proving, which can help detect flaws in AI models that traditional testing might miss. - SRI International has a long history in AI research, including the development of the Siri virtual assistant and early mobile robotics. Their current focus includes creating human-centered, explainable AI and developing systems that can learn continuously, drawing inspiration from biological memory processes for DARPA's Lifelong Learning Machines (L2M) program. - The City University of London's Artificial Intelligence Research Centre (CitAI) specializes in Explainable AI (XAI) and Artificial General Intelligence (AGI), with a focus on real-world applications in sectors like health, transport, and cybersecurity. Their work also addresses the legal, ethical, and social impacts of AI technologies. - A key challenge in AI assurance is "catastrophic forgetting," where a system's performance on previously learned tasks degrades after it is trained on new data. Research is focused on enabling AI to learn continuously and reliably without disrupting prior knowledge, a crucial capability for systems that must adapt after deployment. - The European Union Aviation Safety Agency (EASA) is actively developing a roadmap for AI integration, acknowledging that traditional assurance frameworks are not adapted for machine learning. Their approach focuses on building trust through a systematic process for learning assurance and considering human-factor interactions with AI systems. - The concept of a "safety case" is gaining traction for AI assurance, moving beyond simple testing to build a structured, evidence-based argument that a system is acceptably safe for a specific operational context. This approach, long used in aviation and nuclear energy, helps to formalize and document the reasoning behind safety claims. - Distinguishing between accuracy and reliability is critical; a model can be highly accurate in controlled tests but unreliable in real-world conditions where data may change unexpectedly. Reliability focuses on consistent and predictable behavior, ensuring the system fails gracefully rather than catastrophically.
