New Benchmark for Federated AI Published
A new benchmark for federated learning in distributed AI-of-Things scenarios, known as FedAIoT, was published this week. The benchmark provides a reference for developing and testing on-device learning models for platforms where data privacy and low bandwidth are critical, such as in aerospace systems.
- The benchmark directly addresses the challenge of non-independent and identically distributed (non-IID) data, which is common in aerospace. For example, data from the same model of aircraft can differ due to variations in sensor calibration, operational environments, and maintenance histories. - FedAIoT includes the VisDrone dataset, featuring drone-captured images, to facilitate the development of object detection and tracking algorithms in aerial applications. This allows for testing models that could be adapted for autonomous navigation, surveillance, or inspection tasks on larger aerospace platforms. - The framework evaluates the performance of "IoT-friendly" neural network models, which are designed for resource-constrained edge devices. While specific architectures vary by dataset, the benchmark utilizes models like lightweight LSTMs and ResNets, providing a reference for deploying AI on embedded systems with limited computational power. - A key feature of the benchmark is its support for evaluating the impact of quantized training on model performance. This is highly relevant for aerospace applications where reducing the memory footprint and power consumption of AI models is critical for deployment on flight-certified hardware. - The benchmark provides a unified end-to-end framework that covers the entire federated learning pipeline, from data partitioning to hyperparameter tuning. This allows engineers to simulate and test different federated learning scenarios, such as how a model would perform when trained across a fleet of aircraft with varying data contributions. - By providing standardized methods for data partitioning and evaluation, FedAIoT allows for reproducible research and fair comparison of different federated learning algorithms. This helps in selecting the most robust and efficient algorithms for safety-critical aerospace applications. - The research addresses the issue of data imbalance in federated learning for aviation, which is crucial for tasks like fault detection where anomalies are rare. The benchmark can be used to test strategies for handling such imbalances without centralizing sensitive operational data. - NASA is exploring the use of federated learning for aviation-specific use cases, such as predicting communication demand in the national airspace, by creating cross-silo federated learning scenarios based on data from different airports.
