AI Prototypes Code for ESP32
A developer is prototyping a system that uses local Qwen LLMs to generate and execute code for ESP32-based devices like remote controls and lightbulbs. The AI can create novel behaviors, like rainbow light patterns or Morse code, and make them persist across device reboots.
The use of local Qwen LLMs for on-device code generation on the ESP32 marks a significant step in edge AI. The Qwen models, developed by Alibaba Cloud, are a family of large language and multimodal models. Running these models locally reduces reliance on the cloud, which in turn decreases latency and enhances data privacy. This approach leverages the growing field of TinyML, which focuses on running machine learning models on resource-constrained microcontrollers. Frameworks like TensorFlow Lite for Microcontrollers, Edge Impulse, and Espressif's own ESP-DL have made deploying AI on the ESP32 more accessible. The ESP32-S3 variant is particularly well-suited for this, with vector instructions that accelerate the matrix multiplication and convolution operations common in neural networks. While generating functional code with AI is becoming faster, ensuring its efficiency and reliability on embedded hardware remains a challenge. AI-generated code often prioritizes readability over microarchitectural efficiency, which can lead to issues with timing, memory access, and performance under load. For safety-critical applications, direct use of AI-generated code is still approached with caution. The demonstration of a 260,000-parameter Llama-based model on an ESP32-S3 highlights the feasibility of running smaller LLMs on microcontrollers. This was achieved by using both of the chip's 240 MHz cores and leveraging the ESP-DSP library to speed up crucial calculations. Such advancements are part of a larger trend of moving AI processing from data centers to the edge, a market expected to grow significantly.
