Edge AI Inference Accelerator: Local LLM and Vision Inference on Raspberry Pi with USB NPU

An Edge AI Inference Accelerator is a small computer that runs neural networks locally, without sending data to the cloud. This build pairs a Raspberry Pi 5 with a Hailo-class M.2 NPU module (up to 26 TOPS INT8 at about 2.5 W) that runs a quantized TinyLlama 1.1B chat model and YOLOv8n object detection offline. The problem it solves is the cloud dependency of modern AI: an always-online pipeline leaks private data, adds latency, and fails the moment the network does — a deal-breaker for anything on a drone, a doorbell camera, or a field sensor. This build demonstrates the edge-AI alternative at bench scale: quantized models split between the NPU (vision and compute-heavy layers) and the Pi's CPU (layers the NPU's op set can't run), with honest, student-measured numbers for speed and power instead of vendor claims. The rig runs off a standard 5 V supply at under 20 W total (Expected) and costs under ₹20,000 (Approximate).

Qualcomm's October 2026 announcement sets the reference: Hexagon-NPU-based inference accelerators, from add-in cards to full liquid-cooled racks, with PCIe scale-up, Ethernet scale-out, and 160 kW of rack power. The AI200 card carries up to 768 GB of LPDDR and is commercially available from 2026; the AI250, available 2027, uses near-memory computing claimed to deliver more than 10x effective memory bandwidth at lower power. The same announcement brought the Dragonfly C1000 CPU — custom Oryon cores in a 250+ core-count chiplet claimed at 2x performance-per-watt, available 2028, with a multi-year Meta deployment deal — plus a Saudi-backed Humain plan to deploy 200 MW of capacity starting 2026; Qualcomm shares jumped roughly 11% on the news. This build borrows none of the datacenter hardware and all of the idea: a dedicated inference accelerator bolted onto a general-purpose host, measured honestly at watts-per-token on a student budget.

An inference run flows through the rig like this. A prompt or an image enters the Flask dashboard, which routes it to the right pipeline: text goes to the quantized TinyLlama 1.1B model, images go to YOLOv8n. The Hailo Dataflow Compiler has already mapped each model's supported layers onto the NPU at build time — inside the Hailo-8, each neural-network layer gets its own dedicated compute-and-memory block, so activations flow through the chip in a pipeline instead of shuttling weights in and out of DRAM, and that is where the watts-per-token win comes from. Supported layers execute on the NPU at up to 26 TOPS INT8; layers outside the NPU's op set fall back to the Pi 5's quad-core Cortex-A76 cores. An inline USB power meter measures actual watts during the run, and the dashboard reports tokens/sec, time-to-first-token, watts/token and the NPU-vs-CPU speedup — every number measured on the bench rig by the student, never taken from a datasheet.

1. Power on and bring-up: the M.2 NPU module is detected only at boot — the Pi 5's PCIe link trains once at power-on, so reseat-and-reboot is part of the bring-up checklist; the dashboard confirms the HailoRT device is visible before any model loads.
2. Honest naming note: the title's "USB NPU" is this Hailo-class M.2 module on the official M.2 HAT+ — an add-in peripheral, not soldered silicon — plus the inline USB power metering; the build does not use a USB-stick NPU.
3. Models are prepared once: TinyLlama 1.1B chat is quantized to INT4/INT8 via llama.cpp/ONNX; YOLOv8n is compiled to INT8 through the Hailo Dataflow Compiler, which maps each supported layer onto its own dedicated compute-and-memory block on the chip.
4. A prompt or an image enters the Flask dashboard, which routes text to the LLM pipeline and images to the vision pipeline; one Python inference stack (HailoRT + ONNX Runtime) drives both.
5. Supported layers execute on the NPU at up to 26 TOPS INT8 (manufacturer spec); layers outside the NPU's op set fall back to the Pi 5's quad-core Cortex-A76 @ 2.4 GHz cores (manufacturer spec).
6. The inline USB power meter measures actual watts during the run while the dashboard samples latency.
7. The dashboard reports tokens/sec, time-to-first-token, watts/token and the NPU-vs-CPU speedup; perplexity on WikiText-2 and mAP@0.5 on COCO val are recorded as buyer-measured design targets — every figure measured on the bench rig, never a vendor claim.

Tech stack:

• Raspberry Pi 5 (BCM2712 · quad-core Cortex-A76 @ 2.4 GHz)
• Hailo-8 NPU module (26 TOPS INT8 · ~2.5 W · M.2)
• Quantized LLM inference (TinyLlama 1.1B · INT4/INT8)
• YOLOv8n object detection (COCO-trained · INT8 on NPU)
• Inference benchmark dashboard (Flask · live telemetry)
• USB power meter (inline · watts/token measurement)
• Python inference stack (HailoRT · ONNX Runtime)
• 3D-printed NPU carrier enclosure

The accelerator's capabilities are four. First, offline LLM chat: a TinyLlama 1.1B chat model quantized to INT4/INT8 runs fully on the bench rig — prompt in, answer out, no network call anywhere in the loop. Second, NPU vision: YOLOv8n (trained and evaluated on COCO 2017, 80 object classes) compiled through the Hailo Dataflow Compiler runs on the NPU at up to 26 TOPS INT8 (manufacturer spec), streaming bounding boxes at rates a Pi CPU alone cannot reach. Third, honest benchmarking: a Flask dashboard reports tokens/sec, time-to-first-token, watts/token and NPU-vs-CPU speedup for every run, all measured by the student on the bench rig with an inline USB power meter — never vendor claims. Fourth, hybrid execution: quantized layers the NPU's op set supports run on the accelerator; unsupported layers fall back to the Pi's quad-core Cortex-A76 cores, so real models run end to end instead of only in a demo script.

• [Offline LLM chat — TinyLlama 1.1B] The chat model is quantized to INT4/INT8 (llama.cpp/ONNX) and runs end to end on the bench rig — prompt in, answer out, with no network call anywhere in the loop. Input is a token sequence, output is next-token probabilities, and perplexity on WikiText-2 is reported as a buyer-measured design target.
• [YOLOv8n vision on the NPU] YOLOv8n (Ultralytics), trained and evaluated on COCO 2017's 80 object classes, is compiled through the Hailo Dataflow Compiler to INT8 and runs on the NPU at up to 26 TOPS (manufacturer spec). Input is a 640x640 RGB image; output is bounding boxes plus class scores.
• [Live benchmark dashboard] A Flask web dashboard serves the demo UI and reports tokens/sec, time-to-first-token, watts/token and NPU-vs-CPU speedup for every run — every number measured on the bench rig, never taken from a vendor claim.
• [Inline USB power metering] A USB power meter sits inline on the rig's 5 V supply and measures real watts during each inference run, so the watts/token figures are measured, not estimated.
• [Hybrid NPU + CPU execution] Layers the NPU's op set supports run on the accelerator; unsupported layers fall back to the Pi 5's quad-core Cortex-A76 @ 2.4 GHz cores (manufacturer spec), so real models run end to end instead of only in a demo script.
• [INT4/INT8 quantization pipeline] The build scripts take the base models through quantization — llama.cpp/ONNX for the LLM, Dataflow Compiler INT8 for vision — with a documented accuracy check: perplexity and mAP@0.5 measured as design targets, not claimed.
• [3D-printed NPU carrier enclosure] The M.2 HAT+ and NPU module mount in a 3D-printed carrier enclosure (print files included) that holds the add-in peripheral mechanically secure on the bench.

Documented future scope (not included in the base build):

• [Future scope — Hailo-10H upgrade path] The Hailo-10H (up to 40 TOPS INT4 with 8 GB LPDDR4 on the module — manufacturer spec) is the documented upgrade path for small language and vision-language models; it is not part of the base build.

• Assembled rig: Raspberry Pi 5, M.2 HAT+ with the Hailo-class NPU module, 3D-printed carrier enclosure, inline USB power meter
• Quantized model pipeline: TinyLlama 1.1B (INT4/INT8) and YOLOv8n (INT8, NPU-compiled) with build scripts and a quantization tuning guide
• Flask inference benchmark dashboard with the working demo UI (tokens/sec, time-to-first-token, watts/token, NPU-vs-CPU speedup)
• Wiring diagram, assembly layout and bring-up checklist (including the boot-only PCIe detection note)
• 3D-print files for the NPU carrier enclosure
• Component list with Indian supplier sourcing notes for the Pi 5, M.2 HAT+ and NPU module
• Project report PDF (edge-AI architecture, quantization, NPU dataflow, benchmark methodology)
• PPT presentation for final review
• Viva Q&A preparation document (quantization trade-offs, NPU vs CPU execution, watts/token, LLM vs vision pipelines)