In this guide
A model that only runs on your laptop is a demo. A model running on a Raspberry Pi at the edge — classifying crops in a field, detecting intrusions on a camera feed, predicting motor failure from vibration — is a project. Edge AI means running inference where the data is born instead of streaming everything to the cloud: lower latency, no internet dependency, and no per-inference API bill.
This guide covers the full path: choosing the Pi and model, shrinking the model to fit, the inference runtimes that work on ARM, and the practical realities (thermal throttling, power, latency budgets) that separate working edge projects from dead ones.
Why edge, why Raspberry Pi
| Cloud inference | Edge inference (Pi) |
|---|---|
| Needs constant internet | Works offline |
| Latency: network round-trip | Latency: milliseconds, local |
| Data leaves the device (privacy) | Data stays on device |
| Per-request cost scales | Fixed hardware cost |
| Infinite compute (theoretically) | 4–8 GB RAM, modest ARM CPU |
The Pi's constraints are the point: if your model runs well on a Pi, it runs well anywhere. The Raspberry Pi 4 (4/8 GB) and Pi 5 are the realistic targets — earlier models can work for tiny models but struggle with anything modern.
Note: The Pi has no GPU worth using for ML (the VideoCore GPU lacks ML framework support). Plan for CPU inference, or add a Coral USB TPU accelerator for a real speedup.
The model diet: making models Pi-sized
A model trained on a workstation will not fit a Pi as-is. The standard shrinking pipeline, in order of effort:
- Pick a small architecture first. MobileNetV3, EfficientNet-Lite, YOLOv8n (nano), or TinyML-class models are designed for this. A small architecture at full precision beats a large architecture brutally quantized.
- Quantize to INT8. Converts weights (and activations) to 8-bit integers: ~4x smaller, 2–4x faster on ARM, with typically small accuracy loss. This is the single highest-ROI step.
- Prune (optional): remove near-zero weights. Helps size more than speed on CPUs unless structured pruning is used.
- Distill (advanced): train the small model to mimic a large teacher. Top quality-for-size, but needs training effort.
For most student projects: small architecture + INT8 quantization is the entire strategy, and it is enough.
Inference runtimes on the Pi
| Runtime | Good for | Notes |
|---|---|---|
| ONNX Runtime | General models | Export from PyTorch via ONNX, run INT8 quantized |
| TensorFlow Lite | Mobile/edge classics | Mature, huge model zoo, Coral TPU delegate support |
| NCNN / MNN | Maximum ARM CPU speed | Less friendly APIs, fastest CPU inference |
| llama.cpp | LLMs on the Pi | Yes, small LLMs run — slowly (a few tokens/sec on Pi 5) |
| OpenVINO | Intel only — not for Pi | Listed so you don't waste time on it |
TFLite + INT8 quantization is the most documented path with the most tutorials; ONNX Runtime is the most flexible if your model comes from PyTorch. Both are defensible choices.
A minimal TFLite inference loop:
import numpy as np
import tflite_runtime.interpreter as tflite
interpreter = tflite.Interpreter(model_path="model_int8.tflite", num_threads=4)
interpreter.allocate_tensors()
in_detail = interpreter.get_input_details()[0]
interpreter.set_tensor(in_detail["index"], input_data)
interpreter.invoke()
output = interpreter.get_tensor(interpreter.get_output_details()[0]["index"])
The Coral USB TPU: a legitimate cheat code
Google's Coral USB accelerator (~$60) gives the Pi a real ML coprocessor: ~4 TOPS for INT8 models. Object detection that runs at 3 FPS on the Pi CPU runs at 30+ FPS with the Coral. Constraints: models must be compiled for the Edge TPU (full INT8 quantization required, some ops unsupported), and it only accelerates TFLite models. If your project is vision-heavy and real-time, budget for one.
Latency budgeting: do the math first
Before buying hardware, estimate:
- What FPS do you need? A person-counter needs ~5–10 FPS; a gesture controller needs 15+; a once-per-minute crop classifier needs 0.02.
- What does the model cost? Benchmark on the actual Pi (or find published Pi benchmarks for your architecture — the model zoo pages often list them).
- What does the pipeline cost? Camera capture, preprocessing, and postprocessing (NMS for detection) add up. Profile end-to-end, not just the model.
If the math says 2 FPS and you need 15, your options are: smaller model, more quantization, Coral TPU, or lower input resolution (halving resolution roughly quarters compute for CNNs — the cheapest speedup available).
The physical realities nobody warns you about
- Thermal throttling. A Pi 4 running sustained inference hits 80°C and clocks down — your 10 FPS benchmark becomes 6 FPS after five minutes. Use a heatsink + fan for any sustained workload, and benchmark after warmup, not in the first 30 seconds.
- Power supply. An underpowered supply (phone charger instead of the official 3A/5A supply) causes brownouts under CPU load — random crashes blamed on software. Use the official supply.
- Storage wear. Logging inference results to the SD card 24/7 kills cheap cards. Log to RAM and flush periodically, or use an SSD via USB.
- Camera pipeline. The Pi Camera via libcamera works well; USB webcams add latency and CPU load. For vision projects, the Pi Camera module is worth it.
- Headless operation. Your demo will not have a monitor. Build in remote logging, a status LED or web dashboard, and a watchdog that restarts the inference service on crash.
A worked example: plant disease classifier on Pi
A realistic student project architecture:
- Train MobileNetV3-Small on a plant disease dataset (transfer learning — see the CNN transfer learning guide).
- Export to ONNX, quantize to INT8, verify accuracy drop is acceptable on a held-out set.
- Deploy with ONNX Runtime on a Pi 4 + Pi Camera: capture → preprocess → infer → display result on a small web dashboard.
- Measure: end-to-end latency, accuracy on real field photos (not just the dataset), thermal behavior over 30 minutes.
That last step — measuring on real inputs over time — is what separates a project that demos well from one that works.
Common mistakes
- Benchmarking on the laptop, deploying on the Pi. Always benchmark on the target hardware.
- Skipping quantization. Running FP32 on the Pi wastes 4x memory and speed for accuracy you cannot perceive.
- No thermal plan. Throttling discovered during the demo.
- Full OS + desktop on the Pi. Run Raspberry Pi OS Lite (no desktop) for headless inference — the desktop environment eats RAM and CPU.
- Ignoring the input pipeline. A 4K camera stream downscaled in Python on the CPU can cost more than the model itself. Capture at the resolution you need.
- No fallback when the model is unsure. Edge systems should have a confidence threshold with a defined behavior below it (alert, skip, ask) — silent wrong answers are worse than no answer.
Where to go from here
For the conversion step, read ONNX: converting models between frameworks. For training the model in the first place, see CNN transfer learning and data augmentation for image models. For the IoT side (sensors, ESP32 companions), the ESP32-CAM setup guide is directly relevant. More edge and deployment topics in the AI & Machine Learning branch hub.