In this guide
Your YOLO detector finds every person in each frame. But frame 1's "person" and frame 2's "person" are unrelated detections — the detector has no memory. Counting people, measuring dwell time, or following a vehicle across a parking lot needs object tracking: assigning persistent IDs so the same object keeps its identity frame after frame.
DeepSORT and ByteTrack are the two tracking algorithms you will encounter everywhere. This guide explains how tracking-by-detection works, what each algorithm adds, how to choose between them, and the practical tuning that makes or breaks a tracking project.
Tracking-by-detection: the paradigm
Nearly all practical multi-object tracking follows the same loop:
- Detect objects in the current frame (YOLO, Faster R-CNN — see the YOLO guide).
- Predict where existing tracks should be now (motion model).
- Associate new detections with existing tracks (matching).
- Update tracks with matched detections; create tracks for unmatched detections; age out unmatched tracks.
The detector sees; the tracker remembers. Tracking quality therefore depends on detection quality first — a tracker cannot fix a detector that misses half the objects. Get detection solid before tuning tracking.
Note: Tracking is association, not recognition. The tracker does not know what the object is — only that this blob in frame 100 is probably the same blob as in frame 99.
The pieces: motion model + association
Motion prediction is usually a Kalman filter with a constant-velocity model: given a track's recent positions and velocities, predict its next bounding box. It handles smooth motion well and coasts through brief occlusions (the object keeps moving along its predicted path while hidden).
Association matches predicted track boxes to new detection boxes. The cost matrix is typically built from IoU (intersection over union) between boxes, solved with the Hungarian algorithm for the optimal assignment. Matches below an IoU threshold are rejected — better to start a new track than to glue two identities together.
The classic failure is an ID switch: two people cross, and their IDs swap. Everything in tracking research is, roughly, about reducing ID switches.
SORT: the baseline everything builds on
SORT (Simple Online and Realtime Tracking, 2016) is Kalman filter + Hungarian matching on IoU. No appearance features, no deep learning in the tracker itself. It is blazingly fast and surprisingly decent — and the reference point for everything after. Its weakness: ID switches whenever objects overlap or the detector misses a few frames, because IoU alone cannot re-identify anyone.
DeepSORT: adding appearance
DeepSORT (2017) keeps SORT's motion + IoU matching and adds a deep appearance descriptor: a small CNN embeds each detection's image crop into a feature vector, and association considers cosine distance between embeddings alongside IoU.
Why it matters: when two people cross and separate, motion/IoU is ambiguous — but appearance (clothing colors, mostly) disambiguates. DeepSORT also adds cascade matching (prefer matching recently-seen tracks first) which reduces ID switches further.
Costs: the embedding CNN runs per detection per frame (GPU recommended), and appearance fails when people look alike (uniforms, crowds) or change appearance (lighting shifts).
ByteTrack: using every detection box
ByteTrack (2022) made a deceptively simple observation: trackers were throwing away low-confidence detections. When a person is half-occluded, the detector outputs a low-score box — and classic trackers discard it, breaking the track. ByteTrack instead does two-stage association:
- Match high-confidence detections to tracks (as usual).
- Match the remaining tracks against low-confidence detections.
Those recovered low-score boxes keep tracks alive through occlusions that kill other trackers. No appearance model needed — ByteTrack with a strong detector (YOLOX in the original paper) set new state-of-the-art on MOT benchmarks on association metrics. It is also fast, since there is no embedding network.
DeepSORT vs ByteTrack: choosing
| Aspect | DeepSORT | ByteTrack |
|---|---|---|
| Core idea | Appearance embeddings + motion | Two-stage association incl. low-score boxes |
| Needs | Embedding model, GPU preferred | Just a good detector |
| Speed | Slower (per-detection embedding) | Faster |
| Occlusion handling | Good (appearance re-ID) | Excellent (low-score recovery) |
| Similar-looking objects | Strugges (appearance ambiguous) | Struggles (no appearance at all) |
| Long occlusions | Better (re-ID after seconds) | Worse (motion-only coast) |
| Implementation | DeepSort repos / BoxMOT | Ultralytics YOLO has ByteTrack built in |
Default choice: ByteTrack. It is simpler, faster, and with a modern detector it wins or ties on most benchmarks. Choose DeepSORT (or its successors like StrongSORT/OC-SORT in the BoxMOT library) when objects leave and re-enter the frame and must keep their IDs — re-identification across long gaps is where appearance earns its keep.
Running ByteTrack with Ultralytics is nearly free:
from ultralytics import YOLO
model = YOLO("yolov8n.pt")
results = model.track("traffic.mp4", tracker="bytetrack.yaml", persist=True)
for r in results:
for box in r.boxes:
print(f"id={int(box.id)} class={r.names[int(box.cls)]} conf={float(box.conf):.2f}")
Metrics: how tracking is actually scored
Detection metrics (mAP) do not capture tracking quality. The tracking metrics:
| Metric | Measures | Notes |
|---|---|---|
| MOTA | Detection-ish errors (misses, false positives, ID switches) | Dominated by detection quality |
| IDF1 | Identity preservation over time | The tracker's real report card |
| ID switches | Times an object's ID changed | Lower is better; the most intuitive |
| HOTA | Balances detection and association | The modern preferred single number |
When comparing trackers, IDF1 and HOTA tell you about tracking; MOTA mostly tells you about the detector. Published numbers are only comparable on the same benchmark (MOT17, MOT20) with the same detector — never compare across papers casually.
Tuning that actually matters
- Detector confidence threshold. Too high → missed detections break tracks. Too low → false positives spawn ghost tracks. ByteTrack's whole point is tolerating a lower threshold — tune it deliberately.
- Track age / max time lost. How many frames an unmatched track survives before deletion. Longer = survives occlusions but risks stale tracks grabbing new objects. Start around 30 frames at 30 FPS.
- Min hits to confirm. New tracks require N consecutive matches before being reported — filters flickering false positives. 3 is typical.
- IoU matching threshold. Lower values associate more aggressively (fewer fragments, more ID switches); higher is conservative. 0.3 is a common starting point.
- Frame rate. Tracking at 30 FPS is dramatically easier than at 5 FPS — motion prediction degrades with large inter-frame displacement. If your pipeline is slow, the tracker suffers; optimize the detector first.
Common mistakes
- Tuning the tracker before the detector works. Garbage detections → garbage tracks, regardless of algorithm.
- Evaluating by watching the video. "Looks okay" misses ID switches you did not notice. Count ID switches on a labeled clip.
- Wrong metric for the claim. Reporting MOTA improvement when you changed the detector, not the tracker.
- Ignoring the frame-rate effect. Benchmarks at 30 FPS, deployment at 8 FPS, surprise when IDs fragment.
- Appearance embeddings on uniformed crowds. DeepSORT cannot distinguish ten people in identical uniforms — know when appearance helps and when it cannot.
- No handling of track birth/death at frame edges. Objects entering/exiting the frame need sane policies or IDs churn at the borders (define entry/exit zones).
Where to go from here
Tracking starts with detection — the YOLO object detection guide covers the detector side, and data augmentation for image models helps train robust detectors. For running the full pipeline on device, see edge AI on Raspberry Pi. For labeling video data, data labeling strategies covers video annotation tools. More video-analytics topics in the AI & Machine Learning branch hub.