Data Labeling Strategies for Custom Datasets

Every supervised model is a mirror of its labels. Students obsess over architectures and then train on 500 hastily-labeled images where "damaged" meant three different things to two different labelers. The model learns the inconsistency faithfully — and the student blames the architecture.

Data labeling is the unglamorous foundation of custom-dataset projects. This guide covers designing a labeling scheme, choosing tools, keeping labelers consistent, measuring label quality, and the strategies (including weak supervision and model-assisted labeling) that stretch a small labeling budget.

Start with the label schema, not the tool

Before opening any labeling software, write down:

1. The exact classes with one-sentence definitions and 2–3 canonical examples each.
2. The edge cases — what to do when an example fits two classes, or none. (This is where inconsistency breeds.)
3. The "unsure" policy — is there an explicit "ambiguous" label, or must labelers pick? Forcing a guess on genuinely ambiguous examples injects noise; an explicit escape hatch keeps it quarantined.
4. Who decides disputes — one person owns the final call on edge cases.

Write this as a one-page labeling guide and give it to every labeler (even if the labeler is just you, three weeks from now, having forgotten your own conventions). Most labeling inconsistency is schema ambiguity, not labeler carelessness.

Note: If two careful people cannot agree on a label using your schema, the schema is broken — not the people. Fix the definitions before labeling another example.

Labeling tools: what to use

Tool	Good for	Cost
Label Studio	General purpose — image, text, audio	Free, self-hosted
CVAT	Image/video annotation (boxes, polygons, tracking)	Free, self-hosted
Roboflow	Image datasets end-to-end (label → augment → export)	Free tier generous for students
Prodigy	Text annotation with active learning	Paid (worth knowing, rarely needed)
Spreadsheets	Text classification at small scale	Free, surprisingly viable under ~1000 items

For most student image projects, Roboflow (label in browser, built-in augmentation, one-click export to YOLO/COCO formats) or CVAT (more control, video support) cover everything. For text, Label Studio handles classification, NER, and sentiment out of the box.

Measuring label quality: inter-annotator agreement

If more than one person labels (including you at two different times), measure agreement on a shared sample:

• Simple agreement: percentage of examples where labelers agree. Intuitive; inflates with few classes.
• Cohen's kappa: agreement corrected for chance. The standard for two labelers. Above 0.8 is strong, 0.6–0.8 moderate, below 0.6 means your schema needs work.

Practical protocol: have everyone label the same 50–100 examples first, compute agreement, discuss every disagreement, update the schema, repeat. This calibration round pays for itself many times over — it catches schema problems when they are cheap to fix.

from sklearn.metrics import cohen_kappa_score

kappa = cohen_kappa_score(labeler_a, labeler_b)
print(f"Cohen's kappa: {kappa:.2f}")

Strategies that stretch a labeling budget

Label a little, then let the model help

1. Hand-label 200–300 examples carefully.
2. Train a baseline model.
3. Use it to pre-label the next batch — labelers correct predictions instead of labeling from scratch. Correction is 3–5x faster than fresh labeling.
4. Retrain, repeat.

Each cycle improves the pre-labels, so labeling accelerates. This is the single most effective budget-stretcher and it works for image, text, and audio.

Active learning: label what the model is unsure about

Instead of labeling random examples, label the ones the model finds most uncertain (lowest prediction confidence, or highest disagreement among an ensemble). Uncertain examples carry the most information per label. See the dedicated active learning guide for the full treatment.

Weak supervision: rules instead of labels

Write heuristic labeling functions — keyword rules, regexes, simple thresholds — and combine their noisy votes into probabilistic labels (the Snorkel framework formalized this). Individually weak rules, aggregated, can produce training labels good enough for many tasks. Most useful when domain experts can write good heuristics but cannot hand-label thousands of examples.

What NOT to do

• Crowdsourcing without quality control. Unvetted crowd labels without agreement checks or gold-standard questions are noise with extra steps.
• Labeling the test set sloppily. Your test labels must be the highest-quality labels you have — ideally double-labeled and adjudicated. A noisy test set makes all evaluation meaningless.
• Using model predictions as ground truth uncritically. Pre-labels must be human-verified. Training on unverified model outputs is self-reinforcing error.

How much labeled data do you need?

Honest, rough guidance:

Task	Transfer learning from pretrained model	From scratch
Image classification	100–500 per class can work	Thousands per class
Object detection	500–2000 images total, well-labeled	Tens of thousands
Text classification	A few hundred examples	Thousands
NER / token labeling	200–500 documents	Much more

These assume clean labels and a pretrained backbone. With noisy labels, multiply by several. The transfer learning guide explains why pretrained backbones are so label-efficient.

Note: Label quality beats label quantity past a point. 500 clean examples routinely beat 5000 noisy ones — and the clean set trains faster.

The labeling workflow for a student project

1. Write the one-page schema with edge cases.
2. Set up the tool (Label Studio/Roboflow/CVAT) with the schema baked into the interface.
3. Calibration round: 50 shared examples, measure agreement, fix schema.
4. Label the first 200–300; train a baseline; check that the model learns (if it cannot beat chance on training data, labels may be the problem).
5. Pre-label + correct cycles to scale up.
6. Hold out a pristine test set early — labeled by your most careful process, never used in training, never pre-labeled by the model.
7. Document everything: schema version, who labeled what, agreement scores, dataset version. Your report's methodology section writes itself.

Common mistakes

• No written schema. "We all knew what the classes meant" — you did not, and the kappa score will prove it.
• Labeling train and test with different care. Test labels must be the cleanest.
• Ignoring class imbalance at labeling time. If 95% of examples are "normal," labelers develop a bias toward it. Stratify or oversample the rare class deliberately.
• Leaking between splits. Near-duplicate images (consecutive video frames, augmented copies) split across train and test inflate scores. Split by source/session first, then label.
• Changing the schema mid-project without relabeling. Version the schema; when it changes, revisit old labels.

Where to find existing datasets first

Before labeling anything, check whether a suitable dataset already exists — labeling is the most expensive path. See where to find ML datasets for student projects. Only build custom when nothing fits: novel domain, local language, or your specific sensor/hardware setup.

Where to go from here

Once labels exist, the modeling begins: CNN transfer learning for images, data augmentation to multiply your labeled data, and active learning to label smarter. Track every experiment with MLflow. More dataset and training topics in the AI & Machine Learning branch hub.