In this guide
How to Choose a Dataset for Your IEEE Base Paper Project
Quick answer
Dataset choice means picking the data your base paper's method will actually run on, so that your implementation can be compared fairly against the paper before you add anything of your own.
Choose the dataset that lets you run your base paper's method unchanged, compare against its published behaviour, and still leave room for your own feature. Start from the paper's own dataset, check licence, size, labels and splits before anything else, and only substitute when access, cost or hardware forces you to — then document exactly what changed.
| Dataset option | What it gives you | Main risk | Use it when |
|---|---|---|---|
| The paper's own dataset | The fairest comparison against the paper | Access may be gated, licensed or heavy | You can obtain it legally and run it on your hardware |
| A public substitute on the same task | A runnable baseline with comparable evaluation | Distribution shift makes direct comparison weaker | The original data is gated but a same-task dataset exists |
| Your own collected data | Data that matches your own feature exactly | Small, biased or inconsistently labelled | You keep the paper's benchmark too, and use your data for the new feature only |
Picking the dataset is not a download step. It is the decision that determines whether your later results mean anything at all.
Why the dataset decision comes first
A base-paper build has two jobs: show the paper's method working, then show your own contribution on top of it. Both jobs need data, but they need different things from it. The baseline needs data close enough to the paper's that a mismatch tells you something about your implementation. Your own feature needs data that exercises the new behaviour. One dataset rarely does both jobs well, and teams that discover this late end up either re-running everything or presenting a demo nobody can judge.
The order matters because dataset problems are expensive to fix backwards. A licence restriction found in week eight, a label error found after training, or a test set that leaked into training all invalidate work already done. Checking the dataset before building is the cheapest review your project will ever get.
The seven-question dataset check
Work through these questions in order, against the paper's method section, before you commit to a dataset.
1. What does the method actually consume?
Read the paper's data description, not just its introduction. Note the input type (images, text, signals, graphs), the exact fields or channels, the label format, and any preprocessing the method assumes. A defect-detection method that expects labelled image patches cannot be fed whole factory photographs without a patch-extraction step, and that step is itself work you must plan. If the paper's input format and your candidate dataset's format differ, write down the conversion before you download anything.
2. Can you legally and practically obtain it?
Check four things: whether the dataset is public or gated, what its licence permits (coursework and demonstration use are not the same as redistribution), whether access needs a form, an institutional email or approval, and how large the download is. A dataset you cannot legally use in a college submission, or cannot download on your connection and hardware, is not your dataset however good its results look. Apply for gated access early — approval waits are the most common silent delay in base-paper builds.
3. Are the labels trustworthy?
Labels are the dataset's real content. Look for how labels were produced (expert annotation, crowd annotation, automatic labelling, or labels inherited from another source), whether ambiguous cases were handled, and whether a label guide or annotation rule exists. A small, carefully labelled set beats a large, noisily labelled one for a baseline, because label noise looks exactly like a method failure when you compare against the paper. Inspect samples by hand: open a few dozen examples across classes and check that the labels match what you see.
4. Is the split honest?
A dataset needs a train, validation and test split that matches how the paper evaluated. Check whether splits are provided by the dataset creators or left to you, whether the same item can appear in two splits (a common leak with near-duplicate images, repeated recordings or overlapping time windows), and whether the split is by item or by group — for example, by machine, by patient or by location, where grouping matters. If you must create your own split, fix it once, record the rule, and never re-split to chase a better number.
5. Is it big enough, and small enough?
Size is a range, not a target. Too little data and the method cannot learn or be evaluated meaningfully; too much and every run costs hours you do not have. Estimate from the paper's method, not from the dataset's headline size: a method that trains on patches, sequences or graphs may multiply a modest raw collection into many training examples, or reduce a huge collection to a few usable ones. Plan one full training run on your actual hardware before you assume the full dataset is runnable.
6. Does it run on your hardware and timeline?
Check storage, memory and run time on the machine you will actually use, not on the paper's setup. Large image or video collections, high-resolution inputs and long sequences all cost time twice: once in preprocessing, once in every training run. If the full dataset is too heavy, the honest options are a documented subset for development with full runs reserved for final evaluation, or a lighter substitute dataset — not silently training on a fraction and reporting it as the whole.
7. Does it leave room for your own feature?
Your own contribution needs data that shows it working. If every example in the dataset is already handled well by the paper's method, there is nothing for your feature to improve or extend. Look for the gap: a condition, a class, a region or a failure mode your feature addresses, and confirm the dataset — or a second, smaller dataset you collect or adapt — actually contains it. This is where most own-feature plans fail: the idea is sound, but the data to demonstrate it does not exist.
Licence and access: what to check before downloading
| Access type | What to verify | If it fails |
|---|---|---|
| Open download | Licence terms, citation requirement, version | Use it, and record the version and licence |
| Form or agreement gated | Who must sign, how long approval takes, permitted uses | Apply immediately, and line up a substitute in parallel |
| Licence-restricted | Whether student coursework and college submission are covered | Ask the provider or your guide; do not assume coverage |
| Private or on-request | Whether the authors share data at all, and in what form | Treat as unavailable until a reply arrives; plan the substitute as the main path |
| Collected by you | Consent, privacy and safety for what you record | Design collection with consent and anonymisation from the start |
Keep the licence note with the dataset, not in a chat message. Your review documentation will be asked where the data came from, and "downloaded from the internet" is not an answer.
Splits, leakage and the honest comparison
Leakage is the quietest way to produce an impressive-looking result that means nothing. It happens when information from the test set influences training: near-duplicate examples across splits, preprocessing statistics computed over the whole dataset, or a split made after augmentation so that augmented copies of one example land in both train and test. The defence is procedural, not clever: split first, compute every statistic (normalisation, vocabulary, augmentation parameters) from the training split only, and freeze the test split before any tuning.
A small, practical check is worth building into your pipeline: verify that no identifier appears in more than one split. The snippet below shows the idea in plain Python — adapt the identifier to your data (file name, recording id, machine id, or patient id):
train_ids = set(load_ids("train"))
test_ids = set(load_ids("test"))
leaked = train_ids & test_ids
print("overlapping ids:", len(leaked))
If that count is not zero, your split is broken and every result built on it is suspect. Fix the split, not the number.
When the paper's dataset is gated: the substitution ladder
Substitution is legitimate work, not a shortcut, provided you climb the ladder honestly and re-baseline at each step:
- Same task, public dataset. The closest substitute: the same problem, comparable labels, a published evaluation protocol. Your comparison against the paper becomes qualitative — same method family, similar task — and you must say so.
- Same domain, different distribution. Data from the same field but a different source, sensor or population. Expect a performance gap that reflects the data, not your implementation, and report it as a data difference.
- Your own collected set. Small, targeted and fully under your control. Use it to demonstrate your own feature, not to claim you reproduced the paper — a collected set cannot carry a reproduction claim.
At every step, record what changed and why, and re-run the baseline on the new data before adding your feature. A baseline that was frozen on one dataset does not transfer silently to another.
Size, labels and compute: practical checks before training
Run these checks on a sample before committing to full training:
- Open and inspect examples from every class, including the rarest one. If the rare class has too few examples to split across train, validation and test, your evaluation of that class will be noise.
- Count missing, corrupt or mislabelled examples in the sample and scale the estimate. Cleaning time is real project time; budget it.
- Time one training epoch on your hardware with the real input size, then multiply honestly. A method that needs dozens of epochs on a large dataset may need a smaller development subset for iteration, with full runs kept for final numbers.
- Confirm the evaluation metric matches the paper's. If the paper reports a metric you cannot compute from your dataset's labels, you have a dataset problem, not a metric problem.
None of these checks requires results. They require looking at the data before trusting it.
Two worked examples from verified base papers
Low-label industrial defect detection (Marino et al., IEEE Transactions on Industrial Informatics, 2025, DOI 10.1109/TII.2025.3534443). The point of this paper's setting is that labelled defect examples are scarce — that scarcity is the problem, not an inconvenience. A dataset decision here is therefore a label-budget decision: how many labelled examples you can realistically obtain or annotate, how the unlabelled examples will be used, and how you will evaluate when the labelled test set is small. A team that picks a large, fully labelled defect dataset has changed the problem the paper solves, and should say so rather than present it as the same task.
First-person traffic-sign interpretation (SignEye, IEEE Transactions on Intelligent Transportation Systems, 2025, DOI 10.1109/TITS.2025.3590935). This paper's pipeline interprets signs from a vehicle's first-person view and assigns them to lanes and road positions, so the dataset must carry structure beyond the sign itself: the sign's position relative to the vehicle and the road. A plain collection of cropped sign images cannot support that pipeline. The tracker's own-feature slot for this paper — extending it to Indian traffic signs and Indian driving conditions, which the paper's China-focused data does not cover — shows the dataset logic for own features: keep the original benchmark for the baseline, and add the new-condition data for the new feature, so each claim has data that actually supports it.
Both examples follow the same rule: the dataset must match the method's assumptions and the claim you want to make, or the claim must change.
Documenting the choice for your guide and reviews
Write a one-page dataset note before training starts, and keep it updated. It should record: the dataset name and version, where it came from, its licence and access terms, its size and class distribution, the split rule, the preprocessing applied, the known limitations you found on inspection, and any substitution, with the reason. This note answers the first questions a guide asks about a base-paper build — where the data came from, why this dataset, and what would change the results — and it turns a download into a decision you can defend.
Common mistakes
- Choosing the dataset by headline size instead of by label quality, licence and split.
- Downloading first and reading the paper's data section afterwards.
- Re-splitting after seeing results, or tuning on the test set.
- Reporting a substitute dataset's results as if they were the paper's benchmark.
- Collecting own data without consent, privacy or labelling rules, then discovering it cannot be submitted.
- Leaving the dataset decision to the week training is supposed to start.
Frequently asked questions
Should I always use the paper's exact dataset?
If you can obtain it legally and run it, yes — it gives the fairest baseline. If access, licence or hardware blocks it, use the substitution ladder above and label the comparison honestly. What you must not do is use a substitute silently while implying the paper's benchmark.
What if the dataset is too large for my machine?
Use a documented development subset for iteration and reserve full runs for final evaluation, or choose a lighter same-task dataset as the baseline. Either way, record the choice and its effect on comparability. Quietly training on a fraction and reporting it as the whole dataset misrepresents the work.
How do I handle a dataset with very few examples of one class?
First check whether the scarcity is real or a labelling gap. If it is real, say so in your dataset note, evaluate that class separately rather than hiding it inside an overall score, and consider whether your own feature can address exactly that gap — scarce-class handling is often a legitimate contribution, but only if the evaluation shows it.
Can my own collected data replace the paper's dataset?
No, for the baseline. Your own data can demonstrate your own feature, and it can supplement the baseline, but a reproduction claim needs data comparable to the paper's. Keep the two roles separate in your documentation and your presentation.
When should the dataset decision be final?
Before training starts, and before Phase-1 documentation is submitted if your college requires a data plan. Dataset changes after that point are possible, but they cost a re-baseline, and that cost should be visible in your plan rather than absorbed silently.
Limitations
Dataset advice cannot fix a paper whose data is entirely private, a licence that forbids your intended use, or a task for which no public data exists. In those cases the honest outcome may be a different base paper, a narrower scope, or a collected-data project with no reproduction claim — and that decision belongs early, while changing direction is still cheap. This guide also cannot settle what your college accepts: dataset rules, ethics requirements and submission formats vary, and your guide's word on them overrides any general advice here. Final approval always rests with your own guide and college, and no dataset choice guarantees approval, publication or any particular result.
Get a quotation for your base-paper build
If you want help turning a chosen dataset into a working build — baseline reproduction on the paper's data or an honest substitute, plus your own feature, documented for Phase-1 and Phase-2 submission — message Projectech on WhatsApp for a quotation covering the full scope (dataset selection and baseline reproduction, own-feature design and build, and review documentation), the price (set after your topic, base paper and compute scope are locked), and delivery across the Phase-1 to Phase-2 timeline. Share your base paper and the dataset you are considering, and the quotation will be scoped to that paper and that data, not to a generic build.
Related guides
- Reproduce Your IEEE Base Paper First: A Baseline-Before-Features Guide
- How to Read an IEEE Paper in a Weekend
- IEEE Base Paper vs Regular Project: What Guides Actually Check
- Phase-1 vs Phase-2: What to Submit Each Semester
- Self-Supervised ViT for Low-Label Industrial Defect Detection
- Federated LoRA Fine-Tuning of LLMs