Efficient Transformer via NAS / quantization — IEEE Transactions 2025 Base Paper Project

Vision transformers deliver strong image-classification results, but their size and compute cost make them hard to run on ordinary laptops and edge hardware: a standard ViT-Base model carries roughly 86 million parameters and needs hundreds of megabytes in FP32 precision. Two established efficiency techniques address this — neural architecture search, which finds a smaller architecture instead of hand-picking one, and post-training quantization, which stores weights and activations in fewer bits without retraining. Students usually meet these ideas only as paper summaries and rarely implement either end to end. This project closes that gap: the student reproduces the quantization method of an IEEE Transactions paper, adds their own NAS search over depth, width and patch size, and evaluates the resulting models on a real dataset with honest size, latency and accuracy reporting. The deliverable is a working pipeline plus a lab demo that makes the trade-off visible, so the viva discussion rests on the student's measurements rather than borrowed claims.

This is a built-to-order software project: the NAS search code, quantization pipeline, evaluation notebook and the complete viva kit (report, PPT, Q&A) are prepared fresh for you.

Base paper at a glance

Base Paper

• Paper: Towards Accurate Post-Training Quantization of Vision Transformers via Error Reduction
• Journal: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2025
• DOI: https://doi.org/10.1109/TPAMI.2025.3528042
• What is implemented from the paper: The paper's post-training quantization method for vision transformers — per-layer activation-range calibration on a small unlabelled calibration set, scale and zero-point assignment for weights and activations, and protection of quantization-sensitive layers (patch embedding and classifier) at higher precision. The implementation follows the paper's procedure: the FP32 model is frozen, calibration statistics are collected, weights are rounded to INT8 or INT4, and the quantized model is validated against the FP32 baseline on a held-out set. No retraining or fine-tuning is used in the quantization stage, exactly as in the paper.
• What is NOT claimed from the paper: The paper's reported numbers are not reproduced here as this project's results. All accuracy, size and latency figures in this listing are design-target estimates or procedures the buyer runs; the report documents the actual figures measured during the build.

Own Features

The student's own contribution, clearly distinct from the base paper, is the NAS-driven architecture search wrapped around the paper's quantizer:

• [NAS search over transformer shape] A search over encoder depth (6–16 layers), embedding dimension (192–768), patch size (8/16/32), attention heads and MLP ratio, scored on estimated size, latency and validation accuracy, with Pareto-optimal candidates shortlisted. The paper quantizes fixed architectures; the search is the student's addition.
• [Budget-constrained selection] A deployment-budget rule: given a size cap (e.g. under 25 MB) or a latency cap, the search returns the best candidate that fits, instead of a single hand-picked model.
• [Mixed-precision policy] A per-layer precision assignment: sensitive layers stay at INT8 while robust layers drop to INT4, with the accuracy cost of each assignment measured on the validation set. The paper's method is the quantizer; the mixed-precision policy search is the student's own layer.
• [Interactive efficiency lab] A single-file web demo where bit-width, depth, width, patch size and calibration-set size are adjustable and size, latency and accuracy estimates update live, plus a NAS candidate table with Pareto-optimal rows highlighted. This demo is built for this project and is not part of the paper.
• [Calibration-size study] A measured study of how the accuracy drop changes as the calibration set grows (128 to 4,096 images), giving the viva a genuine experiment with a plotted result the student produced.

The pipeline runs in seven stages from baseline to demonstrable result. First a vision transformer baseline is established so every later saving is measured against a real FP32 reference, not assumed. Then the architecture search explores depth, width and patch size and keeps only Pareto-optimal candidates, so the student can defend why one shape was chosen over the others. Quantization follows the base-paper procedure exactly — freeze the model, calibrate on unlabelled images, round weights to the target bit-width, and protect sensitive layers at higher precision. The quantized model is then evaluated on a held-out test set with size, latency and accuracy drop recorded, and the calibration-size study repeats that measurement across set sizes. Finally the results, the Pareto table and the trade-off plots go into the report, and the demo lab presents the same logic interactively for the viva.

1. A vision transformer (ViT-Tiny/Small scale) is trained or fine-tuned on an image-classification dataset, establishing the FP32 baseline accuracy on a held-out test set.
2. The NAS stage samples candidate architectures from the search space (depth, embedding dimension, patch size, heads, MLP ratio) and scores each on estimated size, latency and validation accuracy.
3. Pareto-optimal candidates are shortlisted — architectures no other candidate beats on all three measures — and the best one under the chosen deployment budget is selected.
4. The selected FP32 model is frozen and a calibration set of unlabelled images is run through it to record per-layer activation ranges.
5. Weights and activations are quantized to INT8 (and, as a second variant, mixed INT8/INT4) using the calibrated scales, with the patch-embedding and classifier layers kept at INT8.
6. The quantized model is evaluated on the held-out test set; size, latency and accuracy drop versus the FP32 baseline are recorded, and the calibration-size study is repeated across set sizes.
7. Results, the Pareto table and the trade-off plots go into the report, and the efficiency-lab demo is wired to the same scaling logic for the viva demonstration.

Tech stack:

• Python 3, PyTorch
• timm vision transformer models
• NumPy, Matplotlib, scikit-learn (metrics)
• Jupyter Notebook (training · search · evaluation)
• ONNX Runtime (quantized inference measurement)
• HTML5 · CSS · JavaScript (efficiency-lab demo)

Dataset & model details

• Dataset: ImageNet-1K validation subset for the efficiency study, with CIFAR-100 as the lightweight alternative for laptop-scale runs — both are standard public benchmarks; the exact subset used is recorded in the report. A selected subset is used for this implementation so the build fits a student laptop.
• Task: Image classification; input = RGB image (224 × 224), output = probability distribution over classes, plus the efficiency measures (model size in MB, latency in ms per image, accuracy drop in percentage points).
• Model: Vision transformer (ViT-Tiny/Small scale) as the FP32 baseline; NAS-selected variants of the same family; post-training quantization to INT8 and mixed INT8/INT4 following the TPAMI 2025 base-paper procedure.
• Metrics: Top-1 accuracy, accuracy drop vs FP32 baseline (percentage points), model size (MB), latency per image (ms), compression ratio (×). No accuracy is claimed as measured until the build run is executed for the order; the report documents the actual achieved figures.

The feature set covers the full efficiency pipeline in one build: a base-paper quantization method that compresses a vision transformer without retraining, a neural architecture search that finds a compact shape instead of hand-picking one, and a budget rule that selects the model fitting a size or latency cap. Around these sit the student's own experiments — a mixed-precision policy that assigns bit-widths per layer, and a calibration-size study that measures how the accuracy drop changes with the calibration set. The deliverables make the work demonstrable: an interactive efficiency-lab demo, a reproducible evaluation notebook, and a complete viva kit with report, presentation and Q&A. Every feature below is part of the delivered build; optional extensions such as quantization-aware training are kept in Future Scope in the report, not listed here.

• [Post-training quantization pipeline] FP32 vision transformer converted to INT8 and INT4 with per-layer calibration, scale/zero-point assignment and sensitive-layer protection, following the TPAMI 2025 base-paper procedure.
• [NAS architecture search] Automated search over depth, width, patch size, heads and MLP ratio, with Pareto-optimal candidates ranked by size, latency and validation accuracy.
• [Budget-based model selection] Pick a model by deployment budget — a size cap or latency cap returns the best fitting candidate instead of a fixed architecture.
• [Mixed-precision assignment] Per-layer INT8/INT4 policy with the accuracy cost of each assignment measured on the validation set.
• [Calibration-size experiment] The accuracy drop is measured across calibration-set sizes from 128 to 4,096 images, plotted for the report.
• [Interactive efficiency lab demo] A working single-file web app: adjust bit-width, depth, width, patch size and calibration size and watch model size, latency and accuracy estimates update live.
• [Full evaluation notebook] Dataset loading, FP32 baseline training or fine-tuning, quantization, and evaluation with confusion matrix and per-class metrics in one reproducible notebook.
• [Complete viva kit] Project report, review PPT and viva Q&A document covering transformers, quantization, NAS and the student's own results.

• Complete source code: NAS search, quantization pipeline and evaluation notebook
• Trained FP32 baseline and quantized INT8 / mixed INT4 model files
• Interactive efficiency-lab web demo (single-file HTML/CSS/JS)
• NAS candidate table with Pareto-optimal shortlist and trade-off plots
• Calibration-size study results and plots
• Project report PDF (background, base-paper summary, methodology, results)
• PPT presentation for final review
• Viva Q&A preparation document (transformers, quantization, NAS, own results)
• Base-paper summary note with DOI link for guide approval