Projectech — Buy project kits, Learn, Earn, GrowOrder a Custom Project
Home/Guides/AI & Machine Learning/ONNX: Convert ML Models Between Frameworks

ONNX: Convert ML Models Between Frameworks

Train in PyTorch, deploy anywhere: ONNX is the open format for portable ML models. This guide covers exporting with dynamic axes and opsets, running models with ONNX Runtime, numerical verification, where conversion breaks, and the path to browser, mobile, and edge deployment.

Written by Projectech6 min readPublished
For B.E./B.Tech Computer Science and AI/ML students who trained a model in PyTorch and need to deploy it somewhere PyTorch doesn't fit — web, mobile, or edge Topics: ONNX, ONNX Runtime, PyTorch, TensorFlow
Illustration of ONNX model conversion showing a PyTorch model transforming into a portable ONNX graph file that runs on laptop, phone, and edge devices.
Illustration generated for this guide.
In this guide

You trained a model in PyTorch. Now you need it running in a web app, on an Android phone, or on a Raspberry Pi — places where dragging the entire PyTorch training framework along is absurd. Rewriting the model in each target framework is not a plan. ONNX is: an open format that represents a trained model as a portable computation graph, convertible from PyTorch, TensorFlow, and scikit-learn, and runnable by optimized engines everywhere.

This guide explains what ONNX actually is, how to convert a model, how to run it with ONNX Runtime, where conversion breaks, and how ONNX fits into a deployment pipeline.

What ONNX is (and is not)

ONNX (Open Neural Network Exchange) defines two things:

  1. A model format — a .onnx file describing the computation graph: operators (Conv, MatMul, Relu...), their connections, and the trained weights.
  2. A runtime — ONNX Runtime executes that graph with hardware-specific optimizations, on CPU, GPU, mobile, and edge devices.

What it is not: a training framework. You train in PyTorch/TensorFlow, convert once, and deploy the ONNX file. The training framework never ships.

Note: Think of ONNX like PDF for models — the authoring tool (PyTorch) and the reader (ONNX Runtime) are different programs, and the file works in any compliant reader.

Converting from PyTorch

The standard path uses torch.onnx.export with a dummy input that defines the expected shapes:

import torch

model.eval()
dummy = torch.randn(1, 3, 224, 224)  # batch=1, RGB, 224x224
torch.onnx.export(
    model, dummy, "model.onnx",
    input_names=["input"], output_names=["output"],
    dynamic_axes={"input": {0: "batch"}, "output": {0: "batch"}},
    opset_version=17,
)

The details that matter:

  • model.eval() first. BatchNorm and Dropout behave differently in train mode; exporting in train mode bakes wrong behavior into the graph.
  • Opset version is the ONNX operator set version — newer opsets support more PyTorch operations. Use a recent one (17+) unless your target runtime is old.
  • Dynamic axes let batch size (and sequence length) vary at inference. Without them, the exported model accepts exactly the dummy input's shape — a classic deployment bug.
  • Trace vs script: export traces actual tensor operations. Data-dependent control flow (if-statements on tensor values) may not survive tracing — restructure such logic before exporting.

From scikit-learn, the skl2onnx package converts most classical models (random forests, SVMs, logistic regression) with a similar one-call API. From TensorFlow, tf2onnx handles the conversion.

Running with ONNX Runtime

import onnxruntime as ort
import numpy as np

sess = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
output = sess.run(["output"], {"input": np.random.rand(1, 3, 224, 224).astype(np.float32)})[0]

Why bother instead of just using PyTorch for inference:

Advantage Explanation
Smaller footprint No autograd, no training code — megabytes instead of gigabytes
Graph optimizations Operator fusion, constant folding, and layout rewrites applied automatically
Hardware coverage CPU, CUDA, TensorRT, CoreML, DirectML, mobile NPUs via execution providers
Quantization tooling Built-in paths to INT8 quantized models for edge deployment

On CPU, ONNX Runtime is frequently faster than PyTorch inference for the same model — the graph optimizer earns its keep.

Verifying the conversion

Never deploy a converted model without checking it. The standard verification: run the same inputs through PyTorch and ONNX Runtime and compare outputs.

import numpy as np

torch_out = model(dummy).detach().numpy()
ort_out = sess.run(["output"], {"input": dummy.numpy()})[0]
print("max abs diff:", np.abs(torch_out - ort_out).max())

Acceptable difference is around 1e-5 for float32 — floating-point reordering between frameworks causes tiny deviations. Differences of 1e-2 or NaNs mean the conversion is broken: usually an unsupported op silently substituted, a missed eval() mode, or dynamic-shape mishandling.

Where conversion breaks (and workarounds)

Problem Symptom Fix
Unsupported/custom op Export error or wrong results Replace with standard ops; or register a custom op in the runtime
Data-dependent control flow Traced graph ignores branches Restructure to static shapes; use scripting where supported
Dynamic shapes Runtime shape errors Set dynamic_axes at export; test with multiple input sizes
Very new PyTorch ops "No converter for..." error Newer opset, or rewrite using older equivalent ops
Huge models (LLMs) Multi-GB files, slow export Shard the export; consider whether ONNX is right (vLLM/TGI often better for LLMs)

For LLMs specifically: ONNX works but the ecosystem around LLM serving (paged attention, continuous batching in vLLM) is usually a better fit. ONNX shines brightest for vision, audio, and classical ML models.

ONNX in a deployment pipeline

A typical student-project deployment flow:

  1. Train in PyTorch on Colab (see the Colab GPU guide).
  2. Export to ONNX with dynamic axes; verify numerically.
  3. (Optional) Quantize to INT8 with ONNX Runtime's quantization tools for edge targets.
  4. Ship the .onnx file with ONNX Runtime in your app — web (ONNX Runtime Web runs in the browser via WebAssembly!), mobile, or Raspberry Pi.

ONNX Runtime Web deserves emphasis: your model can run entirely in the visitor's browser with no server GPU. For a project demo page, that is remarkably compelling — inference happens client-side.

For Raspberry Pi deployment specifically, see Edge AI on Raspberry Pi.

Common mistakes

  • Exporting in train mode. The number-one ONNX bug. model.eval() before export, always.
  • Forgetting dynamic axes. Model works on exactly one batch size, crashes on any other.
  • Skipping numerical verification. "It exported without errors" is not "it works." Compare outputs.
  • Assuming every op converts. Test the export early in the project, not the night before the demo.
  • Shipping the training framework to production. If your deployment installs full PyTorch to run inference, you have a packaging problem ONNX solves.
  • Using ONNX for LLM serving. It works, but you are fighting the ecosystem — evaluate vLLM or llama.cpp first for language models.

Where to go from here

ONNX is one step in getting models into the real world. For the full deployment story, read deploying ML models as a final-year project and Docker for student projects for packaging. For edge targets, the Raspberry Pi edge AI guide continues from the quantized ONNX file. For the training side, the transfer learning guide covers getting a model worth deploying. More deployment topics in the AI & Machine Learning branch hub.

More project guides

More in AI & Machine Learning

Random Forests Explained: Why Decision Trees Vote Better Together

Decision trees are readable but overfit; random forests fix this by training hundreds of varied trees on bootstrapped data with random feature subsets, then letting them vote. This guide explains Gini impurity, bagging, out-of-bag validation and the four hyperparameters that matter, with a complete scikit-learn workflow, honest feature-importance practices, and the mistakes students keep making.

Read guide

Object Tracking: DeepSORT and ByteTrack Explained

Detection finds objects per frame; tracking keeps their identities across frames. This guide explains tracking-by-detection, Kalman motion models, DeepSORT's appearance embeddings vs ByteTrack's low-confidence box recovery, tracking metrics (HOTA, IDF1, ID switches), and the tuning parameters that determine real-world quality.

Read guide

Image Segmentation: U-Net and Mask R-CNN

When projects need pixel-level answers, segmentation delivers. This guide explains semantic vs instance vs panoptic segmentation, U-Net's encoder-decoder with skip connections, Mask R-CNN's parallel mask head, Dice and IoU evaluation, paired augmentation, and how to choose the right architecture for your data and question.

Read guide