In this guide
How AI Vision Guides a Cobot Arm: Cameras, Sorting, Tool Flanges and Human-Safe Pauses
In November 2026 Universal Robots showed its Gen 7 platform in Europe at electronica in Munich: AI-ready arms, a new controller and teach pendant, and a tool flange that delivers data, power and safety at the wrist, demonstrated with AI-based bin picking and AI PCB inspection. The industrial version uses trained neural models, safety-rated controllers and force sensing. A student build cannot copy that stack, but it can demonstrate the same ideas honestly at lab scale: a camera above a workcell, a classifier that names each object, an arm that picks it into the right bin, a wrist that accepts more than one tool, and a pause behaviour when a person comes close. This guide explains how each piece works, where the student version differs from the industrial one, and what to get right before demo day. A companion student build puts these ideas into one working cell: Gen 7-Inspired AI Vision Cobot Arm.
Why a camera changes what an arm can do
A blind arm is a playback machine. It repeats taught poses, and it only works if every object is placed in exactly the same spot every time. That is fine for a welding path on a fixed jig and useless for a bin of mixed parts, because the parts land in random positions and orientations. A camera breaks that constraint: it measures where each object actually is, so the arm can react to the scene instead of replaying a script.
The full loop has five stages, and every stage can fail independently, which is why testing them separately matters. First, the camera captures the workcell from a fixed, known position. Second, the vision software finds objects and names them by colour, shape or class. Third, the software converts each object's pixel position into a real position in the arm's coordinate frame. Fourth, the controller turns that position into joint angles and moves the arm. Fifth, the gripper closes, lifts, moves to the bin and releases. If a sort goes wrong, the fault is in exactly one of those stages, and the fastest debugging habit is to check them in order rather than retuning everything at once.
From pixels to positions: the calibration step students skip
A camera reports positions in pixels. An arm needs positions in millimetres from its own base. The bridge between the two is calibration, and it is the least glamorous and most decisive part of a vision-guided build.
For a fixed overhead camera looking straight down at a flat tray, the simple version works well: place a printed grid or a set of markers at known positions on the tray, record where each marker appears in pixels, and fit a mapping from pixel coordinates to tray coordinates. Once that mapping is stable, every detected object gets a tray position the arm can use. The mapping only stays valid while nothing moves: if the camera shifts, the tray moves or the camera height changes, the mapping is wrong and every pick misses by the same systematic offset. That is a useful diagnostic, by the way. Random misses point at detection; a consistent offset in one direction points at calibration.
Depth adds a second problem. A single overhead camera sees a flat image, so it cannot directly measure how tall an object is. Student builds handle this by fixing the problem away: all demo objects sit on the same flat tray, so height is a known constant, and the gripper always descends to the same pick height. Industrial bin picking solves the general case with depth cameras or 3D sensors, but for a tray of blocks and cylinders the fixed-height assumption is honest, cheap and easy to explain to an examiner.
Colour and shape sorting: the honest student classifier
Industrial bin picking in the Gen 7 demonstrations uses learned models that recognise parts they have seen in training data. A student build on a small board can instead use a classical classifier: colour thresholds plus contour shape. It is worth understanding exactly what that classifier does, because its limits explain most demo failures.
Colour comes first. Each frame is converted to HSV colour space, where hue separates colours more robustly than red-green-blue values do under changing brightness. A threshold range for each colour produces a mask: pixels inside the range are kept, everything else is dropped. Shape comes second. Contours are traced around each kept region, and simple rules classify them: a contour with four clear corners is treated as a block, a near-circular contour as a cylinder. The output for each object is one row: colour, shape, pixel position and target bin.
A short sketch of the core idea, in OpenCV terms:
frame_hsv = convert_to_hsv(camera_frame)
mask_red = pixels_in_range(frame_hsv, red_low, red_high)
cleaned = remove_small_specks(mask_red)
for each contour in cleaned:
shape = block if corner_count == 4 else cylinder
record(colour, shape, contour_centre)
This classifier is fast, runs on a small single-board computer, and needs no training data. Its limits are equally plain. It cannot recognise an object it has no threshold for, it fails when two objects touch and merge into one contour, and it is sensitive to lighting: a warm evening lamp shifts hues enough to break thresholds tuned under white daylight. None of that makes it a bad student choice. It makes it a classifier whose failure modes are understandable, demonstrable and fixable by controlling the scene: even light, separated objects, a fixed set of colours. A report that names those limits scores better than one that calls colour thresholds artificial intelligence and hopes nobody asks.
The upgrade path is real and worth naming. A small trained detector, such as a nano-class object detection model, learns appearance instead of colour rules, so it tolerates lighting changes and new object types far better. It also needs labelled training images, more compute and a deployment step onto the board. That is a strong extension for a second phase, not a requirement for a first working demo. Start with the classifier you can debug in an afternoon; upgrade once the motion loop is already reliable.
How the arm turns a target into motion
Once the vision software has a target position, the arm needs joint angles. That conversion is inverse kinematics: given a desired gripper position, find the angles that put it there. For a student arm with a rotating base plus shoulder and elbow, the geometry is manageable: the base angle points at the target in the horizontal plane, and the shoulder and elbow solve a two-link triangle in the vertical plane. There is usually more than one solution, elbow-up or elbow-down, and the sensible choice is the one closest to the arm's current pose, so the arm does not swing wildly between picks.
Two practical rules keep the motion safe and presentable. First, interpolate between poses: move all joints together along a smooth ramp instead of jumping from one angle set to another. Second, cap the joint speed in firmware. A hobby servo at full speed near a person's hand looks alarming and is genuinely harder to stop; a capped speed keeps the demo calm and gives the safety pause time to act. Speed caps also protect the mechanics: a fast stop at full reach stresses brackets and strips gears far sooner than a gentle one.
Teach mode is the other half of arm control, and it is how industrial operators program cobots without writing code. The operator moves the arm by hand or with joint sliders, records poses, and replays them as a sequence. For a student demo, teach mode does two jobs: it produces the bin positions and pick heights without solving kinematics for every point, and it gives the examiner something to try live. A taught pose that replays repeatably is also evidence the mechanics are sound: backlash, loose brackets and slipping servo horns all show up as drift between the taught pose and the replay.
Tool flanges: why the wrist matters as much as the arm
The Gen 7 tool flange is easy to overlook and worth understanding. An arm without a tool is just a moving stick; the tool decides what work it can do. Industrial flanges standardise the mechanical mount and carry power, data and safety signals to the tool, so a gripper, a camera or a screwdriver can be swapped without rewiring the arm. That swap is what makes a cobot redeployable across short product runs: change the tool and the program, not the robot.
A student version can demonstrate the same idea mechanically. A magnetic plate at the wrist lets the operator pull off a two-finger gripper and fit a pen-holder tool by hand in seconds, then run a taught drawing pose with the new tool. What it does not demonstrate is equally important to state: there is no electrical tool interface, no automatic tool identification and no safety signal through the mount. Naming that gap is not a weakness in the report; it is the difference between a student stand-in and an industrial flange, and examiners respect a build that knows which one it is.
Tool choice also constrains the whole design. A two-finger gripper suits blocks and cylinders of similar size; it cannot handle flat sheets, soft objects or anything heavier than the wrist servo can hold at full reach. Torque falls with reach, and the wrist is always the weakest joint, so the honest payload limit is set by the wrist at maximum extension, not by the shoulder's headline torque. Size the demo objects to that limit and the demo stays boring in the best way: it just works.
Human-safe pausing: demonstration behaviour, not certification
The most misunderstood part of collaborative robotics is the word collaborative itself. An industrial cobot earns that label through a safety system: force and torque sensing, safety-rated controllers, monitored speed and separation, and a risk assessment for the specific task. A student arm with a presence sensor demonstrates one behaviour from that world: it pauses when a hand enters a marked zone and resumes when the zone clears. That behaviour is real, testable and worth building, but it is not certification, and the report must never imply otherwise.
The demonstration version has three layers, each simple on its own. A presence sensor watches a marked zone around the arm; the controller pauses motion the moment the zone is entered and resumes only after it clears. A latching e-stop button overrides everything until it is manually reset, so a fault cannot restart itself. A firmware speed cap keeps the arm slow enough that a pause is meaningful rather than symbolic. Together they let an examiner test the collaboration story live: hand in, arm stops; hand out, arm resumes; e-stop pressed, arm stays stopped until reset.
The limits belong in the same paragraph as the features. There is no force sensing, so the arm cannot detect contact it did not see coming; there is no safety-rated controller, so a software fault could in principle bypass the pause; and no risk assessment covers the build, so it must never work beside a person unsupervised or with anything heavier or sharper than the demo objects. A student who states those limits unprompted is showing engineering judgement, not undermining their own project.
Putting the loop together: a build order that works
The order of construction decides how painful the integration week is. Build and test in this sequence, and do not move on until each stage passes on its own.
First, the mechanics: assemble the arm, confirm every servo moves through its full range without binding, and check that a taught pose replays to the same place three times in a row. Second, the vision: tune colour thresholds under the exact lighting the demo will use, and confirm the classifier names one block and one cylinder correctly before adding more objects. Third, the calibration: map pixels to tray positions with a printed grid, then verify by commanding the gripper to a marked point and measuring the error. Fourth, the loop: run the full pick-and-place cycle with two objects, then scale to eight. Fifth, the safety behaviours: test the pause, the resume and the e-stop as a scripted demonstration, not as an afterthought.
Two demo-day habits save more marks than any extra feature. Log every cycle to a file: pick order, bin counts, pause events and e-stop events, exported for the report. And record the conditions the demo ran under, especially lighting, because a sort that works under one lamp and fails under another is not a mystery once the lighting is written down. The companion build follows exactly this structure, and its dashboard shows the queue, the bin counts and the safety status live: Gen 7-Inspired AI Vision Cobot Arm.
FAQ
Is colour and shape sorting real AI vision?
It is real computer vision and not a learned model. Colour thresholds and contour rules are classical image processing: they work from rules a person wrote, not from patterns a model learned from data. Calling them a neural network would be wrong, and the distinction matters because the failure modes are completely different. A learned detector generalises to new appearances; a colour classifier only knows the colours it was tuned for. For a tray of known coloured objects, the classical approach is the right engineering choice: faster, debuggable and runnable on a small board.
Why does my arm miss objects the camera clearly sees?
Work through the five stages in order. First confirm the detection: does the software list the object with the right colour, shape and position? Then check calibration: command the gripper to a marked tray point and measure the error. A consistent offset means the pixel-to-tray mapping moved or was fitted badly. Random misses usually mean detection is unstable, often from touching objects merging into one contour or from lighting drift. Fix one stage at a time; retuning thresholds will not fix a calibration error.
Can a student arm be genuinely safe to work beside?
It can demonstrate safe behaviours, and it cannot claim certified collaborative safety. A presence pause, a speed cap and an e-stop are meaningful layers, and they should be tested live. But without force sensing, a safety-rated controller and a task risk assessment, the arm is a supervised demonstration device. Keep people out of the zone except during the scripted pause test, keep objects light, and power down before any tool swap or wiring change.
When should I upgrade from colour thresholds to a trained detector?
When the motion loop already works and the task outgrows colour rules: new object types, variable lighting you cannot control, or objects that share colours but differ in shape details a contour rule cannot capture. Budget real time for it: collecting and labelling images, training, and deploying a small model to the board is a project phase of its own. Do not start it in the last two weeks before review.
What makes a vision-guided arm project strong at review?
A complete, testable chain beats a long feature list. Show the camera frame becoming a pick queue, the queue becoming joint motion, and the motion becoming sorted bins, with a log that proves it. Then show the two honest limits: the lighting the demo needs, and the safety behaviours that are demonstrations rather than certification. Examiners remember the student who could explain why a sort failed and what stage caused it.
Limitations
- A colour and shape classifier only recognises what it was tuned for: new colours, touching objects, shadows and coloured room light all break it, and no threshold tuning fixes a scene the classifier was never designed for.
- A single overhead camera cannot measure object height, so the fixed-tray, fixed-pick-height assumption must hold; anything outside that assumption needs a depth sensor and a different design.
- Demonstration safety behaviours are not certified collaborative safety: no force sensing, no safety-rated controller and no risk assessment, so the arm stays a supervised demo device.
- Hobby servos limit the whole build: backlash, limited repeatability, torque that falls with reach, and a wrist that sets the real payload limit, so heavy or large objects are out of scope.
- Calibration is fragile by nature: moving the camera, the tray or the lighting invalidates the pixel-to-position mapping until it is redone, and a demo that skips recalibration after any move will miss systematically.