Sensors and compute

Stage 3 · Electrical, sensing and safety · ~35 min · Before this: 03 System architecture, 10 CAN buses · numbers from the repository as of 2026-11-02

A walking policy is only as good as what it can feel. This page covers the three senses Marvin Bot has (a pelvis IMU, joint encoders and a head-mounted active-stereo depth camera) and the computer that turns them into actions, a Jetson AGX Orin 64 GB. Three playgrounds: how depth error grows with distance, what IMU noise and delay do to a tilt estimate, and what fits on the Orin.

You will learn

• How a stereo camera measures depth from disparity, why its error grows with the square of distance, and what the Orbbec Gemini 335L’s 95 mm baseline buys on stairs.
• What a gyro and an accelerometer measure, how noise, bias, sample rate and delay corrupt them, and how a complementary filter combines them into the “projected gravity” the policy sees.
• What the Jetson AGX Orin offers (CPU, GPU, memory, power modes), why the 50 Hz policy costs almost nothing, and why a vision-language-action model does not.

The idea in one picture

Close one eye and hold a finger in front of your face, then switch eyes: the finger jumps, the far wall barely moves. That jump is disparity, and it is how a stereo camera measures depth. Near things jump a lot, far things a little, so a one-pixel mistake matters little up close and a lot far away. For balance the robot uses a different sense: an inertial measurement unit (IMU), like your inner ear, which feels rotation (a gyroscope) and the pull of gravity plus any acceleration (an accelerometer). Each sense is noisy in its own way, and each reading arrives a little late. The computer has to combine them fast enough for a 20 ms control tick (the walking policy, the trained neural-network controller, decides 50 times a second).

In short, Marvin Bot has three senses and one brain:

• IMU in the pelvis: “which way is down, and how fast am I tipping?” Essential for balance.
• Joint encoders in every motor: “what angle is each joint at?”
• Depth camera in the head: “how far away is each thing in front of me?” Needed to see stairs and obstacles.
• The computer, an NVIDIA Jetson AGX Orin, which reads them all and runs the policy. Its commands reach the motors over the CAN buses of 10 CAN buses.

Playground A: stereo depth error explorer

Start here: click the presets and watch how the blue error curve moves; lower is better. The plot has distance from the camera along the bottom and the expected depth error up the side; the shaded “stairs ahead” area is the 0.5-2 m range where the robot looks for stairs, and the dotted line is your tolerance. Readouts turn amber when the error at that distance is larger than your tolerance.

Depth from disparity is z = f·B / d (d is the disparity, in pixels). Differentiate it and a disparity error σ_d becomes a depth error

σ_z = z² · σ_d / (f · B)

where z is the distance (m), B the baseline between the two infrared cameras (m), f the focal length in pixels (f = (image width ÷ 2) ÷ tan(horizontal field of view ÷ 2)), and σ_d how precisely the camera matches a point between its two images, in pixels (sub-pixel matching gives fractions of a pixel).^[4]

Presets

Marvin Bot today (335L) 335L at 848 × 480 Gemini 335 (50 mm) 150 mm baseline (hypothetical) Poor matching (dark, low texture)

Camera

Baseline B

Distance between the two infrared (IR) cameras, like the gap between your eyes. Wider = better depth far away. Datasheet: 95 mm for the 335L.

Depth image width

1280 px (1280 × 800)848 px (848 × 480)640 px (640 × 400)424 px (424 × 266)

Fewer pixels across the same field of view means a shorter focal length in pixels.

Horizontal depth field of view (FOV)

Wider view, shorter focal length, larger error.

Disparity precision σ_d

How finely the camera can line up the same point in its two images, in pixels (smaller = better). 0.24 px reproduces the datasheet's ≤0.8% at 2 m and ≤1.6% at 4 m.

Disparity search range

64 px128 px256 px

The largest disparity the matcher looks for; sets the minimum distance.

Use case

Tolerance on a stair edge

Depth error you can accept (cm). The terrain map has 10 cm cells.

Focal length f

–

σz at 1 m

–

σz at 2 m (% of z)

–

σz at 4 m (% of z)

–

Range within tolerance

–

Minimum distance

–

σz, chosen cameraone whole pixel of disparity (quantisation step)stair look-ahead 0.5–2 mdatasheet points (335L)

Try this

1. Load “Marvin Bot today”: the curve passes through the green datasheet points at 2 m and 4 m. One σ_d of 0.24 px fits both, which supports the z² law.
2. Load “Gemini 335 (50 mm)”. Same sensor, same σ_d, about half the baseline: about twice the error, 1.5% at 2 m, again what Orbbec’s datasheet gives for that model.
3. Switch Marvin Bot’s camera to 640 px wide. Error doubles, but at 2 m it is still about 3 cm, well under a 17 cm riser. Lower resolution may be fine for a 10 cm terrain map and saves bandwidth.
4. Set the search range to 64 px: the minimum distance doubles. The robot’s own feet, about 1.3 m below the camera, are still in range; anything closer than the minimum is a hole in the depth image.

Playground B: IMU noise, bias and latency

A simulated pelvis rocks by a few degrees, as it does while walking. The gyro sees the rotation rate plus noise and a bias; the accelerometer sees gravity plus noise plus the body’s own accelerations. A complementary filter integrates the gyro for fast changes and leans on the accelerometer’s gravity direction for the slow average. The policy then gets the estimate a few milliseconds late.

How to read this picture: four seconds of pelvis tilt, time along the bottom, tilt angle up the side. The white line is the truth; the aim is for the blue line (what the policy sees) to sit on it. Grey is what the accelerometer alone would say (jumpy with every footstep), violet what the gyro alone would say (smooth but drifting away). Preset names: an AHRS is an IMU with its own built-in orientation filter; BMI088 is a small, cheap IMU chip; “R42” is requirement 42 in docs/requirements.md.

Presets

Marvin Bot today (R42 target) BMI088 on a microcontroller Low-cost USB AHRS Industrial AHRS Trusting the accelerometer

Sensor

Gyro noise density

Random jitter in the rotation-rate reading (°/s/√Hz; lower is quieter). BMI088 datasheet: 0.014.

Gyro bias

A steady false rotation the gyro reports when still, in °/s left after start-up calibration. BMI088 zero offset is up to ±1 °/s before calibration.

Accelerometer noise density

Random jitter in the acceleration reading, in µg/√Hz (millionths of one g, the acceleration of gravity). BMI088: 175 (product page).

Sample rate

R42: at least 500 Hz.

Delay to the policy

Filtering, transport and buffering. Training randomises 0–20 ms.

Motion and filter

Walking bumps (body acceleration)

m/s² peak of forward-back acceleration from footsteps, which the accelerometer cannot tell from tilt.

Filter time constant τ

How long the filter trusts the gyro before the accelerometer pulls it back.

New noise sample

Tilt error the policy sees (RMS)

–

…of which from delay

–

Accelerometer-only (RMS)

–

Gyro-only drift after 4 s

–

Angular-velocity error (RMS)

–

true tiltaccelerometer onlygyro onlyfilter, as the policy sees it (delayed)

Try this

1. Load “Trusting the accelerometer” (τ = 0.02 s): the estimate follows the grey accelerometer line, footsteps and all. Raise τ to 0.5 s and the bumps disappear.
2. Now set τ to 3 s and the bias to 1 °/s. The filter leans on the drifting gyro and the error creeps toward bias × τ = 3° (it gets there after a few τ). Then raise the walking bumps to 3 m/s² with τ back at 0.5 s. The time constant is a trade between bumps and bias.
3. Load “Low-cost USB AHRS” and watch “of which from delay”: at walking rates lateness is the largest part of the error, more than noise. Cutting delay is worth more than a quieter sensor.
4. Compare “BMI088 on a microcontroller” and “Industrial AHRS”: with a few milliseconds of delay, the remaining difference is mostly bias stability over temperature and time, which this toy only shows as a constant bias.

Playground C: what fits on the Orin

Tick the workloads to stack them against the Jetson AGX Orin 64 GB’s CPU (the general-purpose processor), GPU (the graphics processor, which does neural-network maths fast) and memory. Each bar fills as you add jobs; green headroom means room to spare, red means it does not fit. The workloads in plain words:

• Real-time process: the fast loop that talks to the motors and IMU and runs the safety checks.
• Policy MLP: the walking controller, a small neural network (MLP, multi-layer perceptron; “256-128-128” are the sizes of its layers).
• Terrain height map: turning depth images into a grid of ground heights for stairs.
• ROS 2 logging and teleoperation: recording data, and streaming video to a person driving the robot remotely (ROS 2 is the common robot software framework).
• VLA (vision-language-action model, here NVIDIA’s GR00T N1.7, compiled with NVIDIA’s TensorRT for speed): a large model that turns camera images and spoken instructions into arm and hand motions, in “chunks” of several actions at once (18 Whole-body control and VLA).

estimate marks rough figures: per-workload costs are order-of-magnitude guesses unless a source is given, to be replaced by measurements on the robot.

Presets

Marvin Bot today (walking) Stairs from depth VLA onboard at 2 Hz 30 W power mode

Power mode

MAXN (up to 60 W): 12 cores, GPU 1301 MHz50 W: 12 cores at 1.5 GHz, GPU 816 MHz30 W (default): 8 cores, GPU 612 MHz15 W: 4 cores, GPU 408 MHz

Workloads

Real-time process: CAN I/O on 7 buses, IMU, safety est. Policy MLP Depth camera stream (depth on the camera) est. Terrain height map from depth (GPU) est. ROS 2 logging est. Teleoperation video stream est. VLA (GR00T N1.7, TensorRT)

Policy rate

Policy networkFlat: 256-128-128, ~75 inputsRough: 512-256-128, ~210 inputsBigger: 1024-1024-512, 64-step history (hypothetical)

Logging rate

MB/s written (full-resolution depth at 30 fps is about 61 MB/s).

VLA chunk rate

Action chunks per second. One chunk ≈ 0.28 s of GPU at full clocks (one report).

CPU (core-equivalents at 2.2 GHz)

GPU (fraction of MAXN throughput)

Memory (GB of 64, shared CPU/GPU)

Policy inference

–

Max VLA rate left

–

CPU headroom

–

GPU headroom

–

Try this

1. Load “Marvin Bot today”: the policy is a sliver. A 256-128-128 network is about 70 000 multiply-adds, tens of microseconds of arithmetic; the cost is in the plumbing around it.
2. Load “VLA onboard at 2 Hz”, then drag the chunk rate up. Somewhere near 3.5 chunks per second the GPU is full. That is why GR00T N1.7 runs on the desktop GPU first.
3. Load “30 W power mode” (the preset keeps the VLA on): the GPU clock drops to 47% and the same VLA no longer fits. Power mode is a design decision, paid for in battery runtime.

From first principles

The IMU: what it measures

A MEMS (micro-electro-mechanical system: tiny moving structures etched into a silicon chip) gyroscope measures angular velocity ω (°/s or rad/s) about three axes; an accelerometer measures specific force: gravity’s reaction plus the body’s own acceleration (m/s²). Both have:

• Noise density, the white noise per square root of bandwidth. RMS noise ≈ density × √bandwidth: the BMI088’s 0.014 °/s/√Hz over a 100 Hz bandwidth is about 0.14 °/s RMS.^[1]
• Bias, a slowly wandering offset. Integrating a gyro bias b for t seconds gives an angle error b·t, which grows without limit; that is why a gyro alone cannot hold a tilt estimate.
• Sample rate and delay. The reading is up to one sample period old when taken, plus filtering and transport time.

Low-cost MEMS parts such as Bosch’s BMI088 (gyro up to 2 kHz output, 0.014 °/s/√Hz, accelerometer 175 µg/√Hz, bias stability under 2 °/h in-run) are good enough on noise; the differences from an industrial AHRS (attitude and heading reference system: an IMU with its own calibrated orientation filter) are factory temperature calibration, bias stability over time, and a ready-made orientation output.^[1]

Why at least 500 Hz, low latency, at the pelvis

The policy runs at 50 Hz, so why sample ten times faster? Three reasons:

• Fresher data. A faster IMU’s newest sample is younger (a 200 Hz sensor’s reading is on average 2.5 ms old, a 1 kHz one 0.5 ms).
• Less noise. Averaging several samples per tick cuts noise and avoids aliasing of footstep vibration (fast shaking showing up as a false slow wobble).
• Less hidden delay. Many budget USB modules that top out near 200 Hz also buffer and filter internally, which adds delay (the repo notes this as the reason a USB AHRS is only the backup).

The pelvis is the right place because it is the root of the kinematic tree: the policy’s observations are expressed in the pelvis frame, and the pelvis moves least with arm swings. In training, the IMU sits on the pelvis 6 cm above its origin.

Orientation estimation and what the policy observes

The complementary filter in Playground B is the simplest fusion:

θ̂ ← α·(θ̂ + ω_meas·Δt) + (1 − α)·θ_accel, α = τ / (τ + Δt)

with θ̂ the tilt estimate, Δt the sample period, τ the time constant and θ_accel = atan2(a_x, a_z) the tilt implied by the accelerometer. It is a high-pass filter on the gyro plus a low-pass filter on the accelerometer, which add up to one. A steady gyro bias leaves a steady error of about b·τ. Production AHRS filters (Mahony’s nonlinear complementary filter on rotations, or a Kalman filter) do the same job in 3-D.^[2]

The walking policy does not see angles. It sees the gyro’s angular velocity directly and the projected gravity: the unit gravity vector (0, 0, −1) rotated into the pelvis frame, g_b = R^T(0, 0, −1), where R is the pelvis orientation. Standing upright it is (0, 0, −1); tilted forward by θ its forward component is about sin θ. Projected gravity avoids the wrap-around and yaw drift of angles, because heading does not matter for balance.

Joint encoders

Each RobStride actuator has an absolute magnetic encoder (14-bit, 16 384 counts per revolution, on the RS04^[6]) and reports position, velocity, torque and temperature in its CAN reply. The STS3250 servos report position over their serial bus. Joint readings travel with the IMU into the observation and share its 0–20 ms randomised delay in training.

Active stereo and the Orbbec Gemini 335L

Two global-shutter infrared cameras 95 mm apart see the scene; an infrared projector at 850 nm adds a dot pattern so that plain floors and painted risers, which have no texture of their own, still give something to match. Orbbec’s MX6800 chip computes depth on the camera, up to 1280 × 800 at 30 fps over a 90 × 65° field of view, ideal range 0.25–6 m, under 3 W on average, 124 × 29 × 27 mm, 133 g.^[3] On-camera depth leaves the Orin’s GPU free. Its spatial precision is given as ≤0.8% at 2 m and ≤1.6% at 4 m, consistent with the z² law in Playground A.

For stairs, the region that matters is 0.5–2 m ahead: one to three steps of 25 cm treads seen from about 1.3 m up. At 2 m the expected error is about 1.6 cm, a tenth of a riser.

The Jetson AGX Orin 64 GB

In plain terms: a palm-sized computer with a phone-style processor and a capable graphics chip, designed to run neural networks on robots within a 15-60 W power budget.

Go deeper: the Orin’s specifications

From NVIDIA’s technical brief: a 12-core Arm Cortex-A78AE CPU (up to 2.2 GHz), a 2048-core Ampere GPU with 64 Tensor Cores (up to 1.3 GHz, about 5.3 FP32 TFLOPS and 170 sparse INT8 TOPS), two deep-learning accelerators (105 sparse INT8 TOPS together), 64 GB of 256-bit LPDDR5 at 204.8 GB/s shared by CPU and GPU, and configurable power from 15 to 60 W. The headline 275 TOPS is the GPU and DLAs added together, at INT8 with 2:4 sparsity; dense figures are half.^[5] Power modes cap cores and clocks: 15 W runs 4 cores and the GPU at 408 MHz, 30 W (the default) 8 cores at 612 MHz, 50 W 12 cores at 816 MHz, MAXN everything.^[7] (TOPS: trillions of operations per second; INT8: 8-bit whole-number maths, the fast low-precision kind used for running trained networks.)

The developer kit (module, carrier board, heat sink and fan) weighed 0.877 kg in the repo’s weight study and costs $3,499 since NVIDIA raised it from $1,999 in July 2026.^[9] The module alone on a compact third-party carrier board would save about 0.4 kg (description/mass_budget.yaml; older entries said 1.0 kg, from before the kit was weighed), at the cost of a custom cooling and I/O design.

What runs where

A policy MLP is tiny: 256-128-128 hidden units with about 75 inputs and 12 outputs is roughly 2 × 70 000 = 0.14 million floating-point operations per step, 7 MFLOP/s at 50 Hz, a rounding error for any CPU. A vision-language-action model is the opposite. GR00T N1.7 is a multi-billion-parameter model; on an AGX Orin under JetPack 7.2 with TensorRT engines for all its parts, one published run took 0.2755 s per 16-action chunk, about 3.6 chunks per second with the GPU fully busy.^[8] That leaves little room for anything else on the GPU, hence “desktop GPU first, the Orin later”.

How Marvin Bot does it

In the repo

• IMU: ≥500 Hz gyro and accelerometer at the pelvis (docs/requirements.md R42). Primary: an industrial AHRS or a BMI088 read by a microcontroller, decided after a bench test; a low-cost USB AHRS (about 200 Hz) is the backup (docs/decisions.md 2026-10-25, sensing; roadmap open item 6). BOM: $730 budgeted for the primary, $35 for the backup. Placed in the pelvis channel near the hip axes (docs/design/assembly_wiring.md); in simulation on the pelvis at +0.06 m (sim/isaaclab/marvin_bot/robot_cfg.py).
• Observations: pelvis gyro, gravity from the IMU orientation, command, joint positions and velocities, last action; on rough ground a 1.4 × 0.8 m height map at 0.1 m with drift, holes, noise and 10 Hz updates; IMU and joint readings delayed 0–20 ms at random (sim/isaaclab/marvin_bot/tasks/velocity_env_cfg.py).
• Camera: Orbbec Gemini 335L, chosen because it computes depth on the camera, its IR pattern works on plain floors and carpeted stairs, it uses USB-C, and Isaac Sim has a model of it; JetPack 7.2 driver support still to confirm (open item 5). $359 list.
• Camera placement: a 20 mm window in the helmet let through only about a third of the depth view, so the camera now sits with its front level with the face in a light printed frame; a ray cast from both IR cameras and the colour camera clears 100% of the 90 × 65° view. The head is 0.452 kg; the camera’s outline and lens positions come from Orbbec’s own mesh and URDF (docs/decisions.md 2026-10-28).
• Compute: Jetson AGX Orin 64 GB developer kit, JetPack 7.2, Isaac ROS 4.6, ROS 2 Jazzy (docs/requirements.md R40). Weighed 0.877 kg (docs/decisions.md 2026-10-28). Mounted on edge in the torso, fan forward. Policies: 256-128-128 (flat) and 512-256-128 (rough) MLPs (sim/isaaclab/marvin_bot/tasks/agents.py).

Trade-offs and alternatives

Item	Picked	Considered	Why
Depth camera	Gemini 335L (active stereo, depth on camera)	A GMSL stereo camera with depth on the GPU; another USB active-stereo camera	The first needs a capture card and GPU time; the second had USB stability reports on Jetson (docs/decisions.md).
Camera mounting	Front flush with the face, in a printed frame	Window in the helmet; flat face plate; curved IR lens	The window cropped the view to about 33%; a formed lens adds distortion and reflections to stereo.
IMU	Industrial AHRS or BMI088 + microcontroller	Low-cost USB AHRS	Rate and latency; the USB module tops out near 200 Hz.
Computer	AGX Orin 64 GB developer kit	Module on a compact carrier (−0.4 kg); a newer Jetson Thor module	Already owned; the carrier option is a mass reduction held in reserve.
VLA	Desktop GPU first	Onboard from the start	About 3.6 chunks/s fills the Orin’s GPU (open item 10).

Where sources and the repo disagree

• Gemini 335L baseline. The repo’s open item 5 says sources disagree on the baseline. Orbbec’s own datasheet (Gemini 330 series, V1.6) gives 95 mm for the 335L and 336L, and 50 mm for the 335 and 336; Orbbec’s product and shop pages also say 95 mm.^[3] Lower figures found elsewhere probably mix up the 335 and the 335L. The datasheet’s precision figures (0.8% at 2 m for 95 mm, 1.5% for 50 mm) agree with each other under the z² law at the same 0.24 px matching precision, which is a good cross-check of 95 mm.
• Depth range. Orbbec’s shop page lists 0.17–10 m with 0.5–6 m optimal; the datasheet lists 0.17–20 m+ with 0.25–6 m ideal. The near limit also depends on resolution and disparity search range (0.5 m at 1280 × 800 with a 128 px search).
• BMI088 accelerometer noise. Bosch’s product page says 175 µg/√Hz; its flyer and an older datasheet copy say 230 µg/√Hz.^[1]
• Power-mode table. The core counts and clocks above are from NVIDIA’s Jetson Linux r36.4 (JetPack 6) guide; the JetPack 7.2 table was not checked.

Decisions you can make here

• What the IMU bench test measures. It should measure end-to-end delay (motion to reading in the control process), not just noise; this toy says delay dominates. Trade: a microcontroller-read BMI088 is cheap and fast but needs firmware; an industrial AHRS is ready-made but costs about $730.
• Depth resolution for the terrain map. Trade: 640 × 400 roughly doubles the depth error (about 3 cm at 2 m, still far below a 17 cm riser) but cuts the data to a quarter and brings the minimum distance closer.
• Power mode for walking. Trade: 30 W leaves room for the policy and the terrain map and saves battery; the VLA needs MAXN (more runtime lost, see 09 Power) or the desktop GPU.
• Developer kit or module on a carrier board. Trade: about 0.4 kg saved, against designing your own cooling and connectors.

Your decision noteCopy

Built from your settings above. Paste it into an issue, a design discussion or an AI assistant session.

Next: 12 Safety covers what happens when any of this goes wrong: how the robot stops, and how power to the motors is cut.

Further reading

• Orbbec Gemini 330 series datasheet V1.6: the full spec tables, field-of-view formulas and minimum-distance table for the 335L.
• Szeliski, Computer Vision: Algorithms and Applications: the stereo chapter derives depth from disparity and explains matching.
• Mahony, Hamel and Pflimlin, Nonlinear complementary filters on the special orthogonal group (2008): the standard 3-D version of Playground B’s filter.
• NVIDIA Jetson Linux guide: power and performance: power modes, nvpmodel and clocks.

Sources

1. Bosch Sensortec, BMI088 product page and flyer — used for: gyro noise 0.014 °/s/√Hz, accel noise 175 (flyer 230) µg/√Hz, output rates to 2 kHz, bias stability <2 °/h, ±1 °/s zero offset.
2. Mahony, Hamel and Pflimlin, Nonlinear Complementary Filters on the Special Orthogonal Group, IEEE TAC 2008 — used for: complementary filtering of gyro and accelerometer.
3. Orbbec, Gemini 330 series datasheet V1.6 (and product page) — used for: 95 mm baseline, 90 × 65° depth FOV, precision ≤0.8% at 2 m / ≤1.6% at 4 m, ranges, minimum-Z table, MX6800, power, size and mass.
4. Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., 2022 — used for: depth from disparity and its error.
5. NVIDIA, Jetson AGX Orin series technical brief — used for: CPU, GPU, TOPS split, memory and power range.
6. RobStride, Product_Information (RS04 user manual) — used for: 14-bit absolute encoder, feedback contents.
7. NVIDIA, Jetson Linux developer guide r36.4, platform power and performance — used for: AGX Orin 64 GB power modes, cores and clocks.
8. Seeed Studio wiki, GR00T N1.7 with TensorRT on AGX Orin (JetPack 7.2) — used for: 0.2755 s per 16-action chunk.
9. CNX Software, NVIDIA increases the price of Jetson modules and devkits, July 2026 — used for: the $3,499 developer kit price.