Perception & Robots

Assets declare userData.sensors = [{camera, 64×64, 5 Hz}] and the engine renders the declared camera to an offscreen target on a budget of at most one sensor render per engine frame, handing the RGBA buffer back before the asset's animate() runs. Perception is slow (5 Hz), control runs every frame on the latest percept — the same split a real rover uses. The channel is declarative, like the city's spinners and oscillators: one engine mechanism now serves the mine excavator, the undercity humanoid, and whatever comes next.

Graceful degradation is a hard contract. The same module file must work in an engine with no sensor channel at all: until the first frame arrives, both robots play a deterministic baked loop — the excavator its 24 s dig cycle, the humanoid a pure-t figure-eight patrol. Frames arriving flips them (stickily) into closed-loop autonomy; frames stopping for 3 s hands control seamlessly back to the baked path.

The undercity has pitch-black corners where vision alone goes blind, so the humanoid carries a 905 nm micro flash-LiDAR in its chest: 15 beams over a 90° sector, 20 m range, 6 Hz. In the engine it is a raycaster fan — an admitted engineering approximation — but everything about the receiver is computed for real, and that computation rewrote the design (section 04). It now ranges 2.5–20 m to better than 1 cm on 60 nJ pulses into a 1024-microcell SiPM; inside 2 m it saturates and reports proximity only.

The walk is not keyframed. A MuJoCo biped built 1:1 from the visual geometry tracks gait commands under g = 3.71 m/s²; walk/turn/idle cycles (12 channels) are baked and phase-blended by (v, ω) at playback, with the stride shrinking as it slows. Servo overshoot, touchdown knee flex and torso nod are dynamics, not artistry — and because MuJoCo carries torque and rate at every step, so is the power bill: 95 W walking, 62 W standing, CoT 0.64. A second model (flat feet, free pitch, torque-driven ankles) answers the question the bake rig cannot: it stands on its own under capture-point feedback, and once allowed to take capture steps it walks, absorbs a ~30 N·s shove, and lifts its own balance ceiling to 0.45 m/s.

It can also run, which is a genuinely different problem: flight phases mean no support polygon and no ZMP to steer, so the model becomes a spring-loaded inverted pendulum and the law becomes Raibert's — thrust sets hop height, footstep placement sets speed, hip torque holds attitude. Measured: 0.59 m/s at a 26% flight fraction, and a bounding gait to 1.00 m/s at 71%. The comparison that makes it interesting is gravity: the same controller with the same thrust spends 68–71% of the time airborne on Mars and only 33–41% on Earth. Long airtime is not a controller defect, it is 1/g — the same reason Apollo crews switched to loping in one-sixth gravity. Mars makes hopping easy and accelerating hard, since traction is capped by µ·m·g and each step can add only 0.56 m/s against Earth's 1.47.

One more thing low gravity pays for: heat. Walking dumps 86.6 W into a sealed, windless undercity, which for the body as a whole is nothing — a 7.1 K shell rise. Resolved per joint it is another story: the hip must dump 22.9 W into a housing the size of a coffee mug, 71.7 K above ambient on its own surface, so the cooling design is really the conduction path into the thigh casing. A COMSOL FEM later audited the lumped model's weakest assumption — that the thigh is isothermal — and found it 55% optimistic like-for-like: the shell is a fin at 56% efficiency, its tip running at a third of the root's rise. The winding lands at 54 °C against a 120 °C limit (the lumped model had said 50, its area undercount and its perfect-fin assumption almost exactly cancelling). What makes that close is the planet. Joint torque scales with gravity and copper loss with torque squared, so the identical machine on Earth would dissipate 129.9 W at the hip and cook its winding to 189 °C — full T⁴ radiation helps at that temperature, and fin droop takes the help right back. The hand's story turned out differently: a MuJoCo contact audit found its 2.90 grip factor rested on a guessed acceleration (the real walking hand sees 22× more) and on no torque ledger at all — rim-pinching a tray fails on moment on any planet, while carrying it on the palm holds a measured 4× margin on Mars and Earth alike, because the fingers' 24 N press dominates the friction budget and gravity nearly cancels out. Low gravity is what makes this robot's thermal design credible — the hand, it turned out, owes Mars nothing — and the bill for that dispensation arrived earlier in this page, as poor traction and a capture point that runs away from the toe.

The last set of invented numbers was how it behaves around people, and checking those against human factors turned up a real fault rather than a nuance. ISO/TS 15066's separation formula is dominated by the human's approach speed, so a robot that keeps its authority to move is still owed 1.97 m even while standing still — while the comfortable distance for handing something over is 0.65 m. Enforced literally, this robot could never pass anyone a cup of water. The escape is in the standard: separation monitoring is one of four collaboration modes, and a stationary machine may work under power-and-force limiting instead — for which this one already qualifies, since a wave is 13 N and an unintended start 93 N against a 280 N chest limit. It now switches modes on its own speed. The second finding was simpler and more embarrassing: at handover range a 1.65 m robot has to pitch its head down 24.8° to meet a seated person's eyes, and its neck only had a yaw axis. It looks down now.

It also falls over sometimes — the stepping controller's feasible region is patchy, and pretending otherwise would be the same mistake as the guessed numbers. A fall is an inverted pendulum about the ankle, and the number that matters is not the centre of mass but the head: it sits at the far end of the lever, so it lands at 4.46 m/s against the CoM's 2.42, 1.84× faster purely from v = ωr. A fall deposits 132 J; the visor glass can absorb 16. That is why the visor's guards now stand 3 mm proud of the screen — the frame hits the floor, not the glass. For a bystander it is worse than anything walking can do: 13.3 kg of effective mass at 4.46 m/s is 493 N even through a compliant skin, 1.8× the chest limit, and no speed governor can help because the energy comes from height, not from travel. The mitigation is distance, and it already exists: a topple reaches 1.65 m, the SSM separation is 2.40 m. Falling is also the one place where Mars helps without sending a bill — 1.63× longer to react, 1.63× slower landing, 2.64× less energy — and the robot can push itself back up on 29 N·m of hip torque against the 150 available.

And yet the robot never runs on patrol. A 39.8 kg humanoid at 0.6 m/s delivers 478 N in a 50 ms collision — 1.7× the ISO/TS 15066 transient chest limit. So running is retained strictly as an evacuation capability: an explicit action arms it, the arming is refused outright if anyone is inside the SSM radius, and a person entering mid-run drops it back to walking within one control step. Behaviour otherwise: figure-eight patrol → LiDAR and vision avoidance → greeting (stop, face you, ◠ on the visor, wave) → self-docking when the 480 Wh pack drops below 15%, all at 0.35 m/s — not a style choice but a bound, and which physics binds has now changed twice (section 04).

The newest capability is voice, and it was budgeted twice on purpose. A first pass in dB arithmetic (uniform-absorption Sabine, geometric noise scaling) said the foyer rings like a church and the robot is deaf while walking; a second pass at simulation grade checked whether the first was lying. An image-source model with per-surface, per-band absorption raised the mid-band reverberation from 2.4 s to 4.7 s and moved the critical distance to ~1.3 m — which puts the 1.6 m greeting stop slightly outside the direct field, killing the tidy coincidence the first pass had reported. Intelligibility survives anyway: C50 clarity sits at 0 dB (early reflections inside 50 ms reinforce speech rather than smear it), and a waveform-level simulation of the six head mics — speech plus both hip gearboxes' whine, every path convolved with its own room impulse response — shows delay-and-sum beamforming lifting the standing SNR from −3 to +4.7 dB. Walking, the same beamformer recovers −22 to only −14 dB: still deaf, now to simulation grade. The sound-source localizer wrote its own policy: scanning while walking, it locks onto the gearbox (18 dB louder than the talker), so the robot never steers by sound unless it is standing still.

The last unexamined subsystem was the one the robot visits twice a sol: the charger. Auditing it turned up the worst bookkeeping error on this page. The battery ledger had sized the pack against 80% of full capacity while counting cycle life over the 25–95% operating window — two SOC windows in one ledger, never multiplied together. Multiplied: the pack's usable window held 266 Wh against a 296 Wh shift, so the "4 h shift" promise failed on day one, by 25 minutes, no ageing required — and it hid for three rounds because every audit checked its own corner. The self-consistent resize (fifth battery change, 380 → 480 Wh) pays twice: a bigger pack cycles shallower, so cycle life rises from 500 to 643 full-equivalent cycles. Two quieter findings came with it. The dock's support ledge is a power component, not furniture: resting on it lets the actuators de-energize, and without it 37 W of holding torque would burn 170 Wh per sol — half a shift — during charging. And when the robot charges is a lifetime parameter: the pack spends two thirds of every sol parked, so floating at 92% costs 4.3%/yr of calendar fade against 1.4%/yr at 45% — just-in-time charging outlives charge-on-arrival by 57 sols at zero hardware cost — a number that took a second, harder look to get right (section 06). At 0.27 C the charge never even reaches the CV phase below the 95% ceiling; the one real hazard is that 48 V / 3.1 A sits far above the DC arc-sustaining minimum, which is why the contacts stay dead until the mate is verified.

The robot feared its own laser. With ?debug=1 the LiDAR fan renders as bright green rays — and the nav camera saw them. The rays crossed the scan band, dragged the adaptive threshold up, and the whole floor classified as hazard (hazR hit 0.83 on open ground; the failure line is 0.5). A real 905 nm beam is invisible to a silicon visible-light camera, so the fix mirrors physics: the debug overlay is dropped from the nav-camera render pass via a per-camera material hook. False alarms fell from 0.83 to 0.00.

MJCF ranges compile in degrees. First gait bake: every joint amplitude came out at exactly one tenth of the command, and every servo sat saturated. Cause: MuJoCo's MJCF default angle unit is degrees, so range="-1.5 1.5" pinned every joint at ±1.5° — the constraint solver was politely cancelling 150 N·m of servo torque against the limit. One line (<compiler angle="radian"/>) ended a thoroughly confusing debugging session.

"Flat feet deadlock a biped" — retracted. That was written down as a lesson after the hips maxed out their torque without moving. Re-tested once the radian bug was fixed, and it was never true: the deadlock was the degree/radian bug wearing a costume. Worse, the "fix" (round point-contact soles) removed the feet's ability to exert pitch torque, which is the entire basis of the ankle strategy — the model could then only stand by leaning on its rig. The real answer is flat feet and a balance controller.

The rig was holding the robot up, and the ledger caught it. The gait bake runs on a planarizing pitch spring, standing in for a gait-trainer boom. Measure what that spring contributes and it is 55 N·m while standing — divide by the 148 N weight and it is a demanded ZMP shift of 37 cm against a 10 cm foot. The baked posture was not balanceable at all; it just looked fine. (The first attempt to check this was worse than useless: computing the centre of pressure from the simulator's own contact forces and finding it "100% inside the support polygon" is a tautology — the CoP is inside the contact hull by definition.) A second model with flat feet, free pitch and torque-driven ankles now stands under capture-point feedback and takes an 8 N·s shove.

A 1.3 kW walking robot. First pass at billing power from joint torques gave 1.3 kW walking and — absurdly — 1.9 kW standing still. Cause: copper loss evaluated at the joint instead of at the motor. A 44 N·m joint torque behind a 100:1 harmonic drive is 0.44 N·m at the rotor, so I²R was overstated by N² ≈ 10⁴. With the gearbox in the model it lands at 95 W walking, 62 W standing, CoT 0.64 — and the 260 Wh pack that was advertised as a 4 h shift turns out to be 3.2 h, so the design now carries 340 Wh.

"SNR 74, huge margin" meant the detector was blinded. The LiDAR link budget said 5,500 photoelectrons per pulse at 20 m and stopped there, satisfied. A ranging Monte Carlo asked the next question — how accurate is the distance — and the answer was: a metre short. SPADs are Geiger-mode, one shot per microcell, so 5,500 photons over 64 cells is 86 per cell, every cell fires on the pulse's leading edge, and the mean first-arrival lands early. That is range walk, and the standard cure (infer photon count from how many cells fired) is exactly what saturation destroys. The fix was counter-intuitive: transmit less. At 60 nJ into 1024 cells the receiver sits in its linear regime and ranges to under a centimetre.

Three physics vetoed the walking speed. 0.55 m/s was a guess. Motion blur caps it at 0.30 m/s (at the exposure the dim foyer forces, a turn smears the 64-pixel frame by 15 pixels — the camera was useless whenever the robot turned). Ankle-strategy balance caps it at 0.30 m/s, because Mars stretches the pendulum constant to 0.491 s and pushes the capture point past the toe. ISO/TS 15066 contact force caps it at 0.35 m/s for a 3 mm foam shell. Three unrelated calculations, one answer. The avoidance threshold fell too: 1.2 m is 1.1 m short of what speed-and-separation monitoring demands for an approaching human — though the 2.5 m greeting radius, invented as a social distance, turned out to be the right safety number by accident.

Looking too far makes cowards. The undercity scan band initially reached ~7 m ahead, so distant dark door metal read as a permanent threat and the robot crept everywhere at 0.17 m/s with a “!” face. Same trap the mine robot hit with dark strata on the pit wall; same fix — narrow the band to the 1–4.5 m near field. The excavator's variant is even better: it was afraid of the shadowed side of its own spoil mound, and needed a mapped-approach exemption to dump its load.

The 0.3 m/s coincidence did not survive its own fixes. Three physics landing on the same speed made a satisfying story, and it was true — of the first design. Then the f/1.4 lens fixed the exposure problem and the blur limit jumped to 1.33 m/s. Then a capture-step walking controller replaced the no-stepping assumption and the balance limit rose to 0.45 m/s (measured over a 25-cell speed × step-time grid, of which only 0.45 m/s survived five out of five perturbed starts — a lone 0.70 m/s cell was luck, and a shove knocked that one straight over). What remains is a single binding constraint: ISO/TS 15066 contact force, 0.35 m/s for a 3 mm foam shell. So the upgrade path is now specific instead of vague — 15 mm of compliant skin buys 0.84 m/s, after which balance binds again and the gait controller itself has to improve. Each fix surfaced the next limit, which is what convergence looks like; worth saying plainly that the prettiest result on this page had a shelf life.

The gait controller does not port to Earth. Same code, same gains, gravity set to 9.81 — it falls over. T_c drops from 0.434 s to 0.267 s, so every footstep lands in the wrong place and every step time is mistuned. Gait gains are calibrated to a planet.

The acoustics coincidence lasted one day. The voice budget's first pass reported a critical distance of 1.50 m — almost exactly the 1.6 m greeting stop, a satisfying "the geometry already protects the microphones" story. The L2 image-source model, fed per-surface material data instead of one averaged absorption number, moved the crossover to ~1.3 m and put the greeting stop on the reverberant side. The same round caught a subtler trap: the first ego-noise draft billed acoustic output off electrical power and concluded the robot couldn't hear even standing still. Copper loss is silent; gear mesh is loud — noise follows mechanical throughput, which is also why an unturning gear at full holding torque makes no sound. And the DOA scanner, run on raw walking audio, confidently pointed at the robot's own hip.

The battery ledger never multiplied its own two windows. The EOL sizing round — itself a correction — checked the shift against 80% of full capacity, while its cycle count used the 25–95% operating window the FSM actually enforces. Each number was defensible; their product was not: 380 Wh × the window = 266 usable Wh against a 296 Wh shift, so the "4 h shift" promise failed on day one at 3 h 35 min. Three subsequent audits (review, endgame, voice) walked past it because each checked its own corner. The fix is the fifth battery change, 380 → 480 Wh, sized self-consistently this time — window, Wöhler cycle life, and calendar ageing solved together — and the charging ledger that caught it also repriced the dock's "1.7 h recharge" (2.3 h: the dock's 150 W must also feed the robot) and promoted the support ledge from scenery to a 62 kWh/yr power component.

The thermal ledger was right by accident. Its 50 °C hip-winding figure rested on two assumptions that a COMSOL FEM later tested: that the thigh shell is isothermal (it is a fin at 56% efficiency — the tip rises a third of what the root does, so this alone was 55% optimistic) and an exterior area of 0.137 m² (the as-built geometry has 0.172 — 26% pessimistic). The two errors nearly cancel, and the winding moves only 50 → 54 °C. But cancelling errors are not absent errors: on Earth they stop cancelling (183 → 189 °C), and they would drift apart at any other power level. The FEM run passed all three gate classes — energy closure 1.000, the isothermal-limit case matching the analytic answer to 0.1 K, and an independent fin-theory cross-check landing within 2% of the conduction case.

The hand's 2.90 was three errors wearing one number. The grasp factor used a guessed 0.39 m/s² hand acceleration (harvest it from the walking model and the peak is 8.56 — 22×), carried no torque ledger (a tray pinched by its rim hangs its centre of mass 15 cm outside the grip line; gravity's 1.67 N·m against the 0.48 the finger pads can muster means it cannot sit level even statically — MuJoCo tips it 138°), and never said which grasp it priced. The carry configuration measures a 4× margin under the real walking excitation — identical on Mars and Earth, which retires the card's "low gravity forgives the hand" claim: with 24 N of finger press dominating the normal force, gravity nearly cancels from both sides of the ledger. The audit also ate three rig bugs before producing physics — a palm plate moonlighting as a second support, a world-frame trajectory teleporting at its loop wrap, and convolve's zero-padding forging an 11 m/s² whip — each caught because a sweep that is not monotonic in its knob is a rig smell, not a result.

The battery ledger and the thermal ledger had never been multiplied either. The pack's life came from a table of calendar-fade rates in percent per year, multiplied by elapsed years, evaluated at the parked state-of-charge, at room temperature. Every one of those four choices is wrong in a different way. SEI growth is diffusion-limited through the film it builds, so fade goes as √t, not linearly. The pack does not live at room temperature: it sits in a backpack sharing 11 W with the compute board, bolted to a torso dissipating 87 W, and computes out at 28.8 °C while working — 6.8 K hot, which by Arrhenius is 1.6× the calendar fade. And calendar ageing does not pause while the robot works: eight hours a sol pass at a mean 64% charge, at the hottest pack temperature of the day, which the old account simply did not count. Corrected, pack life drops 15% (473 → 404 sols) and the just-in-time charging prize shrinks from 76 sols to 57 — because over the real duty cycle the cycle term at end of life is 7.4× the calendar term. This robot is worn out by working, not by waiting. The consolation: insulating the pack from its hot neighbour buys back only 20 sols against the scheduler's 57, so the fix stays in software.

And then the SRAM version won on area too. The campaign's parting advice — use a macro — was worth building rather than asserting, so the histogram was rebuilt on one OpenRAM 512×32 block, three ten-bit counters packed per word. The same testbench and the same 4,809-stamp golden vectors pass unchanged on both simulators, finishing at the identical tick: the storage architecture changed, the interface and the truth did not. Synthesized, it is 1,753 cells and 0.304 mm² against the flop version's 38,142 cells and 0.555 mm² — 45% less area, 95% fewer cells, 98% fewer flip-flops. Which sharpens the lesson one more turn: "a routability answer, not a density answer" was only half right. At this size the macro wins both, and the thing that unlocks it is a packing decision — 3×10 bits fits 1,050 counters into one 16 kbit macro, where the obvious 2×16 bits would need 525 words, overflow 512, and cost a second macro. And it routes: the same OpenROAD flow that spent eight rounds failing to close the flop version walked this one from 1,563 detailed-route violations to zero in eight iterations and about six minutes, with +0.05 ns of setup slack at 100 MHz and a clean antenna check. There is a GDS. Same function, same golden vectors, two architectures: one does not exist, the other is manufacturable.

Synthesis said the flop histogram wins; the router said it cannot exist. The LiDAR front end's flop-based histogram beat the SRAM-macro area estimate at synthesis — then failed place-and-route three times on the same error: a flat 1050-way read mux converges 12,600 nets on one point, and no placement knob could route it. Banking the histogram per channel (a wiring fix that also saved 19% of area) passed global routing, but detailed-route DRC still would not converge against the pin density of 12,899 enable-flops — on routers five thousand commits apart. Timing was never the problem. The conclusion the campaign bought: SRAM macros in histogram chips are not an area optimization, they are how the read mux gets pre-routed inside the macro — and the original scaling estimate, which assumed one, was right for a reason its author didn't know.

Slow transients leak into “idle”. The gait rig's pitch-support spring settles over many seconds; a 3 s settle left the idle loop drifting 0.4 rad through the recording lap. 8 s settle, three warm-up laps to reach the limit cycle, and a smoothstep end-blend brought all three loops to zero wrap error.