Engineering field notes · Robotics & Automation / AI / Python
Getting a robot to answer a question and getting it to move are two different accomplishments. The interesting engineering starts when the same system has to decide what was requested, whether its body can do it, who currently has control, and what would prove the task succeeded.
That is the territory I have been exploring with AI Robot Mission Control. This kickoff post brings together the designs, working components, experiments, and architecture that have been developing across several projects. The goal is to codify that body of work into a shared direction of an AI-first global and localized fleet orchestration system, and plan a first milestone release, while explaining the larger system I want it to become.
It grew out of practical experiments and conversations: an Arduino-controlled robot named Billy, Raspberry Pi compute, cameras and microphones, local language models, and the question of how these pieces might become a reusable system. Eventually the question stopped being which model to install. It became how to organize the work around the model, the body, and the person standing beside it.
The ambition is a general intelligence network for the robotic systems I build: a framework of smaller frameworks working in concert. Deterministic controllers provide precise actions. Reusable skills make those actions available for more than one task. An interpretation and reasoning layer relates goals to capabilities, while each robot retains its own body description, state, local interaction, and control authority. Generality is the design goal: a common way to coordinate different bodies without rewriting the whole system around each one.
MissionControl provides the central mission service, NEXUS the robot-side runtime, ORACLE the skill registry and compatibility service, and the Arduino controller the hardware-facing layer. REN connects the work to broader supervisory context. The Unity rover adds an earlier strand of the same ambition: a virtual world in which behavior can be inspected, rehearsed, and eventually compared with physical results.
Key questions this project direction addresses
From commands to interpretation. How do I move from “execute this exact pattern” to a layer that can interpret a goal, ask for missing context, and select an appropriate action?
Skills that can be shared. How can a useful behavior become a versioned skill that another compatible robot can discover, adapt, and execute without inventing it again?
A common internal language. How do a central system and smaller robot-local systems describe tasks, observations, constraints, and outcomes so they can understand each other?
Reasoning about the body. How do both sides understand a robot’s geometry, joints, kinematics, sensors, actuators, and current limits well enough to relate an abstract goal to physical movement?
Directed and self-initiated activity. How can explicit instructions, scheduled tasks, and robot-initiated proposals coexist, with clear scope and rules for deciding what happens next?
Local interaction within a shared system. How can each robot respond to the person beside it, explain its state, and accept intervention while remaining part of a coordinated network?
Universal control for a variety of forms and units. How do I make this work for lots of robots and automation forms? Lots, and lots of robots.
Why the architecture became part of the experiment
A small robot makes a very effective critic of software assumptions. A motor command can be syntactically valid while the battery is low, a sensor is missing, someone is holding the chassis, or the robot has already stopped on an obstacle. A fluent explanation does not resolve any of those conditions.
The project’s working hypothesis is that a useful AI robot needs several kinds of reasoning at different speeds and with different authority. Interpreting an ambiguous request can take time. Updating a fleet record can tolerate some latency. Detecting an immediate stop condition belongs close to the machine. Those responsibilities deserve explicit interfaces.
My earlier Pi and UNO Q discussions explored smaller local models, occasional assistance from a more capable central host, and a persistent supervisory system. Those experiments helped shape a distributed approach: keep immediate interaction and body awareness close to each robot, and call on central coordination or more capable reasoning when the activity needs it.
The earlier experiment: giving the robot a virtual world
This project has an earlier branch of the story in Building a Unity-Based Simulator for a Differential-Drive Robot. That post introduced the Monad Unity rover: a virtual testing ground for a rover, with a longer-term intention to connect simulated and physical behavior in both directions.
The intention was to let a physical robot send observations into a virtual environment, use that environment to inspect and test behavior, and eventually investigate learning across both. Mission Control gives that intention a larger structure: which robot is represented, which task is being attempted, which software version is involved, and what authority a simulated result should have.
The earlier Unity repository contains a keyboard-controlled Rigidbody rover, terrain generation, boundary-wall construction, on-screen pose and speed information, and tilt-related movement gating. Its terrain generator combines a Perlin-noise height field with randomly placed local height modifications. These are concrete foundations for exploration, even though a repeatable evaluation harness is another step.
The movement core in RobotController.cs is short enough to show directly:
float move = Input.GetAxis("Vertical") * moveSpeed;
float turn = Input.GetAxis("Horizontal") * turnSpeed;
Vector3 forwardMove = transform.forward * move * Time.fixedDeltaTime;
rb.MovePosition(rb.position + forwardMove);
Quaternion turnRotation = Quaternion.Euler(0, turn * Time.fixedDeltaTime, 0);
rb.MoveRotation(rb.rotation * turnRotation);
That is a useful early abstraction: a forward input and a turning input act on a simulated body. The script also checks pitch and roll and skips further movement commands when the configured stability threshold is exceeded. It detects a return to stability and offers a reset; it does not implement an active self-righting maneuver.
The earlier article discusses wheel colliders and configurable sensors. The controller code shown here specifically uses Rigidbody position and rotation updates; that excerpt does not establish calibrated left/right wheel dynamics or a complete sensor model. As the simulator matures, the relationship between scene configuration, controller behavior, and the physical chassis needs explicit measurement.
Connecting the virtual rover to the larger system
The approach to connecting the earlier Unity work sits in oracle/simulation/sim_bridge.py. The bridge uses newline-delimited JSON over TCP to describe exchanges between NEXUS and a Unity scene: synchronize a pose, transfer a map, request a screenshot, or run a visual scenario. That gives the virtual rover a defined place in the larger system.
For example, the current bridge translates a NEXUS pose into a Unity request like this:
def sync_nexus_position(self, pose: Tuple[float, float, float]) -> bool:
"""Push live NEXUS SLAM pose (x, y, theta_rad) to Unity."""
result = _send_receive({"cmd": "set_robot_pose", "x": pose[0], "y": pose[1], "theta": pose[2]})
return result is not None and result.get("status") == "ok"
To complete this handoff, I plan to add a Unity-side NetworkBridge.cs listener for the TCP messages, using port 9000 by default. Its job will be to validate each request, apply scene changes through Unity’s main-thread update path, and return a clear acknowledgment or result. The Python bridge supplies the starting message vocabulary; the next work is to connect that vocabulary to the rover and the scene.
Several details deserve their own contract. Unity’s rover moves on an X/Z ground plane with Y as height, while the Python request describes a planar X/Y pose and heading in radians. An adapter must define axis mapping, units, heading direction, reference frame, timestamps, and stale-update handling. Otherwise a message can deserialize successfully and still place the robot in the wrong world.
The map handoff needs similar care. The Python bridge currently sends encoded occupancy-grid values under a field named heightmap_b64. Occupancy and elevation describe different properties. A receiving adapter must define whether it is constructing obstacles, rendering an occupancy overlay, or loading a genuine terrain height field.
Visualization, rehearsal, and evidence have different jobs
The direction I want to explore has three useful modes. A visualization mode would mirror measured state so an operator can inspect what the physical robot reports. A rehearsal mode would execute a proposed task on the simulated body. An evaluation mode would run a versioned scenario and return results against explicit criteria. Keeping these modes distinct gives each stage a practical purpose as the integration develops.
They also need separate authority. Mirroring a measured pose should not silently command the physical robot. Resetting a simulated rover should not reset a hardware fault. A rehearsal result should describe the scenario and assumptions under which it succeeded, then return to the same admission and local-control boundaries used by other mission proposals.
This is how Unity and the Gazebo-first roadmap can coexist. Unity remains an approachable visual workbench and a place for interaction experiments. Gazebo is the documented primary target for repeatable physics scenarios and certification work. The useful common element is a clear scenario/result contract; choosing a backend does not by itself establish fidelity or certification.
So, a sensible first step for the Unity connection would be read-only: send a timestamped pose, apply the agreed coordinate transform, and display connection age and robot identity. After that, I’m planning to add a repeatable simulated drive scenario with a fixed starting state, recorded environment parameters, and measured travel. Learning-based control and bidirectional physical operation can then build on what those smaller steps teach us.
The layers, and what each is responsible for

⚙️ Components working together
| Component | Responsibility | Current direction |
|---|---|---|
| MissionControl | Operator API, fleet registration, task records, execution plans and sessions | FastAPI service with SQLite persistence and an operator control board |
| NEXUS | Robot-local state, interaction, hardware communication and control arbitration | Runtime modules and a local Flask control API, with task-consumer integration underway |
| ORACLE | Versioned skills, persisted registry records and compatibility reasoning | SQLite-backed registry service and compatibility APIs |
| Firmware | Motor and servo commands, sensor telemetry and local stop behavior | Arduino Mega-oriented controller, moving toward measured bench and chassis exercises |
| REN | Broader supervisory requests and mission outcomes | API and webhook interfaces being developed for supervisory coordination |
The earlier Unity work supplies the virtual-body experiment; these layers supply mission context, compatibility, authority, and traceable outcomes around it. These names describe responsibilities more than particular computers. The intended topology places MissionControl and ORACLE on a central Windows host, NEXUS on a Raspberry Pi-class edge computer, and time-sensitive hardware behavior on the microcontroller. Moving a service to another host should preserve the contracts and authority rules.
There is also a lightweight robot/ voice-and-vision request loop being developed alongside the richer nexus/ runtime. This work in progress gives me a simpler interaction path to experiment with while the fuller runtime takes shape. Part of the architecture work is deciding which capabilities belong in that small loop and which should be shared with NEXUS.
What distinguishes the approach
The distinguishing idea is the combination of body-aware capability checks, explicit execution records, and robot-local authority. None of these ideas requires a new foundation model. Their value comes from how consistently the implementation applies them.
🪛 Design choices and tradeoffs
| Design question | Choice in this project | Tradeoff |
|---|---|---|
| Where should reasoning happen? | Local interaction and state, with central inference and coordination available | More deployment and connectivity states to handle |
| How does a robot qualify for a task? | Compare skill requirements with a versioned robot profile and optional live capability snapshot | Profiles must stay accurate and snapshots must be fresh |
| Who can interrupt movement? | The design reserves immediate authority for robot-local control and firmware | Every movement path must obey the same arbitration |
| What counts as an outcome? | Separate plans, sessions, lifecycle states and evidence | More records and correlation work than a direct command loop |
| How are different bodies supported? | Shared contracts and embodiment/transport descriptions | Each new body needs an adapter and practical hardware validation |
The approach brings existing robotics ideas together around a common operating model. Controllers, middleware, planners, and simulators can fit beneath or beside these boundaries. I’m keeping the interfaces open to ROS and other adapters so the framework can grow around different hardware and software choices. The challenge is making those choices cooperate through a consistent language of capabilities, tasks, and results.
The robot needs a description of its body
A mission service cannot sensibly assign the same action to a tracked rover, an articulated arm, and a microphone on a desk. The shared models therefore describe the robot class, its actuators and sensors, supported control domains, compute tier, and software versions.
The actuator descriptor in shared/robot_profile.py is one example of that shared language:
class ActuatorDescriptor(BaseModel):
model_config = ConfigDict(extra="forbid")
schema_version: str = "robot_profile.v1"
actuator_id: str = Field(min_length=1)
actuator_type: Literal["motor", "servo", "linear", "gripper", "led", "speaker"]
control_domain: str = Field(min_length=1)
max_value: Optional[float] = None
min_value: Optional[float] = None
units: Optional[str] = None
joint_name: Optional[str] = None
A descriptor makes assumptions inspectable. A motor belongs to a control domain. A range has units. A joint has a name. Rejecting unexpected fields can catch a misspelled configuration key before it becomes an assumption hidden in runtime behavior.
The code also distinguishes a relatively stable RobotProfile from a RobotCapabilitySnapshot. The profile describes the configured machine. The snapshot describes present conditions such as healthy sensors, faulted actuators, available control domains, emergency stop, and local override.
Engineering note / Configured is different from available
A robot may have a camera in its profile while the camera is disconnected today. Capability matching needs both the body description and current health. Freshness checks, trustworthy measurements, and enforcement at execution time are additional requirements; a valid schema alone supplies none of them.
Compatibility can be concrete, inspectable code
ORACLE grew from the idea of central assistance in the earlier experiments. In its current implementation, which is still evolving, the registry service also performs deterministic compatibility checks. That is useful: some questions should have an explainable result without requiring a language model to improvise an answer.
This excerpt from oracle/registry_service.py compares required control domains with those currently available:
available_domains = set(
capability_snapshot.available_control_domains
if capability_snapshot is not None
else robot_profile.supported_control_domains
)
missing_domains = sorted(set(skill_contract.required_control_domains) - available_domains)
if missing_domains:
reasons.append(
CompatibilityMismatchReason(
code="missing_required_control_domain",
field="skill_contract.required_control_domains",
message=f"Missing required control domains: {missing_domains}.",
)
)
The surrounding implementation checks other conditions, including robot class, required sensors and actuators, sensor sample rates, compute tier, and snapshot indications of emergency stop or override. A rejection can return a structured reason rather than a vague refusal.
Compatibility is an admission question: do the skill’s requirements match this robot’s declared capabilities and present state? The next question is execution: can that robot perform the task under the conditions it encounters? I’m working toward keeping these checks connected throughout a mission, so a capability change or local fault can affect a decision after the initial assignment as well as before it.
A mission deserves a plan and a session
Suppose I ask a tracked robot to move forward 1.5 meters. The sentence expresses intent, but leaves questions open: which robot, which skill version, what speed, what stop conditions, and what evidence will establish completion? An execution plan is a place to make those choices explicit.
A session records a particular attempt. That separation matters when a task is paused, retried, rejected, or interrupted. The request may stay the same while the conditions and outcome change.
The repository’s shared/execution.py defines allowable session transitions. Here is a selected portion of that actual transition table:
"accepted": {"running", "aborted"},
"running": {"paused", "completed", "failed", "aborted", "rolled_back"},
"paused": {"running", "aborted", "failed", "rolled_back"},
"completed": set(),
"failed": set(),
A transition table is a small but useful discipline. A completed attempt should not silently become running again. A new attempt should have its own identity and history. The models carry plan, session, robot, skill, and trace identifiers so the service can relate records across handoffs.
The intended admission flow is straightforward: interpret the request, resolve a supported skill, check the robot, create a bounded plan, obtain local acceptance, execute, and record evidence. The current integration work is bringing the task-stream consumer and executor into NEXUS startup so this becomes one continuous mission path.
Who gets to say stop?
A central service can know more about the overall mission while knowing less about what is happening beside a wheel right now. The design therefore gives the local runtime authority to reject or stop movement. Operator interaction can acquire a temporary control lease that blocks autonomous movement through the local control boundary.
The actual predicate in nexus/core/control_api.py is deliberately compact:
def _should_block_movement(self, source: str, command: str) -> bool:
return self.is_override_active() and self._is_autonomy_source(source) and command in _MOVEMENT_COMMANDS
This is arbitration logic, not authentication. The caller’s source label describes how the command should be treated; by itself it does not prove the caller’s identity. A deployed system also needs an appropriate access boundary around its control endpoints.
One of the integration tasks is bringing voice movement and mission-driven movement through the same local arbitration path. The voice pipeline currently talks directly to the Arduino bridge; I’m working toward a common boundary that applies local control leases and guardrail decisions wherever the request originates.
That also makes interpretation a practical engineering problem. The early voice parser uses keyword matching, which can confuse a phrase such as “do not move forward” with a movement request. The next iteration needs deliberate intent handling, stop precedence, and a way to clarify or reject ambiguous requests. This is one place where the larger intelligence layer has a concrete job: understanding what was meant before selecting a physical action.
Engineering note / A stop needs its own path
The firmware includes a motion watchdog and stop commands, while higher layers contain arbitration and guardrail concepts. Those layers need measured tests with the actual controller. Watchdog timing, sensor latency, and emergency-stop wiring are properties to verify on the machine, not guarantees created by naming a function ESTOP.
The most revealing example: what does completed mean?
The current task executor has a drive.forward.safe path. It converts requested distance and speed into an estimated duration, sends a motor command, waits, and then reports completion. The result assignment in nexus/planning/task_executor.py currently looks like this:
await self._drive_forward_cm(cm=cm, speed=speed)
task.result = {
"distance_driven_meters": distance_meters,
"completed": True,
}
That code reports the requested distance under a name that sounds measured. Yet waiting for a timer does not establish how far the chassis traveled. The firmware might have stopped early, a track might have slipped, or the serial bridge might be operating in its no-hardware mock mode.
This is where the architecture guides the next implementation step. I want the result to distinguish commanded distance, estimated distance, measured travel, interruption reason, telemetry freshness, and final status. Completion should follow defined acceptance criteria, including tolerances and timeouts, so the reasoning layer can work with what happened rather than merely what was requested.
I’m working toward those richer outcome checks together with explicit handling for unsupported tasks. If a robot cannot perform an action, that should become a useful response to the planner: select another skill, choose another body, ask for help, or explain the limitation.
Simulation helps when its claims are explicit
The current simulation work I’m doing includes a PyBullet planner path, a Unity bridge contract, and a Gazebo adapter boundary. Each has a role in the design: lightweight planning experiments, visual interaction and rehearsal, and a path toward repeatable physics scenarios. Gazebo is the intended primary physics and certification target; implementing its execution backend is part of the next simulation work.
Some other simulation paths can fall back to mock results when the physics dependency is unavailable. That is useful for exercising a request/response path. A mock reporting a feasible maneuver must remain visibly different from a physics-backed result, and neither should automatically authorize physical execution.

Each physical trial adds context: a chassis, firmware revision, sensor configuration, floor, battery condition, and maneuver. I want to retain that context with the skill’s evidence so compatibility and deployment decisions become better informed over time. Certification and deployment models give this work a starting structure; the longer-term aim is to connect scenario results, physical outcomes, and decisions about where a skill should run.
Memory, personality, and a robot that can explain itself
Interaction is part of the project too. NEXUS includes identity loading, avatar-related modules, speech paths, local model integration, and escalation to central inference. MissionControl stores operational records and conversation data, with optional memory components.
I want a robot to explain why it stopped, what it was attempting, and what it needs next. Personality can make that interaction approachable. Operational state needs stricter treatment: an expressive avatar should reflect a known state rather than make a confident story out of missing telemetry.
A future status response might say, “The requested move was interrupted after the front sensor reported an obstacle; I am waiting for local clearance.” Producing that sentence reliably requires a real event and a linked execution record underneath it. The wording is easy once the system actually knows what happened.
The broader shared-world-model idea remains exploratory: robot-local observations, central mission context, and evidence with provenance and age. Before adding sophisticated shared memory, the first useful step is honest connection status and durable, queryable task outcomes.
Where the project stands today
🌐 Where the project stands today
| State | Examples |
|---|---|
| Working foundations | MissionControl APIs and operational persistence; ORACLE’s persistent registry and compatibility endpoints; shared profiles and execution models; the local control API; voice and vision modules; the serial bridge; and Arduino controller code. |
| Integration in progress | Connecting task intake to physical execution, unifying local arbitration, measuring completion, and developing deployment and recovery behavior. |
| The next expansion | Gazebo scenario execution, evidence-led skill evaluation and rollout, ROS and REN adapters, ESP32 firmware work, and a path toward coordinating different robot bodies. |
The current body of work includes schema validation, API-boundary checks, registry persistence, compatibility reasoning, execution models, and selected executor behaviors. These initial tests are useful foundations, and I’m working toward turning them into practical exercises across the whole system: a request understood centrally, accepted locally, carried out by a body, and returned as a meaningful result.
ORACLE persistence is now implemented, giving the skill registry durable records to build on. The next proofing work is to establish that each layer compiles or starts correctly, that commands and observations are understood across the network, and that the real chassis stops within an acceptable distance. I want these to become repeatable exercises that can be carried forward when a controller, body, or skill changes.
Navigation integration is also a current work in progress. I’m working through the relationship between goto(), path following, low-level movement, and the policies that decide when to stop, replan, or ask for help. The aim is a bounded navigation architecture in which a high-level destination becomes a sequence of actions the local robot can evaluate and execute.
Milestone for the first release
My first release target will be one tracked robot, one supported skill, and one observable mission loop: register, receive a task, obtain local acceptance, move under bounded conditions, report a measured outcome, and respond predictably to interruption. That milestone will bring the central service, the local runtime, the shared language, and the physical controller into one practical exercise.
That requires wiring the mission consumer into startup, routing movement through a consistent local boundary, correcting parsing and navigation behavior, rejecting unsupported tasks, and making disconnected hardware explicit. It also requires proving the stop behavior on the actual controller and keeping the recorded outcome honest when something goes wrong.
The milestone exercises will include nominal travel, operator override, obstacle interruption, lost communication, and restart recovery for stored records. Each result should carry enough context to explain what happened and feed the next decision. This first complete loop is the base on which the broader network can grow.
Engineering note / The engineering question I keep returning to
Can the next person tell what the robot was asked to do, why it accepted, what it actually did, and why it stopped? If the answer depends on reconstructing a chat or guessing from a console message, the system still has work to do.

Where I want to take it
The larger ambition is a coordinated intelligence system that can inhabit many kinds of robotic frames. A rover, a workbench arm, a small conversational robot, and a future sensor-equipped mechanism would each retain a local sense of what they are, what they can do, what they are observing, and who is interacting with them. They would participate in a shared system without needing identical hardware or identical models.
I want to be able to express a goal at the level that makes sense to a person: inspect an area, bring a camera closer, watch an experiment, help organize a workbench. The system would interpret that goal, identify what is missing, and break it into activities that available bodies and skills can support. Sometimes the right answer would be a deterministic sequence already known to work. Sometimes it would require reasoning, a simulated rehearsal, another robot’s observation, or a question back to me.
Skills are central to that ambition. A useful skill should describe its inputs, requirements, constraints, and outcomes in a form other parts of the system can understand. The shared part might be the intent and contract; the physical implementation would depend on the body. Moving to a viewpoint means something different to a tracked rover and an articulated arm, but both could participate in the same inspection task through appropriate embodiment adapters.
The same common language needs to reach down into kinematics and back up into meaning. Joint ranges, geometry, coordinate frames, sensor placement, and actuator limits constrain the possible actions. Observations and measured results then return as context for the next decision. I want central planning and local execution to reason about the same physical situation at different levels of detail, with provenance and uncertainty preserved.
There is room for both directed and self-initiated work. I might assign a specific task, schedule a routine, or let a robot propose an activity within an agreed scope: inspect a change, refresh a map, check a sensor, or request assistance. A proposal would still pass through capability checks, permissions, resource limits, and local arbitration. Initiative becomes useful when the system can explain its reason and accept interruption.
Local interaction is just as important as fleet coordination. A person beside one robot should be able to ask what it is doing, redirect it, or stop it. That robot should be able to explain its part of the shared work without losing its immediate responsibility to the person and environment around it. The conversational layer, expressive interface, deterministic controller, and mission system each contribute to that interaction.
Over time, I want simulation and physical experience to inform one another: rehearse a proposed behavior, compare it with real telemetry, retain what was learned, and use that evidence when choosing the next action or adapting a skill. Learning and improvement would remain tied to versioned changes, evaluation, and an explicit decision about what is allowed to run. This is a longer-term research direction that gives the current integration work a purpose.
The framework of frameworks is how I want to make that ambition manageable. Small, understandable systems can do their own jobs well, while shared contracts let them cooperate: deterministic direction when precision is needed, skill reuse when a behavior is already understood, deeper reasoning when a goal is ambiguous, and immediate local response when a person or the environment changes the situation.
That connects with the Agentic RDE work: requirements, decisions, and evidence need a durable home as these systems evolve. A clear architecture also makes it easier to return to an experiment, change a model or controller, and understand which assumptions must be revisited.
Many bodies, shared intelligence is the direction. The first milestone gives that direction a physical starting point; the larger goal is a network of capable, understandable robotic systems that can work together and with people.
As more repositories and the central body of work become public, I’ll add those links here. The earlier Unity rover repository and its project article are available now.
