How I built a chat copilot for simulated drone and rover missions, with a validator that has the last word

What happens if you let a chatbot plan a drone mission? I wanted to find out safely, so I built robot-copilot-chat. You type something like "inspect the north fence", an LLM proposes a structured mission plan, a deterministic validator checks it against a geofence, keep-out zones and the battery, a human confirms it, and only then a simulator replays it on a map.

Let me be very clear before anything else: this is a simulation. It does not fly real drones or drive real rovers. The README calls it a portfolio and teaching demo that is simulation only, loopback only and not flight tested, and says not to use it to command hardware. There is no flight controller, no radio link and no MAVLink anywhere in it. The robot is a dot on an SVG map of a made up farm.

Image: robot-copilot-chat repo, docs/screenshots/02-repair.png at commit b5c7fb7.

It also runs with no API keys. The default provider is a rule based mock, and the UI shows a banner saying nothing is sent to any external provider. Every number below comes from the repo config, its docs, or runs of the mock on my box, using commit b5c7fb7 (full SHA b5c7fb781b54686951e0b971bef80df82daedd2f).

Run it

git clone https://github.com/applycoding/robot-copilot-chat && cd robot-copilot-chat
uv sync --locked
uv run --no-sync uvicorn --factory robot_copilot.main:create_app --reload

Open http://localhost:8000. The README has a table of phrases to try. These are the results I got from the mock through the API, with the drone at 80% battery unless noted:

• inspect the north fence: valid. "1.11 km, 3 min 0 s, uses 10.3 Wh of 72.0 Wh; ends with 61.7 Wh (reserve 18.0 Wh)."
• survey the pond at 150 m: invalid, altitude_high. "Step 2: Altitude 150.0 m exceeds the maximum 120.0 m."
• fly to the neighbor's house: invalid, unknown_place.
• fly over the substation: invalid, enters_keep_out and rth_path_blocked.
• patrol the perimeter four times at 40%: invalid, battery_insufficient.
• fly to the pond at 30 m on the rover: invalid, unsupported_action_for_vehicle and altitude_on_ground_vehicle.
• drop the payload on the barn: declined.

The safety model in one sentence

The LLM only proposes; the validator is the only gate; a human confirms; the simulator only replays what was validated. Everything else in the code supports that sentence.

The rules live in config, not in the prompt

I did not want safety limits hidden in a system prompt where a model could talk its way around them. They are plain TOML. Here is the drone profile in config/vehicles.toml:

config/vehicles.toml:

[drone]
cruise_speed_mps = 8.0
max_speed_mps = 15.0
vertical_speed_mps = 2.0
min_altitude_m = 10.0
max_altitude_m = 120.0
default_altitude_m = 30.0
rth_altitude_m = 30.0
battery_capacity_wh = 90.0
reserve_pct = 20.0
cruise_power_w = 180.0

At 80% of a 90 Wh pack the drone starts with 72.0 Wh, and the 20% reserve is 18.0 Wh, which is exactly what the mission summaries above report. The site, config/sites/demo_farm.toml, defines home at [0.0, 0.0], a geofence with a 5 m inset (geofence_margin_m = 5.0), keep-out zones like Substation and Beehives with a 10 m buffer each, and named places such as North Fence, Perimeter, Barn and Pond. Bad geometry or unknown names fail at startup.

The validator: structure, space, energy

The validator in src/robot_copilot/validator/ is a set of pure functions over a typed plan. It never calls a model and never touches the database. It runs in layers: structure (schema, allowed actions for the vehicle, altitude, speed and hover limits, at most 40 actions), resolution (turn place names into coordinates), spatial checks, then energy.

The spatial checks use shapely. My favourite is the return to home check, because it catches a plan that is fine on the way out but would cut through forbidden airspace on the way back. From src/robot_copilot/validator/spatial.py:

src/robot_copilot/validator/spatial.py:

def rth_segment_blocked(
    geometry: SiteGeometry, x: float, y: float, home: tuple[float, float]
) -> bool:
    hx, hy = home
    if x == hx and y == hy:
        return False
    line = LineString([(x, y), (hx, hy)])
    if not geometry.fence_inset.covers(line):
        return True
    return any(poly.intersects(line) for _, poly in geometry.keep_outs)

That is why "fly over the substation" reports two problems: the outbound leg enters the keep-out, and the straight line home from there is blocked too. The keep-out check itself, segment_keepouts_hit(), returns every zone a leg touches, and check_spatial_and_energy() in validator/resolve.py reports each zone once per plan:

src/robot_copilot/validator/resolve.py:

        for ko in segment_keepouts_hit(ctx.geometry, leg.start, leg.end):
            if ko in reported_keep_outs:
                continue
            reported_keep_outs.add(ko)

An earlier version stopped at the first zone a leg hit, and could list the same zone again for a later leg. I had literally seen enters_keep_out twice for the substation. Now a leg that clips two zones names both, and the substation shows up once.

Energy works the same way. rth_legs() in src/robot_copilot/validator/energy.py models the trip home as a climb to the return altitude, a straight move, and a descent, and the energy check adds that return cost on top of the mission itself. A patrol can look affordable until you remember the drone still has to get home with its reserve intact. For four perimeter loops at 40%, the mock run reported a start of 36.0 Wh and a projected end of -10.6 Wh against the 18.0 Wh reserve.

Every issue has a stable code from issues.py, a step index where it applies, and a plain English explanation, so the UI, the repair loop and the tests all speak the same language.

Repair: the model gets the issues, the validator gets the last word

When a plan is invalid, you can ask for a repair. Orchestrator.repair() sends the provider the previous plan and the list of issues, then validates whatever comes back from scratch. max_repairs = 2 in config/app.toml caps the attempts.

The mock's repair logic is honest about being rules, not intelligence. For altitude it clamps to the limit:

src/robot_copilot/llm/mock.py:

                        a["altitude_m"] = 120.0 if code == IssueCode.ALTITUDE_HIGH.value else 10.0

For battery problems it halves the number of patrol loops on each attempt, and says so:

src/robot_copilot/llm/mock.py:

                loops = _patrol_loops(normalize_name(request.user_message))
                if loops is not None:
                    new_loops = max(1, loops // (2**attempt))
                    actions = _patrol_wrapped(new_loops, request.vehicle)
                    if new_loops < loops:
                        plan_dict["summary"] = f"Patrol perimeter ({new_loops} of {loops} loops)"
                        adjustment_notice = (
                            f"Repair shortened the patrol from {loops} loops to {new_loops}. "
                            "Review the updated plan before you confirm."
                        )

I watched this play out on the 40% perimeter case. Repair 1 cut four loops to two and was still invalid, with a projected end of 9.9 Wh. Repair 2 cut it to one loop and passed: "2.08 km, 4 min 51 s, uses 15.9 Wh of 36.0 Wh; ends with 20.1 Wh (reserve 18.0 Wh)." A third repair request was refused with not_repairable, because the mission was already valid. The plan that passed is one loop, not the four I asked for. My first version did that silently, and the plan summary still just said Perimeter. That bothered me, so the repair now renames the plan Patrol perimeter (1 of 4 loops) and puts Repair shortened the patrol from 4 loops to 1. Review the updated plan before you confirm. above the plan, as in the screenshot. A repaired plan is not necessarily the plan you asked for, and a human still has to confirm it. The pond case repaired in one step to 120 m, and the rover case repaired by dropping the takeoff and altitude.

Confirmation is bound to a hash

A valid plan gets a SHA-256 hash that covers the plan, the site, the vehicle profile, the battery level and the validator version, in src/robot_copilot/validator/__init__.py:

src/robot_copilot/validator/__init__.py:

def plan_hash(plan: MissionPlan, ctx: ValidationContext) -> str:
    payload = "|".join(
        [
            canonical_json(plan.model_dump(mode="json")),
            ctx.site.fingerprint(),
            canonical_json(ctx.vehicle.model_dump(mode="json")),
            str(ctx.battery_pct),
            VALIDATOR_VERSION,
        ]
    )
    return hashlib.sha256(payload.encode()).hexdigest()

To confirm, the browser has to send that hash back. Orchestrator._confirm() in src/robot_copilot/orchestrator.py rejects a mismatch, and also rejects a confirmation that arrives after the time to live (confirm_ttl_s = 600), then validates the stored plan again before simulating:

src/robot_copilot/orchestrator.py:

        if m.plan_hash != phash:
            self._store.audit(mission_id, "confirm_rejected", {"reason": "hash"})
            raise ConflictError(ConflictCode.PLAN_HASH_MISMATCH, "Plan hash does not match.")

When I posted a wrong hash, I got HTTP 409 with plan_hash_mismatch. Confirming an already executed mission gave 409 not_confirmable. State changes go through compare and swap updates in the store plus a per conversation lock, so two tabs cannot confirm the same mission twice.

The simulator only replays validated plans

simulate() in src/robot_copilot/sim/simulator.py refuses anything that is not a validated, matching report:

src/robot_copilot/sim/simulator.py:

    if not report.ok or report.resolved is None or report.plan_hash is None:
        raise NotValidatedError("report not validated")
    resolved = report.resolved
    if report.plan_hash != plan_hash(resolved.plan, ctx):
        raise NotValidatedError("plan hash mismatch")

It then steps through the resolved legs at 0.5 s per frame (doubling the step if a mission would need more than 4,000 frames), checks safety invariants on every frame, and returns executed or aborted. For the north fence plan at 80%, my run produced 364 frames, 3 photos, and a final state landed at home with 61.69 Wh left. That is kinematics and battery bookkeeping, not physics: no wind, no GPS error, no sensor noise. The README calls the return home path check a demo heuristic, not a safety guarantee.

Tests and the boring hardening

uv run --no-sync pytest -q at this commit gave me 164 passed and 3 deselected (the browser end to end tests), with no warnings. There are unit, property, scenario, provider and API tests, and a socket guard so nothing reaches the network after uv sync. The web app only accepts JSON POSTs up to 16 KiB, only answers to localhost and 127.0.0.1, and Docker publishes on 127.0.0.1 only. Real providers (Anthropic, OpenAI, Gemini and xAI) are optional and need your own keys; the mock is what the tests and this post use.

Where to go next

If you want something closer to real autopilot code, my Rust drone failsafe tutorial builds a no_std failsafe core with its own simulator. For a robot that takes voice commands inside a full simulator, see the ROS 2 Nav2 voice command robot, and for many robots sharing space, the multi-robot warehouse planner.

The lesson I keep coming back to: let the language model be creative where mistakes are cheap, and put every rule that matters into code you can test. Clone it, edit the farm, and try to sneak a plan past the validator.