A ROS 2 Nav2 robot that follows voice commands in Gazebo

The useful version of a voice robot is boring in the best way. You say "go to the kitchen" and the robot drives there. You say "stop" and the current goal is cancelled. The ROS 2 voice-command robot is that loop on ROS 2 Humble: a TurtleBot3 in Gazebo, Nav2 from nav2_bringup, and a small speech stack that can run with no internet.

The speech side follows the design of the offline voice assistant. Every model has a mock, so continuous integration needs no weights, microphone, or GPU. CI builds the package in the ros:humble-ros-base image and tests the logic with mocks. It does not run Gazebo, and the repo does not claim navigation performance numbers. The place coordinates in config/places.yaml were picked from the turtlebot3_world map for the demo. Check them in RViz before you rely on them.

How a sentence becomes a goal

A microphone or a WAV file goes through energy VAD, which cuts utterances with an RMS threshold, hangover, a minimum speech length, and padding. Speech to text is ScriptedSTT by default, which is what CI uses, or WhisperSTT when you pass stt:=whisper and have faster-whisper installed.

The transcript is published on /voice/transcript. voice_command_node turns it into an intent. A keyboard node can publish the same kind of text if you would rather type. The core package has no ROS imports, so the grammar, planner, and session can be tested with pytest alone.

nav_commander_node subscribes to /voice/goal as a PoseStamped and to /voice/cancel as an Empty. It forwards goals to Nav2's navigate_to_pose action (NavigateToPose) and publishes the result on /voice/goal_result. The voice node also listens to /amcl_pose so a relative move can start from the current pose, and it publishes a spoken or logged line on /voice/reply.

The grammar, then the model

RuleIntentParser runs first. It lowercases the text, strips a few filler words (please, robot, hey, okay, can you, could you, now), and matches a small set of patterns. Kinds are navigate, move, rotate, stop, patrol, and unknown.

These phrases are handled by the rules, so they never call a model:

• stop, halt, freeze, cancel, abort, and wait become stop.
• "go home", "come back", "return to the dock", and the charger or base variants navigate to the place named home.
• "patrol the lab and the hallway" splits on commas, the word "and", and the word "then".
• "move forward 2 meters" and "drive 2 meters forward" are relative moves. "Backward" and "back" store a negative distance.
• "turn left", "rotate right 45 degrees", and "spin around" are rotations. Left is positive degrees, right is negative, and "around" is 180. A turn with no number defaults to 90 degrees.
• "go to the kitchen" and "take me to the office" look up a named place.

If the rules return unknown, and you launched with intent_backend:=llm, LocalLLMIntentParser asks a local OpenAI-compatible server. That can be llama.cpp's server, Ollama, or vLLM. The default base URL in config/voice.yaml is http://127.0.0.1:8080/v1 and the model name is local. The request uses temperature 0 and response_format set to a JSON object.

The system prompt asks for one JSON object with kind, place, distance_m, angle_deg, and places. It lists the known places and tells the model not to invent any. The parser then validates the object. A bad kind, a navigate goal whose place is not in the map, a patrol with fewer than two known places, a move without a number, or a rotate without an angle all fall back to the rule result. Any HTTP or JSON failure does the same. The rules are the safety net, not a hint the model is free to ignore.

Places and safety limits

config/places.yaml names five spots in the map frame:

GoalPlanner turns an intent plus the current pose into Nav2 goals. The defaults, also set on the voice node as max_move_m: 5.0, are a maximum relative move of 5 meters and a maximum patrol of 6 stops. A relative move of 0, or a move whose absolute distance is over that limit, raises Relative moves must be between 0 and 5 m. A patrol with one place raises A patrol needs at least two places. A longer patrol raises the stop limit. An unknown place uses difflib.get_close_matches with a cutoff of 0.6 and answers with Did you mean ...? or the list of known places.

Relative moves are applied along the current heading: the new pose is the current x and y plus the distance times the cosine and sine of yaw. Rotations keep x and y and add the angle to yaw. stop plans an empty list, and the session cancels the active goal.

CommandSession keeps one goal at a time. A patrol is a queue. A new command preempts the queue. "Stop" clears it.

Launch the simulation

You need ROS 2 Humble with nav2_bringup and turtlebot3_gazebo installed. The Gazebo model path has to include the TurtleBot3 models or the world will not spawn them.

mkdir -p ~/voice_ws/src && cd ~/voice_ws/src
git clone https://github.com/applycoding/ros2-voice-command-robot
cd .. && colcon build --symlink-install && source install/setup.bash
export TURTLEBOT3_MODEL=waffle
export GAZEBO_MODEL_PATH=$GAZEBO_MODEL_PATH:/opt/ros/humble/share/turtlebot3_gazebo/models
 
ros2 launch voice_command_robot voice_nav_sim.launch.py headless:=False

Set the initial pose in RViz. Nav2 will not navigate cleanly until AMCL has a pose. In another terminal, after sourcing the same workspace:

ros2 run voice_command_robot keyboard

The prompt is you>. Try commands the grammar already understands:

go to the kitchen
patrol the lab and the hallway
stop
move forward 2 meters
turn left

voice_only.launch.py starts the voice nodes without the simulator when you want to test transcripts and replies on their own.

There is a second way in, for a recorded command instead of the keyboard. This path needs faster-whisper:

ros2 run voice_command_robot mic_listener --ros-args -p stt:=whisper -p wav_file:=command.wav

For free-form sentences the rules do not cover, start a local server and flip the intent backend. One example from the README is llama.cpp's server:

llama-server -m qwen2.5-1.5b-instruct-q4_k_m.gguf --port 8080
ros2 launch voice_command_robot voice_nav_sim.launch.py intent_backend:=llm

Leave the backend on rules until that server is actually up. A failed model call falls back to the rules, which means an unknown sentence stays unknown. You will see the reply from the planner (Sorry, I did not understand that command.) rather than a guessed room.

config/voice.yaml also lets you point places_file at another YAML. An empty string uses the installed config/places.yaml. global_frame is map and robot_frame is base_footprint on the nav commander.

What to test before you trust the sim

The Python tests do not need ROS topics or Gazebo. Install the dev requirements and run pytest from the repo:

pip install -r requirements-dev.txt
pytest

They cover the grammar, LLM output validation and fallback with a fake HTTP client, the planner's limits and geometry, the session state machine, VAD on synthetic audio, an audio-to-goals pipeline run with the scripted STT, and the launch and package files. CI also runs colcon build and imports the nodes inside the Humble base image.

If a typed command does the wrong thing, check the intent before you blame Nav2. A place the grammar does not know never becomes a goal. A move longer than 5 meters is rejected before NavigateToPose is called. A patrol is a list of poses, not a single average point. And the coordinates above are demo picks from the map image. If the kitchen marker sits in a wall in your RViz view, edit places.yaml and rebuild or rely on the symlink install. The package will not invent a better coordinate for you.