How I built a ROS 2 YOLO tracker with two-stage IoU matching

I built the ROS 2 vision tracking demo as a portfolio project: a simulation pipeline that reads a video, detects objects with YOLO, carries object IDs between frames, and publishes an annotated image. It runs on CPU without a camera, GPU, or robot.

The checked-in crossing scene makes the pipeline visible. It does not establish tracking accuracy or throughput. There is no benchmark results file in this repo, and I am not treating the GIF as an evaluation. The tracker is my own ByteTrack-style implementation, using a Kalman filter and two stages of IoU matching, rather than a fork of the reference implementation.

Four nodes, one frame header

I split the ROS work into four nodes. video_publisher publishes BGR frames on /image_raw. detector turns those frames into /detections. tracker publishes /tracks, and annotate combines tracks with images to produce /image_annotated for RViz or rqt_image_view.

Both detection topics use vision_msgs/Detection2DArray. On /detections, the detection id is empty. On /tracks, it contains the track ID as a string. The hypothesis class_id contains a class name such as person, rather than the numeric COCO index. Those conversions live in src/vision_tracking/vision_tracking/messages.py.

The detector carries the incoming image header forward. The tracker preserves that header too. The annotator uses ApproximateTimeSynchronizer with a queue of 10 and a configurable slop, defaulting to 0.3 seconds. That keeps image pairing explicit instead of drawing whichever tracks happened to arrive most recently.

I kept the detector, tracker, video reader, and drawing functions separate from their ROS wrappers. The GIF renderer imports the same core components, so I can inspect the visual path without installing Humble.

The code that joins detection to tracking

Keep the image callback small

In src/vision_tracking/vision_tracking/detector_node.py, the callback handles conversion and publication:

    def _on_image(self, message: Image) -> None:
        frame = self._bridge.imgmsg_to_cv2(message, desired_encoding='bgr8')
        detections = self._detector.detect(frame)
        self._publisher.publish(detections_to_msg(detections, message.header))
        self.get_logger().info(
            f'{len(detections)} detections',
            throttle_duration_sec=2.0,
        )

The backend owns inference. The default is OpenCV DNN with YOLOv4-tiny, using a matching .cfg and .weights pair. The loader also accepts ONNX when the installed OpenCV can import it. The demo's OpenCV backend explicitly rejects GPU device requests.

Preprocessing letterboxes the image into a square canvas, preserving aspect ratio. Postprocessing removes padding, maps boxes back to image pixels, clips them, and suppresses overlapping candidates. I kept that math in detectors/yolo_post.py so synthetic tensors can exercise it without model weights.

The optional Ultralytics backend accepts a checkpoint and a device setting. Switching backends does not replace my tracker with Ultralytics tracking. Both backends return the same internal detection records.

Divide scores before matching

In src/vision_tracking/vision_tracking/tracker.py, I separate strong detections from weaker ones:

        high = [item for item in detections if item.score >= self._high_thresh]
        low = [
            item for item in detections
            if self._low_thresh <= item.score < self._high_thresh
        ]

Before this split, every existing track predicts its next box. KalmanBox stores center coordinates, width, height, and a velocity for each of those four values. The update step corrects that state from a detected box. Width and height are clamped to at least one pixel.

The tracker node derives elapsed time from message stamps, falling back to its configured frame rate on the first message or a non-increasing timestamp. The core tracker clamps that step between 0.001 and 1.0 seconds. This is a simple image-plane motion model, with no camera calibration or world coordinates.

High-score detections are matched first. assignment.py builds a cost matrix from one minus IoU, penalizes pairs below the IoU threshold, and solves the assignment with its own Hungarian implementation. Rejected pairs stay unmatched.

Give weaker detections a second chance

The second association in src/vision_tracking/vision_tracking/tracker.py only considers tracks missed by the first:

        remaining = [self._tracks[index] for index in unmatched]
        low_matches, still_open, _unused = associate_boxes(
            [track.box() for track in remaining],
            [item.bbox for item in low],
            self._iou_threshold,
        )

Matched tracks get corrected from those weaker boxes. Tracks still unmatched accumulate a missed-frame count. Only unmatched high-score detections create new tracks, so a weak isolated detection cannot start an ID.

One configuration detail matters here. src/vision_tracking/config/tracking.yaml sets detector confidence to 0.35, tracker high threshold to 0.5, and tracker low threshold to 0.1. With these defaults, detections below 0.35 never reach the tracker. Its usable low-score interval therefore starts at 0.35. Lowering only the tracker threshold will not recover boxes already filtered out by the detector.

Separate keeping a track from drawing it

The visibility rule in src/vision_tracking/vision_tracking/tracker.py is short:

    def _visible(self, track: _Track) -> bool:
        if track.hits < self._min_hits:
            return False
        return track.time_since_update <= self._coast_frames

The default min_hits is 1 and coast_frames is 1. A track can appear immediately and remain visible for one missed update. Separately, max_age defaults to 30, and tracks remain stored while their missed count is at most that value. These are update counts, not seconds.

That distinction lets a temporarily hidden track recover its ID without drawing its predicted box throughout the whole retention period. Increasing retention does not guarantee recovery: the returning detection still needs enough overlap with the prediction.

Run the CPU demo

From the repository root, the Docker commands in README.md are:

docker build -t vision-tracking-demo .
docker run --rm --network host vision-tracking-demo

The Dockerfile starts from ROS 2 Humble on Jammy, installs dependencies, downloads YOLOv4-tiny assets, and builds the Python package with colcon. Weights are downloaded during the build; they are not committed. The default container command launches the four nodes against media/sample_crossing.mp4 with RViz disabled.

In another terminal on the same machine, with Humble installed, README.md gives these inspection commands:

source /opt/ros/humble/setup.bash
ros2 topic hz /image_annotated
ros2 topic echo /tracks vision_msgs/msg/Detection2DArray

I would check that annotated images arrive and that track messages contain IDs before adding a viewer. The topic frequency is something to measure on the machine running the demo, not a published performance number.

For a display, the README includes the container command with X11 forwarding and rviz:=true. It also documents a native Ubuntu 22.04 build. The video path can point to /dev/video0; device sources do not loop at end of input.

For a visual check without ROS, follow the README's GIF instructions using scripts/render_demo.py. That path needs NumPy, OpenCV, ffmpeg, and the downloaded assets. The renderer samples frames and passes the resulting elapsed time to the tracker. Its output frame rate is a rendering setting, not an inference benchmark.

Test the behavior before judging the clip

The core test command from README.md is:

cd src/vision_tracking
python3 -m pytest

The suite checks ID continuity for simple parallel motion, weak detections preserving an existing ID, weak detections failing to create one, gap recovery, expiration, and minimum-hit visibility. Assignment tests compare small rectangular matrices against brute-force solutions. Decode tests use synthetic network outputs to check coordinates, letterboxing, suppression, and class filtering.

Video tests write a tiny clip and check looping. Drawing tests verify that annotation changes pixels without changing the input image. The optional detector forward-pass test needs VISION_TRACKING_WEIGHTS and VISION_TRACKING_CFG; otherwise it skips. It checks a blank image, not tracking quality.

.github/workflows/ci.yml builds and tests inside ros:humble-ros-base-jammy. It runs colcon plus the copyright, flake8, and docstring checks. ROS message round-trips and node imports are covered there. Import checks do not establish that the complete streaming pipeline works under load.

Limits and what I would add next

Association uses geometry alone. It does not gate matches by class or compare appearance embeddings, so nearby overlapping objects can exchange IDs. The OpenCV suppression loop is also class-agnostic. Long occlusions, rapid motion, or poor detections need more evidence than this sample provides.

My next step would be an annotated evaluation clip with recorded settings, ID-switch counts, and timing measurements. I would also test class-aware association and synchronization behavior under delayed inference. Until those runs exist, the claim stays narrow: this repo demonstrates a readable ROS 2 detection and tracking pipeline.

Read and run the implementation

The repository contains the launch file, parameters, core tests, renderer, and Docker setup. I would start with CPU playback, inspect the two output topics, then change one threshold at a time while watching what happens to track creation and recovery.