HFlow: Standardizing Robotics Data Pipelines

HFlow is an open-source SDK designed to streamline data pipelines for robotics development, standardizing and quality-checking multimodal recordings from robots and human operators.

Developed by Brandon and Kingston, founders of Hebbian Robotics, HFlow targets a significant bottleneck in embodied AI and robotics: the often chaotic, script-driven processing of vast sensor data corpuses.

The SDK processes synchronized streams of video, joint states, actions, timestamps, and metadata, ensuring data integrity crucial for training robust robotics models and maintaining operational reliability.

HFlow pipelines are defined as plain Python functions for transformations, checks, labels, and enrichments. These functions can run in-process during development or be packaged as Airflow 3 DAGs for scheduled, large-scale processing.

It leverages MCAP, an open container format, to manage multimodal recordings, ensuring compatibility and synchronized data streams. The output is a canonical MCAP file, complete with provenance tracking detailing how the data was produced.

Quality checks focus on generating reusable evidence for common issues such as black frames, frozen video, or timestamp drift. These measurements are stored in an append-only Parquet catalog, which can be queried using DuckDB SQL to assemble version-pinned manifests.

HFlow does not aim to replace existing tools but integrates with them — connecting MCAP for recordings, Airflow for execution, Parquet for catalog data, and DuckDB for curation. It adds contracts specific to robotics episodes, processing provenance, and quality evidence management.

"processing robotics data is itself one of the bottlenecks to improving robotics models." This observation by the creators highlights the tool's core mission to professionalize a critical, often ad-hoc, stage of robotics development. HFlow is free under the Apache-2.0 license, with future plans for managed workspaces and enterprise support.