Skip to content

ROADMAP.md

Latest commit

 

History

History
128 lines (99 loc) · 10.6 KB

ROADMAP.md

File metadata and controls

128 lines (99 loc) · 10.6 KB
Raw

Robokin Roadmap

Personal / open-source roadmap for robokin. Distilled from collected inspiration notes and a handwritten "ROBOKIN Growth" note (2026-05-15).

robokin's identity: one library to control, teleoperate, and plan motion for robot arms — pluggable kinematics backends (Placo / PyRoki / RoboPlan / …) at the base, with controllers, motion planning, and safety layered on top for manipulation.

Status legend: 💡 idea · 🔬 needs research · 🏗️ ready to build

Theme 1 — Kinematics & solver backends (core)

robokin's core is one interface that can pick any backend (Placo / PyRoki / RoboPlan, + frax, + maybe EXOTica), with trajectory planning (Theme 4) and safety/collision (Theme 5) layered on top, using whatever each backend already provides. Caveat: every backend is real test and maintenance cost — keep 1–2 primary, the rest optional/experimental.

  • K1 · Integrate frax in FK + IK 🏗️ Add StanfordASL/frax (also danielpmorton/frax) as a backend behind the same interface.
  • K2 · Evaluate EXOTica for real-time IK 🔬 Benchmark EXOTica's ~1–5 ms IK against Placo/PyRoki for a teleop loop — adopt only if it wins. EXOTica returns joint positions only, so compute velocities via finite differences before sending. (OMPL is offline path planning only.)
  • K3 · Real-time control-loop & 3D-camera patterns 🔬 Study uynitsuj/robots_realtime: show a 3D stereo camera in Viser and use it to drive the arm; fold useful real-time servo patterns into robokin's executor. Camera to try: IMX219-83 stereo.

Theme 2 — Controllers & control interface

One common "controller / pose-source" interface, many input devices, switchable at runtime.

  • C0 · Easy controller interface + controller picker 🏗️ Ergonomic API (move-to-pose, named poses, gripper) that hides solver/backend details and lets you switch controllers (keyboard, gizmo, joystick, …) at runtime. The base every controller plugs into. In progress: Interactive SO-101 IK in ROS 2 with Viser + robokin
  • C1 · Keyboard IK 🏗️
  • C2 · Gizmo IK 🏗️ — drag a Viser gizmo → EE target; make the gizmo easier to move around. Ref: robokin PR #2
  • C3 · Gamepad / joystick IK 🏗️
  • C4 · MediaPipe / hand IK 🔬→🏗️ Map markerless hand pose → EE target via IK. Survey existing MediaPipe interfaces and make them look good. (three.js arm control = visual reference only, not adopted.)
  • C5 · EMG / nerve-bracelet (Myo) controller 💡 Experiment with an old Myo armband (have the hardware) as an input device.
  • C6 · Hand-control research 🔬 arxiv 2603.11383v1 (likely best paper), plastic_gear, review set, IMU-sensor idea.
  • C7 · VR controller 💡 — parked (no hardware yet). Ref: discourse VR idea

Theme 3 — ROS 2 interfaces

  • N1 · Official ROS 2 API for robokin 🏗️ Promote cartesian_motion_node.py (legalaspro/so101-ros-physical-ai) into a clean, official robokin ROS 2 interface (topics / services / actions).

Theme 4 — Motion planning & sequencing

Higher-level layers on top of today's single-segment MotionPlanner (cartesian / joint-quintic) and single-trajectory TrajectoryExecutor. Most algorithms already live in the backends — this is mostly surfacing and orchestrating them.

  • M1 · Higher-level trajectory-planning interface 🔬→🏗️ One robokin API for multi-waypoint, blended, collision-aware planning, tapping pyroboplan (RRT / RRT-Connect, trajectory generation, optimization, collision checks), Placo (splines / jerk-limited), and PyRoki (trajectory optimization). Today only single-segment cartesian + minimum-jerk quintic exist.
  • M2 · Motion sequencer / arm "programs" 🔬→🏗️ Chain moves + gripper + waits/conditions into a runnable sequence to program the arm. Needs the executor to gain a trajectory queue and blending between segments (it currently holds one trajectory).

Theme 5 — Safety & robustness

  • S1 · Trajectory safeguards 🔬→🏗️ Filter/smooth sudden or jerky inputs during execution so the arm stays stable; clamp dangerous motions.
  • S2 · Safe physical human-robot interaction 🔬 Interact with the robot mid-task without it stopping. Exploratory.
  • S3 · Failsafe / watchdog for remote links 🏗️ On lag, drop, or loss of the remote connection, the arm must stop or hold safely (deadman + watchdog on the robot side). Non-negotiable for anything in Theme 6.

Theme 6 — Remote teleop platform (Viser + ROS 2 + Zenoh)

Biggest standalone build. Kept in this repo for now; may spin into its own package later.

  • R1 · Easy remote robot connection 🏗️ Make connecting to a remote arm simple and secure.
  • R2 · Reuse the Viser interface for any robot (URDF-driven) 🔬→🏗️ Viser is just the UI/visualization; the server hosting Viser loads the robot's URDF and runs IK/FK, so the same UI drives different arms. Simplest target: open the app locally (e.g. on a Mac) and control a connected arm. (Remote serving via nginx/docker is a later option, not required for the local case.)
  • R3 · Connecting the arm to the server — two adapters 🔬 In both cases the real-time loop stays local to the robot; the local side connects up to the server (not remote-CAN — the bus stays local).
    1. Preferred — via the ROS 2 interface (N1): if the arm already speaks ROS 2, integrating controllers is easy.
    2. Local arm CAN/hardware library: a driver on the robot host owns the CAN/hardware loop (e.g. SO101 / Feetech), then connects up to the server. Transport: Zenoh + TCP over a secure network (Zenoh handles NAT / remote links).
  • R4 · Multi-robot registry & remote-control service 💡 Register a robot together with its URDF — enough to drive R2's UI, IK/FK, and command generation. A user picks a robot from the registry and controls it remotely via a connect API (Zenoh or similar). Two-way comms: stream arm state back (latest joint positions) so the model and recorded data stay in sync with the real arm.

Theme 7 — Perception, camera & 3D

  • P1 · Open detection / segmentation models 🔬 Integrate open models (SAM, RF-DETR, …) for live-video segmentation/detection — goal: match jaiselsingh's demo. Also skalskip92, mirrash7, roboflow, climb-mocap.
  • P2 · Object → 3D in Viser 🔬 Combine a 3D camera + Pixel3D-style generation to turn scene objects into 3D and view them in Viser. Ref: WangZhao_0849.
  • P3 · 3D depth-camera support in Rerun 🔬→🏗️ Stream and visualize 3D / depth-camera data (e.g. the IMX219-83 stereo cam) in Rerun, alongside the existing Rerun bridges.
  • P4 · Claude / agent integration on top of perception 💡 A Claude Code–style agent layer that uses perception outputs (detections, segments) to drive higher-level behavior.
  • P5 · Egocentric hand-pose estimation 🔬 Estimate hand pose from an egocentric (first-person / wearable) camera to drive hand-based teleop (feeds C4) and make demonstration capture more seamless. MediaPipe is the natural starting point, but other models/methods are open too.

Theme 8 — Performance & latency

  • L1 · Latency reduction across the control/comms path 🔬 Profile end-to-end latency (UI → server → transport → local driver → motors) and the budget per link (LAN <5 ms vs internet 50–200 ms). Decides what direct teleop vs waypoint control is safe (ties to R3).
  • L2 · Evaluate Rust for comms + local drivers 🔬 — conditional on L1 Only if L1 shows Python is the hot-path bottleneck, consider Rust for the transport layer and the local CAN/hardware driver. Pairs naturally with Zenoh (Rust-native) and ROS 2 via rclrs; keep the high-level / planning side in Python. Note: Zenoh is already Rust, Placo is C++, PyRoki is JAX — the latency-critical bits may already be native.

Theme 9 — Multi-arm, hands & hardware scaling

  • X1 · Run robokin on different arms 🔬 Validate beyond SO101 — drives generality of the core API. Near-term (next weeks): test on the Barcelona arms (Piper, Universal Robots). They're already on ROS 2, so this doubles as the first real test of the ROS 2 interface (N1) and the URDF-driven Viser UI (R2).
  • X2 · Dual-arm SO101 (bimanual) 🔬→🏗️ Build a second SO101 and test coordinated two-arm control through robokin (shared IK/planning across both arms).
  • X3 · Dexterous hand — AmazingHand 🔬 Build the open-source Pollen Robotics AmazingHand, mount it on the arm, and test dexterous manipulation (DEX).

Hardware

  • Have: SO101 arm · Myo armband (defunct startup → C5).
  • To try / build: 2nd SO101 (X2) · IMX219-83 stereo camera (K3) · AmazingHand build (X3).

Suggested issue order

  1. K1 (frax) — small, fits the existing API → good first issue.
  2. C0 (controller interface) — base layer for all controllers.
  3. C1 / C2 / C3 (keyboard / gizmo / gamepad IK) — quick, visible wins on top of C0.
  4. N1 (official ROS 2 API) — exposes robokin to the stack.
  5. S1 (safeguards) — needed before live teleop.
  6. M1 → M2 (surface trajectory planning, then build the sequencer on top).
  7. C4 (MediaPipe IK) — high-value demo.
  8. R1 → R2 → R3 → R4 (remote platform) — epic; S3 failsafe is a hard prerequisite.
  9. X2 (dual-arm SO101) — once single-arm control is solid.
  10. Research / idea: K2, K3, C5 (Myo), C6, C7 (VR, parked), P1, P2, P3, P4, P5, S2, X1, X3, L1, L2.