Skip to content

add observe skill to go2 #1109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
paul-nechifor merged 1 commit into from
Jan 25, 2026
Merged

add observe skill to go2 #1109

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion dimos/robot/unitree/connection/go2.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
from dimos import spec
from dimos.core import DimosCluster, In, LCMTransport, Module, Out, pSHMTransport, rpc
from dimos.core.global_config import GlobalConfig
from dimos.core.skill_module import SkillModule
from dimos.dashboard.rerun_init import connect_rerun
from dimos.msgs.geometry_msgs import (
PoseStamped,
Expand All @@ -35,6 +36,8 @@
)
from dimos.msgs.sensor_msgs import CameraInfo, Image, PointCloud2
from dimos.msgs.sensor_msgs.image_impls.AbstractImage import ImageFormat
from dimos.protocol.skill.skill import skill
from dimos.protocol.skill.type import Output
from dimos.robot.unitree.connection.connection import UnitreeWebRTCConnection
from dimos.utils.data import get_data
from dimos.utils.decorators.decorators import simple_mcache
Expand Down Expand Up @@ -141,7 +144,7 @@ def publish_request(self, topic: str, data: dict): # type: ignore[no-untyped-de
return {"status": "ok", "message": "Fake publish"}


class GO2Connection(Module, spec.Camera, spec.Pointcloud):
class GO2Connection(SkillModule, spec.Camera, spec.Pointcloud):
cmd_vel: In[Twist]
pointcloud: Out[PointCloud2]
odom: Out[PoseStamped]
Expand All @@ -153,6 +156,7 @@ class GO2Connection(Module, spec.Camera, spec.Pointcloud):
camera_info_static: CameraInfo = _camera_info_static()
_global_config: GlobalConfig
_camera_info_thread: Thread | None = None
_latest_video_frame: Image | None = None

@classmethod
def rerun_views(cls): # type: ignore[no-untyped-def]
Expand Down Expand Up @@ -212,6 +216,7 @@ def start(self) -> None:

def onimage(image: Image) -> None:
self.color_image.publish(image)
self._latest_video_frame = image
Copy link
Contributor

@leshy leshy Jan 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as long as we are familiar with this but choosing to do _latest_video_frame thing I'm ok. idk if this API is too obscuring

https://github.com/dimensionalOS/dimos/blob/main/docs/api/sensor_streams/advanced_streams.md#getting-values-synchronously

Copy link
Contributor Author

@paul-nechifor paul-nechifor Jan 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did consider that (getter_hot), (since I reviewed the doc), but it seemed easier to just set the value (like the other skill does) since there already is a subscription for the image here.

if self._global_config.viewer_backend.startswith("rerun"):
rr.log("world/robot/camera/rgb", image.to_rerun())

Expand Down Expand Up @@ -302,6 +307,15 @@ def publish_request(self, topic: str, data: dict[str, Any]) -> dict[Any, Any]:
"""
return self.connection.publish_request(topic, data)

@skill(output=Output.image)
def observe(self) -> Image | None:
"""Returns the latest video frame from the robot camera. Use this skill for any visual world queries.

This skill provides the current camera view for perception tasks.
Returns None if no frame has been captured yet.
"""
return self._latest_video_frame


go2_connection = GO2Connection.blueprint

Expand Down