Oculus quest integration

[leshy]

• Transforms published for left and right controller positions (does head have a position also?)
• Potentailly twist published per controller (maybe when holding a button?) if we want to drive robots
• Can get a video stream into the headset?
• Can we get a pointcloud into the headset?
• Generally Investigate data you can get from oculus that we might want to publish and use

[linear]

DIM-318

[ruthwikdasyam]

class QuestTeleop():
        color_image: In[Image]
        pointcloud: In[PointCloud2]
        left_pose: Out[PoseStamped]
        right_pose: Out[PoseStamped]

For the initial PR, would focus on getting poses of the controllers, have a working pipeline to move arms (xarm/piper)
Later, will get into Image and pointcloud streams.

I plan to have a web server and achieve the benchmark latency that satisfies every In[] and Out[]. A verified alternative would be having a Unity app - an apk running on the VR, managing the streams.

[leshy]

class QuestTeleop():
        color_image: In[Image]
        pointcloud: In[PointCloud2]
        left_pose: Out[PoseStamped]
        right_pose: Out[PoseStamped]

For the initial PR, would focus on getting poses of the controllers, have a working pipeline to move arms (xarm/piper) Later, will get into Image and pointcloud streams.

I plan to have a web server and achieve the benchmark latency that satisfies every In[] and Out[]. A verified alternative would be having a Unity app - an apk running on the VR, managing the streams.

for benchmarking you can talk to @Nabla7 about rerun for latency measurement

[leshy]

this is great, can we actually run unitree with pointcloud / color image inputs? pointcloud being a map?

[ruthwikdasyam]

this is great, can we actually run unitree with pointcloud / color image inputs? pointcloud being a map?

Not including [In] with the initial PR. Tested Video streaming in the browser before, should figure out best way to display pointclouds

[ruthwikdasyam]

Aaded QuestTeleop Module - we can call VRTeleop, as this is expected to work with any VR headset. Considering the format - to have a base_teleop class, and QuestTeleop based on it - curious what other TeleopModule's would we add/make from the base_teleop Module.

More Context:
Support for all headsets is provided in this module. What other form of hardware would we use to perform teleop?
(Should design base class depending on this)

Update

BaseTeleopModule is added: changes/updates can be done as needed, when building for more teleop devices

[ruthwikdasyam]

PR#1 - #962

Implemented:

• Websocket connection with VR
• Passthrough view
• 2x[PoseStamped], 2x[Float32] at 90Hz
• Generalized TeleopRobotController
• Button Control for Start and Stop

Next set of features:

• Implement teleop with xarm (Modification to TeleopRobotController)
• Implement an IK method for Teleop (Prioritizes rate, cartesian planning ability)
• Has working controller_pos readings, Quest also detecting hands without any controllers in hand - checking if we can switch between hands and controllers, and do teleop with both.
• Lay block/frame through the headset when doing the teleoperation - visual feedback on what is being read
• [In] to Quest3TeleopModule -either [PointCloud2] or [Image]
• Investigate the headset head position

Ambitious set of features:

• Multiple views - Image, Pointcloud2

[Feel free to add to the list]