Intelligent Bin Picking in Simulink using Gemini Robotics ER

• Overview
• Setup
• Getting Started
• Model Details
• Limitations
• References
• License
• Feedback
• Community Support

Overview

Gemini Robotics ER is a vision-language model with embodied reasoning (ER) capabilities: spatial understanding, object recognition, and scene interpretation grounded in physical context. Unlike general-purpose vision models, Gemini Robotics ER can interpret a camera image in terms of robot-actionable outputs: object locations, pick sequences, and placement targets derived from a natural-language instruction. This eliminates the need for custom-trained object detectors or hand-crafted pick rules (traditionally weeks of data collection and model training) and lets you set up new task plans in minutes by writing a natural-language prompt instead of programming scheduler logic.

This example uses Gemini Robotics ER for two stages of a pick-and-place loop:

• Task planning: The overhead camera image and a natural-language instruction (e.g. "Place all elbow fittings on the right table") are sent to Gemini. Embodied reasoning grounds the instruction spatially: Gemini identifies each relevant object by type, colour, and shape; localises it with a pixel bounding box; infers the intended destination from the workspace context (bin, left table, right table); and returns a pick priority order. The result is a structured JSON plan (pick boxes, drop boxes, pick order) produced without any pre-trained object detector or object template.
• Task verification: After the robot completes its actions, two images are sent to Gemini: the scene before the task started and the current scene. Embodied reasoning is used to compare object positions across both images and determine whether every object specified in the instruction has moved to its correct destination. Gemini returns a pass/fail judgement with a natural-language explanation (e.g. "Task unsuccessful. The blue tee fitting was not placed.").

The simulation environment is adapted from the Intelligent Bin Picking System in Simulink® example (Robotics System Toolbox™). The original example uses a Mask R-CNN object detector and a rule-based task scheduler; this project replaces the detector with Gemini Robotics ER and extends the scheduler to accept natural-language task instructions with flexible pick ordering and cross-bin placement.

Setup

MathWorks Products

Requires MATLAB® R2026a or newer.

• Robotics System Toolbox™
• Simulink®
• Stateflow®
• Simulink® 3D Animation™
• Optimization Toolbox™

Requires a host platform C compiler; see Supported Compilers.

3rd Party Products

• Python 3.13: MATLAB®-Python cosimulation for Gemini API calls
• google-genai: Gemini API client
• Pillow: image encoding
• numpy: array interchange between MATLAB and Python

All packages are installed automatically via installPythonEnv.

1. Install dependencies (run once after cloning)

In MATLAB®, from the repo root:

installPythonEnv                  % downloads Python 3.13 + packages
installIntelligentBinPicking      % fetches "Intelligent Bin Picking in Simulink" example

2. Start each MATLAB session

projectstartup                    % adds paths, configures Python environment
setenv('GEMINI_API_KEY', 'your-key-here')

Get a free API key from Google AI Studio.

Verify Setup

Verify the Python environment and Gemini API are reachable:

runtests("test/testPythonEnv.m")   % no API key required
runtests("test/testGeminiAPI.m")   % requires GEMINI_API_KEY; auto-skips if unset

Getting Started

Run projectstartup at the beginning of each MATLAB® session to configure paths and the Python environment:

projectstartup

Scene Reference

The overhead camera sees the bin with colored pipe fittings (elbows, crosses, tees, straights). Use this view as a reference when writing task prompts:

Objects are distinguishable by color (red, green, blue, cyan, magenta, black) and shape (elbow, cross, tee, straight). Prompts can reference either attribute, spatial position (leftmost, rightmost), or combinations.

Running a Scenario

Open the model, set a natural-language task prompt, and simulate:

modelName = "IntelligentBinPickingGemini";
open_system(modelName)

taskPrompt = "Place all elbow fittings on the right table. " + ...
             "Place all cross fittings on the left table.";
sim(modelName);

taskPrompt must be set in the MATLAB® base workspace before sim(). The UserPrompt StringConstant block reads this variable. Alternatively, double-click the block and edit the String parameter directly.

More scenarios

• Sort by position: leftmost → left table, rightmost → right table

taskPrompt = "Pick the leftmost fitting and place it on the left table. " + ...
             "Pick the rightmost fitting and place it on the right table.";
sim(modelName);

• Priority + exclusion: cyan crosses first → left table, then green tee and red tee → right table, black and pink fittings stay in bin

taskPrompt =  "Pick the cyan cross fitting and place them on the left table. " + ...
              "Then pick the green tee fitting and place it on the right table. " + ...
              "Then pick the blue cross fitting and place it on the right table. " + ...
              "Leave the remaining fittings in the bin.";
sim(modelName);

• Sort by color temperature: warm (red, magenta) → right table, cool (blue, cyan, green) → left table, black stays in bin

taskPrompt = "Place all warm-colored fittings (red, magenta) on the right table. " + ...
             "Place all cool-colored fittings (blue, cyan, green) on the left table. " + ...
             "Leave any black fittings in the bin.";
sim(modelName);

Randomizing Object Positions

Each simulation starts with a fixed object layout by default. To test robustness across different arrangements, randomize positions before running:

rng(42);           % any seed you like
robotSimParams;    % regenerates spawn positions with the new seed
sim(modelName);

The randomization shuffles the assignment of objects to the 8 predefined spawn locations in the bin and varies their orientations, producing a unique scene each time.

Using the Demo App

To run scenarios interactively, launch the demo app:

geminiDemoApp

The app lets you:

• Select a preset scenario from the dropdown or switch to Custom mode and type any natural-language task prompt
• Randomize Positions — shuffles object locations in the bin before each run, so you can test the same prompt against different arrangements
• Simulate / Stop — starts or halts the Simulink® simulation

Recording Video

To save a video of the simulation, uncomment the Simulation 3D Video Writer block in Simulink_3D_IBP_Target and set its Filename parameter to the desired output path (e.g. results/my_run.avi). Create the output directory first if needed: mkdir results.

Cleanup

When done, run projectshutdown to remove paths:

projectshutdown

Model Details

Gemini Robotics ER Block

geminiERBlock (GeminiRobotics.slx) is a MATLAB® System block that sends a camera image and a natural-language prompt to Gemini Robotics ER and returns a structured Simulink bus. Task-specific behaviour is selected through the mask's Mode parameter.

This example shows two modes. Both share the same block, differing only in the mode plugged into the Mode parameter and in the output bus.

GeminiERPlan produces a pick-and-place plan from an overhead camera image. Given an image and a task prompt (e.g. "Place all elbow fittings on the right table"), it returns pick bounding boxes, drop bounding boxes, and a pick order in pixel coordinates.

GeminiERVerify checks whether the robot completed its task. It anchors a before-image at simulation start and, when triggered at task end, sends the before/after pair to Gemini for comparison. Returns a pass flag and a short diagnostics string.

Extending the block

To adapt the block to a different workspace, subclass GeminiERBase.m and override the parts of the pipeline that change:

Extension point	Purpose
getRole()	Task-specific role sent as part of the system_instruction.
getFormatSchema()	JSON schema Gemini is asked to return.
preprocess()	Optional image transforms (crop, resize) applied before the API call.
postprocess()	Parse Gemini's JSON response into the output bus.
createSimulinkBus()	Default struct that defines the output bus fields and types.
updateImages()	Custom multi-image handling (e.g. anchoring a before-image).
firesOnFirstStep()	Whether a call fires automatically at simulation start.

When adding a new mode, also extend the bus-type switch in geminiERBlock.getOutputDataTypeImpl so Simulink knows which bus to expect at the output port.

For scene-only changes (same task shape, different objects or camera view), no code is needed. Update the Scene context mask parameter to describe the new workspace and re-run.

See GeminiERPlan.m and GeminiERVerify.m for two worked examples.

Other Blocks

The Pixel Plan to World block (pixelPlanToWorldBlock) back-projects the pixel-space pick and drop boxes from geminiPixelDetectionsBus to robot-base-frame XYZ coordinates using the camera intrinsics and a depth map, producing geminiTaskPlannerBus.

The Task Scheduler and CHOMP Trajectory Planner are carried over from the original IBP example with minimal changes: the Task Scheduler accepts a pick priority order and per-object drop positions from the Gemini planner, replacing the fixed rule-based logic. Refer to the Intelligent Bin Picking System in Simulink example for a full description of the scheduler state machine, trajectory planner, and Sim3D scene.

Limitations

• Gemini Robotics ER is currently in preview; APIs and capabilities are subject to change.
• Like other large language models, Gemini Robotics ER can hallucinate, producing incorrect detections or placement plans, especially for ambiguous prompts or unfamiliar object types.
• Vague or underspecified task instructions may produce inconsistent detections or incorrect placement assignments. Clear, specific prompts yield the most reliable results.
• Pixel-level bounding box errors become positional errors in world coordinates, which may cause missed grasps in tightly packed bins.
• API latency is typically 2-5 s per call (the first call in a session may take longer due to Python process warm-up). The planner fires once per task, not at each simulation step. Mid-task replanning on failure is not yet implemented.
• This example runs in a simulated environment with a virtual camera. Deploying on a real robot requires updating the camera intrinsics and pose parameters in pixelPlanToWorldBlock to match the physical camera setup.

References

• Gemini API: Gemini Robotics ER
• MathWorks: Intelligent Bin Picking System in Simulink

License

The license is available in the license.txt file in this GitHub repository.

Security

To report a security vulnerability, see SECURITY.md.

Feedback

Tried this example? We'd love your feedback!

Share your experience to help us prioritize what to improve next - takes less than 5 minutes.

Give feedback ➜ Feedback on Intelligent Bin Picking in Simulink using Gemini Robotics ER

Community Support

MATLAB Central

Copyright 2026 The MathWorks, Inc.