Model Gemini Robotics ER dapat menunjuk objek, melacaknya dalam video, mendeteksinya dengan kotak pembatas, dan membuat lintasan pergerakan.
Untuk kode yang dapat dijalankan sepenuhnya, lihat Cookbook Robotics.
Menunjuk objek
Contoh berikut menemukan objek tertentu dalam gambar dan menampilkan koordinat [y, x] yang dinormalisasi:
Python
from google import genai
PROMPT = """
Point to no more than 10 items in the image. The label returned
should be an identifying name for the object detected.
The answer should follow the json format: [{"point": <point>,
"label": <label1>}, ...]. The points are in [y, x] format
normalized to 0-1000.
"""
client = genai.Client()
uploaded_file = client.files.upload(file="my-image.png")
image_response = client.interactions.create(
model="gemini-robotics-er-2-preview",
input=[
{
"type": "image",
"uri": uploaded_file.uri,
"mime_type": uploaded_file.mime_type
},
{"type": "text", "text": PROMPT}
],
generation_config={"thinking_level": "high"},
)
print(image_response.output_text)
REST
# First, ensure you have the image file locally.
# Encode the image to base64
IMAGE_BASE64=$(base64 -w 0 my-image.png)
curl -X POST \
"https://generativelanguage.googleapis.com/v1beta/interactions" \
-H "x-goog-api-key: $GEMINI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gemini-robotics-er-2-preview",
"input": {
"parts": [
{
"inlineData": {
"mimeType": "image/png",
"data": "'"${IMAGE_BASE64}"'"
}
},
{
"text": "Point to no more than 10 items in the image. The label returned should be an identifying name for the object detected. The answer should follow the json format: [{\"point\": [y, x], \"label\": <label1>}, ...]. The points are in [y, x] format normalized to 0-1000."
}
]
},
"generation_config": {
"thinking_config": {
"thinking_level": "high"
}
}
}'
Outputnya akan berupa array JSON yang berisi objek, masing-masing dengan point (koordinat [y, x] yang dinormalisasi) dan label yang mengidentifikasi objek.
JSON
[
{"point": [376, 508], "label": "small banana"},
{"point": [287, 609], "label": "larger banana"},
{"point": [223, 303], "label": "pink starfruit"},
{"point": [435, 172], "label": "paper bag"},
{"point": [270, 786], "label": "green plastic bowl"},
{"point": [488, 775], "label": "metal measuring cup"},
{"point": [673, 580], "label": "dark blue bowl"},
{"point": [471, 353], "label": "light blue bowl"},
{"point": [492, 497], "label": "bread"},
{"point": [525, 429], "label": "lime"}
]
Gambar berikut adalah contoh cara titik-titik ini dapat ditampilkan:
Melacak objek dalam video
Gemini Robotics ER 2 juga dapat menganalisis frame video untuk melacak objek dari waktu ke waktu. Lihat Input video untuk mengetahui daftar format video yang didukung.
Python
from google import genai
client = genai.Client()
uploaded_file = client.files.upload(file="my-video.mp4")
prompt = """
Point to the red ball in every frame where it appears.
The answer should follow the json format: [{"point": [y, x],
"label": <label>}, ...]. The points are in [y, x] format
normalized to 0-1000. Return one entry per frame that contains
the object.
"""
image_response = client.interactions.create(
model="gemini-robotics-er-2-preview",
input=[
{
"type": "video",
"uri": uploaded_file.uri,
"mime_type": uploaded_file.mime_type
},
{"type": "text", "text": prompt}
],
)
print(image_response.output_text)
Deteksi objek dan kotak pembatas
Selain titik, Anda dapat meminta model untuk menampilkan kotak pembatas 2D, yang memberikan detail spasial yang lebih banyak untuk objek yang terdeteksi.
Python
from google import genai
client = genai.