Segmenting with prompts

Segment Anything Model 3.1 (SAM 3.1) segments by concept. Give it a short noun phrase naming one thing — "yellow school bus", "glasses", "red hat" — and it returns a box and a mask for every match. It reads the phrase as a concept to locate, not as an instruction to follow, so you name the object rather than writing a command or a question.

SAM 3.1 on Meta Model API takes text prompts. Name a concept in input_text and the model detects every instance. The open-weight research model also accepts visual prompts — input boxes, points, and clicks that mark where to look. The Model API works from the text concept alone; the boxes it returns are outputs (located regions), not inputs.

One Responses API call on sam-3.1 carries the prompt and the media, an image or a video, and streams a box and a mask per match. The prompt is the part that decides what you get back, so start there.

Write the concept prompt

The prompt goes in an input_text part as a short noun phrase. SAM 3.1 matches the concept you name, so concrete wording controls the result.

  • Name one concept with a noun phrase: "yellow school bus", "red hat", or "glasses". Commands ("segment the bus") and questions ("where is the bus?") are unreliable and often return nothing. Name the object; don't instruct the model.
  • One concept per request: there is no multi-concept syntax. Comma, semicolon, and plus lists ("bus, bicycle") return zero matches; an "and", a newline, or a second input_text part segments only one of the concepts. To segment several concepts, send one request each.
  • Add attributes to narrow: "square" matches every square; "red square" matches only the red ones. Color, type, and other concrete attributes filter predictably, down to zero when nothing matches.
  • Formatting is ignored: plurals ("red squares"), case, trailing punctuation, and extra whitespace all return the same result. Styling carries no meaning.
  • Skip negation, boolean OR, and spatial phrasing: "person not wearing a hat" and "cat or dog" return inconsistent results, and "person on the left" is only partially honored. Segment the plain concept, then filter by position or attribute in your app using the returned box coordinates.
  • Keep it short: a concise noun phrase works as well as a long description. Extra words give no benefit, and padding a prompt with filler degrades matching.

SAM 3.1 locates the objects you name; it doesn't answer questions or reason about a scene. For scene understanding, use the Muse Spark model family, or a combination: use Muse Spark to caption, answer questions, or identify scene activity via image understanding or video understanding, then pass the concept to SAM 3.1 for the segmentation work.

Attributes change the match set predictably. On an image of six squares, "square" returns six matches, "red square" returns three, and "yellow square" returns none. Each match comes back as one box and one mask; see Segment an image for the wire records and read segmentation output to parse them.

A concept that isn't present returns zero matches: the stream completes with response.completed and an empty output_text. Treat it as a normal result, not an error.

Build the request

A request pairs the concept prompt with one media part inside a single user message. Send the phrase in an input_text part and the media in an input_image or input_video part:

  • input_text: the concept prompt, a short noun phrase for one object (see Write the concept prompt).
  • input_image: a still as image_url.
  • input_video: a clip as video_url, or as file_id for a video uploaded through the Files API.
  • image_url / video_url: a public URL or a base64 data URL (data:image/png;base64,...).

The same model serves both images and video. Swap the media part and keep the rest of the request identical. One concept per request: to segment several object types, send several requests.

Segment an image

Send an input_image part and stream the response to response.completed:

python
import os
​
from openai import OpenAI
​
client = OpenAI(
    base_url="https://api.meta.ai/v1",
    api_key=os.environ["MODEL_API_KEY"],
)
​
with client.responses.stream(
    model="sam-3.1",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {
                    "type": "input_text",
                    "text": "glasses",
                },
                {
                    "type": "input_image",
                    "image_url": "https://example.com/photo.png",
                },
            ],
        },
    ],
) as stream:
    for event in stream:
        if event.type == "response.output_text.delta":
            print(event.delta, end="", flush=True)
print()

An image returns on a single line: one record per match, comma-separated, each record a box then a mask. Here w and h are the source image dimensions, repeated on every record:

text
<0f>0<|box;x1=181;y1=365;x2=556;y2=486;w=1516;h=1600|><|mask;x=0;y=0;data=122,376,~...|>

<0f> is the frame, the leading integer is the object ordinal, then one box then one mask; ~ marks a lossless mask payload. Full grammar in reading segmentation output.

Segment a video

Swap in an input_video part with a video_url. Start with a hosted clip to tune the prompt before uploading anything: