Vision

Vision-capable models accept image inputs alongside text. ai& supports three input modes; choose the one that fits your use case.

Three ways to send an image

{
  "role": "user",
  "content": [
    { "type": "text", "text": "What's in this image?" },
    { "type": "image_url", "image_url": { "url": "https://example.com/cat.png" } }
  ]
}

{
  "role": "user",
  "content": [
    { "type": "text", "text": "What's in this image?" },
    { "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBORw0KG..." } }
  ]
}

{
  "role": "user",
  "content": [
    { "type": "text", "text": "What's in this image?" },
    { "type": "file", "file": { "file_id": "file-abc123" } }
  ]
}

URL requirements

ai& checks media URLs before the request reaches a model. The check covers image_url, video_url and audio_url parts, image_url on Responses input_image, file_url on chat file parts and Responses input_file, Anthropic sources with type: "url", and server_url on Responses MCP tools. A URL passes only if it:

is an absolute https URL. http and other schemes are rejected.
uses the default port, 443.
contains no username or password.
names a host, not an IP address. This applies to IPv4 and IPv6 in any notation.
ends in a public top-level domain, such as .com or .jp. localhost, single-label names, and names under .local or .internal are rejected.

Media sent as a data: URL or by file_id is not checked.

A URL that fails returns 400 with type: "invalid_request_error" and code: "invalid_media_url". param names the field, for example messages.0.content.1.image_url.url. On /v1/messages the error uses Anthropic’s envelope. The request does not reach a model, and you are not charged for it.

To fix a media URL, host the file at a public https URL, upload it with the Files API and pass its file_id, or send it inline as a base64 data: URL. An MCP server_url must itself be a public https URL.

The file_id path

When you send {type: "file", file: {file_id}}, the model receives the bytes you uploaded — no public hosting required. The wire format on your end stays the same regardless of which mode you use.

Upload images via Files with purpose: "vision" — or omit the purpose and let it be inferred from MIME.

Supported MIME types: image/png, image/jpeg, image/webp, image/gif. Max size: 100 MiB.

Capability gating

Vision requests are rejected if the target model lacks the vision capability — both for inline image_url parts and for file_id references whose purpose maps to vision. List models and their capabilities via Models.