Multimodal input and understanding

Overview

Multimodal understanding uses images or files as input so a model can answer questions, compare content, or extract information. It differs from image, video, and audio generation: understanding reads media, while generation creates media.

Common tasks

Task	Example
Image Q&A	Describe a scene and answer questions about the image
Multi-image comparison	Compare the content or differences between images
Image information extraction	Extract text, fields, or structured information from an image
File understanding	Summarize a file or answer questions about it

Protocol representation

Each protocol uses its own input container and content blocks. A protocol may represent a content block even when the selected model does not support that input capability.

Protocol	Input container	Image and file blocks
OpenAI Chat	`messages[].content`	`image_url`, `file`
OpenAI Responses	`input[].content`	`input_image`, `input_file`
Anthropic Messages	`messages[].content`	`image`, `document`
Gemini Generate Content	`contents[].parts`	`inlineData`, `fileData`

API access

Base URL

https://api.agtcloud.cn/v1

Image understanding

The following examples submit the same image through Chat Completions.

curl 'https://api.agtcloud.cn/v1/chat/completions' \
  -H 'Authorization: Bearer your-api-key' \
  -H 'Content-Type: application/json' \
  --data '{
    "model": "MODEL_ID",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "Describe the main content of this image."},
          {
            "type": "image_url",
            "image_url": {"url": "https://example.com/example.jpg"}
          }
        ]
      }
    ]
  }'

The text result is available at choices[0].message.content.

File input

There is no protocol-independent request structure for file input. OpenAI Chat uses file, Responses uses input_file, Anthropic Messages uses document, and Gemini uses inlineData or fileData. Build the request according to the selected API Reference.