Attachments & Content Parts

A message's content is either a string or an array of parts. When a part is an image, audio clip, video or file — an attachment — the SDK treats it as media, not as text. This page explains how attachments are counted, summarized and traced, and which helpers you can use when you build plugins or hosts that flatten content yourself.

What's new in 0.10.6

Attachments are no longer JSON.stringify'd into text. Token estimation, the summarizer's input and trace display now use a short placeholder such as [image: photo.png], and trace message sections carry the structured parts so a UI can render a thumbnail. See What changed in 0.10.6.

When this matters

• Your agent accepts photos, scans, audio or documents from users.
• You set maxContextTokens and saw compaction trigger far too early after an image was attached.
• Summaries said "three files" when four were attached, or could not tell a picture existed.
• You render traces in your own UI and want to show the attached image, not a wall of base64.
• You write a plugin or host code that turns a message into a string.

What counts as an attachment

The SDK recognises these part shapes. Every one of them is covered by the placeholder, the token estimate and the trace display:

Part type	Kind	Notes
image, image_url, input_image, image_file	image	image is the unified shape (source: { type: "base64" | "url", … }); image_url and input_image are the OpenAI shapes
audio, input_audio	audio	input_audio carries { data, format }
video, video_url	video
file, input_file, document	file
(no type) with inlineData / inline_data / fileData / file_data	by MIME type	Gemini's untyped parts; the kind comes from the MIME type (image/*, audio/*, video/*, otherwise file)

Text parts ({ type: "text", text }) and anything else — for example tool_use blocks — are not attachments.

const message = {
  role: "user",
  content: [
    { type: "text", text: "What is in this photo?" },
    { type: "image_url", name: "photo.png", image_url: { url: "data:image/png;base64,iVBORw0…" } },
  ],
};

The placeholder

Anything that has to flatten content to text shows an attachment as a one-line placeholder:

[image: photo.png]
[audio: audio/mp3]
[file: report.pdf]
[video: clip.mp4]
[image]                ← nothing could be named

The label is the first of these the part has:

1. its own name (name, fileName, filename, file_name, file.filename, image_url.name, source.fileName …)
2. its file id (fileId, file_id, file.file_id …)
3. its media type (mediaType, mimeType, a data: URL's type, audio/<format> for input_audio …)
4. the file name at the end of a plain URL — never its query string, so a signed URL's signature is not exposed (https://cdn.test/a/b/cat.png?sig=SECRET → [image: cat.png]), or the id of a session-file:// reference

A label is flattened to one line (newlines and ] become spaces) and cut at 80 characters.

Where the placeholder is used

Surface	Before 0.10.6	Now
Token estimation (default counter and maxContextTokens checks)	the part's JSON, including the whole base64 payload — a ~400-token photo counted as ~18,000 tokens	the placeholder; the estimate is the same for a tiny image and a huge one
Summarizer input	the part's JSON, truncated into a wall of base64	the placeholder, so the summary can say a picture was there
Trace display (content of a message section)	the same JSON wall	the placeholder; text parts show as their text, not as { "type": "text", … }
contentToString(content)	inlined data: URLs	a placeholder; a remote http(s) image URL is still shown ([image:https://x/i.png detail=high])

Tool-call and unknown parts are still counted by their JSON

tool_use blocks and parts the SDK does not recognise are real tokens, so they are still counted from their JSON.

Token estimates and attachments

The default counter is a character heuristic over text, so an attachment costs only its placeholder. That avoids phantom compaction, but it also means the estimate does not include the real per-image cost your provider charges. Decide which side you want:

• Reserve headroom. Lower limits.maxContextTokens a little to leave room for the images the model will receive.
• Count images yourself. Plug a tokenCounter that adds a per-image cost:

import { createSmartAgent } from "@cognipeer/agent-sdk";

const IMAGE_TOKENS = 800; // your provider's per-image cost

const agent = createSmartAgent({
  model,
  limits: { maxContextTokens: 24_000 },
  tokenCounter: (text) => {
    const images = (text.match(/\[image(?::[^\]]*)?\]/g) ?? []).length;
    return Math.ceil(text.length / 4) + images * IMAGE_TOKENS;
  },
});

The counter receives flattened text (the text of the messages being measured, joined), so it sees the placeholders. See Limits & Tokens → Images, audio and files.

Traces: parts on message sections

With tracing.logData: true, each message in a trace is a message section. content stays a readable string with the placeholder in place of the payload. When the message carries a non-text part, the section additionally has a parts array with the structured parts in order, text parts included, so a host can draw the image where content can only name it:

{
  "kind": "message",
  "role": "user",
  "content": "what is this?\n[image: photo.png]",
  "parts": [
    { "type": "text", "text": "what is this?" },
    { "type": "image_url", "name": "photo.png", "image_url": { "url": "data:image/png;base64,…" } }
  ]
}

• A text-only message has no parts field at all.
• The type is TraceMessagePart ({ type?: string; text?: string; [key: string]: any }), exported from the package root, and TraceMessageSection gains the optional parts?: TraceMessagePart[].
• Plain content is unchanged, so a consumer that ignores parts keeps working.

parts carries the part as it was sent

parts keeps the structured part, including an inline data: URL or base64 payload — that is what makes thumbnails possible. A trace file or sink that receives it receives the image bytes. Set logData: false to record metadata only, or strip parts in a customSink if images must not be stored.

See Debugging & Tracing for the trace payload.

Content helpers

These helpers are exported from @cognipeer/agent-sdk for plugins and hosts that flatten content themselves. They produce exactly the text the SDK itself uses.

Helper	Signature	Returns
describeContentPart(part)	(part: unknown) => string | undefined	the placeholder for a media part; undefined for text, strings and non-media parts
contentPartToText(part, fallback?)	(part: unknown, fallback?: (part: unknown) => string) => string	strings and text parts as they are, media as its placeholder, anything else through fallback (default: its text / content, else its JSON)
mediaKindOf(part)	(part: unknown) => MediaKind | undefined	"image" | "audio" | "video" | "file", or undefined for text / non-media
hasMediaPart(content)	(content: unknown) => boolean	true when content is an array holding at least one media part

import { describeContentPart, contentPartToText, hasMediaPart, mediaKindOf } from "@cognipeer/agent-sdk";

const part = { type: "image_url", name: "photo.png", image_url: { url: "data:image/png;base64,QUJD" } };

describeContentPart(part);                       // "[image: photo.png]"
contentPartToText({ type: "text", text: "hi" });  // "hi"
contentPartToText(part);                         // "[image: photo.png]"
mediaKindOf(part);                               // "image"
hasMediaPart([{ type: "text", text: "a" }, part]); // true

MediaKind is exported as a type. The existing content helpers — contentToString, extractMessageText, collectAttachments, mapTextParts, replaceTextContent, textFromContent — are unchanged; use mapTextParts / replaceTextContent to rewrite the text of a message without flattening its attachments away (see Plugins & Hooks).

Example: list what a message carries

collectAttachments normalises both part shapes into one record (kind, mediaType, url, fileName, …), so a policy does not need to know which one it was handed:

import { collectAttachments } from "@cognipeer/agent-sdk";

for (const attachment of collectAttachments(message.content)) {
  console.log(attachment.kind, attachment.mediaType, attachment.fileName);
}

Troubleshooting

Compaction still triggers too early with images. Check the version is 0.10.6 or later, and that images reach the SDK as one of the shapes in the table above. A part with an unrecognised type is not treated as media and is still counted from its JSON.

The estimate looks too low for a conversation full of photos. That is by design: the default counter sees text. Add a per-image cost in a custom tokenCounter or reserve headroom in maxContextTokens.

The summary does not mention image content. The summarizer sees [image: photo.png], not the pixels. It can record that a picture was attached; it cannot describe what is in it. If the image content matters, ask the model to describe it in a normal turn before compaction.

My trace UI shows the placeholder, not the picture. Read the section's parts array and render the image_url / image part yourself. A text-only message has no parts.

parts makes my trace files huge. parts includes inline payloads. Set logData: false, or drop parts in a customSink.

What changed in 0.10.6

• Attachments were JSON.stringify'd into text. Anything that flattened a message to a string fell back to the part's JSON, including its whole base64 payload. Token estimation, the summarizer's input and trace display now use the placeholder, for image, audio, video and file parts in every shape the SDK sees (image, image_url, input_image, input_audio, file, input_file, document, Gemini inlineData).
• contentToString no longer inlines a data: URL (a remote URL is still shown).
• Trace message sections gain parts (new type TraceMessagePart) when a message carries a non-text part.
• New exports: describeContentPart, contentPartToText, mediaKindOf, hasMediaPart (and the MediaKind type).

Related

• Limits & Tokens — budgets and the pluggable token counter
• Summarization & Context — how compaction reads a conversation
• Debugging & Tracing — trace sections and sinks
• Vision example — the message shape for an image