Handling image data from MCP servers

[MoNorth]

Please read this first

• Have you read the docs? Agents SDK docs
• Have you searched for related issues? Others may have faced similar issues.

Describe the bug

The image returned by MCP cannot be recognized by LLM, and the result returned by function call cannot be recognized if it is an image. Can you provide a hook that can rewrite the return result of the function call so that it can be recognized by LLM.

[
    {
        "type": "image",
        "data": "xxxx(base64)",
        "mimeType": "image/jpeg"
    }
]

Debug information

• Agents SDK version: (e.g. v0.0.5)
• Runtime environment (e.g. Node.js 22.16.0)

Repro steps

Ideally provide a minimal JavaScript/TypeScript script that can be run to reproduce the bug.

Expected behavior

A clear and concise description of what you expected to happen.

[seratch]

Hi @MoNorth, thank you for reaching out!

The LLM can actually receive base64 image data, but I generally don't recommend connecting agents with image data responses from an MCP server because:

• Sending the entire base64 data to Responses API (or other models) after the MCP tool calling can consume a large number of tokens
• Running agents with large data inputs may result in slow responses due to the same reason

Here’s a simple example I just quickly wrote: seratch@4109ccc It works, but it's extremely slow and consumes tokens even with this small-sized image data exchange.

I’m not sure if this will suit your use case, but I would suggest having the MCP server generate the image data, save it somewhere else, and then return the URL to the generated data in its MCP tool response.

[MoNorth]

Thank you for your response!
Do we need a hook that can intercept the results of the tool? I can use this hook to process the image data returned by MCP and input it to LLM in other forms

[seratch]

I don't think having hooks on the SDK side is a straightforward approach to handling this, as it would be too complex for developers, especially when working with multi-turn agents. So, I’d suggest either modifying the MCP server to not return image data (if you own the MCP server), or placing a proxy between the MCP and your agents to adjust the tool responses. Hope this helps you out.

[github-actions]

This issue is stale because it has been open for 7 days with no activity.