Build Hour: Responses API — A Practical Guide to Agentic, Multimodal AI

In this Build Hour hosted by OpenAI, I walk you through the Responses API: why it exists, what problems it solves, and how you can migrate from older chat-based endpoints to a new, agentic primitive. I’m Christine from the startup marketing team, and I’m joined by Steve from the API engineering team. Together we demonstrate the Responses API in action, migrate a simple chat app, and build a playful day-in-the-life simulator that highlights hosted tools, stateful reasoning, and multimodal workflows for GPT-5. This article summarizes everything I covered, expands on the technical and design concepts, and provides practical guidance you can use right away.

Overview 🧭

I’ll start with a concise summary: the Responses API is OpenAI’s flagship API for building agents. It replaces and extends older primitives like Completions and Chat Completions by introducing a flexible “item” architecture, built-in hosted tools (search, file search, code execution, ImageGen, and more), and stateful reasoning capabilities that persist chain-of-thought across requests. The API is intentionally designed to support both quick conversational interactions and long, agentic rollouts where a model calls multiple tools and executes multi-step plans in a single logical request.

Throughout this piece I’ll explain key concepts, show how the Responses API changes application architecture, describe migration strategies and tooling, and cover common questions I answered live—like performance characteristics, prompt caching, and best practices for rehydrating previous reasoning.

Why I believe the Responses API was needed 🤖

When I reflect on the evolution of language model APIs, it helps to look back. Early APIs were centered around simple text completions—models took a prompt and continued it. Then the industry moved to conversational APIs with Chat Completions, which modeled dialogues as ordered messages. Those patterns served a generation of applications well.

But model capabilities have changed. Modern models such as GPT-4o and GPT-5 are highly agentic and multimodal. They don’t just output single messages; they plan, call tools, reason visually, and take sequential actions that can span minutes. The older chat primitive—with a single sampling point per request and limited ways to represent actions—became a friction point for richer workflows. The Responses API solves this by:

Modeling everything as items (messages, tool calls, reasoning steps) rather than collapsing actions into messages.
Allowing multiple sampling steps in a single call—so the model can "think, act, see results, think again, and finally answer" without losing the chain of thought.
Providing first-class hosted tools and an MCP (managed code provider) interface for remote, authenticated function execution.
Designing streaming as a finite set of strongly typed events rather than a chaotic stream of deltas.
Making stateful reasoning a core part of the experience: you can persist and rehydrate chain-of-thought tokens across many requests for faster, cheaper, and more reliable multi-step runs.

In short, the Responses API is purpose-built for agents. It reduces engineering friction when your model needs to use tools, manage state, or handle multimodal content.

Core concepts and terminology 🧩

Before diving into demos or migration steps, it’s important to understand a few key concepts I used throughout the session and will refer to here.

Item

An item is the atomic unit of input and output in the Responses API. Items can be messages, function or tool calls, reasoning tokens, or other structured objects. Everything you pass to the model or get back from it is represented as an item. This contrasts with the old chat API where the single central primitive was a message and function calling felt bolted-on.

Agentic loop

The Responses API enables an agentic loop: the model can sample multiple times inside a single logical request. For example, it can generate code, a hosted code interpreter can run that code and return results, and then the model can sample again using those results to produce a final answer. This loop removes the need for the model to fully re-think each step after every round-trip and preserves the plan across steps.

Reasoning items and reasoning summaries

Reasoning items are explicit tokens representing internal chain-of-thought or intermediate reasoning. The API can emit "reasoning" items during a request, and those items can be stored or even summarized into a short, consumable format for users while the model continues to run. This is particularly useful for long-running computations where you want to show progress or insights while the system computes a final answer.

Hosted tools & MCP

Hosted tools are first-class services that the Responses API can call directly (web search, file search, ImageGen, code interpreter, etc.). MCP (Managed Code Provider) is a pattern for remote function calling: the Responses API queries an MCP server to list available functions for a user, passes function definitions to the model, receives a tool invocation, then routes the invocation to the external MCP service for execution and receives the live result.

Stateful vs stateless operation

The Responses API is stateful by default: conversation objects and previous response IDs let you persist context and chain-of-thought on OpenAI’s servers. If you operate statelessly, you can still preserve the chain of thought by requesting encrypted reasoning content to rehydrate locally. For enterprise or compliance setups that require stateless operation, this encrypted content is an important feature.

Key features I highlighted ✨

During the Build Hour I emphasized a set of features that together make the Responses API a major shift in how developers build with LLMs. I’ll unpack each one here and explain practical use cases.

Preserved reasoning

When a model emits a reasoning item and that item is persisted, you can pass it back in later calls. This means the model can remember what it planned to do and take the next action without recomputing the plan. In practice this reduces latency and token cost in multi-step workflows and increases success rate for tool calling. I mentioned a quantitative example live: a ~5% improvement in tool-calling accuracy and ~20% P50 latency improvement on long, multi-turn rollouts.

Multiple samples per request

Because the API can sample multiple times, it supports workflows like:

Write code → execute in a hosted interpreter → ingest execution output → sample again → finalize answer.
Plan a research path → call multiple web searches or file searches → synthesize results → emit final recommendation.
Ask a multimodal task that requires vision and text tools iteratively.

This multi-sample capability is essential when the model needs to act as an agent rather than just respond to a single prompt.

Items in / items out API surface

The item-based model greatly simplifies client-side handling. Instead of mixing function-call objects inside message strings, you get a predictable set of item types in the output. This makes the server-side state machine easier to implement: process each item depending on its type (message, tool call, reasoning, MCP approval request) and persist or present it accordingly.

Multimodal workflows

Vision and multimodal content are first-class. You can pass images via base64 or URLs, upload files (PDFs), and have the model reason over them. For example: pass a utility bill PDF and ask "Why was my bill unusually high in September?" The Responses API will extract content, let the model analyze it, and can produce a detailed breakdown. This is immediately applicable to RAG systems, document assistants, and multimodal customer support bots.

Streaming events redesign

Streaming is cleaner: events are strongly typed and finite. Instead of object deltas that force clients to accumulate everything, you get explicit events like response.started, output.text.delta, output.item.added, output.item.done, response.failed, etc. This enables straightforward switch-case logic on the client to render real-time UI elements, progress indicators, or intermediate reasoning summaries.

Performance and cost considerations 💸

I addressed an important question during the session: how does the Responses API compare with Chat Completions in performance and cost? The short answer: for agentic, multi-step workflows it’s both faster and cheaper at the median.

Why faster? Because the model can preserve a chain-of-thought and avoid re-thinking at each step. A long rollout that calls multiple tools will often reuse the initial planning content instead of re-emitting the plan repeatedly, trimming token output and compute time.
Why cheaper? Fewer emitted tokens and more effective caching reduce costs. Prompt caching benefits when you keep context as an append-only prefix; reusing prior context increases cache hit rates.
Where it matters most: multi-step tasks that call many tools or involve host-executed computations. For short, single-turn chat messages, differences are less pronounced.

From empirical examples I discussed, median end-to-end latencies for long tool-calling rollouts were about 20% shorter on Responses API compared to Chat Completions. That’s material for any product where agents take many steps.

Built-in tools and MCP: practical uses 🧰

One thing I’m excited about is the range of hosted tools that plug into Responses. These are not afterthoughts—they are tightly integrated primitives designed to be safe, performant, and accessible to the model within the same request loop.

Hosted tools I demonstrated

File Search: search through uploaded PDFs and documents.
Code Interpreter: run code server-side and return outputs to the model.
Web Search: fetch live web results for up-to-date information.
ImageGen (Image Generation): create images via prompts and return base64 images streamed back.
MCP (Remote Function Calls): list tools available to the user from an external service (like a project management board) and execute them with proper authorization.

These tools enable common real-world workflows: adding issues to a project board, running experiments and inspecting results, generating images to match textual descriptions, and retrieving fresh facts from the web to reduce hallucination risk.

MCP in practice

MCP deserves special attention because it allows the Responses API to interact with external systems in an authenticated, namespaced manner. Here’s the typical MCP flow I outlined:

At request time, Responses queries the MCP server for the list of functions/tools the user is allowed to call.
Those function definitions are presented to the model as callable tools (namespaced by server label).
The model chooses a tool and outputs structured JSON arguments indicating the intended call.
Responses sends that JSON to the MCP server, which executes the action and returns structured results.
The model rehydrates the result and continues planning or emits a final summary of what it did.

This flow is extremely useful in scenarios like:

Project management automation: creating and updating issues in a task board (e.g., Linear, Jira).
Internal developer tooling: an agent that can run CI commands, fetch logs, and open bug tickets.
Personal assistant capabilities: an agent that can schedule calendar entries or query internal knowledge bases with proper authorization.

Migrating from Chat Completions: the migration pack 🛠️

Migration is often the scariest part for engineering teams. I understand that—changing a core dependency is non-trivial. That’s why I showed a practical migration workflow and the Codex CLI migration pack that automates much of the work.

Here’s how I approached migration in the demo and why the steps make sense.

Migration steps I used in the demo

Run the migration CLI on your repository to identify chat completions usage and standard patterns.
Choose replacement model targets (e.g., GPT-5 mini, GPT-5 nano, or GPT-5 full depending on needs).
Switch the conversation mapping from message-based to items-based input: convert messages to input.items and adapt any function calling to item types.
Update streaming handler code to respond to strongly typed streaming events instead of object deltas.
Optionally add reasoning summary settings to get intermediate reasoning summaries in the UI while the model completes its plan.
Test edge cases: tool calls, error handling, and stateless vs stateful modes.

The migration pack feeds Codex with helpful prompts, docs, and acceptance criteria so it can adapt code patterns automatically. In the demo the automated run completed quickly for a small app; in larger apps it scales and still significantly reduces manual effort.

What the migration tool changes

Examples of practical changes you’ll see in diffs after the migration tool runs:

Switch from chat messages array to input.items with type annotations.
Insert reasoning item handling and encrypted reasoning content fields for local rehydration if you operate statelessly.
Adapt streaming handlers to the Responses streaming event types (output.text.delta, output.item.added, response.created, etc.).
Swap the model name in requests to GPT-5 or other recommended endpoints.

These are concrete, actionable diffs that are easy to understand and test. The goal is to make migration safe and predictable.

Demo: building an interactive simulator — day in the life 🎮

To make the concepts concrete, I built a playful OpenAI Simulator that represents a day on the API engineering floor. The simulator demonstrated how agents can be characters with distinct prompts, tool permissions, and multimodal capabilities. Two characters I used were Sam (Sam Altman) and Wendy Che (an engineer on the API team). The simulator served three main purposes:

Show how different agents can have distinct prompt backstories and behaviors.
Demonstrate tool access and MCP integration (listing, approving, and calling functions in an external project board).
Highlight multimodal tool usage: web search followed by image generation with streamed image data.

Here are the key flows I implemented and what they illustrate.

Reasoning summaries to improve UX

Users don’t want to be left staring at a blank chat while the model thinks for several seconds. Reasoning summaries provide a short, streaming digest of the model’s intermediate chain-of-thought while sampling continues. In the simulator, I enabled a reasoning summary with effort=medium and summary=auto. The model emitted reasoning items, the API ran a sideline summarizer to produce short summaries, and the UI rendered these reasoning deltas as a blue “thinking” message while the model worked on the final answer.

This pattern is ideal for product UIs that want to make multi-step agent actions feel responsive. It’s also an excellent place to add trust and transparency: by surfacing intermediate reasoning you help users understand the agent’s chain-of-thought and intervene if necessary.

MCP tool flow: list issues and create issue

In the game, Sam was given access to a Linear board via an MCP server. The flow went like this:

User asked Sam to list tasks on the AGI board.
Responses queried the Linear MCP server to list authorized functions for that user (getIssue, listIssues, createIssue).
Sam chose to call listIssues; the Responses API invoked the MCP server with the model’s JSON arguments.
Linear returned a list of issues; Sam summarized them and presented them in chat with a friendly, humorous flavor.
User asked Sam to create a new issue (e.g., distinguishing MacBooks vs Windows PCs); Sam called createIssue via MCP, and the API returned the newly-created issue object which Sam then summarized.

This demonstrates how agents can perform real actions in your stack: surface curated tasks, create structured records, or run operations—all while preserving a human-friendly narrative of what they did.

Web search + ImageGen multi-tool flow with Wendy

Wendy got a task to "find what a French bulldog looks like" and then "draw me a picture." I gave Wendy access to web search and the ImageGen tool. The agent’s plan comprised several web searches to collect descriptive facts followed by a call to the image generation tool.

Streaming events captured each state: websearch.call.in_progress, websearch.call.searching, websearch.call.completed, image_generation.call.in_progress, and finally image data as base64 in the completed item. My UI opened the image when the image_generation call was done. This pattern shows how an agent can combine factual retrieval (to avoid hallucinating image features) with generation, producing higher-quality and grounded outputs.

Agent Builder preview: drag-and-drop workflows 🧩

Building agents should not necessarily require hand-writing complicated orchestration logic. The Agent Builder (Agent Kit and Chat Kit launched at Dev Day) provides a visual, node-based way to author agent logic. I gave a quick walkthrough of a simple workflow with three nodes:

Decision node: classify whether a user query needs a web search.
If-else node: route to either a web-enabled agent or a conversational agent.
Worker agents: agents with specific tool permissions (web search enabled or pure chat responder).

The preview showed a "what's the weather near me?" example where a decision node categorized the query as requiring web search and then routed it to a web-enabled agent that knows the user's location. For "tell me a joke" the decision node returned false and the flow went to a pure conversational agent. Watching the nodes light up and the typed outputs appear made it clear how agent builder lowers the bar for teams to create complex behaviors without writing glue code.

Q&A highlights and practical tips ❓

In the Q&A I answered a number of real questions. Here are key takeaways and practical advice that apply to most teams.

How to reduce hallucinations and return structured JSON reliably

Few-shot prompting remains highly effective with the latest models. If you need reliable structured JSON output from GPT-5 mini or similar endpoints, provide varied and clear examples demonstrating the input-output mapping. If hallucinations are due to missing facts, consider adding tools such as web search to fetch live data or file search for internal documents. In short: strengthen the instruction via few-shot examples, and add tools to give the model reliable references.

How to rehydrate previous items across requests

There are multiple options:

Append the full list of items from the prior conversation and pass them in each new request (append-only strategy).
Use the previous_response_id parameter to chain off a single prior response; the API will fetch and rehydrate the prior context automatically.
Create and use a conversation object (Conversations API) and keep passing incremental input items. The conversation grows on the server and you can just reference the conversation ID for subsequent responses.
If you need stateless client behavior, request encrypted reasoning content from Responses and use that to rehydrate reasoning tokens on your side.

Choose the option that matches your privacy, performance, and operational constraints.

Prompt caching and how to maximize it

Prompt caching works by token prefix matching. The model-side cache will provide discounted input token pricing for tokens that match a cached prefix. So the best way to leverage it is:

Keep the early context of your request stable—treat context as append-only rather than editing earlier parts of the conversation.
Avoid heavy reordering of the context or changing items far upstream between calls; changes there alter the tokenized prefix and invalidate the cache.
Use conversation objects or previous_response_id to preserve consistent prefixes across requests.

Common mistakes I see

In my experience, teams often:

Forget to request encrypted reasoning content when operating statelessly and thus lose the benefits of chain-of-thought rehydration.
Try to reimplement hosted tools poorly instead of leveraging the built-in tools; using the built-in options saves time and yields better accuracy and performance.
Fail to use the conversation or prompt objects that help version and iterate on tasks more productively.

These are easy fixes, and the product primitives are there to support good practices.

Best practices and recommended patterns ✅

Drawing on the demos and Q&A, here are practical recommendations I follow when designing agentic applications with the Responses API:

Design prompts as tasks, not monoliths

Create modular prompts (use the Prompt object) that encapsulate behavior such as "summarize reasoning" or "create ticket." Version those prompts in the dashboard and reference them by ID in your API calls. This makes iteration, testing, and rollback much easier than editing inline strings across codebases.

Use conversation objects for chat UIs

When building chat interfaces, use the Conversations API to hold the conversation state. Pass the conversation ID to the Responses API and send incremental input items. It’s simpler and keeps the client code light.

Leverage reasoning summaries for UX

If tasks take perceptible time, enable reasoning summarization so users see a digest of what the agent is thinking. That builds trust and reduces the perception of latency. Place a UI affordance to show detailed chain-of-thought if your product needs transparency for audits or debugging.

Restrict tool permissions and require approvals where needed

When exposing powerful tools, adopt least privilege. Restrict which tools an agent can call depending on user role, and require approval for irreversible or sensitive operations. The Responses API supports allowlists and approval requests (MCP approval flows), which you should wire into admin UIs.

Prefer hosted tools over rolling your own unless necessary

Hosted RAG, file search, and code execution are non-trivial to implement well. Use the built-in tools for a faster path to reliable results and better cost/performance characteristics.

Common migration pitfalls and how to avoid them ⚠️

If you plan to migrate from Chat Completions, watch out for these pitfalls and take corrective steps:

Not updating streaming handlers: change your client code to handle typed streaming events rather than accumulating raw deltas.
Dropping reasoning context accidentally: if you’re stateless by default, make sure to request encrypted reasoning content or adopt the conversation object.
Mishandling function/tool calls: convert embedded function-call semantics into item-typed tool calls and update parsing logic accordingly.
Breaking prompt caching: keep the early context stable and prefer append-only context growth to increase cache hits.

Using the migration pack and running automated tests that exercise tool calling paths makes this transition much smoother.

Resources, links, and next steps 📚

If you want to follow along or reproduce the demos I used in the Build Hour, here are the resources I referenced and how I recommend you proceed:

Build Hours GitHub repo for code examples and demo templates: https://github.com/openai/build-hours
Responses API docs and reference: https://platform.openai.com/docs/api-reference/responses
Migrate to Responses guide: https://platform.openai.com/docs/guides/migrate-to-responses
Sign up for upcoming Build Hours and previews: https://webinar.openai.com/buildhours/

Suggested learning path:

Read the Responses API docs to understand the item model and streaming event types.
Try a simple migration of a toy chat app using the migration pack to see diffs and patterns.
Experiment with hosted tools like web search and file search; test how reasoning summaries feel in a live UI.
Build a small agent that uses MCP to connect to a test endpoint (e.g., a local mock of a project board) and flow from intent classification to function calls to final summary.
Iterate on prompts using the Prompt object to version and improve behavior.

Conclusion 🏁

I’ve found the Responses API to be a meaningful evolution in building agentic applications. The design moves beyond a single-sample, message-first approach and embraces an item-first, multi-sample, tool-integrated architecture that is well-matched to what modern models can do. For teams building agents that call multiple tools, need persistent chain-of-thought, or rely on multimodal inputs, Responses is purpose-built and offers practical gains in performance, cost, and developer ergonomics.

If you’re on Chat Completions today, migration is realistic—use the Codex CLI migration pack to automate common changes, test your workflows, and gradually roll over. If you’re starting fresh, consider designing around items, enabling reasoning summaries for UX, and leveraging hosted tools to avoid reinventing complex infrastructure.

Finally, I enjoyed demonstrating how these concepts come together in a playful simulator and in the Agent Builder preview. My advice: prototype quickly, lean on built-in tools for reliability, and iterate on prompts with few-shot examples and the Prompt object. I’m excited to see what you build next.

Thanks for reading—I hope this guide helps you understand the Responses API and gives you practical steps to get started. If you want to dive deeper, check the GitHub repo for runnable examples and follow up on the next Build Hours focused on Agent Kit, agent RFT, and memory patterns.