Uber uses GPT-5 to balance speed and quality

In a short video produced by OpenAI, I—Jai Malkani, Uber’s Global Head of AI & Product, Customer Obsession—shared how our early work with GPT-5 has begun to reshape customer and partner interactions across our platforms. The conversation explored how advanced models can simultaneously raise the bar for response quality while operating at the scale of hundreds of millions of riders, drivers, eaters, and merchants. In this article I report on those findings, unpack what we've learned, and outline practical lessons and implications for teams building AI-driven conversational experiences.

🚀 The scale and stakes of conversational AI at Uber

I lead AI and product efforts focused on customer obsession at Uber, and I don’t overstate the challenge when I say we operate at a truly massive scale. Every day our systems touch hundreds of millions of people—riders calling for rides, driver partners seeking clarity about routes and incentives, eaters placing orders, and merchants managing menus and fulfillment. That scale creates a unique set of constraints and requirements:

Latency expectations: Interactions must be fast. Customers expect near-instant responses when they’re dealing with time-sensitive issues like a missing driver, a late delivery, or a billing discrepancy.
Consistency: Responses should be uniformly high quality and aligned with our brand voice across languages, regions, and platforms.
Safety and policy compliance: We must ensure model outputs do not contradict safety policies, regulatory requirements, or customer commitments.
Operational scale: Solutions must be maintainable and monitorable across millions of conversations per day.

Working in that environment means that any conversational AI we deploy must be judged not only by how helpful or human-sounding it can be, but also by how reliably it can operate under pressure, at scale, and within guardrails that protect users and the business.

🧭 Why GPT-5 stands out in our evaluations

In early experiments with GPT-5, it became clear that the model offered something important beyond incremental improvements in language generation: it provided richer, more nuanced responses that felt more natural and more helpful in context. I described the evaluation results this way in the video: GPT-5 and the structured critique framework we used “really offer the most thorough and structured critiques of our conversation, of our bots and models.” That insight was critical for us, because the biggest lift in real-world customer experience often comes from the model’s ability to understand context and to be honest about uncertainty.

Key qualities that made GPT-5 notable in our trials included:

Contextual comprehension: The model demonstrated a stronger ability to preserve conversation context across multiple turns, making follow-up answers more coherent and reducing the need for customers to repeat themselves.
Realism and groundedness: It separated what was realistic to accomplish in a conversation from what was not, which reduced hallucinations and minimized the chance of providing misleading information.
Calibration of confidence: GPT-5 often signaled uncertainty when appropriate, allowing our systems to escalate or hand off to human agents in a timely way instead of fabricating answers.
Human-like tone with brand alignment: Responses felt human while remaining “on brand”—concise, empathetic when needed, and consistent across message templates.

Those features made it an attractive candidate for production pilots where the balance between speed and quality is not just a nice-to-have, but a defining factor of success.

⚖️ Balancing speed and quality in live systems

One of the core tensions in production-scale conversational AI is the trade-off between latency and answer quality. On one hand, customers demand fast answers. On the other, high-quality answers often require deeper reasoning, larger context windows, or additional retrieval steps that introduce latency.

Our experiments were explicitly designed to measure this trade-off and to find practical ways to strike a productive balance. I often say that “speed without quality is pointless, and quality without speed is impractical.” We measured both the objective response time and subjective customer satisfaction across different model configurations and orchestration strategies.

Here are the practical approaches we used to manage that balance:

Tiered response architecture: We implemented a multi-tier system that selects a fast, lightweight model for straightforward queries and routes complex or ambiguous requests to GPT-5 for a richer, more deliberative response. This preserves low latency for common cases while enabling depth when needed.
Progressive disclosure: For tasks that could benefit from a multi-step approach, we returned a fast provisional answer and then followed up with a more detailed response from GPT-5 if the user engaged further or if the system detected unresolved ambiguity.
Cached templates and retrieval augmentation: For frequently asked questions or policy-driven responses, we combined retrieval from our knowledge bases with light generation to reduce the amount of computation required while maintaining personalization.
Explicit uncertainty handling: When the model’s confidence was low, we defaulted to concise, transparent handoffs to human agents or to system-level actions (e.g., creating a ticket or scheduling a callback) rather than risk a problematic generated answer.

The result of this approach is a system that delivers speed for the 80% of interactions that are routine, while reserving GPT-5’s deep reasoning for the 20% of conversations that truly require it. This infrastructure-level approach is essential when you are operating at our scale—hundreds of millions of interactions—and cannot accept a one-size-fits-all latency profile.

📈 Scaling quality across hundreds of millions of interactions

Scaling conversational quality is not only about technology; it’s also about reproducible processes and evaluation frameworks. When we started our GPT-5 trials, we prioritized developing robust evaluation pipelines. As I explained during the presentation, the evaluation tools “consistently separated what’s realistic versus what’s not,” and that consistency is what lets us scale.

Key pillars of our scaling strategy:

Automated critique and scoring: We built automated evaluations that score model outputs against a rubric of relevance, accuracy, policy adherence, and brand tone. These scores are used to gate deployments and to identify systematic failure modes.
Human-in-the-loop review: For edge cases and periodic audits, we involved subject matter experts—policy teams, regional ops, and customer support veterans—to review samples and provide corrective instructions to the model or to the prompt templates.
Statistical sampling across conversation types: To avoid blind spots, we sample conversations across languages, geographies, and customer intent buckets. This ensures that improvements are not just local to one use case or region.
Continuous monitoring and alerting: Production metrics—including response time, escalation rate, customer satisfaction, and compliance flags—feed into dashboards and automated alerts so we can quickly identify regressions.

By combining automated critique with human oversight, we established a repeatable, measurable path to scale. When you’re dealing with massive volumes of interactions, the only way to maintain quality is to make the evaluation process scalable and consistent. GPT-5 helped us by producing outputs that were easier to score and analyze because they were often more structured and clearer in their uncertainty estimates.

💬 Human tone, brand voice, and customer experience

One of my favorite outcomes from our GPT-5 work was how consistently the model produced responses that felt both human and on-brand. That’s not an incidental benefit; it’s central to customer trust. A response can be factual but sterile, or it can be warm and helpful while still being accurate. For consumer-facing brands like Uber, the latter is essential.

We defined a brand voice rubric emphasizing clarity, empathy, brevity, and actionability. Then we tuned prompts, system instructions, and template scaffolds so that GPT-5 would reliably meet these standards. The effect was immediate: conversations felt more natural, and customers reported higher satisfaction in pilot tests.

Specific techniques we used to tune brand alignment:

System-level instructions: We crafted concise instructions that anchored the model’s behavior—for example, "Be concise, empathetic, and provide a clear next step." These instructions reduced variance in tone.
Response scaffolding: We structured outputs into predictable segments (acknowledgement, brief explanation, and a next action), which made responses easier to parse for customers and downstream tooling.
Persona anchoring: We used short persona statements for role-specific interactions. For example, a driver-facing assistant uses a slightly more technical, direct tone, while rider-facing responses are more empathetic and reassuring.

When combined with the model’s improved contextual comprehension, this led to a noticeable shift in customer experience. As I put it in the video: “Every conversation feels really more human and on brand.” That sensation matters: it increases trust, reduces friction, and lowers the cognitive load on users navigating service disruptions or complex policy decisions.

🛠️ Practical implementation details and architecture choices

Implementing GPT-5 into a large production ecosystem is a non-trivial engineering challenge. It’s not just about calling an API; it’s about integrating model outputs into deterministic pipelines, preserving compliance, ensuring performance, and enabling rapid iteration. Below I outline the major architectural patterns and operational decisions we adopted.

Hybrid model orchestration

We used a hybrid orchestration layer that dynamically routes requests to different components based on intent classification and confidence thresholds. The orchestration logic includes:

Intent classifier: A fast model or rule-based engine classifies the incoming message and determines whether it is routine or complex.
Policy filter: A deterministic policy layer ensures that responses conform to regulatory and business requirements before they are surfaced to users.
Model router: Depending on the classification, the router selects a lightweight model for quick responses or routes to GPT-5 when deeper reasoning is needed.

Augmentation with retrieval and knowledge grounding

To keep answers factual, we complemented generation with retrieval. This is especially important when the query involves policy details, recent outages, account-specific facts, or menu information for merchants. Our pipeline typically follows this flow:

User query arrives and is classified.
Relevant documents and facts are retrieved from internal knowledge stores or user account records.
Retrieved content is fed to GPT-5 as grounding context with strict instructions to cite sources or refuse to answer if facts conflict.
Generated answer passes through a policy filter and then is returned to the user.

This retrieval-augmented approach reduces hallucination and improves the factuality of responses—particularly important for user-specific or safety-sensitive queries.

Fail-safe mechanisms and human handoff

We do not treat the model as infallible. Instead, we introduced multiple fail-safe mechanisms:

Confidence thresholds: If the model’s internal confidence or the rubric score falls below a threshold, the system prompts escalation to a human agent or to a structured ticketing flow.
Template fallback: For certain policy-bound responses, we use deterministic templates populated with retrieved facts.
Audit trail: We log both the retrieved content and the model output for later audits and compliance checks.

By prioritizing transparent escalation and robust logging, we protect customers and maintain the ability to learn from mistakes when they occur.

🔍 Evaluation, measurement, and continual learning

High-quality conversational AI is an iterative achievement, not a one-time deployment. To improve the system at scale, we built comprehensive measurement and learning systems that close the loop between production behavior and development efforts.

Key measurement pillars:

Customer-centric metrics: Net promoter score (NPS) after interaction, post-conversation satisfaction, perceived helpfulness, and resolution rate were front-and-center.
Operational metrics: Escalation rate to human agents, average handle time, and the proportion of cases that required follow-up actions measured operational impact.
Model performance metrics: Automated rubric scores for relevance, factuality, tone, and policy adherence quantified model behavior across large samples.
Longitudinal analysis: We tracked whether model-driven responses reduced repeat contact or increased re-open rates for tickets over time.

Our continuous learning loop incorporated the following steps:

Collect representative samples of model outputs and user feedback.
Use automated critiques and human review to label failure modes.
Adjust prompts, retrieval strategies, or model routing rules accordingly.
Deploy changes to a small cohort and monitor metrics for regressions before rolling out more broadly.

That iterative cadence—measure, learn, iterate—kept our production system improving without introducing regressions that could impact millions of users.

🌐 Broader implications for ride-hailing and delivery ecosystems

Beyond the immediate engineering wins, our GPT-5 experiments revealed broader strategic implications for on-demand platforms like Uber. AI-driven assistants can transform multiple parts of the user journey:

Onboarding and education: New drivers and merchants can get personalized, interactive onboarding support that reduces drop-off and speeds time-to-productivity.
Issue resolution: Customers facing missing rides or late deliveries can receive context-aware troubleshooting steps and clear next actions without waiting for a human agent.
Proactive notifications: Intelligent summarization can detect patterns—like repeated cancellations—and proactively reach out with targeted help.
Localized support: Improved language comprehension and tone management allow for higher-quality interactions across geographies and cultural contexts.

These capabilities have the potential to reduce friction across the ecosystem, increase retention, and improve operational efficiency by reducing unnecessary escalations to human teams.

🧩 Real customer scenarios and examples

To make this concrete, here are several illustrative scenarios that reflect the kinds of improvements we observed in pilots. I’ll describe each in a realistic, user-facing way.

Scenario 1: Rider with a disrupted trip

Imagine a rider whose driver canceled mid-trip and the app shows an unclear refund status. A quick, well-structured conversation with our GPT-5–enabled assistant might look like this:

"I’m really sorry your trip got canceled—that’s frustrating. I can see from your account that we automatically refunded the booking fee and are issuing a credit within 24 hours. If you’d like, I can request an expedited review and file a ticket now. Would you like me to do that?"

That response accomplishes three things: acknowledges the user’s frustration, cites a concrete action and timeline, and offers a clear next step. The model’s ability to pull the right account-specific facts and to present them in an empathetic, concise way was a clear win.

Scenario 2: Driver partner query about incentives

A driver partner asks whether they qualify for a new incentive program. Instead of offering a generic answer, our system retrieved the driver’s recent trip history and incentive eligibility rules, then produced a tailored response:

"Thanks for checking—based on your recent trips this week you’ve completed 12 qualifying trips. The incentive requires 15 trips by Sunday to unlock the bonus. Would you like me to show a list of times or neighborhoods where demand is currently higher so you can plan the remaining trips?"

This response is actionable, factual, and aligned with the driver’s practical needs—exactly what we aim for with driver-facing support.

A merchant needs to set menu availability for a holiday. The assistant provided a step-by-step guide and offered to make the update automatically once the merchant confirmed the changes:

"I can help set your holiday hours and update the menu. If you confirm, I’ll schedule the menu to switch at 10 PM on Friday and revert to regular hours on Monday morning. Do you want to proceed?"

Here, the assistant reduced friction by proposing a precise, testable action and by offering to execute it on the merchant’s behalf.

🛡️ Governance, safety, and ethical guardrails

When a generative model touches customer accounts and decisions, governance moves to the forefront. We established multiple layers of guardrails to ensure that GPT-5 outputs were safe, compliant, and aligned with our obligations to customers and regulators.

Governance components we implemented:

Policy-first prompt design: Prompts explicitly instruct the model on disallowed content and non-negotiable safety behaviors (e.g., never provide medical advice, never change a user’s payment method without explicit confirmation and verification).
Deterministic checks post-generation: Outputs run through deterministic checks that enforce things like not exposing PII, confirming refund policies, and ensuring legal disclaimers are present when required.
Human review panels: For any new use case or region, human reviewers evaluate model outputs against local laws and cultural norms before the feature is green-lit.
Audit logs and traceability: Full logging ensures we can reconstruct how a decision was made, what facts were retrieved, and what instructions were given to the model.

Safety is not a one-time checklist. It requires ongoing investment in monitoring, policy updates, and responsiveness to new threats or failure modes that surface in production.

🔄 Human-in-the-loop and the role of analysts

While models like GPT-5 can shoulder a lot of the conversational burden, human-in-the-loop processes remain vital. Analysts, policy experts, and frontline agents add value in three primary ways:

Quality assurance: Human reviewers evaluate edge cases, provide nuanced feedback, and help refine prompts and rubrics.
Policy interpretation: Humans translate regulatory or contractual nuance into actionable constraints for the model.
Continuous triage: Agents handle escalations and use model outputs as a draft or context summary that speeds resolution rather than replaces human judgment.

At Uber’s scale, people don’t disappear from the loop; instead, their roles become higher-value—focused on oversight, escalation resolution, and improving the model’s behavior.

🧾 Best practices and recommendations for teams

If you’re building or operating conversational AI for large customer bases, here are practical recommendations distilled from our GPT-5 experience:

Define the metric hierarchy: Prioritize user-facing metrics first (satisfaction, resolution) and then tie model metrics (factuality, tone) to those outcomes.
Adopt hybrid routing: Use a combination of fast models and deeper reasoning models to balance latency and complexity.
Use retrieval to ground answers: Always feed the model relevant, up-to-date facts from internal systems to reduce hallucination.
Implement transparent escalation rules: Be explicit about when to hand off to humans and automate the handoff where possible.
Instrument thoroughly: Log inputs, retrieved context, model output, and final resolution to enable audits and improvements.
Start small, iterate quickly: Pilot in constrained segments and broaden scope after validating against your metrics and safety checks.
Invest in brand voice guides: Short, clear persona and tone rules dramatically reduce variance in output.

These are pragmatic steps that helped us move from promising experiments to reliable production behavior.

🔭 Looking ahead: evolution and future directions

GPT-5’s capabilities highlight a path forward where conversational AI is not just reactive but anticipatory and context-aware. That trajectory opens new possibilities:

Proactive problem detection: Models could detect service degradation patterns and reach out to affected users with preemptive remedies.
Cross-modal assistance: Combining voice, text, and structured data could create richer, more accessible support experiences.
Personalization at scale: Models that respect privacy constraints might adapt tone and content to individual preferences and histories.
Stronger human-model collaboration: Interfaces that present model suggestions as drafts or options could make agents faster and more effective.

These directions are promising, but they come with the responsibility to ensure fairness, privacy, and safety—so measured rollout and continuous governance will remain critical.

📊 Lessons we learned the hard way

No experiment at this scale is without setbacks. Here are some lessons we learned that may help others avoid common pitfalls:

Don’t underestimate localization complexity: Cultural nuances and region-specific regulations require custom rules and human review—one-size-fits-all prompts don’t cut it.
Quantitative metrics can miss nuance: Automated scores are essential, but they must be paired with human audits, especially for low-frequency failure modes.
Logging is non-negotiable: It’s tempting to skip thorough logging for performance reasons, but traceability is vital for debugging and compliance.
Establish clear ownership: AI outputs interact with legal, ops, support, and engineering domains—assign clear cross-functional owners for policy and escalation decisions.

These are practical organizational and engineering realities that shape whether an AI initiative succeeds at scale.

🌟 Impact on customers, partners, and the business

When implemented thoughtfully, generative models like GPT-5 can have a positive downstream impact on customers and partners:

Faster resolutions: Routine and semi-complex issues can be resolved more quickly, improving customer satisfaction.
Lower operational costs: Reductions in unnecessary agent escalations can reallocate human expertise to more complex tasks.
Higher partner retention: Better onboarding and in-context assistance can reduce churn among drivers and merchants.
Brand strengthening: Consistent, empathetic interactions can reinforce trust in the platform’s reliability.

Those outcomes align with our core mission: make the platform easier to use, more reliable, and more trustworthy for everyone who depends on it.

🔗 Closing thoughts and a call to action

Working with GPT-5 has given us a concrete example of how large language models can meaningfully improve conversational experience at scale. The key finding I emphasized is that structured critiques and consistent evaluation are the ingredients that enable models to scale responsibly. As I said in the video, this consistency is what allows us to scale across hundreds of millions of interactions, and it’s what makes each conversation feel more human and on brand.

If you’re building conversational AI, my call to action is straightforward:

Start with clear metrics tied to user outcomes.
Design hybrid systems that combine fast retrieval, deterministic policy, and selective use of powerful models for deep reasoning.
Invest in robust evaluation pipelines that mix automated scoring with human review.
Prioritize safety and governance from day one—don’t retrofit it later.
Iterate with small, measurable pilots and scale only after you’ve validated both quality and compliance.

These steps helped us take a promising model like GPT-5 from experimentation to a reliable component of our conversational stack. I’m optimistic about what’s next: better, faster, and safer interactions that respect users and improve outcomes across the platform.

📚 Appendix: Quick reference checklist for teams

Below is a compact checklist you can use when planning or evaluating a large-scale conversational AI deployment:

Define primary user outcome(s) (e.g., CSAT, resolution rate).
Create a rubric for automated evaluation (factuality, tone, actionability).
Implement hybrid routing (fast model vs. deep model).
Use retrieval-augmented generation for grounded answers.
Design deterministic policy checks to run post-generation.
Build human-in-the-loop review processes for edge cases and audits.
Log inputs, retrieval context, outputs, and final resolution for traceability.
Roll out in small cohorts and expand with monitoring and A/B testing.
Plan for localization and regional compliance from the outset.
Communicate escalation protocols clearly to customers and agents.

✅ Final note

My team’s work with GPT-5 is an example of how advanced generative models, combined with rigorous evaluation and prudent engineering, can help platforms like Uber balance speed and quality at scale. The experiments reaffirmed a principle I value deeply: technology must serve people thoughtfully, reliably, and safely. I hope this detailed report gives you practical insights you can apply to your own deployments, and encourages a measured, user-centered approach to building conversational AI.