Most model pricing pages present inference costs as a straightforward function of token consumption. Production systems often behave differently.

While measuring real-world AI workloads, we found that actual costs were influenced as much by interaction patterns as by token counts. Context accumulation caused conversations to become progressively more expensive over time. Prompt caching produced savings until seemingly minor changes invalidated cache entries. Routing requests through lower-cost models sometimes increased overall spend because of downstream corrections and escalation paths.

This session examines the patterns that emerged as we instrumented and analysed production inference workloads. Using traces, cost measurements, cache statistics, and orchestration flows, we will explore how conversation structure, context growth, cache behaviour, routing decisions, and multi-model workflows influence the economics of operating AI systems.

Attendees will see examples where observed costs diverged substantially from initial estimates, where cache effectiveness differed dramatically from expectations, and where optimisation strategies produced counterintuitive results. Rather than presenting budgeting formulas or cost reduction techniques, the session focuses on building a more accurate mental model of the factors that determine what AI systems actually cost to operate in production.

What You Will Learn:

• How conversation structure and context growth influence inference costs over time
• The impact of cache behaviour, routing decisions, and multi-model workflows on overall spending
• Why observed production costs can differ significantly from initial estimates

Who Should Attend:

• AI engineers
• Platform engineers supporting AI workloads
• ML engineers and practitioners
• Technical leads responsible for AI systems
• Architects evaluating AI infrastructure and operating costs
• Engineering managers overseeing AI adoption