Build agent quality that scales across model families
Coding-agent quality depended on the model, context management, tool execution, and customer environment. Benchmark results alone could not tell us where to invest.
I made the call to prioritize context compaction after reviewing failure patterns and trade-offs with Engineering, focusing investment on shared agent reliability.
Shipped context-compaction and tool-result handling improvements; task completion improved during the quality work. The paid-user growth above reflects broader quality, onboarding, and adoption improvements.
My role, trade-offs, and measurement
As Product DRI for Duo Developer, I worked with Engineering to distinguish model limitations from context, tool, and execution issues. I made the prioritization call for compaction after reviewing the evidence, rather than relying only on more available context or future model improvements. The trade-off was investing in shared context management while protecting information that longer coding tasks depend on.
I tracked task completion weekly and used SWE-bench Pro and Verified alongside latency, costs to us and customers, production telemetry, and developer feedback. Real repositories and execution environments could reveal problems that standardized benchmarks missed.
My focus was the product judgment behind the investment: where shared infrastructure would improve developer outcomes, and how to keep learning as models and production needs evolved.
I evaluated compaction frequency and cost when deciding how to support it across deployment models, prioritizing a consistent implementation over deployment-specific forks. This connected agent reliability with operating cost and implementation complexity.
Related decision: routing economics and billing
I influenced the direction toward token-based or hybrid billing rather than pure request-based pricing. My contribution connected model selection and task routing with quality, latency, and inference economics. A request alone could hide substantial differences in the work an agent performed; the pricing discussion needed to account for that variability.