Overview
Ramp’s engineering team shares lessons from building AI agents for finance operations, covering their journey from building hundreds of separate agents to consolidating into a single agent with thousands of skills. They detail how they built their policy agent, the infrastructure required, and the cultural shifts needed for AI-native product development.
Key Takeaways
- Start simple and iterate quickly - Begin with constrained problems like coffee expenses rather than trying to automate all of finance from day one
- Build your own ground truth dataset through cross-functional labeling sessions - Users are often wrong about policy decisions, so you need your own definition of correctness
- Consolidate to single agent architecture - Instead of building thousands of separate agents, focus on one agent with thousands of skills and shared toolboxes
- Design for auditability from the beginning - As systems become more complex and black-box-like, assume you only know inputs/outputs and ensure you can verify correctness
- Cultural shift required: focus on impact over coding - Teams that understand users, handle ambiguity, and obsess over experience will outperform those who debate libraries and bike-shed details
Topics Covered
- 0:00 - Introduction and Ramp Overview: Introduction to Ramp as finance platform, simple expense workflow example
- 3:30 - Paradigm Shift: From Many Agents to One: Lesson learned about consolidating from thousands of agents to single agent with many skills
- 8:00 - Policy Agent Deep Dive: Detailed walkthrough of how they built their most popular agent for expense policy enforcement
- 12:30 - Iteration Strategy and Starting Small: How they started with simple coffee expenses and gradually added complexity
- 15:30 - Building Ground Truth and Evaluation: Creating datasets, labeling sessions, and evaluation frameworks for agent performance
- 23:00 - AI Infrastructure and Internal Tools: Applied AI service, tool catalog, and internal systems for agent development
- 28:00 - Ramp Inspect: Internal Coding Agent: Their background coding agent that handles 50% of production PRs
- 32:00 - Cultural Shift and Future of Engineering: How AI changes what makes great engineering teams and the skills that matter most