About the role
About the Role
Join a small, senior team building autonomous AI agents that handle real executive workflows. You will develop agent capabilities and the evaluation systems needed to verify reliability before changes reach users.
What You'll Do
- Build agents that complete delegated work end to end without requiring users to check every step.
- Design reliable measures of agent success and maintain representative evaluations for real-world tasks.
- Develop new agent capabilities for tools executives already use, and verify their readiness before launch.
- Investigate production failures, identify root causes, and deliver fixes that prevent recurrence.
- Build scalable failure detection for known issues and unexpected patterns using evaluation and monitoring tools.
- Improve agent recovery when a tool or website fails.
- Create repeatable workflows so engineers can measure whether agent changes improve performance.
What We're Looking For
- At least 3 years in data science or machine learning roles building or delivering evaluation systems, metrics, or product quality measurement.
- Production-level Python and SQL skills for analysis, large datasets, and automated pipelines.
- Experience designing evaluation frameworks, including success criteria, representative datasets, appropriate metrics, and trustworthy benchmarks.
- Experience using statistics, experiment design, and realistic end-to-end simulations to evaluate agent changes.
- Background building or operating agents for real users and improving their reliability; experience with post-training, fine-tuning, or reinforcement learning is valuable.
- Experience owning and shipping systems to users, with demonstrated impact; familiarity with Django, React, and TypeScript.
- A track record of sustained contribution and growth in a role, founding a startup, or earning a promotion.
Compensation & Benefits
Base salary: $200,000 to $300,000 annually. Visa sponsorship is not available.
Location
On-site in Palo Alto, California, with four days per week in the office.