Pay
$89K–$149K is typical
The middle half of what similar US postings state. This one doesn’t list pay.
From 21,366 postings in the last year. Stated ranges, not salaries paid.
San Francisco, California, United States
Actively hiring, first seen by us 4 hours agoCompensation & Benefits Base salary is $200,000 to $300,000 annually, with up to $350,000 for exceptional candidates. Additional commission-style bonus of up to $200,000 or more is tied to the value of delivered environments. Meaningful founding equity is included. Visa sponsorship is available. LocationRead the full posting
Work setup in the employer’s posting: On-siteFrom the employer’s posting
Employment type in the employer’s posting: FullTimeFrom the employer’s posting
Build evaluation environments and benchmarks that measure whether AI agents can perform real-world company work and meet or exceed human performance.
Design synthetic company environments with realistic email, records, and files for agents to use over extended periods.
Build graders and evaluation systems capable of assessing advanced agents, including cases where human reviewers may lack the relevant expertise.
Develop benchmarks that correlate with real-world production performance.
Build evaluation environments and benchmarks that measure whether AI agents can perform real-world company work and meet or exceed human performance. This founding engineering role combines agent evaluation research with production readiness, helping establish reliable evidence of agent capability.
What You'll Do Design synthetic company environments with realistic email, records, and files for agents to use over extended periods. Build graders and evaluation systems capable of assessing advanced agents, including cases where human reviewers may lack the relevant expertise.
Design synthetic company environments with realistic email, records, and files for agents to use over extended periods. Build graders and evaluation systems capable of assessing advanced agents, including cases where human reviewers may lack the relevant expertise. Develop benchmarks that correlate with real-world production performance.
Build graders and evaluation systems capable of assessing advanced agents, including cases where human reviewers may lack the relevant expertise. Develop benchmarks that correlate with real-world production performance. Find and address evaluation failure modes, including reward hacking and benchmark gaming.
From the employer’s posting.
Actively hiringFirst seen by us today, still listed, and not a repost of an earlier posting.
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
$89K–$149K is typical
The middle half of what similar US postings state. This one doesn’t list pay.
From 21,366 postings in the last year. Stated ranges, not salaries paid.
Day 0
Early. Similar jobs usually come down between day 8 and day 57.
From 39,461 similar postings that came down. Coming down is not always a hire.
16 open jobs
Open jobs by the week we first saw them, last 12 weeks.
Hiring recordAdd your resume to compare the role’s scope, tools and requirements with your experience.
Complete your application on jobs.ashbyhq.com. The employer’s form will show what is required.
Already applied? Track this application
Build evaluation environments and benchmarks that measure whether AI agents can perform real-world company work and meet or exceed human performance. This founding engineering role combines agent evaluation research with production readiness, helping establish reliable evidence of agent capability.
Design synthetic company environments with realistic email, records, and files for agents to use over extended periods.
Build graders and evaluation systems capable of assessing advanced agents, including cases where human reviewers may lack the relevant expertise.
Develop benchmarks that correlate with real-world production performance.
Find and address evaluation failure modes, including reward hacking and benchmark gaming.
Simulate software tools such as inboxes and CRMs with enough fidelity to support agent interactions at scale.
Hands-on research experience in synthetic data pipelines, benchmarks, evaluations, or model post-training; this is a research-oriented role, not a pure infrastructure engineering position.
Experience building environments or benchmarks used to train or evaluate agents at a frontier AI lab, and implementing graders or evaluation systems for AI agents.
At least 1 to 2 years at a fast-growing or AI-focused startup, plus mid-level or senior engineering or research experience.
Experience simulating software systems for agent tool use, identifying benchmark flaws, and detecting and fixing reward hacking in evaluation systems.
A degree from a highly selective university or an equivalent credential, such as founding a company or participating in a recognized founder program.
High-level competition experience in mathematics, programming, debate, or games is a plus.
Base salary is $200,000 to $300,000 annually, with up to $350,000 for exceptional candidates. Additional commission-style bonus of up to $200,000 or more is tied to the value of delivered environments. Meaningful founding equity is included. Visa sponsorship is available.
On-site five days per week in San Francisco, California, United States.
Selected passages from the saved posting. Check the full description for conditions and exceptions.
Compensation & Benefits Base salary is $200,000 to $300,000 annually, with up to $350,000 for exceptional candidates. Additional commission-style bonus of up to $200,000 or more is tied to the value of delivered environments. Meaningful founding equity is included. Visa sponsorship is available. Location
San Francisco, California, United States
Location On-site five days per week in San Francisco, California, United States.
Compensation & Benefits Base salary is $200,000 to $300,000 annually, with up to $350,000 for exceptional candidates. Additional commission-style bonus of up to $200,000 or more is tied to the value of delivered environments. Meaningful founding equity is included. Visa sponsorship is available. Location