Back to jobs

Senior / Staff AI Engineer — Evals & Model Quality

San Francisco · San Francisco, California, United States

Live, first seen by us 35 days ago
Pay
Pay not listed
Work setting
On-site — work setting source
Location type: On-site
From the employer’s posting
Employment
Full-time — employment source
Employment type: Full-time
From the employer’s posting
Team
Engineering — team source
Department: Engineering
From the employer’s posting

What you’ll work on

Full posting
  • Evaluate complex agent behavior: Design evaluations for multi-step agent trajectories, tool use, reasoning, and end-to-end task completion — not just individual LLM responses.

  • Build an end-to-end evaluation framework for FurtherAI agents, measuring success across multi-step workflows, tool calls, and complex insurance tasks.

  • Build automated grading systems: Develop LLM-as-judge evaluations, deterministic graders, human evaluation workflows, and rubrics that accurately reflect real-world quality.

From the employer’s posting
Build high-quality datasets: Develop golden datasets, benchmarks, and regression suites, working closely with Forward Deployed Engineers and domain experts to capture real-world insurance tasks and edge cases. Evaluate complex agent behavior: Design evaluations for multi-step agent trajectories, tool use, reasoning, and end-to-end task completion — not just individual LLM responses. Build automated grading systems: Develop LLM-as-judge evaluations, deterministic graders, human evaluation workflows, and rubrics that accurately reflect real-world quality.
Representative Projects Build an end-to-end evaluation framework for FurtherAI agents, measuring success across multi-step workflows, tool calls, and complex insurance tasks. Create a library of golden datasets and benchmarks for common insurance workflows, reusable across customer implementations and out-of-the-box AI skills.
Evaluate complex agent behavior: Design evaluations for multi-step agent trajectories, tool use, reasoning, and end-to-end task completion — not just individual LLM responses. Build automated grading systems: Develop LLM-as-judge evaluations, deterministic graders, human evaluation workflows, and rubrics that accurately reflect real-world quality. Make development evaluation-driven: Integrate offline and online evaluations into development and release processes so engineers can measure improvements, catch regressions, and ship confidently.

Before you apply

Sponsorship
Visa sponsorship not confirmed — sponsorship source
Do you require employer sponsorship to work in the US now or in the future?
From the employer’s application form

About FurtherAI

At FurtherAI, we’re building the AI agents that run insurance — a trillion-dollar industry still powered by manual work.

In the employer’s words · Read in context · Company website ↗

Apply at FurtherAI

Posting record

Report an error
Status in our records
Live
First seen by us
Oct 17, 202635 days ago
Recorded sightings
15
Last seen by us
Nov 21, 2026
Employer says posted
Oct 14, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 35

Up longer than most. Similar jobs usually come down between day 1 and day 15.

From 1,368 similar postings that came down. Coming down is not always a hire.

FurtherAI is hiring

1 open job

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
Where the posting names them
- Strong understanding of modern LLMs, agent architectures, evaluation methodologies, and model performance tradeoffs. - Strong Python engineering skills and the ability to build reliable, production-grade systems. - Ability to independently own ambiguous technical problems, from research and experimentation through production.

Your next step

Complete your application on jobs.ashbyhq.com. The employer’s form will show what is required.

Applying through Ashby

  • No account needed. “Autofill from resume” at the top of the form fills the key fields from your file.

Already applied? Track this application

Job description

View original posting ↗

About Us

At FurtherAI, we’re building the AI agents that run insurance — a trillion-dollar industry still powered by manual work. By combining frontier agentic AI with deep domain expertise, we’re reshaping how the industry’s most critical work gets done.

We’ve raised more than $30M from top investors (Andreessen Horowitz, YC, Nexus, South Park Commons), grown more than XY00% in the last 12 months, and work with some of the largest names in insurance — top 10 carriers/brokers, Fortune 100 companies, and public companies.

We are a deeply technical company. Both founders are engineers, and engineers have an outsized say on product direction.

Now, we’re looking for an exceptional AI Engineer to own how we evaluate and continuously improve the quality of AI agents across our platform.

Why Join Us

  • Outsized ambition: We’re out to build the definitive AI company for insurance — and we’re moving faster than anyone in the space.

  • Elite team: Founders and engineers from the best companies and schools in the world — ex-Apple AI Research, Brex, Coinbase, and Uber; 4 YC founders and 6 ex-founders; grads of Stanford, Berkeley, IITs, and Cornell.

  • Massive market impact: Insurance is the backbone of global commerce. Our work reshapes how the industry operates.

  • Founder’s mindset: Own big pieces of the product from day one — ideal if you want to start your own company one day.

What You’ll Do

  • Own evaluations across FurtherAI: Build the evaluation frameworks and infrastructure that measure the accuracy, quality, and reliability of our AI agents across workflows, Assistant, document processing, and computer use.

  • Build high-quality datasets: Develop golden datasets, benchmarks, and regression suites, working closely with Forward Deployed Engineers and domain experts to capture real-world insurance tasks and edge cases.

  • Evaluate complex agent behavior: Design evaluations for multi-step agent trajectories, tool use, reasoning, and end-to-end task completion — not just individual LLM responses.

  • Build automated grading systems: Develop LLM-as-judge evaluations, deterministic graders, human evaluation workflows, and rubrics that accurately reflect real-world quality.

  • Make development evaluation-driven: Integrate offline and online evaluations into development and release processes so engineers can measure improvements, catch regressions, and ship confidently.

  • Drive model evaluation and selection: Continuously benchmark frontier and open-weight models across accuracy, latency, cost, and reliability. Use results to inform model selection, upgrades, and production releases.

  • Build continuous improvement loops: Turn production failures, customer feedback, and agent traces into new evaluation cases and measurable improvements in agent quality.

Representative Projects

  • Build an end-to-end evaluation framework for FurtherAI agents, measuring success across multi-step workflows, tool calls, and complex insurance tasks.

  • Create a library of golden datasets and benchmarks for common insurance workflows, reusable across customer implementations and out-of-the-box AI skills.

  • Build an automated benchmarking system that evaluates new frontier and open-weight model releases and identifies the best models for different tasks.

  • Develop production evaluation pipelines that detect quality regressions, identify failure patterns, and turn them into actionable improvements.

What We’re Looking For

  • 5+ years of software engineering experience, with strong hands-on experience building production AI/ML systems.

  • Experience building evaluation frameworks, LLM-based applications, model benchmarking systems, or applied AI infrastructure.

  • Strong understanding of modern LLMs, agent architectures, evaluation methodologies, and model performance tradeoffs.

  • Strong Python engineering skills and the ability to build reliable, production-grade systems.

  • Ability to independently own ambiguous technical problems, from research and experimentation through production.

  • Clear communicator — both in person and in writing.

  • Bonus: Experience with LLM-as-judge, human evaluation systems, agent trajectory evaluations, or large-scale benchmarking.

  • Bonus: Experience with open-weight models, fine-tuning, post-training, or model routing.

  • Bonus: Background in B2B SaaS and 0-to-1 experience.

  • Above all: drive, grit, and ownership. If you excel here, other requirements are secondary.

At FurtherAI, we set a high bar. We’re not looking for someone who just wants a job. We’re looking for someone who wants to build something transformative. If you thrive in environments where the pace is fast, expectations are high, and the rewards are outsized, you’ll love it here.

You can look forward to the following benefits:

  • Fully covered, best-in-class health, dental, and vision benefits

  • Competitive compensation, meaningful stock options, company 401(k)

  • Unlimited PTO

  • Outstanding in-office culture in the heart of San Francisco, 5x per week

  • Lunch and dinner onsite

  • Team events, such as happy hours and off-sites

  • Pre-tax commuter benefits

Demand for AI Engineer roles

New AI Engineer postings at the employers we follow are up 340% over the last 6 weeks compared with the 6 before.

Weekly new AI Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 78, the most 803, and the latest week had 438.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under AI Engineer, by the work the posting describes.

AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.

Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.

An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

San Francisco, California, United States

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.