Back to jobs

Staff AI Engineer - Agent Evaluation Platform

Paris · Levallois-Perret, Ile-de-France, France

Actively hiring, first seen by us 2 days ago
Pay
Pay not listed
Work setting
Hybrid — work setting source
Location type: Hybrid
From the employer’s posting
Employment
Full-time · Permanent — employment source
Job details Permanent position Full Time
Permanent position Full Time Workplace: Levallois
Employment type: Full-time
From the employer’s posting
Team
AI — team source

What you’ll work on

Full posting
  • Design, ship and operate production-grade evaluation tooling used across AI product teams

  • Design evaluation strategies for agentic systems mixing checks, LLM judges, simulation and human review

  • Build capabilities for production sampling, distribution-shift detection and test case generation

  • Help product teams define quality metrics, representative datasets and reliable quality gates

  • Turn recurring product needs into reusable, self-service platform capabilities

From the employer’s posting
Design, ship, and operate production-grade evaluation tooling (APIs, services, CI/CD integrations)
Design evaluation strategies for agentic systems, combining deterministic checks, LLM-as-a-judge, simulation-based testing, and human review
Build intelligent capabilities for production sampling, dataset representativeness monitoring, distribution-shift detection, and test case generation
Help teams define meaningful quality metrics, representative datasets, and reliable quality gates throughout the development lifecycle
Turn recurring product needs into reusable, self-service platform capabilities

Benefits in the posting

Full benefits wording
  • Free comprehensive health insurance (basic package) for you and your children
  • 25 days of paid vacation per year, plus up to 14 days of RTT
  • Free mental health and coaching services through our partner Moka.care
  • ParentCare Program: Enjoy full salary coverage (100%) during your first month of birth leave, and 75% during the second, covered by Doctolib
  • For caregivers and workers with disabilities, a package including an adaptation of the remote policy, extra days off for medical reasons, and psychological support
  • The interview process
  • A first interview with a member of the Talent Acquisition team
  • A coding interview
  • An agentics system design
  • A behavioural interview
  • Reference check
  • Job details
  • Permanent position
  • Full Time

From the employer’s posting.

Apply at Doctolib

Posting record

Report an error

Actively hiringFirst seen by us 2 days ago, still listed, and not a repost of an earlier posting.

Status in our records
Live
First seen by us
Nov 19, 20262 days ago
Recorded sightings
3
Last seen by us
Nov 21, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 2

Similar jobs usually come down between day 1 and day 15.

At Doctolib, jobs usually come down by about day 39.

From 1,368 similar postings that came down, and 15 of Doctolib’s. Coming down is not always a hire.

Doctolib is hiring

16 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • AI Engineer hiring is picking up
  • Its jobs tend to stay up a long time
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
Where the posting names them
Before you read on, if you don't have the exact profile described below, but you feel this job description matches your skill set, we still encourage you to apply! You could be our next team mate if you: - Have 6+ years of experience as a software engineer, with significant experience using Python; or as a machine learning engineer - Have built or operated evaluation for LLM-powered or agentic systems: datasets, LLM-as-a-judge, human review, production-driven evaluation

Your next step

Complete your application on jobs.ashbyhq.com. The employer’s form will show what is required.

Applying through Ashby

  • No account needed. “Autofill from resume” at the top of the form fills the key fields from your file.

Already applied? Track this application

Job description

View original posting ↗

Join a team of passionate and hardworking entrepreneurs to transform healthcare!

 

Working in the tech team at Doctolib involves building innovative products and features to improve the daily lives of care teams and patients. We work in feature teams in an agile environment, while collaborating with product, design, and business teams.

 

What you'll do

 

We are looking for a Staff AI Software Engineer to join the AI Evaluation team (EvalX) at Doctolib and help every AI product team measure, understand and improve the quality, reliability and safety of their systems.

 

Our AI products span patient-facing conversational agents, AI-powered phone assistants and clinical assistants supporting healthcare professionals. EvalX builds the platform, tools and methods that let these teams evaluate continuously, and our ambition is to make evaluation a core engineering practice.

 

In this role, you will work at the intersection of platform engineering, applied AI and product teams: you build shared evaluation capabilities, and you work directly with the teams that use them so they actually get adopted.

 

In this role, you will:

  • Design, ship, and operate production-grade evaluation tooling (APIs, services, CI/CD integrations) used by AI product teams across Doctolib

  • Work hands-on with product teams to assess how they evaluate today, define improvement plans, and help them own their evaluation quality

  • Design evaluation strategies for agentic systems, combining deterministic checks, LLM-as-a-judge, simulation-based testing, and human review

  • Build intelligent capabilities for production sampling, dataset representativeness monitoring, distribution-shift detection, and test case generation

  • Help teams define meaningful quality metrics, representative datasets, and reliable quality gates throughout the development lifecycle

  • Turn recurring product needs into reusable, self-service platform capabilities, and measure success through adoption, team autonomy, and evaluation coverage

  • Contribute to cross-functional initiatives that shape Doctolib's AI platform and best practices for building and evaluating agentic systems

 

Who you are

 

Before you read on, if you don't have the exact profile described below, but you feel this job description matches your skill set, we still encourage you to apply! You could be our next team mate if you:

  • Have 6+ years of experience as a software engineer, with significant experience using Python; or as a machine learning engineer

  • Have built or operated evaluation for LLM-powered or agentic systems: datasets, LLM-as-a-judge, human review, production-driven evaluation

  • Understand and promote software development best practices

  • Are product-oriented and enjoy working directly with other teams, challenging assumptions, and turning complex needs into concrete solutions

  • Can influence engineering practices and drive change without hierarchical authority: your job is not done until your solution is adopted

  • Are comfortable speaking and writing in English in your day-to-day with the team

 

Now, it would be fantastic if you:

  • Have experience with applied ML techniques such as embeddings, clustering, statistical sampling, and distribution monitoring

  • Have experience building developer platforms or internal tooling, or in a Forward Deployed Engineering role

  • Have experience in the medical domain or another regulated environment

 

What we offer

  • Free comprehensive health insurance (basic package) for you and your children

  • 25 days of paid vacation per year, plus up to 14 days of RTT

  • Free mental health and coaching services through our partner Moka.care

  • Work from abroad for up to 10 days per year thanks to our flexibility days policy

  • Lunch vouchers (Swile card) worth €8.50 per working day, with €4.50 covered by Doctolib

  • A subsidy from the work council to refund part of the membership to a sport club or a creative class

  • 50% reimbursement of your public transport subscription

  • Enrollment in Doctolib's long-term employee value sharing plan called DoctoGrowth

  • ParentCare Program: Enjoy full salary coverage (100%) during your first month of birth leave, and 75% during the second, covered by Doctolib

  • For caregivers and workers with disabilities, a package including an adaptation of the remote policy, extra days off for medical reasons, and psychological support

 

The interview process

  • A first interview with a member of the Talent Acquisition team

  • A coding interview

  • An agentics system design

  • A behavioural interview

  • Reference check

 

Job details

  • Permanent position

  • Full Time

  • Workplace: Levallois

  • Start date : As soon as possible

 

If you would like to find out more about tech life at Doctolib, feel free to read our latest Medium blog articles!

 

At Doctolib, we believe in improving access to healthcare for everyone - regardless of where you come from, what you look like. This translates into our recruitment process: Doctolib is an equal opportunity employer. We don't just accept diversity at Doctolib, we respect and celebrate it!

 

The more diverse ideas are heard, the more our product will truly improve healthcare for all. You are welcome to apply to Doctolib, regardless of your gender, religion, age, sexual orientation, ethnicity, disability, or place of origin. If you have a disability, let us know if there's any way we can make the interview process smoother for you!

 

All the information transmitted via this form is processed by Doctolib for the purpose of managing applications. For more information on how Doctolib processes your application data, click here.

 

If you wish to exercise your rights or if you have any questions about the processing of your data, you can write to us at hr.dataprivacy@doctolib.com.

Demand for AI Engineer roles

New AI Engineer postings at the employers we follow are up 340% over the last 6 weeks compared with the 6 before.

Weekly new AI Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 78, the most 803, and the latest week had 438.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under AI Engineer, by the work the posting describes.

AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.

Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.

An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Levallois-Perret, Ile-de-France, France

- ParentCare Program: Enjoy full salary coverage (100%) during your first month of birth leave, and 75% during the second, covered by Doctolib - For caregivers and workers with disabilities, a package including an adaptation of the remote policy, extra days off for medical reasons, and psychological support The interview process
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.