Back to jobs

Senior AI Evals Engineer (AI Quality & Safety) - Libra-Legal AI Assistant

Pay
Pay not listed
Work setting
Not stated
Employment
Full-time — employment source
Employment type in the employer’s posting: Full time
From the employer’s posting

What you’ll work on

Full posting
  • Build and own the eval platform: datasets, judges, harnesses, regression gates, cost and quality in one view, on Python, FastAPI and Langfuse.

  • Work out what a defensible measure looks like for each, going first on the ones nobody has evaluated before and then making them repeatable without you.

  • Own guardrails: ungrounded advice, invented or misattributed citations, jurisdiction and language leakage, prompt injection from ingested documents.

From the employer’s posting
What you'll do Build and own the eval platform: datasets, judges, harnesses, regression gates, cost and quality in one view, on Python, FastAPI and Langfuse. Make it self-service, so AI Engineers can evaluate and tune their own features without going through you, and eval-driven development becomes the most effective path rather than a tax. Map the quality landscape: good means something different for research, drafting, summarisation and retrieval, and again per jurisdiction. Work out what a defensible measure looks like for each, going first on the ones nobody has evaluated before and then making them repeatable without you.
Build and own the eval platform: datasets, judges, harnesses, regression gates, cost and quality in one view, on Python, FastAPI and Langfuse. Make it self-service, so AI Engineers can evaluate and tune their own features without going through you, and eval-driven development becomes the most effective path rather than a tax. Map the quality landscape: good means something different for research, drafting, summarisation and retrieval, and again per jurisdiction. Work out what a defensible measure looks like for each, going first on the ones nobody has evaluated before and then making them repeatable without you. Design the rubrics and set the standard for LLM-as-judge: turn Legal Engineers' and subject-matter experts' judgment into version-controlled criteria, keep judges recalibrated as models and jurisdictions change, and make authoring cheap enough to do at volume on privileged material.
Make cheaper models win: treat quality per euro as a first-class metric, instrumented per call, and find the configuration where a smaller model matches or beats an expensive one. Own guardrails: ungrounded advice, invented or misattributed citations, jurisdiction and language leakage, prompt injection from ingested documents. Design them, red-team them, prove they hold. What you'll bring Education

What you’ll bring

All qualifications

Core experience

  • Strong Python: FastAPI, modern tooling, and the data stack (pandas, numpy, notebooks), because much of this job is analysis.
  • Excellent communication in English.
  • Hands-on experience designing evaluations: datasets, rubrics, LLM-as-judge, benchmarking, human labelling.
Qualification wording
Strong Python: FastAPI, modern tooling, and the data stack (pandas, numpy, notebooks), because much of this job is analysis.
Excellent communication in English.
Hands-on experience designing evaluations: datasets, rubrics, LLM-as-judge, benchmarking, human labelling.

Benefits in the posting

Full benefits wording
  • Remote work & flexibility: Work remotely up to 3 days per week (home office) (= 8 days a month in the office) with flexible working hours.
  • Support for development: E-learning via LinkedIn, online language training with goFluent, and other training and development opportunities.
  • Company pension scheme.
  • A great start: structured onboarding with a buddy program.
  • Our Interview Practices

From the employer’s posting.

Apply at Wolters Kluwer

Posting record

Report an error

Actively hiringFirst seen by us 13 days ago, still listed, and not a repost of an earlier posting.

Status in our records
Live
First seen by us
Nov 5, 202613 days ago
Recorded sightings
12
Last seen by us
Nov 16, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 13

Similar jobs usually come down between day 2 and day 14.

At Wolters Kluwer, jobs usually come down by about day 36.

From 1,032 similar postings that came down, and 58 of Wolters Kluwer’s. Coming down is not always a hire.

Wolters Kluwer is hiring

7 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • Hiring at a steady pace
  • Takes jobs down at a usual pace
  • 14% of open jobs were reposted
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • FastAPI
  • NumPy
  • pandas
Where the posting names them
What you'll do - Build and own the eval platform: datasets, judges, harnesses, regression gates, cost and quality in one view, on Python, FastAPI and Langfuse. Make it self-service, so AI Engineers can evaluate and tune their own features without going through you, and eval-driven development becomes the most effective path rather than a tax. - Map the quality landscape: good means something different for research, drafting, summarisation and retrieval, and again per jurisdiction. Work out what a defensible measure looks like for each, going first on the ones nobody has evaluated before and then making them repeatable without you.
- Minimum 5 years in software engineering, at least 1 building LLM-powered products in production. - Strong Python: FastAPI, modern tooling, and the data stack (pandas, numpy, notebooks), because much of this job is analysis. - Hands-on experience designing evaluations: datasets, rubrics, LLM-as-judge, benchmarking, human labelling.

Your next step

Complete your application on wk.wd3.myworkdayjobs.com. The employer’s form will show what is required.

Applying through Workday

  • You create an account with each employer before you can upload anything. Every Workday employer has its own account, so one sign-up doesn’t carry over.
  • “Autofill with Resume” comes right after sign-in, before the My Information and My Experience steps. Check every field it fills in those steps, since that’s what recruiters see.
  • Your application status lives in that employer’s Candidate Home, under the same sign-in.

Already applied? Track this application

Job description

View original posting ↗

About Us
There are hundreds of thousands of lawyers across Europe, and at Libra we're
transforming how they work. Our AI platform combines deep legal reasoning
with cutting-edge generative technology, fundamentally changing how lawyers
research, draft, and deliver legal work.
We operate with the speed and ownership of a startup, while being backed by
the scale and stability of an established leader. Libra is an independent
business unit within the Legal & Regulatory division of Wolters Kluwer, a leading global provider of information, software, and services for professionals. For 180 years, Wolters Kluwer has supported and simplified the work of experts and organizations through innovative solutions, relying today on more than 20,000 colleagues worldwide to bring that vision to life.


About The Role
As an Evals Engineer (m/w/d) at Libra, you'll be the first dedicated hire on a new
team that owns how we measure quality: the datasets, rubrics, judges and
harnesses that decide whether an AI feature is good enough to ship, and the
automated optimization that runs against them.


Optimizing a system against a target is newly automatable: LLM optimizers like
GEPA now propose and test the candidates themselves. Nobody has to invent
the experiments any more, and the system will improve in whatever direction
the evals point, whether or not that's where you meant to go. That makes
defining what good means the highest-leverage work we do, and it has to be
done task by task, from scratch, for a legal AI platform at European scale.
You'll work in a lean, agile environment with AI Engineers, Legal Engineers and
Product, based at the vibrant Merantix AI Campus in Berlin, surrounded by a
community of AI innovators.


What you'll do

  • Build and own the eval platform: datasets, judges, harnesses, regression gates, cost and quality in one view, on Python, FastAPI and Langfuse. Make it self-service, so AI Engineers can evaluate and tune their own features without going through you, and eval-driven development becomes the most effective path rather than a tax.
  • Map the quality landscape: good means something different for research, drafting, summarisation and retrieval, and again per jurisdiction. Work out what a defensible measure looks like for each, going first on the ones nobody has evaluated before and then making them repeatable without you.
  • Design the rubrics and set the standard for LLM-as-judge: turn Legal Engineers' and subject-matter experts' judgment into version-controlled criteria, keep judges recalibrated as models and jurisdictions change, and make authoring cheap enough to do at volume on privileged material.
  • Deep-dive results and traces until you can say why something failed, then find the lever that moves it and automate the fix.
  • Unhobble the optimizer: instrument the app so prompts, hyperparameters and harness architecture become levers a search can safely pull, then run automated optimization (GEPA, DSPy) over them against a fitness function you trust, widening that surface as you go.
  • Make cheaper models win: treat quality per euro as a first-class metric, instrumented per call, and find the configuration where a smaller model matches or beats an expensive one.
  • Own guardrails: ungrounded advice, invented or misattributed citations, jurisdiction and language leakage, prompt injection from ingested documents. Design them, red-team them, prove they hold.

What you'll bring
Education

  • Bachelor's degree or equivalent in a relevant technical field (e.g. Computer Science, Software Engineering, Statistics, Data Science); advanced degree is a plus.

Experience

  • Minimum 5 years in software engineering, at least 1 building LLM-powered products in production.
  • Strong Python: FastAPI, modern tooling, and the data stack (pandas, numpy, notebooks), because much of this job is analysis.
  • Hands-on experience designing evaluations: datasets, rubrics, LLM-as-judge, benchmarking, human labelling.
  • Solid security and data-privacy practice. You'll handle traces, documents and datasets derived from privileged legal material.
  • AI coding agents (Claude Code, Codex, Cursor) in your daily workflow.
  • Genuine interest in the legal domain.
  • Bonus: automated prompt or pipeline optimization (GEPA, DSPy or similar), and a broader data science toolkit, e.g. embedding clustering to check dataset coverage.

Skills

  • A strong engineer who hasn't hand-written code in months. The architecture is yours, the typing isn't. You ship more working software than you ever did alone, reject code that runs but is shaped wrong, and leave less rework and cognitive debt behind you.
  • You think in systems and expect them to be gamed. The app, the evals, the optimizer and the people using them are one loop. Once evals gate releases, everything optimizes toward them, so some judgments stay human.
  • Hard to fool by a single number. "Could these all be within variance?" comes before any ranking, and a judge's reliability before you trust its grades. You report bounds, and retire a result that doesn't hold up, including your own.
  • You get expertise out of people who have no time to give you any, and build tools they choose to use. An eval nobody runs is worth nothing.
  • Runs on macro-management. You ask the sharp questions up front, then come back with options, their trade-offs and the assumptions behind each, and help pick. Good to think out loud with. Entrepreneurial, accountable, pragmatic.
  • Excellent communication in English.

What we offer:

  • Permanent employment from day one.
  • Remote work & flexibility: Work remotely up to 3 days per week (home office) (= 8 days a month in the office) with flexible working hours.
  • Work abroad flexibly: Work from anywhere within the EU for up to 20 days within a twelve-month period.
  • Rest & time off: 30 vacation days starting from 2027 (26 days for 2026).
  • Show your commitment: 1 additional day off per year for your volunteer work (Volunteer Day).
  • Support for development: E-learning via LinkedIn, online language training with goFluent, and other training and development opportunities.
  • Mobility on the go: JobRad leasing for the bicycle or e-bike of your choice.
  • Company pension scheme.
  • Well equipped for working from home: €300 budget for IT equipment
  • Modern on-site working: Attractive office in Berlin.
  • A great start: structured onboarding with a buddy program.

#LI-Hybrid

Our Interview Practices

To maintain a fair and genuine hiring process, we kindly ask that all candidates participate in interviews without the assistance of AI tools or external prompts. Our interview process is designed to assess your individual skills, experiences, and communication style. We value authenticity and want to ensure we’re getting to know you—not a digital assistant. To help maintain this integrity, we ask to remove virtual backgrounds and include in-person interviews in our hiring process. Please note that use of AI-generated responses or third-party support during interviews will be grounds for disqualification from the recruitment process.

Applicants may be required to appear onsite at a Wolters Kluwer office as part of the recruitment process.

About the role

We file this job under AI Engineer, by the work the posting describes.

AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.

Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.

An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

DEU - Berlin

- Permanent employment from day one. - Remote work & flexibility: Work remotely up to 3 days per week (home office) (= 8 days a month in the office) with flexible working hours. - Work abroad flexibly: Work from anywhere within the EU for up to 20 days within a twelve-month period.
More source context
- Well equipped for working from home: €300 budget for IT equipment - Modern on-site working: Attractive office in Berlin. - A great start: structured onboarding with a buddy program. #LI-Hybrid Our Interview Practices
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.