Back to jobs

Senior/Staff AI Engineer, Quality & Evals

Berlin · Berlin, Germany

Live, first seen by us 38 days ago
Pay
Pay not listed
Work setting
On-site — work setting source
Location type: On-site
From the employer’s posting
Employment
Full-time — employment source
Employment type: Full-time
From the employer’s posting
Team
Engineering — team source
Department: Engineering
From the employer’s posting

What you’ll work on

Full posting
  • You will build the evaluation systems that power our multimodal retrieval agents and continuously improve critical quality metrics across current and future pipelines.

  • Build online and offline evaluation systems for LLM agents, including pipelines that use golden datasets, ground-truth data, human review workflows, and experiment results.

  • You’ll work at the edge of applied AI and information retrieval, building multimodal agentic pipelines and solving hard context and agent-harness engineering problems.

From the employer’s posting
We are looking for an engineer with strong backend, data, and AI systems experience to build the evaluation and observability foundation for production-grade LLM agents used in complex audit workflows. This role sits at the intersection of backend engineering, data infrastructure, and AI quality. You will build the evaluation systems that power our multimodal retrieval agents and continuously improve critical quality metrics across current and future pipelines. You’ll work at the edge of applied AI and information retrieval, building multimodal agentic pipelines and solving hard context and agent-harness engineering problems.
You will help build and operate the technical systems around our AI agents, with a focus on data infrastructure, evaluation, observability, and optimization. You will: Build online and offline evaluation systems for LLM agents, including pipelines that use golden datasets, ground-truth data, human review workflows, and experiment results. Create automated quality gates so changes to prompts, context, models, or agent logic can be tested before reaching production.
This role sits at the intersection of backend engineering, data infrastructure, and AI quality. You will build the evaluation systems that power our multimodal retrieval agents and continuously improve critical quality metrics across current and future pipelines. You’ll work at the edge of applied AI and information retrieval, building multimodal agentic pipelines and solving hard context and agent-harness engineering problems. This is not a traditional analytics, BI, or dashboarding role. You should expect to write production code, design data architecture, work inside backend systems, and directly improve the quality, cost, reliability, and performance of LLM-based agents.

About Cortea AI

We build side by side in our Berlin office.

In the employer’s words · Read in context · Company website ↗

Apply at Cortea AI

Posting record

Report an error
Status in our records
Live
First seen by us
Oct 11, 202638 days ago
Recorded sightings
32
Last seen by us
Nov 18, 2026
Employer says posted
Oct 8, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 38

Up longer than most. Similar jobs usually come down between day 2 and day 14.

From 1,002 similar postings that came down. Coming down is not always a hire.

Cortea AI is hiring

1 open job

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • BigQuery
  • ClickHouse
  • GCP
  • MLflow
Where the posting names them
You will fit into the role if you... - Have strong Python and/or backend engineering experience. - Have a solid understanding of how LLM and agent systems are evaluated—including deterministic checks, ground truth, LLM-as-judge, human review, and quality metrics—and can reason about when each approach is appropriate.
- Create automated quality gates so changes to prompts, context, models, or agent logic can be tested before reaching production. - Analyze large volumes of agent traces and executions in columnar and analytical databases such as BigQuery or ClickHouse to identify failure modes, quality regressions, latency issues, reliability gaps, and cost optimization opportunities. - Build reliable data retention and replay mechanisms for long-term analysis of production agent behavior.
- Have a solid understanding of how LLM and agent systems are evaluated—including deterministic checks, ground truth, LLM-as-judge, human review, and quality metrics—and can reason about when each approach is appropriate. - Have deployed and operated systems in the cloud, ideally on GCP. - Have hands-on experience building end-to-end retrieval or ML pipeline evaluation systems and using LLM observability or experimentation tools such as Braintrust, MLflow, Langfuse, or Weights & Biases.
- Have deployed and operated systems in the cloud, ideally on GCP. - Have hands-on experience building end-to-end retrieval or ML pipeline evaluation systems and using LLM observability or experimentation tools such as Braintrust, MLflow, Langfuse, or Weights & Biases. - Are comfortable working with analytical databases, data warehouses, columnar stores, and high-volume event or trace data.

Your next step

Complete your application on jobs.ashbyhq.com. The employer’s form will show what is required.

Applying through Ashby

  • No account needed. “Autofill from resume” at the top of the form fills the key fields from your file.

Already applied? Track this application

Job description

View original posting ↗

About us

We’re Cortea, a Berlin startup transforming audits with AI. Manual, document-heavy audits waste expert time while demand keeps rising. Our AI-powered software and specialized AI agents remove the repetitive work so auditors can focus on judgment.

Backed by top-tier VCs with >15m EUR funding, with a working product and paying customers, we’re rapidly scaling.

We value first-principles thinking, speed, trust, and kindness. We build side by side in our Berlin office.

Your Role

We are looking for an engineer with strong backend, data, and AI systems experience to build the evaluation and observability foundation for production-grade LLM agents used in complex audit workflows.

This role sits at the intersection of backend engineering, data infrastructure, and AI quality. You will build the evaluation systems that power our multimodal retrieval agents and continuously improve critical quality metrics across current and future pipelines.

You’ll work at the edge of applied AI and information retrieval, building multimodal agentic pipelines and solving hard context and agent-harness engineering problems.

This is not a traditional analytics, BI, or dashboarding role. You should expect to write production code, design data architecture, work inside backend systems, and directly improve the quality, cost, reliability, and performance of LLM-based agents.

What you’ll do

You will help build and operate the technical systems around our AI agents, with a focus on data infrastructure, evaluation, observability, and optimization. You will:

  • Build online and offline evaluation systems for LLM agents, including pipelines that use golden datasets, ground-truth data, human review workflows, and experiment results.

  • Create automated quality gates so changes to prompts, context, models, or agent logic can be tested before reaching production.

  • Analyze large volumes of agent traces and executions in columnar and analytical databases such as BigQuery or ClickHouse to identify failure modes, quality regressions, latency issues, reliability gaps, and cost optimization opportunities.

  • Build reliable data retention and replay mechanisms for long-term analysis of production agent behavior.

  • Manage observability tools for tracing, monitoring, debugging, and experiment management of our audit agents.

  • Team up with backend engineers to improve the speed and reliability of our retrieval and reasoning agents.

You will fit into the role if you...
  • Have strong Python and/or backend engineering experience.

  • Have a solid understanding of how LLM and agent systems are evaluated—including deterministic checks, ground truth, LLM-as-judge, human review, and quality metrics—and can reason about when each approach is appropriate.

  • Have deployed and operated systems in the cloud, ideally on GCP.

  • Have hands-on experience building end-to-end retrieval or ML pipeline evaluation systems and using LLM observability or experimentation tools such as Braintrust, MLflow, Langfuse, or Weights & Biases.

  • Are comfortable working with analytical databases, data warehouses, columnar stores, and high-volume event or trace data.

  • Understand system design, reliability, observability, monitoring, logging, debugging, and operational trade-offs.

  • Bring senior-level engineering judgment: you can make architectural decisions, communicate trade-offs, and build systems that other engineers can extend.

  • Are comfortable with ambiguity, able to reason from first principles, and excited to build infrastructure for AI systems that are actively used in production.

Nice-to-haves that are a plus:

  • Designing data pipelines, ETL/ELT workflows, event-processing systems, or feedback loops for production data.

  • Building infrastructure around LLM-based products or agentic systems, including optimizing LLM usage, context windows, reasoning tokens, or model selection.

  • Working with production traces from complex distributed systems.

  • Building internal platforms for engineers, domain experts, or operations teams.

  • Using workflow orchestration systems such as Temporal or similar.

  • Familiarity with audit, finance, compliance, or other high-accuracy domains.

  • Experience in an early-stage startup or fast-moving engineering environment.

No-one checks every box. If you’ve shipped retrieval systems and like owning evaluations and pipelines, let’s talk.

What we offer
  • High impact & growth: Shape strategy at a scaling AI startup from day one.

  • Mission-driven culture: Ambitious team valuing first-principles thinking and bold ideas.

  • Attractive compensation: competitive salary plus significant equity.

  • Best tools for the job: Generous coding tools budget so you can use the best tools for the job.

  • Startup perks: Flexible vacation, team lunches, retreats, central Berlin office.

Interview process
  • First Call — Intro to Cortea with Liza

  • Second Call — Technical interview with Vlad

  • Third Call — Deep dive into our culture with our Co-Founder Philipp

  • On-site Day (Berlin) — Meet the team and work on a real problem together

We’re an equal-opportunity team and encourage women and underrepresented groups to apply.

About the role

We file this job under AI Engineer, by the work the posting describes.

AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.

Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.

An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Berlin, Germany

- Third Call — Deep dive into our culture with our Co-Founder Philipp - On-site Day (Berlin) — Meet the team and work on a real problem together We’re an equal-opportunity team and encourage women and underrepresented groups to apply.
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.