Back to jobs

Senior AI Engineer

Pay
Pay not listed
Work setting
On-site — work setting source
Work setup in the employer’s posting: On-site
From the employer’s posting
Employment
Full-time — employment source
Employment type in the employer’s posting: Full time
From the employer’s posting

What you’ll work on

Full posting
  • Develop and maintain LLM evaluation pipelines

  • Build automated regression tests for LLM applications

  • Build LLM-as-a-judge solutions to assess AI conversations

From the employer’s posting
Responsibilities Develop and maintain LLM evaluation pipelines Build LLM-as-a-judge solutions to assess AI conversations
Measure AI response quality and compare model performance Build automated regression tests for LLM applications Develop monitoring and observability for AI systems
Develop and maintain LLM evaluation pipelines Build LLM-as-a-judge solutions to assess AI conversations Create evaluation datasets and golden test sets

What you’ll bring

All qualifications

Core experience

  • Strong Python development
  • Experience with AI agents, tool calling and multi-turn conversations
  • Experience using AI development tools such as Cursor, Claude Code or GitHub Copilot
  • Experience improving LLM applications using production feedback
  • Hands-on experience evaluating LLM applications
  • SQL and experience working with large datasets
Qualification wording
Strong Python development
Experience with AI agents, tool calling and multi-turn conversations
Experience using AI development tools such as Cursor, Claude Code or GitHub Copilot
Experience improving LLM applications using production feedback
Hands-on experience evaluating LLM applications
SQL and experience working with large datasets

Benefits in the posting

Full benefits wording
  • Comprehensive healthcare

From the employer’s posting.

Apply at Madiff

Posting record

Report an error
Status in our records
Live
First seen by us
Oct 23, 202629 days ago
Recorded sightings
33
Last seen by us
Nov 21, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 29

Up longer than most. Similar jobs usually come down between day 1 and day 15.

At Madiff, jobs usually come down by about day 167.

From 1,376 similar postings that came down, and 56 of Madiff’s. Coming down is not always a hire.

Madiff is hiring

1 open job

Open jobs by the week we first saw them, last 12 weeks.

  • Hiring has slowed
  • Its jobs tend to stay up a long time
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • SQL
  • Docker
Where the posting names them
We are looking for a Senior AI Engineer to lead the development and optimisation of an LLM evaluation platform supporting a production AI-powered shopping assistant. The role focuses on building LLM evaluation pipelines, LLM-as-a-judge systems, golden datasets, and automated regression tests using Python, SQL, and LLM observability tools such as Langfuse, Braintrust, or Weave, while analysing real user conversations to identify hallucinations, quality issues, and potential risks. The engineer will take technical ownership of the evaluation platform, collaborate with product and engineering teams, and improve AI agents, prompt and context engineering, guardrails, and monitoring to ensure reliable, measurable, and continuously improving LLM performance. Responsibilities
• Minimum 2 years working with LLM systems in production • Strong Python development • Hands-on experience evaluating LLM applications
• LLM observability tools such as Langfuse, Braintrust, Weave or similar • SQL and experience working with large datasets • CI/CD, testing, Docker or other container technologies
• SQL and experience working with large datasets • CI/CD, testing, Docker or other container technologies • Cloud infrastructure experience

Your next step

Complete your application on madiffpl.zohorecruit.com. The employer’s form will show what is required.

Already applied? Track this application

Job description

View original posting ↗

We are looking for a Senior AI Engineer to lead the development and optimisation of an LLM evaluation platform supporting a production AI-powered shopping assistant. The role focuses on building LLM evaluation pipelines, LLM-as-a-judge systems, golden datasets, and automated regression tests using Python, SQL, and LLM observability tools such as Langfuse, Braintrust, or Weave, while analysing real user conversations to identify hallucinations, quality issues, and potential risks. The engineer will take technical ownership of the evaluation platform, collaborate with product and engineering teams, and improve AI agents, prompt and context engineering, guardrails, and monitoring to ensure reliable, measurable, and continuously improving LLM performance.

Responsibilities
• Develop and maintain LLM evaluation pipelines
• Build LLM-as-a-judge solutions to assess AI conversations
• Create evaluation datasets and golden test sets
• Analyse real customer conversations and identify recurring problems
• Measure AI response quality and compare model performance
• Build automated regression tests for LLM applications
• Develop monitoring and observability for AI systems
• Identify hallucinations, incorrect responses and unusual user behaviour
• Work on AI guardrails, adversarial testing and abuse detection
• Improve prompts, context management and agent workflows
• Work closely with product managers and engineering teams
• Take technical ownership of the evaluation platform


Requirements

Must have
• Minimum 5 years of software engineering experience
• Minimum 2 years working with LLM systems in production
• Strong Python development
• Hands-on experience evaluating LLM applications
• LLM-as-a-judge and evaluation frameworks
• Prompt engineering and context engineering
• Experience with AI agents, tool calling and multi-turn conversations
• LLM observability tools such as Langfuse, Braintrust, Weave or similar
• SQL and experience working with large datasets
• CI/CD, testing, Docker or other container technologies
• Cloud infrastructure experience
• Experience using AI development tools such as Cursor, Claude Code or GitHub Copilot
• Good English and Polish
Nice to have
• Human evaluation and calibration of LLM results
• Golden datasets and regression testing for generative AI
• Agent orchestration and automated workflows
• MCP and agent-to-agent integrations
• AI red-teaming, jailbreak testing and guardrails
• Experience improving LLM applications using production feedback
• E-commerce, search or recommendation systems
• Experience as a Technical Lead or Staff Engineer

Benefits

• Flexible hybrid mode (1–2 days per week in the office in Warsaw/Kraków)
• Solid, competitive salary
• Work in a multinational environment on international projects
• Comprehensive healthcare
• Long-term B2B contract with a stable project pipeline

Demand for AI Engineer roles

New AI Engineer postings at the employers we follow are up 340% over the last 6 weeks compared with the 6 before.

Weekly new AI Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 78, the most 803, and the latest week had 438.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under AI Engineer, by the work the posting describes.

AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.

Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.

An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Warsaw, Poland

Benefits • Flexible hybrid mode (1–2 days per week in the office in Warsaw/Kraków) • Solid, competitive salary
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.