Back to jobs

Senior Machine Learning Engineer, Evals

Amsterdam · Amsterdam, Noord-Holland, Netherlands

Actively hiring, first seen by us 3 days ago
Pay
Pay not listed
Work setting
Not stated
Employment
Full-time — employment source
Employment type: Full-time
From the employer’s posting
Team
AI & ML — team source

What you’ll work on

Full posting

Evaluation is the core of everything we do.

This work builds on ProLLM, the public benchmarking platform we ran for real-world use cases until our coding assistant benchmarks became saturated.

  • Design evaluations and success criteria with scientists and domain experts

  • Curate evaluation data and calibrate task difficulty

  • Own development and operation of the evaluation platform across checkpoints and deployed models

  • Calibrate automated graders against expert reviews and investigate disagreements

  • Analyse model outputs and agent trajectories to separate model failures from eval problems

From the employer’s posting
Evaluation is the core of everything we do. We build from the evals: we define how success will be measured before developing a capability, and use the results to guide training, model selection and deployment. We already have hundreds of evaluations spanning domain knowledge, commerce and gaming, with plans to expand their coverage, depth and scale.
This work builds on ProLLM, the public benchmarking platform we ran for real-world use cases until our coding assistant benchmarks became saturated. Since then it has been a private benchmarking platform for Prosus companies. We plan to open it again to publish our commerce benchmarks, covering both agentic and decision-making tasks.
Design evaluations with scientists and domain experts
Curate evaluation data and calibrate task difficulty
Own the development and operation of the evaluation platform across training checkpoints, deployed models and external baselines
Calibrate automated judgments against expert reviews, investigate disagreements
Distinguish model failures from problems in the evaluation

What you’ll bring

All qualifications

Core experience

  • At least 5 years of ML or software engineering experience
  • Experience owning evaluation frameworks, benchmarks or model-quality systems
  • Strong Python and PyTorch, Transformers or comparable ML libraries
  • Reliable production software development
  • Practical knowledge of experimental design and statistics

Preferred experience

  • Experience with language-model training or post-training.
  • Experience with agent or gaming environments.
  • Experience with human evaluation, LLM-as-judge calibration, recommendation metrics or online experimentation.
Qualification wording
At least five years of ML or software engineering experience
Experience owning evaluation frameworks, benchmarks or model-quality systems that informed training or deployment decisions
Strong Python and experience with PyTorch, Transformers or comparable ML libraries
alongside reliable production software development
Practical knowledge of experimental design and statistics
Experience with language-model training or post-training.
Experience with agent or gaming environments.
Experience with human evaluation, LLM-as-judge calibration, recommendation metrics or online experimentation.

Benefits in the posting

Full benefits wording
  • A base at our Amsterdam office and AI House, with relocation support. We also consider remote arrangements for candidates with particularly strong experience relevant to the role.
  • Our Diversity & Inclusion Commitment

From the employer’s posting.

About Prosus

We thrive with experimentation; we build, we test, we learn, we iterate.

In the employer’s words · Read in context · Company website ↗

Apply at Prosus

Posting record

Report an error

Actively hiringFirst seen by us 3 days ago, still listed, and not a repost of an earlier posting.

Status in our records
Live
First seen by us
Nov 17, 20263 days ago
Recorded sightings
4
Last seen by us
Nov 20, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 3

Early. Similar jobs usually come down between day 7 and day 76.

From 6,379 similar postings that came down. Coming down is not always a hire.

Prosus is hiring

5 open jobs

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • Hugging Face
  • PyTorch
Where the posting names them
- Experience owning evaluation frameworks, benchmarks or model-quality systems that informed training or deployment decisions. - Strong Python and experience with PyTorch, Transformers or comparable ML libraries, alongside reliable production software development. - Practical knowledge of experimental design and statistics, including sampling, uncertainty, repeated trials and comparisons across model versions.
- Direct collaboration with researchers, engineers and portfolio teams, with scope to publish methods and benchmarks, including through ProLLM, where the work can be shared. - Colleagues who have released Hugging Face models with more than 100 million downloads, authored papers at NeurIPS, created benchmarks and built multiple production AI systems. - Current ML frameworks and unlimited access to coding assistants.

Your next step

Complete your application on jobs.ashbyhq.com. The employer’s form will show what is required.

Applying through Ashby

  • No account needed. “Autofill from resume” at the top of the form fills the key fields from your file.

Already applied? Track this application

Job description

View original posting ↗

Who We Are

You've ordered dinner through Just Eat Takeaway. You've bought and sold on OLX. You've paid with PayU. Chances are, you've already used a Prosus company today.

We are Prosus, a global technology company powering some of the world's leading lifestyle ecommerce businesses. Every day, billions of people across India, Latin America, Europe and beyond use our products and services to buy, sell, pay, connect and discover.

Our AI-driven ecosystem spans 100+ companies across ecommerce, food delivery, fintech and experiences, serving billions of consumers worldwide. AI powers how we operate and innovate, with more than 60,000 active AI agents helping our teams build, scale and move faster. Through our investment team, we also back the founders and technologies shaping the future: From robotics and autonomous systems to synthetic biology and frontier AI.

Guided by our five values Innovation, Results, People, Impact, and Entrepreneurship, culture is our competitive advantage. We thrive with experimentation; we build, we test, we learn, we iterate.

Join us in shaping the future of technology at global scale.

About the team

Join the AI team at Prosus, a global technology group with businesses in food delivery, e-commerce, classifieds, travel and payments. We build LCM, the Large Commerce Model: our own foundation model, trained on billions of transactions and interactions from proprietary data across the portfolio. LCM already powers search, recommendations and marketing in production at iFood. We are extending it across our companies around the world and to businesses outside the group, while broadening its capabilities in agents and real-time prediction tasks. Your work will shape what LCM learns next and feed directly into products used by millions of people.

About the role

Evaluation is the core of everything we do. We build from the evals: we define how success will be measured before developing a capability, and use the results to guide training, model selection and deployment. We already have hundreds of evaluations spanning domain knowledge, commerce and gaming, with plans to expand their coverage, depth and scale.

This work builds on ProLLM, the public benchmarking platform we ran for real-world use cases until our coding assistant benchmarks became saturated. Since then it has been a private benchmarking platform for Prosus companies. We plan to open it again to publish our commerce benchmarks, covering both agentic and decision-making tasks.

We’re looking for a Senior Machine Learning Engineer to own the development of these evaluations and the platform that runs them. The work involves defining meaningful tasks, curating evaluation data, implementing reliable scoring, investigating model behaviour and making results useful for research and product decisions.

In our flat team of around 30 engineers, you’ll work directly with applied scientists, training engineers and domain experts. You’ll help teams contribute evaluations and build the infrastructure and automation needed to run them throughout model development.

In the first three months, we expect you to take technical ownership of part of the evaluation platform, deliver a set of new or improved evaluations with curated data and calibrated difficulty, and work with scientists to use them in a training or release decision.

What the role involves

  • Design evaluations with scientists and domain experts, translating intended capabilities into representative tasks, datasets and success criteria. Cover current applications and capabilities we want future models to acquire.

  • Curate evaluation data and calibrate task difficulty, so that evaluations are neither saturated nor out of reach and still separate models of different quality. Work with domain experts on sourcing, annotation and review, within data provenance, permitted use across companies and privacy requirements.

  • Own the development and operation of the evaluation platform across training checkpoints, deployed models and external baselines. Support distributed execution, reproducible configurations, trace inspection and comparison of results, while managing runtime and cost.

  • Develop programmatic checks, outcome-based scoring and LLM-based graders. Calibrate automated judgments against expert reviews, investigate disagreements and test whether graders reward superficial answers or reject valid solutions.

  • Analyse model outputs and agent trajectories, including tool calls, environment state and task outcomes. Distinguish model failures from problems in the evaluation, quantify uncertainty and explain what changed between versions.

  • Maintain versioned tasks, datasets, graders and environments, with protected held-out tests and contamination checks. Expand coverage as models improve, retaining useful regression tests and adding harder tasks where existing tests no longer distinguish performance.

  • Prepare commerce benchmarks for public release on ProLLM, including public mirror sets and evaluation code, where the data can be shared.

  • Integrate evaluations into training and release workflows, with dashboards and regression checks that support concrete decisions. Work with portfolio teams to establish whether offline improvements correspond to better outcomes in online experiments.

  • Build tools and agents that help generate candidate tasks, run evaluations and investigate failures, validating their outputs before adoption. Review contributions and mentor engineers so teams can extend the suite consistently.

Minimum qualifications

  • At least five years of ML or software engineering experience, including substantial hands-on work building and evaluating ML systems.

  • Experience owning evaluation frameworks, benchmarks or model-quality systems that informed training or deployment decisions.

  • Strong Python and experience with PyTorch, Transformers or comparable ML libraries, alongside reliable production software development.

  • Practical knowledge of experimental design and statistics, including sampling, uncertainty, repeated trials and comparisons across model versions.

  • Experience running inference or evaluation workloads at scale and debugging failures across models, data, scoring code and infrastructure.

  • Clear communication and the ability to turn ambiguous results into defensible recommendations.

Preferred qualifications

  • Experience with language-model training or post-training.

  • Experience with agent or gaming environments.

  • Experience with human evaluation, LLM-as-judge calibration, recommendation metrics or online experimentation.

  • Contributions to evaluation frameworks, public benchmarks or research.

What we offer

  • Ownership of the evaluation platform and the evaluations that guide LCM’s research, training and deployment.

  • Access to proprietary domain data, GPU infrastructure and external models for comparison.

  • Direct collaboration with researchers, engineers and portfolio teams, with scope to publish methods and benchmarks, including through ProLLM, where the work can be shared.

  • Colleagues who have released Hugging Face models with more than 100 million downloads, authored papers at NeurIPS, created benchmarks and built multiple production AI systems.

  • Current ML frameworks and unlimited access to coding assistants.

  • A base at our Amsterdam office and AI House, with relocation support. We also consider remote arrangements for candidates with particularly strong experience relevant to the role.

  • Competitive compensation and a top-spec MacBook Pro.

If you want to build the evaluations that shape what a foundation model learns, we’d like to hear from you. Please include examples of evaluations or platforms you have built, how you established that their results were reliable, and the decisions those results informed.

Why Prosus?

At Prosus, your colleagues include 60,000 AI agents… and your job is to figure out what that actually means. We're one of the largest operators of agentic AI on the planet, inside a $100B+ portfolio across 100+ companies and two billion people. The humans here own what AI can't do alone: judgment, creativity, and the call on what gets built next.

Own the impact. Work the ecosystem. Earn your growth. Build with backing.

Curious about our perks and benefits? Take a look at this page.

Our Diversity & Inclusion Commitment

We respect the dignity and human rights of individuals and communities wherever we operate in the world. Building an inclusive workplace where everyone feels welcome and can thrive is critical for us. We provide access to education, which helps everyone understand the important role they play and the positive impact they can have.

Want to know more about life at Prosus? Explore our culture and values.

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are down 4% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 508, the most 899, and the latest week had 508.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Amsterdam, Noord-Holland, Netherlands

- Current ML frameworks and unlimited access to coding assistants. - A base at our Amsterdam office and AI House, with relocation support. We also consider remote arrangements for candidates with particularly strong experience relevant to the role. - Competitive compensation and a top-spec MacBook Pro.
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.