Back to jobs

Member of Technical Staff, ML Platform (Evals)

Pay
Pay not listedSimilar roles state $166K–$237K
Work setting
Remote — work setting source
Location type: Remote
From the employer’s posting
Employment
Full-time — employment source
Employment type: Full-time
From the employer’s posting
Team
Engineering — team source
Department: Engineering
From the employer’s posting

What you’ll work on

Full posting

We're looking for an ML infrastructure engineer to own model evaluation at Runway, end to end.

Your job will be to design and build the systems that generate samples at scale, score them with automated metrics and human annotations, track results across checkpoints and releases, and put the answers in front of researchers in minutes rather than days.

  • You'll define what "better" means operationally – how we measure it, how confident we are, and how a result becomes a ship/no-ship decision.

  • Own the evaluation platform end to end: the tooling and systems required to generate, annotate, review and adapt at frontier scale

  • Define the CLIs, APIs, GUIs and storage layers required to make this process seamless, fast, sophisticated and collaborative

From the employer’s posting
We're looking for an ML infrastructure engineer to own model evaluation at Runway, end to end. Every decision we make about a model – which checkpoint to keep training, what to ship to millions of users, which datamix and architecture shows the most promise – rests on evals. Today that work is spread across the organization. You'll turn it into one platform.
Your job will be to design and build the systems that generate samples at scale, score them with automated metrics and human annotations, track results across checkpoints and releases, and put the answers in front of researchers in minutes rather than days. You'll define what "better" means operationally – how we measure it, how confident we are, and how a result becomes a ship/no-ship decision.
We're looking for an ML infrastructure engineer to own model evaluation at Runway, end to end. Every decision we make about a model – which checkpoint to keep training, what to ship to millions of users, which datamix and architecture shows the most promise – rests on evals. Today that work is spread across the organization. You'll turn it into one platform. Your job will be to design and build the systems that generate samples at scale, score them with automated metrics and human annotations, track results across checkpoints and releases, and put the answers in front of researchers in minutes rather than days. You'll define what "better" means operationally – how we measure it, how confident we are, and how a result becomes a ship/no-ship decision. You'll be embedded within research teams as a member of ML Platform, and you'll set the technical direction for evals across the company. This is an extremely high-leverage role: the quality of our models is bounded by how well we can measure them.
What you'll do Own the evaluation platform end to end: the tooling and systems required to generate, annotate, review and adapt at frontier scale Define the CLIs, APIs, GUIs and storage layers required to make this process seamless, fast, sophisticated and collaborative
Own the evaluation platform end to end: the tooling and systems required to generate, annotate, review and adapt at frontier scale Define the CLIs, APIs, GUIs and storage layers required to make this process seamless, fast, sophisticated and collaborative Work directly with research teams on video, image, audio, agents, and robotics to understand what they need to measure and build it, then generalize the result into the platform

What you’ll bring

All qualifications

Core experience

  • 5+ years of experience building ML infrastructure or data platforms in production environments, with at least some of that time spent on evaluation, experimentation, or benchmarking systems
  • Experience with online experimentation platforms and connecting offline metrics to product outcomes
  • Strong Python and PyTorch, and hands-on experience running large batch GPU workloads on Kubernetes
  • Experience designing data pipelines and storage for large volumes of media or model outputs, with attention to versioning and reproducibility
  • Familiarity with experimental statistics: paired comparisons, confidence intervals, multiple-comparison pitfalls, inter-rater agreement
  • Ability to lead a broad technical area: gather requirements from many teams, set direction, make tradeoffs, and drive a roadmap without waiting to be told
Qualification wording
5+ years of experience building ML infrastructure or data platforms in production environments, with at least some of that time spent on evaluation, experimentation, or benchmarking systems
Experience with online experimentation platforms and connecting offline metrics to product outcomes
Strong Python and PyTorch, and hands-on experience running large batch GPU workloads on Kubernetes
Experience designing data pipelines and storage for large volumes of media or model outputs, with attention to versioning and reproducibility
Familiarity with experimental statistics: paired comparisons, confidence intervals, multiple-comparison pitfalls, inter-rater agreement
Ability to lead a broad technical area: gather requirements from many teams, set direction, make tradeoffs, and drive a roadmap without waiting to be told
Apply at Runway

Posting record

Report an error

Actively hiringFirst seen by us 9 days ago, still listed, and not a repost of an earlier posting.

Status in our records
Live
First seen by us
Nov 12, 20269 days ago
Recorded sightings
11
Last seen by us
Nov 22, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Pay

$166K–$237K is typical

The middle half of what similar US postings state. This one doesn’t list pay.

From 1,537 postings in the last year. Stated ranges, not salaries paid.

Time since we first saw it

Day 9

Similar jobs usually come down between day 7 and day 76.

From 6,389 similar postings that came down. Coming down is not always a hire.

Runway is hiring

2 open jobs

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • Kubernetes
  • PyTorch
Where the posting names them
A peek at our technical stack Our inference and training code is written in Python (PyTorch) and is cloud-native. We leverage multiple clouds and use a combination of Kubernetes native and home-grown tooling for efficient job orchestration. We’re big users of agentic development and operations. We have a robust internal platform and provide broad access to the latest models and harnesses.
- 5+ years of experience building ML infrastructure or data platforms in production environments, with at least some of that time spent on evaluation, experimentation, or benchmarking systems - Strong Python and PyTorch, and hands-on experience running large batch GPU workloads on Kubernetes - Experience designing data pipelines and storage for large volumes of media or model outputs, with attention to versioning and reproducibility

Your next step

Complete your application on jobs.ashbyhq.com. The employer’s form will show what is required.

Applying through Ashby

  • No account needed. “Autofill from resume” at the top of the form fills the key fields from your file.

Already applied? Track this application

Job description

View original posting ↗

We are building AI to simulate the world through merging art and science.

We believe that world models are at the frontier of progress in artificial intelligence. Language models alone won’t solve the world’s hardest problems – robotics, disease, scientific discovery. Real progress requires models that experience the world and learn from their mistakes, the same way that humans do. And this kind of trial and error can be massively accelerated when done in simulation, rather than in the real world.

World models offer the most clear path to general-purpose simulation, changing how stories are told, how scientific progress is made and how the next frontiers of humanity are reached.

Our team consists of creative, open minded, caring and ambitious people who are determined to change the world. We aspire to continuously build impossible things and our ability to do so relies on building an incredible team. If you are driven to do the same, we'd love to hear from you.

*Titles posted to our careers page are updated regularly to best advertise the role and expected seniority, but internally we have a flat titling system with two types of titles: one for individual contributors and one for people managers.

About the role

We're looking for an ML infrastructure engineer to own model evaluation at Runway, end to end. Every decision we make about a model – which checkpoint to keep training, what to ship to millions of users, which datamix and architecture shows the most promise – rests on evals. Today that work is spread across the organization. You'll turn it into one platform.

Your job will be to design and build the systems that generate samples at scale, score them with automated metrics and human annotations, track results across checkpoints and releases, and put the answers in front of researchers in minutes rather than days. You'll define what "better" means operationally – how we measure it, how confident we are, and how a result becomes a ship/no-ship decision.

You'll be embedded within research teams as a member of ML Platform, and you'll set the technical direction for evals across the company. This is an extremely high-leverage role: the quality of our models is bounded by how well we can measure them.

A peek at our technical stack

Our inference and training code is written in Python (PyTorch) and is cloud-native. We leverage multiple clouds and use a combination of Kubernetes native and home-grown tooling for efficient job orchestration.

We’re big users of agentic development and operations. We have a robust internal platform and provide broad access to the latest models and harnesses.

You’ll have access to best-in-class models, agents, GPUs, storage and cloud services you need to drive a world class evaluation pipeline that helps us deliver frontier models.

What you'll do

  • Own the evaluation platform end to end: the tooling and systems required to generate, annotate, review and adapt at frontier scale

  • Define the CLIs, APIs, GUIs and storage layers required to make this process seamless, fast, sophisticated and collaborative

  • Work directly with research teams on video, image, audio, agents, and robotics to understand what they need to measure and build it, then generalize the result into the platform

  • Help set the standards for how we evaluate models at Runway: reproducibility, metric definitions, reporting formats, and when a result is trustworthy enough to act on

  • Support broad adoption of the platform and its integration throughout Runway’s research efforts across training, production model serving and more

  • Contribute broadly as a member of the ML Platform team to tools and systems that help Runway train and serve frontier models

What you'll need

  • 5+ years of experience building ML infrastructure or data platforms in production environments, with at least some of that time spent on evaluation, experimentation, or benchmarking systems

  • Strong Python and PyTorch, and hands-on experience running large batch GPU workloads on Kubernetes

  • Experience designing data pipelines and storage for large volumes of media or model outputs, with attention to versioning and reproducibility

  • Familiarity with experimental statistics: paired comparisons, confidence intervals, multiple-comparison pitfalls, inter-rater agreement

  • Comfort building internal tools end to end, from the command line to the browser

  • Ability to lead a broad technical area: gather requirements from many teams, set direction, make tradeoffs, and drive a roadmap without waiting to be told

  • Familiarity with the full model development lifecycle: data, training, evaluation, serving

  • Self-starter who can work embedded with research teams and move fast

  • Strong systems thinking and pragmatic approach to production reliability

  • Humility and open mindedness; at Runway we love to learn from one another

Nice to have

  • Track record of building evaluation suites for generative models

  • Hands-on work with LLM- or VLM-as-judge pipelines

  • Experience with online experimentation platforms and connecting offline metrics to product outcomes

  • Prior work evaluating agents or robotics policies

Runway strives to recruit and retain exceptional talent from diverse backgrounds while ensuring pay equity for our team. Our salary ranges are based on competitive market rates for our size, stage and industry, and salary is just one part of the overall compensation package we provide.

There are many factors that go into salary determinations, including relevant experience, skill level and qualifications assessed during the interview process, and maintaining internal equity with peers on the team. The range shared below is a general expectation for the function as posted, but we are also open to considering candidates who may be more or less experienced than outlined in the job description. In this case, we will communicate any updates in the expected salary range.

Lastly, the provided range is the expected salary for candidates in the U.S. Outside of those regions, there may be a change in the range, which again, will be communicated to candidates.

Working at Runway

Great things come from great teams. We’d love to hear from you.

We’re committed to creating a space where our employees can bring their full selves to work and have equal opportunity to succeed. So regardless of race, gender identity or expression, sexual orientation, religion, origin, ability, age, veteran status, if joining this mission speaks to you, we encourage you to apply.

More about Runway

We're excited to be recognized as a best place to work:

Crain's | InHerSight | BuiltIn NYC | INC

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are down 4% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 508, the most 899, and the latest week had 508.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

United States

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.