Pay
$166K–$237K is typical
The middle half of what similar US postings state. This one doesn’t list pay.
From 1,537 postings in the last year. Stated ranges, not salaries paid.
United States
Actively hiring, first seen by us 9 days agoLocation type: RemoteFrom the employer’s posting
Employment type: Full-timeFrom the employer’s posting
Department: EngineeringFrom the employer’s posting
We're looking for an ML infrastructure engineer to own model evaluation at Runway, end to end.
Your job will be to design and build the systems that generate samples at scale, score them with automated metrics and human annotations, track results across checkpoints and releases, and put the answers in front of researchers in minutes rather than days.
You'll define what "better" means operationally – how we measure it, how confident we are, and how a result becomes a ship/no-ship decision.
Own the evaluation platform end to end: the tooling and systems required to generate, annotate, review and adapt at frontier scale
Define the CLIs, APIs, GUIs and storage layers required to make this process seamless, fast, sophisticated and collaborative
We're looking for an ML infrastructure engineer to own model evaluation at Runway, end to end. Every decision we make about a model – which checkpoint to keep training, what to ship to millions of users, which datamix and architecture shows the most promise – rests on evals. Today that work is spread across the organization. You'll turn it into one platform.
Your job will be to design and build the systems that generate samples at scale, score them with automated metrics and human annotations, track results across checkpoints and releases, and put the answers in front of researchers in minutes rather than days. You'll define what "better" means operationally – how we measure it, how confident we are, and how a result becomes a ship/no-ship decision.
We're looking for an ML infrastructure engineer to own model evaluation at Runway, end to end. Every decision we make about a model – which checkpoint to keep training, what to ship to millions of users, which datamix and architecture shows the most promise – rests on evals. Today that work is spread across the organization. You'll turn it into one platform. Your job will be to design and build the systems that generate samples at scale, score them with automated metrics and human annotations, track results across checkpoints and releases, and put the answers in front of researchers in minutes rather than days. You'll define what "better" means operationally – how we measure it, how confident we are, and how a result becomes a ship/no-ship decision. You'll be embedded within research teams as a member of ML Platform, and you'll set the technical direction for evals across the company. This is an extremely high-leverage role: the quality of our models is bounded by how well we can measure them.
What you'll do Own the evaluation platform end to end: the tooling and systems required to generate, annotate, review and adapt at frontier scale Define the CLIs, APIs, GUIs and storage layers required to make this process seamless, fast, sophisticated and collaborative
Own the evaluation platform end to end: the tooling and systems required to generate, annotate, review and adapt at frontier scale Define the CLIs, APIs, GUIs and storage layers required to make this process seamless, fast, sophisticated and collaborative Work directly with research teams on video, image, audio, agents, and robotics to understand what they need to measure and build it, then generalize the result into the platform
5+ years of experience building ML infrastructure or data platforms in production environments, with at least some of that time spent on evaluation, experimentation, or benchmarking systems
Experience with online experimentation platforms and connecting offline metrics to product outcomes
Strong Python and PyTorch, and hands-on experience running large batch GPU workloads on Kubernetes
Experience designing data pipelines and storage for large volumes of media or model outputs, with attention to versioning and reproducibility
Familiarity with experimental statistics: paired comparisons, confidence intervals, multiple-comparison pitfalls, inter-rater agreement
Ability to lead a broad technical area: gather requirements from many teams, set direction, make tradeoffs, and drive a roadmap without waiting to be told
Actively hiringFirst seen by us 9 days ago, still listed, and not a repost of an earlier posting.
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
$166K–$237K is typical
The middle half of what similar US postings state. This one doesn’t list pay.
From 1,537 postings in the last year. Stated ranges, not salaries paid.
Day 9
Similar jobs usually come down between day 7 and day 76.
From 6,389 similar postings that came down. Coming down is not always a hire.
2 open jobs
Open jobs by the week we first saw them, last 12 weeks.
Hiring recordAdd your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
A peek at our technical stack Our inference and training code is written in Python (PyTorch) and is cloud-native. We leverage multiple clouds and use a combination of Kubernetes native and home-grown tooling for efficient job orchestration. We’re big users of agentic development and operations. We have a robust internal platform and provide broad access to the latest models and harnesses.
- 5+ years of experience building ML infrastructure or data platforms in production environments, with at least some of that time spent on evaluation, experimentation, or benchmarking systems - Strong Python and PyTorch, and hands-on experience running large batch GPU workloads on Kubernetes - Experience designing data pipelines and storage for large volumes of media or model outputs, with attention to versioning and reproducibility
Complete your application on jobs.ashbyhq.com. The employer’s form will show what is required.
Already applied? Track this application
We are building AI to simulate the world through merging art and science.
We believe that world models are at the frontier of progress in artificial intelligence. Language models alone won’t solve the world’s hardest problems – robotics, disease, scientific discovery. Real progress requires models that experience the world and learn from their mistakes, the same way that humans do. And this kind of trial and error can be massively accelerated when done in simulation, rather than in the real world.
World models offer the most clear path to general-purpose simulation, changing how stories are told, how scientific progress is made and how the next frontiers of humanity are reached.
Our team consists of creative, open minded, caring and ambitious people who are determined to change the world. We aspire to continuously build impossible things and our ability to do so relies on building an incredible team. If you are driven to do the same, we'd love to hear from you.
*Titles posted to our careers page are updated regularly to best advertise the role and expected seniority, but internally we have a flat titling system with two types of titles: one for individual contributors and one for people managers.
We're looking for an ML infrastructure engineer to own model evaluation at Runway, end to end. Every decision we make about a model – which checkpoint to keep training, what to ship to millions of users, which datamix and architecture shows the most promise – rests on evals. Today that work is spread across the organization. You'll turn it into one platform.
Your job will be to design and build the systems that generate samples at scale, score them with automated metrics and human annotations, track results across checkpoints and releases, and put the answers in front of researchers in minutes rather than days. You'll define what "better" means operationally – how we measure it, how confident we are, and how a result becomes a ship/no-ship decision.
You'll be embedded within research teams as a member of ML Platform, and you'll set the technical direction for evals across the company. This is an extremely high-leverage role: the quality of our models is bounded by how well we can measure them.
Our inference and training code is written in Python (PyTorch) and is cloud-native. We leverage multiple clouds and use a combination of Kubernetes native and home-grown tooling for efficient job orchestration.
We’re big users of agentic development and operations. We have a robust internal platform and provide broad access to the latest models and harnesses.
You’ll have access to best-in-class models, agents, GPUs, storage and cloud services you need to drive a world class evaluation pipeline that helps us deliver frontier models.
Own the evaluation platform end to end: the tooling and systems required to generate, annotate, review and adapt at frontier scale
Define the CLIs, APIs, GUIs and storage layers required to make this process seamless, fast, sophisticated and collaborative
Work directly with research teams on video, image, audio, agents, and robotics to understand what they need to measure and build it, then generalize the result into the platform
Help set the standards for how we evaluate models at Runway: reproducibility, metric definitions, reporting formats, and when a result is trustworthy enough to act on
Support broad adoption of the platform and its integration throughout Runway’s research efforts across training, production model serving and more
Contribute broadly as a member of the ML Platform team to tools and systems that help Runway train and serve frontier models
5+ years of experience building ML infrastructure or data platforms in production environments, with at least some of that time spent on evaluation, experimentation, or benchmarking systems
Strong Python and PyTorch, and hands-on experience running large batch GPU workloads on Kubernetes
Experience designing data pipelines and storage for large volumes of media or model outputs, with attention to versioning and reproducibility
Familiarity with experimental statistics: paired comparisons, confidence intervals, multiple-comparison pitfalls, inter-rater agreement
Comfort building internal tools end to end, from the command line to the browser
Ability to lead a broad technical area: gather requirements from many teams, set direction, make tradeoffs, and drive a roadmap without waiting to be told
Familiarity with the full model development lifecycle: data, training, evaluation, serving
Self-starter who can work embedded with research teams and move fast
Strong systems thinking and pragmatic approach to production reliability
Humility and open mindedness; at Runway we love to learn from one another
Track record of building evaluation suites for generative models
Hands-on work with LLM- or VLM-as-judge pipelines
Experience with online experimentation platforms and connecting offline metrics to product outcomes
Prior work evaluating agents or robotics policies
Runway strives to recruit and retain exceptional talent from diverse backgrounds while ensuring pay equity for our team. Our salary ranges are based on competitive market rates for our size, stage and industry, and salary is just one part of the overall compensation package we provide.
There are many factors that go into salary determinations, including relevant experience, skill level and qualifications assessed during the interview process, and maintaining internal equity with peers on the team. The range shared below is a general expectation for the function as posted, but we are also open to considering candidates who may be more or less experienced than outlined in the job description. In this case, we will communicate any updates in the expected salary range.
Lastly, the provided range is the expected salary for candidates in the U.S. Outside of those regions, there may be a change in the range, which again, will be communicated to candidates.
Great things come from great teams. We’d love to hear from you.
We’re committed to creating a space where our employees can bring their full selves to work and have equal opportunity to succeed. So regardless of race, gender identity or expression, sexual orientation, religion, origin, ability, age, veteran status, if joining this mission speaks to you, we encourage you to apply.
More about Runway
We're excited to be recognized as a best place to work:
Crain's | InHerSight | BuiltIn NYC | INC
New ML Engineer postings at the employers we follow are down 4% over the last 6 weeks compared with the 6 before.
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
We file this job under ML Engineer, by the work the posting describes.
ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.
Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.
A data scientist often works out which model to build; an ML engineer makes it run reliably in production.
The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.
Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.
Selected passages from the saved posting. Check the full description for conditions and exceptions.
No pay amount identified in the saved description.
United States
Working pattern and location restrictions need checking in the full posting.
No clear work-authorization passage found. Eligibility is unconfirmed.