Back to jobs

ML Research Engineer, Performance Optimization

Cambridge, MA USA; San Francisco, CA USA

Actively hiring, first seen by us 8 days ago
Pay
USD 148,000–240,000/year · BaseAnnual period assumed — pay source
Expected Base Salary Range $148,000—$240,000 USD About LILA
Read the full posting
Work setting
Not stated
Employment
Not stated

What you’ll work on

Full posting
  • Profile GPU compute and develop kernels to raise utilization and cut memory use

  • Tune multi-node cluster communication, MoE frameworks and comm/compute overlap

  • Manage mid-training data across storage, streaming, tokenization and processing

  • Optimize the async RL cycle, including rollouts, environment interfacing and weight sync

  • Set tested, documented optimized configurations the whole team can adopt

From the employer’s posting
GPU compute performance: profiling, kernel development, long-context handling, memory footprint optimization
GPU cluster performance: collective/MoE framework optimization, comm/compute overlap, RDMA/IB/EFA usage, P/D disaggregation
Big data handling: manage mid-training scale data across storage, streaming, tokenization, and processing
RL pipeline: optimize the async RL cycle, including environment/sandbox interfacing, long rollout handling, large token metadata (such as router replay), and weight sync
Setting the performance standard: make sure optimized configurations are tested, accessible, and adaptable to research tasks so that the whole team can work at maximum velocity

Before you apply

Employer’s visa record
6 H-1B petitions approved in fiscal 2024 and 2025For the employer as a whole. This posting doesn’t say whether the role is sponsored. US Citizenship and Immigration Services.
Education & alternatives
What You'll Need to Succeed - BS/MS in Computer Science or a related quantitative field and 4+ years working as a software engineer or machine learning engineer, or PhD in a relevant research area and 1+ years work experience. - Experience with LLM inference libraries such as vLLM and training libraries such as Megatron, including usage and throughput optimization.

About Lila Sciences

Lila Sciences is building Scientific Superintelligence™ to solve humankind's greatest challenges.

In the employer’s words · Read in context

Apply at Lila Sciences

Posting record

Report an error

Actively hiringFirst seen by us 8 days ago, still listed, and not a repost of an earlier posting.

Status in our records
Live
First seen by us
Nov 11, 20268 days ago
Recorded sightings
9
Last seen by us
Nov 19, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Pay

In line with similar jobs

$148K–$240K against $166K–$237K, the middle half of what similar US postings state.

From 1,507 postings in the last year. Stated ranges, not salaries paid.

Time since we first saw it

Day 8

Similar jobs usually come down between day 7 and day 76.

At Lila Sciences, jobs usually come down by about day 65.

From 6,078 similar postings that came down, and 16 of Lila Sciences’s. Coming down is not always a hire.

Lila Sciences is hiring

22 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • ML Engineer hiring is steady
  • Its jobs tend to stay up a long time
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Docker
  • Airflow
Where the posting names them
- Experience with distributed systems or compute frameworks (Ray, Flyte, Airflow, or similar). - Familiarity with containerized compute backends (Docker, Modal) and sandbox orchestration. - Experience with Ray actors, Ray Serve, or similar actor-model frameworks.
- Experience with GPU networking frameworks like NCCL and NIXL, as well as fabrics like NVLink, IB, and EFA. - Experience with distributed systems or compute frameworks (Ray, Flyte, Airflow, or similar). - Familiarity with containerized compute backends (Docker, Modal) and sandbox orchestration.

Your next step

Complete your application on job-boards.greenhouse.io. The employer’s form will show what is required.

Applying through Greenhouse

  • No account needed: the form is one page, with an “Autofill my application” option that reads your resume.
  • Greenhouse doesn’t read resumes over 2.5 MB, or details placed in a header, footer, text box, table or second column. The file still attaches, but a recruiter has to type you in by hand.

Already applied? Track this application

Job description

View original posting ↗

Your Impact at LILA

Lila is building a platform where AI and automation co-evolve to accelerate scientific discovery Our AI Research team is looking to hire a ML Research Engineer to maximize iteration speed for mid- and post-training runs on 1T+ parameter LLMs using the latest NVIDIA chips. This involves not only maximizing GPU utilization on an individual training run but also making these optimizations maintainable, reusable, and adaptable so that the rest of the research team always knows how to confidently use the latest optimized stack.

What You'll Be Building

  • GPU compute performance: profiling, kernel development, long-context handling, memory footprint optimization
  • GPU cluster performance: collective/MoE framework optimization, comm/compute overlap, RDMA/IB/EFA usage, P/D disaggregation
  • Big data handling: manage mid-training scale data across storage, streaming, tokenization, and processing
  • RL pipeline: optimize the async RL cycle, including environment/sandbox interfacing, long rollout handling, large token metadata (such as router replay), and weight sync
  • Setting the performance standard: make sure optimized configurations are tested, accessible, and adaptable to research tasks so that the whole team can work at maximum velocity
  • Establishing the Pareto frontier: communicate with researchers about throughput/accuracy tradeoffs (e.g. lower-precision compute, higher async staleness) to establish optimal points

What You'll Need to Succeed

  • BS/MS in Computer Science or a related quantitative field and 4+ years working as a software engineer or machine learning engineer, or PhD in a relevant research area and 1+ years work experience.
  • Experience with LLM inference libraries such as vLLM and training libraries such as Megatron, including usage and throughput optimization.
  • Experience running LLM inference and/or training at scale across multi-node GPU clusters.
  • Familiarity with core concepts in reinforcement learning.
  • Proven track record of software engineering excellence — clean APIs, well-tested code, thoughtful abstractions.
  • Hands-on experience supporting and maintaining production libraries used by other engineers and researchers.
  • Strong CI/CD skills — experience designing integration test suites, build pipelines, and automated quality gates for multi-package ecosystems.
  • A passion for agentic systems and awareness of cutting-edge approaches in LLM tool use and autonomy.

Bonus Points For

  • Experience working with kernel DSLs and frameworks like CUDA, CuTe, CuTile, Triton, etc.
  • Experience with GPU networking frameworks like NCCL and NIXL, as well as fabrics like NVLink, IB, and EFA.
  • Experience with distributed systems or compute frameworks (Ray, Flyte, Airflow, or similar).
  • Familiarity with containerized compute backends (Docker, Modal) and sandbox orchestration.
  • Experience with Ray actors, Ray Serve, or similar actor-model frameworks.

Compensation

We offer competitive base compensation with bonus potential and generous early-stage equity. Your final offer will reflect your background, expertise, and expected impact.

U.S. Benefits. Full-time U.S. employees receive a comprehensive benefits program including medical, dental, and vision coverage; employer-paid life and disability insurance; flexible time off with generous company wide holidays; paid parental leave; an educational assistance program; commuter benefits, including bike share memberships for office based employees; and a company subsidized lunch program.

International Benefits. Full-time employees outside the U.S. receive a comprehensive benefits program tailored to their region. USD salary ranges apply only to U.S.-based positions; international salaries are set to local market.

Expected Base Salary Range
$148,000—$240,000 USD

About LILA

Lila Sciences is building Scientific Superintelligence™ to solve humankind's greatest challenges. We believe science is the most inspiring frontier for AI. Rather than hard-coding expert knowledge into tools, LILA builds systems that can learn for themselves.

LILA combines advanced AI models with proprietary AI Science Factory™ instruments into an operating system for science that executes the entire scientific method autonomously, accelerating discovery at unprecedented speed, scale, and impact across medicine, materials, and energy. Learn more at www.lila.ai.

Guided by our core values of truth, trust, curiosity, grit, and velocity, we move with startup speed while tackling problems of historic importance. If this sounds like an environment you'd love to work in, even if you don't meet every qualification listed above, we encourage you to apply.

We’re All In

Lila Sciences is committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity or Veteran status.

Information you provide during your application process will be handled in accordance with our Candidate Privacy Policy.

A Note to Agencies

Lila Sciences does not accept unsolicited resumes from any source other than candidates. The submission of unsolicited resumes by recruitment or staffing agencies to Lila Sciences or its employees is strictly prohibited unless contacted directly by Lila Science’s internal Talent Acquisition team. Any resume submitted by an agency in the absence of a signed agreement will automatically become the property of Lila Sciences, and Lila Sciences will not owe any referral or other fees with respect thereto.

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are up 12% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 556, the most 1,080, and the latest week had 556.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay
Expected Base Salary Range $148,000—$240,000 USD About LILA
Location & working pattern

Cambridge, MA USA; San Francisco, CA USA

Working pattern and location restrictions need checking in the full posting.

Work authorization
We’re All In Lila Sciences is committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity or Veteran status. Information you provide during your application process will be handled in accordance with our Candidate Privacy Policy.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.