Back to jobs

Distributed LLM Inference Engineer

San Francisco · Palo Alto · San Francisco

Open 5 months, first seen by us 169 days ago
Pay
Pay not listedSimilar roles state $166K–$237K
Work setting
Hybrid — work setting source
Location type: Hybrid
From the employer’s posting
Employment
Full-time — employment source
Employment type: Full-time
From the employer’s posting
Team
Engineering — team source
Department: Engineering
From the employer’s posting

What you’ll work on

Full posting

As a Distributed LLM Inference Engineer, you will help systems and optimizations that push the boundaries of performance for inference at large scale.

Iterate very quickly with product teams to ship the end to end solutions for Batch and Online inference at high scale which will be used by open-source Ray users and customers of Anyscale

Before you apply

Employer’s visa record
34 H-1B petitions approved in fiscal 2024 and 2025For the employer as a whole. This posting doesn’t say whether the role is sponsored. US Citizenship and Immigration Services.
Sponsorship
Visa sponsorship not confirmed — sponsorship source
Would you require visa sponsorship to work in the country in which this position is based?
From the employer’s application form

About Anyscale

At Anyscale, we're on a mission to democratize distributed computing and make it accessible to software developers of all skill levels.

In the employer’s words · Read in context · Company website ↗

Apply at Anyscale

Posting record

Report an error

Open 5 monthsWe first saw this posting 169 days ago and it is still up. Roles open this long are often slow to fill or kept open to collect applications.

Status in our records
Live
First seen by us
Jun 2, 2026169 days ago
Recorded sightings
99
Last seen by us
Nov 19, 2026
Employer says posted
May 27, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Pay

$166K–$237K is typical

The middle half of what similar US postings state. This one doesn’t list pay.

From 1,507 postings in the last year. Stated ranges, not salaries paid.

Time since we first saw it

Day 169

Up longer than most. Similar jobs usually come down between day 7 and day 76.

From 5,848 similar postings that came down. Coming down is not always a hire.

Anyscale is hiring

3 open jobs

  • No new data jobs since July
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • PyTorch
  • TensorFlow
Where the posting names them
- Familiarity with running ML inference at large scale with high throughput and low latency - Familiarity with deep learning and deep learning frameworks (e.g. PyTorch) - Solid understanding of distributed systems, ML inference challenges
- Work closely with community on LLM engines like vLLM, TensorRT-LLM - Contributions to deep learning frameworks (PyTorch, TensorFlow) - Contributions to deep learning compilers (Triton, TVM, MLIR)

Your next step

Complete your application on jobs.ashbyhq.com. The employer’s form will show what is required.

Applying through Ashby

  • No account needed. “Autofill from resume” at the top of the form fills the key fields from your file.

Already applied? Track this application

Job description

View original posting ↗

About Anyscale

At Anyscale, we're on a mission to democratize distributed computing and make it accessible to software developers of all skill levels. We’re commercializing Ray, a popular open-source project that's creating an ecosystem of libraries for scalable machine learning. Companies like OpenAI, Uber, Spotify, Instacart, Cruise, and many more, have Ray in their tech stacks to accelerate the progress of AI applications out into the real world.


With Anyscale, we’re building the best place to run Ray, so that any developer or data scientist can scale an ML application from their laptop to the cluster without needing to be a distributed systems expert.


Proud to be backed by Andreessen Horowitz, NEA, and Addition with $250+ million raised to date.


About the role

As a Distributed LLM Inference Engineer, you will help systems and optimizations that push the boundaries of performance for inference at large scale. This is an incredibly critical role to Anyscale as it allows us to achieve a market leading position for  AI infrastructure.

As part of this role, you will

  • Iterate very quickly with product teams to ship the end to end solutions for Batch and Online inference at high scale which will be used by open-source Ray users and customers of Anyscale

  • Work across the stack integrating Ray Data and LLM engine providing optimizations achieving low cost solutions for large scale ML inference 

  • Integrate with Open source software like vLLM, work closely with the community to adopt these techniques in Anyscale solutions, and also contribute improvements to open source

  • Follow the latest state-of-the-art in the open source and the research community, implementing and extending best practices

We'd love to hear from you if you have

  • Familiarity with running ML inference at large scale with high throughput and low latency

  • Familiarity with deep learning and deep learning frameworks (e.g. PyTorch)

  • Solid understanding of distributed systems, ML inference challenges

Bonus points!

  • ML Systems knowledge

  • Experience using Ray 

  • Work closely with community on LLM engines like vLLM, TensorRT-LLM

  • Contributions to deep learning frameworks (PyTorch, TensorFlow)

  • Contributions to deep learning compilers (Triton, TVM, MLIR)

  • Prior experience working on GPUs / CUDA

Compensation

At Anyscale, we take a market-based approach to compensation. We are data-driven, transparent, and consistent.  As the market data changes over time, the target salary for this role may be adjusted.

This role is also eligible to participate in Anyscale's Equity and Benefits offerings, including the following:

  • Stock Options

  • Healthcare plans, with premiums covered by Anyscale at 99% for both employees and dependents

  • 401k Retirement Plan

  • Education & Wellbeing Stipend

  • Paid Parental Leave

  • Fertility Benefits

  • Paid Time Off

  • Commute reimbursement

  • 100% of in-office meals covered

Anyscale Inc. is an Equal Opportunity Employer. Candidates are evaluated without regard to age, race, color, religion, sex, disability, national origin, sexual orientation, veteran status, or any other characteristic protected by federal or state law. 

Anyscale Inc. is an E-Verify company and you may review the Notice of E-Verify Participation and the Right to Work posters in English and Spanish

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are up 12% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 556, the most 1,080, and the latest week had 556.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

San Francisco

Working pattern and location restrictions need checking in the full posting.

Work authorization
Anyscale Inc. is an Equal Opportunity Employer. Candidates are evaluated without regard to age, race, color, religion, sex, disability, national origin, sexual orientation, veteran status, or any other characteristic protected by federal or state law. Anyscale Inc. is an E-Verify company and you may review the Notice of E-Verify Participation and the Right to Work posters in English and Spanish

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.