Back to jobs

Staff Machine Learning Engineer, Evaluations

Pay
USD 190,000–220,000/year — pay source
Salary 190,000 – 220,000 USD per year ASAPP is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, disability, age, or veteran status. If you have a disability and need assistance with our employment application process, please email us at careers@asapp.com to obtain assistance. #LI-SL1 #LI-Hybrid
Read the full posting
Work setting
Not stated
Employment
Full-time — employment source
Employment type Full-time
Employment type in the employer’s posting: Full-time
From the employer’s posting

What you’ll work on

Full posting
  • Design eval methods: golden test sets, human review, LLM-as-judge, automated metrics

  • Build annotation pipelines, dataset versioning, and data quality checks

  • Partner with Research, Product, and Platform teams to productize experiments

  • Set expectations for model and agent releases and report on platform health

From the employer’s posting
Design eval methodologies appropriate to different stages of the pipeline: golden/regression test sets, human-in-the-loop review workflows, LLM-as-judge approaches, and automated metrics for task success, safety, and hallucinations.
Build the data infrastructure evaluation depends on: annotation and labeling pipelines, dataset versioning, data quality checks
Partner closely with Research, Product, and Platform teams to productize experiments into robust AI solutions
set expectations on what "good" looks like for a model/agent release, and report on platform health and coverage

What you’ll bring

All qualifications

Core experience

  • Demonstrated experience leading the technical direction of a project or small team: setting architecture, driving design reviews, and being accountable for a system's long-term health (not just shipping features).
  • Experience managing a small team of engineers in a fast-paced environment
  • Experience designing data pipelines for ML evaluation- labeling/annotation workflows, dataset versioning and quality control, and reproducible benchmarking.
  • Experience building and evaluating agentic systems at scale.
  • Experience with voice/audio quality evaluations.
  • Familiarity with large-scale ML experimentation, benchmarking, or simulation frameworks.
Qualification wording
Demonstrated experience leading the technical direction of a project or small team: setting architecture, driving design reviews, and being accountable for a system's long-term health (not just shipping features).
Experience managing a small team of engineers in a fast-paced environment
Experience designing data pipelines for ML evaluation- labeling/annotation workflows, dataset versioning and quality control, and reproducible benchmarking.
Experience building and evaluating agentic systems at scale.
Experience with voice/audio quality evaluations.
Familiarity with large-scale ML experimentation, benchmarking, or simulation frameworks.
Education & alternatives
- Experience designing data pipelines for ML evaluation- labeling/annotation workflows, dataset versioning and quality control, and reproducible benchmarking. - A Bachelor’s Degree in CS or other related fields - Demonstrated technical mentorship of junior and mid-level engineers, driving adoption of best practices and architectural alignment for scalability and extensibility.

Benefits in the posting

Full benefits wording
  • Competitive compensation with stock options
  • Comprehensive medical, vision, and dental insurance
  • 401k matching
  • Fitness and wellness stipend
  • Mental well-being benefits
  • Professional learning and development stipend
  • Parental leave, including adoptive and foster parents
  • 3 weeks paid time off (increases with tenure) along with sick leave, bereavement and jury duty

From the employer’s posting.

Apply at Asapp-2

Posting record

Report an error
Status in our records
Live
First seen by us
Sep 12, 202670 days ago
Recorded sightings
53
Last seen by us
Nov 22, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Pay

In line with similar jobs

$190K–$220K against $166K–$237K, the middle half of what similar US postings state.

From 1,537 postings in the last year. Stated ranges, not salaries paid.

Time since we first saw it

Day 70

Similar jobs usually come down between day 7 and day 76.

From 6,398 similar postings that came down. Coming down is not always a hire.

Asapp-2 is hiring

2 open jobs

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • AWS
  • Docker
  • Kafka
  • Kubernetes
Where the posting names them
- Demonstrated experience leading the technical direction of a project or small team: setting architecture, driving design reviews, and being accountable for a system's long-term health (not just shipping features). - Strong architectural skills, with proven experience designing complex, data-intensive software systems and production experience with Python, AWS, Kubernetes, and/or Docker. - Experience designing data pipelines for ML evaluation- labeling/annotation workflows, dataset versioning and quality control, and reproducible benchmarking.
- Knowledge of techniques for optimizing model architectures for faster inference. - Experience with AWS, CI/CD, Kafka, Athena -

Your next step

Complete your application on jobs.lever.co. The employer’s form will show what is required.

Applying through Lever

  • No account needed: one form, and Lever reads your resume into the profile recruiters search.

Already applied? Track this application

Job description

View original posting ↗

At ASAPP, our mission is simple: deliver the best AI-powered customer experience—faster than anyone else. To achieve that, we’re guided by principles that shape how we think, build, and execute. We value customer obsession, purposeful speed, ownership, and a relentless focus on outcomes. ASAPP’s AI Engineering team is seeking an enterprising, talented and curious machine learning engineer.
 
The AI Engineering team is responsible for working closely with the research and modeling teams to create state-of-the-art NLP models for specific tasks, and deploy them in a production setting designed to serve our customers at scale. We are looking for a Machine Learning Engineer to help build and evaluate the core intelligence behind our agentic AI systems. This role will play a key part in designing and owning evaluation frameworks that ensure quality, safety, and performance across complex agentic systems.

We're looking for a Staff Machine Learning Engineer to own and grow the evaluation platform that measures quality, safety, and performance across ASAPP's agentic AI systems- the infrastructure that tells us, with confidence, whether a model or agent change is actually an improvement before it reaches customers.

This a hybrid role with 10-12 days of in-office presence per month to balance flexibility with collaboration.

What you'll do

  • Lead the technical roadmap and architecture for the evaluation platform, from offline benchmarking to online/production monitoring of agentic and LLM-based systems.

  • Design eval methodologies appropriate to different stages of the pipeline: golden/regression test sets, human-in-the-loop review workflows, LLM-as-judge approaches, and automated metrics for task success, safety, and hallucinations.

  • Build the data infrastructure evaluation depends on: annotation and labeling pipelines, dataset versioning, data quality checks, and tooling that lets researchers and product teams run and interpret experiments without needing platform team help.

  • Partner closely with Research, Product, and Platform teams to productize experiments into robust AI solutions

  • Represent the eval platform to stakeholders outside the immediate team- set expectations on what "good" looks like for a model/agent release, and report on platform health and coverage.

  • Stay current with advancements in ML, NLP, voice, and LLM systems, and contribute actively to technical discussions across teams.

  • Mentor and support other engineers through design reviews, feedback, and knowledge sharing.

What you'll need

  • Deep, hands-on experience building and operating evaluation systems for modern ML/LLM/agentic systems- not just consuming existing eval tools.

  • Demonstrated experience leading the technical direction of a project or small team: setting architecture, driving design reviews, and being accountable for a system's long-term health (not just shipping features).

  • Strong architectural skills, with proven experience designing complex, data-intensive software systems and production experience with Python, AWS, Kubernetes, and/or Docker.

  • Experience designing data pipelines for ML evaluation- labeling/annotation workflows, dataset versioning and quality control, and reproducible benchmarking.

  • A Bachelor’s Degree in CS or other related fields

  • Demonstrated technical mentorship of junior and mid-level engineers, driving adoption of best practices and architectural alignment for scalability and extensibility.

  • Desire to learn, teach, and collaborate closely with cross-functional peers.

What we'd like to see

  • Experience building and evaluating agentic systems at scale. 

  • Experience with voice/audio quality evaluations.

  • Production experience with LLM-centric services (e.g., inference, orchestration, evaluation, monitoring)

  • Familiarity with large-scale ML experimentation, benchmarking, or simulation frameworks.

  • Experience with conversational/customer-support AI domains (e.g., containment rate, conversation quality, goal completion).

  • Knowledge of techniques for optimizing model architectures for faster inference.

  • Experience with AWS, CI/CD, Kafka, Athena

  • Experience managing a small team of engineers in a fast-paced environment

Benefits

  • Competitive compensation with stock options
  • Comprehensive medical, vision, and dental insurance
  • 401k matching
  • Fitness and wellness stipend
  • Mental well-being benefits
  • Professional learning and development stipend
  • Parental leave, including adoptive and foster parents
  • 3 weeks paid time off (increases with tenure) along with sick leave, bereavement and jury duty
Compensation package also includes a performance bonus on top of the listed salary range
 
Separately, we also offer a compelling equity grant comprised of stock options

Salary

190,000 – 220,000 USD per year

ASAPP is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, disability, age, or veteran status. If you have a disability and need assistance with our employment application process, please email us at careers@asapp.com to obtain assistance. #LI-SL1 #LI-Hybrid

Employment type

Full-time

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are down 4% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 508, the most 899, and the latest week had 508.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay
Salary 190,000 – 220,000 USD per year ASAPP is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, disability, age, or veteran status. If you have a disability and need assistance with our employment application process, please email us at careers@asapp.com to obtain assistance. #LI-SL1 #LI-Hybrid
Location & working pattern

Mountain View

We're looking for a Staff Machine Learning Engineer to own and grow the evaluation platform that measures quality, safety, and performance across ASAPP's agentic AI systems- the infrastructure that tells us, with confidence, whether a model or agent change is actually an improvement before it reaches customers. This a hybrid role with 10-12 days of in-office presence per month to balance flexibility with collaboration. What you'll do
More source context
190,000 – 220,000 USD per year ASAPP is committed to creating a diverse environment and is proud to be an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, disability, age, or veteran status. If you have a disability and need assistance with our employment application process, please email us at careers@asapp.com to obtain assistance. #LI-SL1 #LI-Hybrid Employment type
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.