Back to jobs

Machine Learning Engineer (Evals and Voice Models)

Pay
USD 181,000–250,000/year · BaseAnnual period assumed — pay source
Base salary range: $181,000—$250,000 USD Why join us?
Read the full posting
Work setting
Not stated
Employment
Not stated

What you’ll work on

Full posting
  • Design and document evaluation frameworks for AI agents across voice, chat and messaging

  • Train and fine-tune TTS, ASR and speech-to-speech models on production and synthetic data

  • Calibrate LLM-as-judge systems against human raters and design annotation guidelines

  • Build live quality monitoring for deployed agents and flag model or data drift

  • Build release gates and offline regression suites that each AI change must pass

From the employer’s posting
Design and document comprehensive evaluation frameworks for Aircall’s AI agents across voice, chat and messaging.
Train and fine-tune voice models (TTS, ASR, speech-to-speech) using production and synthetic data
Design annotation guidelines and workflows for human-labeled evaluation data, and calibrate LLM-as-judge systems against human raters
Build and maintain live quality monitoring for deployed AI agents, tracking accuracy, resolution rate, and safety signals in production, and flagging model or data drift before it impacts customers.
Build release gates, the offline regression suite each AI surface must pass before a prompt, model, or config change ships

Before you apply

Employer’s visa record
1 H-1B petition approved in fiscal 2024 and 2025For the employer as a whole. This posting doesn’t say whether the role is sponsored. US Citizenship and Immigration Services.

What you’ll bring

All qualifications

Core experience

  • 3+ years of experience in ML Engineering or Applied ML with 8+ years of overall experience
  • Strong experience in evaluating supervised, unsupervised, LLMs and deep learning models.
  • Hands-on experience in failure analysis and evaluating LLMs
  • Experience building automated evaluation systems
  • Strong communication skills to articulate complex technical concepts across technical and non-technical audiences
  • Hands-on experience training or fine-tuning voice/speech models (TTS, ASR, or speech-to-speech), including data pipeline construction and experimentation.

Preferred experience

  • Experience evaluating LLMs or agentic systems (e.g., LLM-as-a-judge, RAG evaluation)
  • Experience with synthetic data generation and prompt engineering
  • Experience training or fine-tuning voice models at scale, with familiarity in synthetic data generation, model distillation, or low-latency inference optimization for production voice agents.
Qualification wording
3+ years of experience in ML Engineering or Applied ML with 8+ years of overall experience
Strong experience in evaluating supervised, unsupervised, LLMs and deep learning models.
Hands-on experience in failure analysis and evaluating LLMs
Experience building automated evaluation systems
Strong communication skills to articulate complex technical concepts across technical and non-technical audiences
Hands-on experience training or fine-tuning voice/speech models (TTS, ASR, or speech-to-speech), including data pipeline construction and experimentation.
Experience evaluating LLMs or agentic systems (e.g., LLM-as-a-judge, RAG evaluation)
Experience with synthetic data generation and prompt engineering
Experience training or fine-tuning voice models at scale, with familiarity in synthetic data generation, model distillation, or low-latency inference optimization for production voice agents.
Education & alternatives
Preferred Qualifications - MS / PhD in Computer Science, Machine Learning, Statistics, or related field - Experience evaluating LLMs or agentic systems (e.g., LLM-as-a-judge, RAG evaluation)
Apply at Aircall.io

Posting record

Report an error
Status in our records
Live
First seen by us
Oct 1, 202647 days ago
Recorded sightings
83
Last seen by us
Nov 17, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Pay

In line with similar jobs

$181K–$250K against $166K–$237K, the middle half of what similar US postings state.

From 1,505 postings in the last year. Stated ranges, not salaries paid.

Time since we first saw it

Day 47

Similar jobs usually come down between day 7 and day 77.

From 5,865 similar postings that came down. Coming down is not always a hire.

Aircall.io is hiring

17 open jobs

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume to compare the role’s scope, tools and requirements with your experience.

Your next step

Complete your application on job-boards.greenhouse.io. The employer’s form will show what is required.

Applying through Greenhouse

  • No account needed: the form is one page, with an “Autofill my application” option that reads your resume.
  • Greenhouse doesn’t read resumes over 2.5 MB, or details placed in a header, footer, text box, table or second column. The file still attaches, but a recruiter has to type you in by hand.

Already applied? Track this application

Job description

View original posting ↗

Aircall is a unicorn AI-powered communications platform transforming how teams win with customers.

For over a decade, we’ve believed voice builds the strongest business relationships. We bring voice, SMS, WhatsApp, and 250+ integrations together on one seamless workspace—so every customer conversation drives revenue, resolves issues faster, and builds trust. Our AI Voice Agents handle routine interactions while your team leads the conversations that matter most. Used by 24,000+ businesses worldwide.

We’re 600+ people from 40+ nationalities across Paris, London, Madrid, Berlin, New York, San Francisco, Seattle, Sydney, and Mexico City. We’re ambitious, product-driven, and execution-focused—built for visible impact, fast decisions, and real growth.

Come join us!

Aircall's AI suite includes an AI Voice Agent and AI Messaging Agent that autonomously handle calls, WhatsApp, and SMS, plus AI Assist, which delivers real-time coaching, call summaries, and CRM automation for sales and support teams. We are looking for someone that can build out the evaluation foundation across all of these products and other agentic products. You'll work on voice models, agent capability evals, benchmark design, LLM-as-judge systems, failure analysis, and the infrastructure that ties it together by establishing shared metrics, test sets, and tooling to measure accuracy, resolution quality, and safety consistently across products. You will set up repeatable pipelines for regression testing and benchmarking as models and features evolve so teams can ship confidently without re-inventing evaluation methodology for each product.

Key Responsibilities

  • Design and document comprehensive evaluation frameworks for Aircall’s AI agents across voice, chat and messaging.
  • Train and fine-tune voice models (TTS, ASR, speech-to-speech) using production and synthetic data, iterating on architecture, data mix, and training strategy to improve accuracy, naturalness, and latency.
  • Assess AI generated solutions across training pipelines, experimentation setups, debugging processes, and optimization strategies. 
  • Analyze system design decisions and identify strengths, weaknesses, and potential failure points.
  • Design annotation guidelines and workflows for human-labeled evaluation data, and calibrate LLM-as-judge systems against human raters to ensure automated evals stay trustworthy over time.
  • Build and maintain live quality monitoring for deployed AI agents, tracking accuracy, resolution rate, and safety signals in production, and flagging model or data drift before it impacts customers.
  • Own the metric contract for every published AI metrics, including definition, population, grain, rollup, validity window.
  • Build release gates, the offline regression suite each AI surface must pass before a prompt, model, or config change ships, measuring reliability across repeated trials, not just average pass rates.
  • Build voice-specific evaluation: simulated callers across accents, languages, background noise, barge-in, DTMF, and tool failures, with latency and ASR accuracy as first-class quality metrics.

Minimum Qualifications

  • BS in Computer Science, Machine Learning, Statistics, or related field
  • 3+ years of experience in ML Engineering or Applied ML with 8+ years of overall experience
  • Strong experience in evaluating supervised, unsupervised, LLMs and deep learning models.
  • Hands-on experience in failure analysis and evaluating LLMs
  • Experience building automated evaluation systems
  • Strong communication skills to articulate complex technical concepts across technical and non-technical audiences
  • Hands-on experience training or fine-tuning voice/speech models (TTS, ASR, or speech-to-speech), including data pipeline construction and experimentation.

Preferred Qualifications

  • MS / PhD in Computer Science, Machine Learning, Statistics, or related field
  • Experience evaluating LLMs or agentic systems (e.g., LLM-as-a-judge, RAG evaluation)
  • Experience with synthetic data generation and prompt engineering
  • Experience training or fine-tuning voice models at scale, with familiarity in synthetic data generation, model distillation, or low-latency inference optimization for production voice agents.
Base salary range:
$181,000—$250,000 USD

Why join us?

🚀 Key moment to join Aircall in terms of growth and opportunities

💆‍♀️ Our people matter, work-life balance is important at Aircall

📚 Fast-learning environment, entrepreneurial and strong team spirit

🌍 45+ Nationalities: cosmopolite & multi-cultural mindset

💶 Competitive salary package & benefits

 

DE&I Statement: 

At Aircall, we believe diversity, equity and inclusion – irrespective of origins, identity, background and orientations – are core to our journey. 

We pride ourselves on promoting active inclusion within our business to foster a strong sense of belonging for all. We’re working to create a place filled with diverse people who can enrich and learn from one another. We’re committed to ensuring that everyone not only has a seat at the table but is valued and respected at it by providing equal opportunities to develop and thrive.  

We will constantly challenge ourselves to make sure that we live up to our ambitions around diversity, equity and inclusion, and keep this conversation open. Above all else, we understand and acknowledge that we have work to do and much to learn.

 

Want to know more about candidate privacy? Find our Candidate Privacy Notice here.

We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are up 12% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 556, the most 1,080, and the latest week had 556.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay
Base salary range: $181,000—$250,000 USD Why join us?
Location & working pattern

San Francisco Office

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.