Back to jobs

Machine Learning Engineer

New York · New York, United States

Open 3 months, first seen by us 93 days ago
Pay
Pay not listedSimilar roles state $166K–$237K
Work setting
Not stated
Employment
Full-time — employment source
Employment type: Full-time
From the employer’s posting
Team
Replay › Engineering — team source
Department: Replay › Engineering
From the employer’s posting

What you’ll work on

Full posting
  • Own and improve NER, entity resolution, and document understanding models

  • Turn model failures into a prioritized improvement roadmap

  • Design active-learning loops to pick the examples worth hand-labeling

  • Build representative datasets and benchmarks with decision-relevant metrics

  • Productionize model changes with evaluation evidence and safe rollout

From the employer’s posting
Own and improve NER, entity resolution, structured or tabular detection, document understanding, semantic review, or related de-identification systems
Transform model failures and capability ceilings into a prioritized improvement roadmap
Design active-learning loops that combine model sweeps, LLM-assisted review, clustering, and uncertainty signals to identify the examples most worth hand-labeling
Build representative datasets and benchmarks, and use decision-relevant metrics to reveal strengths, weaknesses, uncertainty, and failure costs
Productionize improvements with reproducible artifacts, evaluation evidence, runtime instrumentation, and safe rollout

Before you apply

Sponsorship
Visa sponsorship not confirmed — sponsorship source
Will you now or in the future require visa sponsorship?
From the employer’s application form

What you’ll bring

All qualifications

Core experience

  • 3+ years of professional ML or software engineering experience
  • Improving models in production
  • Strong Python and software engineering skills for production pipelines
  • Solid grasp of precision, recall, F1, calibration, and class imbalance
Qualification wording
3+ years of professional machine learning or software engineering experience
including improving models in production
a strong Python and software engineer who can work inside data pipelines and production systems, not only notebooks
You have a strong grasp of precision, recall, F1, calibration, thresholding, class imbalance, imperfect labels, distribution shift, and representative evaluation

About Replay

At Replay, we help businesses monetize their historical operating data by licensing it to the frontier AI and data labs for model training.

In the employer’s words · Read in context · Company website ↗

Apply at Replay

Posting record

Report an error

Open 3 monthsWe first saw this posting 93 days ago and it is still up. Roles open this long are often slow to fill or kept open to collect applications.

Status in our records
Live
First seen by us
Aug 20, 202693 days ago
Recorded sightings
56
Last seen by us
Nov 22, 2026
Employer says posted
Aug 13, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Pay

$166K–$237K is typical

The middle half of what similar US postings state. This one doesn’t list pay.

From 1,537 postings in the last year. Stated ranges, not salaries paid.

Time since we first saw it

Day 93

Up longer than most. Similar jobs usually come down between day 7 and day 76.

From 6,398 similar postings that came down. Coming down is not always a hire.

Replay is hiring

3 open jobs

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
Where the posting names them
- You have a strong grasp of precision, recall, F1, calibration, thresholding, class imbalance, imperfect labels, distribution shift, and representative evaluation - You are an applied engineer first: a strong Python and software engineer who can work inside data pipelines and production systems, not only notebooks - You have a bias toward action while maintaining scientific and engineering rigor

Your next step

Complete your application on jobs.ashbyhq.com. The employer’s form will show what is required.

Applying through Ashby

  • No account needed. “Autofill from resume” at the top of the form fills the key fields from your file.

Already applied? Track this application

Job description

View original posting ↗

About Replay

We are building the first-ever bank for enterprise data. There are over 35 million businesses in the US alone, many of which are sitting on an asset they don’t know they have. At Replay, we help businesses monetize their historical operating data by licensing it to the frontier AI and data labs for model training. Replay sits at the intersection of advancing the frontier of AI, while helping companies generate millions in growth capital without having to give up equity, take on debt, or lay off employees.

Why Join Replay Now

  • We have scaled from $0 to a multi-eight-figure run rate in a matter of months

  • We have raised from top-tier investors, including Floodgate, Afore, Ludlow, and Hustle Fund

  • We are small enough that you will carry outsized responsibility and grow as quickly as the company does

  • You will partner with and build for some of the fastest and most important companies in the world

  • You will help build a massive, category-defining business from the ground floor

The Role

Replay turns sensitive internal enterprise data into de-identified datasets without destroying the structure and meaning that make the data valuable. The data does not arrive in one clean modality. It spans messages, documents, tables, files, images, metadata, and provider-specific structures, with important context distributed across all of them.

You will improve how well our system understands and protects that data. Your initial scope will be a prioritized subset of named-entity recognition, entity and identity resolution, structured extraction, classification, semantic review, or other model-backed parts of the de-identification pipeline. We do not expect one person to be an expert in every modality. The goal is measurable improvement in the areas you own: better precision, recall, F1, high-risk coverage, and preserved data utility across the failure modes that matter.

This is an applied, production-facing ML role. You will study errors, form hypotheses, build datasets and experiments, improve or replace models, and ship the result into a live pipeline. Evaluation, reproducibility, observability, and safe releases matter because they let us identify, ship, and verify meaningful model improvements in production.

What You'll Do

  • Own and improve NER, entity resolution, structured or tabular detection, document understanding, semantic review, or related de-identification systems

  • Transform model failures and capability ceilings into a prioritized improvement roadmap

  • Design active-learning loops that combine model sweeps, LLM-assisted review, clustering, and uncertainty signals to identify the examples most worth hand-labeling

  • Build representative datasets and benchmarks, and use decision-relevant metrics to reveal strengths, weaknesses, uncertainty, and failure costs

  • Choose and combine deterministic rules, classical ML, fine-tuning, embeddings, multimodal models, and LLM-based approaches based on the problem and evidence

  • Design experiments, tune thresholds, analyze precision-recall and utility tradeoffs, and explain which changes are real, uncertain, or limited to particular conditions

  • Productionize improvements with reproducible artifacts, evaluation evidence, runtime instrumentation, and safe rollout

  • Optimize inference cost, latency, and throughput without hiding regressions in quality or high-risk recall

  • Build high-fidelity evaluation environments with seeded failure modes and programmatic verifiers that expose subtle regressions

  • Build reliable model- or agent-based harnesses with bounded behavior and explicit output verification when the problem calls for them

  • Partner with Applied Science on measurement and calibration, Data and Product Engineering on pipeline and review systems, and Security and Quality on acceptable risk

  • Use AI engineering tools deeply to accelerate research, implementation, error analysis, and evaluation while verifying their output

What Success Looks Like

  • Model improvements generalize beyond the examples used to develop them and hold up in replay, shadow, and production evidence

  • Priority modalities and entity classes show credible improvements in precision, recall, F1, or other decision-relevant quality measures

  • High-risk misses decline without unacceptable over-redaction or loss of useful structure

  • New formats and modalities can be covered without relying on brittle one-off fixes

  • Improvements reduce meaningful delivery risk, review or rework burden, or loss of data utility rather than moving only an isolated benchmark

  • The team can explain why a model changed, where it improved or regressed across consequential failure modes and data segments, and whether the change should ship

  • The path from error discovery to a trustworthy production improvement becomes faster and more repeatable

  • Quality gains remain inside acceptable inference-cost, latency, and operational constraints

You Might Thrive Here If

  • You have 3+ years of professional machine learning or software engineering experience, including improving models in production

  • You have startup experience, enjoy broad ownership, and thrive when requirements are evolving or incomplete

  • You use modern AI tools fluently and verify their output

  • You have personally moved model quality through error analysis, data work, experimentation, implementation, deployment, and iteration

  • You have a strong grasp of precision, recall, F1, calibration, thresholding, class imbalance, imperfect labels, distribution shift, and representative evaluation

  • You are an applied engineer first: a strong Python and software engineer who can work inside data pipelines and production systems, not only notebooks

  • You have a bias toward action while maintaining scientific and engineering rigor

  • You are curious and stay current with relevant state-of-the-art methods

  • You choose techniques based on the shape of the problem and can combine deterministic, statistical, neural, and LLM-based approaches

  • You communicate uncertainty and tradeoffs clearly to scientists, engineers, and people making delivery or risk decisions

This Role May Not Be for You If

  • You want to focus on research novelty without owning measurable production improvement

  • You prefer optimizing one aggregate benchmark without investigating consequential failure modes, data segments, and failure costs

  • You want data preparation, evaluation, deployment, and production diagnosis to belong entirely to other teams

  • You reach for a larger model before understanding the errors, constraints, and simpler alternatives

  • You do not want AI tools to be part of your daily engineering and research workflow

Bonus

  • Experience with NER, entity resolution, information extraction, document understanding, multimodal systems, or privacy-preserving ML

  • Experience with hyperparameter tuning, data augmentation, model merging, ensembles, knowledge distillation, or multimodal model training

  • Experience fine-tuning or adapting transformer, GLiNER, embedding, vision-language, or small specialized models

  • Experience with active learning, uncertainty sampling, weak supervision, human-in-the-loop review, or LLM-assisted evaluation pipelines

  • Experience building goldens, adversarial corpora, replay systems, model bakeoffs, agentic harnesses, or programmatic evaluation environments

  • Experience with difficult ML or labeling problems

  • Experience with ONNX Runtime, TensorRT, model pruning, quantization, or other CPU/GPU inference optimization

  • Experience with sensitive enterprise data or other high-trust production systems

  • Experience with synthetic data generation and managing the synth-to-real gap

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are down 4% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 508, the most 899, and the latest week had 508.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

New York, United States

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.