Pay
$120K–$194K is typical
The middle half of what similar US postings state. This one doesn’t list pay.
From 3,983 postings in the last year. Stated ranges, not salaries paid.
New York · New York, United States
Open 3 months, first seen by us 93 days agoEmployment type: Full-timeFrom the employer’s posting
Department: Replay › EngineeringFrom the employer’s posting
Build golden, adversarial, and production-like evaluation corpora with labeling policy and versioning
Evaluate detectors, models, judges, and pipeline changes with comparisons that support a ship decision
Break aggregate results into slices like modality, provider, language, and risk tier
Turn vague quality ideas like useful or safe into measurable claims with uncertainty
Build reproducible analysis and evaluation pipelines using Python, SQL, and synthetic data
Design representative samples and build golden, adversarial, replay, and production-like corpora with explicit provenance, labeling policy, agreement, adjudication, and versioning
Evaluate detectors, models, prompts, judges, thresholds, review workflows, and pipeline changes using comparisons that can support a real decision
Break aggregate results into the modalities, providers, entity classes, customer contexts, languages, formats, and risk tiers that reveal consequential failures
Turn ambiguous concepts such as “useful,” “clean,” or “safe” into measurable claims with known uncertainty and clear decision consequences
Build reproducible analysis, evaluation pipelines, and high-fidelity environments using Python, SQL, synthetic data, historical replay, seeded failures, and programmatic verifiers
At Replay, we help businesses monetize their historical operating data by licensing it to the frontier AI and data labs for model training.
In the employer’s words · Read in context · Company website ↗
Open 3 monthsWe first saw this posting 93 days ago and it is still up. Roles open this long are often slow to fill or kept open to collect applications.
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
$120K–$194K is typical
The middle half of what similar US postings state. This one doesn’t list pay.
From 3,983 postings in the last year. Stated ranges, not salaries paid.
Day 93
Up longer than most. Similar jobs usually come down between day 7 and day 60.
From 12,339 similar postings that came down. Coming down is not always a hire.
3 open jobs
Open jobs by the week we first saw them, last 12 weeks.
Hiring recordAdd your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
As Replay's first Data Scientist focused on evaluation, you will establish how we know whether that data is actually getting better. You will build the datasets, experiments, quality measures, and feedback loops that expose hidden failures, accelerate model and pipeline improvement, and give the team confidence in what it delivers. This is a hands-on, zero-to-one role at the intersection of data science, AI, and a real production system. You will write Python and SQL, construct evaluation corpora, study failure patterns, design comparisons, calibrate human and model-based judgments, and turn the result into a clear decision. The questions are scientifically difficult, but the output must be practical enough to change what the team builds and ships. You will work closely with Machine Learning, Product Engineering, Data Engineering, Security, Quality, domain experts, and the team making delivery decisions. Machine Learning Engineers own changing model behavior. You own the credibility of the evidence used to decide whether a model, pipeline, or delivery change actually made the data safer or more useful.
- Connect local measures to escaped sensitive information, avoidable over-redaction, preserved data utility, review burden, rework, and delivery acceptance - Build reproducible analysis, evaluation pipelines, and high-fidelity environments using Python, SQL, synthetic data, historical replay, seeded failures, and programmatic verifiers - Establish holdout and evaluation practices that keep the evidence trustworthy while model and product teams iterate quickly
- You can investigate messy, multi-stage data systems and determine where an apparent gain or loss actually came from - You are comfortable writing Python and SQL and building reproducible technical artifacts rather than handing requirements to an engineering team - You can protect the independence of an evaluation while collaborating closely with the people whose work it evaluates
Complete your application on jobs.ashbyhq.com. The employer’s form will show what is required.
Already applied? Track this application
We are building the first-ever bank for enterprise data. There are over 35 million businesses in the US alone, many of which are sitting on an asset they don’t know they have. At Replay, we help businesses monetize their historical operating data by licensing it to the frontier AI and data labs for model training. Replay sits at the intersection of advancing the frontier of AI, while helping companies generate millions in growth capital without having to give up equity, take on debt, or lay off employees.
We have scaled from $0 to a multi-eight-figure run rate in a matter of months
We have raised from top-tier investors, including Floodgate, Afore, Ludlow, and Hustle Fund
We are small enough that you will carry outsized responsibility and grow as quickly as the company does
You will partner with and build for some of the fastest and most important companies in the world
You will help build a massive, category-defining business from the ground floor
Replay turns sensitive internal enterprise data into de-identified datasets without destroying the structure and meaning that make the data valuable. That creates a difficult measurement problem. A system can improve aggregate F1 while missing a high-risk slice, remove more sensitive information while also destroying useful context, or pass one stage while defects escape somewhere else in the pipeline.
As Replay's first Data Scientist focused on evaluation, you will establish how we know whether that data is actually getting better. You will build the datasets, experiments, quality measures, and feedback loops that expose hidden failures, accelerate model and pipeline improvement, and give the team confidence in what it delivers.
This is a hands-on, zero-to-one role at the intersection of data science, AI, and a real production system. You will write Python and SQL, construct evaluation corpora, study failure patterns, design comparisons, calibrate human and model-based judgments, and turn the result into a clear decision. The questions are scientifically difficult, but the output must be practical enough to change what the team builds and ships.
You will work closely with Machine Learning, Product Engineering, Data Engineering, Security, Quality, domain experts, and the team making delivery decisions. Machine Learning Engineers own changing model behavior. You own the credibility of the evidence used to decide whether a model, pipeline, or delivery change actually made the data safer or more useful.
Did a higher NER or entity-resolution score actually reduce sensitive misses across the messages, documents, tables, and providers that matter?
Is a new model finding more sensitive information, or simply removing more of the useful structure our customers need?
Can we trust a golden dataset, a human review process, or an LLM judge enough to use it for a release decision?
Which customer, modality, entity, language, or format slices are hidden by a strong aggregate result?
Where did a quality loss enter between source data, processing, de-identification, review, and delivery?
What is the smallest credible experiment that would tell us whether to ship, revise, or stop a change?
Define what high-quality and safe-to-deliver data mean across de-identification, structure preservation, semantic coherence, and customer utility
Design representative samples and build golden, adversarial, replay, and production-like corpora with explicit provenance, labeling policy, agreement, adjudication, and versioning
Turn ambiguous concepts such as “useful,” “clean,” or “safe” into measurable claims with known uncertainty and clear decision consequences
Evaluate detectors, models, prompts, judges, thresholds, review workflows, and pipeline changes using comparisons that can support a real decision
Break aggregate results into the modalities, providers, entity classes, customer contexts, languages, formats, and risk tiers that reveal consequential failures
Connect local measures to escaped sensitive information, avoidable over-redaction, preserved data utility, review burden, rework, and delivery acceptance
Build reproducible analysis, evaluation pipelines, and high-fidelity environments using Python, SQL, synthetic data, historical replay, seeded failures, and programmatic verifiers
Establish holdout and evaluation practices that keep the evidence trustworthy while model and product teams iterate quickly
Use modern AI tools deeply for analysis, corpus development, coding, review, and hypothesis generation while independently verifying their output
The team has a decision-grade baseline for a priority Clean Data quality claim and trusts it enough to use in model, pipeline, release, and delivery decisions
Improvements are judged by the slices and failure costs that matter, not only by an aggregate benchmark
The company can distinguish a true gain from label noise, sample bias, leakage, evaluator error, or a shifted workload
Changes that improve one stage cannot hide escaped defects, over-redaction, utility loss, or review burden somewhere else
At least one consequential decision changes because the evidence reveals a risk, tradeoff, or opportunity that was previously unclear
Evaluation becomes faster and more repeatable without sacrificing independence or rigor
Quality claims communicate uncertainty honestly and remain understandable to engineers, customers, and risk owners
You have at least three years of professional experience in applied science, data science, machine learning, quantitative research, or a closely related role
You have designed evaluations or experiments that changed a product, model, release, or operational decision
You understand sampling, uncertainty, precision, recall, F1, calibration, agreement, class imbalance, distribution shift, and imperfect labels
You can investigate messy, multi-stage data systems and determine where an apparent gain or loss actually came from
You are comfortable writing Python and SQL and building reproducible technical artifacts rather than handing requirements to an engineering team
You can protect the independence of an evaluation while collaborating closely with the people whose work it evaluates
You have startup experience and enjoy broad ownership, changing context, and building the measurement foundation while decisions are already moving quickly
You use AI tools fluently but do not confuse an articulate model output with valid evidence
You communicate uncertainty and difficult findings directly, without hiding behind false precision
You want to optimize models as your primary job rather than determine whether changes actually improve delivered data
You prefer descriptive dashboards that stop short of changing a decision
You treat labels, benchmarks, or model-based judges as ground truth without investigating how they fail
You need a perfectly defined dataset and research plan before you can make progress
You are uncomfortable disagreeing with a technically strong team when the evidence does not support its conclusion
You do not want AI tools to be part of your daily scientific and technical workflow
Experience evaluating NER, entity resolution, information extraction, document understanding, multimodal, retrieval, or LLM systems
Experience with privacy, de-identification, data quality, model risk, safety, or other high-trust decision systems
Experience designing human-review, adjudication, weak-supervision, or active-learning systems
Experience building adversarial corpora, replay systems, simulation environments, programmatic verifiers, or model-judge evaluations
Experience connecting offline measures to escaped defects, customer outcomes, review effort, or preserved data utility
Experience measuring quality across multi-stage batch or data pipelines
New Data Scientist postings at the employers we follow are up 3% over the last 6 weeks compared with the 6 before.
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
We file this job under Data Scientist, by the work the posting describes.
Data scientists use statistics and models to answer harder questions, such as whether a change worked, what demand will look like next quarter, or which customers are likely to leave.
Postings usually ask for Python or R, SQL, statistics, experiment design and common machine learning methods.
Unlike an ML engineer, a data scientist's main output is often an analysis or a first model, not software running inside a product.
Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-2051.
Selected passages from the saved posting. Check the full description for conditions and exceptions.
No pay amount identified in the saved description.
New York, United States
Working pattern and location restrictions need checking in the full posting.
No clear work-authorization passage found. Eligibility is unconfirmed.