Back to jobs

Senior Data Scientist AI Evaluation

Pay
Pay not listed
Work setting
Remote — work setting source
Listed location: Remote - Americas
Read the full posting
Employment
Not stated

What you’ll work on

Full posting
  • Define ground truth, metrics, and scoring methods for models and agents

  • Build repeatable eval loops that track quality and catch regressions before release

  • Work with engineering to operationalize eval harnesses

  • Turn eval results into recommendations for improving the systems

  • Set evaluation guidelines, documentation, and review practices

From the employer’s posting
Define ground truth, metrics, and scoring methods for models and agents.
Build repeatable eval loops: Track quality over time and catch regressions before release.
Work with engineering and analytics engineering to operationalize eval harnesses.
Translate eval results into actionable recommendations for system improvements.
Set evaluation guidelines, documentation, and review practices.

What you’ll bring

All qualifications

Core experience

  • 6–10 years in quantitative data science or ML, with focus on measurement or evaluation
  • Strong statistical and ML foundation, including sample sizing, significance and validating automated graders
  • Proficiency in Python and SQL
  • Track record of quantitative measurement rigor, such as LLM evaluation or experimentation
Qualification wording
6–10 years in quantitative data science or ML, with focused experience in measurement or evaluation.
Strong statistical and ML foundation—you treat evaluations as experiments (sample sizing, confidence intervals, significance, handling non-determinism) and validate automated graders against human ground truth.
Proficiency in Python and SQL, with experience evaluating models in production environments.
Track record of quantitative measurement rigor (e.g., LLM/model evaluation, metric validation, or experimentation).

About Alpaca

Our global team is a diverse group of experienced engineers, traders, and brokerage professionals who are working to achieve our mission of opening financial services to everyone on the planet.

In the employer’s words · Read in context

Apply at Alpaca

Posting record

Report an error
Status in our records
Live
First seen by us
Oct 8, 202645 days ago
Recorded sightings
41
Last seen by us
Nov 21, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 45

Similar jobs usually come down between day 7 and day 60.

At Alpaca, jobs usually come down by about day 101.

From 12,339 similar postings that came down, and 8 of Alpaca’s. Coming down is not always a hire.

Alpaca is hiring

5 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • Hiring at a steady pace
  • Takes jobs down at a usual pace
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • SQL
Where the posting names them
- Strong statistical and ML foundation—you treat evaluations as experiments (sample sizing, confidence intervals, significance, handling non-determinism) and validate automated graders against human ground truth. - Proficiency in Python and SQL, with experience evaluating models in production environments. - Strong judgment in defining quality metrics and ground truth for ambiguous outputs.
- Hands-on LLM/agent evaluation in production, including eval harnesses, LLM-as-judge calibration, and CI regression gates. - Experience evaluating text-to-SQL, analytics agents, or other systems where correctness is verifiable against data. - Background in fintech, brokerage, or other domains where a wrong answer has real business or risk consequences.

Your next step

Complete your application on job-boards.greenhouse.io. The employer’s form will show what is required.

Applying through Greenhouse

  • No account needed: the form is one page, with an “Autofill my application” option that reads your resume.
  • Greenhouse doesn’t read resumes over 2.5 MB, or details placed in a header, footer, text box, table or second column. The file still attaches, but a recruiter has to type you in by hand.

Already applied? Track this application

Job description

View original posting ↗

Who We Are:

Alpaca is a US-headquartered, global leader in agent-first brokerage infrastructure for stocks, ETFs, options, crypto, fixed income, 24/5 trading, and more.

Amongst our subsidiaries, Alpaca is a licensed financial services company, serving hundreds of financial institutions across 40 countries with our institutional-grade APIs. This includes broker-dealers, investment advisors, wealth managers, hedge funds, and crypto exchanges, totalling over 10 million brokerage accounts.

Our global team is a diverse group of experienced engineers, traders, and brokerage professionals who are working to achieve our mission of opening financial services to everyone on the planet. We're deeply committed to open-source contributions and fostering a vibrant community, continuously enhancing our award-winning, developer-friendly API and the robust infrastructure behind it.

Alpaca is proudly backed by $400 million in funding from top-tier global investors including Portage Ventures, Spark Capital, Tribe Capital, Social Leverage, Horizons Ventures, Opera Tech Ventures, SBI Group, Derayah Financial, Unbound, Peak XV, Elefund, and Y Combinator.

Our Team Members:

We're a dynamic team of 400+ globally distributed members who thrive working from our favorite places around the world, with teammates spanning the USA, Canada, Japan, Hungary, Nigeria, Brazil, the UK, and beyond!

We're searching for passionate individuals eager to contribute to Alpaca's rapid growth. If you align with our core values—Stay Curious, Have Empathy, and Be Accountable—and are ready to make a significant impact, we encourage you to apply.

Your Role: We're looking for a Senior Data Scientist, AI Evaluation to design how Alpaca measures whether our models and agents are actually right. You'll be a senior individual contributor who turns ambiguous quality questions into ground truth, scoring methods, and eval loops that the company can trust—and uses those results to make the systems better. You'll build on an established data foundation, so the focus is raising quality and speeding up safe rollout. You'll own the quality bar, independent of the teams that build and optimize those systems.

 

This role is for someone who cares as much about whether an answer is correct as about whether a model can generate one. You'll partner with Product, Engineering, Analytics Engineering, and business stakeholders to define what "good" looks like, build the evaluations that test it, and close the loop so evals drive iteration. If you have a strong quantitative background, have shipped rigorous, measurable work (evaluation, experimentation, or model validation), and want ownership over a greenfield eval practice at a fast-growing brokerage-infrastructure company, this is the role.

 

What You'll Do

  • Design AI evaluations: Define ground truth, metrics, and scoring methods for models and agents.
  • Build repeatable eval loops: Track quality over time and catch regressions before release.
  • Partner on infrastructure: Work with engineering and analytics engineering to operationalize eval harnesses.
  • Drive iteration: Translate eval results into actionable recommendations for system improvements.
  • Establish quality standards: Set evaluation guidelines, documentation, and review practices.
  • Mentor and align: Foster evaluation best practices and build a culture of measurable AI quality across the team.

What We're Looking For

  • Track record of quantitative measurement rigor (e.g., LLM/model evaluation, metric validation, or experimentation).
  • Strong statistical and ML foundation—you treat evaluations as experiments (sample sizing, confidence intervals, significance, handling non-determinism) and validate automated graders against human ground truth.
  • Proficiency in Python and SQL, with experience evaluating models in production environments.
  • Strong judgment in defining quality metrics and ground truth for ambiguous outputs.
  • Excellent communication and cross-functional collaboration skills to align technical teams and leadership.
  • Strong problem-solving ability in fast-paced, greenfield environments.
  • 6–10 years in quantitative data science or ML, with focused experience in measurement or evaluation. A quantitative degree is a plus; equivalent industry experience is equally welcome.

Nice to Have

  • Hands-on LLM/agent evaluation in production, including eval harnesses, LLM-as-judge calibration, and CI regression gates.
  • Experience evaluating text-to-SQL, analytics agents, or other systems where correctness is verifiable against data.
  • Background in fintech, brokerage, or other domains where a wrong answer has real business or risk consequences.
  • Fluency with AI tools in research and engineering workflows.

How We Take Care of You:

  • Competitive Salary & Stock Options
  • Health Benefits
  • New Hire Home-Office Setup: One-time USD $500
  • Monthly Stipend: USD $150 per month via a Brex Card

Alpaca is proud to be an equal opportunity workplace dedicated to pursuing and hiring a diverse workforce.

Recruitment Privacy Policy

Demand for Data Scientist roles

New Data Scientist postings at the employers we follow are up 3% over the last 6 weeks compared with the 6 before.

Weekly new Data Scientist postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 1,389, the most 2,932, and the latest week had 1,705.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under Data Scientist, by the work the posting describes.

Data scientists use statistics and models to answer harder questions, such as whether a change worked, what demand will look like next quarter, or which customers are likely to leave.

Postings usually ask for Python or R, SQL, statistics, experiment design and common machine learning methods.

Unlike an ML engineer, a data scientist's main output is often an analysis or a first model, not software running inside a product.

US government figures for data scientists

Median yearly pay, May 2025
$120,230
Expected change, 2025 to 2035
34.6% more jobsagainst 3.5% for all jobs

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-2051.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Remote - Americas

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.