Back to jobs

Staff Data Scientist, Agent Improvement Loop

Pay
Pay not listed
Work setting
Not stated
Employment
Not stated

What you’ll work on

Full posting
  • Own evals strategy: pre-release testing, production monitoring, and conversation judges

  • Run continuous error analysis on agent traces and rank improvement opportunities

  • Turn findings into fixes and new evals, then measure results with before-and-after methods

  • Calibrate and validate LLM judges against human review

  • Write production-quality Python and TypeScript for evals and analysis tools

From the employer’s posting
Own the evals strategy for our agents: pre-release behavioral testing, production monitoring, and the judges that score conversations.
Run continuous error analysis on agent traces at scale. Find failure patterns, attribute them to skills, prompts or releases, and rank improvement opportunities by impact.
Close the loop. Turn findings into fixes and new evals, ship them, and measure the result with sound before-and-after and causal methods.
Calibrate and validate LLM judges against human review.
Write production-quality Python and TypeScript for evals and analysis tooling, with tests and CI
Apply at Yalo

Posting record

Report an error

Actively hiringFirst seen by us 11 days ago, still listed, and not a repost of an earlier posting.

Status in our records
Live
First seen by us
Nov 11, 202611 days ago
Recorded sightings
9
Last seen by us
Nov 21, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 11

Similar jobs usually come down between day 7 and day 60.

From 12,339 similar postings that came down. Coming down is not always a hire.

Yalo is hiring

1 open job

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • SQL
  • TypeScript
  • MLflow
  • SageMaker
Where the posting names them
- Calibrate and validate LLM judges against human review. Measure their stability, report their limits, and set the bar for what counts as evidence at Yalo. - Write production-quality Python and TypeScript for evals and analysis tooling, with tests and CI, so it can live in a shared codebase. - Mentor data scientists and engineers on measurement, experimentation and evaluation practice.
- Hands-on experience evaluating LLM-based systems or agents: eval design, LLM-as-judge, rubric design, and judge calibration. - Expert in Python and SQL. Comfortable in TypeScript, or ready to be. - Strong production engineering habits: version control, code review, testing, CI/CD, and shipping tools others use.
- Causal inference and experimentation skills: A/B tests, difference-in-differences, pre/post analysis, and the judgment to say what the data can't show. - Exposure to MLOps or LLMOps platforms (Vertex, SageMaker, MLflow, or similar) and big data tools. - Familiarity with AI safety and responsible AI practices.

Your next step

Complete your application on job-boards.greenhouse.io. The employer’s form will show what is required.

Applying through Greenhouse

  • No account needed: the form is one page, with an “Autofill my application” option that reads your resume.
  • Greenhouse doesn’t read resumes over 2.5 MB, or details placed in a header, footer, text box, table or second column. The file still attaches, but a recruiter has to type you in by hand.

Already applied? Track this application

Job description

View original posting ↗

Yalo

Hi! This is Yalo! We are on a mission to bring conversational commerce to the world...

Remember how it used to be to interact with businesses that knew and understood you, that could recommend exactly what you needed, and that with a simple message could get you what you wanted??? Yep... neither do we. That is why at Yalo we are marrying the scale of digital commerce with the personalization and simplicity of conversations to help companies delight their users.

We know that traditional SAAS companies focus on first-world problems... we don't! Having started in Latin America, our roots are in Emerging Markets and therefore we care about bringing amazing experiences to a population that traditionally has been underserved, such as the small shop owner in Brazil who is ordering online for the first time.

If you're looking for a place to make things happen, learn fast, and impact emerging markets in a way that hasn't been done before, look no further. 💫

Come Join us in our mission of improving billions of lives through the power of conversational commerce!

Your mission 🚀

As a Staff Data Scientist, you make our AI agents measurably better, and you prove it. You own how Yalo evaluates agents: evals, trace and error analysis, and the improvement loop that turns what we find in production into fixes to our agent skills, harness, CLI and customers' conversational flows. You work with Product, Engineering, Delivery and Customer Success. Your job is to show what moved the metric, not to guess.

What are the responsibilities for this role?

  • Own the evals strategy for our agents: pre-release behavioral testing, production monitoring, and the judges that score conversations.
  • Run continuous error analysis on agent traces at scale. Find failure patterns, attribute them to skills, prompts or releases, and rank improvement opportunities by impact.
  • Close the loop. Turn findings into fixes and new evals, ship them, and measure the result with sound before-and-after and causal methods.
  • Improve our agent skills, harness and CLI using trace evidence. Work through the owning teams' repos and review process.
  • Partner with delivery and customer teams to take real customer flows from underperforming to reliably healthy, and make the process repeatable for others.
  • Calibrate and validate LLM judges against human review. Measure their stability, report their limits, and set the bar for what counts as evidence at Yalo.
  • Write production-quality Python and TypeScript for evals and analysis tooling, with tests and CI, so it can live in a shared codebase.
  • Mentor data scientists and engineers on measurement, experimentation and evaluation practice.
  • Help shape quarterly and annual planning for agent quality.

Job Requirements (Must have)

  • BS or MS in a technical field (Computer Science, Statistics, Applied Math, Operations Research, Economics, etc.).
  • 8+ years delivering data science solutions with measurable business impact in cross-functional teams.
  • Hands-on experience evaluating LLM-based systems or agents: eval design, LLM-as-judge, rubric design, and judge calibration.
  • Expert in Python and SQL. Comfortable in TypeScript, or ready to be.
  • Strong production engineering habits: version control, code review, testing, CI/CD, and shipping tools others use.
  • Experience with GenAI systems in production: prompt and tool design, agent frameworks (e.g., LangGraph), and LLM observability tooling (e.g., LangSmith).
  • Skilled at turning messy trace and conversation data into clear findings for technical and non-technical audiences.
  • Strong product and business sense. You frame quality work in terms of customer outcomes.

Nice to have:

  • Experience with conversational AI, customer-facing agents or tool-using agents.
  • NLP and clustering for failure-pattern discovery.
  • Causal inference and experimentation skills: A/B tests, difference-in-differences, pre/post analysis, and the judgment to say what the data can't show.
  • Exposure to MLOps or LLMOps platforms (Vertex, SageMaker, MLflow, or similar) and big data tools.
  • Familiarity with AI safety and responsible AI practices.
  • Comfort in an ambiguous, fast-changing environment.
  • Experience mentoring other data scientists.

What do we offer? 

  • Unlimited PTO policy
  • Competitive rewards on the market range
  • Remote working is available (-+3 hours CT)
  • Flexible time (driven by results)
  • Start-up environment
  • International teamwork
  • You and nothing else limit your career here

 

We care,

We keep it simple,

We make it happen,

We strive for excellence. 

At Yalo, we are dedicated to creating a workplace that embodies our core values: caring, initiative, excellence, and simplicity. We believe in the power of diversity and inclusivity, where everyone's unique perspectives, experiences, and talents contribute to our collective success. As we embrace and respect our differences, we strive to create something extraordinary for the benefit of all.
We are proud to be an Equal Opportunity Employer, providing equal opportunities to individuals regardless of race, color, religion, national or ethnic origin, gender, sexual orientation, gender identity or expression, age, disability, protected veteran status, or any other legally protected characteristic. Our commitment to fairness and equality is a fundamental pillar of our company.


At Yalo, we uphold a culture of excellence. We constantly challenge ourselves to go above and beyond, delivering remarkable results and driving innovation. We encourage each team member to take initiative and make things happen, empowering them to bring their best ideas forward and contribute to our shared goals.

Demand for Data Scientist roles

New Data Scientist postings at the employers we follow are up 3% over the last 6 weeks compared with the 6 before.

Weekly new Data Scientist postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 1,389, the most 2,932, and the latest week had 1,705.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under Data Scientist, by the work the posting describes.

Data scientists use statistics and models to answer harder questions, such as whether a change worked, what demand will look like next quarter, or which customers are likely to leave.

Postings usually ask for Python or R, SQL, statistics, experiment design and common machine learning methods.

Unlike an ML engineer, a data scientist's main output is often an analysis or a first model, not software running inside a product.

US government figures for data scientists

Median yearly pay, May 2025
$120,230
Expected change, 2025 to 2035
34.6% more jobsagainst 3.5% for all jobs

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-2051.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

LATAM

- Competitive rewards on the market range - Remote working is available (-+3 hours CT) - Flexible time (driven by results)
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.