Time since we first saw it
Day 12
Similar jobs usually come down between day 4 and day 77.
From 51 similar postings that came down. Coming down is not always a hire.
Senior ML / Evaluation Engineer
- Pay
- Pay not listed
- Work setting
Remote — work setting source
Work setup in the employer’s posting: TELECOMMUTE
From the employer’s posting- Employment
- Not stated
What you’ll work on
Full postingDesign and implement evaluation frameworks for LLM apps and AI agents
Develop LLM-as-a-Judge and deterministic, code-based evaluators
Evaluate agent behavior across responses, tool calls, and full workflows
Integrate evaluations into CI/CD pipelines and automated deployment gates
Enable continuous quality monitoring of solutions in production
From the employer’s posting
Design and implement evaluation frameworks for LLM-based applications and AI agents
Develop LLM-as-a-Judge and deterministic, code-based evaluators
Evaluate agent behavior across individual responses, tool calls, and complete workflows
Integrate evaluations into CI/CD pipelines and automated deployment gates
Enable continuous quality monitoring of solutions in production
What you’ll bring
All qualificationsCore experience
- 5+ years in ML, AI, or AI platform engineering
- Strong Python development experience
- Hands-on experience with LLM/GenAI evaluation
- Experience designing and implementing evaluation frameworks
- Experience integrating AI/ML quality checks into CI/CD
Qualification wording
5+ years of experience in ML Engineering, AI Engineering, or AI Platform Engineering
Strong Python development experience
Hands-on experience with LLM/GenAI evaluation
Experience designing and implementing evaluation frameworks
Experience integrating AI/ML quality checks into CI/CD
Posting record
Report an errorActively hiringFirst seen by us 12 days ago, still listed, and not a repost of an earlier posting.
- Status in our records
- Live
- First seen by us
- Nov 1, 202612 days ago
- Recorded sightings
- 13
- Last seen by us
- Nov 14, 2026
- Employer says posted
- Sep 29, 2026
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
How this job compares
EPAM Systems is hiring
673 open jobs
Open jobs by the week we first saw them, last 12 weeks.
Hiring recordSee how this role fits your experience
Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
- Python
- AWS
Where the posting names them
Join the Enterprise Agent Development Platform project at EPAM. We are building a cloud-native platform that enables engineering teams to develop, deploy, and operate AI agents in production. The platform combines modern agent frameworks, AWS infrastructure, CI/CD, observability, and governance to make AI development faster and more reliable. As a Python AI Evaluation Engineer , you will own a key part of this platform: building the capabilities that measure and validate the quality of AI solutions. You will design evaluation approaches, develop custom evaluators, and integrate quality checks into the software delivery lifecycle. This role is a strong fit for an engineer who enjoys solving new GenAI challenges and turning them into practical, automated engineering solutions. Responsibilities Design and implement evaluation frameworks for LLM-based applications and AI agents Develop LLM-as-a-Judge and deterministic, code-based evaluators Build custom Python evaluators for quality and behavioral checks Define evaluation criteria, metrics, thresholds, and acceptance rules Evaluate agent behavior across individual responses, tool calls, and complete workflows Work with OpenTelemetry traces and spans as evaluation data Integrate evaluations into CI/CD pipelines and automated deployment gates Enable continuous quality monitoring of solutions in production Establish reusable evaluation patterns and engineering standards Work closely with AI Engineers, Architects, and Platform Engineers to embed quality into the development process Requirements 5+ years of experience in ML Engineering, AI Engineering, or AI Platform Engineering Strong Python development experience Hands-on experience with LLM/GenAI evaluation Experience designing and implementing evaluation frameworks Experience developing custom or deterministic evaluators Experience integrating AI/ML quality checks into CI/CD Good understanding of LLM and AI agent architectures Nice to have Hands-on experience with AWS AgentCore Evaluation Experience with AWS Bedrock Guardrails, including PII detection Knowledge of CloudWatch metrics and production monitoring Experience with OpenTelemetry Familiarity with LangGraph, Strands Agents, or similar agent frameworks
Your next step
Complete your application on careers.epam.com. The employer’s form will show what is required.
Already applied? Track this application
Has EPAM Systems posted this job before?
Yes. We’ve followed 2 other Senior ML / Evaluation Engineer jobs there since November 2026. All 2 are still up.
About EPAM Systems
- People
- About 3,012 in 2024.Benefit plan filings (Form 5500)
- Company
- Listed as EPAM.
- H-1B visas
- 21 approved for new hires in fiscal 2025.US Citizenship and Immigration Services
Demand for AI Engineer roles
New AI Engineer postings at the employers we follow are up 134% over the last 6 weeks compared with the 6 before.
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
About the role
We file this job under AI Engineer, by the work the posting describes.
AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.
Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.
An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.
Job description
Join the Enterprise Agent Development Platform project at EPAM. We are building a cloud-native platform that enables engineering teams to develop, deploy, and operate AI agents in production. The platform combines modern agent frameworks, AWS infrastructure, CI/CD, observability, and governance to make AI development faster and more reliable. As a Python AI Evaluation Engineer , you will own a key part of this platform: building the capabilities that measure and validate the quality of AI solutions. You will design evaluation approaches, develop custom evaluators, and integrate quality checks into the software delivery lifecycle. This role is a strong fit for an engineer who enjoys solving new GenAI challenges and turning them into practical, automated engineering solutions. Responsibilities Design and implement evaluation frameworks for LLM-based applications and AI agents Develop LLM-as-a-Judge and deterministic, code-based evaluators Build custom Python evaluators for quality and behavioral checks Define evaluation criteria, metrics, thresholds, and acceptance rules Evaluate agent behavior across individual responses, tool calls, and complete workflows Work with OpenTelemetry traces and spans as evaluation data Integrate evaluations into CI/CD pipelines and automated deployment gates Enable continuous quality monitoring of solutions in production Establish reusable evaluation patterns and engineering standards Work closely with AI Engineers, Architects, and Platform Engineers to embed quality into the development process Requirements 5+ years of experience in ML Engineering, AI Engineering, or AI Platform Engineering Strong Python development experience Hands-on experience with LLM/GenAI evaluation Experience designing and implementing evaluation frameworks Experience developing custom or deterministic evaluators Experience integrating AI/ML quality checks into CI/CD Good understanding of LLM and AI agent architectures Nice to have Hands-on experience with AWS AgentCore Evaluation Experience with AWS Bedrock Guardrails, including PII detection Knowledge of CloudWatch metrics and production monitoring Experience with OpenTelemetry Familiarity with LangGraph, Strands Agents, or similar agent frameworks
Source details
Source notes
Source excerptsSelected passages from the saved posting. Check the full description for conditions and exceptions.
- Pay
No pay amount identified in the saved description.
- Location & working pattern
Location not supplied.
Working pattern and location restrictions need checking in the full posting.
- Work authorization
No clear work-authorization passage found. Eligibility is unconfirmed.