Time since we first saw it
Day 2
Early. Similar jobs usually come down between day 6 and day 69.
From 4,296 similar postings that came down. Coming down is not always a hire.
Senior ML / Evaluation Engineer
- Pay
- Pay not listed
- Work setting
- Not stated
- Employment
- Not stated
What you’ll work on
Full postingBuild LLM-as-judge evaluators and deterministic Lambda-based checks
Define enterprise evaluation standards with mandatory dimensions and pass/fail thresholds
Implement CI/CD deployment gates that enforce quality metrics before release
Generate metrics and dashboards from evaluation results in CloudWatch
From the employer’s posting
Build LLM-as-judge evaluators leveraging AWS AgentCore built-in modules and custom logic for correctness and helpfulness checks
Define enterprise evaluation standards, including mandatory dimensions, scoring criteria, and pass/fail thresholds
Implement CI/CD deployment gates using on-demand evaluation modes to enforce quality in automated pipelines
Generate metrics, logs, and dashboards from evaluation outcomes via CloudWatch or equivalent monitoring platforms
Posting record
Report an errorActively hiringFirst seen by us 2 days ago, still listed, and not a repost of an earlier posting.
- Status in our records
- Live
- First seen by us
- Nov 1, 20262 days ago
- Recorded sightings
- 4
- Last seen by us
- Nov 4, 2026
- Employer says posted
- Sep 29, 2026
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
How this job compares
EPAM Systems is hiring
656 open jobs
Open jobs by the week we first saw them, last 12 weeks.
Hiring recordSee how this role fits your experience
Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
- Python
- AWS
Where the posting names them
We're looking for a Senior ML / Evaluation Engineer to join our team in Portugal in a fully remote working mode. In this role, you will own the design and implementation of advanced evaluation frameworks for an Enterprise Agent Development Platform—a production-grade, cloud-native ecosystem enabling scalable, secure AI agent deployment. You will create evaluation strategies that combine LLM-as-judge grading with deterministic checks, define enterprise evaluation standards, and implement CI/CD deployment gates to enforce quality metrics prior to release. This position requires strong expertise in ML system testing, evaluation design, and integration into automated pipelines for agentic environments. Responsibilities Design and implement multi-layer evaluation frameworks for agentic workflows and AI-driven applications Build LLM-as-judge evaluators leveraging AWS AgentCore built-in modules and custom logic for correctness and helpfulness checks Develop deterministic evaluators as AWS Lambda functions for rule-based validation Define enterprise evaluation standards, including mandatory dimensions, scoring criteria, and pass/fail thresholds Implement CI/CD deployment gates using on-demand evaluation modes to enforce quality in automated pipelines Enable online evaluation in production by integrating sampling-based evaluation strategies and PII detection guardrails Incorporate observability signals (OpenTelemetry spans) from AWS AgentCore into grading frameworks for trace-level assessment Generate metrics, logs, and dashboards from evaluation outcomes via CloudWatch or equivalent monitoring platforms Collaborate with platform, orchestration, and DevOps teams to maintain evaluation reliability and scalability Requirements 5+ years of experience in ML engineering, AI evaluation frameworks, or AI platform development Hands-on expertise designing LLM evaluation frameworks (LLM-as-judge and deterministic graders) Practical experience implementing CI/CD deployment gates for ML model or AI agent quality assurance Proficiency in Python for building evaluation logic (deterministic Lambda-based evaluators) Strong understanding of advanced validation dimensions, including multi-turn context integrity and workflow-level scoring Nice to have Familiarity with AWS AgentCore Evaluations API (CreateEvaluation, GetEvaluationResult) Exposure to AWS Bedrock Guardrails for compliance and sensitive data validation Experience integrating evaluation metrics into AWS CloudWatch for monitoring and alerting Knowledge of OTel instrumentation and trace ingestion for quality scoring inputs
Your next step
Complete your application on careers.epam.com. The employer’s form will show what is required.
Already applied? Track this application
Has EPAM Systems posted this job before?
Yes. We’ve followed 2 other Senior ML / Evaluation Engineer jobs there since November 2026. All 2 are still up.
About the role
We file this job under ML Engineer, by the work the posting describes.
ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.
Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.
A data scientist often works out which model to build; an ML engineer makes it run reliably in production.
US government figures for computer and information research scientists
- Median yearly pay, May 2025
- $140,300
- Expected change, 2025 to 2035
- 21.8% more jobsagainst 3.5% for all jobs
The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.
Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.
Job description
We're looking for a Senior ML / Evaluation Engineer to join our team in Portugal in a fully remote working mode. In this role, you will own the design and implementation of advanced evaluation frameworks for an Enterprise Agent Development Platform—a production-grade, cloud-native ecosystem enabling scalable, secure AI agent deployment. You will create evaluation strategies that combine LLM-as-judge grading with deterministic checks, define enterprise evaluation standards, and implement CI/CD deployment gates to enforce quality metrics prior to release. This position requires strong expertise in ML system testing, evaluation design, and integration into automated pipelines for agentic environments. Responsibilities Design and implement multi-layer evaluation frameworks for agentic workflows and AI-driven applications Build LLM-as-judge evaluators leveraging AWS AgentCore built-in modules and custom logic for correctness and helpfulness checks Develop deterministic evaluators as AWS Lambda functions for rule-based validation Define enterprise evaluation standards, including mandatory dimensions, scoring criteria, and pass/fail thresholds Implement CI/CD deployment gates using on-demand evaluation modes to enforce quality in automated pipelines Enable online evaluation in production by integrating sampling-based evaluation strategies and PII detection guardrails Incorporate observability signals (OpenTelemetry spans) from AWS AgentCore into grading frameworks for trace-level assessment Generate metrics, logs, and dashboards from evaluation outcomes via CloudWatch or equivalent monitoring platforms Collaborate with platform, orchestration, and DevOps teams to maintain evaluation reliability and scalability Requirements 5+ years of experience in ML engineering, AI evaluation frameworks, or AI platform development Hands-on expertise designing LLM evaluation frameworks (LLM-as-judge and deterministic graders) Practical experience implementing CI/CD deployment gates for ML model or AI agent quality assurance Proficiency in Python for building evaluation logic (deterministic Lambda-based evaluators) Strong understanding of advanced validation dimensions, including multi-turn context integrity and workflow-level scoring Nice to have Familiarity with AWS AgentCore Evaluations API (CreateEvaluation, GetEvaluationResult) Exposure to AWS Bedrock Guardrails for compliance and sensitive data validation Experience integrating evaluation metrics into AWS CloudWatch for monitoring and alerting Knowledge of OTel instrumentation and trace ingestion for quality scoring inputs
Source details
Source notes
Source excerptsSelected passages from the saved posting. Check the full description for conditions and exceptions.
- Pay
No pay amount identified in the saved description.
- Location & working pattern
Location not supplied.
This passage needs a closer read in the full description.
- Work authorization
No clear work-authorization passage found. Eligibility is unconfirmed.