Time since we first saw it
Day 2
Early. Similar jobs usually come down between day 6 and day 69.
From 4,296 similar postings that came down. Coming down is not always a hire.
Senior ML/Evaluation Engineer
- Pay
- Pay not listed
- Work setting
Remote — work setting source
Work setup in the employer’s posting: TELECOMMUTE
From the employer’s posting- Employment
- Not stated
What you’ll work on
Full postingSet up AWS AgentCore Evaluation in on-demand mode for CI/CD gates and online mode for production
Design LLM-as-judge evaluators such as helpfulness and correctness
Build custom Python Lambda evaluators for deterministic quality checks
Define evaluation levels for per-response, per-invocation, and workflow-session checks
Write the enterprise evaluation standard with mandatory dimensions and pass/fail criteria
From the employer’s posting
AWS AgentCore Evaluation (on-demand mode for CI/CD gates, online mode for production sampling)
LLM-as-judge evaluator design (built-in AgentCore evaluators — helpfulness, correctness)
Custom code-based Lambda evaluators (Python — deterministic checks)
Evaluation levels (TRACE for per-response, TOOL_CALL for per-invocation, SESSION for workflow)
Enterprise evaluation standard authoring (mandatory dimensions, pass/fail criteria)
Tools in this posting
- Python
- AWS
Source — Tool mentions in context
We are building an Enterprise Agent Development Platform, a production-grade, cloud-native ecosystem that enables engineering teams to define, orchestrate, deploy, and observe AI agents at scale. The platform standardizes agent development across the organization. The initiative spans agent framework design, runtime architecture, marketplace integration, CI/CD automation, and enterprise-grade observability, reducing agent development from months to days while enforcing consistent security, quality, and governance standards. Responsibilities AWS AgentCore Evaluation (on-demand mode for CI/CD gates, online mode for production sampling) LLM-as-judge evaluator design (built-in AgentCore evaluators — helpfulness, correctness) Custom code-based Lambda evaluators (Python — deterministic checks) Evaluation levels (TRACE for per-response, TOOL_CALL for per-invocation, SESSION for workflow) OTel spans from AWS AgentCore Observability as evaluation input Enterprise evaluation standard authoring (mandatory dimensions, pass/fail criteria) Requirements 5+ years ML engineering or AI platform engineering LLM evaluation framework design and implementation Custom evaluator implementation for deterministic quality checks CI/CD deployment gate design for ML model or agent quality Nice to have AWS AgentCore Evaluation API hands-on (CreateEvaluation, GetEvaluationResult) AWS Bedrock Guardrails for PII detection evaluator integration CloudWatch metrics output from AgentCore Evaluation for online mode
Posting record
Report an errorActively hiringFirst seen by us 2 days ago, still listed, and not a repost of an earlier posting.
- Status in our records
- Live
- First seen by us
- Nov 1, 20262 days ago
- Recorded sightings
- 4
- Last seen by us
- Nov 4, 2026
- Employer says posted
- Sep 29, 2026
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
How this job compares
EPAM Systems is hiring
656 open jobs
Open jobs by the week we first saw them, last 12 weeks.
Hiring recordYour next step
Check the employer’s posting for the current role and application details.
Already applied? Track this application
About the role
We file this job under ML Engineer, by the work the posting describes.
ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.
Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.
A data scientist often works out which model to build; an ML engineer makes it run reliably in production.
US government figures for computer and information research scientists
- Median yearly pay, May 2025
- $140,300
- Expected change, 2025 to 2035
- 21.8% more jobsagainst 3.5% for all jobs
The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.
Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.
Job description
We are building an Enterprise Agent Development Platform, a production-grade, cloud-native ecosystem that enables engineering teams to define, orchestrate, deploy, and observe AI agents at scale. The platform standardizes agent development across the organization. The initiative spans agent framework design, runtime architecture, marketplace integration, CI/CD automation, and enterprise-grade observability, reducing agent development from months to days while enforcing consistent security, quality, and governance standards. Responsibilities AWS AgentCore Evaluation (on-demand mode for CI/CD gates, online mode for production sampling) LLM-as-judge evaluator design (built-in AgentCore evaluators — helpfulness, correctness) Custom code-based Lambda evaluators (Python — deterministic checks) Evaluation levels (TRACE for per-response, TOOL_CALL for per-invocation, SESSION for workflow) OTel spans from AWS AgentCore Observability as evaluation input Enterprise evaluation standard authoring (mandatory dimensions, pass/fail criteria) Requirements 5+ years ML engineering or AI platform engineering LLM evaluation framework design and implementation Custom evaluator implementation for deterministic quality checks CI/CD deployment gate design for ML model or agent quality Nice to have AWS AgentCore Evaluation API hands-on (CreateEvaluation, GetEvaluationResult) AWS Bedrock Guardrails for PII detection evaluator integration CloudWatch metrics output from AgentCore Evaluation for online mode
Source details
Source notes
Source excerptsSelected passages from the saved posting. Check the full description for conditions and exceptions.
- Pay
No pay amount identified in the saved description.
- Location & working pattern
Location not supplied.
Working pattern and location restrictions need checking in the full posting.
- Work authorization
No clear work-authorization passage found. Eligibility is unconfirmed.
AI answers unavailable
We couldn’t identify enough role detail in this saved description to support an AI answer. Read the full posting