Back to jobs

AI Evaluation Engineer

Pay
Pay not listed
Work setting
Not stated
Employment
Not stated

What you’ll bring

All qualifications

Core experience

  • Experience 4+ years in software, ML, or data engineering, with at least 1-2 years working hands-on with LLM-based applications (adjust to your market).
  • Strong Python skills and software engineering fundamentals (testing, version control, CI/CD, code review).
  • Experience evaluating agentic systems (multi-step reasoning, tool calling, trajectory evaluation).
  • Practical experience evaluating LLM/RAG systems, including retrieval metrics (recall, precision, MRR/nDCG) and generation metrics.
  • Experience with LLMOps, monitoring and observability, and production feedback loops.
  • Hands-on experience with LLM-as-a-judge design, prompt engineering, and judge validation.
Qualification wording
Experience 4+ years in software, ML, or data engineering, with at least 1-2 years working hands-on with LLM-based applications (adjust to your market).
Strong Python skills and software engineering fundamentals (testing, version control, CI/CD, code review).
Experience evaluating agentic systems (multi-step reasoning, tool calling, trajectory evaluation).
Practical experience evaluating LLM/RAG systems, including retrieval metrics (recall, precision, MRR/nDCG) and generation metrics.
Experience with LLMOps, monitoring and observability, and production feedback loops.
Hands-on experience with LLM-as-a-judge design, prompt engineering, and judge validation.
Education & alternatives
QUALIFICATIONS Education & Licensing Master’s degree in a computer science or a quantitative discipline from an accredited college or university preferred. Experience 4+ years in software, ML, or data engineering, with at least 1-2 years working hands-on with LLM-based applications (adjust to your market).
Apply at Sedgwick

Posting record

Report an error
Status in our records
Live
First seen by us
Nov 4, 202618 days ago
Recorded sightings
13
Last seen by us
Nov 16, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 18

Up longer than most. Similar jobs usually come down between day 1 and day 15.

At Sedgwick, jobs usually come down by about day 20.

From 1,376 similar postings that came down, and 23 of Sedgwick’s. Coming down is not always a hire.

Sedgwick is hiring

16 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • Hiring at a steady pace
  • Takes jobs down at a usual pace
  • 6% of open jobs were reposted
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • AWS
  • Azure
  • GCP
  • MLflow
Where the posting names them
- A strong understanding of statistical validation, model evaluation metrics, and the ability to partner with Data Scientists to optimize model performance - Strong Python skills and software engineering fundamentals (testing, version control, CI/CD, code review). - Practical experience evaluating LLM/RAG systems, including retrieval metrics (recall, precision, MRR/nDCG) and generation metrics.
- Familiarity with at least one eval or observability tool (e.g., Ragas, DeepEval, promptfoo, LangSmith, Langfuse, MLflow, or equivalent). - Experience with cloud platforms (Azure, AWS, or GCP) and their managed LLM services. - Clear communication skills and the ability to work with engineers, product owners, SMEs, and risk stakeholders.
- Working knowledge of statistics and experimental design: sampling, confidence intervals, significance testing, inter-rater agreement. - Familiarity with at least one eval or observability tool (e.g., Ragas, DeepEval, promptfoo, LangSmith, Langfuse, MLflow, or equivalent). - Experience with cloud platforms (Azure, AWS, or GCP) and their managed LLM services.

Your next step

Complete your application on sedgwick.wd1.myworkdayjobs.com. The employer’s form will show what is required.

Applying through Workday

  • You create an account with each employer before you can upload anything. Every Workday employer has its own account, so one sign-up doesn’t carry over.
  • “Autofill with Resume” comes right after sign-in, before the My Information and My Experience steps. Check every field it fills in those steps, since that’s what recruiters see.
  • Your application status lives in that employer’s Candidate Home, under the same sign-in.

Already applied? Track this application

Job description

View original posting ↗

By joining Sedgwick, you'll be part of something truly meaningful. It’s what our 33,000 colleagues do every day for people around the world who are facing the unexpected. We invite you to grow your career with us, experience our caring culture, and enjoy work-life balance. Here, there’s no limit to what you can achieve.

Newsweek Recognizes Sedgwick as America’s Greatest Workplaces National Top Companies

Certified as a Great Place to Work®

Fortune Best Workplaces in Financial Services & Insurance

AI Evaluation Engineer

PRIMARY PURPOSE: The AI Evaluation Engineer designs, builds, and operates the evaluation systems that determine whether our GenAI solutions are accurate, safe, reliable, and ready for production. You will serve multiple product pods, establishing consistent evaluation standards, automated pipelines, and quality gates, and you will help teams measure and improve LLM, RAG, and agentic applications throughout their lifecycle.


Fully remote role, open to U.S. based candidates.


ESSENTIAL FUNCTIONS AND RESPONSIBILITIES

  • Establishes standardized, automated evaluation frameworks for LLMs and traditional ML models, including LLM as a judge patterns, ROUGE, METEOR, and statistical validation to ensure performance, reliability, and readiness prior to production release.
  • Deploys comprehensive observability solutions to monitor model drift, hallucination risk, latency, throughput, and token consumption, balancing model quality with infrastructure cost and operational efficiency.
  • Define evaluation methodologies and metrics for LLM, RAG, and agent-based applications (e.g., faithfulness, groundedness, relevance, task completion, tool-use correctness, safety).
  • Build reusable evaluation frameworks, templates, and libraries that pods can adopt with minimal effort.
  • Partner with product owners and SMEs to translate business requirements into measurable quality criteria and acceptance thresholds.
  • Develop automated evaluation pipelines integrated into CI/CD, with regression testing and quality gates for prompt, model, and retrieval changes.
  • Implement offline evaluation (benchmark and golden datasets) and online evaluation (production monitoring, sampling, drift and quality alerts).
  • Support A/B and shadow testing for model, prompt, and architecture changes.
  • Design, validate, and maintain LLM-as-a-judge evaluators, including rubric design, bias mitigation (position, verbosity, self-preference), and calibration against human labels.
  • Establish human annotation workflows, labeling guidelines, and inter-annotator agreement practices with SMEs.
  • Track judge reliability over time and re-calibrate as models and use cases change.
  • Build and curate golden datasets, adversarial and edge-case suites, and synthetic test data generation approaches.
  • Apply approved data-handling practices (de-identification, synthetic data, access controls) for any dataset derived from sensitive or regulated data.
  • Implement red-teaming, hallucination, toxicity, PII leakage, prompt-injection, and jailbreak testing.
  • Produce evaluation evidence and reporting that supports security, privacy, compliance, and model risk reviews.
  • Contribute to responsible AI standards and release-readiness criteria.
  • Embed with pods to onboard them to evaluation practices, then coach teams toward ownership of their own eval suites.
  • Build dashboards and reporting that give engineering and business leaders transparent quality and cost/latency trends.
  • Evaluate and recommend eval tooling and platforms; document standards and best practices.

ADDITIONAL FUNCTIONS and RESPONSIBILITIES

  • Performs other duties as assigned.
  • Travel as required.

QUALIFICATIONS

Education & Licensing
Master’s degree in a computer science or a quantitative discipline from an accredited college or university preferred.

Experience
4+ years in software, ML, or data engineering, with at least 1-2 years working hands-on with LLM-based applications (adjust to your market).

Skills & Knowledge

  • Professional experience with evaluation and observability frameworks like LangSmith, Arize Phoenix, or WhyLabs
  • A strong understanding of statistical validation, model evaluation metrics, and the ability to partner with Data Scientists to optimize model performance
  • Strong Python skills and software engineering fundamentals (testing, version control, CI/CD, code review).
  • Practical experience evaluating LLM/RAG systems, including retrieval metrics (recall, precision, MRR/nDCG) and generation metrics.
  • Hands-on experience with LLM-as-a-judge design, prompt engineering, and judge validation.
  • Working knowledge of statistics and experimental design: sampling, confidence intervals, significance testing, inter-rater agreement.
  • Familiarity with at least one eval or observability tool (e.g., Ragas, DeepEval, promptfoo, LangSmith, Langfuse, MLflow, or equivalent).
  • Experience with cloud platforms (Azure, AWS, or GCP) and their managed LLM services.
  • Clear communication skills and the ability to work with engineers, product owners, SMEs, and risk stakeholders.
  • Excellent oral and written communication skills, including presentation skills
  • Ability to work in a team environment
  • Ability to meet or exceed Performance Competencies

Preferred

  • Experience evaluating agentic systems (multi-step reasoning, tool calling, trajectory evaluation).
  • Background in ML evaluation, search/relevance, NLP, or QA automation.
  • Experience with LLMOps, monitoring and observability, and production feedback loops.
  • Familiarity with AI risk and governance frameworks (e.g., NIST AI RMF, ISO/IEC 42001) and secure development practices in regulated environments.
  • Experience with data annotation workflows and working with SMEs.
  • Knowledge of enterprise controls: SSO, RBAC, private networking, logging, and encryption.

SUCCESS MEASURES (first 12 months)

  • Standard evaluation framework adopted by pods.
  • Automated quality gates in CI/CD for production-bound solutions.
  • Calibrated LLM judges with documented agreement against human labels.
  • Reduction in regressions reaching production and in time to validate model or prompt changes.

WORK ENVIRONMENT
When applicable and appropriate, consideration will be given to reasonable accommodations.
Mental: Clear and conceptual thinking ability; excellent judgment, troubleshooting, problem solving, analysis, and discretion; ability to handle work-related stress; ability to handle multiple priorities simultaneously; and ability to meet deadlines
Physical: Computer keyboarding, travel as required
Auditory/Visual: Hearing, vision and talking

The statements contained in this document are intended to describe the general nature and level of work being performed by a colleague assigned to this description. They are not intended to constitute a comprehensive list of functions, duties, or local variances. Management retains the discretion to add or to change the duties of the position at any time.

Sedgwick is an Equal Opportunity Employer and a Drug-Free Workplace.

If you're excited about this role but your experience doesn't align perfectly with every qualification in the job description, consider applying for it anyway! Sedgwick is building a diverse, equitable, and inclusive workplace and recognizes that each person possesses a unique combination of skills, knowledge, and experience. You may be just the right candidate for this or other roles.

Demand for AI Engineer roles

New AI Engineer postings at the employers we follow are up 340% over the last 6 weeks compared with the 6 before.

Weekly new AI Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 78, the most 803, and the latest week had 438.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under AI Engineer, by the work the posting describes.

AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.

Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.

An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Telecommuter FL

PRIMARY PURPOSE: The AI Evaluation Engineer designs, builds, and operates the evaluation systems that determine whether our GenAI solutions are accurate, safe, reliable, and ready for production. You will serve multiple product pods, establishing consistent evaluation standards, automated pipelines, and quality gates, and you will help teams measure and improve LLM, RAG, and agentic applications throughout their lifecycle. Fully remote role, open to U.S. based candidates. ESSENTIAL FUNCTIONS AND RESPONSIBILITIES
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.