Back to jobs

AI Engineer: Evals

Pay
Pay not listed
Work setting
Not stated
Employment
Not stated

What you’ll work on

Full posting
  • You will build the evaluation discipline that lets teams design, deploy and improve trustworthy agents in production.

  • Lead the technical strategy and implementation of how PhysicsX evaluates agents.

  • Build reusable evaluation frameworks, reference implementations and golden paths on top of our existing tooling.

From the employer’s posting
The role You will build the evaluation discipline that lets teams design, deploy and improve trustworthy agents in production. You will report into the overall engineering lead for our Agentic strategy. This is a hands-on engineering role with an organisation-wide mandate. Product, delivery and customer teams own the evals for the agents they ship. You will give them the shared infrastructure, frameworks, practices and coaching to do that well. You will also own cross-product evaluation and develop public benchmarks for agents doing real engineering work.
What you will do Lead the technical strategy and implementation of how PhysicsX evaluates agents. Build reusable evaluation frameworks, reference implementations and golden paths on top of our existing tooling.
Lead the technical strategy and implementation of how PhysicsX evaluates agents. Build reusable evaluation frameworks, reference implementations and golden paths on top of our existing tooling. Help product, delivery and customer teams establish robust eval suites, then turn what works into repeatable guidance.

What you’ll bring

All qualifications

Core experience

  • Experience with Pydantic Evals, Pydantic Logfire or comparable code-first evaluation and observability tools.
  • Experience evaluating agents in engineering, simulation, design, manufacturing or other expert domains.
  • Experience creating public benchmarks or open-source evaluation tooling.
  • Familiarity with post-training, synthetic data, reward signals or model selection.
  • Experience in regulated, customer-operated or air-gapped environments.
Qualification wording
Experience with Pydantic Evals, Pydantic Logfire or comparable code-first evaluation and observability tools.
Experience evaluating agents in engineering, simulation, design, manufacturing or other expert domains.
Experience creating public benchmarks or open-source evaluation tooling.
Familiarity with post-training, synthetic data, reward signals or model selection.
Experience in regulated, customer-operated or air-gapped environments.

About PhysicsX

PhysicsX is the physics AI company for industrials.

In the employer’s words · Read in context

Apply at PhysicsX

Posting record

Report an error

Actively hiringFirst seen by us 8 days ago, still listed, and not a repost of an earlier posting.

Status in our records
Live
First seen by us
Nov 13, 20268 days ago
Recorded sightings
9
Last seen by us
Nov 21, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 8

Similar jobs usually come down between day 1 and day 15.

At PhysicsX, jobs usually come down by about day 143.

From 1,376 similar postings that came down, and 6 of PhysicsX’s. Coming down is not always a hire.

PhysicsX is hiring

15 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • Hiring is picking up
  • Its jobs tend to stay up a long time
Hiring record

See how this role fits your experience

Add your resume to compare the role’s scope, tools and requirements with your experience.

Your next step

Complete your application on job-boards.eu.greenhouse.io. The employer’s form will show what is required.

Applying through Greenhouse

  • No account needed: the form is one page, with an “Autofill my application” option that reads your resume.
  • Greenhouse doesn’t read resumes over 2.5 MB, or details placed in a header, footer, text box, table or second column. The file still attaches, but a recruiter has to type you in by hand.

Already applied? Track this application

Job description

View original posting ↗

About us

Re-architecting Engineering for the Age of Intelligence

PhysicsX is the physics AI company for industrials. The company’s mission is to accelerate hardware innovation by overhauling what industrial engineering and manufacturing look like today. PhysicsX is building a new simulation software stack to deliver deep physics AI enablement across the entire engineering lifecycle. The company partners with leading organisations in aerospace & defence, automotive, semiconductors, materials, and energy & renewables, supporting them on some of their most critical and complex challenges. PhysicsX is headquartered in the United Kingdom, with offices in London, New York, and Singapore and an expanding presence in the Bay Area.

The role

You will build the evaluation discipline that lets teams design, deploy and improve trustworthy agents in production. You will report into the overall engineering lead for our Agentic strategy.

This is a hands-on engineering role with an organisation-wide mandate. Product, delivery and customer teams own the evals for the agents they ship. You will give them the shared infrastructure, frameworks, practices and coaching to do that well. You will also own cross-product evaluation and develop public benchmarks for agents doing real engineering work.

What you will do

  • Lead the technical strategy and implementation of how PhysicsX evaluates agents.
  • Build reusable evaluation frameworks, reference implementations and golden paths on top of our existing tooling.
  • Help product, delivery and customer teams establish robust eval suites, then turn what works into repeatable guidance.
  • Develop datasets and evaluators that combine deterministic checks, domain-expert judgement, model-based grading and human review.
  • Connect offline evaluation, regression testing and production telemetry so teams can detect drift and learn from real use.
  • Define cross-product scorecards and shared benchmarks without taking ownership away from the teams shipping each agent.
  • Create and publish credible open benchmarks for agents performing real engineering and manufacturing workflows.
  • Work with Research and Post-Training on model and agent evaluation. Own cross-product, production-agent and public benchmark integrity; share approval when results support product or external claims.
  • Build institutional muscle through reviews, examples, training and direct collaboration, making rigorous evaluation part of how teams think rather than a final release step.
  • Help determine how this capability should grow and, over time, shape the team around it.

What you bring

Where you need to be strongest

  • You have built and operated evaluation systems for agents or LLM-powered products in production.
  • You can turn an ambiguous product goal into datasets, evaluators, experiments and decision criteria that teams trust.
  • You understand the failure modes of probabilistic systems and use appropriate statistical judgement rather than treating benchmark scores as exact.
  • You know when to use deterministic evaluators, domain experts, human review and LLM-as-a-judge, including the limitations of each.
  • You are a strong engineer who builds usable frameworks and production-quality tooling, not only analyses or notebooks.
  • You have helped multiple teams adopt a technical practice through hands-on delivery, clear standards and pragmatic coaching.
  • You can work directly with customers and domain experts, learn what correctness means in their workflows and encode it without losing important nuance.

Foundations

  • Evaluation of multi-step agents, tool calls, structured outputs and long-running workflows.
  • Trace-based observability and OpenTelemetry concepts.
  • Dataset design, sampling, experiment design, regression detection and reproducibility.
  • CI and release gates for systems whose behaviour is non-deterministic.
  • Clear technical writing and communication across product, research, engineering, delivery and customer teams.
  • Product judgement: knowing which measurements should drive a decision and which are merely interesting.

Nice to have

  • Experience with Pydantic Evals, Pydantic Logfire or comparable code-first evaluation and observability tools.
  • Experience evaluating agents in engineering, simulation, design, manufacturing or other expert domains.
  • Experience creating public benchmarks or open-source evaluation tooling.
  • Familiarity with post-training, synthetic data, reward signals or model selection.
  • Experience in regulated, customer-operated or air-gapped environments.

 

 
We value diversity and are committed to equal employment opportunity regardless of sex, race, religion, ethnicity, nationality, disability, age, sexual orientation or gender identity. We strongly encourage individuals from groups traditionally underrepresented in tech to apply. To help make a change, we sponsor bright women from disadvantaged backgrounds through their university degrees in science and mathematics. 
 
We collect diversity and inclusion data solely for the purpose of monitoring the effectiveness of our equal opportunities policies and ensuring compliance with employment and equality legislation. This information is confidential, used only in aggregate form, and will not influence the outcome of your application. 
 

Demand for AI Engineer roles

New AI Engineer postings at the employers we follow are up 340% over the last 6 weeks compared with the 6 before.

Weekly new AI Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 78, the most 803, and the latest week had 438.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under AI Engineer, by the work the posting describes.

AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.

Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.

An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

London

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.