Back to jobs

AI Engineer (Harness)

Pay
Pay not listed
Work setting
Hybrid — work setting source
Work setup in the employer’s posting: hybrid
From the employer’s posting
Employment
Full-time — employment source
Employment type Full time
Employment type in the employer’s posting: Full time
From the employer’s posting

What you’ll work on

Full posting
  • Build & Calibrate LLM-as-a-Judge Pipelines: Develop automated judging systems validated against human ground truth.

From the employer’s posting
Design Experiments & Validate Results: Apply rigorous statistical methods—including hypothesis testing, confidence intervals, bootstrapping, and power analysis—to determine whether performance changes are true system improvements or stochastic noise. Build & Calibrate LLM-as-a-Judge Pipelines: Develop automated judging systems validated against human ground truth. Measure alignment using agreement metrics (Cohen’s kappa, correlation, precision/recall) and systematically detect judge failure modes such as position bias, verbosity bias, self-preference, and prompt sensitivity. Curate Benchmark Datasets & Rubrics: Define sampling strategies, detailed annotation guidelines, and scoring rubrics. Measure inter-annotator agreement and manage label noise to ensure benchmark integrity over time.

What you’ll bring

All qualifications

Core experience

  • Experience with agent and evaluation frameworks (e.g., LangChain, LangGraph, AutoGen, lm-evaluation-harness, Promptfoo, or Ragas).
Qualification wording
Experience with agent and evaluation frameworks (e.g., LangChain, LangGraph, AutoGen, lm-evaluation-harness, Promptfoo, or Ragas).

Benefits in the posting

Full benefits wording
  • A competitive salary – because we value your unique skills
  • Employer pension contributions – because you deserve a secure future
  • Private health insurance - because your health is important to us
  • Hybrid work setup – because everyone has different needs
  • Gym memberships - because staying active fuels productivity
  • Life at Valarian

From the employer’s posting.

Apply at Valarian

Posting record

Report an error
Status in our records
Live
First seen by us
Sep 17, 202663 days ago
Recorded sightings
40
Last seen by us
Nov 19, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 63

Up longer than most. Similar jobs usually come down between day 2 and day 14.

From 1,011 similar postings that came down. Coming down is not always a hire.

Valarian is hiring

4 open jobs

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • Kubernetes
Where the posting names them
What you’ll do: - Architect Evaluation Runtimes & Harnesses: Design and maintain scalable Python evaluation harnesses that measure task completion, trajectory quality, tool-use correctness, cost, latency, and variance across multi-step agentic systems. - Design Experiments & Validate Results: Apply rigorous statistical methods—including hypothesis testing, confidence intervals, bootstrapping, and power analysis—to determine whether performance changes are true system improvements or stochastic noise.
- Rigor in Error Analysis: Ability to dissect complex execution logs, categorize failure modes, and synthesize clear, data-driven recommendations. - Hands-On LLM Integration: Practical proficiency with LLM APIs, prompt engineering, structured outputs, function/tool calling, and retrieval systems in Python. - Agent Architecture Awareness: Clear understanding of planning loops, tool orchestration, memory management, and multi-agent coordination, including their trade-offs and common failure modes.
- Agent Architecture Awareness: Clear understanding of planning loops, tool orchestration, memory management, and multi-agent coordination, including their trade-offs and common failure modes. - Harness & Scaffolding Contribution: Ability to write clean, maintainable Python code to iterate on harness components such as guardrails, retry logic, state handling, and context management. Nice to have:
- Experience with agent and evaluation frameworks (e.g., LangChain, LangGraph, AutoGen, lm-evaluation-harness, Promptfoo, or Ragas). - Exposure to running evaluation pipelines or agent workloads within secure, enclave, or Kubernetes environments. Benefits:

Your next step

Complete your application on jobs.lever.co. The employer’s form will show what is required.

Applying through Lever

  • No account needed: one form, and Lever reads your resume into the profile recruiters search.

Already applied? Track this application

Job description

View original posting ↗

Valarian: A New Era of Sovereign Intelligence 

Valarian is a dual use technology company building critical tools to safeguard the future in an era of evolving global security challenges. Backed by a Series A funding round of $50m led by NEA, we are rethinking security beyond traditional military domains to address asymmetric threats that impact our technological advantage, economic strength, and democratic institutions. We provide the control infrastructure that allows government and regulated enterprise to put frontier-class AI to work safely.

We are bringing a frontier AI experience inside secure boundaries for the first time. Our technology supports mission critical workflows across government, healthcare, and enterprise, including:

  • Scaling open-weight models without external inference endpoints.

  • Putting frontier class AI to work across organisations through partnerships with other leading AI startups such as Perplexity. 

  • Increasing operational resiliency for air gapped, on prem, and cloud based systems.

  • Securing intellectual property and maintaining auditability for high stakes enterprise environments.

  • Facilitating secure information flow during disaster response, M&A due diligence, and privileged legal communications.

We’ve only scratched the surface. Built in Britain by veterans of the Ministry of Defence, British Forces, Darktrace, and elite engineering backgrounds, we are driven by a shared mission to safeguard western democracy . Whether you join our London HQ or our wider global organisation, you’ll be part of a collaborative, high performing team creating the cutting edge software and infrastructure that defines the frontier of enterprise AI.

The Role

As an AI Harness Engineer at Valarian, you will own the experimental design, evaluation methodologies, and benchmark infrastructure for our AI models and autonomous agentic workloads. In an emerging domain with no standard playbook, you will serve as the bridge between rigorous data science, statistical validation, and production agent scaffolding.

You will design robust evaluation datasets, architect calibrated LLM-as-a-judge pipelines, and build metrics that accurately capture multi-step agent performance under non-deterministic conditions. Your insights and error analyses will directly drive iterative improvements to our harness scaffolding, tool orchestration, and system safety guardrails.

What you’ll do:
  • Architect Evaluation Runtimes & Harnesses: Design and maintain scalable Python evaluation harnesses that measure task completion, trajectory quality, tool-use correctness, cost, latency, and variance across multi-step agentic systems.

  • Design Experiments & Validate Results: Apply rigorous statistical methods—including hypothesis testing, confidence intervals, bootstrapping, and power analysis—to determine whether performance changes are true system improvements or stochastic noise.

  • Build & Calibrate LLM-as-a-Judge Pipelines: Develop automated judging systems validated against human ground truth. Measure alignment using agreement metrics (Cohen’s kappa, correlation, precision/recall) and systematically detect judge failure modes such as position bias, verbosity bias, self-preference, and prompt sensitivity.

  • Curate Benchmark Datasets & Rubrics: Define sampling strategies, detailed annotation guidelines, and scoring rubrics. Measure inter-annotator agreement and manage label noise to ensure benchmark integrity over time.

  • Deep Error & Trajectory Analysis: Conduct hands-on failure analysis on agent runs, cluster root causes into actionable error categories, and clearly communicate findings to both engineering teams and stakeholders.

  • Drive Harness Scaffolding Iteration: Translate evaluation results directly into architectural improvements across agent scaffolding, prompt design, tool execution loops, context window compaction, retries, and safety guardrails.

What we are looking for: 
  • Grounding in Statistics & Experimental Design: Demonstrated expertise in hypothesis testing, bootstrapping, power analysis, and statistical significance within probabilistic machine learning systems.

  • LLM-as-a-Judge Validation: Proven experience building, auditing, and validating LLM judge setups against human annotations, including tracking agreement metrics and mitigating judge biases.

  • Evaluation Dataset Engineering: Strong track record designing evaluation datasets, crafting precise scoring rubrics, managing label noise, and measuring inter-annotator agreement.

  • Evaluation of Non-Deterministic Multi-Step Systems: Deep comfort evaluating complex agent trajectories, tool execution sequences, latency, cost, and variance across repeated runs rather than relying solely on single-shot benchmarks.

  • Metric Design & Integrity: Exceptional judgment in defining metrics tied to real-world outcomes, with a keen eye for detecting benchmark gaming, target drift, or shortcut learning.

  • Rigor in Error Analysis: Ability to dissect complex execution logs, categorize failure modes, and synthesize clear, data-driven recommendations.

  • Hands-On LLM Integration: Practical proficiency with LLM APIs, prompt engineering, structured outputs, function/tool calling, and retrieval systems in Python.

  • Agent Architecture Awareness: Clear understanding of planning loops, tool orchestration, memory management, and multi-agent coordination, including their trade-offs and common failure modes.

  • Harness & Scaffolding Contribution: Ability to write clean, maintainable Python code to iterate on harness components such as guardrails, retry logic, state handling, and context management.

Nice to have:
  • Experience with agent and evaluation frameworks (e.g., LangChain, LangGraph, AutoGen, lm-evaluation-harness, Promptfoo, or Ragas).

  • Exposure to running evaluation pipelines or agent workloads within secure, enclave, or Kubernetes environments.

Benefits:

 

Our benefits are designed to ensure our employees feel taken care of and are proud to be a part of the Valarian team. We are committed to consistently enhancing our benefit package, taking into account the overall well-being and needs of our teammates. Here are the key benefits accessible to all employees at Valarian Technologies:

  • Equity – because you have the right to own what you’re building

  • A competitive salary – because we value your unique skills

  • Employer pension contributions – because you deserve a secure future

  • Private health insurance - because your health is important to us

  • Hybrid work setup – because everyone has different needs

  • Gym memberships - because staying active fuels productivity

  • Rewarding company retreats and meetups that respect your work/life balance – because we love getting to know each other

 

Life at Valarian

 

Our culture is built on inclusivity, compassion and flexibility – we want everyone to be empowered to achieve their goals at Valarian.

 

The work we do is vital, but so are the connections that make it happen. We thrive on the shared energy, spontaneous conversations, and mutual trust built when we spend time together. We operate on an office first hybrid model, regularly gathering in our London office to support one another and collaborate. 

 

Valarian Technologies Limited is an equal opportunity employer and welcomes applications from invdividuals regardless of race, colour, religion, sex, sexual orientation, gender, identity or expression, national origin, age, disability, genetic information, marital status, veteran, amnesty, or any other legally protected characteristic.
 
We are committed to ensuring a fair and inclusive recruitment process and providing employment opportunities to all applicants. Decision recruitment, hiring, and employment are based solely on qualifications, skills, and experience relevant to the job requirements.
 
We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analysing resumes, or assessing responses and identifying potential inconsistencies or verification signals in application materials based on available information. These tools assist our recruitment team but do not replace human judgement. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us. 

 

Employment type

Full time

About the role

We file this job under AI Engineer, by the work the posting describes.

AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.

Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.

An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

London

- Private health insurance - because your health is important to us - Hybrid work setup – because everyone has different needs - Gym memberships - because staying active fuels productivity
More source context
Our culture is built on inclusivity, compassion and flexibility – we want everyone to be empowered to achieve their goals at Valarian. The work we do is vital, but so are the connections that make it happen. We thrive on the shared energy, spontaneous conversations, and mutual trust built when we spend time together. We operate on an office first hybrid model, regularly gathering in our London office to support one another and collaborate. Valarian Technologies Limited is an equal opportunity employer and welcomes applications from invdividuals regardless of race, colour, religion, sex, sexual orientation, gender, identity or expression, national origin, age, disability, genetic information, marital status, veteran, amnesty, or any other legally protected characteristic.
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.