Back to jobs

AI Engineer (Evaluations) - Hawthorn East

Hawthorn East Melbourne, VIC, AU, 3123

Actively hiring, first seen by us 2 days ago
Pay
Pay not listed
Work setting
Not stated
Employment
Full-time — employment source
Job ID: 200884 Employment Type: Full time LI-MM1
Read the full posting

What you’ll work on

Full posting

As the AI Engineer (Evaluations) in Coles’ AI Commerce team, you’ll be responsible for how we measure the quality of the AI agent in the Coles website and apps that helps customers find products, build their basket, manage orders and sort out issues.

This is an AI engineering role focused on evaluation.

  • Build and maintain golden datasets, test scenarios and LLM-as-judge metrics for the agent

  • Run evals on every significant change and give a clear go/no-go release signal

  • Analyse results to find failure patterns, root causes and regressions

  • Calibrate LLM judges against human review so scores can be trusted

  • Grow eval coverage for edge cases, safety and red-team scenarios

From the employer’s posting
As the AI Engineer (Evaluations) in Coles’ AI Commerce team, you’ll be responsible for how we measure the quality of the AI agent in the Coles website and apps that helps customers find products, build their basket, manage orders and sort out issues.
This is an AI engineering role focused on evaluation. You’ll build and manage our evals, keep them trustworthy, and turn the results into clear actions for the team. When the agent’s quality shifts, you’ll be the first to know why. This is a hands-on role. You’ll work AI-first, using agentic development tools to ship high-quality work at pace. You’ll work closely with the Technical Lead, AI engineers, product managers and designers to make sure what we build balances UX, quality and safety.
Build and maintain the evaluation framework for the AI Commerce Agent: golden datasets, test scenarios and LLM-as-judge metrics
Run evals against every significant change and make the results a clear go/no-go signal for release
Analyse results to find failure patterns, root causes and regressions
Calibrate LLM judges against human review so the scores can be trusted
Grow coverage as the agent grows, including edge cases, safety and adversarial (red-team) scenarios

What you’ll bring

All qualifications

Core experience

  • 2 to 3 years in AI engineering, ML or data science, including LLM or agentic system evaluation
  • Strong Python and SQL
  • Hands-on LLM evaluation: golden datasets, LLM-as-judge, rubric design and tracing
  • Solid grounding in statistics and experiment design, such as sampling and significance
  • Understanding of AI agents: tool use, orchestration, RAG and guardrails
Qualification wording
Proven (2 to 3 years) experience in AI engineering, ML or data science, including hands-on work evaluating LLM or agentic systems
Strong Python and SQL
Hands-on experience with LLM evaluation: golden datasets, LLM-as-judge, rubric design and tracing
A solid grounding in statistics and experiment design, such as sampling, significance and inter-rater agreement
An understanding of how AI agents work: tool use, orchestration, RAG and guardrails

About Coles Group

We’re continuing to build a gender equitable team, and a culture that is just as diverse, inclusive and welcoming as the communities we serve.

In the employer’s words · Read in context

Apply at Coles Group

Posting record

Report an error

Actively hiringFirst seen by us 2 days ago, still listed, and not a repost of an earlier posting.

Status in our records
Live
First seen by us
Nov 20, 20262 days ago
Recorded sightings
3
Last seen by us
Nov 22, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 2

Similar jobs usually come down between day 1 and day 15.

From 1,368 similar postings that came down. Coming down is not always a hire.

Coles Group is hiring

6 open jobs

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • SQL
  • Azure
Where the posting names them
- Proven (2 to 3 years) experience in AI engineering, ML or data science, including hands-on work evaluating LLM or agentic systems - Strong Python and SQL - Hands-on experience with LLM evaluation: golden datasets, LLM-as-judge, rubric design and tracing
- Experience with red-teaming or AI safety testing - Experience with evaluation tooling or observability platforms, such as Azure AI Foundry, App Insights or similar - A visible footprint in the AI engineering community, such as open source, papers, talks or technical writing

Your next step

Complete your application on careers.colesgroup.com.au. The employer’s form will show what is required.

Applying through SAP SuccessFactors

  • Images (.jpg) aren’t accepted as a resume.

Already applied? Track this application

Job description

View original posting ↗

We’ve been proudly serving Australian communities since 1914 and have grown to become a top 30-listed company on the ASX with a portfolio of iconic homegrown brands. At the heart of our continued success is over 115,000 team members working across the country, all contributing to bringing our customers smiles. You’ll not only get to see your work make a difference, you’ll be part of a team working together to help Australians eat and live better every day.

 

About the team

 

Our Technology team is the backbone of the business, building and maintaining the information technology infrastructure that support Coles Group. As a part of the Technology team, you get the opportunity to work through complex, challenging, meaningful problems using large data sets and new technologies to expand your skills and experience.

 

About the role

 

As the AI Engineer (Evaluations) in Coles’ AI Commerce team, you’ll be responsible for how we measure the quality of the AI agent in the Coles website and apps that helps customers find products, build their basket, manage orders and sort out issues.

 

This is an AI engineering role focused on evaluation. You’ll build and manage our evals, keep them trustworthy, and turn the results into clear actions for the team. When the agent’s quality shifts, you’ll be the first to know why. This is a hands-on role. You’ll work AI-first, using agentic development tools to ship high-quality work at pace. You’ll work closely with the Technical Lead, AI engineers, product managers and designers to make sure what we build balances UX, quality and safety.

 

If you want to shape the quality of a production AI agent used by customers across the nation, with the scale and backing of a major retailer behind you, this is that role.

 

 

Key responsibilities will include:

 

  • Build and maintain the evaluation framework for the AI Commerce Agent: golden datasets, test scenarios and LLM-as-judge metrics
  • Run evals against every significant change and make the results a clear go/no-go signal for release
  • Analyse results to find failure patterns, root causes and regressions, and turn them into clear, prioritised actions for engineers and product
  • Calibrate LLM judges against human review so the scores can be trusted
  • Grow coverage as the agent grows, including edge cases, safety and adversarial (red-team) scenarios
  • Help build online evaluation from production conversations, tracing and customer feedback
  • Tie eval metrics to product success metrics and report on agent quality to the team and stakeholders
  • Use agentic development tools to ship high-quality code at pace
  • Stay across developments in AI evaluation and bring what’s useful into our approach

 

 

About you and your skills:

 

We're looking for a sharp, curious engineer who cares about getting quality right. You're rigorous with data, you don't trust a number until you understand it, and you can explain what the results mean and what to do next. You take ownership of your work, learn quickly and back your views with evidence. You work well in a team and you're comfortable pushing back when the data says something isn't ready.

 

 

You'll bring:

 

  • Proven (2 to 3 years) experience in AI engineering, ML or data science, including hands-on work evaluating LLM or agentic systems
  • Strong Python and SQL
  • Hands-on experience with LLM evaluation: golden datasets, LLM-as-judge, rubric design and tracing
  • A solid grounding in statistics and experiment design, such as sampling, significance and inter-rater agreement
  • An understanding of how AI agents work: tool use, orchestration, RAG and guardrails
  • AI-first ways of working. Agentic coding tools are core to how you build, and you know how to use them without compromising quality
  • Clear communication. You can turn complex results into a story that engineers and stakeholders act on
  • Pragmatism. You apply the 80/20 rule and know when good enough is right for now and when quality is non-negotiable

 

What will make you stand out:

 

  • Experience with red-teaming or AI safety testing
  • Experience with evaluation tooling or observability platforms, such as Azure AI Foundry, App Insights or similar
  • A visible footprint in the AI engineering community, such as open source, papers, talks or technical writing
  • Experience in e-commerce, search, recommendations or conversational commerce
  • Show us what you've built. Links to GitHub, shipped products or your writing are welcome with your application.

 

What’s in it for you?

 

  • Flexible working options: We know that work is only one part of your life, so we actively encourage a positive work-life balance and provide hybrid working options to help you achieve it.
  • Office perks: Take advantage of our gym facility and fitness classes, free parking, BBQ area, mini-Coles supermarket, fooderie hub where you can sample new products before they hit the shelves, school holiday program and so much more when you come in.
  • Discounts: Eligible team members receive 5% discount all year round on your Supermarket and Liquor online and in-store purchases. We also offer additional periods of double discount (10%) at various times throughout the year. As well as hotdeals exclusive to team members that translate into additional savings.
  • Reward through recognition: Give and receive recognition, linked to our Coles values, through our digital recognition platform ‘mythanks’. You can accumulate points to redeem in the online shop for exciting gifts and electronic gift cards from an extensive range of retailers.
  • Opportunities for learning and development: No matter where you start within our diverse business, you’ll have experiences, exposure, and education to satisfy you. Discover and explore a variety of career development programs and job-specific training.
  • Paid parental leave: We understand how important your life outside work is and offer permanent team members paid parental leave to support you in balancing work and family.
  • Investment in your future: Our annual team member share plan offer allows eligible team members to make regular pre-tax salary sacrifice deductions to purchase Coles Group shares.

 

About the recruitment process

 

We’re continuing to build a gender equitable team, and a culture that is just as diverse, inclusive and welcoming as the communities we serve. We are committed to creating a workplace that is safe and respectful for our team. We encourage applications from people of all ages, cultures (including Aboriginal and Torres Strait Islander peoples), abilities, sexual orientation and gender identities.

 

We’re happy to adjust our recruitment process to support candidates with disability. For further information and additional contact details visit the ‘Our Recruitment Process’ section of our careers site or email inclusionrecruitment@coles.com.au

 

Job ID: 200884

 

Employment Type: Full time

LI-MM1

Demand for AI Engineer roles

New AI Engineer postings at the employers we follow are up 340% over the last 6 weeks compared with the 6 before.

Weekly new AI Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 78, the most 803, and the latest week had 438.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under AI Engineer, by the work the posting describes.

AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.

Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.

An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Hawthorn East Melbourne, VIC, AU, 3123

What’s in it for you? - Flexible working options: We know that work is only one part of your life, so we actively encourage a positive work-life balance and provide hybrid working options to help you achieve it. - Office perks: Take advantage of our gym facility and fitness classes, free parking, BBQ area, mini-Coles supermarket, fooderie hub where you can sample new products before they hit the shelves, school holiday program and so much more when you come in.
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.