Time since we first saw it
Day 2
Similar jobs usually come down between day 1 and day 15.
From 1,368 similar postings that came down. Coming down is not always a hire.
Hawthorn East Melbourne, VIC, AU, 3123
Actively hiring, first seen by us 2 days agoJob ID: 200884 Employment Type: Full time LI-MM1Read the full posting
As the AI Engineer (Evaluations) in Coles’ AI Commerce team, you’ll be responsible for how we measure the quality of the AI agent in the Coles website and apps that helps customers find products, build their basket, manage orders and sort out issues.
This is an AI engineering role focused on evaluation.
Build and maintain golden datasets, test scenarios and LLM-as-judge metrics for the agent
Run evals on every significant change and give a clear go/no-go release signal
Analyse results to find failure patterns, root causes and regressions
Calibrate LLM judges against human review so scores can be trusted
Grow eval coverage for edge cases, safety and red-team scenarios
As the AI Engineer (Evaluations) in Coles’ AI Commerce team, you’ll be responsible for how we measure the quality of the AI agent in the Coles website and apps that helps customers find products, build their basket, manage orders and sort out issues.
This is an AI engineering role focused on evaluation. You’ll build and manage our evals, keep them trustworthy, and turn the results into clear actions for the team. When the agent’s quality shifts, you’ll be the first to know why. This is a hands-on role. You’ll work AI-first, using agentic development tools to ship high-quality work at pace. You’ll work closely with the Technical Lead, AI engineers, product managers and designers to make sure what we build balances UX, quality and safety.
Build and maintain the evaluation framework for the AI Commerce Agent: golden datasets, test scenarios and LLM-as-judge metrics
Run evals against every significant change and make the results a clear go/no-go signal for release
Analyse results to find failure patterns, root causes and regressions
Calibrate LLM judges against human review so the scores can be trusted
Grow coverage as the agent grows, including edge cases, safety and adversarial (red-team) scenarios
Proven (2 to 3 years) experience in AI engineering, ML or data science, including hands-on work evaluating LLM or agentic systems
Strong Python and SQL
Hands-on experience with LLM evaluation: golden datasets, LLM-as-judge, rubric design and tracing
A solid grounding in statistics and experiment design, such as sampling, significance and inter-rater agreement
An understanding of how AI agents work: tool use, orchestration, RAG and guardrails
We’re continuing to build a gender equitable team, and a culture that is just as diverse, inclusive and welcoming as the communities we serve.
In the employer’s words · Read in context
Actively hiringFirst seen by us 2 days ago, still listed, and not a repost of an earlier posting.
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
Day 2
Similar jobs usually come down between day 1 and day 15.
From 1,368 similar postings that came down. Coming down is not always a hire.
6 open jobs
Open jobs by the week we first saw them, last 12 weeks.
Hiring recordAdd your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
- Proven (2 to 3 years) experience in AI engineering, ML or data science, including hands-on work evaluating LLM or agentic systems - Strong Python and SQL - Hands-on experience with LLM evaluation: golden datasets, LLM-as-judge, rubric design and tracing
- Experience with red-teaming or AI safety testing - Experience with evaluation tooling or observability platforms, such as Azure AI Foundry, App Insights or similar - A visible footprint in the AI engineering community, such as open source, papers, talks or technical writing
Complete your application on careers.colesgroup.com.au. The employer’s form will show what is required.
Already applied? Track this application
We’ve been proudly serving Australian communities since 1914 and have grown to become a top 30-listed company on the ASX with a portfolio of iconic homegrown brands. At the heart of our continued success is over 115,000 team members working across the country, all contributing to bringing our customers smiles. You’ll not only get to see your work make a difference, you’ll be part of a team working together to help Australians eat and live better every day.
Our Technology team is the backbone of the business, building and maintaining the information technology infrastructure that support Coles Group. As a part of the Technology team, you get the opportunity to work through complex, challenging, meaningful problems using large data sets and new technologies to expand your skills and experience.
As the AI Engineer (Evaluations) in Coles’ AI Commerce team, you’ll be responsible for how we measure the quality of the AI agent in the Coles website and apps that helps customers find products, build their basket, manage orders and sort out issues.
This is an AI engineering role focused on evaluation. You’ll build and manage our evals, keep them trustworthy, and turn the results into clear actions for the team. When the agent’s quality shifts, you’ll be the first to know why. This is a hands-on role. You’ll work AI-first, using agentic development tools to ship high-quality work at pace. You’ll work closely with the Technical Lead, AI engineers, product managers and designers to make sure what we build balances UX, quality and safety.
If you want to shape the quality of a production AI agent used by customers across the nation, with the scale and backing of a major retailer behind you, this is that role.
Key responsibilities will include:
About you and your skills:
We're looking for a sharp, curious engineer who cares about getting quality right. You're rigorous with data, you don't trust a number until you understand it, and you can explain what the results mean and what to do next. You take ownership of your work, learn quickly and back your views with evidence. You work well in a team and you're comfortable pushing back when the data says something isn't ready.
You'll bring:
What will make you stand out:
We’re continuing to build a gender equitable team, and a culture that is just as diverse, inclusive and welcoming as the communities we serve. We are committed to creating a workplace that is safe and respectful for our team. We encourage applications from people of all ages, cultures (including Aboriginal and Torres Strait Islander peoples), abilities, sexual orientation and gender identities.
We’re happy to adjust our recruitment process to support candidates with disability. For further information and additional contact details visit the ‘Our Recruitment Process’ section of our careers site or email inclusionrecruitment@coles.com.au
Job ID: 200884
Employment Type: Full time
LI-MM1
New AI Engineer postings at the employers we follow are up 340% over the last 6 weeks compared with the 6 before.
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
We file this job under AI Engineer, by the work the posting describes.
AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.
Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.
An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.
Selected passages from the saved posting. Check the full description for conditions and exceptions.
No pay amount identified in the saved description.
Hawthorn East Melbourne, VIC, AU, 3123
What’s in it for you? - Flexible working options: We know that work is only one part of your life, so we actively encourage a positive work-life balance and provide hybrid working options to help you achieve it. - Office perks: Take advantage of our gym facility and fitness classes, free parking, BBQ area, mini-Coles supermarket, fooderie hub where you can sample new products before they hit the shelves, school holiday program and so much more when you come in.
No clear work-authorization passage found. Eligibility is unconfirmed.