Time since we first saw it
Day 1
Similar jobs usually come down between day 1 and day 14.
From 1,255 similar postings that came down. Coming down is not always a hire.
Reads like an AI Engineer role: building products on existing AI models.
Amsterdam, Noord-Holland, Netherlands
Actively hiring, first seen by us 1 day agoTurn customer QA rubrics into precise, testable LLM scoring instructions
Run evaluation experiments on large datasets using LangSmith and BigQuery
Curate golden datasets and generate synthetic data for rare cases
Build LLM-as-a-judge pipelines to make evaluation repeatable
Validate retrieval, tool calling, and speech transcription and diarization quality
Translate customer rubrics into AutoQA instructions.
Design and execute evaluation experiments on large, representative datasets using LangSmith and BigQuery
Curate raw production data into golden datasets with balanced coverage, and generate synthetic data to cover the rare cases that matter most.
Build LLM-as-a-judge pipelines to assess system quality internally and make evaluation repeatable.
Beyond scoring quality, you'll help validate retrieval (RAG/IR), tool calling, and speech pipelines (transcription and diarization quality).
You can be early in your career (recent graduates with strong LLM fundamentals are welcome), we have room and a clear path for you to grow
recent graduates with strong LLM fundamentals are welcome
We're looking for a sharp, analytical data scientist
At Kaizo, we build a performance development and quality platform for customer support teams.
In the employer’s words · Read in context
Actively hiringFirst seen by us 1 day ago, still listed, and not a repost of an earlier posting.
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
Day 1
Similar jobs usually come down between day 1 and day 14.
From 1,255 similar postings that came down. Coming down is not always a hire.
1 open job
Open jobs by the week we first saw them, last 12 weeks.
Hiring recordAdd your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
About Kaizo At Kaizo, we build a performance development and quality platform for customer support teams. Our AutoQA product uses LLMs to review support conversations against each customer's own quality rubric, automatically and at scale. Behind it sits a microservices-based stream processing platform handling over 200 million events per day (Kafka, Kubernetes on Google Cloud, ElasticSearch, MongoDB, BigQuery), and an LLMOps stack built around LangSmith for experimentation, prompt management, and tracing. The hard part isn't calling an LLM. It's knowing, with evidence, how well the system performs on every customer's unique rubric, and having a reliable, repeatable way to improve it. That's where you come in.
- Translate customer rubrics into AutoQA instructions. Work with real customer quality criteria and turn them into precise, testable instructions that LLMs can score reliably. - Run experiments that move accuracy. Design and execute evaluation experiments on large, representative datasets using LangSmith and BigQuery, and track quality with our performance metrics. - Build the datasets that make evaluation possible. Curate raw production data into golden datasets with balanced coverage, and generate synthetic data to cover the rare cases that matter most. QA is a discipline of rare occurrences: distributions are skewed, positives are scarce, and resourcefulness beats volume.
Complete your application on kaizo.recruitee.com. The employer’s form will show what is required.
Already applied? Track this application
Do you want to work out how you actually measure whether an AI system is doing a good job, and then make it better? We're looking for a sharp, analytical data scientist to own the evaluation and quality loop of our AutoQA product. You can be early in your career (recent graduates with strong LLM fundamentals are welcome), we have room and a clear path for you to grow.
Join a fast-growing SaaS company in an international environment (steep learning curve guaranteed).
Own a high-impact problem: making LLM-powered quality assurance measurably accurate at scale.
Sit at the intersection of AI and CX, working directly with enterprise customers and their real-world QA rubrics.
Grow into a Senior Data Scientist, AI Engineer, or ML Engineer role. We invest in progression.
Enjoy the perks: flexible hours, open holiday policy, an office in the heart of Amsterdam with hybrid flexibility, visa sponsorship, great gear, workations, and team events.
At Kaizo, we build a performance development and quality platform for customer support teams. Our AutoQA product uses LLMs to review support conversations against each customer's own quality rubric, automatically and at scale. Behind it sits a microservices-based stream processing platform handling over 200 million events per day (Kafka, Kubernetes on Google Cloud, ElasticSearch, MongoDB, BigQuery), and an LLMOps stack built around LangSmith for experimentation, prompt management, and tracing.
The hard part isn't calling an LLM. It's knowing, with evidence, how well the system performs on every customer's unique rubric, and having a reliable, repeatable way to improve it. That's where you come in.
Translate customer rubrics into AutoQA instructions. Work with real customer quality criteria and turn them into precise, testable instructions that LLMs can score reliably.
Run experiments that move accuracy. Design and execute evaluation experiments on large, representative datasets using LangSmith and BigQuery, and track quality with our performance metrics.
Build the datasets that make evaluation possible. Curate raw production data into golden datasets with balanced coverage, and generate synthetic data to cover the rare cases that matter most. QA is a discipline of rare occurrences: distributions are skewed, positives are scarce, and resourcefulness beats volume.
Build LLM-as-a-judge pipelines to assess system quality internally and make evaluation repeatable.
Make results actionable. Your experiments should end in a decision: change a prompt, adjust which tools the system uses, surface context the AI is missing, or flag where new capabilities are needed. You'll work with the AI team to ship those decisions.
Evaluate across the full stack. Beyond scoring quality, you'll help validate retrieval (RAG/IR), tool calling, and speech pipelines (transcription and diarization quality).
Join customer calls with the team to understand how QA leaders define quality, and feed what you learn back into the product.
Shaping how customers monitor quality themselves, catch drift, and keep their AutoQA setup improving over time.
Smarter categorization and routing of conversations to power analytics and get the right tickets to the right evaluation.
We file this job under AI Engineer, by the work the posting describes.
AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.
Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.
An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.
Selected passages from the saved posting. Check the full description for conditions and exceptions.
No pay amount identified in the saved description.
Amsterdam, Noord-Holland, Netherlands
- Grow into a Senior Data Scientist, AI Engineer, or ML Engineer role. We invest in progression. - Enjoy the perks: flexible hours, open holiday policy, an office in the heart of Amsterdam with hybrid flexibility, visa sponsorship, great gear, workations, and team events. About Kaizo
- Grow into a Senior Data Scientist, AI Engineer, or ML Engineer role. We invest in progression. - Enjoy the perks: flexible hours, open holiday policy, an office in the heart of Amsterdam with hybrid flexibility, visa sponsorship, great gear, workations, and team events. About Kaizo