Time since we first saw it
Day 11
Similar jobs usually come down between day 1 and day 15.
From 1,376 similar postings that came down. Coming down is not always a hire.
Mangaluru Office · Mangaluru, Karnataka, India
Actively hiring, first seen by us 11 days agoLocation type: On-siteFrom the employer’s posting
Employment type: Full-timeFrom the employer’s posting
Department: Product & Engineering › EngineeringFrom the employer’s posting
Application deadline: 2026-10-30T18:30:00.000ZFrom the employer’s posting
As an AI Evaluation Engineer, you'll own the quality bar for both our AI-powered features and the broader product experience they sit inside.
Define golden datasets, rubrics, and scoring methodologies (human-in-the-loop and automated/LLM-as-judge) to measure model and product quality objectively.
Build automated eval pipelines that run in CI/CD, flag regressions before release, and produce clear, trackable quality metrics over time.
Partner with product managers and engineers to translate ambiguous quality bars ("is this good enough to ship?") into measurable, repeatable evaluation criteria.
As an AI Evaluation Engineer, you'll own the quality bar for both our AI-powered features and the broader product experience they sit inside. You'll design and run evaluation frameworks that catch regressions in model behavior (accuracy, hallucination, safety, compliance-sensitive edge cases) as well as classic product/QA issues — so that what we ship to customers handling real securities and compliance data is trustworthy every time. This is a hands-on, blended role: part evaluation engineering for LLM/AI systems, part product quality ownership.
Design, build, and maintain evaluation frameworks and benchmark suites for AI/LLM-powered features (e.g., document extraction, compliance checks, automated workflows), covering accuracy, consistency, hallucination rate, and safety. Define golden datasets, rubrics, and scoring methodologies (human-in-the-loop and automated/LLM-as-judge) to measure model and product quality objectively. Build automated eval pipelines that run in CI/CD, flag regressions before release, and produce clear, trackable quality metrics over time.
Define golden datasets, rubrics, and scoring methodologies (human-in-the-loop and automated/LLM-as-judge) to measure model and product quality objectively. Build automated eval pipelines that run in CI/CD, flag regressions before release, and produce clear, trackable quality metrics over time. Extend evaluation coverage beyond the model layer into full product/QA testing — functional, regression, and end-to-end testing of AI-powered features and the surrounding product.
Extend evaluation coverage beyond the model layer into full product/QA testing — functional, regression, and end-to-end testing of AI-powered features and the surrounding product. Partner with product managers and engineers to translate ambiguous quality bars ("is this good enough to ship?") into measurable, repeatable evaluation criteria. Investigate failures and edge cases, perform root-cause analysis across the model/product boundary, and drive fixes with engineering.
From the employer’s posting.
We're building the next generation of capital-markets infrastructure, starting with a modern, digital-native transfer agent designed for today's public companies.
In the employer’s words · Read in context · Company website ↗
Actively hiringFirst seen by us 11 days ago, still listed, and not a repost of an earlier posting.
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
Day 11
Similar jobs usually come down between day 1 and day 15.
From 1,376 similar postings that came down. Coming down is not always a hire.
2 open jobs
Open jobs by the week we first saw them, last 12 weeks.
Hiring recordAdd your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
- Hands-on experience testing or evaluating AI/LLM-powered features — building eval sets, scoring rubrics, or benchmark harnesses (or strong adjacent automation/QA experience with a demonstrated interest in AI evaluation). - Solid automation/testing fundamentals: scripting (Python and/or TypeScript/Java), API testing, SQL for data validation, and CI/CD integration. - Comfort working with LLMs and AI tooling directly — prompt engineering, RAG pipelines, or AI-assisted development tools — either as a builder or a rigorous evaluator of them.
Complete your application on jobs.ashbyhq.com. The employer’s form will show what is required.
Already applied? Track this application
We're building the next generation of capital-markets infrastructure, starting with a modern, digital-native transfer agent designed for today's public companies. Our platform replaces slow, manual processes with secure, automated workflows — delivering the first API-driven, real-time infrastructure for public companies.
Backed by Index Ventures, Spark Capital, Jump, MUFG, and Infinity Ventures, our team includes veterans from NYSE, Equiniti, Computershare, and AngelList. Our Engineering team is headquartered in Mangaluru.
About the RoleAs an AI Evaluation Engineer, you'll own the quality bar for both our AI-powered features and the broader product experience they sit inside. You'll design and run evaluation frameworks that catch regressions in model behavior (accuracy, hallucination, safety, compliance-sensitive edge cases) as well as classic product/QA issues — so that what we ship to customers handling real securities and compliance data is trustworthy every time. This is a hands-on, blended role: part evaluation engineering for LLM/AI systems, part product quality ownership.
What You'll DoDesign, build, and maintain evaluation frameworks and benchmark suites for AI/LLM-powered features (e.g., document extraction, compliance checks, automated workflows), covering accuracy, consistency, hallucination rate, and safety.
Define golden datasets, rubrics, and scoring methodologies (human-in-the-loop and automated/LLM-as-judge) to measure model and product quality objectively.
Build automated eval pipelines that run in CI/CD, flag regressions before release, and produce clear, trackable quality metrics over time.
Extend evaluation coverage beyond the model layer into full product/QA testing — functional, regression, and end-to-end testing of AI-powered features and the surrounding product.
Partner with product managers and engineers to translate ambiguous quality bars ("is this good enough to ship?") into measurable, repeatable evaluation criteria.
Investigate failures and edge cases, perform root-cause analysis across the model/product boundary, and drive fixes with engineering.
Maintain traceability and reporting on eval/QA results for compliance-sensitive workflows, given the regulated nature of the data we handle.
5 plus years of experience in QA/test engineering, ML evaluation, or a related quality-focused engineering role.
Hands-on experience testing or evaluating AI/LLM-powered features — building eval sets, scoring rubrics, or benchmark harnesses (or strong adjacent automation/QA experience with a demonstrated interest in AI evaluation).
Solid automation/testing fundamentals: scripting (Python and/or TypeScript/Java), API testing, SQL for data validation, and CI/CD integration.
Comfort working with LLMs and AI tooling directly — prompt engineering, RAG pipelines, or AI-assisted development tools — either as a builder or a rigorous evaluator of them.
Strong analytical mindset: comfortable defining metrics for fuzzy, subjective quality questions and defending them with data.
Clear written communication — you'll be documenting failure modes and quality bars for both engineers and non-technical stakeholders.
Bonus: experience in fintech, compliance, or another regulated domain where correctness and auditability matter.
Experience with vector databases / semantic search (FAISS, ChromaDB, or similar) and RAG evaluation.
Familiarity with eval tooling/frameworks (e.g., Ragas, DeepEval, promptfoo, custom LLM-as-judge pipelines).
Background in Agile/Scrum environments and defect-management tooling (Jira or similar).
Competitive compensation based on experience
Equity participation
Comprehensive health insurance (self + family)
Paid leave and wellness benefits
To apply: send your résumé to careers@vinylequity.com.
New AI Engineer postings at the employers we follow are up 340% over the last 6 weeks compared with the 6 before.
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
We file this job under AI Engineer, by the work the posting describes.
AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.
Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.
An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.
Selected passages from the saved posting. Check the full description for conditions and exceptions.
No pay amount identified in the saved description.
Mangaluru, Karnataka, India
Working pattern and location restrictions need checking in the full posting.
No clear work-authorization passage found. Eligibility is unconfirmed.