Time since we first saw it
Day 33
Similar jobs usually come down between day 8 and day 60.
From 11,991 similar postings that came down. Coming down is not always a hire.
Bangalore
Live, first seen by us 33 days agoThe Senior AI Evals Test Engineer is a hands-on technical role responsible for defining, developing, and executing robust evaluation frameworks and test strategies for AI agents built using the Google Agent Development Kit (ADK) or similar sort of agentic frameworks.
Design and build agent evaluation frameworks — create end-to-end infrastructure for evaluating LLM-based agents across correctness, latency, cost, safety, and task completion rate
Build golden datasets and test harnesses — curate domain-specific evaluation sets for Rubrik's product verticals (data security, SaaS protection, cyber recovery, compliance)
Evaluate new technologies and patterns (e.g., new tools for Agent Evals), run proofs-of-concept, and influence the platform roadmap
The Senior AI Evals Test Engineer is a hands-on technical role responsible for defining, developing, and executing robust evaluation frameworks and test strategies for AI agents built using the Google Agent Development Kit (ADK) or similar sort of agentic frameworks. This role bridges AI development and reliable deployment — ensuring our agents are safe, ethical, effective, and meet high-quality performance standards. A significant focus of the role is on Responsible AI and Safety Evals, including adversarial testing, red-teaming, bias detection, and toxicity measurement. The engineer will design scalable evaluation pipelines, build synthetic test environments, and work closely with AI/ML Engineers, Researchers, and policy teams to ensure agents are production-ready and compliant with regulatory and ethical standards. The role is 70% Automation and 30% Manual Testing, and sits within the Applied AI: Test Engineering team as a key technical authority.
Key Responsibilities Design and build agent evaluation frameworks — create end-to-end infrastructure for evaluating LLM-based agents across correctness, latency, cost, safety, and task completion rate Build synthetic testing environments and simulation strategies to stress-test AI agents under diverse real-world conditions, with specific focus on agents built on Google ADK and related Google AI/ML services (e.g., Vertex AI, Gemini models)
Build synthetic testing environments and simulation strategies to stress-test AI agents under diverse real-world conditions, with specific focus on agents built on Google ADK and related Google AI/ML services (e.g., Vertex AI, Gemini models) Build golden datasets and test harnesses — curate domain-specific evaluation sets for Rubrik's product verticals (data security, SaaS protection, cyber recovery, compliance) Establish eval-as-a-service — build and enhance centralized evaluation platform that any IT team members can self-serve to test their agents against standardized and custom benchmarks
Identify, document, prioritize, and track defects, performance regressions, and alignment failures using Jira, collaborating closely with Applied AI Engineers to analyze root causes and validate fixes Evaluate new technologies and patterns (e.g., new tools for Agent Evals), run proofs-of-concept, and influence the platform roadmap Partner with policy and legal teams to ensure agent evaluations systematically cover regulatory compliance, fairness objectives, and emerging AI governance requirements
Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience
Strong understanding of agentic AI patterns — tool use, function calling, multi-step planning, ReAct/CoT reasoning chains, retrieval-augmented generation (RAG)
8+ years of overall software Test engineering experience, with 5+ years focused on test engineering, QA automation, or developer tooling, and demonstrated ability to build evaluation systems across multiple AI/ML domains, conversational agents, or Large Language Models (LLMs)
Proficiency in Python — strong scripting and tooling skills; experience with pytest, async patterns, and data processing at scale
2+ years working with ML/AI systems, building, testing, or evaluating models in production environments, including hands-on experience with LLM-based applications, prompt engineering, RAG pipelines, or agentic workflows
Familiarity with Google Cloud Platform (GCP) or any Cloud Service relevant to AI/ML (e.g., Vertex AI) and experience integrating testing into MLOps workflows
Experience testing complex APIs, especially for tool-integration and function-calling scenarios in agentic workflows
Proven ability to lead without direct authority, influencing cross-functional teams and driving technical decisions through clarity and data rather than hierarchy.
Experience evaluating agents in cybersecurity, data protection, or infrastructure domains
Experience with prompt injection (Must) and data poisoning skills is a plus
Education & Experience - Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience - 8+ years of overall software Test engineering experience, with 5+ years focused on test engineering, QA automation, or developer tooling, and demonstrated ability to build evaluation systems across multiple AI/ML domains, conversational agents, or Large Language Models (LLMs)
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
Day 33
Similar jobs usually come down between day 8 and day 60.
From 11,991 similar postings that came down. Coming down is not always a hire.
6 open jobs
Open jobs by the week we first saw them, last 12 weeks.
Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
Tech Stack: - Languages: Python(Must), TypeScript, Java (Optional) - Test Automation:Selenium, Playwright
- Production experience with eval pipelines — building automated, CI-integrated evaluation infrastructure (not just notebooks) - Proficiency in Python — strong scripting and tooling skills; experience with pytest, async patterns, and data processing at scale - Familiarity with Google Cloud Platform (GCP) or any Cloud Service relevant to AI/ML (e.g., Vertex AI) and experience integrating testing into MLOps workflows
- Deep Learning/NLP Knowledge: Fundamental understanding of how LLMs work, including concepts like RAG, prompting, and the technical challenges related to alignment - Data Analysis: Experience with data analysis and visualization tools (e.g., Pandas, SQL) for analyzing evaluation and safety results - Policy Background: Familiarity with AI ethics principles and industry standards for responsible AI development
- Experience with prompt injection (Must) and data poisoning skills is a plus - Certifications in AI/ML, cloud platforms (GCP, AWS), AWS Machine Learning Specialty, Google Professional ML Engineer, DeepLearning.AI specializations and software testing methodologies Soft Skills
- Test Automation:Selenium, Playwright - LLM Providers: Anthropic Claude, OpenAI, Azure OpenAI, open-source models - Eval Tooling: DeepEvals(Must), LLM as a Judge (Must), Promptfoo/RAGAS/LangSmith/ Custom harnesses
- Agent Evals Frameworks/libraries: LangGraph, Claude Agent SDK, custom orchestrators - Infrastructure:DevOps (Basic), Kubernetes, CI/CD (Jenkins/GitHub Actions), observability stacks (Datadog/Grafana) Preferred Qualifications
- Proficiency in Python — strong scripting and tooling skills; experience with pytest, async patterns, and data processing at scale - Familiarity with Google Cloud Platform (GCP) or any Cloud Service relevant to AI/ML (e.g., Vertex AI) and experience integrating testing into MLOps workflows - Statistical literacy — ability to design experiments, interpret eval metrics with appropriate confidence, and avoid common pitfalls (overfitting to benchmarks, metric gaming)
Technical Skills - Deep hands-on experience with LLM evaluation — familiarity with frameworks and libraries such as LangSmith, DeepEval,LLM as a Judge, Promptfoo, RAGAS, Hugging Face evaluation tooling, custom eval harnesses - Strong understanding of agentic AI patterns — tool use, function calling, multi-step planning, ReAct/CoT reasoning chains, retrieval-augmented generation (RAG)
Complete your application on www.rubrik.com. The employer’s form will show what is required.
Already applied? Track this application
The Senior AI Evals Test Engineer is a hands-on technical role responsible for defining, developing, and executing robust evaluation frameworks and test strategies for AI agents built using the Google Agent Development Kit (ADK) or similar sort of agentic frameworks. This role bridges AI development and reliable deployment — ensuring our agents are safe, ethical, effective, and meet high-quality performance standards. A significant focus of the role is on Responsible AI and Safety Evals, including adversarial testing, red-teaming, bias detection, and toxicity measurement. The engineer will design scalable evaluation pipelines, build synthetic test environments, and work closely with AI/ML Engineers, Researchers, and policy teams to ensure agents are production-ready and compliant with regulatory and ethical standards. The role is 70% Automation and 30% Manual Testing, and sits within the Applied AI: Test Engineering team as a key technical authority.
Rubrik (NYSE: RBRK) is the Security and AI Operations Company, delivering one platform built at the intersection of data, identity, and AI. The platform enables agentic cyber resilience and trusted AI operations for enterprises worldwide, built to recover before disruption strikes and stay a step ahead, so security is never the reason ambition stalls.
Linkedin | X (formerly Twitter) | Instagram | Rubrik.com
At Rubrik, we are dedicated to fostering a culture where people from all backgrounds are valued, feel they belong, and believe they can succeed. Our commitment to inclusion is at the heart of our mission to secure the world’s data.
Our goal is to hire and promote the best talent, regardless of background. We continually review our hiring practices to ensure fairness and strive to create an environment where every employee has equal access to opportunities for growth and excellence. We believe in empowering everyone to bring their authentic selves to work and achieve their fullest potential.
Our Company: We are committed to building a merit-based organization that offers equal access to growth and success for all employees globally. Your potential is limitless here.
Our Culture: We strive to create an inclusive atmosphere where individuals from all backgrounds feel a strong sense of belonging, can thrive, and do their best work. Your contributions help us innovate and break boundaries.
Our Communities: We are dedicated to expanding our engagement with the communities we operate in, creating opportunities for underrepresented talent and driving greater innovation for our clients. Your impact extends beyond Rubrik, contributing to safer and stronger communities.
Rubrik is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, genetic information, protected veteran status, or any other status protected by applicable law. Rubrik provides equal employment opportunities to employees and applicants in all terms and conditions of employment, including recruiting, hiring, promotion, compensation, training, leave, and termination.
Rubrik provides reasonable accommodations to qualified applicants and employees with disabilities in accordance with applicable law. If you need an accommodation to participate in the application process or to perform your job, please contact hr@rubrik.com. Accommodations may include changes to the application process, documents in an alternate format, a sign language interpreter, or specialized equipment.
New Data Scientist postings at the employers we follow are up 4% over the last 6 weeks compared with the 6 before.
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
We file this job under Data Scientist, by the work the posting describes.
Data scientists use statistics and models to answer harder questions, such as whether a change worked, what demand will look like next quarter, or which customers are likely to leave.
Postings usually ask for Python or R, SQL, statistics, experiment design and common machine learning methods.
Unlike an ML engineer, a data scientist's main output is often an analysis or a first model, not software running inside a product.
Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-2051.
Selected passages from the saved posting. Check the full description for conditions and exceptions.
No pay amount identified in the saved description.
Bangalore
Working pattern and location restrictions need checking in the full posting.
No clear work-authorization passage found. Eligibility is unconfirmed.