Back to jobs

AI Testing Engineer, Amazon Bedrock

Pay
Pay not listed
Work setting
Remote — work setting source
Location: 100% remote, seeking candidates located in LATAM. Contract Length: 1 year, with potential for extension.
Read the full posting
Employment
Not stated

What you’ll work on

Full posting
  • Define testing and evaluation strategy for GenAI applications and AI agents

  • Build automated evaluation pipelines and add AI-quality checks to CI/CD

  • Design single-turn and multi-turn conversational test cases

  • Design adversarial tests for prompt injection, jailbreaks, and PII leakage

  • Perform root-cause analysis of incorrect AI outputs with AI engineers

From the employer’s posting
Define the testing and evaluation strategy for Generative AI applications and AI agents.
Build automated evaluation pipelines and integrate AI-quality checks into CI/CD and release processes.
Design single-turn and multi-turn conversational test cases, including context retention and workflow transitions.
Design negative and adversarial test scenarios covering prompt injection, jailbreaks, hallucinations, PII leakage, unsafe responses, and out-of-scope requests.
Perform root-cause analysis for incorrect or undesirable AI outputs and collaborate with AI engineers on remediation.

What you’ll bring

All qualifications

Core experience

  • 5+ years of experience in software testing, quality engineering, or test automation with hands-on automation capabilities.
  • Experience with AI evaluation frameworks such as RAGAS, DeepEval, Promptfoo, LangSmith, TruLens, or equivalent tools.
  • Bachelor’s degree in Computer Science, Artificial Intelligence, Machine Learning, Data Science, Software Engineering, or a related technical field.
  • 2+ years of experience with Amazon Bedrock.
  • Experience evaluating RAG systems, retrieval quality, or enterprise knowledge assistants.
  • Practical understanding of Generative AI / LLM behavior and the challenges of testing non-deterministic systems.
Qualification wording
5+ years of experience in software testing, quality engineering, or test automation with hands-on automation capabilities.
Experience with AI evaluation frameworks such as RAGAS, DeepEval, Promptfoo, LangSmith, TruLens, or equivalent tools.
Bachelor’s degree in Computer Science, Artificial Intelligence, Machine Learning, Data Science, Software Engineering, or a related technical field.
2+ years of experience with Amazon Bedrock.
Experience evaluating RAG systems, retrieval quality, or enterprise knowledge assistants.
Practical understanding of Generative AI / LLM behavior and the challenges of testing non-deterministic systems.
Education & alternatives
Education & Certifications: - Bachelor’s degree in Computer Science, Artificial Intelligence, Machine Learning, Data Science, Software Engineering, or a related technical field. Required Equipment: This role requires the use of a personal laptop. Candidates must be comfortable with company security software and device management tools being installed on their laptop as part of the onboarding process and for the duration of the engagement.

About NTT DATA, Europe & LATAM, Branch in USA

NTT DATA is a Top 5 global IT services provider with more than 190,000 professionals across 50+ countries.

In the employer’s words · Read in context

Posting record

Report an error

Actively hiringFirst seen by us 3 days ago, still listed, and not a repost of an earlier posting.

Status in our records
Live
First seen by us
Nov 18, 20263 days ago
Recorded sightings
4
Last seen by us
Nov 21, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 3

Similar jobs usually come down between day 1 and day 15.

From 1,368 similar postings that came down. Coming down is not always a hire.

NTT DATA, Europe & LATAM, Branch in USA is hiring

1 open job

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • JavaScript
  • Python
  • TypeScript
  • AWS
Where the posting names them
- Experience designing test scenarios for conversational applications, chatbots, virtual assistants, or AI agents. - Experience with Python and/or JavaScript / TypeScript for test automation and data processing. - Experience testing REST APIs and backend services.
- Experience evaluating RAG systems, retrieval quality, or enterprise knowledge assistants. - Experience with Amazon Bedrock and AWS services. - Experience testing Amazon Connect, Contact Center, agent-assist, or Customer Service applications.

Your next step

Complete your application on job-boards.greenhouse.io. The employer’s form will show what is required.

Applying through Greenhouse

  • No account needed: the form is one page, with an “Autofill my application” option that reads your resume.
  • Greenhouse doesn’t read resumes over 2.5 MB, or details placed in a header, footer, text box, table or second column. The file still attaches, but a recruiter has to type you in by hand.

Already applied? Track this application

Job description

View original posting ↗

Location: 100% remote, seeking candidates located in LATAM.

Contract Length: 1 year, with potential for extension.

Contract Rate: $20-25 per hour in USD. Final rate will be determined based on relevant experience and overall fit for the position.

NTT DATA is seeking an AI Testing Engineer, Amazon Bedrock to define and execute the quality strategy for Generative AI applications and AI agents used in Customer Service and Contact Center scenarios. This role will evaluate AI outputs across deterministic and non-deterministic scenarios using automation, semantic and statistical analysis, human review, and adversarial testing while partnering with AI engineers and functional QA teams.

Responsibilities:

  • Define the testing and evaluation strategy for Generative AI applications and AI agents.
  • Create and maintain evaluation datasets, golden datasets, and regression suites covering representative business scenarios.
  • Design single-turn and multi-turn conversational test cases, including context retention and workflow transitions.
  • Evaluate AI outputs for accuracy, relevance, completeness, groundedness, faithfulness, consistency, tone, safety, and compliance.
  • Build automated evaluation pipelines and integrate AI-quality checks into CI/CD and release processes.
  • Detect regressions introduced by changes to prompts, models, knowledge sources, retrieval strategies, tools, or guardrails.
  • Design negative and adversarial test scenarios covering prompt injection, jailbreaks, hallucinations, PII leakage, unsafe responses, and out-of-scope requests.
  • Validate fallback behavior, uncertainty handling, tool execution, error handling, and escalation to human agents.
  • Perform root-cause analysis for incorrect or undesirable AI outputs and collaborate with AI engineers on remediation.
  • Define measurable quality thresholds and release acceptance criteria for AI-agent capabilities.
  • Produce quality metrics, evaluation reports, dashboards, and trend analysis for project and product stakeholders.
  • Support UAT preparation, defect triage, production monitoring, and continuous improvement of AI behavior.

Required Experience:

  • 5+ years of experience in software testing, quality engineering, or test automation with hands-on automation capabilities.
  • 2+ years of experience with Amazon Bedrock.
  • Practical understanding of Generative AI / LLM behavior and the challenges of testing non-deterministic systems.
  • Experience designing test scenarios for conversational applications, chatbots, virtual assistants, or AI agents.
  • Experience with Python and/or JavaScript / TypeScript for test automation and data processing.
  • Experience testing REST APIs and backend services.
  • Experience working with structured test data in formats such as JSON and CSV.
  • Understanding of evaluation concepts such as groundedness, faithfulness, relevance, hallucination detection, semantic similarity, and human evaluation.
  • Experience with Git, CI/CD, automated regression testing, and defect-management workflows.
  • Strong analytical skills and ability to investigate why an AI output failed rather than only identifying that it failed.
  • Professional proficiency in English.

Nice-to-Have:

  • Experience with AI evaluation frameworks such as RAGAS, DeepEval, Promptfoo, LangSmith, TruLens, or equivalent tools.
  • Experience evaluating RAG systems, retrieval quality, or enterprise knowledge assistants.
  • Experience with Amazon Bedrock and AWS services.
  • Experience testing Amazon Connect, Contact Center, agent-assist, or Customer Service applications.

Education & Certifications:

  • Bachelor’s degree in Computer Science, Artificial Intelligence, Machine Learning, Data Science, Software Engineering, or a related technical field.

Required Equipment: This role requires the use of a personal laptop. Candidates must be comfortable with company security software and device management tools being installed on their laptop as part of the onboarding process and for the duration of the engagement.

About NTT DATA

NTT DATA is a Top 5 global IT services provider with more than 190,000 professionals across 50+ countries. We combine industry expertise with capabilities in consulting, technology, AI, cloud, applications, infrastructure, and connectivity to help organizations innovate, optimize, and transform. Through responsible innovation, we deliver meaningful business outcomes, accelerate client success, and create a positive impact on society.

Equal Opportunity Employer

NTT DATA is committed to hiring and retaining a diverse workforce. We are proud to be an Equal Opportunity/Affirmative Action-Employer, making decisions without regard to race, color, religion, creed, sex, sexual orientation, gender identity, marital status, national origin, age, veteran status, disability, or any other protected class. NTT DATA is an Equal Opportunity Employer Male/Female/Disabled/Veteran and a VEVRAA Federal Contractor.

Req ID: 1355

Demand for AI Engineer roles

New AI Engineer postings at the employers we follow are up 340% over the last 6 weeks compared with the 6 before.

Weekly new AI Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 78, the most 803, and the latest week had 438.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under AI Engineer, by the work the posting describes.

AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.

Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.

An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

LATAM

Location: 100% remote, seeking candidates located in LATAM. Contract Length: 1 year, with potential for extension.
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.