Back to jobs

Associate Director AI Product Engineer - Financial Institution Operations & Technology

Pay
$180,000–250,000/yearAnnual period assumed — pay source
Success means reliable evidence of what a product gets right, where it fails, and whether it is ready for use. Evaluations catch important errors before release, guide measurable improvements, and detect declining quality in live operations. Testing supports sound decisions, not simply a passing score. Associate Director Salary range: $180,000 - $250,000 Job title and level will be determined based on qualifications and experience.
Read the full posting
Work setting
Remote — work setting source
Listed location: Remote - USA
Read the full posting
Employment
Full-time — employment source
Employment type in the employer’s posting: Full time
From the employer’s posting

What you’ll work on

Full posting
  • You will build automated tests, investigate failures, and provide evidence for release decisions while contributing to product development.

  • Own the Evaluation Approach

  • Test whether the system finds the right information and whether its answer reflects that information.

From the employer’s posting
The Role We are hiring a hands-on AI Product Engineer focused on evaluation: proving whether AI products are accurate, reliable, and safe for banking operations. You will build automated tests, investigate failures, and provide evidence for release decisions while contributing to product development. Working with banking experts and engineers, you will define correct results, measure performance, and turn findings into product improvements. Your core responsibility is evaluation from early development through live use. Client stakeholders retain release and risk-acceptance authority.
What You’ll Do Own the Evaluation Approach Define what each product must get right, how errors could affect customers or operations, and what evidence is needed before use.
Verify Accuracy and Explain Failures Test whether the system finds the right information and whether its answer reflects that information. Separate source problems from answer errors. Check extracted facts against original records. Independently verify amounts, dates, and totals, and test whether citations actually support each important claim.

Before you apply

Sponsorship
Visa sponsorship not confirmed — sponsorship source
Candidates must be able to verify their legal right to work in the United States without employer sponsorship now or in the future.
Read the full posting

What you’ll bring

All qualifications

Core experience

  • Demonstrated ownership of AI evaluations that informed real release decisions, including test design, expert-reviewed answers, scoring, and failure investigation.
  • Experience independently evaluating AI products, checking automated scoring against expert judgment, or testing security and fairness in financial institutions.
  • Strong Python skills and experience building automated test software, connecting data sources, and recording repeatable results across product versions.
  • Experience evaluating document review, summaries, information search, or automated workflows, including comparisons across AI services and conventional software.
  • Ability to select representative samples, interpret error rates and uncertainty, and distinguish genuine improvement from weak tests or inconsistent scoring.
  • Experience building or improving AI products used in operations, with practical controls for sensitive data, output accuracy, and human review.
Qualification wording
Demonstrated ownership of AI evaluations that informed real release decisions, including test design, expert-reviewed answers, scoring, and failure investigation.
Experience independently evaluating AI products, checking automated scoring against expert judgment, or testing security and fairness in financial institutions.
Strong Python skills and experience building automated test software, connecting data sources, and recording repeatable results across product versions.
Experience evaluating document review, summaries, information search, or automated workflows, including comparisons across AI services and conventional software.
Ability to select representative samples, interpret error rates and uncertainty, and distinguish genuine improvement from weak tests or inconsistent scoring.
Experience building or improving AI products used in operations, with practical controls for sensitive data, output accuracy, and human review.

About BRG

Berkeley Research Group is building a Financial Services Operations & Technology practice focused initially on Commercial and Consumer Banking.

In the employer’s words · Read in context

Apply at BRG

Posting record

Report an error
Status in our records
Live
First seen by us
Nov 2, 202619 days ago
Recorded sightings
15
Last seen by us
Nov 17, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 19

Up longer than most. Similar jobs usually come down between day 1 and day 15.

At BRG, jobs usually come down by about day 28.

From 1,368 similar postings that came down, and 10 of BRG’s. Coming down is not always a hire.

BRG is hiring

6 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • Hiring is picking up
  • Takes jobs down at a usual pace
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
Where the posting names them
- Demonstrated ownership of AI evaluations that informed real release decisions, including test design, expert-reviewed answers, scoring, and failure investigation. - Strong Python skills and experience building automated test software, connecting data sources, and recording repeatable results across product versions. - Ability to select representative samples, interpret error rates and uncertainty, and distinguish genuine improvement from weak tests or inconsistent scoring.

Your next step

Complete your application on thinkbrg.wd5.myworkdayjobs.com. The employer’s form will show what is required.

Applying through Workday

  • You create an account with each employer before you can upload anything. Every Workday employer has its own account, so one sign-up doesn’t carry over.
  • “Autofill with Resume” comes right after sign-in, before the My Information and My Experience steps. Check every field it fills in those steps, since that’s what recruiters see.
  • Your application status lives in that employer’s Candidate Home, under the same sign-in.

Already applied? Track this application

Job description

View original posting ↗

We Do Consulting Differently


Preferred locations: New York, Chicago, Washington, D.C., Dallas, and Boston. Other locations considered case by case.


About the Practice

Berkeley Research Group is building a Financial Services Operations & Technology practice focused initially on Commercial and Consumer Banking. We combine banking expertise, technology, and AI to help clients improve operations.


The Role

We are hiring a hands-on AI Product Engineer focused on evaluation: proving whether AI products are accurate, reliable, and safe for banking operations. You will build automated tests, investigate failures, and provide evidence for release decisions while contributing to product development.

Working with banking experts and engineers, you will define correct results, measure performance, and turn findings into product improvements. Your core responsibility is evaluation from early development through live use. Client stakeholders retain release and risk-acceptance authority.


What You’ll Do

Own the Evaluation Approach

  • Define what each product must get right, how errors could affect customers or operations, and what evidence is needed before use.
  • Build reusable software that runs tests, scores results, and compares versions. Make testing part of everyday development, not a final checkpoint.
  • Work with engineers to reproduce failures, identify causes, and verify fixes. Contribute product code where needed to improve accuracy and control.

Design Tests and Measure Results

  • Set pass criteria and scoring rules with business and risk teams. Distinguish minor defects from errors that must block release.
  • Maintain representative test cases, including incomplete records and rare, high-impact errors. Reserve unseen examples for final testing to avoid overstating performance.
  • Have banking experts establish expected answers and resolve scoring disagreements. Measure factual accuracy, missing information, incorrect additions, and support from cited sources.
  • Check automated scoring against expert review. Test whether reviewers agree and whether AI-based scoring misses important errors.
  • Compare versions on the same tests. Report results by task and error severity, with sample sizes and uncertainty, not just averages.

Verify Accuracy and Explain Failures

  • Test whether the system finds the right information and whether its answer reflects that information. Separate source problems from answer errors.
  • Check extracted facts against original records. Independently verify amounts, dates, and totals, and test whether citations actually support each important claim.
  • Test when the product should answer, request clarification, or refer work to a person. Verify that reviewers can detect and correct material errors.

Manage Risk and Monitor Performance

  • Test for invented facts, unsupported recommendations, unfair outcomes, disclosure of sensitive information, and attempts to make the system ignore its safeguards.
  • Protect client data, restrict access and permitted actions, and use only approved services. Verify that safeguards work in practice.
  • Document repeatable test results, unresolved failures, and limitations. Recommend release, restricted use, or further development; support independent review and required client approvals.
  • Sample live outputs and track error rates and user corrections. Add confirmed failures to repeatable tests; trigger investigation or suspension when quality falls below agreed standards.
  • Explain findings clearly, coach colleagues, and develop reusable tests and controls. Support proposals, technical estimates, and recruiting.

Qualifications

Required

  • Minimum 5 years of working experience with a minimum of 1 year experience in AI evaluation.
  • Demonstrated ownership of AI evaluations that informed real release decisions, including test design, expert-reviewed answers, scoring, and failure investigation.
  • Strong Python skills and experience building automated test software, connecting data sources, and recording repeatable results across product versions.
  • Ability to select representative samples, interpret error rates and uncertainty, and distinguish genuine improvement from weak tests or inconsistent scoring.
  • Experience building or improving AI products used in operations, with practical controls for sensitive data, output accuracy, and human review.
  • Experience in financial services or another regulated environment, with the ability to translate business requirements into practical tests and controls.
  • Strong communication and teamwork skills, ability to mentor colleagues, and willingness to travel to client sites as needed.

Preferred

  • Banking experience in lending, payments, servicing, collections, or default operations.
  • Experience independently evaluating AI products, checking automated scoring against expert judgment, or testing security and fairness in financial institutions.
  • Experience evaluating document review, summaries, information search, or automated workflows, including comparisons across AI services and conventional software.
  • Consulting or product delivery experience, including client discovery, implementation planning, and user adoption.

What Success Looks Like

Success means reliable evidence of what a product gets right, where it fails, and whether it is ready for use. Evaluations catch important errors before release, guide measurable improvements, and detect declining quality in live operations. Testing supports sound decisions, not simply a passing score.


Associate Director Salary range: $180,000 - $250,000


Job title and level will be determined based on qualifications and experience.

Candidates must be able to verify their legal right to work in the United States without employer sponsorship now or in the future.



About BRG

BRG combines world-leading academic credentials with world-tested business expertise and purpose-built emerging technologies. Our culture centers on agility and connectivity which sets us apart and gets you ahead.  

 

At BRG, our professionals include specialist consultants, industry experts, renowned academics, and leading-edge data scientists. Together, they bring a diversity of real-world experience, data, and human and artificial intelligence, to economics, disputes, and investigations; corporate finance; and performance improvement services that address the most complex challenges facing organizations across the globe.

 

Our unique structure nurtures the interdisciplinary relationships that give us the edge, laying the groundwork for more informed insights and more original, incisive thinking.  When paired with our global reach and resources, our diverse perspectives and technical capabilities make us uniquely capable to address our clients’ challenges. We get results because we know how to apply our thinking to your world.

 

At BRG, we don’t just show you what’s possible. We’re built to help you make it happen. 


BRG is proud to be an Equal Opportunity Employer. Our hiring practices provide equal opportunity for employment without regard to race, religion, color, sex, gender, national origin, age, United States military veteran status, ancestry, sexual orientation, marital status, family structure, medical condition including genetic characteristics or information, veteran status, or mental or physical disability so long as the essential functions of the job can be performed with or without reasonable accommodation, or any other protected category under federal, state, or local law.


Demand for AI Engineer roles

New AI Engineer postings at the employers we follow are up 340% over the last 6 weeks compared with the 6 before.

Weekly new AI Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 78, the most 803, and the latest week had 438.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under AI Engineer, by the work the posting describes.

AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.

Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.

An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay
Success means reliable evidence of what a product gets right, where it fails, and whether it is ready for use. Evaluations catch important errors before release, guide measurable improvements, and detect declining quality in live operations. Testing supports sound decisions, not simply a passing score. Associate Director Salary range: $180,000 - $250,000 Job title and level will be determined based on qualifications and experience.
Location & working pattern

Remote - USA

Working pattern and location restrictions need checking in the full posting.

Work authorization
Job title and level will be determined based on qualifications and experience. Candidates must be able to verify their legal right to work in the United States without employer sponsorship now or in the future. About BRG

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.