Back to jobs

Senior AI Engineer for Observability

Reads like an ML Engineer role: training models and running them in production.

Pay
$80,000–175,000/yearAnnual period assumed — pay source
Salary: $80,000.00 - $175,000.00 Pay Type:
Read the full posting
Work setting
Not stated
Employment
Full-time — employment source
Employment type in the employer’s posting: Full time
From the employer’s posting

What you’ll bring

All qualifications

Core experience

  • Bachelor's Degree in Computer Science, Engineering or other relevant domain
  • 7-10 years experience in Site Reliability Engineering
  • Experience in Artificial Intelligence Observability Engineering, facilitating the ability to Automate Detect, Investigate, and Remediate
  • Experience using Python for custom evaluations and SQL or LogQL/PromQL for querying telemetry, logs, and metrics data.
Qualification wording
Bachelor's Degree in Computer Science, Engineering or other relevant domain
7-10 years experience in Site Reliability Engineering
Experience in Artificial Intelligence Observability Engineering, facilitating the ability to Automate Detect, Investigate, and Remediate
Experience using Python for custom evaluations and SQL or LogQL/PromQL for querying telemetry, logs, and metrics data.
Education & alternatives
Qualifications: - Bachelor's Degree in Computer Science, Engineering or other relevant domain - 7-10 years experience in Site Reliability Engineering

About Bank of Montreal

At BMO we are driven by a shared Purpose: Boldly Grow the Good in business and life.

In the employer’s words · Read in context

Apply at Bank of Montreal

Posting record

Report an error
Status in our records
Live
First seen by us
Oct 21, 202628 days ago
Recorded sightings
47
Last seen by us
Nov 16, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 28

Similar jobs usually come down between day 7 and day 75.

At Bank of Montreal, jobs usually come down by about day 15.

From 5,846 similar postings that came down, and 146 of Bank of Montreal’s. Coming down is not always a hire.

Bank of Montreal is hiring

44 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • Hiring at a steady pace
  • Takes jobs down quickly
  • 9% of open jobs were reposted
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • SQL
  • Datadog
  • Grafana
Where the posting names them
- Highly proficient in automation, monitoring, observability, buildouts, AI engineering, understand model architectures, evaluation metrics, token usage, and how to spot concept or data drift - Experience using Python for custom evaluations and SQL or LogQL/PromQL for querying telemetry, logs, and metrics data. - Provision, configure, and standardize observability pipelines, logging sidecars, and centralized dashboards using Prometheus, Grafana, Datadog) to track SLIs/SLOs, handle alert fatigue, and run root cause analysis.
- Experience using Python for custom evaluations and SQL or LogQL/PromQL for querying telemetry, logs, and metrics data. - Provision, configure, and standardize observability pipelines, logging sidecars, and centralized dashboards using Prometheus, Grafana, Datadog) to track SLIs/SLOs, handle alert fatigue, and run root cause analysis. - Exposure to heavily regulated industry organizations

Your next step

Complete your application on bmo.wd3.myworkdayjobs.com. The employer’s form will show what is required.

Applying through Workday

  • You create an account with each employer before you can upload anything. Every Workday employer has its own account, so one sign-up doesn’t carry over.
  • “Autofill with Resume” comes right after sign-in, before the My Information and My Experience steps. Check every field it fills in those steps, since that’s what recruiters see.
  • Your application status lives in that employer’s Candidate Home, under the same sign-in.

Already applied? Track this application

Job description

View original posting ↗

Application Deadline:

11/29/2026

Address:

33 Dundas Street West

Job Family Group:

Data Analytics & Reporting

Hybrid role

Designs, builds, and maintains systems that track, evaluate, and debug non-deterministic AI models and agent workflows. Trace LLM calls, monitor token usage, detect drift, set up automated evaluations, and govern autonomous agent behaviors in production.

  • Designs and develops machine learning (ML) and deep learning systems.
  • Runs machine learning tests and experiments. Trains and retrain systems to prevent drift and optimize results.
  • Solves complex problems with multi-layered data sets, extends existing ML frameworks and optimizes existing machine learning libraries.
  • Develops Machine Learning apps, implements algorithms, and builds tools to apply ML frameworks.
  • Turns unstructured data into useful information by auto-tagging images and text-to-speech conversions.
  • Develops ML algorithms to analyze huge volumes of historical data to make predictions.
  • Runs tests, performs statistical analysis, and interprets test results.
  • Focus is primarily on business/group within BMO; may have broader, enterprise-wide focus.
  • Provides specialized consulting, analytical and technical support.
  • Exercises judgment to identify, diagnose, and solve problems within given rules.
  • Works independently and regularly handles non-routine situations.
  • Broader work or accountabilities may be assigned as needed.
  • Take measured risks while protecting the bank by applying our Risk Management Framework in the execution of your role, in line with our Risk Culture and within our approved Risk Appetite, making sound and risk informed decisions that align to business strategy, protect assets, and adhere to applicable policy documents (Frameworks, Policies, Standards, Procedures and Supporting documents), laws and regulations.

Qualifications:


  • Bachelor's Degree in Computer Science, Engineering or other relevant domain
  • 7-10 years experience in Site Reliability Engineering
  • Experience in Artificial Intelligence Observability Engineering, facilitating the ability to Automate Detect, Investigate, and Remediate
  • Exposure to Platform Engineering, building AI capabilities
  • Highly proficient in automation, monitoring, observability, buildouts, AI engineering, understand model architectures, evaluation metrics, token usage, and how to spot concept or data drift
  • Experience using Python for custom evaluations and SQL or LogQL/PromQL for querying telemetry, logs, and metrics data.
  • Provision, configure, and standardize observability pipelines, logging sidecars, and centralized dashboards using Prometheus, Grafana, Datadog) to track SLIs/SLOs, handle alert fatigue, and run root cause analysis.
  • Exposure to heavily regulated industry organizations

    • Agent & LLM Tracing: Implement OpenTelemetry-based frameworks to capture step-by-step agent trajectories, spans, prompts, tool calls, and intermediate outputs.
    • Evaluation & Quality Assurance: Design automated evaluation pipelines for relevance, groundedness, hallucination detection, and prompt adherence using LLM-as-a-judge or custom code.
    • Drift & Performance Monitoring: Track model drift, data quality degradation, latency spikes, and error rates across production pipelines.
    • Cost & Token Optimization: Build telemetry and sampling strategies to control high data and inference costs, managing log retention and cardinality.
    • Agent Governance & Guardrails: Define operational boundaries for autonomous AI agents, setting rules for when they can self-remediate.
    • Alert Hygiene & Incident Response: Reduce alert fatigue by replacing static thresholds with dynamic, intelligent anomaly detection and building runbooks for AI failures.



Salary:

$80,000.00 - $175,000.00

Pay Type:

Salaried

The above represents BMO Financial Group’s pay range and type.

Salaries will vary based on factors such as location, skills, experience, education, and qualifications for the role, and may include a commission structure. Salaries for part-time roles will be pro-rated based on number of hours regularly worked. For commission roles, the salary listed above represents BMO Financial Group’s expected target for the first year in this position.

BMO Financial Group’s total compensation package will vary based on the pay type of the position and may include performance-based incentives, discretionary bonuses, as well as other perks and rewards. BMO also offers health insurance, tuition reimbursement, accident and life insurance, and retirement savings plans. To view more details of our benefits, please visit: https://jobs.bmo.com/global/en/Total-Rewards

About Us

At BMO we are driven by a shared Purpose: Boldly Grow the Good in business and life. It calls on us to create lasting, positive change for our customers, our communities and our people. By working together, innovating and pushing boundaries, we transform lives and businesses, and power economic growth around the world.

As a member of the BMO team you are valued, respected and heard, and you have more ways to grow and make an impact. We strive to help you make an impact from day one – for yourself and our customers. We’ll support you with the tools and resources you need to reach new milestones, as you help our customers reach theirs. From in-depth training and coaching, to manager support and network-building opportunities, we’ll help you gain valuable experience, and broaden your skillset.

To find out more visit us at https://jobs.bmo.com/ca/en.

BMO is committed to an inclusive, equitable and accessible workplace. By learning from each other’s differences, we gain strength through our people and our perspectives. Accommodations are available on request for candidates taking part in all aspects of the selection process. To request accommodation, please contact your recruiter.

Note to Recruiters: BMO does not accept unsolicited resumes from any source other than directly from a candidate. Any unsolicited resumes sent to BMO, directly or indirectly, will be considered BMO property. BMO will not pay a fee for any placement resulting from the receipt of an unsolicited resume. A recruiting agency must first have a valid, written and fully executed agency agreement contract for service to submit resumes.

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are up 12% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 556, the most 1,080, and the latest week had 556.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay
Salary: $80,000.00 - $175,000.00 Pay Type:
Location & working pattern

Toronto, ON, CAN

Data Analytics & Reporting Hybrid role Designs, builds, and maintains systems that track, evaluate, and debug non-deterministic AI models and agent workflows. Trace LLM calls, monitor token usage, detect drift, set up automated evaluations, and govern autonomous agent behaviors in production.
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.