Back to jobs

AI/ML Training Performance Engineer

Pay
Pay not listedSimilar roles state $166K–$237K
Work setting
On-site — work setting source
Working Location: This is an on-site role based at Blackrock Neurotech's headquarters in Salt Lake City, Utah. Occasional travel may be required. How We Work
Read the full posting
Employment
Full-time — employment source
Employment type in the employer’s posting: SALARIED_FT
From the employer’s posting

What you’ll work on

Full posting
  • Profile the training path to find compute, memory, communication and I/O bottlenecks

  • Write and tune custom GPU kernels in CUDA, Triton, or HIP/ROCm

  • Design and tune distributed training across data, tensor, pipeline or sharded parallelism

  • Optimize precision, memory allocation, and checkpointing to fit larger models

  • Build checkpoint, restart, and recovery workflows and performance regression checks

From the employer’s posting
Profile the full training path to distinguish compute, memory, communication, CPU, and I/O bottlenecks
Write, tune, and validate custom GPU kernels using CUDA, Triton, HIP/ROCm, or the appropriate platform tools
Design and tune distributed training strategies, including data, tensor, pipeline, or sharded parallelism
Optimize tensor layouts, precision, memory allocation, activation checkpointing, operator fusion, and execution graphs
Build robust checkpoint, restart, and recovery workflows and performance regression checks

What you’ll bring

All qualifications

Core experience

  • Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field, or equivalent practical experience with 3+ years of relevant industry experience OR Master's degree with 1+ years of relevant industry experience OR PhD in a related field
  • Demonstrated experience improving the performance of substantial deep learning training workloads, with measured gains in speed, memory efficiency, or compute cost
  • Deep understanding of GPU execution, including memory hierarchies, memory coalescing, thread blocks, warps or wavefronts, occupancy, synchronization, and bandwidth limits
  • Hands-on experience developing and profiling GPU kernels with CUDA or HIP/ROCm, and the ability to diagnose correctness and performance at the hardware level
  • Deep knowledge of PyTorch or an equivalent framework, including automatic differentiation, computation graphs, tensor storage, compilation, and mixed-precision training
  • Strong understanding of deep learning architectures and the underlying computations that drive training performance

Preferred experience

  • Experience with Triton, compiler optimization, advanced GPU profiling tools, multiple accelerator generations, long-sequence or multimodal models, neural time series, or large-scale model training is a plus
Qualification wording
Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field, or equivalent practical experience with 3+ years of relevant industry experience OR Master's degree with 1+ years of relevant industry experience OR PhD in a related field
Demonstrated experience improving the performance of substantial deep learning training workloads, with measured gains in speed, memory efficiency, or compute cost
Deep understanding of GPU execution, including memory hierarchies, memory coalescing, thread blocks, warps or wavefronts, occupancy, synchronization, and bandwidth limits
Hands-on experience developing and profiling GPU kernels with CUDA or HIP/ROCm, and the ability to diagnose correctness and performance at the hardware level
Deep knowledge of PyTorch or an equivalent framework, including automatic differentiation, computation graphs, tensor storage, compilation, and mixed-precision training
Strong understanding of deep learning architectures and the underlying computations that drive training performance
Experience with Triton, compiler optimization, advanced GPU profiling tools, multiple accelerator generations, long-sequence or multimodal models, neural time series, or large-scale model training is a plus
Education & alternatives
What You Bring - Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field, or equivalent practical experience with 3+ years of relevant industry experience OR Master's degree with 1+ years of relevant industry experience OR PhD in a related field - Demonstrated experience improving the performance of substantial deep learning training workloads, with measured gains in speed, memory efficiency, or compute cost
Apply at Blackrock Neurotech

Posting record

Report an error
Status in our records
Live
First seen by us
Sep 20, 202657 days ago
Recorded sightings
36
Last seen by us
Nov 16, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Pay

$166K–$237K is typical

The middle half of what similar US postings state. This one doesn’t list pay.

From 1,501 postings in the last year. Stated ranges, not salaries paid.

Time since we first saw it

Day 57

Similar jobs usually come down between day 8 and day 77.

From 5,690 similar postings that came down. Coming down is not always a hire.

Blackrock Neurotech is hiring

2 open jobs

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • C++
  • Python
  • PyTorch
Where the posting names them
- A measurement-driven approach to performance optimization, using profiling and benchmarking to validate meaningful end-to-end improvements - Exceptional programming ability in Python and C++ or a comparable systems language, with strong debugging, testing, and performance analysis practices - Deep understanding of GPU execution, including memory hierarchies, memory coalescing, thread blocks, warps or wavefronts, occupancy, synchronization, and bandwidth limits
The ML Training Performance Engineer will own the efficiency and scalability of training models on GPU and cloud infrastructure. You will turn available compute into faster, more capable experiments as our model training scales in complexity and compute requirements. As a hands-on individual contributor on a small research team, you will work across the training stack, from Python and model execution to GPU kernels, distributed communication, and runtime environments. Partnering with model researchers, data engineers, and infrastructure and IT teams, you will identify and implement performance improvements while preserving numerical correctness and scientific intent. You will have significant ownership over how we measure, optimize, and scale training performance, establishing the baselines, tooling, and technical approaches that will support our AI/ML work as it grows.
- Write, tune, and validate custom GPU kernels using CUDA, Triton, HIP/ROCm, or the appropriate platform tools when existing implementations limit performance - Improve Python training scripts, framework and compiler settings, batching, gradient accumulation, and optimizer execution while preserving intended training behavior - Design and tune distributed training strategies, including data, tensor, pipeline, or sharded parallelism, based on model structure, memory limits, and interconnect topology
- Hands-on experience developing and profiling GPU kernels with CUDA or HIP/ROCm, and the ability to diagnose correctness and performance at the hardware level - Deep knowledge of PyTorch or an equivalent framework, including automatic differentiation, computation graphs, tensor storage, compilation, and mixed-precision training - Strong understanding of deep learning architectures and the underlying computations that drive training performance

Your next step

Complete your application on ats.rippling.com. The employer’s form will show what is required.

Applying through Rippling

  • “The résumé will be parsed to fill in the application details”: check what it filled before you send.

Already applied? Track this application

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are up 8% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 2. The fewest in a week was 612, the most 1,080, and the latest week had 975.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Job description

View original posting ↗

Build the systems that expand human capability


At Blackrock Neurotech, we’ve spent decades making the impossible possible – helping people move, speak, and reconnect with the world when they otherwise could not. We’ve seen that restoring function restores more than ability. It restores independence, identity, and agency.

Today, we are building the next generation of human capability: brain-computer interfaces that are designed to be safe, scalable, and trusted in the real world. Our work is not only about reconnecting people to what was lost, but about expanding what is possible – creating a seamless interface between human intent and technology.

This is foundational work in a category-defining field. You will help build the infrastructure for a future where neural interfaces are invisible, reliable, and deeply human-centered.


Working at Blackrock Neurotech means:
  • Owning meaningful, high-impact problems at the frontier of science and engineering
  • Building alongside experienced, thoughtful peers across disciplines
  • Solving technically complex challenges grounded in real human outcomes
  • Contributing to a culture that values rigor, clarity, and long-term thinking over noise



The Role

The ML Training Performance Engineer will own the efficiency and scalability of training models on GPU and cloud infrastructure. You will turn available compute into faster, more capable experiments as our model training scales in complexity and compute requirements.


As a hands-on individual contributor on a small research team, you will work across the training stack, from Python and model execution to GPU kernels, distributed communication, and runtime environments. Partnering with model researchers, data engineers, and infrastructure and IT teams, you will identify and implement performance improvements while preserving numerical correctness and scientific intent.


You will have significant ownership over how we measure, optimize, and scale training performance, establishing the baselines, tooling, and technical approaches that will support our AI/ML work as it grows.

What You'll Do

  • Own training performance across single-GPU, multi-GPU, and multi-node workloads, establishing reproducible baselines for throughput, memory use, utilization, time to target quality, and cost
  • Profile the full training path to distinguish compute, memory, communication, CPU, and I/O bottlenecks and prioritize changes with measurable end-to-end impact
  • Optimize tensor layouts, precision, memory allocation, activation checkpointing, operator fusion, and execution graphs to fit larger or longer-context models within available resources
  • Write, tune, and validate custom GPU kernels using CUDA, Triton, HIP/ROCm, or the appropriate platform tools when existing implementations limit performance
  • Improve Python training scripts, framework and compiler settings, batching, gradient accumulation, and optimizer execution while preserving intended training behavior
  • Design and tune distributed training strategies, including data, tensor, pipeline, or sharded parallelism, based on model structure, memory limits, and interconnect topology
  • Partner with model researchers on hardware-aware architecture and hyperparameter changes, measuring their effects on convergence, model quality, and compute requirements
  • Coordinate with the neural data infrastructure engineer on prefetching, pinned memory, host-to-device transfer, and I/O overlap so data delivery keeps pace with training
  • Work with infrastructure and IT on GPU selection, cloud instance configurations, networking, drivers, containers, scheduling, and capacity planning as training needs and compute capacity scale
  • Build robust checkpoint, restart, and recovery workflows and performance regression checks that keep long-running experiments reproducible and productive
  • Communicate benchmark evidence, numerical tradeoffs, scaling limits, and resource recommendations clearly to researchers and organizational stakeholders

What You Bring

  • Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field, or equivalent practical experience with 3+ years of relevant industry experience OR Master's degree with 1+ years of relevant industry experience OR PhD in a related field
  • Demonstrated experience improving the performance of substantial deep learning training workloads, with measured gains in speed, memory efficiency, or compute cost
  • A measurement-driven approach to performance optimization, using profiling and benchmarking to validate meaningful end-to-end improvements
  • Exceptional programming ability in Python and C++ or a comparable systems language, with strong debugging, testing, and performance analysis practices
  • Deep understanding of GPU execution, including memory hierarchies, memory coalescing, thread blocks, warps or wavefronts, occupancy, synchronization, and bandwidth limits
  • Hands-on experience developing and profiling GPU kernels with CUDA or HIP/ROCm, and the ability to diagnose correctness and performance at the hardware level
  • Deep knowledge of PyTorch or an equivalent framework, including automatic differentiation, computation graphs, tensor storage, compilation, and mixed-precision training
  • Strong understanding of deep learning architectures and the underlying computations that drive training performance
  • Experience with distributed training, collective communication, sharding, and the interaction between model partitioning and GPU interconnects
  • Strong understanding of numerical stability and the ability to validate gradients, convergence, and model quality after performance changes
  • Experience configuring and diagnosing Linux-based GPU environments, containers, cloud compute, and high-throughput storage or networking
  • Ability to collaborate closely with researchers and infrastructure teams and make clear tradeoffs between implementation effort, performance, reliability, and scientific value
  • Experience with Triton, compiler optimization, advanced GPU profiling tools, multiple accelerator generations, long-sequence or multimodal models, neural time series, or large-scale model training is a plus


Working Location:

This is an on-site role based at Blackrock Neurotech's headquarters in Salt Lake City, Utah. Occasional travel may be required.



How We Work

We are a small, experienced team working on consequential problems.

  • We take ownership of outcomes and follow through with clarity and accountability
  • We prioritize sustained, high-quality work over performative urgency
  • We value rigor, sound judgement and thoughtful decision-making
  • We collaborate deliberately: low ego, high trust and high context

This is a high-ownership role, but it is not an "always-on" one. We expect strong work and our people to have a life outside of it.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Salt Lake City, UT

Working Location: This is an on-site role based at Blackrock Neurotech's headquarters in Salt Lake City, Utah. Occasional travel may be required. How We Work
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.