Back to jobs

Machine Learning Performance Engineer, Training

Pay
$200,000/year · BasePlus bonus — pay source
Prior experience in financial trading is not required. Anticipated New York annual base salary of $200,000, plus eligible for discretionary bonus. Benefits
Read the full posting
Work setting
Not stated
Employment
Not stated

What you’ll work on

Full posting
  • Benchmark training workloads across CPUs, GPUs and other accelerators to find bottlenecks

  • Design and optimize distributed training strategies across multi-GPU, multi-node setups

  • Develop and optimize GPU kernels and performance-critical framework components

  • Analyze and improve the full training pipeline from data loading to checkpointing

  • Partner with HPC and infrastructure teams on scheduling, observability and fault tolerance

From the employer’s posting
Benchmark model-training workloads across CPUs, GPUs, and other accelerator platforms to identify bottlenecks
Design and optimize distributed training strategies, including data, tensor, pipeline, and model parallelism.
Develop and optimize GPU kernels and performance-critical framework components for quantitative machine learning workloads.
Analyze and improve the full training pipeline, including data loading, preprocessing, memory management, forward and backward passes, optimizer execution, checkpointing, and experiment recovery.
Partner with HPC and infrastructure teams to optimize workload scheduling, resource allocation, observability, fault tolerance, and reproducibility

Before you apply

Employer’s visa record
50 H-1B petitions approved in fiscal 2024 and 2025For the employer as a whole. This posting doesn’t say whether the role is sponsored. US Citizenship and Immigration Services.

What you’ll bring

All qualifications

Core experience

  • 3+ years of experience optimizing machine learning training workloads in high-performance, distributed, or large-scale computing environments.
  • Deep knowledge of machine learning frameworks such as PyTorch or JAX, including their execution models, compilation paths, autograd systems, and distributed-training capabilities.
  • Proven experience with GPU kernel development and optimization using technologies such as CUDA, Triton, CUTLASS, cuBLAS, cuDNN, or related libraries.
  • Strong understanding of GPU architecture, including streaming multiprocessor execution, warp scheduling, tensor cores, and the memory hierarchy from registers through HBM.
  • Experience with distributed-training technologies and communication libraries such as NCCL, FSDP, DeepSpeed, Megatron-LM, XLA, or equivalent systems.
  • Proficiency with performance-analysis tools such as Nsight Systems, Nsight Compute, PyTorch Profiler, or comparable tracing and profiling platforms.

Preferred experience

  • Experience optimizing training workloads for transformer-based, time-series, reinforcement-learning, or other computationally intensive models.
  • Experience with cluster orchestration and scheduling technologies such as Kubernetes, Slurm, Ray, or similar platforms.
  • Familiarity with fault-tolerant distributed training, large-scale checkpointing, experiment reproducibility, and GPU-cluster observability.
  • Practical experience with specialized accelerators, custom hardware, or compiler technologies for machine learning.
Qualification wording
3+ years of experience optimizing machine learning training workloads in high-performance, distributed, or large-scale computing environments.
Deep knowledge of machine learning frameworks such as PyTorch or JAX, including their execution models, compilation paths, autograd systems, and distributed-training capabilities.
Proven experience with GPU kernel development and optimization using technologies such as CUDA, Triton, CUTLASS, cuBLAS, cuDNN, or related libraries.
Strong understanding of GPU architecture, including streaming multiprocessor execution, warp scheduling, tensor cores, and the memory hierarchy from registers through HBM.
Experience with distributed-training technologies and communication libraries such as NCCL, FSDP, DeepSpeed, Megatron-LM, XLA, or equivalent systems.
Proficiency with performance-analysis tools such as Nsight Systems, Nsight Compute, PyTorch Profiler, or comparable tracing and profiling platforms.
Experience optimizing training workloads for transformer-based, time-series, reinforcement-learning, or other computationally intensive models.
Experience with cluster orchestration and scheduling technologies such as Kubernetes, Slurm, Ray, or similar platforms.
Familiarity with fault-tolerant distributed training, large-scale checkpointing, experiment reproducibility, and GPU-cluster observability.
Practical experience with specialized accelerators, custom hardware, or compiler technologies for machine learning.

Benefits in the posting

Full benefits wording
  • Our benefits include:
  • Generous paid time off policies
  • Savings plans and other financial wellness tools available in each region
  • Hybrid working opportunities
  • In-office wellness experiences and reimbursement for select wellness expenses (e.g., gym, personal training and more)
  • Volunteer opportunities and charitable giving
  • Workshops and continuous learning opportunities

From the employer’s posting.

Apply at Tower Research Capital

Posting record

Report an error
Status in our records
Live
First seen by us
Oct 7, 202645 days ago
Recorded sightings
48
Last seen by us
Nov 21, 2026
Employer says posted
Sep 24, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Pay

In line with similar jobs

$200K against $166K–$237K, the middle half of what similar US postings state.

From 1,537 postings in the last year. Stated ranges, not salaries paid.

Time since we first saw it

Day 45

Similar jobs usually come down between day 7 and day 76.

From 6,398 similar postings that came down. Coming down is not always a hire.

Tower Research Capital is hiring

7 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • Hiring has slowed
  • Its jobs tend to stay up a long time
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • C++
  • Python
  • Kubernetes
  • PyTorch
Where the posting names them
- Deep knowledge of machine learning frameworks such as PyTorch or JAX, including their execution models, compilation paths, autograd systems, and distributed-training capabilities. - Strong programming skills in Python and C++, with experience developing or optimizing performance-critical systems. - Proven experience with GPU kernel development and optimization using technologies such as CUDA, Triton, CUTLASS, cuBLAS, cuDNN, or related libraries.
- Experience optimizing training workloads for transformer-based, time-series, reinforcement-learning, or other computationally intensive models. - Experience with cluster orchestration and scheduling technologies such as Kubernetes, Slurm, Ray, or similar platforms. - Familiarity with fault-tolerant distributed training, large-scale checkpointing, experiment reproducibility, and GPU-cluster observability.
- 3+ years of experience optimizing machine learning training workloads in high-performance, distributed, or large-scale computing environments. - Deep knowledge of machine learning frameworks such as PyTorch or JAX, including their execution models, compilation paths, autograd systems, and distributed-training capabilities. - Strong programming skills in Python and C++, with experience developing or optimizing performance-critical systems.
- Experience with distributed-training technologies and communication libraries such as NCCL, FSDP, DeepSpeed, Megatron-LM, XLA, or equivalent systems. - Proficiency with performance-analysis tools such as Nsight Systems, Nsight Compute, PyTorch Profiler, or comparable tracing and profiling platforms. - Understanding of high-performance networking, storage, and accelerator interconnects, including technologies such as InfiniBand, RDMA, NVLink, or NVSwitch.

Your next step

Complete your application on www.tower-research.com. The employer’s form will show what is required.

Applying through Greenhouse

  • No account needed: the form is one page, with an “Autofill my application” option that reads your resume.
  • Greenhouse doesn’t read resumes over 2.5 MB, or details placed in a header, footer, text box, table or second column. The file still attaches, but a recruiter has to type you in by hand.

Already applied? Track this application

Job description

View original posting ↗

Tower Research Capital is a leading quantitative trading firm founded in 1998. Tower has built its business on a high-performance platform and independent trading teams. We have a 25+ year track record of innovation and a reputation for discovering unique market opportunities.

Tower is home to some of the world’s best systematic trading and engineering talent. We empower portfolio managers to build their teams and strategies independently while providing the economies of scale that come from a large, global organization. 

Engineers thrive at Tower while developing electronic trading infrastructure at a world class level. Our engineers solve challenging problems in the realms of low-latency programming, FPGA technology, hardware acceleration and machine learning. Our ongoing investment in top engineering talent and technology ensures our platform remains unmatched in terms of functionality, scalability and performance.

At Tower, every employee plays a role in our success. Our Business Support teams are essential to building and maintaining the platform that powers everything we do — combining market access, data, compute, and research infrastructure with risk management, compliance, and a full suite of business services. Our Business Support teams enable our trading and engineering teams to perform at their best.

At Tower, employees will find a stimulating, results-oriented environment where highly intelligent and motivated colleagues inspire each other to reach their greatest potential.

Summary

You will bridge the gap between quantitative research and high-performance computing, building and optimizing the systems used to train machine learning models at scale. You will focus on accelerating the end-to-end training lifecycle—from data ingestion and distributed execution to kernel performance and hardware utilization—enabling researchers to iterate more quickly across increasingly complex models and datasets.

Responsibilities

  • Training Performance and Benchmarking
    • Benchmark model-training workloads across CPUs, GPUs, and other accelerator platforms to identify bottlenecks and guide Tower’s compute infrastructure decisions.
    • Develop performance models and standardized benchmarks for measuring throughput, utilization, scalability, and time to convergence.
  • Distributed Training Optimization
    • Design and optimize distributed training strategies, including data, tensor, pipeline, and model parallelism.
    • Improve communication efficiency across multi-GPU and multi-node environments by optimizing collective operations, topology awareness, and computation–communication overlap.
  • End-to-End Training Efficiency
    • Analyze and improve the full training pipeline, including data loading, preprocessing, memory management, forward and backward passes, optimizer execution, checkpointing, and experiment recovery.
    • Identify bottlenecks across compute, memory, storage, networking, and interconnects to increase accelerator utilization and researcher productivity.
  • GPU Kernel and Framework Development
    • Develop and optimize GPU kernels and performance-critical framework components for quantitative machine learning workloads.
    • Integrate specialized libraries, compilers, and execution techniques to improve throughput, memory efficiency, and numerical performance.
  • Model and Numerical Optimization
    • Apply techniques such as mixed-precision training, gradient accumulation, activation checkpointing, operator fusion, and memory-efficient optimizers.
    • Evaluate tradeoffs among training speed, numerical stability, reproducibility, model quality, and infrastructure cost.
  • Training Infrastructure
    • Partner with HPC and infrastructure teams to optimize workload scheduling, resource allocation, observability, fault tolerance, and reproducibility across shared compute environments.
    • Help define the architecture and tooling required to support large-scale experimentation across on-premises and cloud-based infrastructure.
  • Cross-Functional Collaboration
    • Work closely with ML Researchers, Quantitative Researchers, HPC Engineers, Systems Engineers, and hardware specialists to translate research requirements into highly efficient training systems.

Qualifications

  • 3+ years of experience optimizing machine learning training workloads in high-performance, distributed, or large-scale computing environments.
  • Deep knowledge of machine learning frameworks such as PyTorch or JAX, including their execution models, compilation paths, autograd systems, and distributed-training capabilities.
  • Strong programming skills in Python and C++, with experience developing or optimizing performance-critical systems.
  • Proven experience with GPU kernel development and optimization using technologies such as CUDA, Triton, CUTLASS, cuBLAS, cuDNN, or related libraries.
  • Strong understanding of GPU architecture, including streaming multiprocessor execution, warp scheduling, tensor cores, and the memory hierarchy from registers through HBM.
  • Experience with distributed-training technologies and communication libraries such as NCCL, FSDP, DeepSpeed, Megatron-LM, XLA, or equivalent systems.
  • Proficiency with performance-analysis tools such as Nsight Systems, Nsight Compute, PyTorch Profiler, or comparable tracing and profiling platforms.
  • Understanding of high-performance networking, storage, and accelerator interconnects, including technologies such as InfiniBand, RDMA, NVLink, or NVSwitch.
  • Demonstrated ability to benchmark heterogeneous compute platforms and make rigorous, data-driven recommendations about performance, scalability, and cost.

Preferred Qualifications

  • Experience optimizing training workloads for transformer-based, time-series, reinforcement-learning, or other computationally intensive models.
  • Experience with cluster orchestration and scheduling technologies such as Kubernetes, Slurm, Ray, or similar platforms.
  • Familiarity with fault-tolerant distributed training, large-scale checkpointing, experiment reproducibility, and GPU-cluster observability.
  • Practical experience with specialized accelerators, custom hardware, or compiler technologies for machine learning.
  • Prior experience in financial trading is not required.

Anticipated New York annual base salary of $200,000, plus eligible for discretionary bonus.

Benefits

Tower’s headquarters are in the historic Equitable Building, right in the heart of NYC’s Financial District and our impact is global, with over a dozen offices around the world. 

At Tower, we believe work should be both challenging and enjoyable. That is why we foster a culture where smart, driven people thrive – without the egos. Our open concept workplace, casual dress code, and well-stocked kitchens reflect the value we place on a friendly, collaborative environment where everyone is respected, and great ideas win.

Our benefits include:

  • Generous paid time off policies
  • Savings plans and other financial wellness tools available in each region
  • Hybrid working opportunities
  • Free breakfast, lunch and snacks daily 
  • In-office wellness experiences and reimbursement for select wellness expenses (e.g., gym, personal training and more) 
  • Volunteer opportunities and charitable giving 
  • Social events, happy hours, treats and celebrations throughout the year
  • Workshops and continuous learning opportunities

At Tower, you’ll find a collaborative and welcoming culture, a diverse team and a workplace that values both performance and enjoyment. No unnecessary hierarchy. No ego. Just great people doing great work – together.

Tower Research Capital is an equal opportunity employer.

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are down 4% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 508, the most 899, and the latest week had 508.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay
Prior experience in financial trading is not required. Anticipated New York annual base salary of $200,000, plus eligible for discretionary bonus. Benefits
Location & working pattern

New York, NY

- Savings plans and other financial wellness tools available in each region - Hybrid working opportunities - Free breakfast, lunch and snacks daily
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.