Back to jobs

Senior Staff Machine Learning Performance Engineer

Pay
$244,140–413,160/year · BaseAnnual period assumed — pay source
Snacks, lunches, dinners, and fun activities. The base salary range for this full-time position is $244,140, - $413,160, in addition to bonus, equity and benefits. Our salary ranges are determined by role, level, and location. The range displayed on each job posting reflects the minimum and maximum target for new hire salaries for the position across all US locations. Within the range, individual pay is determined by work location and additional factors, including job-related skills, experience, and relevant education or training. We are an Equal Opportunity Employer. It is our policy to provide equal employment opportunities to all qualified persons without regard to race, age, color, sex, sexual orientation, religion, national origin, disability, veteran status or marital status or any other prescribed category set forth in federal or state regulations.
Read the full posting
Work setting
Not stated
Employment
Not stated

What you’ll work on

Full posting
  • Own step time, scaling efficiency and stability for large multi-node model training runs

  • Work with modeling and infrastructure teams on their hardest performance bottlenecks

  • Turn one-off speedups into modular components that become defaults across workloads

  • Extend profiling and metrics coverage with low instrumentation overhead

  • Build agents that reproduce jobs, check numerical parity and separate regressions from noise

From the employer’s posting
Own step time, scaling efficiency and long-run stability for foundation-model training across thousands of accelerators
Work directly with the modeling, infrastructure, simulation, data-mining and robotics teams on their hardest bottlenecks
Turn one-off wins into modular components that become defaults across most models and workloads
Extend performance tooling and metric coverage across the stack, while keeping instrumentation overhead low
Build agents that reproduce any job on demand, verify numerical parity and separate a real regression from run-to-run noise
Education & alternatives
Minimum Skill Requirements: - Master's or PhD in CS, CE or EE, or equivalent practical experience, with 8+ years of industry experience, the majority of it in performance engineering. - Hands-on experience optimizing GPU training or inference workloads in production, at multi-node scale.

Tools in this posting

  • C++
  • Python
  • PyTorch
Source — Tool mentions in context
- In-depth understanding of computer architecture and high-performance computing. - Strong C++ and Python, with a solid command of data structures and algorithms. Preferred Skill Requirements:
- Hands-on experience writing and tuning CUDA or Triton kernels. - Fluent in PyTorch, with concrete insight into what makes PyTorch code fast or slow — graph breaks, host-side overhead, synchronization points and memory layout. - Training at the scale of thousands of accelerators, including multi-node failure modes and long-running job stability.
Check the employer’s page ↗

Posting record

Report an error
Status in our records
Live
First seen by us
Oct 27, 202625 days ago
Recorded sightings
28
Last seen by us
Nov 22, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Pay

Higher than most similar jobs

$244K–$413K against $166K–$237K, the middle half of what similar US postings state.

From 1,537 postings in the last year. Stated ranges, not salaries paid.

Time since we first saw it

Day 25

Similar jobs usually come down between day 7 and day 76.

At XPENG, jobs usually come down by about day 72.

From 6,398 similar postings that came down, and 9 of XPENG’s. Coming down is not always a hire.

XPENG is hiring

6 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • Its jobs tend to stay up a long time
Hiring record

Your next step

Check the employer’s posting for the current role and application details.

Applying through Greenhouse

  • No account needed: the form is one page, with an “Autofill my application” option that reads your resume.
  • Greenhouse doesn’t read resumes over 2.5 MB, or details placed in a header, footer, text box, table or second column. The file still attaches, but a recruiter has to type you in by hand.

Already applied? Track this application

Job description

View original posting ↗

XPENG is a leading smart technology company at the forefront of innovation, integrating advanced AI and autonomous driving technologies into its vehicles, including electric vehicles (EVs), electric vertical take-off and landing (eVTOL) aircraft, and robotics. With a strong focus on intelligent mobility, XPENG is dedicated to reshaping the future of transportation through cutting-edge R&D in AI, machine learning, and smart connectivity.
 
The Mission: We are building the next generation of end-to-end autonomous driving on physical AI foundation models — vision-language-action policies and world models — and the same training stack carries our robotics program. These training runs span thousands of accelerators and days of wall-clock — how fast we train them is how fast we learn. As a Senior Staff Machine Learning Performance Engineer, you will own that speed: working with the modeling teams on their hardest bottlenecks, setting the technical direction for performance, and turning one-off wins into defaults — including the agentic workflows that find the next round of them automatically.
 
Job Responsibilities:
  • Training Performance at Scale: Own step time, scaling efficiency and long-run stability for foundation-model training across thousands of accelerators — from the data path to kernels, parallelism and communication.
  • Cross-Team Technical Leadership: Work directly with the modeling, infrastructure, simulation, data-mining and robotics teams on their hardest bottlenecks, and set the performance direction those teams build against.
  • Reusable by Design: Turn one-off wins into modular components that become defaults across most models and workloads, rather than a patch that helps a single job.
  • Observability at Scale: Extend performance tooling and metric coverage across the stack, while keeping instrumentation overhead low enough to stay on for production jobs fleet-wide.
  • Agentic Automation: Build agents that reproduce any job on demand, verify numerical parity and separate a real regression from run-to-run noise — so that most of the optimizations that land are found and fixed by AI rather than by hand.
Minimum Skill Requirements:
  • Master's or PhD in CS, CE or EE, or equivalent practical experience, with 8+ years of industry experience, the majority of it in performance engineering.
  • Hands-on experience optimizing GPU training or inference workloads in production, at multi-node scale.
  • A track record of diagnosing performance problems in real production workloads and landing the fixes — often in systems owned by other teams — with the gain measured and verified in production.
  • In-depth understanding of computer architecture and high-performance computing.
  • Strong C++ and Python, with a solid command of data structures and algorithms.
Preferred Skill Requirements: 
  • Hands-on experience writing and tuning CUDA or Triton kernels.
  • Fluent in PyTorch, with concrete insight into what makes PyTorch code fast or slow — graph breaks, host-side overhead, synchronization points and memory layout.
  • Training at the scale of thousands of accelerators, including multi-node failure modes and long-running job stability.
  • Building profiling, tracing or benchmarking infrastructure that other teams depend on — CUPTI, Perfetto, or equivalent.
  • Using LLM agents to automate parts of an engineering workflow.
  • Autonomous driving, robotics, or another domain with hard latency, power and memory constraints.
  • Publications or open-source contributions in ML systems or performance.
What do we provide:
  • A fun, supportive and engaging environment.
  • Infrastructures and computational resources to support your work.
  • Opportunity to work on cutting edge technologies with the top talents in the field.
  • Opportunity to make a significant impact on the transportation revolution by the means of advancing autonomous driving.
  • Competitive compensation package.
  • Snacks, lunches, dinners, and fun activities.
 
The base salary range for this full-time position is $244,140, - $413,160, in addition to bonus, equity and benefits. Our salary ranges are determined by role, level, and location. The range displayed on each job posting reflects the minimum and maximum target for new hire salaries for the position across all US locations. Within the range, individual pay is determined by work location and additional factors, including job-related skills, experience, and relevant education or training.
 
We are an Equal Opportunity Employer. It is our policy to provide equal employment opportunities to all qualified persons without regard to race, age, color, sex, sexual orientation, religion, national origin, disability, veteran status or marital status or any other prescribed category set forth in federal or state regulations.

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are down 4% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 508, the most 899, and the latest week had 508.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay
Snacks, lunches, dinners, and fun activities. The base salary range for this full-time position is $244,140, - $413,160, in addition to bonus, equity and benefits. Our salary ranges are determined by role, level, and location. The range displayed on each job posting reflects the minimum and maximum target for new hire salaries for the position across all US locations. Within the range, individual pay is determined by work location and additional factors, including job-related skills, experience, and relevant education or training. We are an Equal Opportunity Employer. It is our policy to provide equal employment opportunities to all qualified persons without regard to race, age, color, sex, sexual orientation, religion, national origin, disability, veteran status or marital status or any other prescribed category set forth in federal or state regulations.
Location & working pattern

Santa Clara, CA

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

AI answers unavailable

We couldn’t identify enough role detail in this saved description to support an AI answer. Read the full posting