Back to jobs

Founding Senior AI Infrastructure Engineer

Palo Alto, CA, USA · Palo Alto, California, United States

Actively hiring, first seen by us 1 day ago
Pay
Pay not listedSimilar roles state $166K–$237K
Work setting
On-site — work setting source
Location type: On-site
From the employer’s posting
Employment
Full-time — employment source
Employment type: Full-time
From the employer’s posting
Team
Engineering — team source
Department: Engineering
From the employer’s posting

What you’ll work on

Full posting

Running modern AI workloads at scale creates systems problems that rarely fit within a single layer of the stack.

You will identify these bottlenecks and build systems that improve the throughput, efficiency, and robustness of our largest distributed workloads.

  • Profile end-to-end AI workloads to find limiting resources across kernels, memory, networking, and storage

  • Build low-latency, high-throughput sampling and inference systems for large language models

  • Optimize GPU execution through kernel profiling, reduced-precision computation, and targeted CUDA or Triton work

  • Design fault-tolerant distributed systems that detect, isolate, and recover from failures in long jobs

  • Build benchmarks, profiling tools, dashboards, and regression tests for system performance

From the employer’s posting
Running modern AI workloads at scale creates systems problems that rarely fit within a single layer of the stack. A slowdown that appears in a training job may originate in a GPU kernel, collective communication, data movement, container runtime, storage path, scheduler, or interaction between model architecture and hardware topology.
You will identify these bottlenecks and build systems that improve the throughput, efficiency, and robustness of our largest distributed workloads. Your scope will span post-training, agentic reinforcement learning, model training, rollout inference, and the GPU cluster platform beneath them. You will work closely with researchers and systems engineers, develop a quantitative understanding of performance, and turn one-off investigations into durable infrastructure improvements.
Profile end-to-end AI workloads and identify limiting resources across model code, GPU kernels, memory, collective communication, networking, storage, orchestration, and environment execution.
Build low-latency, high-throughput sampling and inference systems for large language models, including batching, scheduling, caching, load balancing, and efficient weight updates.
Optimize GPU execution through kernel and graph profiling, memory-layout improvements, reduced-precision computation, communication overlap, compilation, and targeted CUDA or Triton work.
Design fault-tolerant distributed systems that detect failures early, isolate their impact, recover efficiently, and preserve correctness during long-running jobs.
Develop benchmarks, profiling tools, performance dashboards, and regression tests that make system behavior visible and allow improvements to be measured under realistic workloads.

Before you apply

Sponsorship
Visa sponsorship not confirmed — sponsorship source
Do you need visa sponsorship?
From the employer’s application form

About Goaly

Our mission is to dramatically lower the cost, time, and talent barriers to building proprietary AI — and make each generation of models faster and cheaper to build than the last.

In the employer’s words · Read in context · Company website ↗

Apply at Goaly

Posting record

Report an error

Actively hiringFirst seen by us 1 day ago, still listed, and not a repost of an earlier posting.

Status in our records
Live
First seen by us
Nov 18, 20261 day ago
Recorded sightings
3
Last seen by us
Nov 20, 2026
Employer says posted
Aug 28, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Pay

$166K–$237K is typical

The middle half of what similar US postings state. This one doesn’t list pay.

From 1,507 postings in the last year. Stated ranges, not salaries paid.

Time since we first saw it

Day 1

Early. Similar jobs usually come down between day 7 and day 76.

From 6,269 similar postings that came down. Coming down is not always a hire.

Goaly is hiring

4 open jobs

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • C++
  • Python
  • Rust
  • Kubernetes
  • PyTorch
  • Go
Where the posting names them
- Significant software-engineering, distributed-systems, high-performance-computing, or ML-infrastructure experience, particularly with performance-critical systems operating at large scale. - Exceptional programming and debugging ability in Python and at least one systems language such as C++, Rust, or Go. - Strong systems fundamentals, including operating systems, concurrency, memory, networking, storage, scheduling, containerization, and failure handling.
- Knowledge of NCCL or RCCL, RDMA, InfiniBand, RoCE, NVLink, collective algorithms, network topology, or topology-aware workload placement. - Experience with Linux or operating-system internals, container runtimes, Kubernetes, cluster schedulers, storage systems, or production observability. - Familiarity with transformer architectures, language-model training, reinforcement learning, online sampling, or the performance characteristics of mixture-of-experts models.
- Experience with GPU or accelerator programming, including CUDA, Triton, custom kernels, low-precision computation, memory optimization, or compiler stacks. - Familiarity with ML framework internals and distributed stacks such as PyTorch, JAX, torch.distributed, FSDP, Megatron, DeepSpeed, Ray, vLLM, SGLang, or TensorRT-LLM. - Knowledge of NCCL or RCCL, RDMA, InfiniBand, RoCE, NVLink, collective algorithms, network topology, or topology-aware workload placement.

Your next step

Complete your application on jobs.ashbyhq.com. The employer’s form will show what is required.

Applying through Ashby

  • No account needed. “Autofill from resume” at the top of the form fills the key fields from your file.

Already applied? Track this application

Job description

View original posting ↗

About us

We’re building toward a world where every company can become its own AI lab.
Goaly is a stealth AI startup founded by ex-Meta MSL engineers and researchers. Our mission is to dramatically lower the cost, time, and talent barriers to building proprietary AI — and make each generation of models faster and cheaper to build than the last.

Backed by leading AI investors and endorsed by frontier AI researchers and builders, we’re looking for exceptional new grads who want to work on hard, foundational AI systems problems with outsized ownership from day one.

About the role

Running modern AI workloads at scale creates systems problems that rarely fit within a single layer of the stack. A slowdown that appears in a training job may originate in a GPU kernel, collective communication, data movement, container runtime, storage path, scheduler, or interaction between model architecture and hardware topology.

You will identify these bottlenecks and build systems that improve the throughput, efficiency, and robustness of our largest distributed workloads. Your scope will span post-training, agentic reinforcement learning, model training, rollout inference, and the GPU cluster platform beneath them. You will work closely with researchers and systems engineers, develop a quantitative understanding of performance, and turn one-off investigations into durable infrastructure improvements.

This role is a strong fit for an exceptional systems or performance engineer who wants to work on frontier AI infrastructure. Deep ML experience is valuable but not required; we care most about a track record of solving difficult systems problems at scale and the ability to become fluent in new parts of the ML stack quickly.

What you'll do

  • Profile end-to-end AI workloads and identify limiting resources across model code, GPU kernels, memory, collective communication, networking, storage, orchestration, and environment execution.

  • Build low-latency, high-throughput sampling and inference systems for large language models, including batching, scheduling, caching, load balancing, and efficient weight updates.

  • Optimize GPU execution through kernel and graph profiling, memory-layout improvements, reduced-precision computation, communication overlap, compilation, and targeted CUDA or Triton work.

  • Improve distributed training and reinforcement-learning performance across heterogeneous GPU and CPU workloads, variable-length rollouts, complex network topologies, and changing model architectures.

  • Design quantitative performance and capacity models that predict bottlenecks, explain scaling behavior, guide hardware and topology choices, and prioritize engineering work.

  • Build scheduling and load-balancing mechanisms that improve accelerator utilization while respecting memory, locality, topology, latency, and fault-domain constraints.

  • Design fault-tolerant distributed systems that detect failures early, isolate their impact, recover efficiently, and preserve correctness during long-running jobs.

  • Investigate difficult production issues such as kernel-level stalls, network-latency spikes, collective timeouts, memory fragmentation, stragglers, and performance regressions in containerized environments.

  • Develop benchmarks, profiling tools, performance dashboards, and regression tests that make system behavior visible and allow improvements to be measured under realistic workloads.

  • Partner with researchers to understand new models and algorithms, remove infrastructure constraints from the experimental loop, and translate successful optimizations into reusable platform capabilities.

What success looks like

  • Within your first three months, you have built a quantitative understanding of at least one critical workload, identified its dominant bottlenecks, and shipped a measurable performance or reliability improvement.

  • Within six to twelve months, you have delivered sustained gains in throughput, accelerator utilization, latency, scaling efficiency, or cost across real training, rollout, or inference workloads.

  • Performance investigations become faster and more rigorous because the team has better benchmarks, models, profiles, and observability—not just undocumented fixes.

  • Large distributed jobs run predictably across complex hardware and network topologies, and failures or regressions can be detected, explained, and recovered from with minimal researcher intervention.

You may be a good fit if you have

  • Significant software-engineering, distributed-systems, high-performance-computing, or ML-infrastructure experience, particularly with performance-critical systems operating at large scale.

  • Exceptional programming and debugging ability in Python and at least one systems language such as C++, Rust, or Go.

  • Strong systems fundamentals, including operating systems, concurrency, memory, networking, storage, scheduling, containerization, and failure handling.

  • A track record of using measurement to solve ambiguous performance problems: forming hypotheses, designing representative benchmarks, reading profiles and traces, identifying root causes, and validating improvements under production conditions.

  • The ability to reason across abstraction boundaries, from model architecture and framework execution to accelerator behavior, distributed runtimes, cluster topology, and infrastructure services.

  • A results-oriented mindset, flexibility about where in the stack to work, and a willingness to take ownership beyond a narrowly defined job description.

  • Strong communication and collaboration skills, including the ability to work directly with researchers, explain complex systems behavior clearly, and turn repeated investigations into maintainable tools and abstractions.

  • Interest in developing deep expertise in machine learning systems, even if your prior work has been primarily in distributed systems, HPC, operating systems, compilers, databases, or networking.

Strong pluses

  • Experience building or operating high-performance, large-scale ML training, post-training, or inference systems.

  • Experience with GPU or accelerator programming, including CUDA, Triton, custom kernels, low-precision computation, memory optimization, or compiler stacks.

  • Familiarity with ML framework internals and distributed stacks such as PyTorch, JAX, torch.distributed, FSDP, Megatron, DeepSpeed, Ray, vLLM, SGLang, or TensorRT-LLM.

  • Knowledge of NCCL or RCCL, RDMA, InfiniBand, RoCE, NVLink, collective algorithms, network topology, or topology-aware workload placement.

  • Experience with Linux or operating-system internals, container runtimes, Kubernetes, cluster schedulers, storage systems, or production observability.

  • Familiarity with transformer architectures, language-model training, reinforcement learning, online sampling, or the performance characteristics of mixture-of-experts models.

  • Meaningful contributions to open-source systems, ML frameworks, compilers, kernels, networking software, or performance tooling.

How we work

  • Mission first. We choose work for its impact on the mission and take responsibility for the outcome, not just our assigned tasks.

  • High agency. We identify what is missing, form a plan, and move without waiting for perfect clarity.

  • Speed with rigor. We ship, measure, and iterate quickly while protecting correctness, safety, and reliability.

  • Flexible scope. We cross team and technical boundaries when that is the fastest way to solve the real problem.

  • Low ego, high standards. We give direct feedback, change our minds when the evidence changes, and help the whole team win.

  • Continuous learning. The stack changes quickly; we are willing to learn unfamiliar systems, methods, and domains as the work demands.

A note on qualifications. We care more about exceptional evidence than a perfect keyword match. If the work excites you and you can show unusual strength, learning speed, or ownership, we encourage you to apply even if your background does not match every preferred qualification.

Equal opportunity

We are an equal opportunity employer. We consider qualified applicants without regard to race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, disability, veteran status, genetic information, or any other characteristic protected by applicable law. We provide reasonable accommodations for candidates who need them during the hiring process.

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are up 12% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 556, the most 1,080, and the latest week had 556.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Palo Alto, California, United States

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.