Pay
$166K–$237K is typical
The middle half of what similar US postings state. This one doesn’t list pay.
From 1,505 postings in the last year. Stated ranges, not salaries paid.
Boston, Massachusetts, United States
Live, first seen by us 71 days agoWork setup in the employer’s posting: On-siteFrom the employer’s posting
Employment type in the employer’s posting: FullTimeFrom the employer’s posting
Serving frontier LLMs efficiently at scale is a moving target: every new model and workload reshuffles the tradeoffs across parallelism, batching, KV caching, and scheduling.
This role will be performed onsite from one of our offices in Santa Clara, CA or Boston, MA (relocation assistance available).
You’ll work from first principles, move quickly from insight to execution, and see your work show up in what we ship, not just in a roadmap.
Optimize performance: Tune parallelism strategies (TP/EP/DP/PP), batching, KV caching, routing, and scheduling to meet latency, throughput, and SLA targets.
Support diverse workloads: Enable tool calling, agentic and coding workflows, RAG, and long-context applications.
Serving frontier LLMs efficiently at scale is a moving target: every new model and workload reshuffles the tradeoffs across parallelism, batching, KV caching, and scheduling. You’ll own Netpreme’s inference-serving stack end to end, from tuning vLLM, SGLang, and TensorRT-LLM to architecting distributed serving with NVIDIA Dynamo (prefill/decode disaggregation, KV-cache-aware routing). Working directly with our hardware and systems engineers, you’ll build the benchmarks and drive the optimizations that take new models from research to production.
This role will be performed onsite from one of our offices in Santa Clara, CA or Boston, MA (relocation assistance available).
Impact: AI is advancing faster than the infrastructure underneath it. We’re working at that layer, unlocking more capability from every GPU while dramatically improving efficiency. The work you do here compounds across state-of-the-art models, systems, and real-world applications. Timing: Joining now means real ownership, including equity, and a direct say in product direction. You’ll work from first principles, move quickly from insight to execution, and see your work show up in what we ship, not just in a roadmap. Culture: We care about rigor, clarity, and impact. Disagreement is welcome when it’s thoughtful, the best idea wins regardless of who it comes from, and you’ll learn something new most weeks.
Own the serving infrastructure: Build and maintain distributed inference systems with NVIDIA Dynamo, Kubernetes, Helm, and Ansible, including upgrades, monitoring, troubleshooting, and reliability. Optimize performance: Tune parallelism strategies (TP/EP/DP/PP), batching, KV caching, routing, and scheduling to meet latency, throughput, and SLA targets. Benchmark and measure: Build reproducible benchmarks for latency (TTFT, TPOT, E2E), throughput, concurrency scaling, and system efficiency.
Benchmark and measure: Build reproducible benchmarks for latency (TTFT, TPOT, E2E), throughput, concurrency scaling, and system efficiency. Support diverse workloads: Enable tool calling, agentic and coding workflows, RAG, and long-context applications. Bring new tech into production: Evaluate new models, frameworks, and optimizations, and partner with hardware, model, and systems engineers to ship improvements.
2+ years of engineering experience in LLM inference, ML systems, GPU systems, or related infrastructure.
Experience with Helm, Ansible, or similar deployment and configuration tools.
Hands-on experience deploying and optimizing at least one LLM inference engine (e.g., vLLM, SGLang, or TensorRT-LLM).
Experience with distributed inference systems like NVIDIA Dynamo, including parallelism strategies (TP/EP/DP/PP), prefill/decode disaggregation, and KV-cache-aware routing.
Solid understanding of LLM inference fundamentals: prefill/decode, batching, KV caching, and latency/throughput tradeoffs.
Experience with advanced inference optimizations such as prefix caching, speculative decoding, and quantization.
Qualifications - BS, MS, or PhD in Computer Science, Computer Engineering, or a related field, or equivalent practical experience. - 2+ years of engineering experience in LLM inference, ML systems, GPU systems, or related infrastructure.
From the employer’s posting.
The biggest bottleneck in AI systems today is memory.
In the employer’s words · Read in context
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
$166K–$237K is typical
The middle half of what similar US postings state. This one doesn’t list pay.
From 1,505 postings in the last year. Stated ranges, not salaries paid.
Day 71
Similar jobs usually come down between day 7 and day 76.
From 5,830 similar postings that came down. Coming down is not always a hire.
1 open job
Open jobs by the week we first saw them, last 12 weeks.
Hiring recordAdd your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
- Solid understanding of LLM inference fundamentals: prefill/decode, batching, KV caching, and latency/throughput tradeoffs. - Strong Python and Linux skills, plus hands-on experience with Kubernetes and containerized deployments. - Comfortable benchmarking and troubleshooting systems, and communicating clearly in a small engineering team.
- Deploy and operate models in production: Run large language and multimodal models using inference engines like vLLM, SGLang, and TensorRT-LLM. - Own the serving infrastructure: Build and maintain distributed inference systems with NVIDIA Dynamo, Kubernetes, Helm, and Ansible, including upgrades, monitoring, troubleshooting, and reliability. - Optimize performance: Tune parallelism strategies (TP/EP/DP/PP), batching, KV caching, routing, and scheduling to meet latency, throughput, and SLA targets.
Complete your application on jobs.ashbyhq.com. The employer’s form will show what is required.
Already applied? Track this application
The biggest bottleneck in AI systems today is memory. Chips compute faster than they can be fed data—a problem known as the memory wall—and Netpreme is building the solution that allows AI memory to scale independently of AI compute. Our approach spans the full stack, from a proprietary memory management framework at the XPU kernel and OS levels to X-Mem™, the world’s first Memory Processing Unit (MPU) silicon. Founded by MIT researchers at the intersection of optical networking, ML systems, and GPU kernel optimization, we’re based in Cambridge, MA, and Santa Clara, CA. We’re looking for people who want to own hard problems end to end—and help shape the next generation of AI systems.
Click here to read our latest blog!
Serving frontier LLMs efficiently at scale is a moving target: every new model and workload reshuffles the tradeoffs across parallelism, batching, KV caching, and scheduling. You’ll own Netpreme’s inference-serving stack end to end, from tuning vLLM, SGLang, and TensorRT-LLM to architecting distributed serving with NVIDIA Dynamo (prefill/decode disaggregation, KV-cache-aware routing). Working directly with our hardware and systems engineers, you’ll build the benchmarks and drive the optimizations that take new models from research to production.
This role will be performed onsite from one of our offices in Santa Clara, CA or Boston, MA (relocation assistance available).
Impact: AI is advancing faster than the infrastructure underneath it. We’re working at that layer, unlocking more capability from every GPU while dramatically improving efficiency. The work you do here compounds across state-of-the-art models, systems, and real-world applications.
Timing: Joining now means real ownership, including equity, and a direct say in product direction. You’ll work from first principles, move quickly from insight to execution, and see your work show up in what we ship, not just in a roadmap.
Culture: We care about rigor, clarity, and impact. Disagreement is welcome when it’s thoughtful, the best idea wins regardless of who it comes from, and you’ll learn something new most weeks.
Deploy and operate models in production: Run large language and multimodal models using inference engines like vLLM, SGLang, and TensorRT-LLM.
Own the serving infrastructure: Build and maintain distributed inference systems with NVIDIA Dynamo, Kubernetes, Helm, and Ansible, including upgrades, monitoring, troubleshooting, and reliability.
Optimize performance: Tune parallelism strategies (TP/EP/DP/PP), batching, KV caching, routing, and scheduling to meet latency, throughput, and SLA targets.
Benchmark and measure: Build reproducible benchmarks for latency (TTFT, TPOT, E2E), throughput, concurrency scaling, and system efficiency.
Support diverse workloads: Enable tool calling, agentic and coding workflows, RAG, and long-context applications.
Bring new tech into production: Evaluate new models, frameworks, and optimizations, and partner with hardware, model, and systems engineers to ship improvements.
BS, MS, or PhD in Computer Science, Computer Engineering, or a related field, or equivalent practical experience.
2+ years of engineering experience in LLM inference, ML systems, GPU systems, or related infrastructure.
Hands-on experience deploying and optimizing at least one LLM inference engine (e.g., vLLM, SGLang, or TensorRT-LLM).
Solid understanding of LLM inference fundamentals: prefill/decode, batching, KV caching, and latency/throughput tradeoffs.
Strong Python and Linux skills, plus hands-on experience with Kubernetes and containerized deployments.
Comfortable benchmarking and troubleshooting systems, and communicating clearly in a small engineering team.
Experience with Helm, Ansible, or similar deployment and configuration tools.
Experience with distributed inference systems like NVIDIA Dynamo, including parallelism strategies (TP/EP/DP/PP), prefill/decode disaggregation, and KV-cache-aware routing.
Experience with advanced inference optimizations such as prefix caching, speculative decoding, and quantization.
Experience analyzing LLM architectures and tuning deployments across different hardware platforms to meet SLAs.
Contributions to vLLM, SGLang, TensorRT-LLM, or related open-source projects.
Experience owning production infrastructure, ideally at an early-stage startup.
Competitive salary with performance-based bonus and early-stage equity grant
100% employer-paid Health, Dental, and Vision coverage for you and your dependents
401(k) match with immediate vesting, and access to financial advisors to help you reach your financial goals
100% employer-paid Life, Disability, and AD&D insurance, plus a fitness stipend and wellness & mental health perks
Generous PTO: 20 vacation days, 15 company holidays (including 3 floating days of your choosing)
Daily lunch stipend
Enterprise-level Claude & ChatGPT access with a generous token budget
Well-equipped, sunny offices in Santa Clara, CA & Cambridge, MA with on-site parking and EV charging; on-site fitness center in Santa Clara; gym discounts near our Cambridge office
Visa sponsorship and relocation assistance to one of our office hubs
A collaborative, continuous-learning environment with smart, dedicated colleagues building the next generation of high-performance computing architecture
New ML Engineer postings at the employers we follow are up 12% over the last 6 weeks compared with the 6 before.
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
We file this job under ML Engineer, by the work the posting describes.
ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.
Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.
A data scientist often works out which model to build; an ML engineer makes it run reliably in production.
The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.
Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.
Selected passages from the saved posting. Check the full description for conditions and exceptions.
No pay amount identified in the saved description.
Boston, Massachusetts, United States
- Enterprise-level Claude & ChatGPT access with a generous token budget - Well-equipped, sunny offices in Santa Clara, CA & Cambridge, MA with on-site parking and EV charging; on-site fitness center in Santa Clara; gym discounts near our Cambridge office - Visa sponsorship and relocation assistance to one of our office hubs
- Well-equipped, sunny offices in Santa Clara, CA & Cambridge, MA with on-site parking and EV charging; on-site fitness center in Santa Clara; gym discounts near our Cambridge office - Visa sponsorship and relocation assistance to one of our office hubs - A collaborative, continuous-learning environment with smart, dedicated colleagues building the next generation of high-performance computing architecture