Back to jobs

Software Engineer, ML Infrastructure

Pay
Pay not listedSimilar roles state $166K–$237K
Work setting
On-site — work setting source
Work setup in the employer’s posting: IN_OFFICE
From the employer’s posting
Employment
Full-time — employment source
Employment type in the employer’s posting: FULL_TIME
From the employer’s posting

What you’ll work on

Full posting

We are hiring a Founding ML Infrastructure Engineer to own the end-to-end deployment, optimization, and operation of our suits of models in production.

This is a core founding role focused on building and operating production-grade LLM systems.

  • Own the end-to-end LLM inference stack, including:- Model loading and execution

  • Design and operate high-performance LLM serving systems using technologies such as:- vLLM, TensorRT / TensorRT-LLM, Triton Inference Server, SGLang

  • Work hands-on with PyTorch and TensorFlow models, including:- Model graph understanding

From the employer’s posting
We are hiring a Founding ML Infrastructure Engineer to own the end-to-end deployment, optimization, and operation of our suits of models in production.
This is a core founding role focused on building and operating production-grade LLM systems. You will apply deep knowledge of model internals to deploy, optimize, and run modern LLMs at scale, owning performance end-to-end across latency, throughput, and reliability.
Key Responsibilities Own the end-to-end LLM inference stack, including:- Model loading and execution GPU utilization and memory efficiency
Production deployment and scaling Design and operate high-performance LLM serving systems using technologies such as:- vLLM, TensorRT / TensorRT-LLM, Triton Inference Server, SGLang Optimize inference across:- Latency
Cost efficiency Work hands-on with PyTorch and TensorFlow models, including:- Model graph understanding Attention mechanisms, KV cache behavior, batching strategies

Before you apply

Sponsorship
Visa sponsorship not confirmed — sponsorship source
Visa sponsorship: We do sponsor visas! However, we aren't able to successfully sponsor visas for every role and candidate. But if we make you an offer, we will make all reasonable effort to get you a visa, and we retain an immigration lawyer to help with this.
Read the full posting

What you’ll bring

All qualifications

Preferred experience

  • Experience deploying LLMs on Kubernetes and GPU clusters.
  • Knowledge of observability for ML systems (metrics, latency breakdowns, GPU monitoring).
  • Familiarity with CUDA, NCCL, or low-level GPU performance concepts.
  • Experience working at startups or owning systems with minimal abstraction layers.
Qualification wording
Experience deploying LLMs on Kubernetes and GPU clusters.
Knowledge of observability for ML systems (metrics, latency breakdowns, GPU monitoring).
Familiarity with CUDA, NCCL, or low-level GPU performance concepts.
Experience working at startups or owning systems with minimal abstraction layers.

Benefits in the posting

Full benefits wording
  • Market aligned compensation and benefits
  • Medical, Dental, Vision, Life insurance, 401-K, In-office lunch etc.

From the employer’s posting.

About realm-labs

We help enterprises detect, debug, and prevent AI’s misbehaviors in production.

In the employer’s words · Read in context

Apply at realm-labs

Posting record

Report an error

Actively hiringFirst seen by us today, still listed, and not a repost of an earlier posting.

Status in our records
Live
First seen by us
Nov 18, 2026today
Recorded sightings
2
Last seen by us
Nov 19, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Pay

$166K–$237K is typical

The middle half of what similar US postings state. This one doesn’t list pay.

From 1,507 postings in the last year. Stated ranges, not salaries paid.

Time since we first saw it

Day 0

Early. Similar jobs usually come down between day 7 and day 75.

From 5,846 similar postings that came down. Coming down is not always a hire.

realm-labs is hiring

3 open jobs

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Kubernetes
  • PyTorch
  • TensorFlow
Where the posting names them
Preferred Qualifications - Experience deploying LLMs on Kubernetes and GPU clusters. - Familiarity with CUDA, NCCL, or low-level GPU performance concepts.
- Cost efficiency - Work hands-on with PyTorch and TensorFlow models, including:- Model graph understanding - Attention mechanisms, KV cache behavior, batching strategies
- Strong software engineering fundamentals; ability to write robust, maintainable production code. - Deep hands-on experience with LLM inference infrastructure, including:- PyTorch (required) - TensorFlow (working knowledge)
- Deep hands-on experience with LLM inference infrastructure, including:- PyTorch (required) - TensorFlow (working knowledge) - Proven experience with GPU inference optimization, including:- TensorRT / TensorRT-LLM

Your next step

Complete your application on jobs.gem.com. The employer’s form will show what is required.

Already applied? Track this application

Job description

View original posting ↗

Role Overview


We are hiring a Founding ML Infrastructure Engineer to own the end-to-end deployment, optimization, and operation of our suits of models in production.
This is a core founding role focused on building and operating production-grade LLM systems. You will apply deep knowledge of model internals to deploy, optimize, and run modern LLMs at scale, owning performance end-to-end across latency, throughput, and reliability.

You will design and operate the full ML serving stack from model artifacts to GPU execution, and work closely with Product and ML teams to ensure our models can support high QPS, strict SLAs, and production correctness.
This role is ideal for someone who deeply understands how LLMs work internally, but chooses to specialize in making them fast, stable, and production-ready.


About Realm Labs


Realm Labs is an AI trust and security startup. We help enterprises detect, debug, and prevent AI’s misbehaviors in production. We are backed by top VCs and serve some of the most iconic global enterprises.

Key Responsibilities


  • Own the end-to-end LLM inference stack, including:
    • Model loading and execution
    • GPU utilization and memory efficiency
    • Runtime performance tuning
    • Production deployment and scaling
  • Design and operate high-performance LLM serving systems using technologies such as:
    • vLLM, TensorRT / TensorRT-LLM, Triton Inference Server, SGLang
  • Optimize inference across:
    • Latency
    • Throughput (QPS)
    • GPU memory footprint
    • Cost efficiency
  • Work hands-on with PyTorch and TensorFlow models, including:
    • Model graph understanding
    • Attention mechanisms, KV cache behavior, batching strategies
    • Precision tradeoffs (FP16, BF16, INT8, etc.)
  • Build and maintain production-grade GPU services:
    • Multi-model serving
    • Autoscaling strategies
    • Fault isolation and graceful degradation
  • Collaborate with application and platform teams to:
    • Define serving APIs
    • Ensure correctness and safety of outputs
    • Debug production issues end-to-end
  • Build a reproducible model training and versioning system for customer deployments
  • Establish best practices for:
    • Model versioning
    • Rollouts and rollbacks
    • Performance benchmarking
    • Production validation


Expected Qualifications


  • 5+ years of professional experience in ML infrastructure, systems engineering, or production ML roles.
  • Strong software engineering fundamentals; ability to write robust, maintainable production code.
  • Deep hands-on experience with LLM inference infrastructure, including:
    • PyTorch (required)
    • TensorFlow (working knowledge)
  • Proven experience with GPU inference optimization, including:
    • TensorRT / TensorRT-LLM
    • vLLM
    • Triton Inference Server
    • SGLang or similar serving runtimes
  • Strong understanding of LLM internals, such as:
    • Transformer architectures
    • Attention and KV caching
    • Batching, streaming, and token-level generation
  • Experience running ML systems in production with high traffic and SLAs
  • Comfortable working in Linux-based, cloud production environments


Preferred Qualifications


  • Experience deploying LLMs on Kubernetes and GPU clusters.
  • Familiarity with CUDA, NCCL, or low-level GPU performance concepts.
  • Experience with:
    • Model sharding and parallelism strategies
    • Multi-GPU inference
    • Streaming inference systems
  • Knowledge of observability for ML systems (metrics, latency breakdowns, GPU monitoring).
  • Experience working at startups or owning systems with minimal abstraction layers.


Additional Information


  • This is a founding, high-ownership role with direct impact on core product capabilities.
  • You will be expected to build, run, and own systems end-to-end.
  • The role may include limited on-call responsibilities aligned with production ownership.

Compensation & Benefits


  • Market aligned compensation and benefits
  • Founding engineer equity (Equity is a significant component of this role and will be discussed)
  • Medical, Dental, Vision, Life insurance, 401-K, In-office lunch etc.

Visa sponsorship: We do sponsor visas! However, we aren't able to successfully sponsor visas for every role and candidate. But if we make you an offer, we will make all reasonable effort to get you a visa, and we retain an immigration lawyer to help with this.

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are up 12% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 556, the most 1,080, and the latest week had 556.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Sunnyvale, CA

Working pattern and location restrictions need checking in the full posting.

Work authorization
- Medical, Dental, Vision, Life insurance, 401-K, In-office lunch etc. Visa sponsorship: We do sponsor visas! However, we aren't able to successfully sponsor visas for every role and candidate. But if we make you an offer, we will make all reasonable effort to get you a visa, and we retain an immigration lawyer to help with this.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.