Time since we first saw it
Day 45
Up longer than most. Similar jobs usually come down between day 1 and day 15.
From 1,376 similar postings that came down. Coming down is not always a hire.
Boston Hub · Remote · San Francisco Hub · Washington DC Office · Boston, Massachusetts, United States
Live, first seen by us 45 days agoLocation type: HybridFrom the employer’s posting
Employment type: Full-timeFrom the employer’s posting
Department: EngineeringFrom the employer’s posting
Own the design docs and RFCs, help with build-vs-buy decisions, and align with other technical stakeholders.
Own the model gateway that teams use to reach self-hosted and commercial models, with consistent auth, routing, failover, quotas, and cost attribution.
Design reusable agent harnesses and orchestration primitives that product teams can compose into reliable, verifiable workflows.
Core Responsibilities Set the technical direction and architecture for Code Metal's AI platform and the team building it. Own the design docs and RFCs, help with build-vs-buy decisions, and align with other technical stakeholders. Deploy, benchmark, and tune elastic, production inference for open-weight models on runtimes such as vLLM, SGLang, and TensorRT-LLM.
Deploy, benchmark, and tune elastic, production inference for open-weight models on runtimes such as vLLM, SGLang, and TensorRT-LLM. Own the model gateway that teams use to reach self-hosted and commercial models, with consistent auth, routing, failover, quotas, and cost attribution. Design reusable agent harnesses and orchestration primitives that product teams can compose into reliable, verifiable workflows.
Own the model gateway that teams use to reach self-hosted and commercial models, with consistent auth, routing, failover, quotas, and cost attribution. Design reusable agent harnesses and orchestration primitives that product teams can compose into reliable, verifiable workflows. Build context-engineering services for memory, retrieval, and data discovery, so agents get the information specific to their tasks.
Experience with context engineering: retrieval-augmented generation, embeddings, vector / hybrid search, and memory for agents, ideally over code or large technical corpora.
Experience instrumenting services (for example, with OpenTelemetry tracing and metrics) and operating AI services against SLOs.
Hands-on experience deploying and tuning LLM inference engines such as vLLM, SGLang, or TensorRT-LLM on GPU infrastructure, with measurable gains in throughput, latency, or cost per token.
Familiarity with inference optimization: speculative decoding, prefix caching, tensor/pipeline/expert parallelism, disaggregated prefill and decode, GPU profiling.
Experience building evaluation harnesses for LLMs and agents, and experiment-tracking or artifact management systems such as MLflow or Weights & Biases.
Experience taking an internal platform to an external product: multi-tenancy, SDKs, versioned APIs, and documentation.
From the employer’s posting.
Code Metal is the leader in automated software engineering you can trust.
In the employer’s words · Read in context · Company website ↗
These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.
Day 45
Up longer than most. Similar jobs usually come down between day 1 and day 15.
From 1,376 similar postings that came down. Coming down is not always a hire.
5 open jobs
Open jobs by the week we first saw them, last 12 weeks.
Hiring recordAdd your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.
Required Qualifications - Production-grade Python and strong platform engineering fundamentals: API and service design, distributed systems, containers and Kubernetes, CI/CD, and testing. - Shipped production agentic systems, with a clear sense of where they break and how to make them reliable.
- Familiarity with inference optimization: speculative decoding, prefix caching, tensor/pipeline/expert parallelism, disaggregated prefill and decode, GPU profiling. - Experience building evaluation harnesses for LLMs and agents, and experiment-tracking or artifact management systems such as MLflow or Weights & Biases. - Experience taking an internal platform to an external product: multi-tenancy, SDKs, versioned APIs, and documentation.
- Experience instrumenting services (for example, with OpenTelemetry tracing and metrics) and operating AI services against SLOs. - Solid data science and AI research fundamentals: how transformers and LLM inference work, experiment design, benchmarking, and model evaluation. Working familiarity with PyTorch and Hugging Face, and experience fine-tuning, evaluating, or serving language models. - Staff-level technical leadership: owned architecture across multiple systems or teams, written design docs and RFCs, turned ambiguous needs from several internal customers into a roadmap, and mentored engineers.
Complete your application on jobs.ashbyhq.com. The employer’s form will show what is required.
Already applied? Track this application
Code Metal is the leader in automated software engineering you can trust. As AI writes more of the world's code, the bottleneck in software has shifted from writing code to verifying it works, and AI cannot verify its own work with certainty. Code Metal takes a fundamentally different approach: constrain AI to what it does reliably, verify every step independently of the model using formal methods, and keep engineers in the loop on the decisions that matter. The result isn't code that probably works — it's code that is provably correct, with auditable proof. Customers including the U.S. Air Force, L3Harris, RTX, and Toshiba use Code Metal to modernize legacy code, optimize performance on real hardware, and move prototypes to production, fast. Founded in 2023 with offices in Boston and San Francisco, Code Metal is funded by Accel, Salesforce Ventures, B Capital, Smith Point Capital, J2 Ventures, Shield Capital, Overmatch, RTX, and others.
Learn more at codemetal.ai.
The Role
Code Metal's engineering teams are building AI-driven code transpilation, modernization, optimization, modeling and simulation solutions. They all need the same foundations: models to serve, agents to run, context to manage, and results to measure. Our AI Platform team builds those foundations.
As a Staff AI Platform Engineer, you'll be a technical lead on this team. You'll architect and build the AI enablement stack our engineers depend on, from GPU inference serving and model gateway up through agent harnesses, context engineering, observability, and AI experimentation management. It is an internal platform, but we're building it to product standard.
This is an engineering role first. Most of your time goes to designing, building, and operating production systems. You'll also need solid data science and AI research fundamentals: you'll work closely with our AI Research team, and you'll sometimes run experiments yourself when a platform decision needs evidence.
Core Responsibilities
Set the technical direction and architecture for Code Metal's AI platform and the team building it. Own the design docs and RFCs, help with build-vs-buy decisions, and align with other technical stakeholders.
Deploy, benchmark, and tune elastic, production inference for open-weight models on runtimes such as vLLM, SGLang, and TensorRT-LLM.
Own the model gateway that teams use to reach self-hosted and commercial models, with consistent auth, routing, failover, quotas, and cost attribution.
Design reusable agent harnesses and orchestration primitives that product teams can compose into reliable, verifiable workflows.
Build context-engineering services for memory, retrieval, and data discovery, so agents get the information specific to their tasks.
Instrument the stack end to end, for example, with OpenTelemetry traces and service metrics. Build the experiment-tracking and artifact layer that lets engineers and researchers reproduce and compare results.
Design for productization from day one (versioned APIs, security, deployment in customer environments), and communicate with business and product teams about the roadmap.
Required Qualifications
Production-grade Python and strong platform engineering fundamentals: API and service design, distributed systems, containers and Kubernetes, CI/CD, and testing.
Shipped production agentic systems, with a clear sense of where they break and how to make them reliable.
Experience with context engineering: retrieval-augmented generation, embeddings, vector / hybrid search, and memory for agents, ideally over code or large technical corpora.
Experience instrumenting services (for example, with OpenTelemetry tracing and metrics) and operating AI services against SLOs.
Solid data science and AI research fundamentals: how transformers and LLM inference work, experiment design, benchmarking, and model evaluation. Working familiarity with PyTorch and Hugging Face, and experience fine-tuning, evaluating, or serving language models.
Staff-level technical leadership: owned architecture across multiple systems or teams, written design docs and RFCs, turned ambiguous needs from several internal customers into a roadmap, and mentored engineers.
Preferred Qualifications
Highly desirable: Production experience running an LLM gateway or proxy such as SMG or Bifrost, or equivalent experience building an API gateway, including routing, auth, rate limiting, quotas, failover, and cost attribution.
Hands-on experience deploying and tuning LLM inference engines such as vLLM, SGLang, or TensorRT-LLM on GPU infrastructure, with measurable gains in throughput, latency, or cost per token.
Familiarity with inference optimization: speculative decoding, prefix caching, tensor/pipeline/expert parallelism, disaggregated prefill and decode, GPU profiling.
Experience building evaluation harnesses for LLMs and agents, and experiment-tracking or artifact management systems such as MLflow or Weights & Biases.
Experience taking an internal platform to an external product: multi-tenancy, SDKs, versioned APIs, and documentation.
Experience deploying AI systems on-prem or in regulated domains such as defense or aerospace.
Experience Level
Typically 8+ years of software engineering experience, including 4+ years building and operating ML/AI or LLM systems in production, with demonstrated Staff-level scope and impact.
BenefitsPay depends on experience, but we strive to be at the upper end of the salary range
Health care plan with 100% premium coverage, including medical, dental, and vision
401k with 5% matching
Paid Time Off (uncapped vacation, plus sick and public holidays)
Flexible hybrid or remote work arrangement
Relocation assistance for qualifying employees
Wage Transparency - The salary range for this role is not a guarantee of compensation or salary, as the final offer amount may vary based on factors including, but not limited to, individual proficiency, skills, experience, and location.
We are an equal opportunity employer. US Citizenship may be required for certain project assignments involving security clearance.
Pursuant to the San Francisco Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.
New AI Engineer postings at the employers we follow are up 340% over the last 6 weeks compared with the 6 before.
Each bar is one week of new postings. The darkest bars are the last 6 weeks.
Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.
We file this job under AI Engineer, by the work the posting describes.
AI engineers build products on top of AI models that already exist, most often large language models: they connect a model to a company's data and tools and ship the assistants, agents and search features people use.
Postings ask for strong Python or TypeScript, experience with model APIs, retrieval and agent frameworks, and ways to test whether the model's answers are good enough.
An ML engineer trains and serves models; an AI engineer usually starts from a model someone else trained and builds the product around it, though titles often mix the two.
Selected passages from the saved posting. Check the full description for conditions and exceptions.
No pay amount identified in the saved description.
Boston, Massachusetts, United States
- Shipped production agentic systems, with a clear sense of where they break and how to make them reliable. - Experience with context engineering: retrieval-augmented generation, embeddings, vector / hybrid search, and memory for agents, ideally over code or large technical corpora. - Experience instrumenting services (for example, with OpenTelemetry tracing and metrics) and operating AI services against SLOs.
- Paid Time Off (uncapped vacation, plus sick and public holidays) - Flexible hybrid or remote work arrangement - Relocation assistance for qualifying employees
Wage Transparency - The salary range for this role is not a guarantee of compensation or salary, as the final offer amount may vary based on factors including, but not limited to, individual proficiency, skills, experience, and location. We are an equal opportunity employer. US Citizenship may be required for certain project assignments involving security clearance. Pursuant to the San Francisco Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.