Back to jobs

Senior ML Engineer - Inference and Model Serving

Pay
Pay not listed
Work setting
Not stated
Employment
Not stated

What you’ll work on

Full posting

We are hiring a senior ML engineer to own how our language and speech models run in production.

agents depend on fast, reliable inference: every spoken reply passes through speech recognition, a

  • Serve open-weight language models with vLLM, TensorRT-LLM or SGLang, tuned for latency and throughput

  • Serve streaming speech recognition and speech synthesis models with low time to first audio

  • Quantise models and choose precision based on measured quality and speed

  • Check every optimisation against quality evals and block changes that hurt quality

  • Build CI/CD pipelines that deploy models from registries into Kubernetes

From the employer’s posting
We are hiring a senior ML engineer to own how our language and speech models run in production. Our
agents depend on fast, reliable inference: every spoken reply passes through speech recognition, a
Serve open-weight language models with engines such as vLLM, TensorRT-LLM or SGLang, tuned for both time to first token and throughput.
Serve streaming speech recognition and speech synthesis models on GPUs, with low time to first partial transcript and time to first audio.
Quantise models (INT8, INT4, FP8) and choose precision per model based on measured quality and speed.
block any change that makes a model faster but worse.
Build CI/CD pipelines that profile, regression-test and deploy models from registries such as MLflow or Hugging Face Hub into Kubernetes.

What you’ll bring

All qualifications

Preferred experience

  • Deep familiarity with recent NVIDIA architectures such as Ampere, Hopper and Blackwell.
Qualification wording
Deep familiarity with recent NVIDIA architectures such as Ampere, Hopper and Blackwell.
Education & alternatives
• Serving vision or multimodal models. • A masters degree or PhD in a relevant field. Success in the role
Apply at YAL

Posting record

Report an error

Actively hiringFirst seen by us 3 days ago, still listed, and not a repost of an earlier posting.

Status in our records
Live
First seen by us
Nov 18, 20263 days ago
Recorded sightings
4
Last seen by us
Nov 21, 2026
Employer says posted
Oct 26, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

More ML Engineer jobs in Hyderabad

All ML Engineer jobs in Hyderabad →

How this job compares

Time since we first saw it

Day 3

Early. Similar jobs usually come down between day 7 and day 76.

From 6,399 similar postings that came down. Coming down is not always a hire.

YAL is hiring

4 open jobs

Open jobs by the week we first saw them, last 12 weeks.

Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • C++
  • Python
  • Kafka
  • Kubernetes
  • MLflow
  • Hugging Face
  • PyTorch
Where the posting names them
Required qualifications and skills • Strong Python and working C++ skills. • Hands-on experience serving language models in production with engines such as vLLM,
budgets. • API gateways and rate limiting (Envoy, Kong), and streaming systems (Kafka, Redpanda). • Air-gapped or on-premises GPU deployments for enterprise clients.
• Build CI/CD pipelines that profile, regression-test and deploy models from registries such as MLflow or Hugging Face Hub into Kubernetes. • Monitor latency, throughput, GPU utilisation and errors with OpenTelemetry and Prometheus,
PyTorch Profiler. • Experience running GPU workloads on Kubernetes, with CI/CD, monitoring and autoscaling. • Ability to measure quality alongside speed, and to explain trade-offs between latency,
then find and fix bottlenecks. • Use compilers and runtimes such as TensorRT, TVM or PyTorch Inductor to fuse operations and reduce memory traffic.
• Experience profiling and fixing performance problems with tools such as Nsight Systems or PyTorch Profiler. • Experience running GPU workloads on Kubernetes, with CI/CD, monitoring and autoscaling.

Your next step

Complete your application on echelonera.keka.com. The employer’s form will show what is required.

Already applied? Track this application

Job description

View original posting ↗

Role overview

We are hiring a senior ML engineer to own how our language and speech models run in production. Our
agents depend on fast, reliable inference: every spoken reply passes through speech recognition, a
language model and speech synthesis within a tight latency budget. You will serve these models on
GPUs at the lowest cost per call without losing quality, build the security and metering layers around
them, and automate how models move from registry to production. You will work with the architect on
model-serving interfaces and with the AI engineers who build agents on top.

Key responsibilities

Language model serving
• Serve open-weight language models with engines such as vLLM, TensorRT-LLM or SGLang, tuned
for both time to first token and throughput.
• Apply continuous batching, prefix caching, speculative decoding and multi-LoRA serving to cut
latency and cost per request.
• Manage GPU memory with PagedAttention, FlashAttention, KV-cache quantisation and tensor
or pipeline parallelism, to maximise batch size and context length without out-of-memory
failures.
• Quantise models (INT8, INT4, FP8) and choose precision per model based on measured quality
and speed.
Speech model serving
• Serve streaming speech recognition and speech synthesis models on GPUs, with low time to first
partial transcript and time to first audio.
• Batch and schedule many concurrent audio streams on shared GPUs without letting one stream
delay another.
• Work with AI engineers to optimise end-to-end latency of the speech pipeline, from audio
chunking to model execution to streaming output.
• Place speech and language models on GPUs to balance latency, utilisation and cost when they
run side by side.

Performance and quality

• Profile models and serving stacks with tools such as NVIDIA Nsight Systems and Nsight Compute,
then find and fix bottlenecks.
• Use compilers and runtimes such as TensorRT, TVM or PyTorch Inductor to fuse operations and
reduce memory traffic.
• Check every optimisation against quality evals — task accuracy, word error rate and voice
quality — and block any change that makes a model faster but worse.
• Benchmark candidate models and hardware on latency, throughput, quality and cost, and
recommend what to serve.

Scale, cost and deployment

• Autoscale GPU capacity with traffic, plan capacity for peaks, and track cost per request, per call
and per tenant.
• Deploy to cloud, on-premises and restricted-network environments, including air-gapped installs
for enterprise clients.
• Build fallback paths across model sizes, replicas and providers, so traffic keeps flowing when a
model or node fails.

Access, metering and MLOps

• Design secure model endpoints with OAuth2, JWT and role-based access control, and enforce
strict multi-tenant isolation.
• Build high-throughput metering for input and output tokens, audio seconds and GPU time,
feeding rate limits, billing and cost allocation.
• Build CI/CD pipelines that profile, regression-test and deploy models from registries such as
MLflow or Hugging Face Hub into Kubernetes.
• Monitor latency, throughput, GPU utilisation and errors with OpenTelemetry and Prometheus,
and respond to production incidents.

Required qualifications and skills

• Strong Python and working C++ skills.
• Hands-on experience serving language models in production with engines such as vLLM,
TensorRT-LLM, SGLang or Triton Inference Server.
• Experience serving or optimising speech or other streaming models under strict real-time
latency budgets.
• Solid understanding of GPU architecture — streaming multiprocessors, memory hierarchy,
asynchronous copies — and how it drives inference performance.
• Practical experience with quantisation, batching strategies, KV-cache management and model
parallelism.
• Experience profiling and fixing performance problems with tools such as Nsight Systems or
PyTorch Profiler.
• Experience running GPU workloads on Kubernetes, with CI/CD, monitoring and autoscaling.
• Ability to measure quality alongside speed, and to explain trade-offs between latency,
throughput, cost and accuracy.
• A degree in computer science, computer or electrical engineering, or a related field, or
equivalent practical experience, with clear examples of delivered work.

Preferred experience

• Writing or modifying custom GPU kernels in CUDA or Triton, including kernel fusion.
• Deep familiarity with recent NVIDIA architectures such as Ampere, Hopper and Blackwell.
• Serving large models (70B+ parameters) across multiple GPUs or nodes under strict latency
budgets.
• API gateways and rate limiting (Envoy, Kong), and streaming systems (Kafka, Redpanda).
• Air-gapped or on-premises GPU deployments for enterprise clients.
• Serving vision or multimodal models.
• A masters degree or PhD in a relevant field.

Success in the role

Success means our language and speech models respond fast enough for natural conversation, at a cost
per call that scales with the business. In your first six months, that looks like:
• Latency: speech and language models meet their per-hop budgets at p95 under peak traffic.
• Cost: cost per call and GPU utilisation are tracked, and both improve measurably.
• Quality: no optimisation ships without passing quality evals, and quality never drops unnoticed.
• Reliability: model endpoints stay available through node and model failures, with fallbacks that
work.
• Deployment: new models reach production through an automated, tested pipeline, in the cloud
or on-premises.
• Metering: every tenants usage is measured accurately for rate limits and billing.

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are down 4% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 508, the most 899, and the latest week had 508.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Hyderabad, TG, India

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.