Senior ML Operations Engineer II - Ads Platform

About the Role

Role Responsibilities

• Own the complete model lifecycle from experimentation through validation, deployment, versioning, retraining, rollback, and retirement.
• Establish clear model promotion criteria, release processes, and rollback paths that enable teams to ship improvements confidently.
• Design and build scalable ML data and feature engineering pipelines, including feature generation, historical aggregations, and training dataset construction, using Spark/PySpark and modern lakehouse technologies such as S3 and Apache Iceberg.
• Develop reliable, incremental data processing pipelines with support for data quality validation, lineage, point-in-time correctness, and training-serving feature consistency.
• Build monitoring for data quality, drift, prediction quality, service reliability, latency, and infrastructure cost.
• Partner with data scientists to define meaningful offline and production evaluation signals, including for use cases with delayed outcome labels.
• Use production evidence to improve model performance, platform reliability, deployment speed, and infrastructure efficiency.
• Build reusable training, evaluation, and deployment pipelines that support multiple models and modeling approaches.
• Develop dependable orchestration workflows covering dependencies, scheduling, retries, backfills, and failure recovery.
• Translate scientific requirements into maintainable production systems with clear interfaces, ownership, and operational expectations.
• Build and maintain an accessible model hub connecting model documentation, evaluation results, release history, versioning, and monitoring.
• Create tested CI/CD workflows, reproducible environments, actionable alerts, incident-response procedures, and operational runbooks.
• Define MLOps priorities in partnership with Data Science and Engineering, identifying risks before they become production issues.
• Establish reusable engineering standards, mentor teammates, and reduce the operational burden placed on data scientists.

Minimum Requirements

• 5+ years of experience building and operating production ML systems beyond initial deployment, including monitoring, retraining, incident response, and continuous improvement.
• Strong production software development skills in Python and proficiency with SQL.
• Hands-on experience with Spark/PySpark for large-scale distributed data processing, including feature engineering, historical aggregations, incremental processing, and training dataset construction.
• Experience with modern data lake or lakehouse architectures, including S3, Apache Iceberg, or equivalent technologies.
• Experience with a workflow orchestration engine for scheduling, dependency management, retries, backfills, and recovery.
• Hands-on experience building infrastructure with Terraform.
• Experience deploying and operating containerized production workloads on Kubernetes.
• Experience building reusable training, evaluation, or deployment infrastructure used by multiple models, scientists, or teams.
• Experience implementing model and dataset versioning, experiment tracking, lineage, and reproducible releases.
• Ability to distinguish service or infrastructure failures from changes in model effectiveness.
• Experience building CI/CD pipelines, monitoring, alerting, rollback mechanisms, and recovery procedures for production ML systems.
• Demonstrated ability to balance model quality, delivery speed, reliability, and infrastructure cost.
• Strong cross-functional communication and experience partnering with data scientists to preserve scientific validity as models move into production.
• Ability to explain a production incident or model degradation issue, how it was detected, and what improvements followed.

Preferred Requirements

• Experience developing platform services, infrastructure, or internal tooling in Go.
• Experience building or owning a model registry, model hub, or similar system used by technical and nontechnical stakeholders.
• Experience supporting ML systems where outcome labels are delayed or model quality cannot be measured immediately.
• Track record of improving deployment speed, model quality, system reliability, infrastructure cost, or manual operational effort.
• Experience designing configurable ML infrastructure that supports several modeling frameworks or approaches.
• Evidence that reusable infrastructure, standards, or documentation were adopted across multiple teams.
• Experience mentoring engineers or data scientists and influencing technical priorities across an ML portfolio.
• Experience working in environments with sensitive information and designing documentation or monitoring around appropriate access boundaries.