Senior Staff Applied ML Engineer - Consumer Agents

Senior Staff Applied Machine Learning Engineer, Consumer Agents

Location: Remote — North America

About the role

The Consumer Agent team builds Shopify's consumer-facing agentic shopping experiences — the AI agents inside the Shop app and on merchant storefronts that help millions of buyers search, discover, and buy. As a Senior Staff Applied Machine Learning Engineer, you'll be the person we point at the hardest problems on the surface — distillation to own-GPU inference, agentic memory, whole-page optimization, propensity modeling, or the evaluation rigor that ties them together — and trust to drive it to a shipped, measured outcome. This is a senior individual-contributor seat with deep autonomy: you'll own ambiguous, high-leverage problems end to end, raise the bar for evaluation across the team, and ship weekly on a fast-moving consumer surface. You'll work at the intersection of LLM orchestration, model training and distillation, search and retrieval, and cost/latency engineering, partnering with frontier model labs and ML teams across Shopify.

Key Responsibilities

• Build and improve student models through supervised fine-tuning and knowledge distillation, including the serving, generation-termination, and throughput-economics work that own-GPU inference demands.

• Design and raise the bar for ML evaluation across the team — judge design, offline and online metrics, and the judgment to know when a number is real.

• Develop agentic memory and personalization signals — buyer and user representations, profile generation — that feed predictive layers.

• Work on whole-page optimization: page composition, ranking, and blending for the agent results surface.

• Build propensity and predictive models for intent and purchase propensity across agent surfaces.

• Move fluidly between distillation, memory, ranking, propensity, and evaluation as priorities shift, without lengthy re-tooling.

• Apply rigorous experimentation on a high-traffic surface — proper holdouts, A/B rigor, and counterfactual or off-policy evaluation where online tests aren't enough.

• Integrate cleanly with an LLM orchestration stack already in flight — tool calling, structured output, and retrieval-augmented patterns.

• Become the technical reference point the team routes its hardest model and evaluation calls to.

• Contribute to cross-team ML architecture conversations across Shopify's ML organization and external model-lab partnerships.

• Ship weekly, absorbing ambiguity rather than adding coordination overhead.

Qualifications

• You've shipped LLM systems to production at real scale — systems with real users, not research prototypes.

• Hands-on depth in model training and distillation: supervised fine-tuning and knowledge distillation in practice.

• Comfort with the serving and throughput-economics side of inference, not just the training run.

• Rigorous evaluation methodology: judge design, offline and online metrics, and the judgment to distinguish a real result from an artifact.

• You operate as a senior individual contributor with high autonomy — you take an ambiguous, hard problem and drive it quickly to a measured outcome without hand-holding.

• Breadth over a single specialty: you can move across distillation, memory, whole-page optimization, propensity, and evaluation.

• Strong communication and collaboration in a fast-paced, cross-functional environment.

• Able to work with significant overlap with North America time zones.

Nice to have

• E-commerce, marketplace, commerce-search, recsys, or ads background — with instincts for selection bias, position bias, and the gap between offline and online metrics.

• Inference and serving internals: vLLM or similar engines, structured and constrained decoding, KV-cache and GPU throughput economics, and latency budgets.

• Reinforcement learning experience — RLHF, RLAIF, or preference optimization.

• Propensity, ranking, or classical applied ML at scale: gradient boosting, neural ranking, calibration, position-bias correction.

• Counterfactual evaluation experience — inverse propensity scoring, doubly-robust estimators, off-policy evaluation.

• Experience working alongside or with frontier AI products and agentic systems.