Back to jobs

Senior Principal Machine Learning Engineer (Advanced Threat Protection)

Pay
$190,400–308,000/year — pay source
The compensation offered for this position will depend on qualifications, experience, and work location. For candidates who receive an offer at the posted level, the starting base salary (for non-sales roles) or base salary + commission target (for sales/com-missioned roles) is expected to be the annual range listed below. The offered compensation may also include restricted stock units and a bonus. A description of our employee benefits may be found here. $190,400.00 - $308,000.00/yr Our Commitment
Read the full posting
Work setting
Not stated
Employment
Not stated

What you’ll work on

Full posting
  • Define the multi-year engineering roadmap and technical direction for AI adoption

  • Design distributed cloud systems that process security data streams at petabyte scale

  • Architect production agentic frameworks with memory, secure tool use and guardrails

  • Diagnose and fix underperforming models via fine-tuning such as SFT, DPO, RL, LoRA

  • Troubleshoot distributed GPU training and inference bottlenecks

From the employer’s posting
define the multi-year engineering roadmap, set the technical direction for foundation model adoption
Lead the macro- and micro-level architecture of highly distributed, fault-tolerant cloud systems that process real-time security data streams at petabyte scale
Architect robust, production-grade agentic frameworks featuring dynamic multi-step planning, stateful memory persistence, secure tool execution, and deterministic guardrails
lead targeted remediation via fine-tuning (SFT, DPO, RL, LoRA/PEFT)
Troubleshoot low-level compute bottlenecks on distributed GPU clusters

Before you apply

Work authorization
Sponsors visas
Employer’s visa record
1,046 H-1B petitions approved in fiscal 2024 and 2025For the employer as a whole. US Citizenship and Immigration Services.

What you’ll bring

All qualifications

Core experience

  • Experience & Scale: 10+ years of progressive software engineering and systems design experience, with a documented history of technical leadership delivering high-availability, enterprise-scale platforms.
Qualification wording
Experience & Scale: 10+ years of progressive software engineering and systems design experience, with a documented history of technical leadership delivering high-availability, enterprise-scale platforms.
Education & alternatives
- Strong foundation in data engineering with Apache Spark, distributed processing, SQL, BigQuery, and GCP orchestration (Dataproc, GKE). - Education: BS, MS, or Ph.D. in Computer Science, Data Science, Artificial Intelligence, or a related technical discipline. Preferred Qualifications (Strong Bonus)

About Palo Alto Networks

At Palo Alto Networks®, we’re united by a shared mission—to protect our digital way of life.

In the employer’s words · Read in context

Apply at Palo Alto Networks

Posting record

Report an error

Actively hiringFirst seen by us 2 days ago, still listed, and not a repost of an earlier posting.

Status in our records
Live
First seen by us
Nov 18, 20262 days ago
Recorded sightings
3
Last seen by us
Nov 20, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Pay

Higher than most similar jobs

$190K–$308K against $166K–$237K, the middle half of what similar US postings state.

From 1,507 postings in the last year. Stated ranges, not salaries paid.

Time since we first saw it

Day 2

Early. Similar jobs usually come down between day 7 and day 76.

At Palo Alto Networks, jobs usually come down by about day 36.

From 6,269 similar postings that came down, and 92 of Palo Alto Networks’s. Coming down is not always a hire.

Palo Alto Networks is hiring

64 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • Hiring at a steady pace
  • Takes jobs down at a usual pace
  • 6% of open jobs were reposted
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Java
  • Python
  • SQL
  • BigQuery
  • GCP
  • Spark
  • Go
Where the posting names them
- Languages & Data Engineering:- Advanced proficiency in Python for AI/ML pipelines and scientific computing. - Production-level fluency in Go or Java for low-latency, concurrent backend microservices. - Strong foundation in data engineering with Apache Spark, distributed processing, SQL, BigQuery, and GCP orchestration (Dataproc, GKE).
- Hardware & Systems Debugging: Practical proficiency profiling and debugging distributed workloads on GPUs, diagnosing memory leaks, concurrency locks, and cluster communication bottlenecks. - Languages & Data Engineering:- Advanced proficiency in Python for AI/ML pipelines and scientific computing. - Production-level fluency in Go or Java for low-latency, concurrent backend microservices.
- Agentic Ecosystems: Architect robust, production-grade agentic frameworks featuring dynamic multi-step planning, stateful memory persistence, secure tool execution, and deterministic guardrails. - Data & Telemetry Backbone: Drive the architectural blueprint for ingestion, preprocessing, and feature engineering pipelines using Apache Spark, SQL, and BigQuery across Google Cloud platforms (Dataproc, GKE, and GCP SDKs). Frontier Modeling, Diagnosis & Remediation
- Production-level fluency in Go or Java for low-latency, concurrent backend microservices. - Strong foundation in data engineering with Apache Spark, distributed processing, SQL, BigQuery, and GCP orchestration (Dataproc, GKE). - Education: BS, MS, or Ph.D. in Computer Science, Data Science, Artificial Intelligence, or a related technical discipline.

Your next step

Complete your application on jobs.paloaltonetworks.com. The employer’s form will show what is required.

Applying through Radancy (TalentBrew)

  • Radancy (TalentBrew) runs the employer’s careers site, but the application itself usually runs on another system, often Workday or Taleo. The advice for that system applies once you click Apply.

Already applied? Track this application

Job description

View original posting ↗

Our Mission

At Palo Alto Networks®, we’re united by a shared mission—to protect our digital way of life. We thrive at the intersection of innovation and impact, solving real-world problems with cutting-edge technology and bold thinking. Here, everyone has a voice, and every idea counts. If you’re ready to do the most meaningful work of your career alongside people who are just as passionate as you are, you’re in the right place.

Who We Are

In order to be the cybersecurity partner of choice, we must trailblaze the path and shape the future of our industry. This is something our employees work at each day and is defined by our values: Disruption, Collaboration, Execution, Integrity, and Inclusion. We weave AI into the fabric of everything we do and use it to augment the impact every individual can have. If you are passionate about solving real-world problems and ideating beside the best and the brightest, we invite you to join us!

We believe collaboration thrives in person. That’s why most of our teams work from the office full time, with flexibility when it’s needed. This model supports real-time problem-solving, stronger relationships, and the kind of precision that drives great outcomes.

Job Summary

Your Career

As a Senior Principal Machine Learning Engineer, you will serve as the chief technical authority driving architectural strategy, design excellence, and AI innovation across cloud security systems. In this role, you will define the multi-year engineering roadmap, set the technical direction for foundation model adoption, and lead high-stakes architectural design across complex, distributed environments.

You will act as an organizational force multiplier—partnering across product, security research, infrastructure, and engineering teams to transform cutting-edge AI breakthroughs and high-throughput telemetry into resilient, production-grade Security-as-a-Service (SECaaS) offerings.


Your Impact


Strategic Leadership & Team Direction

  • Set Technical Direction: Serve as the primary technical compass for the organization. Evaluate frontline AI research, filter transient hype from production-viable technology, and steer engineering investments toward high-leverage architectural paths.
  • Cultivate Engineering Culture: Champion engineering rigor by instituting RFC standards, leading architectural review boards (ARBs), and mentoring Staff and Principal engineers to scale technical excellence across teams.
  • Cross-Functional Partnership: Bridge the gap between Product Management, Security Researchers, Data Science, and Infrastructure teams. Translate ambiguous business objectives and complex threat landscapes into concrete technical requirements, milestone architectures, and executable designs.

Large-Scale System Design & Architecture

  • End-to-End System Design: Lead the macro- and micro-level architecture of highly distributed, fault-tolerant cloud systems that process real-time security data streams at petabyte scale.
  • Agentic Ecosystems: Architect robust, production-grade agentic frameworks featuring dynamic multi-step planning, stateful memory persistence, secure tool execution, and deterministic guardrails.
  • Data & Telemetry Backbone: Drive the architectural blueprint for ingestion, preprocessing, and feature engineering pipelines using Apache Spark, SQL, and BigQuery across Google Cloud platforms (Dataproc, GKE, and GCP SDKs).

Frontier Modeling, Diagnosis & Remediation

  • Frontier Model Craftsmanship: Maintain deep, hands-on mastery of the evolving model landscape. Guide architectural choices across Large Language Models (LLMs), Small Language Models (SLMs), and multimodal systems to optimize for cost, latency, privacy, and domain-specific efficacy.
  • Root-Cause Diagnosis & Quality Recovery: Establish systematic frameworks to diagnose underperforming models in production (e.g., drift, context dilution, alignment collapse, hallucination spikes). Pinpoint failures across training data distribution, loss convergence, or prompt/context limits, and lead targeted remediation via fine-tuning (SFT, DPO, RL, LoRA/PEFT).
  • GPU & Accelerator Debugging: Troubleshoot low-level compute bottlenecks on distributed GPU clusters, resolving distributed training/inference issues, CUDA out-of-memory (OOM) states, NCCL communication overhead, and kernel performance stalls.

Qualifications

Your Experience

  • Experience & Scale: 10+ years of progressive software engineering and systems design experience, with a documented history of technical leadership delivering high-availability, enterprise-scale platforms.
  • Architectural Mastery: Proven ability to author complex system designs from scratch, establish data contracts, navigate intricate architectural trade-offs, and steer large-scale legacy-to-modern system migrations.
  • Cross-Functional Influence: Demonstrated track record of aligning cross-functional teams (Product, Data, Applied Research, SecOps, Infrastructure) around unified technical visions and resolving competing design priorities.
  • Frontier AI/ML Depth: Deep hands-on knowledge of modern AI architectures (Transformers, attention mechanisms, tokenization strategies) and practical operational experience deploying and serving both LLMs and SLMs.
  • Model Triage & Fine-Tuning: Expert-level capability in analyzing why models degrade and executing targeted recovery pipelines (data curation, synthetic data generation, supervised fine-tuning, preference alignment, and parameter-efficient tuning).
  • Hardware & Systems Debugging: Practical proficiency profiling and debugging distributed workloads on GPUs, diagnosing memory leaks, concurrency locks, and cluster communication bottlenecks.
  • Languages & Data Engineering:
    • Advanced proficiency in Python for AI/ML pipelines and scientific computing.
    • Production-level fluency in Go or Java for low-latency, concurrent backend microservices.
    • Strong foundation in data engineering with Apache Spark, distributed processing, SQL, BigQuery, and GCP orchestration (Dataproc, GKE).
  • Education: BS, MS, or Ph.D. in Computer Science, Data Science, Artificial Intelligence, or a related technical discipline.

Preferred Qualifications (Strong Bonus)

  • Cybersecurity Domain Expertise: Deep background in threat modeling, endpoint detection and response (EDR), SIEM architectures, or cloud security posture management.
  • Network Topologies: Working knowledge of L3–L7 network protocols, packet telemetry, and deep packet inspection systems.
  • Industry Thought Leadership: Contributions to open-source foundation model tooling, published research, or active participation in the broader AI systems community.

#LI-TD1

Compensation Disclosure

The compensation offered for this position will depend on qualifications, experience, and work location. For candidates who receive an offer at the posted level, the starting base salary (for non-sales roles) or base salary + commission target (for sales/com-missioned roles) is expected to be the annual range listed below. The offered compensation may also include restricted stock units and a bonus. A description of our employee benefits may be found here.

$190,400.00 - $308,000.00/yr

Our Commitment

We’re trailblazers that dream big, take risks, and challenge cybersecurity’s status quo. It’s simple: we can’t accomplish our mission without diverse teams innovating, together.

We are committed to providing reasonable accommodations for all qualified individuals with a disability. If you require assistance or accommodation due to a disability or special need, please contact us at  accommodations@paloaltonetworks.com.

Palo Alto Networks is an equal opportunity employer. We celebrate diversity in our workplace, and all qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected characteristics.

All your information will be kept confidential according to EEO guidelines.

Is role eligible for Immigration Sponsorship?: Yes

 

 

About Palo Alto Networks

People
About 8,804 in 2024.Benefit plan filings (Form 5500)
Company
Listed as PANW.
H-1B visas
261 approved for new hires in fiscal 2025.US Citizenship and Immigration Services

Demand for ML Engineer roles

New ML Engineer postings at the employers we follow are up 12% over the last 6 weeks compared with the 6 before.

Weekly new ML Engineer postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 556, the most 1,080, and the latest week had 556.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under ML Engineer, by the work the posting describes.

ML engineers put machine learning models into products. They write the code that trains, ships, serves and monitors models so a recommendation or prediction works for real users.

Postings ask for strong Python and software engineering, a framework such as PyTorch, cloud services, and more and more often experience with large language models.

A data scientist often works out which model to build; an ML engineer makes it run reliably in production.

US government figures for computer and information research scientists

Median yearly pay, May 2025
$140,300
Expected change, 2025 to 2035
21.8% more jobsagainst 3.5% for all jobs

The government has no code for ML engineers, so these figures are for the closest group, computer and information research scientists.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 15-1221.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay
The compensation offered for this position will depend on qualifications, experience, and work location. For candidates who receive an offer at the posted level, the starting base salary (for non-sales roles) or base salary + commission target (for sales/com-missioned roles) is expected to be the annual range listed below. The offered compensation may also include restricted stock units and a bonus. A description of our employee benefits may be found here. $190,400.00 - $308,000.00/yr Our Commitment
Location & working pattern

Santa Clara, CA

Working pattern and location restrictions need checking in the full posting.

Work authorization
All your information will be kept confidential according to EEO guidelines. Is role eligible for Immigration Sponsorship?: Yes

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.