Back to jobs

Senior Principal Architect-Data

Pay
Pay not listed
Work setting
Not stated
Employment
Not stated

What you’ll work on

Full posting
  • Architect multi-agent AI systems on Databricks, including orchestration and tool-use patterns

  • Own the Databricks Lakehouse architecture, including catalog design and medallion layers

  • Define Unity Catalog standards for AI assets such as model registries and vector indexes

  • Write architecture decision records and reference architectures for engineering teams

  • Mentor senior data and ML engineers on architecture and AI engineering practices

From the employer’s posting
Architect multi-agent AI systems on Databricks, defining agent orchestration frameworks, tool-use patterns, memory/state management, and inter-agent communication protocols.
Own the Databricks Lakehouse architecture end-to-end: catalog/schema design, medallion architecture (raw → refined → curated → AI-ready), storage optimization, and multi-workspace governance.
Define and enforce Unity Catalog standards for AI assets—model registries, feature tables, vector search indexes, and function-as-a-tool registrations—enabling governed AI consumption across the enterprise.
Own architectural decision records (ADRs), reference architectures, and platform blueprints that engineering teams build against independently.
Mentor senior data engineers and ML engineers, elevating architectural thinking, system design rigor, and AI engineering best practices across the organization.

What you’ll bring

All qualifications

Core experience

  • 10+ years of experience in data engineering, data platform architecture, or AI/ML engineering.
  • 4+ years of hands-on, production-grade Databricks experience: Unity Catalog, Delta Lake, Model Serving, Workflows, Databricks SQL, cluster lifecycle management, and workspace administration.
  • 5+ years designing and delivering AI/ML systems at scale—covering model training, feature engineering, model serving, and monitoring.
  • Demonstrated experience architecting agentic AI systems: multi-agent orchestration, tool-use patterns, LLM integration, RAG pipelines, and autonomous workflow design.
  • Deep knowledge of data modeling, medallion architecture, Delta Lake internals, and data lakehouse design patterns.
  • Experience leading architecture at the principal or distinguished level—setting standards, mentoring engineers, and influencing technical direction organization-wide.

Preferred experience

  • Bachelor's degree in Computer Science, Data Engineering, Artificial Intelligence, or equivalent; advanced degree preferred.
  • Experience with agentic orchestration frameworks:
  • Hands-on experience with Mosaic AI, Databricks Apps, AI/BI Genie, and DBRX or similar open-weight LLMs.
  • Experience with MLOps and LLMOps practices: CI/CD for ML, model monitoring, prompt versioning, evaluation pipelines.
Qualification wording
10+ years of experience in data engineering, data platform architecture, or AI/ML engineering.
4+ years of hands-on, production-grade Databricks experience: Unity Catalog, Delta Lake, Model Serving, Workflows, Databricks SQL, cluster lifecycle management, and workspace administration.
5+ years designing and delivering AI/ML systems at scale—covering model training, feature engineering, model serving, and monitoring.
Demonstrated experience architecting agentic AI systems: multi-agent orchestration, tool-use patterns, LLM integration, RAG pipelines, and autonomous workflow design.
Deep knowledge of data modeling, medallion architecture, Delta Lake internals, and data lakehouse design patterns.
Experience leading architecture at the principal or distinguished level—setting standards, mentoring engineers, and influencing technical direction organization-wide.
Bachelor's degree in Computer Science, Data Engineering, Artificial Intelligence, or equivalent; advanced degree preferred.
Experience with agentic orchestration frameworks:
Hands-on experience with Mosaic AI, Databricks Apps, AI/BI Genie, and DBRX or similar open-weight LLMs.
Experience with MLOps and LLMOps practices: CI/CD for ML, model monitoring, prompt versioning, evaluation pipelines.
Apply at Lilly

Posting record

Report an error

Open 3 monthsWe first saw this posting 102 days ago and it is still up. Roles open this long are often slow to fill or kept open to collect applications.

Status in our records
Live
First seen by us
Aug 11, 2026102 days ago
Recorded sightings
59
Last seen by us
Nov 21, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Time since we first saw it

Day 102

Up longer than most. Similar jobs usually come down between day 8 and day 57.

Well past Lilly’s usual: its jobs come down by about day 27.

From 9,995 similar postings that came down, and 101 of Lilly’s. Coming down is not always a hire.

Lilly is hiring

40 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • Data Leadership hiring is steady
  • Takes jobs down at a usual pace
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • SQL
  • Databricks
  • Delta Lake
  • MLflow
  • Terraform
  • PySpark
  • Great Expectations
Where the posting names them
- Demonstrated experience architecting agentic AI systems: multi-agent orchestration, tool-use patterns, LLM integration, RAG pipelines, and autonomous workflow design. - Expert-level SQL and PySpark; strong Python for AI/ML pipeline development. - Deep knowledge of data modeling, medallion architecture, Delta Lake internals, and data lakehouse design patterns.
- 10+ years of experience in data engineering, data platform architecture, or AI/ML engineering. - 4+ years of hands-on, production-grade Databricks experience: Unity Catalog, Delta Lake, Model Serving, Workflows, Databricks SQL, cluster lifecycle management, and workspace administration. - 5+ years designing and delivering AI/ML systems at scale—covering model training, feature engineering, model serving, and monitoring.
Principal Architect-R4 Databricks AI Design & Architecture | Agentic AI Systems As a Principal Architect for Databricks AI & Agentic Systems, you will own the end-to-end technical strategy and architecture of Lilly's AI data platform. You will design the systems that enable autonomous, multi-agent AI workflows—spanning data ingestion, semantic modeling, model orchestration, and real-time decision intelligence—while setting the architectural standards the engineering organization builds against.
Databricks AI Design & Architecture | Agentic AI Systems As a Principal Architect for Databricks AI & Agentic Systems, you will own the end-to-end technical strategy and architecture of Lilly's AI data platform. You will design the systems that enable autonomous, multi-agent AI workflows—spanning data ingestion, semantic modeling, model orchestration, and real-time decision intelligence—while setting the architectural standards the engineering organization builds against. This is a senior individual contributor and technical leadership role. You will be the definitive technical authority on Databricks architecture, agentic AI design patterns, and AI/ML platform engineering—directly influencing how Lilly uses AI to transform pharmaceutical R&D, commercial operations, and patient care.
As a Principal Architect for Databricks AI & Agentic Systems, you will own the end-to-end technical strategy and architecture of Lilly's AI data platform. You will design the systems that enable autonomous, multi-agent AI workflows—spanning data ingestion, semantic modeling, model orchestration, and real-time decision intelligence—while setting the architectural standards the engineering organization builds against. This is a senior individual contributor and technical leadership role. You will be the definitive technical authority on Databricks architecture, agentic AI design patterns, and AI/ML platform engineering—directly influencing how Lilly uses AI to transform pharmaceutical R&D, commercial operations, and patient care. KEY RESPONSIBILITIES
AI Architecture & Agentic System Design - Architect multi-agent AI systems on Databricks, defining agent orchestration frameworks, tool-use patterns, memory/state management, and inter-agent communication protocols. - Design agentic workflows that combine LLMs, retrieval-augmented generation (RAG), and structured data reasoning to support autonomous decision-making pipelines.
- Design agentic workflows that combine LLMs, retrieval-augmented generation (RAG), and structured data reasoning to support autonomous decision-making pipelines. - Establish architecture patterns for agent-to-agent orchestration using frameworks such as LangGraph, AutoGen, CrewAI, or equivalent, integrated within the Databricks ecosystem. - Define how AI agents interact with Unity Catalog assets—tables, models, volumes, and functions—as tool surfaces for autonomous data retrieval and action execution.
- Define how AI agents interact with Unity Catalog assets—tables, models, volumes, and functions—as tool surfaces for autonomous data retrieval and action execution. - Architect real-time and near-real-time agentic pipelines leveraging Databricks Structured Streaming, Delta Live Tables, and event-driven triggers. Databricks Platform Architecture
- Architect real-time and near-real-time agentic pipelines leveraging Databricks Structured Streaming, Delta Live Tables, and event-driven triggers. Databricks Platform Architecture - Own the Databricks Lakehouse architecture end-to-end: catalog/schema design, medallion architecture (raw → refined → curated → AI-ready), storage optimization, and multi-workspace governance.
Databricks Platform Architecture - Own the Databricks Lakehouse architecture end-to-end: catalog/schema design, medallion architecture (raw → refined → curated → AI-ready), storage optimization, and multi-workspace governance. - Define and enforce Unity Catalog standards for AI assets—model registries, feature tables, vector search indexes, and function-as-a-tool registrations—enabling governed AI consumption across the enterprise.
- Set Delta Lake standards: table properties, Z-ordering, partitioning strategies, liquid clustering, and cost-based optimization for petabyte-scale workloads. - Design Databricks Model Serving architectures, including real-time inference endpoints, batch inference pipelines, and A/B model deployment patterns. AI/ML Platform & LLM Integration
AI/ML Platform & LLM Integration - Define the enterprise architecture for LLM integration on Databricks—covering prompt management, context assembly, vector stores (Databricks Vector Search), embedding pipelines, and LLM fine-tuning workflows. - Architect MLflow-based model lifecycle management: experiment tracking, model versioning, champion/challenger patterns, and governance-compliant model promotion.
- Architect MLflow-based model lifecycle management: experiment tracking, model versioning, champion/challenger patterns, and governance-compliant model promotion. - Design feature engineering and feature serving architectures using Databricks Feature Store, ensuring AI agents have low-latency access to curated, consistent features. - Evaluate and adopt emerging Databricks AI capabilities—Mosaic AI, AI/BI Genie, DBRX, and Databricks Apps—as first-class architectural components.
- Design feature engineering and feature serving architectures using Databricks Feature Store, ensuring AI agents have low-latency access to curated, consistent features. - Evaluate and adopt emerging Databricks AI capabilities—Mosaic AI, AI/BI Genie, DBRX, and Databricks Apps—as first-class architectural components. - Define patterns for responsible AI: model monitoring, drift detection, explainability, audit trails, and bias mitigation within regulated pharmaceutical contexts.
Governance, Security & Data Quality - Define and enforce data governance, access control, lineage, and auditing across all Databricks workspaces using Unity Catalog and integration with enterprise identity providers. - Architect data quality frameworks (Great Expectations, Databricks DQ rules, or equivalent) embedded into AI pipelines to ensure model inputs meet reliability standards.
- Define and enforce data governance, access control, lineage, and auditing across all Databricks workspaces using Unity Catalog and integration with enterprise identity providers. - Architect data quality frameworks (Great Expectations, Databricks DQ rules, or equivalent) embedded into AI pipelines to ensure model inputs meet reliability standards. - Define security architecture for AI workloads: network isolation, secret management, encryption-at-rest/in-transit, IP restriction, and compliance with GxP and GDPR requirements.
- Mentor senior data engineers and ML engineers, elevating architectural thinking, system design rigor, and AI engineering best practices across the organization. - Influence and guide vendor partners (Databricks, cloud providers, ISVs) on solution direction, ensuring alignment with Lilly's technical and data governance strategy. - Represent Tech@Lilly in external architecture forums, Databricks advisory councils, and industry working groups.
- Influence and guide vendor partners (Databricks, cloud providers, ISVs) on solution direction, ensuring alignment with Lilly's technical and data governance strategy. - Represent Tech@Lilly in external architecture forums, Databricks advisory councils, and industry working groups. HOW YOU WILL SUCCEED
HOW YOU WILL SUCCEED - A production-grade agentic AI platform on Databricks that powers autonomous workflows across commercial, R&D, and patient services domains. - Architecture standards clear and comprehensive enough that distributed engineering teams execute with confidence within well-defined boundaries.
- Sustained adoption of AI best practices—evidenced by engineers citing and following the architectural standards you define. - Recognized as the go-to technical authority by both engineering and senior business stakeholders on Databricks and AI architecture decisions. BASIC QUALIFICATIONS
PREFERRED QUALIFICATIONS - Databricks Certified Data Engineer (Professional) and/or Databricks Certified Machine Learning Professional. - Experience with agentic orchestration frameworks:
- Experience with agentic orchestration frameworks: - Hands-on experience with Mosaic AI, Databricks Apps, AI/BI Genie, and DBRX or similar open-weight LLMs. - Domain expertise in pharmaceutical, healthcare, or regulated industries (GxP, GDPR, HIPAA)—including Customer Master, Product Master, HCP/HCO data, and commercial alignment data.
- Experience with MLOps and LLMOps practices: CI/CD for ML, model monitoring, prompt versioning, evaluation pipelines. - Familiarity with infrastructure-as-code (Terraform, Databricks Asset Bundles) and GitOps for platform configuration. - Experience with vector databases (Databricks Vector Search, Pinecone, Weaviate) and embedding model management.
- Familiarity with infrastructure-as-code (Terraform, Databricks Asset Bundles) and GitOps for platform configuration. - Experience with vector databases (Databricks Vector Search, Pinecone, Weaviate) and embedding model management. - Understanding of responsible AI principles, model risk management, and AI governance frameworks in enterprise settings.
- Architect the semantic and metrics layer so business logic is expressed once, consistently consumed by BI tools, AI agents, and downstream APIs. - Set Delta Lake standards: table properties, Z-ordering, partitioning strategies, liquid clustering, and cost-based optimization for petabyte-scale workloads. - Design Databricks Model Serving architectures, including real-time inference endpoints, batch inference pipelines, and A/B model deployment patterns.
- Define security architecture for AI workloads: network isolation, secret management, encryption-at-rest/in-transit, IP restriction, and compliance with GxP and GDPR requirements. - Own data sharing architecture—Delta Sharing, external connectors, API gateway patterns—ensuring downstream AI consumers receive governed, versioned data surfaces. Technical Leadership & Standards
- Expert-level SQL and PySpark; strong Python for AI/ML pipeline development. - Deep knowledge of data modeling, medallion architecture, Delta Lake internals, and data lakehouse design patterns. - Proven track record of translating complex business requirements into scalable, governed, production-ready architectures.
- Define the enterprise architecture for LLM integration on Databricks—covering prompt management, context assembly, vector stores (Databricks Vector Search), embedding pipelines, and LLM fine-tuning workflows. - Architect MLflow-based model lifecycle management: experiment tracking, model versioning, champion/challenger patterns, and governance-compliant model promotion. - Design feature engineering and feature serving architectures using Databricks Feature Store, ensuring AI agents have low-latency access to curated, consistent features.

Your next step

Complete your application on lilly.wd115.myworkdayjobs.com. The employer’s form will show what is required.

Applying through Phenom

  • Phenom runs the employer’s careers site, but the application itself usually runs on another system, often Workday or Taleo. The advice for that system applies once you click Apply.

Already applied? Track this application

Job description

View original posting ↗

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us. 


Principal Architect-R4

Databricks AI Design & Architecture | Agentic AI Systems

As a Principal Architect for Databricks AI & Agentic Systems, you will own the end-to-end technical strategy and architecture of Lilly's AI data platform. You will design the systems that enable autonomous, multi-agent AI workflows—spanning data ingestion, semantic modeling, model orchestration, and real-time decision intelligence—while setting the architectural standards the engineering organization builds against.

This is a senior individual contributor and technical leadership role. You will be the definitive technical authority on Databricks architecture, agentic AI design patterns, and AI/ML platform engineering—directly influencing how Lilly uses AI to transform pharmaceutical R&D, commercial operations, and patient care.

KEY RESPONSIBILITIES

AI Architecture & Agentic System Design

  • Architect multi-agent AI systems on Databricks, defining agent orchestration frameworks, tool-use patterns, memory/state management, and inter-agent communication protocols.
  • Design agentic workflows that combine LLMs, retrieval-augmented generation (RAG), and structured data reasoning to support autonomous decision-making pipelines.
  • Establish architecture patterns for agent-to-agent orchestration using frameworks such as LangGraph, AutoGen, CrewAI, or equivalent, integrated within the Databricks ecosystem.
  • Define how AI agents interact with Unity Catalog assets—tables, models, volumes, and functions—as tool surfaces for autonomous data retrieval and action execution.
  • Architect real-time and near-real-time agentic pipelines leveraging Databricks Structured Streaming, Delta Live Tables, and event-driven triggers.

Databricks Platform Architecture

  • Own the Databricks Lakehouse architecture end-to-end: catalog/schema design, medallion architecture (raw → refined → curated → AI-ready), storage optimization, and multi-workspace governance.
  • Define and enforce Unity Catalog standards for AI assets—model registries, feature tables, vector search indexes, and function-as-a-tool registrations—enabling governed AI consumption across the enterprise.
  • Architect the semantic and metrics layer so business logic is expressed once, consistently consumed by BI tools, AI agents, and downstream APIs.
  • Set Delta Lake standards: table properties, Z-ordering, partitioning strategies, liquid clustering, and cost-based optimization for petabyte-scale workloads.
  • Design Databricks Model Serving architectures, including real-time inference endpoints, batch inference pipelines, and A/B model deployment patterns.

AI/ML Platform & LLM Integration

  • Define the enterprise architecture for LLM integration on Databricks—covering prompt management, context assembly, vector stores (Databricks Vector Search), embedding pipelines, and LLM fine-tuning workflows.
  • Architect MLflow-based model lifecycle management: experiment tracking, model versioning, champion/challenger patterns, and governance-compliant model promotion.
  • Design feature engineering and feature serving architectures using Databricks Feature Store, ensuring AI agents have low-latency access to curated, consistent features.
  • Evaluate and adopt emerging Databricks AI capabilities—Mosaic AI, AI/BI Genie, DBRX, and Databricks Apps—as first-class architectural components.
  • Define patterns for responsible AI: model monitoring, drift detection, explainability, audit trails, and bias mitigation within regulated pharmaceutical contexts.

Governance, Security & Data Quality

  • Define and enforce data governance, access control, lineage, and auditing across all Databricks workspaces using Unity Catalog and integration with enterprise identity providers.
  • Architect data quality frameworks (Great Expectations, Databricks DQ rules, or equivalent) embedded into AI pipelines to ensure model inputs meet reliability standards.
  • Define security architecture for AI workloads: network isolation, secret management, encryption-at-rest/in-transit, IP restriction, and compliance with GxP and GDPR requirements.
  • Own data sharing architecture—Delta Sharing, external connectors, API gateway patterns—ensuring downstream AI consumers receive governed, versioned data surfaces.

Technical Leadership & Standards

  • Own architectural decision records (ADRs), reference architectures, and platform blueprints that engineering teams build against independently.
  • Lead architecture reviews, design critiques, and proof-of-concept evaluations for platform investments and AI product initiatives.
  • Mentor senior data engineers and ML engineers, elevating architectural thinking, system design rigor, and AI engineering best practices across the organization.
  • Influence and guide vendor partners (Databricks, cloud providers, ISVs) on solution direction, ensuring alignment with Lilly's technical and data governance strategy.
  • Represent Tech@Lilly in external architecture forums, Databricks advisory councils, and industry working groups.

HOW YOU WILL SUCCEED

  • A production-grade agentic AI platform on Databricks that powers autonomous workflows across commercial, R&D, and patient services domains.
  • Architecture standards clear and comprehensive enough that distributed engineering teams execute with confidence within well-defined boundaries.
  • Measurable platform outcomes: query performance benchmarks met, AI agent latency SLOs achieved, data quality scores at or above targets.
  • Governance posture that passes internal audit and supports regulatory submissions in validated GxP environments.
  • Sustained adoption of AI best practices—evidenced by engineers citing and following the architectural standards you define.
  • Recognized as the go-to technical authority by both engineering and senior business stakeholders on Databricks and AI architecture decisions.

BASIC QUALIFICATIONS

  • Bachelor's degree in Computer Science, Data Engineering, Artificial Intelligence, or equivalent; advanced degree preferred.
  • 10+ years of experience in data engineering, data platform architecture, or AI/ML engineering.
  • 4+ years of hands-on, production-grade Databricks experience: Unity Catalog, Delta Lake, Model Serving, Workflows, Databricks SQL, cluster lifecycle management, and workspace administration.
  • 5+ years designing and delivering AI/ML systems at scale—covering model training, feature engineering, model serving, and monitoring.
  • Demonstrated experience architecting agentic AI systems: multi-agent orchestration, tool-use patterns, LLM integration, RAG pipelines, and autonomous workflow design.
  • Expert-level SQL and PySpark; strong Python for AI/ML pipeline development.
  • Deep knowledge of data modeling, medallion architecture, Delta Lake internals, and data lakehouse design patterns.
  • Proven track record of translating complex business requirements into scalable, governed, production-ready architectures.
  • Experience leading architecture at the principal or distinguished level—setting standards, mentoring engineers, and influencing technical direction organization-wide.

PREFERRED QUALIFICATIONS

  • Databricks Certified Data Engineer (Professional) and/or Databricks Certified Machine Learning Professional.
  • Experience with agentic orchestration frameworks:
  • Hands-on experience with Mosaic AI, Databricks Apps, AI/BI Genie, and DBRX or similar open-weight LLMs.
  • Domain expertise in pharmaceutical, healthcare, or regulated industries (GxP, GDPR, HIPAA)—including Customer Master, Product Master, HCP/HCO data, and commercial alignment data.
  • Experience with MLOps and LLMOps practices: CI/CD for ML, model monitoring, prompt versioning, evaluation pipelines.
  • Familiarity with infrastructure-as-code (Terraform, Databricks Asset Bundles) and GitOps for platform configuration.
  • Experience with vector databases (Databricks Vector Search, Pinecone, Weaviate) and embedding model management.
  • Understanding of responsible AI principles, model risk management, and AI governance frameworks in enterprise settings.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly

About Lilly

People
About 50,000 in 2025, up 6% from the year before.Annual report (SEC Form 10-K)
Company
Listed as LLY.

Demand for Data Leadership roles

New Data Leadership postings at the employers we follow are up 8% over the last 6 weeks compared with the 6 before.

Weekly new Data Leadership postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 751, the most 1,165, and the latest week had 775.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under Data Leadership, by the work the posting describes.

Data leaders run data teams or design how a company's data fits together. Managers hire, set priorities and answer for whether the data work pays off; architects choose the systems and the shape of the data.

Expect years of hands-on data work, experience leading people or projects, and the ability to explain tradeoffs to executives.

Unlike a senior engineer or analyst, most of a data leader's week goes to people, planning and decisions rather than building.

US government figures for computer and information systems managers

Median yearly pay, May 2025
$175,140
Expected change, 2025 to 2035
15.8% more jobsagainst 3.5% for all jobs

The government has no code for data leaders, so these figures are for the closest group, computer and information systems managers, which includes data processing and data operations managers.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 11-3021.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay

No pay amount identified in the saved description.

Location & working pattern

Hyderabad, Telangāna, India

Working pattern and location restrictions need checking in the full posting.

Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.