Associate Director, Data Engineering

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us. 

The Clinical & Non-Clinical Data Organization at Eli Lilly and Company is responsible for the design, build, and operation of enterprise data platforms that power drug discovery, clinical development, and regulatory submissions. Data Hub is building a robust Data Strategy to make Lilly's Clinical and Non-Clinical data AI-ready and audit-ready, delivering scalable, governed, and reusable data products that accelerate how medicines reach patients. Our data engineering organization sits at the intersection of science, technology, and patient impact — connecting Clinical and Non-Clinical data across the full chain, from ingestion to consumption.

The Associate Director, Data Engineering is a senior product-ownership, platform-management, and people-leadership role that owns the quarterly roadmap and backlog for one or more data engineering delivery teams, the health and cost of the underlying data platform, and the data strategy and AI roadmap for a Databricks-oriented practice. Hands-on data engineering experience is expected and non-negotiable — this is a practitioner who has built pipelines, models, and platforms directly, and stays technically current enough to review architecture and pressure-test build-vs-buy calls first-hand.

This role defines and implements the data KPI and metrics framework the organization runs on, enables AI adoption across data teams beyond its own delivery pods, and owns the data products built on the Databricks lakehouse. A further mandate is accelerating data onboarding and access, so new sources and consumers move from request to trusted, governed availability in days, not months. As an M1-level people leader, this individual manages a team of data engineers and/or product owners and partners closely with architecture, platform, and senior business leaders on scope and multi-year direction.

Strategic Leadership & Data/AI Roadmap (Databricks-Oriented)

• Proactively surface strategic topics to senior leadership based on first-hand data engineering experience — platform constraints, technical debt, build-vs-buy trade-offs, and emerging AI/engineering tooling — rather than waiting to be asked.
• Define the data strategy and AI roadmap for a Databricks-oriented data engineering team, identifying the data products, lakehouse assets, and AI/ML capabilities that matter most over the next 12–24 months.
• Own the product roadmap for Databricks-native capabilities — lakehouse architecture, Unity Catalog governance, Delta Live Tables pipelines, MLflow/feature-store patterns— with hands-on proof points, not vendor-slide recommendations.

Product Ownership & Roadmap Leadership

• Own sprint- and quarter-level backlog prioritization applying product-management rigor (RICE/WSJF-style scoring, stakeholder discovery, success metrics) to balance business value, technical debt, effort, and risk.
• Bring forward strategic topics — informed by direct engineering experience — that shape the roadmap rather than simply respond to it (e.g., platform limitations, emerging AI capabilities).
• Facilitate PI Planning, backlog grooming, and sprint reviews; serve as the voice of both the internal customer and the platform in delivery ceremonies.

Platform Management

• Own the data platform as a product — define its roadmap, SLAs/OLAs, and reliability targets, treating internal engineering teams and data consumers as its customers.
• Manage platform capacity, cost, and performance (compute/storage sizing, FinOps discipline, cost-per-workload tracking) to keep the platform scalable and cost-efficient.
• Standardize and rationalize the tooling landscape — reduce redundant components and own the platform's build-vs-buy and vendor evaluations.
• Own lifecycle management of platform components — versioning, deprecation, and migration planning — so platform evolution doesn't disrupt delivery teams.

Data KPIs & Metrics

• Define and implement the data KPI and metrics framework for the organization — data quality, pipeline reliability, freshness/latency, lineage coverage, and cost-to-serve — with clear owners and targets.
• Instrument dashboards and reporting (e.g., in Databricks AI/BI or equivalent) so KPIs are visible to engineering teams and business stakeholders, not buried in a backlog tool.
• Tie KPIs to business outcomes and review trends regularly with the team and stakeholders, using them to drive continuous improvement, not static reporting.

AI Adoption Enablement Across Data Teams

• Champion and enable AI adoption across data teams enterprise-wide, not only within owned pods — building the playbooks, training, and support that let other teams adopt AI-assisted engineering with confidence.
• Stand up a community of practice for AI-assisted and agentic engineering across data teams, sharing reusable patterns, guardrails, and lessons learned.
• Track engineering efficiency and AI-adoption metrics (cycle time, automation coverage, tool adoption rate) and prioritize backlog items that improve them.

Data Acceleration & Solution Delivery

• Prioritize data product delivery work that reduces time-to-data and time-to-insight for downstream consumers.
• Own backlog items that advance Data Capability Maturity — data contracts, self-service access, and reusable data products at scale.

Data Onboarding & Access Acceleration

• Redesign and own the data onboarding pipeline so new sources are cataloged, quality-checked, and available as trusted data products in days, not months.
• Streamline data access provisioning — self-service, role/attribute-based requests (e.g., via Unity Catalog), and automated approvals — to cut time-to-access.
• Track onboarding and access-acceleration metrics (time-to-onboard, time-to-access) and prioritize work that measurably shortens them.

People Leadership

• Directly manage a team of data engineers and/or product owners, including hiring, onboarding, and performance appraisals.
• Coach team members on product thinking and platform ownership, and serve as their career advisor, partnering with HR on development and IDPs.

Stakeholder Engagement & Collaboration

• Partner with architecture, test engineering, and business/product leaders to align priorities and dependencies; represent the team in PI Planning.
• Communicate roadmap, platform trade-offs, and priorities clearly to both technical and business audiences.

Process Leadership & Continuous Improvement

• Implement and refine a standardized framework for backlog management, platform governance, and delivery execution (Agile/Kanban).
• Contribute to a growing community of practice among data product owners and platform leads; support Lilly's evolution toward a federated data mesh model.

• Quantitative background in engineering, computer science, data, mathematics, or a related field.
• 10+ years of relevant data/software engineering experience, including 4+ years of people-leadership experience and a demonstrated track record of product ownership.
• Deep, current, hands-on data engineering background — actively able to build, review, and troubleshoot pipelines and platform components, not solely credentialed from past roles.
• Demonstrated platform management experience: reliability/SLA ownership, capacity/cost management, tooling rationalization, and vendor/build-vs-buy decisions.
• Working product-management skillset: prioritization frameworks, stakeholder discovery, and defining/tracking adoption and outcome metrics.
• Strong working knowledge of data pipelines, data modeling, and cloud data platforms (AWS); hands-on experience with Databricks (Delta Lake, Unity Catalog, Delta Live Tables, MLflow) strongly preferred.
• Demonstrated experience defining data KPI/metrics frameworks and instrumenting dashboards that make quality, reliability, and cost visible to stakeholders.
• Demonstrated experience driving AI/ML-assisted engineering adoption beyond a single team — playbooks, training, or communities of practice that scale across data teams.
• Agile/Kanban methodologies with hands-on experience using Jira, Confluence, and ServiceNow (SNOW).
• Excellent analytical, communication, and executive-presence skills; proficiency in Python and/or SQL.

• Track record of shaping technology or platform strategy based on direct engineering experience, not just business requirements.
• Experience with SRE/FinOps practices for platform cost and reliability management.
• Experience with pharmaceutical, clinical, or life-sciences data; awareness of GxP / 21 CFR Part 11 requirements.
• Databricks certification (e.g., Data Engineer Professional) or equivalent depth of hands-on lakehouse experience.
• Experience with data access-governance and self-service provisioning tooling (e.g., Unity Catalog, Immuta, Collibra, or equivalent IAM automation).

• Bachelor's degree in Computer Science, Data Engineering, Information Systems, or a related field; advanced degree a plus.
• 10+ years of relevant experience in data/software engineering, delivery management, product ownership, or platform management, including 4+ years of people leadership.
• Demonstrated, current hands-on data engineering practice, alongside a history of contributing to or shaping technology/platform strategy based on that direct experience.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly