Back to jobs

Senior Director, NLP & Market Research Data Engineering Lead

Reads like a Data Leadership role: running a data team.

Indianapolis, Indiana, United States of America

Actively hiring, first seen by us 7 days ago
Pay
$175,500–257,400/yearAnnual period assumed — pay source
Actual compensation will depend on a candidate’s education, experience, skills, and geographic location. The anticipated wage for this position is $175,500 - $257,400 Full-time equivalent employees also will be eligible for a company bonus (depending, in part, on company and individual performance). In addition, Lilly offers a comprehensive benefit program to eligible employees, including eligibility to participate in a company-sponsored 401(k); pension; vacation benefits; eligibility for medical, dental, vision and prescription drug benefits; flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts); life insurance and death benefits; certain time off and leave of absence benefits; and well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities).Lilly reserves the right to amend, modify, or terminate its compensation and benefit programs in its sole discretion and Lilly’s compensation practices and guidelines will apply regarding the details of any promotion or transfer of Lilly employees.
Read the full posting
Work setting
Not stated
Employment
Not stated

What you’ll work on

Full posting
  • Design and operate NLP and text pipelines on Databricks medallion layers

  • Build entity, theme, sentiment and embedding enrichment for raw text

  • Own the chunking, embedding and vector indexing behind research retrieval (RAG)

  • Author data contracts and enforce PII, consent and provenance rules

  • Set NLP data quality checks, monitoring and alerting

From the employer’s posting
Design and operate production NLP and text pipelines on Databricks (Unity Catalog, Delta)
Entity and theme extraction, topic modeling, sentiment, summarization, and embedding / vectorization
Chunking, embedding, vector indexing, and grounding
Author and steward data contracts (ODCS); ensure PII, consent, classification, and provenance travel with the data
Establish data quality & evaluation for NLP.
Apply at Lilly

Posting record

Report an error

Actively hiringFirst seen by us 7 days ago, still listed, and not a repost of an earlier posting.

Status in our records
Live
First seen by us
Nov 10, 20267 days ago
Recorded sightings
9
Last seen by us
Nov 18, 2026

These dates show when we found the posting. Check the employer’s website to confirm it is still accepting applications.

How this job compares

Pay

In line with similar jobs

$176K–$257K against $155K–$227K, the middle half of what similar US postings state.

From 3,228 postings in the last year. Stated ranges, not salaries paid.

Time since we first saw it

Day 7

Early. Similar jobs usually come down between day 8 and day 57.

At Lilly, jobs usually come down by about day 29.

From 9,712 similar postings that came down, and 97 of Lilly’s. Coming down is not always a hire.

Lilly is hiring

46 open jobs

Open jobs by the week we first saw them, last 12 weeks.

  • Data Leadership hiring is steady
  • Takes jobs down at a usual pace
Hiring record

See how this role fits your experience

Add your resume and we’ll mark which of these tools you already use, and point out anything in the posting that could rule you out.

  • Python
  • SQL
  • Azure
  • Databricks
  • dbt
  • Delta Lake
  • Terraform
  • PySpark
  • Great Expectations
Where the posting names them
Additional Skills / Preferences ● Demonstrated experience designing lakehouse / medallion architectures (Databricks, Unity Catalog, Delta). ● Hands-on NLP and text engineering — parsing and normalization, entity and topic extraction, embeddings / vectorization, and retrieval / RAG. ● Advanced SQL and Python (PySpark), with data-quality and testing practices for production pipelines. ● Working knowledge of data governance, classification, and access control (row- and column-level security), including the handling of sensitive and unstructured data. ● Ability to influence and align technical and business partners without direct authority in an ambiguous, matrixed environment. ● Experience with vector databases and search, LLM grounding, and agentic / RAG architectures on enterprise content. ● Familiarity with market-research data — social listening, survey and verbatim data, qualitative transcripts, and secondary / medical-scientific text. ● Data-contract frameworks (ODCS), policy-as-code (OPA / Rego), and Git-based CI/CD (Azure DevOps or equivalent); transformation- and infrastructure-as-code (dbt, Terraform). ● Experience in regulated or healthcare data environments; awareness of HIPAA, consent, and MLR / content-governance obligations. ● Readiness to build and lead a small engineering team as the capability scales; experience mentoring or leading engineers, formally or in a matrix, preferred. ● Comfort building a capability that does not exist today, and translating clearly for research, engineering, and governance audiences. Additional Information
Key Objectives ● Build & run the NLP / unstructured-data pipelines. Design and operate production NLP and text pipelines on Databricks (Unity Catalog, Delta) that ingest, parse, structure, and enrich Market Research's unstructured sources across the medallion (bronze → silver → gold). ● Stand up the enrichment & embedding layer. Entity and theme extraction, topic modeling, sentiment, summarization, and embedding / vectorization — turning raw text into structured, queryable, feature-ready signal. ● Own the retrieval / RAG data foundation. Chunking, embedding, vector indexing, and grounding so research agents reason over the full MR evidence base, in partnership with the Agent Factory and Head of AI. ● Make unstructured data AI-ready & governed. Author and steward data contracts (ODCS); ensure PII, consent, classification, and provenance travel with the data into agentic consumption; partner with the Policy-as-Code function to render governance intent as enforced policy (OPA / Rego), with no leakage. ● Establish data quality & evaluation for NLP. Accuracy, coverage, drift, and grounding / hallucination checks (pytest, DLT / Great Expectations), with monitoring and alerting so text data stays trustworthy and auditable. ● Serve Market Research at scale. Engineer the pipelines behind social listening, brand-health, and secondary-MR text analysis, partnering with Research Science and the shared LBI Market Research pool that runs the analysis; establish reference architecture, reusable patterns, and self-service. ● Set technical direction & standards. Own the standards for NLP and unstructured-data engineering across CIA; provide architectural review and gatekeeping for new data flows, integrations, and agent-consumption paths; provide technical direction and mentorship to engineers and the vendor / LBI bench. ● Other key initiatives and special projects as needed. Leadership

Your next step

Complete your application on lilly.wd115.myworkdayjobs.com. The employer’s form will show what is required.

Applying through Phenom

  • Phenom runs the employer’s careers site, but the application itself usually runs on another system, often Workday or Taleo. The advice for that system applies once you click Apply.

Already applied? Track this application

Job description

View original posting ↗

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us. 


NLP & Market Research Data Engineering Lead
Customer Insights and Analytics (CIA)


At Lilly, we unite caring with discovery to make life better for people around the world. We are a global healthcare leader headquartered in Indianapolis, Indiana. Our 50,000+ employees around the world work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities through philanthropy and volunteerism. We give our best effort to our work, and we put people first. We're looking for people who are determined to make life better for people around the world.


The Customer Insights and Analytics (CIA) organization is Lilly's business-facing insight and analytics capability, uniting Market Research and Analytics into a single, integrated partnership model that helps our business partners make faster, better, more customer-centered decisions.


Organization Overview
Customer Insights and Analytics (CIA) is Lilly's enterprise insight and analytics organization, built in three tiers: Strategic Intelligence teams embedded with each therapeutic area, functional-area Integrated Insights teams that own the insight craft, and foundational teams built for scale. Data Engineering is one of those foundational teams — it builds the governed, AI-ready data foundation once, centrally, so it can be reused across the business as Lilly embeds agentic AI at scale. Within Data Engineering, this role owns the unstructured-data and NLP foundation for Market Research.


Position Overview
The NLP & Market Research Data Engineering Lead builds and runs the engineered data foundation for unstructured and research data — the non-tabular sources that carry the consumer, patient, and HCP “why”: social-listening streams, survey verbatims, qualitative transcripts, call and advisory notes, secondary reports, and medical and scientific text. As Lilly extends its agentic platform beyond structured data, this is the engineering role that delivers the unstructured-data half of the AI-ready foundation for Market Research — the NLP and text pipelines, the enrichment and embedding layer, and the retrieval (RAG) foundation on which research agents and MR analytics depend.


The role sits inside CIA's central Data Engineering function, reporting to the Head of Data Engineering & Risk Mitigation, and is a joint priority of Data Engineering and Agentic Intelligence & Products. It builds the capability centrally, once, and makes it reusable across the business — partnering with the embedded Research & Unstructured Data Product Owners, the Agent Factory, Research Science (which owns social listening and MR methods as central capabilities), the Strategic Intelligence Market Research teams, and the governance and stewardship functions.


This role builds and runs the pipelines, the unstructured-data products, and the retrieval foundation; it does not frame or interpret the research, and it does not own the agents or Skills. Where the Agent Factory builds the agent and the Market Research teams frame the question, this role builds the governed, AI-ready data foundation they both depend on.



Key Objectives


● Build & run the NLP / unstructured-data pipelines. Design and operate production NLP and text pipelines on Databricks (Unity Catalog, Delta) that ingest, parse, structure, and enrich Market Research's unstructured sources across the medallion (bronze → silver → gold).
● Stand up the enrichment & embedding layer. Entity and theme extraction, topic modeling, sentiment, summarization, and embedding / vectorization — turning raw text into structured, queryable, feature-ready signal.
● Own the retrieval / RAG data foundation. Chunking, embedding, vector indexing, and grounding so research agents reason over the full MR evidence base, in partnership with the Agent Factory and Head of AI.
● Make unstructured data AI-ready & governed. Author and steward data contracts (ODCS); ensure PII, consent, classification, and provenance travel with the data into agentic consumption; partner with the Policy-as-Code function to render governance intent as enforced policy (OPA / Rego), with no leakage.
● Establish data quality & evaluation for NLP. Accuracy, coverage, drift, and grounding / hallucination checks (pytest, DLT / Great Expectations), with monitoring and alerting so text data stays trustworthy and auditable.
● Serve Market Research at scale. Engineer the pipelines behind social listening, brand-health, and secondary-MR text analysis, partnering with Research Science and the shared LBI Market Research pool that runs the analysis; establish reference architecture, reusable patterns, and self-service.
● Set technical direction & standards. Own the standards for NLP and unstructured-data engineering across CIA; provide architectural review and gatekeeping for new data flows, integrations, and agent-consumption paths; provide technical direction and mentorship to engineers and the vendor / LBI bench.
● Other key initiatives and special projects as needed.


Leadership


While there are no direct reports today, this person will work through Lilly's teams in India and across teams in CIA and Tech@Lilly to build out the Market Research data products (bronze and silver) needed to power agentic solutions for the teams. Leadership is exercised through technical direction, standards, mentorship, and influence across this matrixed set of partners rather than direct people management. The capability this role builds is new and expected to grow — as unstructured-data and Market Research engineering demand scales, this role is positioned to build and lead a small engineering team, and the successful candidate should be able to grow into people leadership over time.


● Lead through influence today. Set technical standards and provide architectural review across CIA's unstructured-data and NLP engineering; direct and quality-assure the vendor / LBI engineering bench that helps deliver the work.
● Mentor and develop. Coach engineers and analysts on NLP and unstructured-data engineering practice, and raise the technical bar across the extended team.
● Build for a future team. Establish the standards, playbooks, reusable architecture, and onboarding patterns a future in-house team will inherit, so the capability can scale without re-engineering.
● Grow into people leadership. As the function matures, take on direct reports — hiring, developing, and retaining engineering talent and shaping the operating model for the Market Research / unstructured-data engineering squad.
● Team Lilly leadership. Lead through change, translate strategy into clear technical goals, remove barriers, and model an inclusive, high-trust team environment.
Key Relationships
● Head of Data Engineering & Risk Mitigation — direct manager; sets the central engineering agenda and standards this role operates within.
● Head of AI / Agentic Intelligence & Products and the Agent Factory — joint owners of the agentic roadmap and primary consumers of the retrieval / RAG foundation.
● Research Science — owns social listening and Market Research methods as central capabilities; defines what “good” looks like for the signal this role engineers.
● Strategic Intelligence Market Research teams (by therapeutic area) — frame the business and research questions the data foundation must serve.
● Research & Unstructured Data Product Owners — translate research demand into the product backlog this role builds against.
● Governance, Stewardship & Policy-as-Code — partner to ensure PII, consent, classification, and provenance are enforced in code.
● LBI / vendor engineering bench — extended delivery capacity this role directs and quality-assures.
● Tech@Lilly — platform and enterprise-engineering partner.


Basic Qualifications


● Bachelor's degree in Computer Science, Data Engineering, or a related technical discipline.
● 8+ years in data engineering, including building and operating production NLP, text-processing pipelines at enterprise scale.

Additional Skills / Preferences


● Demonstrated experience designing lakehouse / medallion architectures (Databricks, Unity Catalog, Delta).
● Hands-on NLP and text engineering — parsing and normalization, entity and topic extraction, embeddings / vectorization, and retrieval / RAG.
● Advanced SQL and Python (PySpark), with data-quality and testing practices for production pipelines.
● Working knowledge of data governance, classification, and access control (row- and column-level security), including the handling of sensitive and unstructured data.
● Ability to influence and align technical and business partners without direct authority in an ambiguous, matrixed environment.
● Experience with vector databases and search, LLM grounding, and agentic / RAG architectures on enterprise content.
● Familiarity with market-research data — social listening, survey and verbatim data, qualitative transcripts, and secondary / medical-scientific text.
● Data-contract frameworks (ODCS), policy-as-code (OPA / Rego), and Git-based CI/CD (Azure DevOps or equivalent); transformation- and infrastructure-as-code (dbt, Terraform).
● Experience in regulated or healthcare data environments; awareness of HIPAA, consent, and MLR / content-governance obligations.
● Readiness to build and lead a small engineering team as the capability scales; experience mentoring or leading engineers, formally or in a matrix, preferred.
● Comfort building a capability that does not exist today, and translating clearly for research, engineering, and governance audiences.

Additional Information


● Indianapolis, IN — hybrid.
● 5–15% travel depending on business need.
● Lilly is an EEO/Affirmative Action Employer and does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability, or any other legally protected status.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.


Lilly is proud to be an EEO Employer and does not discriminate on the basis of age, race, color, religion, gender identity, sex, gender expression, sexual orientation, genetic information, ancestry, national origin, protected veteran status, disability, or any other legally protected status.


Our employee resource groups (ERGs) offer strong support networks for their members and are open to all employees. Our current groups include: Africa, Middle East, Central Asia (AMECA), Black Employees at Lilly (BE@Lilly), Chinese Culture Network (CCN), EnAble, Evolve, Lilly Indian Network (LIN), Organization of Latinx at Lilly (OLA), Pride (LGBTQ+ Allies), Veterans Leadership Network (VLN) and Women’s Initiative for Leading at Lilly (WILL).


Actual compensation will depend on a candidate’s education, experience, skills, and geographic location.  The anticipated wage for this position is

$175,500 - $257,400

Full-time equivalent employees also will be eligible for a company bonus (depending, in part, on company and individual performance). In addition, Lilly offers a comprehensive benefit program to eligible employees, including eligibility to participate in a company-sponsored 401(k); pension; vacation benefits; eligibility for medical, dental, vision and prescription drug benefits; flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts); life insurance and death benefits; certain time off and leave of absence benefits; and well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities).Lilly reserves the right to amend, modify, or terminate its compensation and benefit programs in its sole discretion and Lilly’s compensation practices and guidelines will apply regarding the details of any promotion or transfer of Lilly employees.

#WeAreLilly

About Lilly

People
About 50,000 in 2025, up 6% from the year before.Annual report (SEC Form 10-K)
Company
Listed as LLY.

Demand for Data Leadership roles

New Data Leadership postings at the employers we follow are up 8% over the last 6 weeks compared with the 6 before.

Weekly new Data Leadership postings at the same employers, from the week of August 17 to the week of November 9. The fewest in a week was 752, the most 1,164, and the latest week had 785.
August 17Last week

Each bar is one week of new postings. The darkest bars are the last 6 weeks.

Counted from the hiring pages we’ve followed for at least five weeks, so employers we add don’t look like growth. Updated weekly.

About the role

We file this job under Data Leadership, by the work the posting describes.

Data leaders run data teams or design how a company's data fits together. Managers hire, set priorities and answer for whether the data work pays off; architects choose the systems and the shape of the data.

Expect years of hands-on data work, experience leading people or projects, and the ability to explain tradeoffs to executives.

Unlike a senior engineer or analyst, most of a data leader's week goes to people, planning and decisions rather than building.

US government figures for computer and information systems managers

Median yearly pay, May 2025
$175,140
Expected change, 2025 to 2035
15.8% more jobsagainst 3.5% for all jobs

The government has no code for data leaders, so these figures are for the closest group, computer and information systems managers, which includes data processing and data operations managers.

Source: US Bureau of Labor Statistics, Occupational Employment and Wage Statistics (May 2025) and Employment Projections (2025 to 2035). Occupation code 11-3021.

Source details

View original posting ↗

Source notes

Source excerpts

Selected passages from the saved posting. Check the full description for conditions and exceptions.

Pay
Actual compensation will depend on a candidate’s education, experience, skills, and geographic location. The anticipated wage for this position is $175,500 - $257,400 Full-time equivalent employees also will be eligible for a company bonus (depending, in part, on company and individual performance). In addition, Lilly offers a comprehensive benefit program to eligible employees, including eligibility to participate in a company-sponsored 401(k); pension; vacation benefits; eligibility for medical, dental, vision and prescription drug benefits; flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts); life insurance and death benefits; certain time off and leave of absence benefits; and well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities).Lilly reserves the right to amend, modify, or terminate its compensation and benefit programs in its sole discretion and Lilly’s compensation practices and guidelines will apply regarding the details of any promotion or transfer of Lilly employees.
Location & working pattern

Indianapolis, Indiana, United States of America

Additional Information ● Indianapolis, IN — hybrid. ● 5–15% travel depending on business need. ● Lilly is an EEO/Affirmative Action Employer and does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability, or any other legally protected status. Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.
Work authorization

No clear work-authorization passage found. Eligibility is unconfirmed.

Find answers in the posting

AI
How answers work

AI selects complete passages from this posting. Check them for conditions and exceptions.

Uses this posting and your question. No profile needed.