Data Engineer — AI & Agentic Solutions

At Lilly, the work is demanding because patients are waiting. We unite caring with discovery to help make life better for people around the world, knowing that every decision, every detail, and every day matters. Headquartered in Indianapolis, Indiana, our over 50,000 employees around the globe take on complex challenges to discover and deliver life-changing medicines, strengthen how health is understood and managed, and support the communities we serve. This is hard, urgent, selfless work—but it’s work worth doing. If you’re driven by purpose and ready to bring your best to work that truly matters for patients, we invite you to join us. 

At Lilly, we believe in the talent of our workforce. One of the best ways to utilize and develop that talent is to use our existing workforce to fill new and/or open positions. If you are looking for a new position within Lilly, you can view and apply to open roles posted in the internal job posting system. You must meet the minimum qualifications outlined in the job description and have/obtain work authorization in the country the position is located in order to be considered. When applying internally for a position, your current supervisor will receive notification that you have applied to the position. We encourage employees to discuss the opportunity with their supervisor prior to applying. 

R1-R3

Note:  Roles are posted at the lowest level of a band, however, employees should search across all levels of the band to identify all opportunities.  Employees hired on banded positions (ex: P1-P3, R-path, B1-B3, etc.) transfer at their current level, despite the level indicated on the job posting.  For example, if a P2 candidate is selected for a P1-P3 banded position, the candidate will remain a P2 in the new role.
 

The Data Engineer — AI & Agentic Solutions is a hands-on engineer who builds the data pipelines and agentic AI components that make Lilly's Clinical and Non-Clinical data AI-ready and audit-ready. Working to the patterns and standards set by the data architect and senior engineers, this role writes production pipeline code, builds retrieval and tool-calling agent components, and ships them with the tests, evaluations, and observability a regulated environment requires. This is a builder role — approximately 90% hands-on engineering including daily coding, and 10% design contribution and documentation.

The role contributes directly to the Right Model, Right Task mandate — implementing and tuning the routing configurations that send each agentic and LLM workload to the model best suited to it on cost, latency, and accuracy grounds, using Lilly's internal data platform for lineage and metadata rather than standing up disconnected stores. Agentic and AI-assisted ways of working are the expected default across every engineering, analysis, and documentation activity.

Data Engineering & Pipeline Development (Hands-On)

• Build, test, and maintain production data pipelines across Clinical and Non-Clinical domains — ingestion, transformation, and publication into governed lakehouse layers (Bronze/Silver/Gold).
• Implement pipeline patterns defined by senior engineers and the architect — incremental loads, CDC, schema handling, and idempotent reprocessing — and extend them to new sources.
• Monitor and troubleshoot pipeline runs, resolve data quality and performance issues, and improve reliability against agreed SLAs.
• Contribute to curated, analytics-ready data products with documented schemas and contracts that downstream consumers and agentic systems depend on.

Agentic AI & LLM Application Development (Hands-On)

• Build agentic AI components against established patterns — tool-calling agents, retrieval-augmented generation (RAG) chains, and steps within multi-agent workflows — for Clinical and Non-Clinical use cases.
• Implement prompt, tool, and state-handling logic with attention to token/cost and latency budgets, retries, and graceful failure handling.
• Write and run evaluations, guardrail checks, and regression tests for agentic components, and act on the results to improve accuracy and groundedness.
• Instrument agents with logging, tracing, and observability so behaviour is traceable and auditable in a regulated environment.

Right Model, Right Task — Contributing to Data Engineering

• Implement and tune routing configurations that direct each agentic task to the appropriate model on complexity, cost, latency, and accuracy grounds, rather than defaulting to one model for every job.
• Wire agentic components into Lilly's internal data platform (Data Hub catalog, lineage, and metadata services) rather than building parallel metadata stores.
• Contribute to lineage-aware knowledge graph pipelines that capture data provenance, sensitivity, and domain context for agent reasoning.
• Apply routing and access policy correctly in code — ensuring regulatory-sensitive Clinical data is only handled by approved/validated models.
• Run benchmarks and cost/accuracy comparisons across candidate models and share the findings with the squad.

Semantic, Retrieval & Knowledge Graph Engineering

• Build and tune retrieval components — embeddings, chunking strategies, vector indexes, and hybrid search — that ground LLMs in governed Clinical and Non-Clinical data.
• Contribute to semantic models, taxonomies, and knowledge graph structures under the guidance of senior engineers and the architect.
• Measure retrieval quality against defined benchmarks such as groundedness, recall, and citation accuracy, and iterate on the results.

Automation, Reusability & AI-Native Ways of Working

• Use AI-assisted and agentic tooling by default across the data lifecycle — pipeline generation, schema alignment, data-quality scoring, test generation, and documentation.
• Adopt and extend the team's reusable accelerators — agent templates, orchestration patterns, evaluation harnesses, and pipeline frameworks — and contribute improvements back.
• Automate repetitive checks such as lineage capture, schema-drift detection, and data quality validation instead of handling them manually.

Quality, Governance & Responsible AI

• Apply access control, encryption, and prompt/data-leakage safeguards correctly when building on regulated data.
• Build data quality, lineage, and compliance checks into pipelines and agentic components as part of the delivery, not afterwards.
• Follow engineering discipline consistently: version control, code review, automated testing, and CI/CD practices appropriate to a GxP environment.

Collaboration & Knowledge Sharing

• Work with senior engineers, the data architect, business SMEs, and platform teams to turn Clinical and Non-Clinical requirements into working solutions.
• Ask questions early, raise blockers and risks promptly, and seek review on design decisions before building.
• Share learnings, document what is built, and support teammates as the squad scales its AI-native delivery.

Required — Data Engineering (Must-Have, Hands-On)

• Strong hands-on Python and SQL, with working experience in Spark/PySpark for distributed data processing.
• Hands-on experience building and maintaining production data pipelines on a cloud lakehouse platform (Databricks, or equivalent), including Delta/Parquet and orchestration tooling.
• Practical data modeling experience — relational, dimensional, and lakehouse/medallion patterns.
• Working proficiency with Git, code review, automated testing, and CI/CD practices.

Required — AI & Agentic Development (Must-Have, Hands-On)

• 2+ years hands-on experience building AI/agentic components that reached production or pilot: RAG, tool-calling/function-calling agents, or steps within a multi-agent workflow, using LLM application frameworks (e.g., LangGraph, LlamaIndex, Semantic Kernel, or equivalent).
• Hands-on experience with vector databases/semantic search, embeddings, chunking, and retrieval patterns that ground LLMs in governed data.
• Experience writing evaluations, guardrail checks, and prompt iterations against measurable quality criteria — not subjective review alone.
• Practical understanding of model cost, latency, and accuracy trade-offs, and of why different tasks warrant different models.

Required — Platform Integration & Governance

• Working experience with a cloud platform (AWS preferred) and catalog-based access control, encryption, and lineage patterns (Unity Catalog or equivalent).
• Experience consuming enterprise data-catalog and lineage services rather than building parallel metadata stores.
• Awareness of semantic modeling concepts — ontologies, taxonomies, knowledge graphs — and willingness to build depth in them.
• Self-starter who learns new technologies quickly, follows team standards, and actively shares learnings back to the squad.

Preferred

• Certifications in Databricks, AWS, or an equivalent cloud data platform; coursework or certification in LLM/agentic application development.
• Exposure to knowledge graphs, ontologies, or semantic modeling concepts.
• Awareness of GxP / 21 CFR Part 11 compliance and responsible-AI principles in a validated data environment.
• Familiarity with clinical data standards (CDISC, SDTM, ADaM) or non-clinical lab, safety, and manufacturing data.

Minimum

• Bachelor's degree in Computer Science, Information Systems, Engineering, or related discipline.
• 6+ years of hands-on data engineering experience, including 2+ years building AI/agentic or LLM-based solutions
• Demonstrated examples of data pipelines and agentic components personally built and shipped, with evidence of testing, evaluation, and production support.

Preferred

• Master's degree in a quantitative, computing, or engineering discipline.
• Exposure to a regulated industry (pharma, life sciences, finance) with GxP or equivalent compliance requirements.

Note: When applying internally for a position your current supervisor receives notification that you have applied to the position. We encourage employees to discuss the opportunity with their supervisor prior to applying.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form (https://careers.lilly.com/us/en/workplace-accommodation) for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.