Sr. Data Engineer - Lakehouse Architecture

At Lilly, we unite caring with discovery to make life better for people around the world. We are a global healthcare leader headquartered in Indianapolis, Indiana. Our employees around the world work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities through philanthropy and volunteerism. We give our best effort to our work, and we put people first. We're looking for people who are determined to make life better for people around the world.

Tech@Lilly is seeking a highly skilled Senior Data Engineer who can implement and optimize large-scale Lakehouse solutions and drive the evolution of our modern data platform while providing technical leadership to a growing team. The ideal candidate will have hands-on experience with modern data engineering technology stack and a proven track record of managing engineering talent in fast-paced environments.

What you will be doing

• Design and implement comprehensive Lakehouse architecture solutions using technologies like Databricks, Snowflake, or equivalent platforms
• Build and maintain real-time and batch data processing systems using Apache Spark, Kafka, and similar technologies
• Architect scalable data pipelines that handle structured, semi-structured, and unstructured data to deliver AI ready data.
• Develop data transformation workflows using tools like DBT, Airflow, or Databricks
• Lead the technical strategy for data lake and data warehouse integration, ensuring optimal performance and cost efficiency.
• Implement data governance frameworks, including data quality monitoring, lineage tracking, data time travel and security protocols.
• Implement centralized data catalog system and enhance data discovery using technologies like Elastic Search / Open Search.
• Establish monitoring and alerting systems for data pipeline health using technologies like Apache Superset
• Drive adoption of modern data engineering best practices including Infrastructure as Code, CI/CD, and automated testing
• Collaborate with data scientists, analysts, and business stakeholders to translate requirements into robust technical solutions
• Mentor a team of 3-5 data engineers
• Foster a collaborative team culture focused on continuous learning and innovation

How you will succeed

• Proven ability to mentor junior engineers and facilitate knowledge sharing
• Strong project management skills with experience leading cross-functional initiatives
• Coordinate cross-functional projects and ensure effective communication between technical and business teams
• Demonstrated ability to make architectural decisions and drive technical consensus
• Embrace a growth mindset and actively seek opportunities to expand your leadership capabilities and technical mastery

What you should bring

• Experience with streaming data technologies (Kafka)
• Familiarity with data cataloging tools (Apache Atlas or DataHub)
• Familiarity with high performance data service framework (Arrow Flight)
• Industry certifications in cloud platforms or big data technologies
• Expert-level proficiency in Python and SQL for data transformation and pipeline development
• Strong experience with Apache Spark for big data processing and analytics
• Hands-on experience with cloud platforms (AWS or Azure) and their data services
• Proficiency with Infrastructure as Code tools (Terraform, CloudFormation)
• Experience with containerization (Docker, Kubernetes) and orchestration platforms
• Knowledge of data modeling techniques for both analytical and operational workloads
• Understanding of data governance, security, and compliance requirements
• Knowledge in the pharmaceutical or life sciences domain

Basic Qualification

• Master's degree in computer science, Engineering, or related technical field
• 3+ years of hands-on experience with Lakehouse architectures (Databricks, Snowflake, or similar)
• 7+ years of overall data engineering experience with large-scale distributed systems

Organization Overview

Lilly IT builds and maintains capabilities using cutting edge technologies like most prominent tech companies. What differentiates Lilly IT is that we redefine what's possible through tech to advance our purpose - creating medicines that make life better for people around the world, like data driven drug discovery and connected clinical trials. We hire the best technology professionals from a variety of backgrounds, so they can bring an assortment of knowledge, skills, and diverse thinking to deliver innovative solutions in every area of our business.

Research Tech unites science with technology to accelerate the Research and Development of medicines and to deliver therapeutic innovations. The team leverages technology and platforms to streamline scientific experimentation to help Researchers follow the science, to understand the disease and identify potential therapies. They are at the forefront of advanced analytics to enable data driven drug discovery, to innovate so Scientists can rapidly analyze and accelerate the discovery of new medicines.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form ( for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lilly is proud to be an EEO Employer and does not discriminate on the basis of age, race, color, religion, gender identity, sex, gender expression, sexual orientation, genetic information, ancestry, national origin, protected veteran status, disability, or any other legally protected status.

Our employee resource groups (ERGs) offer strong support networks for their members and are open to all employees. Our current groups include: Africa, Middle East, Central Asia Network, Black Employees at Lilly, Chinese Culture Network, Japanese International Leadership Network (JILN), Lilly India Network, Organization of Latinx at Lilly (OLA), PRIDE (LGBTQ+ Allies), Veterans Leadership Network (VLN), Women's Initiative for Leading at Lilly (WILL), enAble (for people with disabilities). Learn more about all of our groups.

Actual compensation will depend on a candidate's education, experience, skills, and geographic location. The anticipated wage for this position is
$70,500 - $200,200

Full-time equivalent employees also will be eligible for a company bonus (depending, in part, on company and individual performance). In addition, Lilly offers a comprehensive benefit program to eligible employees, including eligibility to participate in a company-sponsored 401(k); pension; vacation benefits; eligibility for medical, dental, vision and prescription drug benefits; flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts); life insurance and death benefits; certain time off and leave of absence benefits; and well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities).Lilly reserves the right to amend, modify, or terminate its compensation and benefit programs in its sole discretion and Lilly's compensation practices and guidelines will apply regarding the details of any promotion or transfer of Lilly employees.

#WeAreLilly