Lead Data Engineer with strong experience in Databricks / Remote

Role :- Lead Data Engineer with strong experience in Databricks ( Industry Preference ---Insurance (P&C, Specialty, Brokerage) or broader BFSI preferred)

Location :- - Remote (Unites states)

Job Description :-

Role Overview

About the Role

We are looking for a strong, hands-on Data Engineer with deep Databricks expertise to design, build and scale enterprise data platforms. You will own end-to-end delivery across ingestion, ETL/ELT, data warehousing, lakehouse architecture and BI enablement, working on at least one major hyperscaler (Azure, AWS or Google Cloud Platform). This role suits someone who can work directly with business and technology leaders, turn requirements into production-grade pipelines, and deliver trusted, analytics-ready data at scale.

Key Responsibilities

Design, build and optimize scalable batch and streaming ETL/ELT pipelines on Databricks (PySpark, Spark SQL, Delta Lake, Delta Live Tables / Lakeflow, Workflows).

Architect and implement Lakehouse solutions using the Medallion (Bronze / Silver / Gold) pattern, with Unity Catalog for governance, lineage and access control.

Build and maintain enterprise data warehouse models (dimensional modeling, star/snowflake schemas, SCDs, Data Vault where relevant) on Databricks SQL, Snowflake, Synapse, Redshift or BigQuery.

Migrate legacy ETL and on-prem warehouses (Informatica, SSIS, DataStage, Teradata, Oracle, SQL Server) to modern cloud lakehouse platforms.

Enable BI and reporting by delivering semantic layers and curated datasets for Power BI (primary), plus Tableau, Looker or Qlik; tune performance for dashboards at scale.

Integrate data from core business systems, APIs, files and third-party sources using ADF, AWS Glue, Fivetran, Kafka/Event Hubs or similar.

Implement data quality, observability and testing frameworks (e.g., DLT expectations, Great Expectations, dbt tests).

Apply CI/CD and DataOps practices (Git, Azure DevOps / GitHub Actions, Databricks Asset Bundles, Terraform).

Optimize cost and performance across clusters, SQL warehouses, storage and queries.

Partner with architects, analysts, data scientists and business stakeholders; mentor junior engineers and contribute to engineering standards.

Required Skills & Experience

8 12 years of data engineering experience delivering large-scale ETL and data warehouse solutions.

Strong, hands-on Databricks expertise (must-have): PySpark, Spark SQL, Delta Lake, Unity Catalog, Workflows/Jobs, DLT/Lakeflow, Databricks SQL, performance tuning.

Advanced SQL and Python; Scala is a plus.

Proven data warehousing and data modeling experience (Kimball dimensional modeling, SCDs, fact/dimension design).

Hands-on experience with at least one hyperscaler, ideally multi-cloud:

Azure: ADLS Gen2, Azure Data Factory, Synapse, Fabric, Event Hubs

AWS: S3, Glue, Redshift, EMR, Lambda, Kinesis

Google Cloud Platform: BigQuery, Dataflow, Dataproc, Cloud Storage

BI tools: strong Power BI (data modeling, DAX, dataset optimization); exposure to Tableau, Looker or Qlik.

Experience with orchestration (Databricks Workflows, Airflow, ADF) and modern tools such as dbt.

Solid understanding of data governance, security, lineage and metadata management.

Experience with CI/CD, Git-based workflows and Agile delivery.

Excellent communication skills and comfort working directly with client stakeholders.

Good to Have

Insurance domain experience (P&C, Specialty/E&S, brokerage, MGAs): policy, claims, billing, underwriting, loss runs, bordereaux, exposure and reinsurance data. Broader BFSI experience is also valued.

Familiarity with insurance platforms and data standards (Guidewire, Duck Creek, Majesco, Applied Epic, ACORD).

Databricks certifications: Data Engineer Associate / Professional.

Cloud certifications (Azure Data Engineer, AWS Data Engineer, Google Cloud Platform Professional Data Engineer).

Snowflake and/or Microsoft Fabric experience.

Exposure to ML/AI data pipelines, feature stores, MLflow, or GenAI/RAG data preparation.

Experience with real-time streaming (Structured Streaming, Kafka).

Education

Bachelor's or Master's degree in Computer Science, Engineering, Information Systems or a related field, or equivalent practical experience.