Lead Data Engineer with strong experience in Databricks | Remote (Unites states)---Visa-independent candidates
Cyberobotix
Dice Job Match Score™
🔢 Crunching numbers...
Job Details
Skills
- Data Engineer
- Databricks
- Insurance
- BFSI
- Brokerage
- ETL
- BI
- Data Warehouse
Summary
Job Title: Lead Data Engineer with strong experience in Databricks
Location: Remote (Unites states)--- Visa-independent candidates
Duration: Long Term
Job Description:
Role Overview
Title | Senior / Lead Data Engineer – Databricks (ETL, Data Warehouse, Lakehouse, BI) |
Experience | 8–12 years in data engineering, with at least 3–4 years hands-on with Databricks |
Industry Preference | Insurance (P&C, Specialty, Brokerage) or broader BFSI preferred |
About the Role
We are looking for a strong, hands-on Data Engineer with deep Databricks expertise to design, build and scale enterprise data platforms. You will own end-to-end delivery across ingestion, ETL/ELT, data warehousing, lakehouse architecture and BI enablement, working on at least one major hyperscaler (Azure, AWS or Google Cloud Platform). This role suits someone who can work directly with business and technology leaders, turn requirements into production-grade pipelines, and deliver trusted, analytics-ready data at scale.
Key Responsibilities
- Design, build and optimize scalable batch and streaming ETL/ELT pipelines on Databricks (PySpark, Spark SQL, Delta Lake, Delta Live Tables / Lakeflow, Workflows).
- Architect and implement Lakehouse solutions using the Medallion (Bronze / Silver / Gold) pattern, with Unity Catalog for governance, lineage and access control.
- Build and maintain enterprise data warehouse models (dimensional modeling, star/snowflake schemas, SCDs, Data Vault where relevant) on Databricks SQL, Snowflake, Synapse, Redshift or BigQuery.
- Migrate legacy ETL and on-prem warehouses (Informatica, SSIS, DataStage, Teradata, Oracle, SQL Server) to modern cloud lakehouse platforms.
- Enable BI and reporting by delivering semantic layers and curated datasets for Power BI (primary), plus Tableau, Looker or Qlik; tune performance for dashboards at scale.
- Integrate data from core business systems, APIs, files and third-party sources using ADF, AWS Glue, Fivetran, Kafka/Event Hubs or similar.
- Implement data quality, observability and testing frameworks (e.g., DLT expectations, Great Expectations, dbt tests).
- Apply CI/CD and DataOps practices (Git, Azure DevOps / GitHub Actions, Databricks Asset Bundles, Terraform).
- Optimize cost and performance across clusters, SQL warehouses, storage and queries.
- Partner with architects, analysts, data scientists and business stakeholders; mentor junior engineers and contribute to engineering standards.
Required Skills & Experience
- 8–12 years of data engineering experience delivering large-scale ETL and data warehouse solutions.
- Strong, hands-on Databricks expertise (must-have): PySpark, Spark SQL, Delta Lake, Unity Catalog, Workflows/Jobs, DLT/Lakeflow, Databricks SQL, performance tuning.
- Advanced SQL and Python; Scala is a plus.
- Proven data warehousing and data modeling experience (Kimball dimensional modeling, SCDs, fact/dimension design).
- Hands-on experience with at least one hyperscaler, ideally multi-cloud:
- Azure: ADLS Gen2, Azure Data Factory, Synapse, Fabric, Event Hubs
- AWS: S3, Glue, Redshift, EMR, Lambda, Kinesis
- Google Cloud Platform: BigQuery, Dataflow, Dataproc, Cloud Storage
- BI tools: strong Power BI (data modeling, DAX, dataset optimization); exposure to Tableau, Looker or Qlik.
- Experience with orchestration (Databricks Workflows, Airflow, ADF) and modern tools such as dbt.
- Solid understanding of data governance, security, lineage and metadata management.
- Experience with CI/CD, Git-based workflows and Agile delivery.
- Excellent communication skills and comfort working directly with client stakeholders.
Good to Have
- Insurance domain experience (P&C, Specialty/E&S, brokerage, MGAs): policy, claims, billing, underwriting, loss runs, bordereaux, exposure and reinsurance data. Broader BFSI experience is also valued.
- Familiarity with insurance platforms and data standards (Guidewire, Duck Creek, Majesco, Applied Epic, ACORD).
- Databricks certifications: Data Engineer Associate / Professional.
- Cloud certifications (Azure Data Engineer, AWS Data Engineer, Google Cloud Platform Professional Data Engineer).
- Snowflake and/or Microsoft Fabric experience.
- Exposure to ML/AI data pipelines, feature stores, MLflow, or GenAI/RAG data preparation.
- Experience with real-time streaming (Structured Streaming, Kafka).
Education
- Bachelor's or Master's degree in Computer Science, Engineering, Information Systems or a related field, or equivalent practical experience.
Best Regards,
Ravi M | Senior Recruiter
Cyberobotix Technology
6500 River Place Blvd, Building 7,
Suites 250, Austin, Texas 78730
E: ravi@cyberobotix.com
W:
Technology Consulting & Talent Solutions | United States & India
This email and any attachments are confidential and intended only for the recipient.
- Dice Id: 91172251
- Position Id: 9138979
- Posted 1 hour ago
Company Info
About Cyberobotix
Cyberobotix emerges as a dynamic force in the IT consulting and staffing industry, poised to redefine excellence in talent acquisition and technology-driven solutions. With an in-depth understanding of diverse organizational needs, we specialize in delivering tailored solutions designed to match your unique requirements. Whether you need specialized short-term expertise, long-term strategic partners, or permanent team members, Cyberobotix is dedicated to exceeding your expectations.
At Cyberobotix, we understand that people and technology are at the heart of every success. Driven by our unwavering commitment to your growth, we place your needs at the forefront of everything we do. Our innovative approach combines cutting-edge technology with personalized service, ensuring that each placement or solution seamlessly aligns with your organization’s culture and objectives. Transparency is at our core, offering you full visibility to make well-informed decisions. From initial consultation to ongoing support, we monitor feedback and performance metrics to ensure your satisfaction.
