Lead Data Engineer - Databricks
Full Time
No Travel Required
Remote
Depends on Experience
Wise Skulls Corp.
Fitment
Dice Job Match Score™
🔢 Crunching numbers...
Job Details
Skills
- Databricks
- Snowflake
- Python
- PySpark
- SQL
- ETL/ELT
- Data Pipelines
- Apache Spark
- Data Modeling
- Apache Airflow
- AWS
- Azure
- GCP
- Lakehouse
- Data Warehousing
Summary
Responsibilities
- Data Engineering & Platform Development
- Design, develop, and maintain scalable data pipelines and data products supporting analytics, reporting, AI, and operational use cases.
- Build and optimize ETL/ELT frameworks for large-scale data ingestion, transformation, validation, and consumption.
- Develop and manage cloud-native data platforms leveraging Snowflake, AWS, Apache Airflow and modern data architectures.
- Create scalable data models, data marts, semantic layers, and curated datasets that support enterprise analytics initiatives.
- Optimize SQL workloads, transformation logic, and query performance to improve scalability and cost efficiency.
- Establish reusable engineering frameworks, accelerators, and best practices to improve delivery consistency across projects.
- Ensure high standards of data quality, reliability, governance, and observability throughout the data lifecycle.
- Develop and maintain Snowflake-based data ecosystems, leveraging advanced features for performance optimization and data sharing.
- Build and orchestrate data workflows using Airflow and other workflow scheduling platforms.
- Collaborate directly with client stakeholders to gather requirements, define roadmaps, and develop scalable technical solutions.
- Present solution designs, technical recommendations, and project updates to both technical and business audiences.
- Prepare and maintain comprehensive project documentation, technical specifications, architecture diagrams, and operational runbooks.
Qualifications
Required Qualifications
- 6+ years of experience in Data Engineering, Big Data Engineering, or Cloud Data Platform development.
- Bachelor's or Master's degree in Computer Science, Engineering, Analytics, Mathematics, Information Systems, or related disciplines.
- Strong hands-on expertise in SQL, Python, and PySpark.
- Extensive experience working with Snowflake, Databricks, or similar cloud-native data platforms.
- Proven experience building and supporting large-scale ETL/ELT data pipelines.
- Strong understanding of data warehousing concepts, dimensional modeling, and modern Lakehouse architectures.
- Experience implementing Medallion Architecture and enterprise-grade data modeling practices.
- Hands-on experience with workflow orchestration tools such as Apache Airflow or equivalent scheduling frameworks.
- Experience working with cloud ecosystems including AWS, Azure, or Google Cloud Platform.
- Strong knowledge of performance tuning, optimization, monitoring, and operational support for data platforms.
- Demonstrated experience leading engineering teams and coordinating with client and internal stakeholders.
- Excellent analytical, problem-solving, communication, and stakeholder management skills.
- Ability to work independently and lead complex initiatives in fast-paced consulting environments.
Preferred Qualifications
- Experience with streaming and real-time data processing frameworks.
- Familiarity with DataOps, CI/CD, Infrastructure as Code, and DevOps practices.
- Experience with data governance, data quality frameworks, and metadata management.
- Exposure to AI/ML data pipelines and feature engineering workflows.
- Experience with visualization tools such as Tableau, Power BI, or Looker.
- Hands-on experience with Big Data technologies including Spark, Hadoop, Hive, HBase, Kafka, or related platforms.
- Consulting or client-facing delivery experience in enterprise-scale environments.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 91097138
- Position Id: JPC -5137
- Posted 2 hours ago
Company Info
Since 2019, Wise Skulls has been redefining the landscape of contractual staffing by connecting exceptional talent with industry-leading opportunities. Specializing in the IT sector and expanding into diverse industries, we help build impactful careers and high-performing teams.
Our mission is clear: to align the right people with the right roles while fostering long-term success for individuals and organizations alike. By leveraging cutting-edge technology, industry expertise, and a deep understanding of both on-site and remote work dynamics, we provide staffing solutions that go beyond filling positions and create pathways to growth and innovation.
At Wise Skulls, we aspire to set new benchmarks for efficiency and excellence in staffing. Our vision is to become the go-to platform for contractual hiring, empowering professionals to achieve their career aspirations and enabling businesses to thrive with dynamic teams.
.png%3Fformat%3Dwebp&w=1080&q=75)
Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.