Overview
On Site
Up to $120,000
Full Time
Skills
data engineer
spark
databricks
hadoop
cloud
azure
Job Details
Data Engineering Tech Lead Spark & Databricks
Job Description:
We are seeking a Data Engineering Tech Lead with strong hands-on experience in building and optimizing scalable data processing solutions using Apache Spark, Databricks, and cloud-based big data technologies. The ideal candidate has a solid foundation in Java or Python programming, and a proven track record of leading teams in data engineering projects, delivering efficient, high-performance solutions in enterprise environments.
Key Responsibilities:
- Lead the design, development, and optimization of large-scale data processing applications using Spark on Databricks.
- Collaborate with architects, data scientists, and stakeholders to define and implement scalable data engineering solutions.
- Guide and mentor a team of developers, providing technical leadership and ensuring best practices are followed.
- Design and manage data pipelines leveraging Hadoop, HDFS, and other Big Data ecosystem tools.
- Ensure the reliability, performance, and security of data workflows and storage.
- Write clean, scalable, and reusable code primarily in Java and Python for data processing.
- Work with cloud-based Big Data platforms (e.g., Azure, AWS, or Google Cloud Platform) to implement robust data solutions.
- Monitor and troubleshoot Spark jobs and data pipelines to ensure optimal performance.
Required Skills and Experience:
- 8+ years of overall experience in data engineering or software development roles.
- Strong hands-on expertise with Apache Spark and Databricks.
- Solid programming experience in Java or Python.
- Proven leadership experience, including managing and mentoring technical teams.
- In-depth knowledge of Hadoop, HDFS, and Big Data architectural patterns.
- Experience working with cloud data platforms (Azure/AWS/Google Cloud Platform).
- Excellent communication and problem-solving skills.
Preferred Qualifications:
- Certification in Databricks or cloud data services (e.g., Azure Data Engineer Associate).
- Experience with CI/CD pipelines, data lake architectures, and performance tuning in Spark.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.