Overview
On Site
Up to $120,000
Full Time
Skills
Azure
ADLS
CI/CD
DATABRICKS
Data Engineer
Job Details
Senior Data Engineer Spark (Java) & Databricks with ADLS Expertise
Job Description:
We are looking for a highly skilled Senior Data Engineer with 8 to 12 years of experience in building and optimizing large-scale data processing pipelines. The ideal candidate will have strong hands-on expertise in Apache Spark using Java, proficiency in Databricks, and solid knowledge of Azure Data Lake Storage (ADLS) concepts and implementations. Experience with Python is also valuable, but Spark with Java is preferred.
Key Responsibilities:
- Design, develop, and optimize scalable data pipelines and ETL processes using Apache Spark (Java preferred)on Databricks.
- Work closely with architects, data scientists, and business stakeholders to translate data requirements into scalable data solutions.
- Implement robust data storage and retrieval strategies using Azure Data Lake Storage (ADLS).
- Ensure high performance and reliability of Spark jobs, optimizing for memory, compute, and scalability.
- Integrate data from various sources and manage data ingestion pipelines effectively.
- Participate in code reviews, enforce best practices, and mentor junior team members.
- Collaborate with DevOps and cloud teams to ensure efficient deployment and monitoring of data pipelines.
Required Skills:
- 8 12 years of experience in Data Engineering or Software Development roles.
- Strong hands-on expertise in Apache Spark, preferably using Java(Python is a plus).
- Experience with Databricksfor developing and managing Spark jobs.
- Deep understanding of Azure Data Lake Storage (ADLS)and its best practices.
- Experience in data modeling, schema design, and data performance tuning.
- Familiarity with big data tools and data lake architecture on Azure.
- Strong problem-solving, communication, and collaboration skills.
Preferred Skills:
- Experience with CI/CD pipelines for data workflows.
- Exposure to Delta Lake, Azure Synapse, or other Azure-based data services.
- Certifications in Azure or Databricks would be an advantage.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.