Overview
Remote
$50 - $80
Contract - Independent
Contract - 6 Month(s)
No Travel Required
Skills
Amazon Web Services
Apache Spark
SQL
Python
Databricks
Data Lake
Amazon Redshift
Microsoft Azure
Job Details
Job Title: Databricks Engineer
Location: Remote - US
Job Type: Contract 6 + Months
Experience Level: 7+ yrs
About the Role
We are seeking a highly skilled Databricks Engineer to join our growing data team. You will play a key role in building and optimizing our big data pipelines, implementing scalable data solutions, and collaborating with cross-functional teams to unlock insights from large datasets using Databricks, Apache Spark, and Azure/AWS cloud technologies.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
- Implement ETL/ELT workflows for processing structured and unstructured data.
- Optimize and fine-tune Spark jobs for performance and cost-efficiency.
- Collaborate with Data Scientists, Analysts, and Engineers to integrate data from various sources.
- Work with Delta Lake, notebooks, and MLflow to support analytics and machine learning initiatives.
- Ensure data quality, governance, and compliance standards are met.
- Monitor and troubleshoot data pipelines in production environments.
- Automate data validation and pipeline testing processes.
- Participate in code reviews, architecture discussions, and Agile ceremonies.
Required Skills & Qualifications
- Bachelor s or Master s degree in Computer Science, Engineering, or a related field.
- 3 6 years of experience working with big data technologies.
- Hands-on experience with Databricks, Apache Spark, and Delta Lake.
- Proficiency in Python and SQL; familiarity with Scala is a plus.
- Strong understanding of data warehousing concepts and performance optimization.
- Experience with cloud platforms like Azure (preferred), AWS, or Google Cloud Platform.
- Familiarity with CI/CD tools and DevOps practices in a data engineering context.
- Good understanding of data security and governance frameworks.
Preferred Qualifications
- Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional).
- Experience working with Azure Data Lake, Azure Synapse, or AWS Redshift.
- Knowledge of ML flow, Unity Catalog, or similar tools.
- Experience with data orchestration tools like Airflow, Azure Data Factory, or dbt.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.