Data engineer

Remote • Posted 1 hour ago • Updated 1 hour ago
Contract W2
12 Months
No Travel Required
Remote
$50 - $60/hr
Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

  • Data Bricks

Summary

Job Summary

We are seeking an experienced Data Engineer with strong hands-on expertise in Databricks, Apache Spark, Python, and cloud-based data engineering technologies. The ideal candidate should have experience designing, developing, and maintaining scalable data pipelines, implementing ETL/ELT processes, and building robust data platforms using the Databricks Lakehouse architecture.

Candidates with a valid Databricks certification and strong experience in data processing, data integration, and cloud data platforms are highly preferred.

Key Responsibilities

  • Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
  • Build and optimize ETL/ELT workflows for processing large volumes of structured and unstructured data.
  • Develop data engineering solutions using Python, PySpark, and SQL.
  • Work with Databricks Lakehouse architecture, Delta Lake, and Delta Live Tables.
  • Implement data ingestion, transformation, cleansing, and validation processes.
  • Develop and manage workflows using Databricks Workflows and Jobs.
  • Optimize Spark jobs, cluster performance, and SQL queries for better performance.
  • Implement data governance, security, and access control using Unity Catalog.
  • Integrate data from multiple sources, including databases, APIs, and cloud storage.
  • Collaborate with data architects, data analysts, data scientists, and business stakeholders.
  • Implement CI/CD pipelines and deployment processes for data engineering solutions.
  • Ensure data quality, reliability, scalability, and performance across data platforms.

Required Technical Skills

  • Strong hands-on experience with Databricks.
  • Databricks certification (Associate or Professional level preferred).
  • Strong programming experience in Python and PySpark.
  • Advanced SQL skills and experience with relational databases.
  • Hands-on experience with Apache Spark and distributed data processing.
  • Experience with Delta Lake and Lakehouse architecture.
  • Strong knowledge of ETL/ELT development and data pipeline implementation.
  • Experience with Databricks Workflows, Jobs, and notebooks.
  • Knowledge of data modeling, data warehousing, and data integration.

Cloud Platform Experience

Experience with at least one of the following cloud platforms:

  • Microsoft Azure – Azure Databricks, ADLS Gen2, Azure Data Factory.
  • AWS – S3, Glue, EMR, and related data services.
  • Google Cloud Platform – Google Cloud Storage, BigQuery, and Dataproc.

Preferred Skills

  • Experience with Databricks Unity Catalog and Delta Live Tables.
  • Knowledge of Medallion Architecture (Bronze, Silver, and Gold layers).
  • Experience with Terraform, Git, Jenkins, and CI/CD pipelines.
  • Familiarity with Kafka or other real-time data streaming technologies.
  • Experience with data governance, lineage, and data quality frameworks.
  • Experience working in Agile/Scrum environments.

 

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 91141690
  • Position Id: 9142992
  • Posted 1 hour ago
Contact the job poster
JK

Jay Kumar

Recruiter @ SN Cloud Solutions

Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.