Data Engineer Databricks & AWS (Only W2)

Overview

On Site
Depends on Experience
Contract - W2
Contract - 2 Month(s)

Skills

Data Pipelines
PySpark
Data Governance
Spark
Kafka
APIs
Medallion

Job Details

Key Responsibilities
Build and Maintain Data Pipelines: Develop scalable data pipelines using PySpark and Spark within the Databricks environment.
Implement Medallion Architecture: Design workflows using raw, trusted, and refined layers to drive reliable data processing.
Integrate Diverse Data Sources: Connect data from Kafka streams, extract channels, and APIs.
Data Cataloging & Governance: Model and register datasets in enterprise data catalogs, ensuring robust governance and accessibility.
Access Control: Manage secure, role-based access patterns to support analytics, AI, and ML needs.
Team Collaboration: Work closely with peers to achieve required code coverage and deliver high-quality, well-tested solutions. Required Skills & Experience
Databricks: Expert-level proficiency
PySpark/Spark: Advanced hands-on experience
AWS: Strong competency, including S3 and Terraform for infrastructure-as-code
Data Architecture: Solid knowledge of the medallion pattern and data warehousing best practices
Data Pipelines: Proven ability to build, optimize, and govern enterprise data pipelines

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.