Senior Data Engineer : W2 Position

  • Indianapolis, IN
  • Posted 6 hours ago | Updated 6 hours ago

Overview

Hybrid
Depends on Experience
Contract - W2
Contract - 12 Month(s)
10% Travel
Unable to Provide Sponsorship

Skills

Senior Data Engineer
Machine Learning Operations (ML Ops)
Machine Learning (ML)
Google Cloud Platform
Good Clinical Practice
Python
SQL
Microsoft Azure
Data Engineering
Databricks
DevOps
Data Processing
Continuous Delivery
Continuous Integration
Amazon Web Services
Apache Spark
Auditing

Job Details

Senior Data Engineer

Introduction:
As a Senior Data Engineer, you will be responsible for designing and implementing scalable data pipelines and lakehouse architectures using Databricks. You will lead the deployment and configuration of Databricks solutions on Azure, AWS, and optimize Spark jobs for efficient performance. Your role will also involve implementing Delta Lake, role-based access controls, and audit mechanisms to ensure reliable and performant data lakes. Additionally, you will evaluate emerging technologies and tools to enhance the Databricks ecosystem and continuously improve the performance, cost-efficiency, and scalability of data solutions.

Responsibilities:
- Design and implement scalable data pipelines and lakehouse architectures using Databricks
- Define best practices for data ingestion, transformation, and storage
- Architect Databricks solutions on Azure, AWS
- Lead deployment and configuration of Databricks including Unity Catalog
- Optimize Spark jobs and cluster performance
- Implement Delta Lake for reliable and performant data lakes
- Implement role-based access controls and audit mechanisms
- Evaluate emerging technologies and tools to enhance the Databricks ecosystem
- Continuously improve performance, cost-efficiency, and scalability of data solutions

Requirements:
Required Qualifications:
- Bachelor’s or Master’s degree in Computer Science, Engineering, or related field
- 7+ years of experience in data engineering or architecture roles
- 3+ years of hands-on experience with Databricks and Apache Spark
- Strong proficiency in Python, SQL, and Spark
- Experience with Delta Lake, MLflow, and Unity Catalog
- Familiarity with CI/CD pipelines and DevOps practices in data environments
- Certifications in Databricks, Azure/AWS/Google Cloud Platform
- Experience with real-time data processing (Kafka, Structured Streaming)
- Knowledge of machine learning workflows and MLOps

Preferred Skills:
- Lakehouse Architecture Design
- Real-Time Media Content Analytics
- MLOps Pipeline Orchestration
- Delta Lake
- Unity Catalog
- MLflow
- Databricks SQL
- Apache Kafka
- Terraform

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.