Senior Data Engineer

Position: Senior Data Engineer

Location: Indianapolis, IN (person to travel to Indianapolis, Indiana to meet with client when required)

Duration: 3 months Contract to hire

Looking for people who does not require visa sponsorship.

Key Responsibilities:

• Experience in designing and implementation of scalable data pipelines and lakehouse architectures using Databricks
• Experience in defining best practices for data ingestion, transformation, and storage
• Experience in architecting Databricks solutions on Azure, AWS
• Lead the deployment and configuration of Databricks including Unity Catalog
• Experience in optimizing Spark jobs and cluster performance
• Implement Delta Lake for reliable and performant data lakes
• Implement role-based access controls and audit mechanisms
• Evaluate emerging technologies and tools to enhance the Databricks ecosystem
• Continuously improve performance, cost-efficiency, and scalability of data solutions

Skills and Competencies

• Lakehouse Architecture Design
• Real-Time Media Content Analytics
• MLOps Pipeline Orchestration
• Delta Lake
• Unity Catalog
• MLflow
• Databricks SQL
• Apache Kafka
• Terraform

Required Qualifications:

• Bachelor s or Master s degree in Computer Science, Engineering, or related field.
• 7+ years of experience in data engineering or architecture roles.
• 3+ years of hands-on experience with Databricks and Apache Spark.
• Strong proficiency in Python, SQL, and Spark.
• Experience with Delta Lake, MLflow, and Unity Catalog.
• Familiarity with CI/CD pipelines and DevOps practices in data environments.
• Certifications in Databricks, Azure/AWS/Google Cloud Platform.
• Experience with real-time data processing (Kafka, Structured Streaming).
• Knowledge of machine learning workflows and MLOps.