Senior Data Engineer

Overview

Remote
Depends on Experience
Full Time
No Travel Required

Skills

Amazon Redshift
Amazon Web Services
Analytical Skill
Analytics
Apache Flink
Apache Hadoop
Apache Kafka
Apache Spark
Big Data
Business Intelligence
Cloud Computing
Collaboration
Communication
Conflict Resolution
Data Engineering
Data Modeling
Data Processing
Data Quality
Database
Distributed Computing
Docker
ELT
Extract
Transform
Load
Good Clinical Practice
Google Cloud Platform
HIPAA
Java
Kubernetes
Leadership
Machine Learning (ML)
Machine Learning Operations (ML Ops)
Mentorship
Meta-data Management
Microsoft Azure
NoSQL
Open Source
Orchestration
Problem Solving
Python
Query Optimization
Real-time
Regulatory Compliance
SQL
Scala
Storage
Streaming
Team Leadership
Unstructured Data
Use Cases
Workflow

Job Details

Job Title: Senior Data Engineer (10+ Years Experience)

About the Role

We are seeking an experienced Senior Data Engineer with a strong track record of designing, building, and optimizing large-scale data platforms. The ideal candidate will have deep expertise in modern data architectures, distributed computing, and real-time data processing, as well as leadership experience guiding teams and projects from concept to production.

Key Responsibilities

  • Architecture & Design

    • Design and implement scalable, secure, and high-performance data pipelines and architectures.

    • Define best practices for data ingestion, storage, transformation, and delivery.

    • Build robust data models supporting analytics, machine learning, and business intelligence use cases.

  • Data Pipeline Development

    • Develop ETL/ELT processes for both batch and real-time data ingestion.

    • Integrate structured, semi-structured, and unstructured data from multiple sources.

    • Optimize existing data pipelines for speed, cost, and reliability.

  • Collaboration & Leadership

    • Partner with Data Scientists, Analysts, and Product teams to understand requirements and deliver data solutions.

    • Mentor junior engineers, providing technical guidance and fostering a culture of excellence.

    • Lead proof-of-concept and pilot projects for new data technologies.

  • Governance & Quality

    • Implement data quality, security, and compliance frameworks (e.g., GDPR, HIPAA).

    • Establish and maintain metadata management and data lineage practices.

Required Qualifications

  • Experience

    • 10+ years of hands-on data engineering experience.

    • Proven track record working with big data technologies such as Hadoop, Spark, Kafka, Flink, or equivalent.

    • Strong experience in cloud platforms (AWS, Azure, or Google Cloud Platform) with data services like AWS Glue, Redshift, BigQuery, or Azure Data Factory.

  • Technical Skills

    • Proficiency in Python, SQL, and Scala/Java.

    • Expertise in data modeling, query optimization, and database systems (SQL & NoSQL).

    • Experience with workflow orchestration tools (Airflow, Dagster, Prefect).

    • Solid understanding of streaming architectures and event-driven data systems.

  • Soft Skills

    • Strong problem-solving, analytical, and communication skills.

    • Ability to work cross-functionally and lead initiatives independently.

    • Demonstrated mentorship and team leadership capabilities.

Preferred Qualifications

  • Experience with data lakehouse architectures (Delta Lake, Iceberg, Hudi).

  • Knowledge of containerization and orchestration (Docker, Kubernetes).

  • Exposure to machine learning pipelines and MLOps.

  • Contributions to open-source data projects.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.