Senior Data Engineer

Overview

On Site
Depends on Experience
Full Time
No Travel Required

Skills

Amazon Web Services
Apache Airflow
Apache Spark
Cloud Computing
Collaboration
Continuous Delivery
Continuous Integration
Data Extraction
Data Integration
Data Manipulation
Data Modeling
Data Processing
Data Quality
Data Security
Data Visualization
Data Warehouse
Databricks
Extract
Transform
Load
Git
Good Clinical Practice
Google Cloud Platform
Microsoft Azure
Microsoft Power BI
Orchestration
Pandas
Performance Tuning
Privacy
PySpark
Python
ROOT
Regulatory Compliance
SQL
Tableau
Version Control
Workflow

Job Details

Work location: Bellevue WA, Atlanta GA, Dallas TX or Overland Park KS
Job Summary:
Should have extensive experience in building robust ETL pipelines, implementing efficient data ingestion strategies, and working with tools like Python, Spark, and Databricks.
Key Responsibilities:
- Design, develop, and maintain scalable ETL pipelines to ingest and transform data from multiple sources.
- Implement data integration strategies using Python, Spark, and Databricks to enable seamless data ingestion and transformation.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.
- Develop and optimize data workflows to improve data quality, consistency, and performance.
- Ensure data security, governance, and compliance best practices are followed.
- Conduct performance tuning of Spark jobs and optimize data queries to minimize latency and improve efficiency.
- Automate data pipelines using CI/CD tools and ensure continuous data delivery.
- Troubleshoot data issues, identify root causes, and implement effective solutions.
Required Qualifications:
- Strong proficiency in Python, with experience in data manipulation libraries like Pandas and PySpark.
- Extensive experience with Apache Spark for data processing and analysis.
- Hands-on experience with Databricks for building scalable data pipelines.
- Experience with data lakes, data warehouses, and cloud platforms like AWS, Azure, or Google Cloud Platform.
- Strong SQL skills for data extraction, transformation, and querying.
- Familiarity with version control systems (e.g., Git) and CI/CD processes.
Preferred Skills:
- Experience with workflow orchestration tools like Apache Airflow, Dagster, or Prefect.
- Knowledge of data modeling techniques and best practices.
- Familiarity with data visualization tools such as Power BI, Tableau.
- Understanding of data privacy regulations and security protocols."

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.