Overview
Hybrid
Depends on Experience
Contract - Independent
Contract - W2
Contract - 12 Month(s)
Skills
Amazon Redshift
Collaboration
Communication
Conflict Resolution
Continuous Delivery
Continuous Integration
Amazon S3
Amazon Web Services
Analytical Skill
Apache Kafka
Cloud Computing
Data Governance
Data Processing
Data Quality
Data Warehouse
ELT
Real-time
SQL
Streaming
Job Details
We are seeking an experienced Data Engineer with strong expertise in Python, PySpark, AWS, Glue, Redshift, and Kafka. The ideal candidate will be responsible for designing and implementing large-scale data pipelines, ensuring data quality, and enabling real-time and batch data processing in a cloud environment. This is a highly technical role requiring strong problem-solving skills and the ability to collaborate with business and technology stakeholders.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Python, PySpark, and AWS Glue.
- Build and manage data warehouses and data lakes on AWS (Redshift, S3, Glue Catalog).
- Implement real-time streaming pipelines using Apache Kafka.
- Optimize performance of data pipelines and queries for large datasets.
- Work closely with analysts, data scientists, and business teams to deliver clean, reliable, and well-structured data.
- Ensure data governance, quality, and security standards are followed.
- Automate data workflows and monitoring to improve reliability and efficiency.
- Troubleshoot and resolve complex data pipeline issues in production.
Required Skills & Qualifications
- 5+ years of experience as a Data Engineer in cloud-based environments.
- Strong hands-on expertise with Python and PySpark for data transformation.
- Proven experience with AWS services (Glue, Redshift, S3, Lambda, EMR, IAM).
- Expertise in Kafka for real-time event streaming and processing.
- Solid knowledge of data warehousing concepts, schema design, and performance tuning.
- Strong experience with SQL (Redshift/PostgreSQL/MySQL).
- Knowledge of CI/CD pipelines and version control (Git, Jenkins, etc.).
- Excellent analytical and communication skills.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.