Overview
Remote
$40 - $50
Contract - W2
Contract - 4 Month(s)
No Travel Required
Skills
UDP
Python
PySpark
Big Data
Extract
Transform
Load
Continuous Delivery
Continuous Integration
Amazon Web Services
Job Details
Job Title: Lead Data Engineer
Location: Remote
JobType: Contract
Duration: 4 Months
Interview process: 3 interviews
Responsibilities:
- Develop and enhance ETL pipelines used by UDP platform
- Contribute to the implementation of key requirements as defined in Phase 1 discovery.
- Automate implemented requirements as appropriate, including infrastructure as code and runbooks.
- Document processes and standard operating procedures.
- Ensure the platform meets defined requirements.
Skills/Technology Needed:
- Experienced with Python, PySpark and SQL
- Automation and CI/CD skills, unit and integration testing
- Familiarity with orchestration and ETL solutions
- Hands-on experience with AWS Services (Redshift, Glue, LakeFormation, Lambda)
- Understanding of AWS Least Privilege Security Model and AWS Security Model (IAM)
- Experience with Microsoft Office (Word, Excel, Powerpoint)
- Experience with handling data in Open Table formats (Iceberg, Parquet, Delta Format). Big Data Handling Understanding of such as Buckets and Partitioning
- (Lead) Understanding of PySpark, Glue job optimization techniques to improve operations
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.