Overview
Remote
Depends on Experience
Contract - W2
Contract - 12 Month(s)
Skills
AWS
Lambda
Scala
Python
NoSQL
Job Details
Description:
Must Haves:
- 5-8 years of hands on experience with AWS (Kubernetes).
- 5-8 years of experience with data storage platforms (data lakes, data warehouses etc.) The candidate specifically needs to have experience with cloud storage service- S3.
- Strong experience developing and maintaining data pipelines supporting the ETL processes leveraging Python and Lambda.
- Must be well versed in file ingestion, extraction, storage and generation
- 5-8 years of hands on experience with Python coding. additional programming languages needed are Java and Scala
- Strong experience leveraging NoSQL and Postgress data management systems
- Strong understanding of data security best practices and compliance.
- Experience in regulated environments preferred
- Excellent ability to analyze data to identify patterns, trends and troubleshoot issues
- Experience working in a CI/CD environment using Git Hub Action
Plusses:
- Experience working in the financial sector
Day to Day Responsibilities:
- Designing and implementing scalable and secure data storage solutions in the cloud, ensuring optimal performance and accessibility
- Developing and maintaining robust data pipelines for the ingestion, transformation, and distribution of large datasets.
- Automating data processes and integrating third-party services
- Utilizing cloud services and tools to automate data workflows and streamline the data engineering process.
- Ensuring compliance with data governance and security policies, including data encryption and access controls.
- Monitoring cloud data systems' performance (Cloud Watch), identifying bottlenecks, and implementing improvements to enhance efficiency.
- Conducting data quality checks and implementing measures to ensure data accuracy and integrity.
- Optimizing data retrieval and developing APIs for data consumption by various enterprise consumers.
- Providing technical expertise and support for data-related issues, including troubleshooting and resolving data pipeline failures.
- Collaborating with IT and security teams to plan and execute disaster recovery strategies for cloud-based data systems.
- Documenting data engineering processes, creating data flow diagrams, and maintaining metadata for data lineage and cataloging.
- Collaborating with architects, analysts, and other engineers to support data modeling, analysis, and reporting needs.
- Staying current with emerging cloud technologies and data engineering practices to recommend and adopt innovations that improve data systems.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.