Overview
Remote
Hybrid
$70 - $80
Accepts corp to corp applications
Contract - W2
Contract - Independent
Contract - 12 Month(s)
Skills
AWS
Glue
EMR
Redshfit
Python
PySpark
Kakfka
Hive
ETL
SQL
Job Details
Below is the JD for Sr Cloud Engineer role.
Responsibilities:
- Collaborate as part of a development team to design and enhance large scale applications developed using Python, Spark.
- Evaluates and plans software designs, test results and technical manuals using AWS
- Confer with business units and development staff to understand both the business and technical requirements for producing technical solutions.
- Create and review technical and user-focused documentation for data solutions (data models, data dictionaries, business glossaries, process and data flows, architecture diagrams, etc.).
- Extend and enhance the business Data Lake
- Create or implement solutions for metadata management
- Solve for complex data integrations across multiple systems.
- Design and execute strategies for real-time data analysis and decisioning.
- Build robust data processing pipelines using AWS Services and integrate with multiple data sources
- Translating client user requirements into data flows, data mapping, etc.
- Analyses and determines data integration needs and follows Agile practices.
Required Skills:
- At least 5+ years of experience on designing and developing Data Pipelines for Data Ingestion or Transformation using Scala or Python
- At least 4 years of experience with Python, Spark.
- At least 3 years of experience working on AWS technologies.
- Experience of designing, building, and deploying production-level data pipelines using tools from AWS Glue, Lamda, Kinesis using databases Aurora and Redshift.
- Experience with Spark programming (pyspark or scala).
- Hands on experience with AWS components like (EMR, S3, Redshift, Lamdba, API Gateway, Kinesis ) in production environments
- Strong analytical skills and advanced SQL knowledge, indexing, query optimization techniques.
- Experience using ETL tools for data ingestion..
- Experience with Change Data Capture (CDC) technologies and relational databases such as MS SQL, Oracle and DB
- Ability to translate data needs into detailed functional and technical designs for development, testing and implementation
5+ Years of Development / Data Engineering Experience
4+ Years Developing with AWS Cloud solutions
4+ Years Developing in Spark/PySpark
4+ Years Developing AWS Glue ETL
4 + Years AWS Storage models (S3 and DynamoDB)
Some hands-on developing with on-prem ETL tooling (Ab Initio preferred, Informatica)
Additional Skills:
- Strong experience in AWS GLUE, EMR, Hudi, experience extracting data from multiple sources & loading data into Data Lakes, AWS Redshift
- Experience working with AWS Elastic search, RDS, PostgreSQL preferred
- Experience w/ AWS services such as Lambda, EMR, SNS/SQS, Event Bridge, Lake Formation & Athena
- Experience w/ integrating applications/systems (data producers) with Enterprise Kafka topics (Confluent Kafka integration w/ AWS S3, Redshift)
- Experience in Requirements Analysis, Data Analysis, Application Design, Application Development & Integration Testing
- Working knowledge of on-prem Extraction, Transformation, Cleansing and Loading methodology and principles
- Experience implementing and adding to DevOps principles (GitLab, maintaining CI/CD pipelines)
- Experience in Java svcs is nice to have