Data Engineer

Role: Data Engineer

Top Skills: Glue, Python, PySpark, Database is a MUST for them to hit the ground running

This position will focus on developing various database and data loading/transformation efforts. Initiatives could include multiple source files and/or files sourced both internally and externally. Candidate is required to interact with databases using AWS Lambda, AWS Glue (for transformations) Python and Spark. Possible work leveraging SnapLogic for ETL, depending upon use case. Work will be applicable to multiple applications. Knowledge of automated testing of data a plus.

Responsibilities:

Design and built systems for analytical reporting

Analyze data and design ETL frameworks using Python and Spark technologies

Collaborate with business partners to gather requirements and translate that to technical specificationsuse

Adhere to lean agile methodologies

Production Support including trouble shooting and problem-solving capabilities

Implement data pipelines to meet business requirements and adhere to best practices.

Required Skills:

Proficient in Python, Spark and database technologies

Experience with various AWS services like AWS Glue, IAM, RDS, S3, SQS, SNS, Lambdas, Cloud Formation, Cloud Watch, Event Bridge, CLI

Experience with Unix/Linux

Experience with SQL

Experience with source code control, preferably Git.

Experience in CI (Continuous Integration)/ CD (Continuous Delivery) software development pipeline stages like Commit, Build, Automated Tests, and Deploy

Desired Skills:

Experience in creating AWS Glue components - Jobs, Triggers, Workflows

Experience with AWS CLI

Experience working with databases such as Snowflake and/or Aurora

Experience with Hashicorp tool suite - Terraform, Vault

Experience with soap and rest services.

Experience working with XML, JSON, YAML

Experience with logging tools such as Cloudwatch and Splunk