Sr. Data Engineer

Sr. Data Engineer 

Primary Responsibilities:

• Oversee daily operations and ensure efficiency of data analytics applications.
• Respond to and resolve operational issues and incidents promptly of data analytics applications.
• Recognize, diagnose, research, and resolve system and design concerns of data analytics applications.
• Document root cause analysis and track incidents to identify recurring problems.
• Responsible for development and providing development lifecycle support for any code changes or enhancements.
• Ensure appropriate Change Management processes/procedures and documentation are followed.
• Interface with external partners and assist them with testing and resolution of production issues.
• Help with system validation once the software has been developed.
• Regularly monitor and measure data analytics performance with a focus on minimizing job runtimes and cost.
• Perform data audits to ensure data accuracy and improve data quality.
• Identify potential risks and challenges by analyzing data trends.
• Ensure data privacy and security using industry best practices.
• Implement automated processes to monitor the data analytics system including data quality.
• Work closely with the project manager, customers, technical architect, functional, testing, admins and other development staff.
• Collaborate with data producers, data scientists, modelers and data consumers to understand their requirements and design innovative solutions to empower them.
• Design and develop new code, review existing code changes, and implement automated tests.
• Actively seek opportunities to continuously improve the technical quality and architecture to improve the product s business value.
• Operate the data pipelines in production including release management and production support.

Basic Qualifications:

• Bachelor s degree from an accredited college in a related discipline, or equivalent experience/combined education, with 8 years of professional experience, or 6 years of professional experience with a related master s degree.
• Experience working in software development lifecycle tools, with strong experience in ETL based development.
• Design, build, maintain and support ETL/ELT data pipelines using AWS Services and primarily on Glue studio, EC2, S3, DMS, DMS, DMS, CloudFormation, Step Functions, Lambda, Athena, DynamoDB, QuickSight, Event Bridge, SNS, VPC, CloudWatch and CloudTrail.
• Design and maintain data ingestion libraries written using PySpark/Python
• Experience working with relational databases like PostgreSQL, RDS, Big Data and Oracle
• Experience with using version control systems (Git Hub, CodeCommit, Bit Bucket)
• Experience with automation tools and scripting to streamline workflows
• Experience with using issue/problem tracking systems (Jira preferred)
• Working knowledge of distributed application environments.
• Ability to multi-task and work in a dynamic, fast-paced team environment with drive to completion.
• Experience with building customer relationships to prioritize and analyze system issues.
• Candidates must have lived in the USA for the past 3 consecutive years.
• Excellent oral and written communication skills.

Preferred Qualifications:

• Knowledge or experience with Software Engineering (IEEE) Capability Maturity Model (CMM) standards.
• Knowledge or experience with Full Development Lifecycle (FDLC) methodologies.
• Familiarity with SQL tools and knowledge of object-oriented programming analysis methods.
• Experience with data warehouse, data visualization, statistical techniques and predictive modeling.
• Experience with Agile methodologies and tools to manage iterative development processes.
• Ability to define and track key performance indicators to ensure project success and team productivity
• Experience with SSRS reports.
• Knowledge of Azure cloud platform