Lead Data Engineer

Position: Lead Data Engineer

Location: Atlanta, GA (Hybrid)

Duration: 3 to 6 Months contract to hire 

Job Responsibilities

·       Design and implement scalable data frameworks to manage end-to-end data pipelines for workforce data analytics.

·       Develop secure, high-quality production code and data pipelines on Databricks, reviewing and debugging processes implemented by others.

·       Identify opportunities to eliminate or automate remediation of recurring issues to improve operational stability of software applications and systems.

·       Lead evaluation sessions with external vendors, startups, and internal teams to drive outcomes-oriented probing of architectural designs, technical credentials, and applicability for use within existing systems and information architecture.

·       Work with business stakeholders to understand requirements and design appropriate solutions, producing architecture and design artifacts for complex applications and implement the right ER Model with right fact and dimension tables.

·       Implement robust monitoring and alerting systems to proactively identify and address data ingestion issues, optimizing performance and throughput.

·       Implement data quality checks and validation processes to ensure accuracy and reliability of data.

Cloud Functions

·       Design and implement end-to-end data solutions using Databricks Pipelines and Azure Data Lake Storage (ADLS), Azure Databricks.

·       Develop, monitor, and maintain scalable data pipelines to support analytics and data science projects.

·       Understand and perform Data Integrity checks and Data Quality checks. Research and utilize Databricks test suites and make them absorbed into data eco system.

·       Utilize Azure Databricks to perform complex data transformation, aggregation, and machine learning tasks.

·       Assist with designing the architecture and workflow for an Enterprise Data Lakehouse in Databricks who wish to build their own Data Marts

Required Knowledge, Skills, & Abilities:

·       Understanding of reporting and/or visualization tools (e.g., Tableau, SSRS, SQL Server)

·       Knowledge of a variety of technologies, data models, and insights across all relevant data sources

·       Understanding of Governance principles

·       Understands concepts like data mining, extraction and analysis as it pertains to a specific bank pillar (e.g., Commercial, Retail, Wealth)

·       Analytical and critical thinking skills

·       Ability to quickly shift between technology stacks

·       Ability to mentor and train team members

·       Strong verbal and written communication skills