Lead Data Engineer

Role: Lead Data Engineer

Location: Atlanta, GA (Hybrid)

Job Type: 3 to 6 Months contract to hire

Tax Term: W2



Exp: 11+ Years



Job Responsibilities

Design and implement scalable data frameworks to manage end-to-end data pipelines for workforce data analytics.

Develop secure, high-quality production code and data pipelines on Databricks, reviewing and debugging processes implemented by others.

Identify opportunities to eliminate or automate remediation of recurring issues to improve operational stability of software applications and systems.

Lead evaluation sessions with external vendors, startups, and internal teams to drive outcomes-oriented probing of architectural designs, technical credentials, and applicability for use within existing systems and information architecture.

Work with business stakeholders to understand requirements and design appropriate solutions, producing architecture and design artifacts for complex applications and implement the right ER Model with right fact and dimension tables.

Implement robust monitoring and alerting systems to proactively identify and address data ingestion issues, optimizing performance and throughput.

Implement data quality checks and validation processes to ensure accuracy and reliability of data.





Cloud Functions

Design and implement end-to-end data solutions using Databricks Pipelines and Azure Data Lake Storage (ADLS), Azure Databricks.

Develop, monitor, and maintain scalable data pipelines to support analytics and data science projects.

Understand and perform Data Integrity checks and Data Quality checks. Research and utilize Databricks test suites and make them absorbed into data eco system.

Utilize Azure Databricks to perform complex data transformation, aggregation, and machine learning tasks.

Assist with designing the architecture and workflow for an Enterprise Data Lakehouse in Databricks who wish to build their own Data Marts





Required Knowledge, Skills, & Abilities:

Understanding of reporting and/or visualization tools (e.g., Tableau, SSRS, SQL Server)

Knowledge of a variety of technologies, data models, and insights across all relevant data sources

Understanding of Governance principles

Understands concepts like data mining, extraction and analysis as it pertains to a specific bank pillar (e.g., Commercial, Retail, Wealth)

Analytical and critical thinking skills

Ability to quickly shift between technology stacks

Ability to mentor and train team members

Strong verbal and written communication skills