Lead Data Engineer / Architect

* NO C2C allowed - W2 only

10+ years of experience as a Data Engineer and 5+ years as lead

• Hands-on experience with Azure Databricks, Spark, and Python

• Experience with Delta Live Tables (DLT) or Databricks SQL

• Strong SQL and database background

• Experience with Azure Functions, messaging services, or orchestration tools

• Familiarity with data governance, lineage, or cataloging tools (e.g., Purview, Unity Catalog)

• Experience monitoring and optimizing Databricks clusters or workflows

• Experience working with Azure cloud data services and understanding how they integrate with Databricks and enterprise data platforms

• Experience with Terraform for cloud infrastructure provisioning

• Experience with GitHub and GitHub Actions for version control and CI/CD automation

• Strong understanding of distributed computing concepts (partitions, joins, shuffles, cluster behavior)

• Familiarity with SDLC and modern engineering practices

• Ability to balance multiple priorities, work independently, and stay organized

Key Responsibilities

• Analyze, design, and develop enterprise data solutions with a focus on Azure, Databricks, Spark, Python, and SQL

• Develop, optimize, and maintain Spark/PySpark data pipelines, including managing performance issues such as data skew, partitioning, caching, and shuffle optimization

• Build and support Delta Lake tables and data models for analytical and operational use cases

• Apply reusable design patterns, data standards, and architecture guidelines across the enterprise, including collaboration with 84.51° when needed

• Use Terraform to provision and manage cloud and Databricks resources, supporting Infrastructure as Code (IaC) practices

• Implement and maintain CI/CD workflows using GitHub and GitHub Actions for source control, testing, and pipeline deployment

• Manage Git-based workflows for Databricks notebooks, jobs, and data engineering artifacts

• Troubleshoot failures and improve reliability across Databricks jobs, clusters, and data pipelines

• Apply cloud computing skills to deploy fixes, upgrades, and enhancements in Azure environments

• Work closely with engineering teams to enhance tools, systems, development processes, and data security

• Participate in the development and communication of data strategy, standards, and roadmaps

• Draft architectural diagrams, interface specifications, and other design documents

• Promote the reuse of data assets and contribute to enterprise data catalog practices

• Deliver timely and effective support and communication to stakeholders and end users

• Mentor team members on data engineering principles, best practices, and emerging technologies