Principal Azure Data Engineer

Job Description:
  

Role Overview

We are seeking a highly experienced Azure Data Engineer to support and evolve a mission‑critical data platform within a regulated financial‑services environment. The successful candidate will have deep expertise in Azure Databricks, Azure Data Factory (ADF), SQL, Python, Dataflows, and emerging AI‑enabled analytics capabilities such as Databricks Genie.

This role requires a strong focus on data integrity, reconciliation, auditability, and operational stability, while enabling scalable analytics and AI use cases for business, finance, and advisory stakeholders. You will play a key role in designing and maintaining trusted data pipelines that support financial reporting, fee and revenue processing, analytics, and decision‑making.

Key Responsibilities

Enterprise Data Engineering

• Design, build, and maintain robust, production‑grade data pipelines on Azure using Databricks and Azure Data Factory (ADF).
• Develop scalable ETL / ELT processes using Python, SQL, and Spark, with a focus on correctness, repeatability, and performance.
• Implement ingestion frameworks for custodian feeds, vendor data, SFTP files, APIs, and internal source systems.
• Apply Delta Lake capabilities (ACID transactions, schema enforcement, time travel) to ensure financial data accuracy and audit readiness.

Databricks & Analytics Enablement

• Lead hands‑on development in Azure Databricks, including notebooks, workflows, job orchestration, and cluster optimization.
• Experience in Databricks Genie / AI‑assisted analytics for governed, business‑friendly insights while adhering to data‑security standards.
• Ensure platform reliability through performance tuning, monitoring, and cost controls.

Azure Orchestration & Integration

• Build and manage ADF pipelines with strong dependency handling, error management, alerting, and restartability.
• Develop and maintain Azure / Power Platform Dataflows to support downstream reporting and analytics use cases.
• Integrate with Azure Data Lake Storage, Key Vault, Azure AD, Power BI, and related Azure services.

Data Quality, Controls & Compliance

• Design and implement data validation, reconciliation, and exception‑handling frameworks, particularly for financial and transactional data.
• Support regulatory, audit, and compliance requirements through strong controls, logging, lineage, and traceability.
• Partner with risk, finance, and compliance teams to ensure data handling aligns with enterprise standards.

Leadership & Stakeholder Partnership

• Collaborate with finance, advisory, operations, and analytics teams to translate business requirements into scalable technical solutions.
• Contribute to platform architecture, roadmap planning, and long‑term modernization initiatives.

Required Qualifications

• 15+ years of experience in data engineering, data platforms, or analytics engineering, preferably within financial services.
• Deep hands‑on experience with:

• Azure Databricks (Spark, Delta Lake, Unity Catalog)
• Azure Data Factory (ADF)
• Advanced SQL (complex transformations, performance tuning)
• Python for data processing and orchestration

• Strong experience designing enterprise‑scale, highly controlled data pipelines.
• Solid understanding of financial data concepts, reconciliations, and reporting dependencies.
• Experience working in regulated or audit‑sensitive environments.
• Excellent analytical, problem‑solving, and communication skills.

Preferred Qualifications

• Experience with Databricks Genie, AI/BI, or generative‑AI‑enabled analytics in governed environments.
• Background in wealth management, asset management, broker‑dealer, or banking platforms.
• Experience supporting fee processing, revenue reporting, or custodian integrations.
• Familiarity with Power BI semantic models and enterprise reporting layers.
• Exposure to data governance frameworks, lineage tools, and metadata management.

What This Role Offers

• Ownership of core financial data pipelines supporting critical business functions.
• Opportunity to modernize analytics and AI capabilities while maintaining strict controls.
• A platform‑focused role with high visibility and long‑term impact.
• Collaboration with senior technology, finance, and business leadership