Sr. Data Engineer

Role Description:

The Senior Data Engineer will play a pivotal role in designing, architecting, and optimizing cloud-native data integration and Lakehouse solutions on Azure, with a strong emphasis on Microsoft Fabric adoption, PySpark/Spark-based transformations, and orchestrated pipelines. This role will lead end-to-end data engineering from ingestion through APIs and Azure services to curated Lakehouse/warehouse layers while ensuring scalable, secure, well-governed, and well-documented data products. The ideal candidate is hands-on in delivery and also brings data architecture knowledge to help shape patterns, standards, and solution designs.

Key Responsibilities

• Design and implement end-to-end data pipelines and ELT/ETL workflows using Azure Data Factory (ADF), Synapse, and Microsoft Fabric.
• Build and optimize PySpark/Spark transformations for large-scale processing, applying best practices for performance tuning (partitioning, joins, file sizing, incremental loads).
• Develop and maintain API-heavy ingestion patterns, including REST/SOAP integrations, authentication/authorization handling, throttling, retries, and robust error handling.
• Architect scalable ingestion, transformation, and serving solutions using Azure Data Lake / OneLake, Lakehouse patterns (Bronze/Silver/Gold), and data warehouse modeling practices.
• Partner with stakeholders to define solution approaches and provide architecture input (data flow design, integration patterns, target-state roadmap, and platform best practices).
• Implement monitoring, logging, alerting, and operational runbooks for production pipelines; support incident triage and root-cause analysis.
• Apply governance and security practices across the lifecycle, including access controls, data quality checks, lineage, and compliance requirements.
• Write complex SQL, develop data models, and enable downstream consumption through analytics tools and curated datasets.
• Drive engineering standards: reusable patterns, code reviews, documentation, source control, and CI/CD practices.

Requirements:

• Bachelor's degree (or equivalent experience) in Computer Science, Engineering, or a related field.
• 5+ years of experience in data engineering with strong focus on Azure Cloud.
• Strong experience with Azure Data Factory pipelines, orchestration patterns, parameterization, and production support.
• Strong hands-on experience with Synapse (pipelines, SQL pools and/or Spark), and modern cloud data platform patterns.
• Advanced PySpark/Spark experience for complex transformations and performance optimization.
• Heavy experience with API-based integrations (building ingestion frameworks, handling auth, pagination, retries, rate limits, and resiliency).
• Strong knowledge of SQL and data warehousing concepts (dimensional modeling, incremental processing, data quality validation).
• Strong understanding of cloud data architectures including Data Lake, Lakehouse, and Data Warehouse patterns.
• Demonstrated ability to troubleshoot and optimize complex data workflows in production environments.
• Excellent communication and collaboration skills.
  
  
  Preferred Skills
• Experience with Microsoft Fabric (Lakehouse/Warehouse/OneLake, Pipelines, Dataflows Gen2, notebooks).
• Architecture experience (formal or informal), such as contributing to solution designs, reference architectures, integration standards, and platform governance.
• Experience with DevOps/CI-CD for data engineering using Azure DevOps or GitHub (deployment patterns, code promotion, testing).
• Experience with Power BI and semantic model considerations for Lakehouse/warehouse-backed reporting.
• Familiarity with data catalog/governance tooling (e.g., Microsoft Purview).