Senior Azure Data Engineer – Databricks & Microsoft Fabric
Employment Type: Full-Time
Job Title: Senior Azure Data Engineer
Experience Level: Senior-Level
Work Arrangement: [Onsite]
Location: [Houston TX]
Job Type: Full-Time, Permanent
Job Description
We are seeking an experienced Senior Azure Data Engineer with strong hands-on expertise in Azure Databricks, Microsoft Fabric, and modern cloud data engineering technologies. The ideal candidate will be responsible for designing, developing, and maintaining scalable enterprise data platforms, implementing robust data pipelines, and delivering governed, analytics-ready data solutions.
The candidate should have extensive experience with PySpark, Python, SQL, Delta Lake, Unity Catalog, Microsoft Fabric, Azure integration services, and DevOps automation. Experience integrating enterprise ERP and legacy systems is highly desirable.
Key Responsibilities
- Design, develop, and maintain scalable data engineering solutions using Azure Databricks and Microsoft Fabric.
- Implement data governance, security, and access controls using Unity Catalog.
- Develop and optimize data pipelines using PySpark, SQL, Python, Delta Lake, and Lakeflow Declarative Pipelines (formerly DLT).
- Build and manage lakehouse architectures using the Bronze, Silver, and Gold medallion framework.
- Implement Microsoft Fabric solutions leveraging OneLake, Lakehouses, Data Pipelines, and Dataflows.
- Develop enterprise data ingestion frameworks integrating REST APIs, ERP systems, and legacy data sources.
- Integrate enterprise applications and systems such as JD Edwards (JDE), DB2/IBM i, Oracle Primavera P6, and similar platforms.
- Implement Azure integration solutions using Durable Functions, Logic Apps, and Service Bus.
- Establish CI/CD pipelines and infrastructure automation using Azure DevOps or GitHub Actions.
- Implement Infrastructure as Code (IaC) using Terraform or Bicep.
- Develop and deploy Databricks Asset Bundles for automated application deployment and environment promotion.
- Collaborate with BI teams to deliver optimized Power BI semantic models, including Direct Lake architectures.
- Ensure data quality, reliability, performance optimization, security, and compliance across enterprise data platforms.
Required Technical Skills
Azure Databricks
- Hands-on experience with Azure Databricks, Unity Catalog, Delta Lake, and PySpark.
- Experience developing Lakeflow Declarative Pipelines (DLT).
- Knowledge of Databricks Asset Bundles and automated deployment strategies.
Microsoft Fabric
- Experience with Microsoft Fabric Lakehouse architecture and OneLake.
- Hands-on experience with Fabric Data Pipelines and Dataflows.
- Understanding of Fabric networking, including Managed Private Endpoints and VNet Gateways.
Programming & Data Engineering
- Strong SQL and Python programming skills.
- Experience building production-grade data pipelines.
- Knowledge of dbt is preferred.
- Strong understanding of dimensional modeling and medallion architecture.
Enterprise Data Integration
- Experience with REST API-based data ingestion.
- Familiarity with ERP and legacy systems such as JDE, DB2/IBM i, Oracle Primavera P6, or similar platforms.
- Experience with Azure Durable Functions, Logic Apps, and Service Bus.
DevOps & Infrastructure Automation
- Experience with Azure DevOps or GitHub Actions.
- Knowledge of Terraform or Bicep.
- Experience with service principals, workload identity federation, and secure authentication.
- Understanding of CI/CD, environment management, and deployment promotion strategies.
Power BI & Analytics
- Understanding of Power BI semantic models and Direct Lake.
- Experience with semantic layer governance and downstream BI integration.
- Exposure to Power Platform and Dataverse is a plus.
Preferred Qualifications
- Experience designing enterprise-scale cloud data platforms.
- Strong understanding of data architecture, governance, and security best practices.
- Experience migrating traditional data warehouse solutions to modern cloud lakehouse platforms.
- Excellent troubleshooting, communication, and problem-solving skills.
What We Are Looking For
We are looking for a hands-on data engineering professional who can take ownership of enterprise data solutions, work across modern Azure data technologies, and collaborate with architecture, engineering, and analytics teams to deliver scalable and reliable data platforms.
Employment Type: Full-Time