Overview
Hybrid
Depends on Experience
Full Time
Accepts corp to corp applications
No Travel Required
Skills
Analytical Skill
Analytics
Apache Spark
Collaboration
Communication
Conflict Resolution
Continuous Delivery
Continuous Integration
Data Architecture
Data Engineering
Data Governance
Data Integration
Data Lake
Data Manipulation
Data Processing
Data Quality
Data Storage
Data Warehouse
Database
Decision-making
DevOps
ELT
Extract
Transform
Load
FOCUS
Git
Management
Microsoft
Microsoft Azure
Microsoft Power BI
Performance Tuning
Problem Solving
PySpark
Python
Real-time
SQL
Streaming
Version Control
Workflow
Job Details
Job Summary
We are seeking a skilled and experienced Data Engineer to join our team, focusing on the Microsoft data platform. The ideal candidate will have extensive hands-on experience in designing, building, and maintaining robust and scalable data pipelines using Microsoft Fabric, Azure Synapse Analytics, and Python. You will be responsible for orchestrating the flow of data from diverse sources into our analytical platform, enabling data-driven decision-making across the organization.
Key Responsibilities
- Pipeline Development: Design, develop, and maintain efficient ETL/ELT data pipelines using Microsoft Fabric and its components, including Fabric Pipelines and Spark jobs.
- Notebook-based Transformation: Write, test, and optimize Python notebooks (PySpark) within the Fabric/Synapse environment to perform complex data ingestion, cleansing, and transformation tasks.
- Data Warehouse and Lakehouse Management: Implement and manage data storage solutions, leveraging Azure Data Lake and Fabric's OneLake for a modern lakehouse architecture.
- Data Integration: Integrate data from a wide variety of sources, including transactional databases, APIs, and streaming platforms, into the centralized data platform.
- Performance Tuning: Optimize data pipelines and Spark workloads for performance and cost efficiency, especially when dealing with large-scale datasets in Synapse Analytics.
- Data Quality and Governance: Establish and enforce data quality and governance standards to ensure the accuracy, consistency, and reliability of data. Implement policies like row-level security and data classification.
- Collaboration: Work closely with data analysts, data scientists, and business stakeholders to understand data requirements and deliver high-quality data assets.
- Automation and CI/CD: Ensure monitoring, logging, and implement CI/CD automation for data workflows using tools like Azure DevOps.
Required Skills and Qualifications
- Proven experience as a Data Engineer with a focus on the Microsoft Azure ecosystem.
- Strong hands-on experience with Microsoft Fabric, specifically its data engineering workloads (Notebooks, Pipelines, and Lakehouses).
- Demonstrable expertise in Azure Synapse Analytics, including Synapse Notebooks (PySpark) and SQL pool resources.
- Proficiency in Python for data manipulation, transformation, and automation.
- Experience in building and maintaining ETL/ELT pipelines using Azure Data Factory and Fabric Pipelines.
- Solid understanding of modern data architecture concepts, including data lakehouses and medallion architecture.
- Strong SQL skills for querying and optimizing data workloads.
- Familiarity with data governance tools and concepts, such as Microsoft Purview.
- Experience with version control systems, such as Git.
- Excellent problem-solving, analytical, and communication skills.
Preferred Qualifications
- Experience with real-time or streaming data processing.
- Knowledge of Power BI for building datasets and reports.
- Relevant Microsoft certifications (e.g., Microsoft Certified: Fabric Data Engineer Associate).
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.