6 results (1 new)
6 results (1 new)
Role: Data Engineer /Lead Data engineer (Full Time/FTE)
Location:Newark, NJ
Experience: 3+ Years
Work Model: Hybrid (3 days in a week from office)
Primary Skill: Synapse Data Engineering, Azur Data Factory, Python (PySpark),Delta Lake formats
Mandatory :Microsoft Fabric
Job Overview
We are seeking a Lead Data Engineer to help us architect and implement modern data solutions using the Microsoft Fabric ecosystem. You will be responsible for streamlining data workflows, implementing OneLake architectures, and enabling seamless data integration and analytics across the enterprise.
Key Responsibilities
Unified Storage: Implement and optimize OneLake structures to create a seamless, single source of truth for diverse data assets.
Data Engineering: Develop scalable ETL/ELT workflows using Synapse Data Engineering and Data Factory.
Performance Tuning: Optimize Fabric capacity and compute usage to ensure cost-efficient, high-speed data processing.
Pipelines & Ingestion: Build metadata-driven pipelines using Fabric Pipelines to ingest and process JSON, ZIP, XLSX, and CSV files effectively. Implement pattern-based incremental refreshes and SCD Type 1 & Type 2 logic.
Architecture & Performance: Design scalable Medallion Architectures (Bronze/Silver/Gold) across Fabric Lakehouses and Data Warehouses. Apply PySpark optimization (broadcast joins, partition strategy, data skew, memory tuning), SQL query tuning, and Delta Lake features (V-Order, Liquid Clustering, Time Travel).
Automation & Scripting: Write advanced Python and leverage Fabric Utils (notebookutils/mssparkutils) alongside Fabric REST APIs to automate inter-notebook workflows, dynamic parameterization, and workspace management.
Governance & Data Quality: Build automated Data Quality practices (schema enforcement, dynamic validation, reconciliation logging). Implement Fabric Governance frameworks, Microsoft Purview integration, data lineage, and security controls (RLS/CLS). Managing staged environments Dev. UAT and PROD with CICD Principals.
DevOps, Capacity & Analytics: Manage Fabric DevOps/ALM (Git integration and deployment pipelines) and monitor compute using the Fabric Capacity Metrics App to optimize CUs. Support basic Power BI semantic modeling (Direct Lake mode) and real-time ingestion via Eventhouse/KQL.
Collaboration: Partner with Data Scientists and BI teams to deliver structured, high-fidelity datasets for GenAI and advanced analytics.
Required Skills and Qualifications
Languages: Advanced proficiency in Python (PySpark) and Complex SQL.
Frameworks: Hands-on experience with Delta Lake formats, Medallion Architecture, and Spark-based processing.
Integrations: Working knowledge of Power BI (Direct Lake connectivity), Git-based CI/CD, and Fabric Capacity Management.
Orchestration: Proven ability to build and monitor pipelines in Data Factory and manage CI/CD via Azure DevOps.
Data Modeling: Good understanding of Star/Snowflake schemas and semantic modeling.
Certifications (Preferred): DP-600 (Fabric Analytics Engineer) or DP-203 (Azure Data Engineer), DP-700 (Fabric Data Engineer Associate).
The opportunity of setting up SG Analytics in 2007 came at the most unexpected moment. It arrived in the form of a phone call from a friend in the US who said he wanted to outsource some analytics-related work to a company. Since he had not been able to finalize any deal, he suggested that Sushant take up the work. A choice to meet one client’s investment research requirement has grown exponentially from there.
With our ability to continuously learn and evolve according to market needs, today, we are a 1600-plus global team that offers AI/ML-led data-centric insights and contextual analytics services.
To see how well you match this job, please log in or create an account.
Once logged in, be sure to complete your profile to get the most accurate match score.
It looks like there aren't any Similar Jobs for this job yet.
Search all similar jobs.png%3Fformat%3Dwebp&w=1080&q=75)
