Overview
Remote
Accepts corp to corp applications
Contract - Independent
Contract - Full Time
Skills
Pipelines
Spark Pools
SQL Pools
Job Details
Contact Details:
1.Saravanan Ganesan
Email:
Cell:
Email:
Cell:
Job Title: Azure Data Engineer
Location: CA, Remote (PST)
Duration: Full Time
Years of Experience: 10+ Yrs.
Required Hours/Week: 40hrs./Week
Notes:
- Must Have: Azure Synapse, PySpark Notebook, Devops
- Candidate must work in PST time
Key Responsibilities:
- Design, develop, and maintain end-to-end data pipelines using Azure Synapse Pipelines and PySpark Notebooks.
- Work with Spark Pools for distributed processing and SQL Pools (Dedicated/Serverless) for transformation and querying.
- Ingest and transform large datasets from SQL databases, Blob/ADLS, REST APIs, and Cosmos DB.
- Author data flow activities, configure Linked Services, and manage Integration Runtimes for pipeline orchestration.
- Implement ETL/ELT transformations using PySpark, SQL, and Data Flows within Synapse.
- Optimize pipeline performance using partitioning, caching, and indexing strategies.
- Manage Synapse Workspace security (Managed Identities, Key Vault integration, Role-based Access Control).
- Monitor jobs using Synapse Monitor Hub, Log Analytics, and Kusto (KQL) queries for debugging and health checks.
- Collaborate with data modelers, BI, and analytics teams to ensure data accuracy, lineage, and consistency.
Required Skills:
- Proficiency in Azure Synapse Analytics Pipelines, Spark Pools, SQL Pools, and Notebooks.
- Strong PySpark programming skills for batch data processing and transformation.
- Deep understanding of Azure Data Lake (ADLS Gen2) and data lifecycle management.
- Hands-on with Synapse Data Flows, Triggers, Linked Services, and pipeline orchestration.
- Advanced SQL for data transformation and validation.
- Experience with Azure DevOps (Repos, CI/CD) for Synapse code deployment.
- Familiarity with performance tuning and cost optimization techniques in Synapse.
- Understanding of data security and governance concepts in Azure (Key Vault, Purview, RBAC).
Good to Have:
- Exposure to Power BI or downstream analytics models built on Synapse datasets.
- Knowledge of Delta format, parquet optimizations, and PolyBase external tables.
- Python scripting for automation of metadata or deployment tasks.
- Experience in data quality automation and lineage tracking using Purview or custom metadata stores.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.