Overview
Remote
50 - 55
Contract - W2
Contract - 12 Month(s)
No Travel Required
Unable to Provide Sponsorship
Skills
Agile
ARM
Analytical Skill
Business Intelligence
Apache Parquet
Data Engineering
Data Quality
Data Warehouse
Databricks
DevOps
Data Security
Data Flow
Extract, Transform, Load
ELT
Continuous Integration
Git
Machine Learning (ML)
Microsoft Azure
Orchestration
Python
Terraform
Scala
Storage
Workflow
RBAC
Modeling
Data Lake
Caching
Data Bricks
Azure
Job Details
Key responsibilities
• Design, build, and maintain scalable batch and streaming data pipelines on Azure Databricks using Spark, Delta Lake, and related Azure services.
• Implement and manage data lake / lakehouse architectures integrating multiple enterprise data sources into Azure Data Lake Storage and Databricks.
• Develop robust ETL/ELT workflows, including ingestion, transformation, cleansing, and enrichment to produce analytics ready data models.
• Optimize Spark jobs and Databricks notebooks for performance, reliability, and cost, leveraging partitioning, caching, and efficient file formats such as Parquet/Delta.
• Implement data quality checks, validation rules, and monitoring to ensure accuracy, completeness, and timeliness of data.
• Apply data security and governance best practices, including RBAC, encryption, secrets management, and compliance controls on Azure and Databricks.
• Collaborate with BI and analytics teams to design curated datasets and semantic layers for tools like Power BI and other downstream consumers.
Implement observability, monitoring, and alerting for pipelines and clusters; troubleshoot incidents and drive root cause analysis.
• Contribute to CI/CD and Infrastructure as Code for data solutions using Azure DevOps, Git, and tools such as Terraform or ARM/Bicep.
• Document data flows, technical designs, standards, and best practices for Databricks based data engineering.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.