Senior Azure DataBricks Engineer at Dallas, TX onsite

Overview

On Site
Depends on Experience
Full Time

Skills

(Data Engineer or Data Analytics Engineer or BI Data Engineer or Azure Data Engineer or Azure Cloud Data Engineer)
ETL
Azure
(PowerBI or Power BI)
SQL
("DataBricks" or "Data Bricks")

Job Details

 

Roles & Responsibilities 1. Data Engineering & Pipeline Development

Automation Framework Development,Azure Data Bricks,Devops,PySpark,Terraform

  • Design, develop, and maintain scalable ETL/ELT pipelines using Apache Spark (PySpark/Scala) on Databricks.
  • Implement Delta Lake for data reliability, ACID transactions, and versioning.
  • Integrate data from various sources including cloud storage (e.g., GCS, S3), relational databases, APIs, and streaming platforms.
  1. Cloud Integration (Google Cloud Platform Focus)
  • Connect Databricks with Google Cloud Platform services like BigQuery, Cloud Storage, Pub/Sub, and Vertex AI.
  • Optimize data movement between Databricks and Google Cloud Platform using connectors and APIs.
  • Ensure seamless authentication and authorization using Google Cloud Platform IAM roles and Databricks workspace configurations.
  1. Performance Optimization
  • Tune Spark jobs for performance and cost-efficiency.
  • Implement best practices for partitioning, caching, and job scheduling.
  • Monitor cluster performance and resource utilization.
  1. Automation & CI/CD
  • Automate deployment of notebooks, jobs, and clusters using Terraform, GitHub Actions, or Azure DevOps.
  • Implement CI/CD pipelines for version control and continuous integration of Databricks assets.
  1. Security & Governance
  • Implement role-based access control (RBAC) and manage secrets securely.
  • Ensure data encryption at rest and in transit.
  • Collaborate with security teams to meet compliance and governance standards.

 

  1. Cross-Functional Collaboration
  • Work closely with data scientists, analysts, and business stakeholders to understand data requirements.
  • Translate business needs into technical solutions using Databricks and Google Cloud Platform.
  1. Mentorship & Leadership
  • Mentor junior developers and conduct code reviews.
  • Lead technical discussions and contribute to architectural decisions.
  • Promote best practices in coding, testing, and documentation.
  1. Innovation & Continuous Improvement
  • Stay updated with the latest features in Databricks and Google Cloud Platform.
  • Evaluate new tools and frameworks for potential adoption.
  • Continuously improve existing pipelines and workflows for efficiency and scalability.

 

 

Regards,

Deepak Kumar

Infinite | Exciting times...infinite possibilities…

Tel: +1- 

Email: 

 

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.