Senior DevOps Engineer with Databricks

Senior DevOps Engineer

Iselin, NJ (Need Onsite day 1, hybrid 3 days from office).

Contract- w2. H1b transfers are fine

Job Description:

Responsibilities:

• Design and implement scalable and secure cloud infrastructure using Azure services.
• Manage and optimize cloud resources to ensure high availability and performance.
• Monitor and troubleshoot cloud infrastructure and services.
• Develop and manage Databricks environments for data processing and analytics.
• Implement ETL pipelines using Databricks and integrate with other data sources.
• Optimize Databricks performance and cost.
• Write and maintain Terraform scripts to automate infrastructure provisioning.
• Implement infrastructure as code (IaC) practices to ensure reproducibility and consistency.
• Collaborate with development teams to integrate Terraform into CI/CD pipelines.
• Implement and manage CI/CD pipelines for automated deployments and testing.
• Use tools like Jenkins, GitHub Actions, or Azure DevOps for continuous integration and continuous deployment.
• Ensure security and compliance in DevOps workflows.
• Utilize Azure services such as Azure VMs, Azure Kubernetes Service (AKS), Azure SQL Database, Azure Blob Storage, etc.
• Implement and manage Azure Networking, including VNETs, VPNs, and ExpressRoute.
• Manage Azure Active Directory and role-based access control (RBAC).
• Work closely with cross-functional teams including developers, data scientists, and IT operations.
• Document infrastructure designs, configurations, and procedures.
• Provide training and support to team members on cloud technologies and best practices.

Requirements:

• 8+ years of experience in cloud engineering with a focus on Azure.
• Proven experience with Databricks and data engineering.
• Strong knowledge of Terraform and infrastructure as code principles.
• Hands-on experience with DevOps tools and practices.
• Experience in containerization and orchestration tools like Docker and Kubernetes.
• Proficiency in scripting languages such as Python, PowerShell, or Bash.
• Solid understanding of networking concepts and cloud security best practices.
• Excellent problem-solving skills and the ability to work in a fast-paced environment.

Preferred, but not required:

Certifications in Azure (e.g., Azure Solutions Architect, Azure DevOps Engineer).

Familiarity with Big Data technologies and frameworks.