AWS Databricks Consultant

Job Description:

• As a Software Engineer, the AWS Databrick Consultant will deliver high-quality, efficient technology solutions by developing new applications or supporting existing ones.
• The role focuses on AWS Databricks platform administration, performance optimization, security compliance, and integration with other systems.

Requirement/Must Have:

• Expertise in AWS and Databricks.
• Experience in Databricks workspace deployment and configuration.
• Familiarity with Databricks Runtime upgrades and compatibility testing.
• Ability to configure notebooks, Repos, and Delta Lake workflows.

Responsibilities:

• Deploy, configure, and maintain AWS Databricks workspaces.
• Candidateage clusters, pools, and jobs for performance and scalability.
• Implement IAM using Azure AD / Azure IAM / ClientIM.
• Set up RBAC, network security, and manage personal access tokens (PATs).
• Ensure compliance with data governance policies using Unity Catalog.
• Apply grants for catalogs and schemas.
• Monitor cluster performance and optimize for cost and speed.
• Fine-tune Spark jobs and use Databricks SQL for diagnostics.
• IntegXX Databricks with Snowflake, Teradata, Oracle, and others.
• Automate cluster management with Terraform.
• Implement CI/CD pipelines using GitHub.
• Troubleshoot job failures, crashes, and performance issues.
• Provide L2/L3 infrastructure support.
• CollaboXX with Databricks support for issue resolution.