Databricks Architect

Sensiple Inc is a New Jersey corporation with over two decades of expertise in technology-driven solutions specialising in Customer Experience, Contact Center Solutions, Digital Transformation, Cloud Computing & Independent Testing. With an expert team that has enriched experience in executing & developing sustainable IT strategies in Healthcare, Technology, Retail, Logistics, Education, Telecommunications, Government and Media, we help our diverse customers to envision the future. By developing highly scalable and consistent solutions, our primary goal is to deliver excellence at all levels and delight our customers and drive them to a better future. One of our client is looking for a Databricks Architect @St. Louis, MO
Please find below the details of the position.
Title: Databricks Architect
Location: St. Louis, MO
Job Type: 6 months Contract to hire
Experience: 8 12 years
Job Summary:
We are seeking a highly skilled Databricks Architect with hands-on experience in migrating Azure Data Factory (ADF) pipelines to Databricks, and a strong foundation in data governance and compliance. The ideal candidate will play a key role in modernizing our data platform, optimizing performance, and ensuring data integrity, security, and regulatory compliance.
Key Responsibilities : -

• Lead the migration of data pipelines and workflows from Azure Data Factory to Databricks, ensuring minimal downtime and performance optimization.
• Architect and implement scalable data processing frameworks using Apache Spark, PySpark, Delta Lake, and Databricks notebooks.
• Define and implement best practices for data governance, including data lineage, quality, cataloging, classification, and access control.
• Collaborate with stakeholders to understand data needs and translate them into efficient Databricks-based solutions.
• Optimize data pipelines for cost-efficiency, scalability, and performance.
• Integrate Databricks with Azure services (e.g., Azure Data Lake Storage, Synapse, Purview, Key Vault).
• Establish and maintain CI/CD pipelines, DevOps processes, and version control for Databricks environments.
• Work with data stewards, compliance teams, and security architects to align with data governance and regulatory standards (e.g., GDPR, HIPAA).
• Document architecture decisions, migration plans, and support knowledge transition.

Required Skills & Qualifications : -

• Bachelor s or Master s degree in Computer Science, Data Engineering, or related field.
• 8+ years of experience in data engineering/architecture, with 2+ years in Databricks.
• Proven experience in migrating ADF pipelines to Databricks, including orchestration, transformation, and dependency management.
• Strong expertise in Apache Spark, PySpark, SQL, and Delta Lake.
• Good understanding of data governance frameworks including metadata management, data cataloging, data masking, and RBAC.
• Experience with Azure data ecosystem ADF, ADLS Gen2, Synapse, Key Vault, and Azure DevOps.
• Familiarity with CI/CD practices in Databricks using tools like Databricks CLI, Repos, Terraform, or ARM templates.
• Experience with performance tuning and cost optimization in Databricks environments.
• Strong problem-solving skills, documentation discipline, and communication ability.

Preferred Skills : -

• Experience working with Unity Catalog in Databricks for fine-grained access control.
• Knowledge of data quality tools.
• Familiarity with enterprise data warehouse concepts and modern data stack.
• Certifications such as Databricks Certified Data Engineer, Azure Data Engineer Associate, or Azure Solutions Architect.
  
  Kindly send your updated resume and expected hourly rate to sugapriyan dot g at sensiple dot comRegards
  Sugapriyan | Sensiple Inc.,
  sugapriyan dot g at sensiple dot com
  555 US Highway 1 S, Ste 330 Iselin, NJ 08830