Sr Data Engineer - Palantry Foundry to Databricks Migration

Introduction:

We are seeking a skilled Sr. Data Engineer to lead the migration of our data infrastructure from Palantry Foundry to Databricks. The successful candidate will play a critical role in ensuring a smooth transition and optimizing the new platform for improved data processing and analysis.

Responsibilities:

• Lead the migration of data pipelines and workflows from Palantry Foundry to Databricks.
• Design and implement scalable data solutions using PySpark and Python on the Databricks platform.
• Collaborate with cross-functional teams to understand data requirements and develop solutions to meet business needs.
• Optimize data storage and processing on Databricks to improve performance and efficiency.
• Ensure data quality and integrity throughout the migration process.
• Provide technical expertise and guidance on best practices for data engineering on Databricks.
• Document processes, workflows, and configurations to support future maintenance and enhancements.

Requirements:

Required Skills:

• Strong experience with Palantry Foundry and Databricks.
• Proficiency in PySpark and Python for data processing and analytics.
• Hands-on experience with Azure cloud platform.

Preferred Skills:

• Previous experience in data migration projects.
• Familiarity with data modeling and ETL processes.
• Certifications in Databricks or related technologies.