Senior Data Engineer(Databricks)

Job Description:

• Design, develop, and maintain scalable data pipelines and ETL processes using Python, Databricks, and AWS Glue.
• Build and optimize data warehouses and data lakes on PySpark and AWS platforms.
• Implement data ingestion, transformation, and orchestration workflows using AWS Step Functions, Lambda, and Glue Jobs.
• Monitor, troubleshoot, and optimize pipeline performance using AWS CloudWatch and other monitoring tools.
• Collaborate with data analysts, data scientists, and business stakeholders to define data requirements and deliver reliable datasets.
• Ensure data quality, governance, and security across all data systems and workflows.
• Automate repetitive data engineering tasks and contribute to building reusable frameworks and templates.
• Support continuous improvement by adopting best practices for CI/CD, version control, and infrastructure-as-code (IaC).