Lead Data Engineer

We are seeking a Lead Data Engineer with deep expertise in Databricks, PostgreSQL, and Liquibase to design, develop, and optimize scalable data platforms and ETL/ELT pipelines. The ideal candidate will lead data engineering initiatives, implement database version control, optimize data processing performance, and mentor engineering teams while delivering reliable, high-quality data solutions.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Build and optimize ETL/ELT workflows for large-scale structured and unstructured data.
• Design, develop, and optimize PostgreSQL databases, schemas, stored procedures, and queries.
• Manage database schema migrations and version control using Liquibase.
• Implement data quality, governance, validation, and monitoring frameworks.
• Optimize Spark jobs, SQL queries, and database performance for high-volume workloads.
• Collaborate with architects, analysts, and business stakeholders to define data requirements.
• Lead code reviews, establish engineering best practices, and mentor junior data engineers.
• Develop CI/CD pipelines for automated deployment of data applications and database changes.
• Ensure security, scalability, reliability, and compliance across the data platform.
• Troubleshoot production issues and drive continuous improvements.

Required Skills

• 8+ years of experience in Data Engineering.
• Strong hands-on experience with Databricks, Apache Spark, PySpark, and Spark SQL.
• Advanced experience with PostgreSQL database development, tuning, and optimization.
• Hands-on experience with Liquibase for database versioning and schema migrations.
• Strong SQL programming and data modeling skills.
• Experience building ETL/ELT pipelines using Python or Scala.
• Experience with cloud platforms such as Azure, AWS, or Google Cloud Platform.
• Knowledge of Delta Lake, Medallion Architecture, and data warehousing concepts.
• Experience with Git, CI/CD pipelines, and Agile development methodologies.
• Strong analytical, problem-solving, and communication skills.

Preferred Skills

• Delta Live Tables (DLT)
• Unity Catalog
• Airflow or Azure Data Factory
• Kafka or other streaming technologies
• Terraform or Infrastructure as Code
• Docker and Kubernetes
• Azure DevOps, GitHub Actions, or Jenkins
• Data governance and security best practices

Nice to Have

• Experience with Lakehouse architecture.
• Experience leading enterprise-scale data modernization projects.
• Exposure to real-time streaming and event-driven data pipelines.
• Databricks, Azure, or AWS certifications are a plus.

Primary Technologies:

• Databricks
• Apache Spark / PySpark
• PostgreSQL
• Liquibase
• Python
• SQL
• Delta Lake
• Git
• CI/CD
• Azure / AWS / Google Cloud Platform

This role is ideal for a technical leader who can architect modern data platforms, optimize enterprise data pipelines, and lead engineering teams in delivering scalable, high-performance data solutions.