Lead Data Engineer
Remote
Fulltime
Job Summary
We are seeking an experienced, technically strong Lead Data Engineer to design, develop, implement, and support enterprise-scale data engineering platforms and integration solutions.
The ideal candidate will bring deep hands-on experience in Python, PySpark, SQL, Apache Spark, Databricks, Snowflake, ETL/ELT, and modern Data Lakehouse architectures, along with the ability to guide engineering teams and drive technical delivery.
The role involves close collaboration with Data Architects, Data Engineering teams, AI/ML teams, business stakeholders, and technology leadership to deliver scalable, reliable, secure, and high-performance data solutions.
Key Responsibilities
1. Data Engineering & Pipeline Development
- Design, develop, and maintain scalable batch and real-time data pipelines using Python, PySpark, SQL, Spark, and ETL/ELT technologies.
- Build and optimize data ingestion, transformation, integration, and processing workflows across enterprise systems.
- Develop and maintain enterprise data warehouses, data lakes, databases, tables, and data processing systems.
- Write complex SQL queries and optimize database and pipeline performance.
- Troubleshoot production data issues and improve pipeline reliability, efficiency, and quality.
2. Data Architecture & Platform Engineering
- Design and implement data structures, schemas, data models, and integration frameworks.
- Work with modern cloud data platforms including Databricks, Snowflake, OCI, and similar technologies.
- Build and support Delta Lake, Lakehouse, and enterprise data warehouse solutions.
- Apply data engineering best practices to ensure scalability, performance, maintainability, and reliability.
- Evaluate technical designs, identify potential issues, and recommend improvements.
3. Technical Leadership & Delivery
- Provide technical guidance to data engineers and collaborate with Data Architects and engineering teams.
- Translate business requirements into technical specifications, implementation plans, and test plans.
- Coordinate technical activities across multiple teams and enterprise platforms.
- Support implementation, testing, deployment, and production stabilization.
- Manage competing technical priorities and ensure timely delivery of engineering solutions.
4. Data Governance, Quality & Security
- Collaborate with data governance, data quality, and information security teams.
- Implement data quality checks, validation processes, and monitoring frameworks.
- Support metadata management, data modeling, and data security requirements.
- Ensure pipelines meet applicable governance, security, and certification standards before production deployment.
- Ensure compliance with HIPAA regulations and appropriate handling of Protected Health Information (PHI).
5. Analytics, AI & Data Science Enablement
- Work closely with analytics, AI/ML, and data science teams to develop reliable datasets and data processing solutions.
- Prepare structured and unstructured data for predictive and prescriptive analytics.
- Support data transformation and integration requirements for machine learning and AI applications.
- Identify opportunities to improve automation, data processing, and analytical capabilities.
6. DevOps & DataOps
- Apply Agile, DevOps, and DataOps principles to data engineering development and delivery.
- Improve deployment processes, pipeline monitoring, validation, and production reliability.
- Support automation of engineering workflows and continuous improvement initiatives.
Required Qualifications
- 12+ years of relevant IT/data engineering experience, including at least 3 years in data modeling, schema design, and building and maintaining data processing systems.
- Strong hands-on programming experience with Python, PySpark, SQL, and Spark.
- Extensive experience designing, developing, and optimizing ETL/ELT pipelines.
- Experience with Databricks, Snowflake, Delta Lake, data warehouses, or comparable enterprise data platforms.
- Strong knowledge of data modeling, database development, data transformation, and large-scale data processing.
- Experience troubleshooting end-to-end data pipelines and resolving production data issues.
- Understanding of data governance, data quality, metadata, and security controls.
- Ability to collaborate with engineering, architecture, analytics, and business teams.
- Strong analytical, problem-solving, communication, and technical leadership skills.
- Bachelor's degree in Computer Science, Information Technology, or a related discipline preferred.
