Senior Data Engineer

Job Summary

We are seeking an experienced Senior Data Engineer with 12+ years of expertise in designing, developing, and maintaining scalable data platforms, data pipelines, and enterprise data solutions. The ideal candidate will have strong experience in cloud-based data architectures, ETL/ELT processes, data warehousing, big data technologies, and data governance. This role requires close collaboration with business stakeholders, data scientists, analysts, and engineering teams to deliver high-quality data solutions that support strategic business initiatives.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines for batch and real-time data processing.
• Build and maintain enterprise data warehouses, data lakes, and lakehouse architectures.
• Develop ETL/ELT workflows to ingest, transform, validate, and load data from multiple sources.
• Implement data integration solutions across on-premises and cloud environments.
• Collaborate with business analysts, product owners, and stakeholders to understand data requirements and translate them into technical solutions.
• Ensure data quality, integrity, security, and governance across data platforms.
• Optimize database performance, query execution, and data processing frameworks.
• Develop and maintain data models, metadata repositories, and data dictionaries.
• Support advanced analytics, machine learning, and AI initiatives by providing reliable data infrastructure.
• Monitor and troubleshoot data pipelines, ensuring high availability and performance.
• Implement CI/CD practices and automation for data engineering workflows.
• Mentor junior engineers and establish best practices for data engineering development.
• Participate in architecture reviews, technical design discussions, and strategic planning initiatives.

Required Qualifications

• Bachelor''s degree in Computer Science, Information Technology, Engineering, or a related field.
• 12+ years of experience in Data Engineering, Data Warehousing, or Big Data development.
• Strong experience with SQL and relational databases such as Oracle, SQL Server, PostgreSQL, or MySQL.
• Extensive experience building ETL/ELT pipelines using tools such as Informatica, Talend, DataStage, SSIS, or Matillion.
• Hands-on experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Experience with modern data warehousing solutions including Snowflake, Redshift, Synapse Analytics, or BigQuery.
• Strong programming skills in Python, Scala, Java, or Spark.
• Experience with Apache Spark, Hadoop ecosystem, Kafka, and distributed data processing frameworks.
• Expertise in data modeling techniques including dimensional modeling and Data Vault.
• Experience with workflow orchestration tools such as Airflow, Azure Data Factory, or AWS Glue.
• Strong understanding of data governance, data quality, and master data management concepts.
• Experience with Git, CI/CD pipelines, and DevOps practices.

Preferred Qualifications

• Experience with Databricks and Lakehouse architecture.
• Knowledge of machine learning data pipelines and MLOps concepts.
• Experience implementing real-time streaming solutions using Kafka, Spark Streaming, or Flink.
• Familiarity with containerization technologies such as Docker and Kubernetes.
• Cloud certifications such as AWS Certified Data Analytics, Azure Data Engineer Associate, or Google Professional Data Engineer.
• Experience working in Agile/Scrum environments.

Technical Skills

Programming Languages: Python, SQL, Scala, Java
Cloud Platforms: AWS, Azure, Google Cloud Platform
Data Warehousing: Snowflake, Redshift, Synapse, BigQuery
Big Data Technologies: Spark, Hadoop, Hive, Kafka, Flink
ETL/ELT Tools: Informatica, Talend, DataStage, SSIS, Matillion, dbt
Orchestration Tools: Airflow, Azure Data Factory, AWS Glue
Databases: Oracle, SQL Server, PostgreSQL, MySQL, MongoDB
Version Control: Git, GitHub, GitLab, Bitbucket
DevOps: Jenkins, Azure DevOps, Docker, Kubernetes

Soft Skills

• Strong analytical and problem-solving abilities.
• Excellent communication and stakeholder management skills.
• Ability to work independently and lead technical initiatives.
• Strong mentoring and leadership capabilities.
• Experience collaborating with cross-functional teams in fast-paced environments.

Nice to Have:

• Data Mesh and Data Fabric architecture experience.
• AI/ML data engineering experience.
• Experience with Generative AI data platforms and vector databases.
• Knowledge of regulatory compliance standards such as GDPR, HIPAA, or SOX.