Senior Data Engineer

Experience: 12+ Years
Employment Type: W2

Job Description

We are looking for an experienced Senior Data Engineer with 12+ years of strong experience in designing, developing, and implementing scalable data engineering solutions. The ideal candidate will have extensive hands-on experience with cloud data platforms, ETL/ELT pipelines, data warehousing, big data technologies, and modern data architecture.

The candidate will be responsible for building reliable and high-performance data pipelines, integrating data from multiple sources, developing cloud-based data solutions, and supporting enterprise-scale data platforms. The role requires strong programming, analytical, problem-solving, and communication skills.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines for large and complex datasets.
• Develop high-performance data processing solutions using Python, SQL, PySpark, and Apache Spark.
• Build and optimize data pipelines for both batch and real-time data processing.
• Work with cloud platforms such as AWS, Azure, or Google Cloud Platform to develop and manage modern data solutions.
• Design and implement data warehouses, data lakes, and lakehouse architectures.
• Develop data solutions using technologies such as Snowflake, Databricks, Redshift, BigQuery, or equivalent platforms.
• Implement data ingestion and transformation processes from databases, APIs, files, applications, and streaming sources.
• Work with Apache Kafka or similar technologies for real-time/event-driven data processing.
• Develop and manage workflows using Apache Airflow, AWS Glue, or other orchestration tools.
• Perform data modeling, data profiling, data validation, and data quality checks.
• Optimize SQL queries, Spark jobs, ETL processes, and data pipelines for performance and scalability.
• Implement monitoring, logging, error handling, and troubleshooting mechanisms for production data pipelines.
• Collaborate with Data Architects, Data Scientists, Business Analysts, Application Developers, and other stakeholders.
• Participate in requirements gathering, technical design, development, testing, deployment, and production support.
• Implement CI/CD practices for data engineering applications using Git, Jenkins, Azure DevOps, or similar tools.
• Follow data governance, security, privacy, and compliance standards while working with enterprise data.
• Troubleshoot production issues and provide root-cause analysis and permanent solutions.
• Mentor junior and mid-level Data Engineers and provide technical guidance on best practices.

Required Technical Skills

• 12+ years of experience in Data Engineering / Big Data / Data Platform development.
• Strong programming experience in Python.
• Advanced SQL skills with experience working on complex queries and large datasets.
• Strong hands-on experience with PySpark / Apache Spark.
• Extensive experience developing ETL/ELT pipelines.
• Strong experience with at least one major cloud platform: AWS, Azure, or Google Cloud Platform.
• Experience with modern data platforms such as Snowflake, Databricks, Redshift, or BigQuery.
• Experience with data orchestration tools such as Airflow, AWS Glue, Azure Data Factory, or equivalent.
• Experience with Kafka or other real-time streaming technologies is preferred.
• Strong understanding of data warehousing, data lakes, data modeling, and dimensional modeling.
• Experience with relational and NoSQL databases.
• Strong understanding of distributed computing and large-scale data processing.
• Experience with Git and CI/CD processes.

Preferred Skills

• Snowflake and DBT
• Databricks / Delta Lake
• AWS Glue, S3, Lambda, Redshift
• Azure Data Factory, ADLS, Synapse
• Google Cloud Platform BigQuery, Dataflow, Pub/Sub
• Apache Kafka / Kafka Streams
• Terraform / Infrastructure as Code
• Data governance and data quality frameworks
• Real-time and streaming data pipelines
• Experience with AI/ML data pipelines or GenAI data platforms
• Experience working in Agile/Scrum environments

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field is preferred.