Senior Data Engineer
Experience: 12+ Years
Employment Type: W2
Job Description
We are looking for an experienced Senior Data Engineer with 12+ years of strong experience in designing, developing, and implementing scalable data engineering solutions. The ideal candidate will have extensive hands-on experience with cloud data platforms, ETL/ELT pipelines, data warehousing, big data technologies, and modern data architecture.
The candidate will be responsible for building reliable and high-performance data pipelines, integrating data from multiple sources, developing cloud-based data solutions, and supporting enterprise-scale data platforms. The role requires strong programming, analytical, problem-solving, and communication skills.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines for large and complex datasets.
- Develop high-performance data processing solutions using Python, SQL, PySpark, and Apache Spark.
- Build and optimize data pipelines for both batch and real-time data processing.
- Work with cloud platforms such as AWS, Azure, or Google Cloud Platform to develop and manage modern data solutions.
- Design and implement data warehouses, data lakes, and lakehouse architectures.
- Develop data solutions using technologies such as Snowflake, Databricks, Redshift, BigQuery, or equivalent platforms.
- Implement data ingestion and transformation processes from databases, APIs, files, applications, and streaming sources.
- Work with Apache Kafka or similar technologies for real-time/event-driven data processing.
- Develop and manage workflows using Apache Airflow, AWS Glue, or other orchestration tools.
- Perform data modeling, data profiling, data validation, and data quality checks.
- Optimize SQL queries, Spark jobs, ETL processes, and data pipelines for performance and scalability.
- Implement monitoring, logging, error handling, and troubleshooting mechanisms for production data pipelines.
- Collaborate with Data Architects, Data Scientists, Business Analysts, Application Developers, and other stakeholders.
- Participate in requirements gathering, technical design, development, testing, deployment, and production support.
- Implement CI/CD practices for data engineering applications using Git, Jenkins, Azure DevOps, or similar tools.
- Follow data governance, security, privacy, and compliance standards while working with enterprise data.
- Troubleshoot production issues and provide root-cause analysis and permanent solutions.
- Mentor junior and mid-level Data Engineers and provide technical guidance on best practices.
Required Technical Skills
- 12+ years of experience in Data Engineering / Big Data / Data Platform development.
- Strong programming experience in Python.
- Advanced SQL skills with experience working on complex queries and large datasets.
- Strong hands-on experience with PySpark / Apache Spark.
- Extensive experience developing ETL/ELT pipelines.
- Strong experience with at least one major cloud platform: AWS, Azure, or Google Cloud Platform.
- Experience with modern data platforms such as Snowflake, Databricks, Redshift, or BigQuery.
- Experience with data orchestration tools such as Airflow, AWS Glue, Azure Data Factory, or equivalent.
- Experience with Kafka or other real-time streaming technologies is preferred.
- Strong understanding of data warehousing, data lakes, data modeling, and dimensional modeling.
- Experience with relational and NoSQL databases.
- Strong understanding of distributed computing and large-scale data processing.
- Experience with Git and CI/CD processes.
Preferred Skills
- Snowflake and DBT
- Databricks / Delta Lake
- AWS Glue, S3, Lambda, Redshift
- Azure Data Factory, ADLS, Synapse
- Google Cloud Platform BigQuery, Dataflow, Pub/Sub
- Apache Kafka / Kafka Streams
- Terraform / Infrastructure as Code
- Data governance and data quality frameworks
- Real-time and streaming data pipelines
- Experience with AI/ML data pipelines or GenAI data platforms
- Experience working in Agile/Scrum environments
Education
Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field is preferred.
Never miss an opportunity! Create an alert based on the job you applied for.
