Senior Data Engineer – Job Description

Experience: 10+ Years
Job Type: W2 Contract Only

Job Summary

We are seeking a Senior Data Engineer with 10+ years of experience in designing, developing, and supporting scalable data solutions. The ideal candidate will have strong experience with Python, SQL, ETL/ELT, cloud platforms, data pipelines, data warehousing, and distributed data processing.

Required Qualifications & Technical Skills

• 10+ years of experience in Data Engineering, ETL, data integration, or related technologies.

• Strong hands-on experience with Python and advanced SQL.

• Experience designing and developing scalable ETL/ELT data pipelines.

• Strong understanding of data warehousing, data lakes, data modeling, and data integration concepts.

• Experience with Apache Spark / PySpark for large-scale data processing.

• Hands-on experience with Databricks or similar modern data platforms.

• Experience working with cloud platforms such as AWS, Azure, or Google Cloud Platform.

• Experience with relational databases such as SQL Server, Oracle, PostgreSQL, MySQL, Snowflake, or similar.

• Experience developing data ingestion and transformation processes from databases, APIs, files, and other sources.

• Strong knowledge of data quality, validation, reconciliation, and error handling.

• Experience with workflow orchestration tools such as Airflow, Azure Data Factory, AWS Glue, or similar.

• Experience with Git, CI/CD, and automated deployment processes.

• Strong troubleshooting and performance optimization skills.

• Experience working in Agile/Scrum environments.

Responsibilities

• Design, develop, and maintain enterprise-scale data pipelines and integration solutions.

• Develop complex data transformations using Python, SQL, Spark, and PySpark.

• Build data ingestion processes from structured and unstructured data sources.

• Design and maintain data warehouse and data lake solutions.

• Optimize SQL queries, Spark jobs, and data pipelines for performance and scalability.

• Implement data validation, quality checks, monitoring, and reconciliation processes.

• Develop and manage data workflows using orchestration tools.

• Work with cloud storage, databases, APIs, and enterprise data platforms.

• Support production data pipelines and troubleshoot data-processing issues.

• Implement CI/CD practices for data engineering code and deployment.

• Collaborate with data architects, analysts, developers, QA teams, and business stakeholders.

• Participate in technical design, code reviews, sprint planning, and other Agile activities.

• Document data flows, technical solutions, data mappings, and operational processes.

• Mentor junior and mid-level data engineers and contribute to data engineering best practices.

Preferred Skills

• Experience with Delta Lake / Lakehouse architecture.

• Knowledge of AWS S3, Azure Data Lake, ADLS, or Google Cloud Storage.

• Experience with Snowflake or other cloud data warehouses.

• Knowledge of Kafka, Event Hubs, or other streaming technologies.

• Experience with Terraform or Infrastructure as Code.

• Knowledge of Docker and Kubernetes.

• Experience with data governance, metadata, lineage, and security.

• Exposure to real-time and near-real-time data processing.

Domain Knowledge

• Strong understanding of enterprise data integration, reporting, analytics, and data warehouse environments.

• Experience working with large-volume transactional and analytical data.

• Understanding of business requirements, data mapping, data transformation, and downstream reporting needs.

• Experience working with cross-functional technical and business teams.