ETL Tester

Job Summary

We are seeking a mid-level ETL Tester / Data QA Engineer with strong hands-on experience in Python-based automation, PySpark, and SQL to validate large-scale data pipelines. The ideal candidate has worked with modern cloud data platforms such as Azure, AWS, or Snowflake and is comfortable testing complex ETL/ELT workflows, data transformations, and incremental loads.

This role focuses on data quality, automation, and validation, working closely with Data Engineers, Analytics teams, and business stakeholders.

Key Responsibilities

• Design and execute ETL test strategies for batch and incremental data pipelines

• Validate source-to-target data transformations, aggregations, and business rules

• Write complex SQL queries to perform data reconciliation, count checks, and data comparisons

• Build and maintain automation frameworks using Python for data validation and regression testing

• Test PySpark-based pipelines running on Azure Databricks, AWS EMR, or similar platforms

• Validate data stored in Snowflake, Azure Synapse, Redshift, or BigQuery

• Perform data quality checks (nulls, duplicates, schema drift, data type mismatches)

• Collaborate with Data Engineers to debug data issues and optimize test coverage

• Generate test reports, defect documentation, and automation run results

Required Skills & Qualifications

• 4–7 years of experience in ETL / Data Warehouse / Big Data testing

• Strong proficiency in SQL (joins, CTEs, window functions, performance tuning)

• Strong Python experience for automation and data validation

• Hands-on experience with PySpark for validating large datasets

• Experience working in at least one cloud platform:

  • Azure (Databricks, ADLS, Synapse) OR

  • AWS (S3, Glue, EMR, Redshift) OR

  • Snowflake

• Experience testing batch and incremental ETL pipelines

• Solid understanding of data warehousing concepts (facts, dimensions, SCDs)

• Experience working with structured and semi-structured data (CSV, JSON, Parquet)

Nice-to-Have Skills

• Experience with data comparison tools (DataComPy, Great Expectations, custom frameworks)

• Familiarity with CI/CD pipelines for data testing

• Exposure to Robot Framework, pytest, or unittest

• Knowledge of Kafka / streaming data testing

• Experience in healthcare, finance, or large enterprise data platforms

Tools & Technologies

• Python, PySpark

• SQL (Snowflake SQL, T-SQL, PostgreSQL, etc.)

• Azure Databricks / AWS EMR / Snowflake

• ADLS / S3 / Cloud storage

• Git, CI/CD pipelines

• Excel / CSV / Parquet validation