Data Quality Engineer

Location: New York, NY

Duration : Long Term

Position Overview

We are looking for an experienced Data Quality Engineer with strong expertise in cloud-based data platforms, data pipeline testing, API automation, performance testing, and database validation. The ideal candidate will be responsible for ensuring the quality, reliability, accuracy, and performance of large-scale data platforms and pipelines, with a strong focus on Google Cloud Platform and Python-based test automation.

Key Responsibilities

• Design and execute comprehensive data quality and testing strategies for cloud-based data platforms, pipelines, APIs, and applications.
• Validate end-to-end data pipelines, including data ingestion, transformation, processing, storage, and downstream consumption.
• Perform data validation across streaming and batch-processing pipelines, ensuring data accuracy, completeness, consistency, and integrity.
• Develop and maintain Python-based automation frameworks for backend, REST API, and data validation testing.
• Develop UI automation tests using Playwright and perform manual validation where required.
• Perform RESTful API testing, including functional, integration, negative, regression, and end-to-end validation.
• Design and execute performance, load, stress, latency, and throughput testing using tools such as k6.
• Validate data across relational, document, analytical, and columnar databases, including PostgreSQL/AlloyDB, MongoDB, BigQuery, and optionally ClickHouse.
• Work extensively within the Google Cloud Platform (Google Cloud Platform) ecosystem, including Pub/Sub, GCS, Dataflow, Dataproc, and Cloud Composer.
• Troubleshoot data discrepancies, pipeline failures, API issues, and performance bottlenecks, collaborating closely with Data Engineering, Development, DevOps, and Product teams.
• Build automated reconciliation and data-validation checks to identify missing, duplicate, inconsistent, incorrect, or delayed data.
• Contribute to CI/CD processes by integrating automated data, API, and application quality checks into deployment pipelines.
• Support data profiling, governance, cataloging, lineage, and automated data-quality controls, preferably using Dataplex.

Required Skills

• Strong experience as a Data Quality Engineer, SDET, Data Test Engineer, or QA Automation Engineer working with modern data platforms.
• Strong hands-on experience with Python-based test automation.
• Strong experience testing RESTful APIs and backend services.
• Experience with Playwright or similar UI automation frameworks.
• Hands-on experience with Google Cloud Platform data technologies, particularly Pub/Sub, GCS, Dataflow, Dataproc, Cloud Composer, and BigQuery.
• Strong SQL and data-validation skills with experience working across large and complex datasets.
• Experience with PostgreSQL/AlloyDB, MongoDB, and BigQuery.
• Hands-on performance testing experience using k6 or comparable modern load-testing tools.
• Strong understanding of ETL/ELT, batch and streaming pipelines, data reconciliation, schema validation, and data-quality principles.
• Strong analytical, debugging, problem-solving, and cross-functional communication skills.

Nice to Have

• Experience with ClickHouse or other high-performance columnar databases.
• Experience with Dataplex for data profiling, cataloging, governance, and automated data-quality controls.
• Experience integrating automated tests into CI/CD pipelines.
• Knowledge of data observability, data lineage, metadata management, and cloud-native monitoring