Sr. Data QA Engineering Lead

Apache Hadoop, Apache Hive, Apache Kafka, Apache Pig, Apache Spark, Automated testing, Automation, Big data, Python, SQL, Scala, Scripting, Snow flake schema, Java, QA
Contract W2, Contract Independent, 12 Months
Depends on Experience

Job Description

Sr. Big Data Quality/Test Engineering Lead – 10 years + Experience: 12-18 Months Contract

Looking for a Lead Big Data Engineer with strong experience in Big Data Testing and Data Quality Engineering. Someone who can see the entire testing and quality environment, knowing where to look for failure points and find blind spots that are ignored.

Most important aspect of this role is communication and organizational skills. Must be able to recognize a situation, distill component parts and report that back to Mgr. what she/he needs to know is how to make the right decisions.

Ideally looking for experience leading projects, who have an automation focus, must have strong experience in the big data space.   Must be a critical thinker, and have done heavy deep diving into the data themselves.  Must have strong understanding or Quality and Engineering and have experience translating requirements, and have conducted project work based upon metrics.

Must be able to translate work into Quality Automation, understanding of automation such as Talend and Informatica but experienced with calculations in SQL so not dependent on tools such as informatica. Must know how to establish quality metrics, define requirements and certify.   Gathering requirements and must be capable of extracting information from the users, and extract the requirements from user, trust that data coming in that is good enough to generate the appropriate reports.

  • Contribute to the design, architecture and implementation of a data quality framework
  • Implement data catalog solutions for data systems and services
  • Implement data quality and metadata management solutions using industry standard practices as well as custom solutions built in house
  • Provide quality and metadata management best practices for data analysts
  • Document, automate and enforce data quality
  • Identify any data quality issues and solve the issues in both a proactive and reactive way
  • Collaborate with subject matter experts across different business units to deliver trusted data to our users


  • Extreme attention to detail with excellent problem solving, and communication skills
  • Expert knowledge of data quality architecture with open-minded approach
  • Independent, energetic, highly technical, and strive for results under pressure and ambiguity
  • Strong analytical skills - ability to read/write complex SQL queries to analyze data, investigate systemic defects and validate data quality
  • Strong programming with Scala/Java
  • Hands-on knowledge and implementation experience on data quality practices, such as profiling, scorecards and dashboards, KPIs
  • Hands-on knowledge and implementation experience of metadata managements solutions and practices
  • Experienced working in a Big Data environment (Cloudera, Hortonworks, Hadoop/Spark) using batch and real-time processing
Dice Id : 10115346
Position Id : 5488
Originally Posted : 2 months ago
Have a Job? Post it