Databricks Automation Engineer - W2 Position

Only on W2

The following requirement is open with our client. 

Client                                     : TCS

Title                                       : Databricks Automation Engineer

Location                               : Seattle WA/ Dallas, TX (Onsite)

Duration                              : 12 Months

Rate                                       : $53/hr on W2

Relevant Experience      :

Detailed Job Description:

We are looking for a Databricks Automation Engineer.

Job Responsibilities:

·         Define and own cross-team test architecture and frameworks for ETL/ELT, Databricks/Spark, APIs, and UIs that scale across multiple product teams and environments.

·         Design and implement comprehensive data validation and data-quality strategies for large-scale telemetry processing (schema validation, row counts, statistical checks, delta comparisons, anomaly detection, drift detection).

·         Architect and enforce CI/CD-integrated automated testing pipelines (Azure DevOps / GitHub Actions) supporting pre-deploy gating, contract testing, canary verification, and regression protection.

·         Lead synthetic test data generation strategies and robust mocking/stubbing approaches for

·         Downstream dependencies; ensure environment-aware and secure test harnesses for sensitive telemetry.

·         Drive performance, scalability, and distributed load testing for Spark jobs and supporting microservices, quantify bottlenecks and propose system- and code-level remediation.

·         Collaborate with data engineering, platform, DevOps, and product leadership to define acceptance criteria, release quality gates, and automated verification SLAs.

·         Implement and maintain observability for test executions; publish automation health metrics (passrates, flakiness, coverage, mean time to detection) and maintain executive-facing dashboards.

·         Own test documentation, run books, failure triage playbooks, and on-call handover for production incidents related to data correctness and test infrastructure.

·         Lead and grow a high-performing QA/test automation capability — hire, coach, establish standards, and promote best practices organization-wide

Must Have Skills:

·         8+ years of professional software engineering and testing experience, including technical ownership of QA/test practices across the SDLC.

·         3+ years leading or mentoring engineering/QA teams in technical areas (Test Lead, Principal QA Engineer, or similar).

·         Deep knowledge of data engineering and lake house patterns: partitioning strategies, ACID/Delta semantics, schema evolution, delta table optimization, CDC, time-travel, and data lineage; able to design verification approaches for Data bricks/Spark/Delta Lake systems.

·         Proven experience architecting and building scalable automated test frameworks and embedding them into enterprise CI/CD using Azure DevOps or GitHub Actions.

·         Strong software engineering skills with extensive coding experience in Python, Java, or Scala for test development and automation; ability to review and contribute production-grade code.

·         Extensive experience testing data pipelines and big-data processing (Databricks, Spark/PySpark, Delta Lake, ADLS Gen2) or equivalent streaming/batch platforms at scale.

Thanks and Regards, 

Goutham Eluri

Technical Recruiter 

ASCII Group LLC. 

38345 W. 10 Mile Rd, Ste.#365; Farmington, MI  48335 

Office:

Email:  Website: