Data Bricks/ Data Engineer

Key Responsibilities

• Validate batch and streaming data pipelines for correctness, completeness, consistency, and timeliness.
• Create and maintain data quality checks (nulls, duplicates, schema drift, referential integrity).
• Verify business rules and transformations using SQL‑based validations.
• Design, develop, and execute test strategies for Databricks-based data pipelines and analytics workflows
• Establish data reconciliation and end‑to‑end traceability between source and downstream systems.
• Test ETL/ELT pipelines built using AWS services (Glue, Lambda, EMR, Step Functions).
• Validate transformations written in SQL and Python.
• Ensure correctness across data ingestion, enrichment, aggregation, and publishing layers.
• Test reprocessing, backfills, and historical data loads.
• Validate ETL/ELT processes built using Apache Spark (PySpark/Scala) in Databricks
• Validate Kafka-based streaming pipelines for data integrity, ordering, and exactly‑once/at‑least‑once semantics.
• Test producer and consumer logic, serialization formats (Avro, JSON, Protobuf).
• Validate topic configurations, partitions, offsets, retention policies, and schema changes.
• Simulate and test late arrivals, duplicate events, and consumer failures.
• Test data workflows using AWS S3, Glue, Lambda, Redshift, Athena, Kinesis, DynamoDB, or similar services.
• Validate IAM roles, permissions, and secure data access.
• Verify data lifecycle policies, encryption, and storage optimizations.
• Build and maintain automated data testing frameworks using Python.
• Develop reusable test utilities, fixtures, and synthetic datasets.
• Integrate data tests into CI/CD pipelines for pre‑merge, scheduled, and post‑deployment validation.
• Enable automated alerts for data quality failures.
• Validate pipeline performance for large‑scale datasets.
• Test throughput, latency, and concurrency under peak workloads.
• Validate retry logic, error handling, idempotency, and recovery mechanisms.
• Perform soak, regression, and failover testing
• Validate data pipeline metrics, logs, and alerts using CloudWatch, Prometheus, Grafana, or equivalent tools.
• Partner with teams to define data SLAs and SLOs.
• Participate in incident response, root‑cause analysis, and postmortems related to data quality issues

• 7+ years of experience in QA, SDET, or Data Quality Engineering roles.
• 3 +Years of Experience in Data Bricks
• Strong hands‑on experience with SQL for complex data validation and analysis.
• Proficiency in Python for test automation and data validation.
• Experience testing data pipelines and ETL/ELT workflows.
• Hands‑on experience with Kafka or other streaming platforms.
• Solid understanding of AWS data services (S3, Glue, Redshift, Lambda, Athena, etc.).
• Experience working with large datasets and distributed systems.
• Strong debugging, analytical, and problem‑solving skills.