Data Engineer with Data Bricks (F2F) Need Locals - Dallas TX
Contract Independent
Contract W2
12 Months
No Travel Required
On-site
$50 - $55/hr
TechCafeHub LLC
Fitment
Dice Job Match Score™
👾 Reticulating splines...
Job Details
Skills
- Amazon DynamoDB
- Amazon Kinesis
- Amazon S3
- Amazon Web Services
- Analytics
- Continuous Integration
- Data Integrity
- Data Quality
- Data Validation
- Databricks
- Business Rules
- Electronic Health Record (EHR)
- Encryption
- Extract, Transform, Load
- Concurrent Computing
- Amazon Redshift
- Analytical Skill
- Apache Avro
- Incident Management
- Optimization
- ELT
- Python
- Root Cause Analysis
- Workflow
Summary
- Validate batch and streaming data pipelines for correctness, completeness, consistency, and timeliness.
- Create and maintain data quality checks (nulls, duplicates, schema drift, referential integrity).
- Verify business rules and transformations using SQL-based validations.
- Design, develop, and execute test strategies for Databricks-based data pipelines and analytics workflows
- Establish data reconciliation and end-to-end traceability between source and downstream systems.
- Test ETL/ELT pipelines built using AWS services (Glue, Lambda, EMR, Step Functions).
- Validate transformations written in SQL and Python.
- Ensure correctness across data ingestion, enrichment, aggregation, and publishing layers.
- Test reprocessing, backfills, and historical data loads.
- Validate ETL/ELT processes built using Apache Spark (PySpark/Scala) in Databricks
- Validate Kafka-based streaming pipelines for data integrity, ordering, and exactly-once/at-least-once semantics.
- Test producer and consumer logic, serialization formats (Avro, JSON, Protobuf).
- Validate topic configurations, partitions, offsets, retention policies, and schema changes.
- Simulate and test late arrivals, duplicate events, and consumer failures.
- Test data workflows using AWS S3, Glue, Lambda, Redshift, Athena, Kinesis, DynamoDB, or similar services.
- Validate IAM roles, permissions, and secure data access.
- Verify data lifecycle policies, encryption, and storage optimization.
- Build and maintain automated data testing frameworks using Python.
- Develop reusable test utilities, fixtures, and synthetic datasets.
- Integrate data tests into CI/CD pipelines for pre-merge, scheduled, and post-deployment validation.
- Enable automated alerts for data quality failures.
- Validate pipeline performance for large-scale datasets.
- Test throughput, latency, and concurrency under peak workloads.
- Validate retry logic, error handling, idempotency, and recovery mechanisms.
- Perform soak, regression, and failover testing
- Validate data pipeline metrics, logs, and alerts using CloudWatch, Prometheus, Grafana, or equivalent tools.
- Partner with teams to define data SLAs and SLOs.
- Participate in incident response, root-cause analysis, and postmortems related to data quality issues
Required Qualifications
- 7+ years of experience in QA, SDET, or Data Quality Engineering roles.
- Strong hands-on experience with SQL for complex data validation and analysis.
- Proficiency in Python for test automation and data validation.
- Experience testing data pipelines and ETL/ELT workflows.
- Hands-on experience with Kafka or other streaming platforms.
- Solid understanding of AWS data services (S3, Glue, Redshift, Lambda, Athena, etc.).
- Experience working with large datasets and distributed systems.
- Strong debugging, analytical, and problem-solving skills.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 91172926
- Position Id: 9040242
- Posted 3 hours ago
Company Info
About TechCafeHub LLC
Welcome to TechCafeHub , we are one stop shop for all your IT needs , whether you need any Software Solution (websites, mobile apps, applications, etc.) for your business needs, you want to start your journey in IT (get trained and placed in leading IT industries), enhance /update your already existing applications, increase your Cloud Footprint , we are here for all of it
Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.Similar Jobs
It looks like there aren't any Similar Jobs for this job yet.
Search all similar jobs