Location: United States Experience: 8+ years Job Type: Permanent
### Top Skills & Technologies AWS | Redshift | Airflow | Kafka | SQL | Python | Data Modeling | Data Quality
### Job Summary Thinking Minds' client is seeking a senior data engineer to own the data foundation supporting product analytics, personalization, and AI initiatives. This role spans application instrumentation, streaming and batch ingestion, warehouse architecture, and production ML data infrastructure. Working with product engineering, data science, ML engineering, analytics, and experimentation teams, you will deliver reliable datasets for business decisions and model training and inference. This is an in-office opportunity with relocation assistance available.
### Ideal Candidate Should Have
Strong ownership of production data platforms, sound architectural judgment, and an ability to translate ambiguous needs into practical designs.
Clear communication, collaborative technical leadership, and a commitment to mentoring, documentation, and engineering quality.
Experience connecting event data, analytics, and production ML workflows while balancing reliability, performance, and cost.
### Key Responsibilities
Build scalable pipelines on AWS using Airflow (MWAA), Glue, Kafka, S3, and Redshift; evolve event schemas, taxonomy, versioning, and migrations.
Establish Data Quality and observability through schema validation, instrumentation testing, anomaly detection, freshness monitoring, completeness checks, and lineage.
Correctly process late-arriving and out-of-order events using lookback reprocessing and idempotent incremental loads.
Develop personalization datasets and feature pipelines, including a batch and low-latency Feature Store using Redshift or S3 to DynamoDB serving patterns.
Support AWS Batch and SageMaker workflows through training datasets, offline evaluation data, and model-output delivery to downstream APIs.
Apply dimensional and event Data Modeling, instrument AI interaction and feedback events, optimize warehouse workloads, and establish code review, testing, CI/CD, documentation, and on-call standards.
### Required Qualifications
8+ years in data engineering, including ownership of production pipelines at scale, with strong SQL and Python skills.
Deep experience with AWS services including S3, Glue, Redshift, DynamoDB, Lambda, and IAM.
Production Apache Airflow experience covering DAG design, dependencies, templating, alerting, and backfills.
Streaming ingestion experience with Kafka, Kinesis, SQS, or comparable technologies, plus product analytics data and observability frameworks.
Experience supporting production ML through feature engineering, training datasets, feature stores, or model-serving data flows.
### Preferred Qualifications
Experience with PySpark or Spark, recommendation systems, embeddings, and Embedding pipelines supporting candidate generation or ranking.
Familiarity with LLM and Generative AI applications, vector stores, retrieval pipelines, evaluation data, and feedback loops.
Exposure to MLOps tooling, model registries, experiment tracking, instrumentation governance, A/B testing, and experiment metric pipelines.
Experience with Iceberg, Delta, Hudi, or Redshift data sharing.
Thinking Minds' client offers architectural ownership, collaborative technical work, medical, dental, and vision coverage, retirement matching, paid time off, tuition reimbursement, lifestyle benefits, and relocation assistance.
#Dice
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 10109241
- Position Id: 86714
- Posted 11 hours ago
