Senior Big Data Engineer
Data Engineer
Owns the data pipelines that move data from operational source systems onto the data platform extraction, ingestion, transformation, orchestration, and the day-to-day operational health of those pipelines.
Also plays a role as a hands-on builder of Foundational Data Products (raw record-of-truth) and potentially Derivative Data Products (composed from upstream products).
Core skills
SQL & Python (table stakes); Scala/Java for high-throughput streaming
Pipeline build & operations: design, develop, deploy, monitor, and remediate batch and streaming pipelines that land source data on the platform; manage backfills, replays, late-arriving data, schema drift, and SLA breaches
Ingestion patterns: full-load, incremental, change-data-capture (Debezium, Fivetran, Qlik Replicate, GoldenGate), event-driven ingest, API and file-based
intake
Pipeline frameworks: dbt, Apache Spark, Apache Beam, Airflow, Dagster, Prefect
Streaming: Kafka, Kinesis, Flink, Spark Structured Streaming
Data product packaging: schema contracts (Avro/Protobuf/JSON Schema), versioning, SLAS/SLOs, data contracts, output-port design (SQL, file, API, event)
Storage formats: loeberg, Delta Lake, Hudi (open table formats are now the mesh default)
Quality & observability: Great Expectations, Soda, Monte Carlo, dbt tests, data lineage (OpenLineage)
CI/CD for data: GitOps pipelines, unit + integration tests, environment promotion
Al-adjacent vector store ingestion (pgvector, Pinecone), feature stores (Feast, Tecton), RAG-ready chunking and embedding pipelines
Mesh-specific: knowing when to build a derivative product vs. extending an existing one; consuming upstream products through governed input ports rather than reaching into source systems
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 90933573
- Position Id: cd3e7366afd9219cf8faa13c1b0a1263
- Posted 1 hour ago
Never miss an opportunity! Create an alert based on the job you applied for.
