We are looking for a Senior Data Engineer to design, implement and optimize scalable data pipelines and infrastructure. You will play a key role in ensuring data availability, reliability and quality for analytics, machine learning and business intelligence applications. Your work will involve collaborating with data scientists, analysts and software engineers to develop robust ETL/ELT workflows, real-time streaming solutions and cloud-native data architectures. Responsibilities Architect, build and optimize scalable data pipelines for batch and real-time data processing Develop and implement ETL/ELT workflows, ensuring efficient data ingestion, transformation and storage Leverage modeling (Party model, Datavault) methodologies to enable scalable and flexible data modeling Ensure data consistency, reliability and governance across data lakes, warehouses and operational data stores Optimize performance and cost efficiency of data infrastructure on Azure Implement and manage big data processing frameworks, such as Databricks, Kafka Enhance data security and compliance, integrating RBAC & ABAC, encryption and regulatory frameworks (GDPR) into data infrastructure Develop automation tools for data pipeline orchestration, using Airflow, Azure Data Factory or Prefect Monitor, troubleshoot and optimize data pipelines, ensuring minimal downtime and quick recovery from failures Collaborate with federated engineering teams to align data architecture with business and engineering goals Provide clean, reliable and scalable datasets for advanced analytics and machine learning in partnership with data scientists and business analysts Evaluate and adopt emerging technologies, ensuring continuous improvement in data engineering best practices Mentor and guide junior engineers, fostering technical excellence and knowledge sharing within the team Requirements 5+ years of experience in data engineering, ETL development or big data technologies Expertise in designing and optimizing ETL/ELT workflows, using tools such as dbt, Airflow, Azure Data Factory or Apache NiFi Hands-on experience with cloud-native data platforms, including Azure Synapse, Databricks, Snowflake or BigQuery Knowledge of data modeling techniques, including Data Vault 2.0, star schema and normalization strategies Experience with large-scale distributed computing frameworks (Apache Spark, Hadoop, Kafka, Event Hub) Advanced proficiency in SQL and programming languages, such as Python, Scala or Java Understanding of Infrastructure as Code (Terraform, Pulumi, ARM templates) for managing cloud-based data infrastructure Skills in data security and governance best practices, including RBAC, encryption and data lineage Experience working in a federated engineering environment, ensuring seamless integration across teams Proficiency in observability and monitoring tools for data pipelines, such as Monte Carlo, DataDog and Great Expectations Familiarity with Agile and DevSecOps methodologies, ensuring continuous integration, deployment and monitoring of data solutions Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems or a related field Upper-Intermediate English language proficiency (B2+)
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 10330481
- Position Id: b645261356319cff8f701c7c87aeb382
- Posted 5 hours ago
Never miss an opportunity! Create an alert based on the job you applied for.
