DataOps & Build Engineer

Job description:

Key Responsibilities:

• Establish DataOps Framework: Define, document, and champion the organizational framework and guidelines for DataOps—including release management processes, environment promotion strategy, and data quality standards.
• Best Practice Dissemination: Create and enforce standard operating procedures (SOPs) for data pipeline development, CI/CD, and testing across the engineering teams, ensuring consistency and adherence to architectural standards
• Data Pipeline Automation: Design and implement robust continuous integration and continuous delivery (CI/CD) pipelines for data code and infrastructure
• Workflow Orchestration Implementation: Configure, optimize, and manage the deployment of data workflows using orchestrators such as Dagster or Talend, focusing on automated testing and deployment steps.
• Version Control & Repository Management: Enforce best practices for source code management (e.g., Gitflow), branching strategies, and repository organization across all data projects.
• Infrastructure as Code (IaC): Work with Infrastructure teams to automate provisioning and management of data platform resources efficiently within AWS.
• Resilience and Failure Recovery: Design and implement automated rollback and self-healing mechanisms within pipelines to quickly recover from transient failures.
• Monitoring and Logging: Set up comprehensive monitoring, logging, and alerting using Cloud native tools, or other tools to ensure visibility into pipeline performance and quickly identify and resolve issues
• Security and Compliance: Ensure data security and compliance by implementing IAM policies, encryption, and other security measures in AWS, adhering to best practices for handling sensitive data
• Testing Frameworks: Implement automated testing strategies across the data lifecycle, including unit tests, integration tests, and data quality validation checks (e.g., column integrity, schema drift) to ensure data reliability before deployment
• Resource and Cost Optimization: Implement automated policies and monitoring to track and control cloud resource consumption, ensuring that pipelines run efficiently and cost-effectively

Candidate Profile:

• 10+ years of hands-on experience across Data Engineering & DevOps, or a dedicated DataOps role, focused heavily on automation and operational excellence
• Proven experience implementing CI/CD practices specifically for data pipelines and data infrastructure
• Strong conceptual understanding of data warehousing, ETL/ELT methodologies, and cloud-native architecture.
• Automation First Mindset: A strong drive to automate repetitive tasks and eliminate manual intervention in the data lifecycle
• Collaboration: Excellent communication skills, capable of working effectively with Data Engineers, Data Scientists, and Infrastructure teams
• Insurance industry experience preferred but not mandatory
• Tools:
  • Core Technology : Airflow, Dagster, Prefect, Monte Carlo, Acceldata
  • Cloud Environment: AWS (S3, IAM, VPC, etc.)
  • Ingest & Transform: dbt Core, AWS Glue