Key Responsibilities
Lead the end-to-end architecture for migrating Talend ETL/ELT workloads to Databricks
Define target-state Databricks Lakehouse architecture, including data ingestion, transformation, storage, orchestration, governance, and consumption layers
Design scalable, secure, highly available, and cost-optimized data solutions
Define architecture standards, design patterns, coding standards, and best practices for Databricks development
Establish reusable migration frameworks and patterns for converting Talend jobs into Databricks/Spark-based pipelines
Evaluate existing Talend jobs and determine appropriate migration strategies, including re-platforming, re-engineering, consolidation, or retirement
Analyze Talend workflows, jobs, mappings, dependencies, schedules, and data transformations
Develop migration strategies for batch and incremental data processing workloads
Translate Talend transformations and business rules into PySpark, SQL, and Databricks implementations
Identify opportunities to simplify and optimize legacy ETL processes during migration
Define data reconciliation and validation strategies to ensure parity between Talend and Databricks outputs
Establish migration sequencing based on business criticality, dependencies, complexity, and risk
Support migration of high-volume and business-critical data pipelines
Design and implement solutions using Databricks, Apache Spark, Delta Lake, PySpark, and SQL
Design Medallion Architecture using Bronze, Silver, and Gold layers
Implement incremental processing, CDC, SCD Type 1/Type 2, data quality, error handling, and audit frameworks
Optimize Spark workloads, Delta tables, SQL queries, partitioning, clustering, and job execution
Leverage Databricks Workflows/Jobs, Unity Catalog, and modern Databricks data engineering capabilities
Design data pipelines integrating structured and semi-structured data from databases, files, APIs, and enterprise applications
Establish monitoring, logging, alerting, operational support, and performance management practices
Design integrations between Databricks and cloud storage, databases, messaging platforms, APIs, and enterprise applications
Define secure connectivity, networking, secrets management, and access-control patterns
Work with business and technology stakeholders to understand Retail/Grocery data requirements and translate them into scalable data architecture
Provide technical leadership and mentoring to data engineers and developers
Conduct architecture, design, and code reviews
Troubleshoot complex production and performance issues
Collaborate with onshore/offshore teams, enterprise architects, infrastructure teams, security teams, and business stakeholders
Prepare architecture diagrams, technical design documents, migration plans, and implementation standards
Communicate technical risks, dependencies, and recommendations to client leadership
Required Qualifications
10+ years of experience
Bachelor's Degree
Skills
Databricks
Apache Spark
PySpark
SQL
Delta Lake
Talend
Cloud Data Platforms
Change Data Capture (CDC)
Azure Data Lake Storage (ADLS)
Amazon S3
Azure Data Factory
Kafka
REST APIs
Data Pipeline Integration
Data Migration Strategy
ETL/ELT Architecture
Data Architecture
Data Modeling
Data Quality and Reconciliation Frameworks
Retail/Grocery Domain Expertise
Technical Leadership
Mentoring and Coaching
Technical Documentation
Stakeholder Management
Cross-team Dependency Management
Performance Tuning and Optimization
Data Governance and Security
Schedule
Start date: 2026-10-01
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: compun
- Position Id: MDC5922599
- Posted 3 days ago
Never miss an opportunity! Create an alert based on the job you applied for.
