Azure Databricks Engineer (PySpark) Tech Lead
Contract Corp To Corp
Contract Independent
Contract W2
Redrouthu's Inc
Fitment
Dice Job Match Score™
🔢 Crunching numbers...
Job Details
Skills
- Data Integration
- Routers
- Data Validation
- Scalability
- Collaboration
- Optimization
- Mentorship
- Informatica PowerCenter
- Mapping
- Workflow
- Databricks
- Data Engineering
- Python
- Database
- Code Refactoring
- Stored Procedures
- SQL
- Data Lake
- Storage
- Unity
- Extract
- Transform
- Load
- ELT
- Data Modeling
- Data Quality
- Apache Spark
- Performance Tuning
- Caching
- Data Processing
- Recovery
- Problem Solving
- Conflict Resolution
- Communication
- Stakeholder Management
- Supply Chain Management
- Analytics
- Business Data
- Microsoft Azure
- DevOps
- Git
- Continuous Integration
- Continuous Delivery
- Production Support
- Release Management
- IT Management
- Informatica
- Oracle
- PySpark
- Migration
- Testing
Summary
# Job Description: Azure Databricks Engineer (PySpark) - Tech Lead
**Location:** Dallas, TX / Atlanta, GA
**Work Mode:** 5 Days Onsite
**Joining Requirement:** Day 1 Onsite
## Role Summary
We are seeking an experienced **Azure Databricks Engineer (PySpark) - Tech Lead** to lead the end-to-end migration of enterprise data integration workloads from Informatica PowerCenter to Azure Databricks. The role requires strong hands-on expertise in PySpark, Azure Databricks, SQL, and data engineering, along with technical leadership, architecture design, and delivery ownership.
The ideal candidate will have proven experience analyzing complex Informatica mappings, migrating batch ETL pipelines, and modernizing Oracle-based data processing workloads into scalable, reliable, and high-performance Databricks solutions. Experience with Unity Catalog, Delta Live Tables (DLT), ADLS Gen2, and enterprise data platform modernization is essential.
## Key Responsibilities
- Lead the end-to-end migration of data pipelines, workflows, and integration patterns from Informatica PowerCenter to Azure Databricks.
- Analyze existing Informatica PowerCenter 10.4.1 workflows, mappings, sessions, and batch ETL processes to define migration strategies and target-state architecture.
- Convert complex Informatica transformations, including lookups, joins, filters, routers, expressions, and aggregations, into equivalent PySpark and Spark SQL implementations.
- Analyze Oracle stored procedures, functions, packages, and database dependencies to determine whether to migrate, refactor, or recreate business logic in Databricks.
- Design and develop scalable ETL/ELT pipelines using PySpark, Python, and Spark SQL.
- Implement data engineering solutions using Azure Databricks Runtime 14.4 LTS, Unity Catalog, Delta Live Tables (DLT), and Azure Data Lake Storage Gen2 (ADLS Gen2).
- Modernize legacy ETL processes into optimized, maintainable, and reusable data engineering frameworks.
- Establish engineering standards covering architecture, coding practices, testing, deployment, performance optimization, and production readiness.
- Define data validation, reconciliation, and testing strategies to ensure functional parity between legacy Informatica workflows and Databricks pipelines.
- Optimize Spark jobs, data processing workloads, and SQL queries for performance, scalability, and cost efficiency.
- Collaborate with business stakeholders, solution architects, infrastructure, security, and operations teams to ensure successful migration delivery.
- Identify technical risks, dependencies, and migration challenges, and implement mitigation plans.
- Lead deployment, cutover, production stabilization, monitoring, and post-migration optimization activities.
- Mentor engineering teams on PySpark development, Databricks best practices, reusable frameworks, and modern data engineering methodologies.
## Required Technical Skills
- Strong hands-on experience with **Azure Databricks and PySpark**.
- Extensive experience with **Informatica PowerCenter**, including mapping analysis, workflow execution, transformations, sessions, and batch ETL processes.
- Proven experience migrating Informatica workloads to Databricks or other modern data engineering platforms.
- Strong programming skills in Python, PySpark, and Spark SQL.
- Strong SQL expertise and experience working with Oracle databases.
- Experience analyzing and refactoring Oracle stored procedures, functions, packages, and complex SQL logic.
- Hands-on experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
- Experience with Unity Catalog, Delta Lake, and Delta Live Tables (DLT).
- Knowledge of ETL/ELT architecture, data modeling, data quality, and data reconciliation.
- Experience with Spark performance tuning, partitioning, caching, and distributed data processing.
- Experience designing data pipelines with appropriate error handling, logging, monitoring, and recovery mechanisms.
- Strong technical leadership, problem-solving, communication, and stakeholder management skills.
## Preferred Qualifications
- Experience with enterprise-scale data modernization and legacy platform migration programs.
- Experience working with supply chain data, enterprise analytics, or complex business data integration environments.
- Familiarity with Azure DevOps, Git, CI/CD pipelines, and automated deployment practices.
- Experience with production support, incident resolution, release management, and operational readiness.
- Experience establishing reusable migration frameworks and engineering standards across multiple teams.
## Work Location and Availability
- **Locations:** Dallas, Texas, or Atlanta, Georgia.
- **Work Arrangement:** Fully onsite, five days per week.
- **Joining:** Must be available to work onsite from Day 1.
## Ideal Candidate Profile
The ideal candidate is a hands-on technical lead who can independently assess existing Informatica and Oracle workloads, develop migration strategies, implement equivalent PySpark pipelines, and lead the migration through testing, deployment, and production stabilization. Candidates should demonstrate both deep technical expertise and the ability to guide engineering teams through enterprise-scale data platform modernization.
**Location:** Dallas, TX / Atlanta, GA
**Work Mode:** 5 Days Onsite
**Joining Requirement:** Day 1 Onsite
## Role Summary
We are seeking an experienced **Azure Databricks Engineer (PySpark) - Tech Lead** to lead the end-to-end migration of enterprise data integration workloads from Informatica PowerCenter to Azure Databricks. The role requires strong hands-on expertise in PySpark, Azure Databricks, SQL, and data engineering, along with technical leadership, architecture design, and delivery ownership.
The ideal candidate will have proven experience analyzing complex Informatica mappings, migrating batch ETL pipelines, and modernizing Oracle-based data processing workloads into scalable, reliable, and high-performance Databricks solutions. Experience with Unity Catalog, Delta Live Tables (DLT), ADLS Gen2, and enterprise data platform modernization is essential.
## Key Responsibilities
- Lead the end-to-end migration of data pipelines, workflows, and integration patterns from Informatica PowerCenter to Azure Databricks.
- Analyze existing Informatica PowerCenter 10.4.1 workflows, mappings, sessions, and batch ETL processes to define migration strategies and target-state architecture.
- Convert complex Informatica transformations, including lookups, joins, filters, routers, expressions, and aggregations, into equivalent PySpark and Spark SQL implementations.
- Analyze Oracle stored procedures, functions, packages, and database dependencies to determine whether to migrate, refactor, or recreate business logic in Databricks.
- Design and develop scalable ETL/ELT pipelines using PySpark, Python, and Spark SQL.
- Implement data engineering solutions using Azure Databricks Runtime 14.4 LTS, Unity Catalog, Delta Live Tables (DLT), and Azure Data Lake Storage Gen2 (ADLS Gen2).
- Modernize legacy ETL processes into optimized, maintainable, and reusable data engineering frameworks.
- Establish engineering standards covering architecture, coding practices, testing, deployment, performance optimization, and production readiness.
- Define data validation, reconciliation, and testing strategies to ensure functional parity between legacy Informatica workflows and Databricks pipelines.
- Optimize Spark jobs, data processing workloads, and SQL queries for performance, scalability, and cost efficiency.
- Collaborate with business stakeholders, solution architects, infrastructure, security, and operations teams to ensure successful migration delivery.
- Identify technical risks, dependencies, and migration challenges, and implement mitigation plans.
- Lead deployment, cutover, production stabilization, monitoring, and post-migration optimization activities.
- Mentor engineering teams on PySpark development, Databricks best practices, reusable frameworks, and modern data engineering methodologies.
## Required Technical Skills
- Strong hands-on experience with **Azure Databricks and PySpark**.
- Extensive experience with **Informatica PowerCenter**, including mapping analysis, workflow execution, transformations, sessions, and batch ETL processes.
- Proven experience migrating Informatica workloads to Databricks or other modern data engineering platforms.
- Strong programming skills in Python, PySpark, and Spark SQL.
- Strong SQL expertise and experience working with Oracle databases.
- Experience analyzing and refactoring Oracle stored procedures, functions, packages, and complex SQL logic.
- Hands-on experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
- Experience with Unity Catalog, Delta Lake, and Delta Live Tables (DLT).
- Knowledge of ETL/ELT architecture, data modeling, data quality, and data reconciliation.
- Experience with Spark performance tuning, partitioning, caching, and distributed data processing.
- Experience designing data pipelines with appropriate error handling, logging, monitoring, and recovery mechanisms.
- Strong technical leadership, problem-solving, communication, and stakeholder management skills.
## Preferred Qualifications
- Experience with enterprise-scale data modernization and legacy platform migration programs.
- Experience working with supply chain data, enterprise analytics, or complex business data integration environments.
- Familiarity with Azure DevOps, Git, CI/CD pipelines, and automated deployment practices.
- Experience with production support, incident resolution, release management, and operational readiness.
- Experience establishing reusable migration frameworks and engineering standards across multiple teams.
## Work Location and Availability
- **Locations:** Dallas, Texas, or Atlanta, Georgia.
- **Work Arrangement:** Fully onsite, five days per week.
- **Joining:** Must be available to work onsite from Day 1.
## Ideal Candidate Profile
The ideal candidate is a hands-on technical lead who can independently assess existing Informatica and Oracle workloads, develop migration strategies, implement equivalent PySpark pipelines, and lead the migration through testing, deployment, and production stabilization. Candidates should demonstrate both deep technical expertise and the ability to guide engineering teams through enterprise-scale data platform modernization.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 91174039
- Position Id: OOJ - 1127-128-1792760850
- Posted 5 hours ago
Company Info
About Redrouthu's Inc
Redrouthu's is a premier, dual-capability talent and technology partner dedicated to helping organizations scale with confidence. We bridge the critical gap between complex business needs and flawless execution by delivering top-tier technical talent and engineering resilient, scalable IT solutions.
We do not believe in a "one-size-fits-all" mentality. Whether an organization requires urgent contract staffing, strategic full-time hires, or custom software architecture built from scratch, Redrouthu’s operates as a dependable extension of our clients' teams. With offices in Austin, Texas, and robust operational capabilities in India, we seamlessly support domestic scaling and international talent expansion.
What We Do: Core Pillars
1. Tech & Engineering Talent Solutions
Hiring often breaks down when recruiters prioritize resume volume over contextual alignment. At Redrouthu's, we focus deeply on quality and precision hiring. We spend time understanding the explicit role requirements, team challenges, and long-term delivery timelines before presenting a single profile.
- Permanent Hires: Identifying and embedding permanent talent that matches your company's technical standards and internal culture.
- Contract Staffing: Rapidly deploying heavily-screened contract professionals to meet immediate project timelines and fluctuating workloads.
- On-Demand RPO: Embedding our elite recruitment specialists directly into your ecosystem to manage talent acquisition at any scale.
- India Talent Solutions: Utilizing our deep on-the-ground expertise to help US firms build, transition, and manage highly effective technical teams within the Indian market.
2. Systems & Custom IT Solutions
We engineer adaptive ecosystems and scalable software architectures built to support evolving business growth. We prioritize clean structure, data security, and long-term maintainability over superficial, rushed deliveries.
- Custom Software Engineering: Translating your complex business logic into custom, cloud-native digital assets mapped precisely to your workflows.
- Managed IT Services: Taking full ownership of daily IT operations, security protocols, and infrastructure efficiency so your internal teams can remain focused on core milestones.
- Managed Projects (SOW): Fully managing end-to-end technical deliverables under clear Statement of Work (SOW) frameworks, deploying specialized tech experts to hit your exact goals.
Why Technologists Choose Redrouthu’s
For developers, engineers, and IT professionals, Redrouthu’s is more than an agency. It is a career accelerator.
- Impactful Projects: We place talent in roles across dynamic industries, including Finance, Healthcare, Retail, and Manufacturing.
- Diverse Technical Ecosystems: Our placements and projects span a massive technical spectrum, from developers and QA specialists to Cloud Engineers and Data Architects.
- Clarity First: We do not ghost candidates or send them into ambiguous interview loops. We ensure you have extreme clarity regarding project scope, expectations, and compensation from day one.
Our Mission & Vision
- Mission: To empower global organizations to execute their digital strategies with absolute confidence by delivering the exact talent and robust technical infrastructure they need.
- Vision: To serve as a globally recognized unified partner where people, process, and modern technology converge to unlock operational excellence and sustained business growth.
Core Values We Live By
- Outcome Over Activity: We do not count hours or measure success by volume. We measure it exclusively by successful placements, stable system uptimes, and realized business value.
- Accountability & Trust: We stand behind the software we build and the technical candidates we place.
- Simplicity & Speed: We avoid unnecessary corporate jargon and technical complexity, choosing instead to deliver streamlined, high-speed execution.
Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.