L3 Senior Iceberg DBA \/ Lakehouse Operations Engineer

Role: L3 Senior Iceberg DBA / Lakehouse Operations Engineer
Work location: Remote work accepted from anywhere in US

Job Description:

10+ years of experience in Big Data / Data Engineering / DBA / Data Operations roles
Minimum 2+ years of hands-on experience with Apache Iceberg in production environments
6+ years of experience working with Cloudera ecosystem (CDP Ecosystem)

Strong expertise in:
o Iceberg table optimization (compaction, metadata management, partition evolution)
o Multi-engine performance tuning (Spark, Hive, Impala)
o Troubleshooting complex data and query performance issues

Proven experience handling:
o P1/P2 production incidents
o Large-scale environments (TB/PB scale)
o Data migration initiatives (Hive/Teradata Iceberg)

Lead enforcement of data modeling and Lakehouse standards across applications

Guide teams on:
o Medallion architecture implementation
o Balancing normalization vs performance

Review and resolve complex data modeling and performance trade-offs
Ensure consistency of data structures across domains and workloads
Mentor and guide L2 resources in operational best practices and troubleshooting


Required Skills
Strong hands-on experience with Apache Iceberg and/or Hive-based data lakes
Understanding of data modeling concepts (normal forms) and modern Lakehouse patterns (Medallion architecture)
Expertise in:
o Table-level optimization and performance tuning
o Large-scale data management (TB/PB scale)
Experience with:
o Spark SQL, Hive, Impala, NiFI, Trino
Strong understanding of:
o Partitioning strategies
o File formats (Parquet/ORC)
o Distributed query processing


Preferred Skills
Experience with:
o Hive-to-Iceberg or Teradata-to-Iceberg migration
o Cloudera CDP (CDE/CDW)
Familiarity with:
o Cloud platforms (AWS, Azure)
Scripting/automation (Python, Shell)


What You'll Work On
Enterprise-scale Iceberg Lakehouse platform supporting multiple applications
Large-scale data modernization initiatives
Performance optimization and stability of mission-critical analytical workloads


Why This Role Matters
Ensures data correctness and performance for downstream analytics and business-critical reporting
Enables successful modernization from legacy platforms to Iceberg
Maintains high availability and reliability of the enterprise data layer


Job Summary
We are seeking a highly skilled Iceberg DBA / Lakehouse Operations Engineer to own the reliability, performance, and operational integrity of the Iceberg data layer powering enterprise analytics and business-critical applications.

This role operates in a large-scale, multi-engine Lakehouse environment, supporting workloads across Spark, Hive, and Impala, and plays a key role in enterprise data modernization initiatives (Hive and Teradata Iceberg).

The ideal candidate brings deep expertise in Iceberg table operations, metadata management, and query performance optimization, ensuring consistent, high-performance data access across platforms in a cloud-based environment.

This role is critical to ensuring data accuracy and performance-any degradation directly impacts downstream reporting, analytics, and business-critical decision-making.


Key Responsibilities:

Iceberg Data Layer Ownership & Operations
Own day-to-day operations of Apache Iceberg tables supporting multiple enterprise applications
Ensure data reliability, consistency, and availability across all Lakehouse workloads
Maintain operational integrity for datasets at multi-terabyte to petabyte scale

Advanced Table Management & Optimization
Execute advanced Iceberg table maintenance and optimization strategies:
o Compaction (minor/major) and small file mitigation
o Snapshot expiration and metadata compaction to control metadata growth
o Orphan file cleanup (vacuum) to maintain storage efficiency
Optimize data layout and performance through:
o File size tuning and distribution strategies
o Partition evolution and pruning optimization
o Clustering and ordering techniques (e.g., Z-ordering or similar patterns)

Data Modeling Standards & Lakehouse Design Alignment
Support and enforce data modeling best practices aligned with:
o Normalized data structures (3NF) for source-aligned datasets
o Medallion architecture (Bronze / Silver / Gold layers) for curated data flows
Ensure Iceberg table design aligns with:
o Data ingestion patterns (raw vs curated layers)
o Downstream consumption and performance requirements
Assist in structuring datasets to balance:
o Data integrity and normalization
o Query performance and analytical efficiency
Work with data engineering teams to ensure consistent implementation of layered data architecture across multiple applications

Multi-Engine Query Performance & Consistency
Ensure consistent and performant query behavior across:
o Spark (CDE)
o Hive / Impala (CDW)
Troubleshoot and resolve:
o Query performance bottlenecks
o Metadata inconsistencies across engines
o Inefficient execution plans and scan patterns

Hive & Teradata Modernization Support
Play a key role in enterprise data platform modernization (Hive and Teradata Iceberg)
Support:
o Schema alignment and data type mapping
o Data validation and reconciliation
Troubleshoot migration-related issues and ensure post-migration stability and performance

Metadata & Data Lifecycle Management
Manage Iceberg metadata to ensure:
o Efficient scaling and performance
o Consistent table state across engines
Execute lifecycle operations:
o Data retention and archival policies
o Snapshot lifecycle management and cleanup
o Time-travel optimization and maintenance

Production Support, Incident Resolution & On-Call
Provide L2/L3 support for data-related production issues across Iceberg-based Lakehouse workloads
Participate in on-call rotation to support critical data platforms and ensure timely response to incidents
Respond to and resolve P1/P2 production incidents within defined SLAs, minimizing impact to downstream applications and reporting

Troubleshoot:
o Data inconsistencies and reporting discrepancies
o Query failures and performance degradation
Perform root cause analysis (RCA) and implement preventive measures to avoid recurring issues
Collaborate with platform and application teams during incident triage and resolution

Security & Data Governance Support
Support fine-grained access control using:
o Ranger policies and RBAC
Own and ensure data validation, reconciliation, and accuracy between source and Iceberg datasets
Ensure secure and compliant access to data across applications