Role: L2 – Senior Iceberg DBA / Lakehouse Operations Engineer
Location: Remote work accepted from anywhere in US
Send profiles to
Job Description:
Isoft is seeking a highly skilled Iceberg DBA / Lakehouse Operations Engineer to own the reliability, performance, and operational integrity of the Iceberg data layer powering enterprise analytics and business-critical applications.
This role operates in a large-scale, multi-engine Lakehouse environment, supporting workloads across Spark, Hive, and Impala, and plays a key role in enterprise data modernization initiatives (Hive and Teradata → Iceberg).
The ideal candidate brings deep expertise in Iceberg table operations, metadata management, and query performance optimization, ensuring consistent, high-performance data access across platforms in a cloud-based environment.
This role is critical to ensuring data accuracy and performance—any degradation directly impacts downstream reporting, analytics, and business-critical decision-making.
Key Responsibilities:
Iceberg Data Layer Ownership & Operations
Own day-to-day operations of Apache Iceberg tables supporting multiple enterprise applications
Ensure data reliability, consistency, and availability across all Lakehouse workloads
Maintain operational integrity for datasets at multi-terabyte to petabyte scale
Advanced Table Management & Optimization
Execute advanced Iceberg table maintenance and optimization strategies:
Compaction (minor/major) and small file mitigation
Snapshot expiration and metadata compaction to control metadata growth
Orphan file cleanup (vacuum) to maintain storage efficiency
Optimize data layout and performance through:
File size tuning and distribution strategies
Partition evolution and pruning optimization
Clustering and ordering techniques (e.g., Z-ordering or similar patterns)
Data Modeling Standards & Lakehouse Design Alignment
Support and enforce data modeling best practices aligned with:
Ensure Iceberg table design aligns with:
Assist in structuring datasets to balance:
Data integrity and normalization
Query performance and analytical efficiency
Work with data engineering teams to ensure consistent implementation of layered data architecture across multiple applications
Multi-Engine Query Performance & Consistency
Ensure consistent and performant query behavior across:
Spark (CDE)
Hive / Impala (CDW)
Troubleshoot and resolve:
Query performance bottlenecks
Metadata inconsistencies across engines
Inefficient execution plans and scan patterns
Hive & Teradata Modernization Support
Support:
Schema alignment and data type mapping
Data validation and reconciliation
Troubleshoot migration-related issues and ensure post-migration stability and performance
Metadata & Data Lifecycle Management
Manage Iceberg metadata to ensure:
Execute lifecycle operations:
Data retention and archival policies
Snapshot lifecycle management and cleanup
Time-travel optimization and maintenance
Production Support, Incident Resolution & On-Call
Provide L2/L3 support for data-related production issues across Iceberg-based Lakehouse workloads
Participate in on-call rotation to support critical data platforms and ensure timely response to incidents
Respond to and resolve P1/P2 production incidents within defined SLAs, minimizing impact to downstream applications and reporting
Troubleshoot:
Data inconsistencies and reporting discrepancies
Query failures and performance degradation
Perform root cause analysis (RCA) and implement preventive measures to avoid recurring issues
Collaborate with platform and application teams during incident triage and resolution
Security & Data Governance Support
Support fine-grained access control using:
Ranger policies and RBAC
Own and ensure data validation, reconciliation, and accuracy between source and Iceberg datasets
Ensure secure and compliant access to data across applications
