Hello,
Good morning,
I'm Rahul from Floga Technologies. We are a growing IT services organization delivering scalable technology solutions and are expanding our team with qualified talent.
We have an immediate opening for a Iceberg DBA / Lakehouse Operations Engineer Experience with strong hands-on experience to support a long-term engagement with our client. Must collaborate effectively with cross-functional teams to deliver high-quality solutions.
Job Title: Iceberg DBA / Lakehouse Operations Engineer
Location: Remote (Based on the business needs candidates needs to be flexible to work from office, Dallas, TX)
Experience: 10+ years
Duration: Long Term
Job Summary
We are seeking a highly skilled Iceberg DBA / Lakehouse Operations Engineer to manage the reliability, performance, and operational integrity of the Apache Iceberg data layer supporting enterprise analytics and business-critical applications.
The role involves managing a large-scale, multi-engine Lakehouse environment supporting workloads across Spark, Hive, and Impala. The engineer will play a key role in enterprise data modernization initiatives, particularly migrations from Hive and Teradata to Apache Iceberg.
The ideal candidate will have strong hands-on experience with Iceberg table operations, metadata management, data modeling, and query performance optimization in cloud-based environments.
This role is critical to maintaining data accuracy, availability, and performance, as any degradation can directly impact downstream reporting, analytics, and business-critical decision-making.
Key Responsibilities
1. Iceberg Data Layer Ownership & Operations
- Own day-to-day operations and administration of Apache Iceberg tables supporting multiple enterprise applications.
- Ensure data reliability, consistency, and availability across Lakehouse workloads.
- Maintain operational integrity of datasets at multi-terabyte to petabyte scale.
- Monitor table health, metadata growth, and storage utilization.
- Ensure consistent data access and operational stability across environments.
2. Advanced Table Management & Optimization
- Perform advanced Iceberg table maintenance and optimization activities, including:
- Minor and major compaction.
- Small file mitigation and optimization.
- Snapshot expiration and metadata compaction.
- Orphan file cleanup and storage management.
- Optimize data layout and query performance through file sizing, distribution, and partitioning strategies.
- Implement partition evolution and partition pruning techniques.
- Apply clustering and data ordering strategies, such as Z-ordering or equivalent approaches.
- Monitor and optimize table maintenance operations to improve query performance and storage efficiency.
3. Data Modeling & Lakehouse Architecture
- Support and enforce data modeling best practices, including:
- Third Normal Form (3NF) for source-aligned datasets.
- Medallion architecture (Bronze, Silver, and Gold) for curated data pipelines.
- Ensure Iceberg table designs align with ingestion patterns, downstream consumption, and performance requirements.
- Balance data integrity, normalization, and analytical query performance.
- Collaborate with data engineering teams to ensure consistent implementation of layered data architecture across applications.
4. Multi-Engine Query Performance & Consistency
- Ensure consistent and optimized query performance across:
- Apache Spark / Spark SQL (CDE).
- Hive.
- Impala (CDW).
- Troubleshoot query performance bottlenecks, metadata inconsistencies, and inefficient execution plans.
- Analyze query execution plans, scan patterns, and data access behavior.
- Ensure consistent table metadata and query results across multiple processing engines.
5. Hive & Teradata Modernization Support
- Support enterprise data modernization initiatives involving Hive-to-Iceberg and Teradata-to-Iceberg migrations.
- Assist with schema alignment, data type mapping, and table structure conversion.
- Perform data validation, reconciliation, and accuracy checks between source and target datasets.
- Troubleshoot migration-related issues and ensure post-migration stability and performance.
- Collaborate with data engineering teams to optimize migrated workloads.
6. Metadata & Data Lifecycle Management
- Manage Iceberg metadata to ensure scalability, consistency, and performance.
- Maintain consistent table states and metadata across multiple query engines.
- Implement data retention and archival policies.
- Manage snapshot lifecycle operations, expiration, and cleanup.
- Optimize time-travel capabilities and metadata maintenance.
- Monitor metadata growth and prevent performance degradation.
7. Production Support & Incident Management
- Provide L2/L3 operational support for production issues across Iceberg-based Lakehouse workloads.
- Participate in on-call rotations to support critical data platforms.
- Respond to and resolve P1/P2 incidents within defined SLAs.
- Troubleshoot data inconsistencies, reporting discrepancies, query failures, and performance degradation.
- Perform Root Cause Analysis (RCA) and implement permanent corrective and preventive actions.
- Collaborate with platform, infrastructure, and application teams during incident triage and resolution.
8. Security & Data Governance
- Support fine-grained data access controls using Apache Ranger policies and Role-Based Access Control (RBAC).
- Ensure data validation, reconciliation, and accuracy between source systems and Iceberg datasets.
- Maintain secure and compliant data access across applications and processing environments.
- Collaborate with security and governance teams to enforce enterprise data access policies.
Required Skills & Qualifications
- Strong hands-on experience with Apache Iceberg and/or Hive-based data lake environments.
- Strong understanding of Iceberg table architecture, metadata management, and table maintenance operations.
- Experience with table-level optimization, compaction, snapshot management, and performance tuning.
- Understanding of data modeling concepts, including normalization and 3NF.
- Knowledge of modern Lakehouse architecture and Medallion architecture (Bronze/Silver/Gold).
- Hands-on experience with Spark SQL, Hive, and Impala.
- Working knowledge of NiFi and Trino.
- Strong understanding of partitioning strategies, partition evolution, and partition pruning.
- Experience with columnar file formats such as Parquet and ORC.
- Understanding of distributed query processing and large-scale data management.
- Experience supporting multi-terabyte to petabyte-scale data platforms.
Preferred Skills
- Experience with Hive-to-Iceberg and/or Teradata-to-Iceberg migrations.
- Experience with Cloudera CDP, particularly CDE and CDW.
- Familiarity with cloud platforms such as AWS and Azure.
- Experience with Apache Ranger, data governance, and access management.
- Scripting and automation experience using Python and Shell.
- Experience supporting enterprise-scale Lakehouse modernization initiatives.
Thanks & Regards,
Rahul H | Floga Technologies.
Sr. Talent Acquisition Specialist
E: rahul
