In this talk, we’ll dive deep into Apache Iceberg and discover practical techniques for maintaining large-scale data tables efficiently. We’ll guide you through the core challenges of table maintenance in modern data lakes and share strategies to optimize performance and ensure data integrity. You’ll learn how to apply best practices to manage metadata, handle table compaction, and automate maintenance tasks to minimize downtime and improve query performance.

Session Outline

Introduction

Brief overview of Apache Iceberg and its role in modern data lakehouse architecture. Overview of Iceberg’s architecture and its advantages over traditional table formats. Key challenges and why table maintenance is critical.

The Importance of Table Maintenance

The impact of table bloat, metadata sprawl, and stale snapshots on performance. Why routine maintenance is essential for operational efficiency.

Core Maintenance Strategies

Metadata Management: Techniques to clean and optimize table metadata. Compaction: Best practices for merging small files and cleaning up obsolete data. Partitioning Strategies: How proper partitioning can reduce maintenance overhead. Automation: Tools and scripts to automate routine maintenance tasks.

Q&A and Interactive Discussion

This session is designed to empower data engineers and software professionals with the tools and techniques necessary to ensure the long-term health and performance of their Apache Iceberg tables. Join us to enhance your data management strategies and drive better outcomes in your data-intensive applications.

Speakers:

With over 7 years of dedicated experience in the dynamic field of data engineering, Helped architecting and implementing robust data infrastructure for a variety of organizations, contributing to their success in harnessing the power of data, and now managing product strategy for Cloudera’s Data Lakehouse

Akshat has made contributions towards Apache Hive and Tez