Following the merger of LINE and Yahoo! JAPAN, we faced the monumental task of enabling seamless data access between two massive, completely independent Hadoop ecosystems—each with its own isolated networks and disparate account systems. This session provides a deep dive into the technical strategies we used to overcome these architectural barriers and achieve cross-cluster interoperability.

Furthermore, we will explore our journey of scaling and performance tuning using NameNode Federation (including Router-based Federation) and Observer NameNodes to push past the inherent limits of a single NameNode. We will share the real-world operational hurdles we encountered and the specific tuning required to maintain stability in one of the world’s largest Hadoop deployments. Finally, we will discuss our experience contributing patches back to the OSS community to ensure these features are battle-tested for enterprise-scale requirements.

Speakers:

Akira Ajisaka: Principal Engineer, LY Corporation

Akira Ajisaka is a Principal Engineer at LY Corporation, with over 10 years of expertise in big data technologies across both on-premises and public cloud environments. Apache Member, Apache Hadoop PMC member, and Apache Kyuubi PMC member.

Kai Hirayama: Data Engineer, LY Corporation

Data Platform Engineer working on Hadoop ecosystem, including YARN and HDFS. Experienced in operating and troubleshooting large-scale distributed systems.