Message queues are essential for real-time use cases like payment processing, fraud detection, and AI-powered support systems—but traditional queues often lack scalability, durability, and replayability. In this talk, we explore how Kafka 4.0 brings native queue semantics to the world of distributed streaming, enabling fair, concurrent, and isolated message processing at scale.

We’ll show how Apache Flink’s native LLM integration leverages this queue model to perform real-time Large Language Model (LLM) inference—like sentiment analysis or summarization—and how enriched results can be written directly to Apache Iceberg, a powerful data lakehouse for long-term analytics, data versioning & time travel and ML feedback loops.

Through a demo and architecture walkthrough, you’ll learn how to build intelligent, scalable pipelines that combine Kafka queues, Flink, LLMs, and Iceberg into a unified real-time analytics stack.

Speakers:

Shekhar is deeply passionate about open source software and actively contributes to various projects, including SymPy, Ruby gems like daru and daru-view (which he authored), Bundler, NumPy, and SciPy. He successfully completed Google Summer of Code in 2016 and 2017 and has served as an admin for SciRuby, mentoring multiple organizations. Shekhar has spoken at prominent conferences such as RubyConf 2018, PyCon 2017, ApacheCon 2020, and Community Over Code 2024, as well as numerous regional meetups. Currently, he works at Apple as a Software Development Engineer.