Apache IoTDB, as a high‑performance time‑series database, introduced the in‑DB stream processing framework Pipe in recent years, providing a solid foundation for real‑time data handling. However, in data integration and backup scenarios, Pipe‑based solutions still suffer from complex development, unstable operation, and limited flexibility. To address these issues, we’ve designed and implemented an in‑DB stream‑processing–based time‑series data subscription module with the following advantages:

• Minimal onboarding: Provides a Kafka‑style subscription interface—one line of code is all it takes to start the data stream, with no extra plugin development required.
• Dual‑mode flexible consumption: Supports both real‑time, row‑level streaming and batch TsFile export, adapting to different business scenarios on demand.
• Performance optimization: Leverages the operating system’s Page Cache for near‑zero‑latency real‑time transmission; exports historical data directly as TsFiles, boosting export efficiency by over 10×.
• Ecosystem integration: Seamlessly integrates with stream processing platforms like Apache StreamPipes, making it easy to fit into existing big‑data architectures. In this talk, we will walk through the mechanisms and applications of time‑series data subscription based on the above design.

Speakers:

Apache IoTDB Contributor, Tsinghua University