Streaming from Apache Iceberg - Building Low-Latency and Cost-Effective Data Pipelines

InfoQ Dev Summit Boston, a two-day conference of actionable advice from senior software developers hosted by InfoQ, will take place on June 24-25, 2024 Boston, Massachusetts.
Deep-dive into 20+ talks from senior software developers over 2 days with parallel breakout sessions. Clarify your immediate dev priorities and get practical advice to make development decisions easier and less risky.
Register now: bit.ly/47tNEWv
-----------------------------------------------------------------------------------------------------
Video with transcript included on InfoQ: bit.ly/46Kz7W3
Steven Wu discusses the design of the Flink Iceberg, comparing the Kafka and Iceberg sources for streaming and how the Iceberg streaming source can power many common stream processing use cases.
#ApacheIceberg #Streaming #DataPipelines
------------------------------------------------------------------------------------------------------
Follow InfoQ:
- Mastodon: techhub.social/@infoq
- Twitter: / infoq
- LinkedIn: / infoq
- Facebook: / infoqdotcom
- Instagram: @infoqdotcom

[@vidak92]

I've been having issues with my Iceberg source in Flink with SQL and Joins not reacting if the source table has write.upsert enabled. It only seems to stream / react on append snapshots. I suppose this is can be explained by the FLIP-27 source not yet supporting Update / Delete and that my upserts are considered updates in this case.