Is Flink the answer to the ETL problem? (with Robert Metzger)

Integration is probably the last, hardest, and least well thought-out part of any large software project. So anything that makes the data-streaming job easier is worth knowing about. So this week we turn our attention to Apache Flink, a flexible system for grabbing, transforming and shipping data between systems using Java, Python or good ol’ SQL.
So this week Robert Metzger-Apache Flink expert and PMC member-joins us to explain what problems Flink solves and how it solves them reliably. We cover the range from simple use cases to realtime aggregations & joins to its high availability strategy.
If you’re working on systems that include more than one database, then you’re definitely going to face the kinds of problems that Flink tackles.
-
Apache Flink: flink.apache.org/
Robert on Twitter: / rmetzger_
Robert on LinkedIn: / metzgerrobert
Kris on Twitter: / krisajenkins
Kris on LinkedIn: / krisjenkins
-
#software #programming #podcast #flink #apacheflink #dataintegration

[@jomohogames]

Great content. Just discovered the channel and started binge-watching!

[@rallokkcaz]

Same listening while I work today!

[@RaviTomar7]

Amazing talk, Thankyou :)

[@mattchaney2559]

Awesome video

[@dataecstasy]

Nice Great talk. More and more developers should be allowed to talk. he didn't pitched decodable, his company for managed flink.

[@cj-ip3zh]

It would be nice to have savepoint triggering available from the web ui.

[@nxx210]

With Ververica platform you can

[@cj-ip3zh]

@@nxx210 I might attempt to make my first open source contribution to OS flink with that feature...

[@IvanGeorgiev]

Can you please cover apache spark streaming?

[@DeveloperVoices]

Yes! Good idea. I'll add that to my list of guests to research. Any suggestions for who would make a good interviewee?

[@IvanGeorgiev]

@@DeveloperVoices I'd say somebody from databricks that could put more color on the internal workings and point out some pros and cons of using it. I'm sure they will be glad to send some of their experts to attend the podcast.

[@simonmassey8850]

a vendor wil point out the cons…? 😂

[@IvanGeorgiev]

@@simonmassey8850 at least will point out all the cons for Apache Flink :)

[@AndreCarneiro666]

Apache Flink brings the worst in terms of maintainability. It's basically the essence of Spark RDD. A trully throwback, in my opinion.

[@poojanpatel2437]

Asking a 12 year veteran working in single tool whether tool support reporting like pdf, csv. Holy please stop doing any content at all. This is just embarrassing for anyone even coming to your channel for speaking.

[@DeveloperVoices]

I think what you've missed is how many people doing data engineering are stuck with unclean data in crappy data sources. I was asking if Flink would be any help in that ugly but all-too-common reality.