Real-Time Data Pipelines Made Easy with Structured Streaming in Apache Spark | Databricks

[@tejusization]

28:45 to 29:40 is the best!!! :D just don't miss that. sets the context

[@mohitmehta3788]

Very simplified approach of explaining streaming.

[@ashwinkumar5223]

Superb Explanation

[@HridyanshiB.]

Good explanation about streaming..Thanks

[@danielmackie82]

What would be an open source equivalent of DB delta?

[@yourstruly5DA]

Good presentation. Would like to understand more how it could integrate and scale with Apache Kafka.

[@satria5403]

Hi, please let me know if u have good resources for this. thank you

[@venkat.k4392]

Appreciated. Thanks you for a great knowledge share.

[@zhengfang303]

When the data has entered the dataframe, if the data has been updated or deleted, how can I update or delete it in the dataframe?

[@parthadeb3723]

A dataframe is immutable. You cannot update a dataframe. You have to create an new dataframe.

[@thesleepyhead7273]

How about integrating this with Tensorflow Serving for end to end Analytics paradigm

[@michaelbrenndoerfer9908]

Google Hydrogen

[@sujithkumar804]

@@michaelbrenndoerfer9908 lol

[@GrayMatterSoftware]

Want to know about the best Practices for Real-Time Analytics Architecture on Big Data? Read here: www.graymatter.co.in/real-time-analytics-bigdata-architecture/ Know more: www.graymatter.co.in/real-time-analytics/ Watch here: kzbin.info/www/bejne/oonHip5pncaea5Y

[@JanekBogucki]

23:30 A single rogue timestamp which is one hour ahead of the second max timestamp would drop all earlier buckets except one bucket corresponding to this single anomalous value. This is fragile.

[@onewithsixonewithsix601]

Unless there is crazy issue in code manipulating timestamp. It is not a probable scenario to get timestamp ahead of actual unix time.

[@albertoandreotti7940]

This is a big disappointment. You cannot stream pipelines built with dataframes. Unified processing framework?? come on! You have to build new versions of all your algorithms so now they can work with a DStream? What a waste of time.