What are different type of input sources in Spark Structed Streaming. How different Triggers works.
Пікірлер: 2
@madhurrusia32763 жыл бұрын
During same spark execution session 1. What if we 2-3 delete files from folder? 2. If we update content of any file?
@dataengineeringforeveryone3 жыл бұрын
Hey. Spark is maintaining state of previous file aggregation in state store. So it doesn’t matter if you delete the processed file or changed the content. As far as they are processed spark doesn’t care about them.