Databricks | Pyspark| AutoLoader: Incremental Data Load |with Demo

[@ajaykiranchundi9979]

Thanks for the kickstart. But I have a question once the python script is executed..it is continuously running and not stopping at all. Is it a general behavior of autoloader ? Also, if I have to run it as a part of ingestion pipeline..what are your suggestions ?

[@gudiatoka]

Not sure about autoloader as it used stream and in structured streaming we can use trigger available now (it is same as event based trigger in adf)

[@SHARATH2596]

For a beginner would you suggest to learn AWS or azure?

[@ShilpaDataInsights]

You can start with any cloud service AWS or Azure. The basics remains the same.