Time Series Forecasting Using Recurrent Neural Network and Vector Autoregressive Model: When and How

The General Data Protection Regulation (GDPR), which came into effect on May 25, 2018, establishes strict guidelines for managing personal and sensitive data, backed by stiff penalties. GDPR's requirements have forced some companies to shut down services and others to flee the EU market altogether. GDPR's goal to give consumers control over their data and, thus, increase consumer trust in the digital ecosystem is laudable. However, there is a growing feeling that GDPR has dampened innovation in machine learning & AI applied to personal and/or sensitive data. After all, ML & AI are hungry for rich, detailed data and sanitizing data to improve privacy typically involves redacting or fuzzing inputs, which multiple studies have shown can seriously affect model quality and predictive power. While this is technically true for some privacy-safe modeling techniques, it's not true in general. The root cause of the problem is two-fold. First, most data scientists have never learned how to produce great models with great privacy. Second, most companies lack the systems to make privacy-safe machine learning & AI easy. This talk will challenge the implicit assumption that more privacy means worse predictions. Using practical examples from production environments involving personal and sensitive data, the speakers will introduce a wide range of techniques--from simple hashing to advanced embeddings--for high-accuracy, privacy-safe model development. Key topics include pseudonymous ID generation, semantic scrubbing, structure-preserving data fuzzing, task-specific vs. task-independent sanitization and ensuring downstream privacy in multi-party collaborations. Special attention will be given to Spark-based production environments.
Talk by Jeffrey Yau.
About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: databricks.com/product/unifie...
Connect with us:
Website: databricks.com
Facebook: / databricksinc
Twitter: / databricks
LinkedIn: / databricks
Instagram: / databricksinc Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. databricks.com/databricks-nam...

[@michaelcao9483]

Thanks so much for a great presentation, Jeff Yau! I've been looking for techniques to model multivariate time series data, and found this video to be extremely helpful!

[@lewismambo8043]

This Lecture in TMSA is very useful. Thank very much Prof.

[@nicok3345]

Thanks for this outstanding presentation :-).

[@cagataymelan1407]

Very helpful. Thank you..! Just noticed that in 20:22 you are multiplying by lag 3 for inverse transformation although you differenced by lag 12

[@siabikebenezer]

Hello sir, can i please get the script for your presentation. I will really glad if you provide your codes to me. Thanks

[@mikiallen7733]

but the problem with sign autocorrelations are known to be non-linear more like XOR function which when we apply the vector autoregressions to it , will fail miserably ! so do you have any special advice as to which method works better with sign AND magnitude autocorrelations your input is highly appreciated

[@PikkaKok]

At 20:18 aren't you inversing the diff with the same values you are trying to forecast? (... * series['beer'][-3:])

[@BrooklynBambi]

Share the source code please?

[@apicasharma2499]

Could you please explain the process of generating IRFs and Variance decomposition in both methods

[@WahranRai]

25:48 You forgot Water gate !

[@snivesz32]

1) Has anyone found a link to Jeffrey Yau's hour-and-a-half version of this talk? 2) The description on this video is incorrect, this video is not about GDPR.

[@dagma3437]

This perhaps? kzbin.info/www/bejne/qnuQgGaeoJapiNk

[@dagma3437]

github.com/SimiY/pydata-sf-2016-arima-tutorial

[@alexanderskusnov5119]

Is there an example of Reinforcement Learning?

[@Avinaash15]

Could anyone explain the part where he puts the RMSE into context. Im not sure how that fits into forecasting future values

[@dataEvo]

RMSE is on absolute units, which without context cannot tell by itself how good the model is. For instance, if RMSE is 100 when predicting values around 200, your % error is 50%. On the hand, if you are predicting values around 1.000.000, an RMSE of 100 is only 0.01% error. Therefore, just by looking at RMSE from two different scenarios you can't tell which one has a better fitted model.

[@ImranKhan-fi2sm]

Hii How to handle persistent model problem. While doing time series analysis i get the output which seems to be one time step ahead of the actual series. How to rectify this problem?? This thing i am getting with several ML, DL, and as well as with statistical algos. Please do reply??

[@Human2023v1]

apply a lead transformation of the forecasted series.

[@eytansuchard8640]

How about using transformers ?

[@jorjodimitrov]

yeah , put a link to github repository captain america. Scraping letter by letter from the video will take me a hole day.