Using Fast Weights to Attend to the Recent Past, NIPS 2016 | Jimmy Ba, University of Toronto

Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs. There is no good reason for this restriction. Synapses have dynamics at many different time-scales and this suggests that artificial neural networks might benefit from variables that change slower than activities but much faster than the standard weights. These ``fast weights'' can be used to store temporary memories of the recent past and they provide a neurally plausible way of implementing the type of attention to the past that has recently proven helpful in sequence-to-sequence models. By using fast weights we can avoid the need to store copies of neural activity patterns.

[@johannesgh90]

Here's the paper, everyone: arxiv.org/abs/1610.06258 This is really interesting and I hope to see more about different implementations of fast weights soon. Just the elaboration in the Q&A here is really cool. Also, I really like that, in the paper, in reference to LSTMs you say, "remember gates", not "forget gates"... because the inaccuracy of that name always bugs me.

[@revimfadli4666]

Looks like it greatly outperforms LSTMs, so I wonder what's keeping it from being the next gold standard. Also bit of a shame it only blew up after Transformers replaced RNNs for mainstream purposes. With the recent surge of graph nets and massively multi agent learning, hope it'll get another chance to be used