Reformer: The Efficient Transformer

[@CristianGarcia]

Thanks, recently read it, enjoyed the video! BTW: I think they claim to reduce memory from O(n^2) to O(n log n), not O(n).

[@connor-shorten]

Thank you!! I was commenting on indexing the individual queries vs. the entire dot product with the title O(L^2) to O(L). I apologize for not making this clear, hopefully future watchers can see this comment and understand the mistake! I didn't mention the n log n in the video because I wasn't really able to understand it myself, maybe you could help clarify this! Is this the memory cost of the LSH bucketing?

[@CristianGarcia]

Mi intuition was that because of the bucketing defined by LSH they only had to store the dot-product of these small matrices, but I really don't know the details of the implementation. Today I saw the Google Research team has an implementation of the Reformer on this jax-based library called trax they created (terrible documentation): github.com/google/trax/tree/master/trax/models/reformer

[@connor-shorten]

@@CristianGarcia Yeah, it was a tough paper to read, really vague about exactly how they implemented it. I saw a lot of criticism on the paper on reddit / hackernews. Thanks for sharing, I'll check this out!

[@jielyu4943]

nice summary

[@hihiendru]

greatr explanation, thank you.

[@planktonfun1]

I've seen this on paper, never thought they would use it