USENIX ATC '13 - TAO: Facebook’s Distributed Data Store for the Social Graph

TAO: Facebook’s Distributed Data Store for the Social Graph
Nathan Bronson, Zach Amsden, George Cabrera, Prasad Chakka, Peter Dimov, Hui Ding, Jack Ferris, Anthony Giardullo, Sachin Kulkarni, Harry Li, Mark Marchukov, Dmitri Petrov, Lovro Puzar, Yee Jiun Song, and Venkat Venkataramani, Facebook, Inc.
We introduce a simple data model and API tailored for serving the social graph, and TAO, an implementation of this model. TAO is a geographically distributed data store that provides efficient and timely access to the social graph for Facebook’s demanding workload using a fixed set of queries. It is deployed at Facebook, replacing memcache for many data types that fit its model. The system runs on thousands of machines, is widely distributed, and provides access to many petabytes of data. TAO can process a billion reads and millions of writes each second.
View the full USENIX ATC '13 program at www.usenix.org/conference/atc...

[@gsb22]

This is GOLDMINE. Amazed with how the system is laid. Respect for all the engineers involved.

[@dilipkumar2k6]

awesome design and presentation. very helpful. Thx for sharing.

[@yogita_garud]

This is awesome! Thanks for sharing. Great talk.

[@ingenious-records]

Simple and Elegant

[@bakkks]

How does write-through cache and/or async replication help with 'see your own writes/ write timeliness' ? My understanding is in order to see your own writes(given replication/sync lag) requests by the same writer have to be served by the same web server -> cache -> db chain.

[@grandhirahul]

It helps because the reads are served immediately from the cache. And also the webservers that write to the follower cache are in the same zone/ecosystem.

[@panhejia]

due to load balancer level sticky session, the same writer client will use the same follower cache. A write request returns as long as the client region cache is populated. Thus the write sees what they write pretty much immediately. But I also do not fully understand the async replication. If a client request -> local follower cache update -> local lead cache update -> local db write is completed, then there is no need to wait for the local lead cache update -> master region lead cache update -> master region db update -> async local region db replication to happen. Is the db async replication only done for consistency purpose? If so can we just use CRDT?

[@bakkks]

How does the cache leader help reduce a hotspot ?

[@panhejia]

i believe the partition of follower cache helps the hotspot issue because you are basically dividing the user groups that are suppose to come to the same hotspot into different cache partitions. As to the leader cache, it is more for the thundering herd issue. Image you have us-west-2 goes offline for 5 mins, then goes back online, if there is no leader cache, then all of the west coast users will hit multiple follower cache, generating a huge wave of duplicate request to the TAO dbs. Using a leader cache that has a smaller data eviction time, it adds another layer that dedupes those duplicate requests. Thus protecting the TAO dbs. Happy to discuss more

[@HandBanana333]

Meep