Maps, Arrays and Subcollections, Oh My! | Get to know Cloud Firestore #4

Is your Cloud Firestore data structure set up correctly? Cloud Firestore supports Subcollections, Maps, and Arrays, all of which can be useful, but it's sometimes difficult to know when to use which one. Why create a subcollection when you could just create an array of maps? Or a map of maps? When is it appropriate to use one over the other? And what would work best in your app?
In this episode of Get to Know Cloud Firestore, Todd does his best to get your questions answered and gives you all the pros and cons, so you can structure your data the right way.
Cloud Firestore Data model → bit.ly/2PZtQnD
More episodes on Cloud Firestore → goo.gle/Cloud-Firestore
Subscribe to Firebase → goo.gle/Firebase

[@IsaacAsante17]

You guys have mastered the art of teaching.

[@codinginflow]

The animations are amazing

[@MrCamster999]

So true

[@eduardosalles9212]

NOOOO.. we learn by seeing the codes. Show us examples.. we want the codes..

[@Zakariat89]

This is one of the best explanations I've ever seen on data structures. Not just for firebase, but noSQL in general. Thank you.

[@bahadirbalban8773]

love the pace of these videos. All teaching videos must be made like this.

[@sexyeur]

(2018, 2019, 2020...) Todd is cool in any year. Thank you for this video!

[@Ballistah]

Great explanation! I LOVE that you also update the videos! That's great!

[@roquefore]

Love love love love love all these Firebase videos! Awesome! Keep going!

[@GustavoDiSalvo]

Great video, thank you! I'm just learning and designing my NoSQL model and every one of these videos teach me exactly what I wanted to know. I have one complain: when you update the original video please update the subtitles too!

[@RobinYoulton]

Thanks Todd, I REALLY appreciate these insights on Cloud Firestore.

[@nickadams2361]

I swear this guy is the only part of any google software I like.

[@asdfghyter]

So the basic rule of thumb might be "Shape your data like your queries". Data that are usually queried together should go together and hierarchies that exist in your app should probably also be mirrored in the data.

[@DanielSadjadian]

This guy is just brilliant and the ending was so funny.

[@klutch4198]

"This tends to put your app in a state I like to call 'uninstalled'" hahahaha

[@sheelajain370]

getting uninstalled

[@reycoseguma2184]

This is a very good explanation for Firestore. Now I know which I should use or not use on Cloud Functions by knowing these rules on Firestore. I think I can learn more by watching the whole series.

[@dilshan3372]

This firestore video series is so fantastic 👌

[@samzmann]

Thanks for array-contains. This is really good!

[@rafidhoda]

Amazing video! That ending was HILARIOUS

[@lucaban4146]

This is the most important video about Firestore imo.

[@AymanShaabanELsayedDEVELOPER]

Hey todd from 2019 you forgot to make a new caption for the video after edit

[@adamm3122]

Ayman Shaaban ELsayed DEVELOPER this was really impressive how they did this!

[@tylim88]

you can also switch to auto generate caption

[@troythompson2]

Right On Time.. Again! Keep it up!

[@n00bstude]

Thanks for the quick education!

[@rolikaseventysix]

Cool as always! Cheers from Slovakia!

[@princehamza890]

Absolutely Great Video ! Firebase is Awesome

[@user-ms3zr4cy2b]

Just keep them videos coming!!!

[@KrishAgrawal]

So Todd can time travel. great work google 👍

[@CuriousMotor]

You're a superb teacher!

[@ecofriend93]

Security rules need to be taken into account when setting up data structures as well. I just found this out the hard way. Luckily I'm still in the dev phase so can pivot to a different data structure that allows me to set different security rules for sub-collections vs the parent collection.

[@AbhishekKumar-mq1tt]

Thank u for this awesome video

[@digimbyte]

can this 20k document field limit be bypassed if we can turn off indexing?

[@Fanaro]

Very helpful video. Thanks!

[@Firebase]

We really appreciate the positive feedback, Philippe! Thanks for watching 😀 If you liked this, be sure to check out more episodes on Cloud Firestore here: goo.gle/Cloud-Firestore

[@qm3ster]

How can I prevent indexing? Should I store a serialized JSON object (as firestore String)?

[@brunorocha9898]

Awesome! Congratz to the team

[@boon_keng]

Could you please next time cover topic of timestamps, dates in Firestore / Cloud functions! I mean getting, storing and sending to clients! Thanks in advance!

[@michaelscothon9871]

,so

[@marcelomms4021]

When to use simple string to store a reference vs a real reference (type) ?

[@steverobertson2732]

Great video. I recently ran into the current limitation of not being able to delete collections (or sub-collections) so deleting a document with sub-collections would orphan the sub-collections. Seems like another reason to use arrays within documents instead of sub-collections - at least for now. Your thoughts?

[@zizaart1199]

Clarified a lot of things, tysm

[@PeriPC]

Great teaching experience guys, great job Todd. The only thing that i am skeptical all this time with Firestore is the limitations of upgrading. Let me explain better. We build an app today with specific needs, let's say for example a Charles Dickens' app where we need to get the books titles only. After users feedback we decide to upgrade our app and add "search by character's occupation" function. In these changes (upgrades) of our app, we need to re-design our entire db "schema" and maybe migrate old db structure to a new one... It sounds tricky

[@sveinsoermo9883]

Awesome info...

[@vanyvanov]

Thanks, it really helped!

[@Firebase]

You're welcome!

[@zensic]

Thank you so much!

[@michaelblum4557]

Great video, thanks

[@Firebase]

Glad you liked it!

[@blessing7695]

I was asking myself this question just yesterday bcz we need to scale up our app.

[@AndreaLombardo84]

So, in the end, what's the best data structure for an Agenda like application to store customers appointments 😄😄😄? The app have a collection of customers and I need to show the full month appointments in a "Google calendar style", but also need to count how many times a client has an appointment with me. Any suggestion? Thanks a lot.

[@mika2666]

🤣 the shark tank refrence at the end

[@LuisSierra42]

ikr, that cracked me up

[@MC-db5gp]

2:20 Firestore would redo index when created a new document, make a change and so on, and thus a limit on field number is necessary? I cant get it :'(

[@marcelogamer3214]

Geopoint will have their own querys soon? Or not at all? Like distance between geopoints or querys by proximity? Really waiting for those!

[@michaeloosthuizen2383]

I would be interested in this too as I have a project that might need to calculate the distance between points. I am dreading having to look up that crazy formula and trying to figure out how make it work. The Geopoint object has a compareTo method that might be used for this, but the docs don't really elaborate on what it returns: firebase.google.com/docs/reference/android/com/google/firebase/firestore/GeoPoint#compareTo(com.google.firebase.firestore.GeoPoint)

[@MichaelEvanspishposh]

I’m running a side Postgres dB with one table that just keeps track of the Firestore Id and a point lat long. So I query the Postgres when I need geospatial but other 99% is direct to firestore

[@lukecross5439]

July of 2020, this is still not supported officially, but, i got filter my documents adding they on Adapter and using Collections.sort(); But, the disavantage is the realtime data not appears more.

[@oghenefokethompson7407]

NICE TEACHING

[@louisart92]

Great!! and how can I do a query like "get all characters from each book into 'dickens_book'" I mean get all subcollections 'characters' from all dickens's books in orther to save them into a array after that?

[@marcelomms4021]

If I want to store user's tenats, should I create an array of references?

[@pkop4]

If array_contains is like map field === true, why can we only query on 1 value at a time, and not multiple ?

[@rockdrummer3999]

Excellent tutorial

[@Firebase]

Glad you liked it

[@acche-rc]

I'm surprised the video didn't cover one big limitation of ArrayContain query: they cannot be chained.

[@danielbutterfield2309]

I would love to have a pint with Todd!

[@ManishPatel3030]

It's very optimize way to store data in collection and send/receive in JSON.

[@johnalto6517]

this guy is awesome

[@adityapalavajjhala1841]

Which among these is more efficient? Updating an array field inside a document or listener on a sub-collection with documents?

[@Quetzalcoatl0]

I don't know what is the best way of creating a voting system for example for "movies". I want people to vote from 1 to 5 stars but also keep track if they voted or not. I'm showing the vote count number and score of each "movie" so i want it to be in the "movies" collection. I can make an array of users in the "movies" collection, and add the UID there so i can check if the current user voted or not, maybe this is the correct way ? I don't think more than 1000 people are gonna use the web app i'm making, it's for a very specific non popular thing, so i think it should be fine even if there are 20k people in the next 5 years.?

[@shashikantdurge]

#askfirebase Are arrays ordered as they are sent? Suppose I write [ b , a , c] are they stored in the same order or [ a , b , c ] ?

[@Bodesuri252]

Hey, thanks for the guidelines/rules. But a question here on arrays in documents: How do I validate the data entered to the firestore. How can I define the type of the elements that get inserted via the security rules? For example, if I only want to have objects inside an array with some specific values.

[@michaeloosthuizen2383]

Security rules does have some support for this, but its not extensive. Depending on your use case I would suggest security rules as a first line and cloud functions as a second line. If it is critical stuff then you can disallow writes in the rules and create a cloud HTTP function that parses a request and writes to the database directly (the admin SDK bypasses rules), more detail here: cloud.google.com/functions/docs/writing/http

[@luciusrex]

As a hobbyist with no professional background, this was immensely helpful! I was wondering the use case of subcollections were if we could just map them into the top level document (I used them anyway for better organization). Especially since one of the key features of nosql is to denormalized data, this was highlighted in video 1 as well ithink. I am currently using a subdocument (e.g,"users/blogs") to copy a top level document (e.g.,"blogs") into a "users" document but currently need 2 api calls to retrieve user and the user's blogs data, a userDocRef and a userBlogsRef. I will be rewriting my code to make it so that the user's blog data is in the user's document and reduce the number of server calls when retrieving data. Posting/updating data will unlikely be improved. (btw, its not really a blog app im making, it's an operations (management) program but seemed like blogs would be easier to understand).

[@michaeloosthuizen2383]

I wouldn't get to focused on reducing database calls, but if you wanted to make things quicker and reduce calls you could do something like this: Have an array in the Users document that lists basic details of each blog, say the document ID, the title and extract. Then have the full blog documents in a sub collection. That way when you retrieve the User document you get a list of blog posts without the second query but still have enough detail to know exactly which blog document to query if you need to. Then maintain consistency using cloud functions to update the array when ever a blog is created or updated. Here are the docs on those triggers: firebase.google.com/docs/functions/firestore-events

[@philip2.042]

does this mean that if a document only contains one array, the size of the array is also limited to 20,000 elements (like maps) as well?

[@devmeta1478]

What if I have situation where ,logically speaking, I should have one document with few subcolletions and each subcollection have documents with many subcolletions. And i always wanna show whole document or just root (top level without any subcolletions) But i know that many users are gonna write and change that document very often, so should I split it to top level collections to avoid write failing ?

[@johnbalvin5401]

Good video, now let's wait 3 moths for the next one :O

[@johnbalvin5401]

@@ToddKerpelmanCorp it was all a lie :C

[@nevinjohn]

I want to list the unread "news" if the "viewed" array within the news documents doesn't contain the user id... Any ideas on how to implement the logic "notarrayContains" ? I guess firestore does not support it? Thanks for any help!

[@nevinjohn]

For those who are wondering how to achieve this: groups.google.com/forum/?#!msg/google-cloud-firestore-discuss/wXn4nd2RBIc/jsWhxy3FBgAJ

[@OMARVOLCOM1]

I can't find any documentation to allow me to update my array of maps in Firestore from flutter. Any ideas on how to find this?

[@Transmediaproducer]

I have the answer and your back in I promise! Dickens and all the rest.

[@codinginflow]

5:50 Aren't 2 million reads = 1.20$?

[@anonwithamnesia]

yes and 2 billion reads are 1200$

[@tilakmadichettitheappdeveloper]

fawk !

[@qm3ster]

So, arrays should be used like sets?

[@-no-handle]

I think they just index the arrays as objects. I just tried adding duplicate strings to the array and it worked just fine.

[@toastrecon]

Is 8:54 a typo? I'm guessing that both subcollections should be named "characters". Thanks, this video helps!

[@GiammaCarioca]

Nice video but it would be great to have an article to accompany it

[@alexanderhemming6148]

Does anyone know why pressing enter in a material UI text field submitting info to firestore does not add enters/ returns into the database?

[@geraldalvarez6466]

Nice

[@laxy1070]

hey but how can I get an array from the collection I did not find any documentation in a firebase site??

[@mikemagss]

Is it possible to write security rules for arrays or is a sub collection the only way?

[@michaeloosthuizen2383]

A rule applies to an entire document (ie ALL its fields including arrays) so if you needed to protect data differently to the rest of the document then I would put it in a sub collection.

[@IAmSamuelCharpentier]

How do they do these video edits from the future?

[@michaeloosthuizen2383]

I find that using sub-collections are great for data privacy because you can have different rules for a document and its sub-collections. Good data structures plus strong rules equals wicked privacy kung fu.

[@benbenzyhome]

Do you have videos for firebase 9

[@karthikshekar2715]

hey, how would I query if I had a collection called videos tat contain the video url etc and another collection called users tat contains the username which needs to be displayed during the video play.

[@GiridharPSPKPSPK]

I have a doubt regarding deleting an array of data in Arrays. The Structure looks like image: [ 0 : imageurl : ".........", isliked : "true", 1: imageurl : ".........", isliked : "false", ] now I'd like to delete index 1 how can I delete using the Cloud Firestore ArrayRemove method or is there anything else to remove based on Index position. Thanks & Regards :)

[@TedHopp]

The distinguishing feature of arrays is index-based access, which Firebase doesn't support. Seems like a better name for "arrays" would have been "sets" (or perhaps "multisets" or "bags" if repeated values are supported). Or am I missing something?

[@michaeloosthuizen2383]

In Firestore rules the arrays are called lists and object are called maps. I think lists would be a more accurate description of them, I am not sure why the difference in terminology between the data structure and the rules... maybe because arrays and objects are more web dev terminology and lists and maps are more "traditional" terminology (at least from my perspective) and probably more correct from a CS point of view.

[@TedHopp]

Interesting. Like arrays, the abstract data type "list" corresponds to a number of ordered values. Lists usually support, at a minimum, ordered traversal of elements (i.e., a "first" and "next" operation) as well as ordered insertion (e.g., "append", "prepend"). From what I gathered from the video and my (admittedly thin) knowledge of the API, Firebase doesn't support any order-based operations on what they call arrays (in this video) or lists (in the rules). Therefore, I don't think "list" is any more appropriate than "array" for what Firebase does offer. If they were to ask me, I might have suggested they use "map" and "collection", but they already assigned a different meaning to "collection", so that's out. Oh well.

[@michaeloosthuizen2383]

Iteration is supported, its just not supported on the database (as far as I am aware). If you use the Javascript SDK an array from the database is an array object, and a Javascript array is a list. Also the support methods of a data structure do not make it a list or not, the actual structure determines that. Another also: when you query a document you get the ENTIRE document, so there is no real need to iterate over an array or manipulate it on the database, you would do that in whatever code made the query (be it client side with a client SDK or API or server side with the admin SDK or API). For example, if you wanted to do server side manipulation you would do that with Cloud Functions using the admin SDK, not in Firestore itself.

[@TedHopp]

Thanks for the info, Todd. What you're describing sounds to me like a set implemented as an array which is in turn simulated using a map. But the exposed functionality sounds like a set, not an array or a map. Yes, values are entered by supplying them in some order (that looks like an array) and they are returned in some order (that looks like an array). However, from what I've read, duplicate values are not supported (the array ["a", "b", "c", "a", "b", "c"] gets stored as ["a", "b", "c"]). In this case (and in others) there's no order guarantee. From the api user's perspective, it doesn't have the characteristics one would expect of an array. I'm not sure I understand your comment. On the one hand you say I have indexed-based access and ordering, but on the other hand I don't have that because the only operators I have don't allow for indexed-based access or ordering. (For instance, I can't insert an item at position 3 if the same value already exists at position 0. Actually, I can't insert an item at position 3 at all, since the api doesn't allow it.) I understand from the blogs why the api is designed this way, and I think it's a good design. A true array model would have lots of issues. I just object to it being called an array when the only thing it has in common with an array is that sets of values are transmitted in both directions in an apparently ordered form (which is not guaranteed to be preserved).

[@Shikschauhan]

Is it possible to update a map in an array ?

[@almeale317]

It could be nice if we could check if a date or number is what we expected(in rage or higher or lower) into a object in an array

[@TikTok-iv5fe]

hey guys,is there any one have information on flutter on this firestore concept

[@rijatru]

If I query a collection with 40 documents, will I get charged for each of those documents reads?

[@vikasrai1316]

Hey,Please make a video on how to maintain data consistency across all collection and documents.I found write batch for this problem but there is limit of only 500 per one write batch. What if i want to write more than 500 at once.

[@DenisTRUFFAUT]

NoSQL implies you, the dev, handle manually data consistency at write time. Once you edit a record, you have to apply changes on all collections where it is referenced. That said, you really don't want to use foreign keys, which is an artefact of SQL. Generally; you separate stuff and you end up with top level collections such : /users/{userId} describing your user, or /friends/{userId} describing the friends of your user, or /messages/{userId} describing messages of your user.

[@vikasrai1316]

Yeah cool thanks.. But still there is a limit of 500 write batch at once what if there are more than 500 duplicate Data and i want to update them, i can only update 500 at once not more than that.

[@michaeloosthuizen2383]

I found that splitting my triggers across multiple functions helped a lot with that limit. Of course if you still have such large batches then write all updates to an array then step through the array to create separate batches of 500 each and then use Promise.all() to run them.

[@aashishranjansingh4124]

Now we need Todd from 2022

[@jasonsebring3983]

I keep thinking just use firestore for realtime updates but use another database for real search capability and return ids for firebase to use. Use the right tool for the job but requires keeping data in sync.

[@alexgu3257]

Caught a small error at 11:24, missing a "s" after "keyword". Thanks for making this btw.

[@user-lw4zo8ip9z]

Problem with subtitles from 6:23 :(

[@tilakmadichettitheappdeveloper]

use google voice translate

[@weilieschok2001]

Could you make some tutorial if i need to store data historically? e.g. employee | date | department | job_title E001 | 1-Jan-2018 | D01 | Trainee E001 | 1-Mar-2019 | D02 | Manager department | date | dept_name D01 | 1-Jan-2000 | Sales D01 | 1-Jan-2017 | Sales & Investment D02 | 1-Jan-2000 | Legal D02 | 1-Jan-2017 | Audit * D01 was first introduced as Sales but later renamed to "Sales & Investment" in 2017, same applied to D02 From the relationship above, i know the employee first hired , 1 Jan 2018 and joined in 'Sales & Investment' department and not 'Sales' Then promoted as manager in 1 Mar 2019 and joined "Audit" department and no longer 'legal'. It's all very easy in SQL/RDBMS. But it's quite tough to model it in NoSQL. Hopefully there is some tutorial on these topic. Thanks

[@stofstik]

Very helpful vid! But only one write per document per second... WAT

[@DodaGarcia]

"In a state I like to call 'getting uninstalled'" 🤣🤣🤣

[@jayantnarula1157]

If any one tell me , How to use 'Forward Slash ' as a character in document Id in swift

[@SingularityLabsAI]

What's the best way to store say I am expecting to store 10m keys. The workflow goes like this If I encounter a key in my app, I check firestore if that key exists, if it does I do some thing if not I put the key in the firestore. I don't really care for the value of the key if there is provision to delete the key or else let the value be true/false, such that I can use false if I wish to delete the key from firestore. I need any of key insertion/lookup/delete in O(1) or O(logN) .

[@klaudiaadamus3353]

hey, guys, just typing this to say your "subscribe" and "watch more" linked windows are popping up way too early - that may be because of your new content addition ;) anyway, a great video collection, thanks!

[@youmeandeveryone5893]

Ok, wheres the video about collectionGroup?

[@Klazyo]

1 - 1 mb per document 2 - you can not retrieve data partially 3 - Quering are shallow 4 - better put ur data top lvl if u want search thru all document or retrieve

[@lukecross5439]

If a document has n sub-collections, can this document reach its maximum size?

[@Frank_van_Puffelen]

The size limit on a document applies to individual documents. The subcollections under a document do not count towards the size of that document.

[@lukecross5439]

@@Frank_van_Puffelen So, subCollections can serve as subFolders

[@v_iancu]

8:44 Can you please point to an example of how such an index should look like?

[@pkop4]

When GraphQL ?

[@poomwett]

8:04 Top-level Collections