Maps, Arrays and Subcollections, Oh My! | Get to know Cloud Firestore #4

[@IsaacAsante17]

You guys have mastered the art of teaching.

[@codinginflow]

The animations are amazing

[@MrCamster999]

So true

[@eduardosalles9212]

NOOOO.. we learn by seeing the codes. Show us examples.. we want the codes..

[@Zakariat89]

This is one of the best explanations I've ever seen on data structures. Not just for firebase, but noSQL in general. Thank you.

[@nickadams2361]

I swear this guy is the only part of any google software I like.

[@bahadirbalban8773]

love the pace of these videos. All teaching videos must be made like this.

[@asdfghyter]

So the basic rule of thumb might be "Shape your data like your queries". Data that are usually queried together should go together and hierarchies that exist in your app should probably also be mirrored in the data.

[@GustavoDiSalvo]

Great video, thank you! I'm just learning and designing my NoSQL model and every one of these videos teach me exactly what I wanted to know. I have one complain: when you update the original video please update the subtitles too!

[@reycoseguma2184]

This is a very good explanation for Firestore. Now I know which I should use or not use on Cloud Functions by knowing these rules on Firestore. I think I can learn more by watching the whole series.

[@Ballistah]

Great explanation! I LOVE that you also update the videos! That's great!

[@roquefore]

Love love love love love all these Firebase videos! Awesome! Keep going!

[@lucaban4146]

This is the most important video about Firestore imo.

[@AymanShaabanELsayedDEVELOPER]

Hey todd from 2019 you forgot to make a new caption for the video after edit

[@adamm3122]

Ayman Shaaban ELsayed DEVELOPER this was really impressive how they did this!

[@tylim88]

you can also switch to auto generate caption

[@ecofriend93]

Security rules need to be taken into account when setting up data structures as well. I just found this out the hard way. Luckily I'm still in the dev phase so can pivot to a different data structure that allows me to set different security rules for sub-collections vs the parent collection.

[@rafidhoda]

Amazing video! That ending was HILARIOUS

[@DanielSadjadian]

This guy is just brilliant and the ending was so funny.

[@RobinYoulton]

Thanks Todd, I REALLY appreciate these insights on Cloud Firestore.

[@dilshan3372]

This firestore video series is so fantastic 👌

[@klutch4198]

"This tends to put your app in a state I like to call 'uninstalled'" hahahaha

[@sheelajain370]

getting uninstalled

[@AndreaLombardo84]

So, in the end, what's the best data structure for an Agenda like application to store customers appointments 😄😄😄? The app have a collection of customers and I need to show the full month appointments in a "Google calendar style", but also need to count how many times a client has an appointment with me. Any suggestion? Thanks a lot.

[@codinginflow]

5:50 Aren't 2 million reads = 1.20$?

[@anonwithamnesia]

yes and 2 billion reads are 1200$

[@tilakmadichettitheappdeveloper]

fawk !

[@marcelogamer3214]

Geopoint will have their own querys soon? Or not at all? Like distance between geopoints or querys by proximity? Really waiting for those!

[@michaeloosthuizen2383]

I would be interested in this too as I have a project that might need to calculate the distance between points. I am dreading having to look up that crazy formula and trying to figure out how make it work. The Geopoint object has a compareTo method that might be used for this, but the docs don't really elaborate on what it returns: firebase.google.com/docs/reference/android/com/google/firebase/firestore/GeoPoint#compareTo(com.google.firebase.firestore.GeoPoint)

[@MichaelEvanspishposh]

I’m running a side Postgres dB with one table that just keeps track of the Firestore Id and a point lat long. So I query the Postgres when I need geospatial but other 99% is direct to firestore

[@lukecross5439]

July of 2020, this is still not supported officially, but, i got filter my documents adding they on Adapter and using Collections.sort(); But, the disavantage is the realtime data not appears more.

[@boon_keng]

Could you please next time cover topic of timestamps, dates in Firestore / Cloud functions! I mean getting, storing and sending to clients! Thanks in advance!

[@michaelscothon9871]

,so

[@PeriPC]

Great teaching experience guys, great job Todd. The only thing that i am skeptical all this time with Firestore is the limitations of upgrading. Let me explain better. We build an app today with specific needs, let's say for example a Charles Dickens' app where we need to get the books titles only. After users feedback we decide to upgrade our app and add "search by character's occupation" function. In these changes (upgrades) of our app, we need to re-design our entire db "schema" and maybe migrate old db structure to a new one... It sounds tricky

[@MC-db5gp]

2:20 Firestore would redo index when created a new document, make a change and so on, and thus a limit on field number is necessary? I cant get it :'(

[@brunorocha9898]

Awesome! Congratz to the team

[@KrishAgrawal]

So Todd can time travel. great work google 👍

[@toastrecon]

Is 8:54 a typo? I'm guessing that both subcollections should be named "characters". Thanks, this video helps!

[@samzmann]

Thanks for array-contains. This is really good!

[@nevinjohn]

I want to list the unread "news" if the "viewed" array within the news documents doesn't contain the user id... Any ideas on how to implement the logic "notarrayContains" ? I guess firestore does not support it? Thanks for any help!

[@nevinjohn]

For those who are wondering how to achieve this: groups.google.com/forum/?#!msg/google-cloud-firestore-discuss/wXn4nd2RBIc/jsWhxy3FBgAJ

[@philip2.042]

does this mean that if a document only contains one array, the size of the array is also limited to 20,000 elements (like maps) as well?

[@TedHopp]

The distinguishing feature of arrays is index-based access, which Firebase doesn't support. Seems like a better name for "arrays" would have been "sets" (or perhaps "multisets" or "bags" if repeated values are supported). Or am I missing something?

[@michaeloosthuizen2383]

In Firestore rules the arrays are called lists and object are called maps. I think lists would be a more accurate description of them, I am not sure why the difference in terminology between the data structure and the rules... maybe because arrays and objects are more web dev terminology and lists and maps are more "traditional" terminology (at least from my perspective) and probably more correct from a CS point of view.

[@TedHopp]

Interesting. Like arrays, the abstract data type "list" corresponds to a number of ordered values. Lists usually support, at a minimum, ordered traversal of elements (i.e., a "first" and "next" operation) as well as ordered insertion (e.g., "append", "prepend"). From what I gathered from the video and my (admittedly thin) knowledge of the API, Firebase doesn't support any order-based operations on what they call arrays (in this video) or lists (in the rules). Therefore, I don't think "list" is any more appropriate than "array" for what Firebase does offer. If they were to ask me, I might have suggested they use "map" and "collection", but they already assigned a different meaning to "collection", so that's out. Oh well.

[@michaeloosthuizen2383]

Iteration is supported, its just not supported on the database (as far as I am aware). If you use the Javascript SDK an array from the database is an array object, and a Javascript array is a list. Also the support methods of a data structure do not make it a list or not, the actual structure determines that. Another also: when you query a document you get the ENTIRE document, so there is no real need to iterate over an array or manipulate it on the database, you would do that in whatever code made the query (be it client side with a client SDK or API or server side with the admin SDK or API). For example, if you wanted to do server side manipulation you would do that with Cloud Functions using the admin SDK, not in Firestore itself.

[@TedHopp]

Thanks for the info, Todd. What you're describing sounds to me like a set implemented as an array which is in turn simulated using a map. But the exposed functionality sounds like a set, not an array or a map. Yes, values are entered by supplying them in some order (that looks like an array) and they are returned in some order (that looks like an array). However, from what I've read, duplicate values are not supported (the array ["a", "b", "c", "a", "b", "c"] gets stored as ["a", "b", "c"]). In this case (and in others) there's no order guarantee. From the api user's perspective, it doesn't have the characteristics one would expect of an array. I'm not sure I understand your comment. On the one hand you say I have indexed-based access and ordering, but on the other hand I don't have that because the only operators I have don't allow for indexed-based access or ordering. (For instance, I can't insert an item at position 3 if the same value already exists at position 0. Actually, I can't insert an item at position 3 at all, since the api doesn't allow it.) I understand from the blogs why the api is designed this way, and I think it's a good design. A true array model would have lots of issues. I just object to it being called an array when the only thing it has in common with an array is that sets of values are transmitted in both directions in an apparently ordered form (which is not guaranteed to be preserved).

[@Fanaro]

Very helpful video. Thanks!

[@Firebase]

We really appreciate the positive feedback, Philippe! Thanks for watching 😀 If you liked this, be sure to check out more episodes on Cloud Firestore here: goo.gle/Cloud-Firestore

[@GiridharPSPKPSPK]

I have a doubt regarding deleting an array of data in Arrays. The Structure looks like image: [ 0 : imageurl : ".........", isliked : "true", 1: imageurl : ".........", isliked : "false", ] now I'd like to delete index 1 how can I delete using the Cloud Firestore ArrayRemove method or is there anything else to remove based on Index position. Thanks & Regards :)

[@karthikshekar2715]

hey, how would I query if I had a collection called videos tat contain the video url etc and another collection called users tat contains the username which needs to be displayed during the video play.

[@devmeta1478]

What if I have situation where ,logically speaking, I should have one document with few subcolletions and each subcollection have documents with many subcolletions. And i always wanna show whole document or just root (top level without any subcolletions) But i know that many users are gonna write and change that document very often, so should I split it to top level collections to avoid write failing ?

[@courageous-schools]

Just keep them videos coming!!!

[@almeale317]

It could be nice if we could check if a date or number is what we expected(in rage or higher or lower) into a object in an array

[@rolikaseventysix]

Cool as always! Cheers from Slovakia!

[@alexanderhemming6148]

Does anyone know why pressing enter in a material UI text field submitting info to firestore does not add enters/ returns into the database?

[@laxy1070]

hey but how can I get an array from the collection I did not find any documentation in a firebase site??

[@Klazyo]

1 - 1 mb per document 2 - you can not retrieve data partially 3 - Quering are shallow 4 - better put ur data top lvl if u want search thru all document or retrieve

[@CuriousMotor]

You're a superb teacher!

[@Bodesuri252]

Hey, thanks for the guidelines/rules. But a question here on arrays in documents: How do I validate the data entered to the firestore. How can I define the type of the elements that get inserted via the security rules? For example, if I only want to have objects inside an array with some specific values.

[@michaeloosthuizen2383]

Security rules does have some support for this, but its not extensive. Depending on your use case I would suggest security rules as a first line and cloud functions as a second line. If it is critical stuff then you can disallow writes in the rules and create a cloud HTTP function that parses a request and writes to the database directly (the admin SDK bypasses rules), more detail here: cloud.google.com/functions/docs/writing/http

[@louisart92]

Great!! and how can I do a query like "get all characters from each book into 'dickens_book'" I mean get all subcollections 'characters' from all dickens's books in orther to save them into a array after that?

[@michaeloosthuizen2383]

I find that using sub-collections are great for data privacy because you can have different rules for a document and its sub-collections. Good data structures plus strong rules equals wicked privacy kung fu.

[@digimbyte]

can this 20k document field limit be bypassed if we can turn off indexing?

[@ManishPatel3030]

It's very optimize way to store data in collection and send/receive in JSON.

[@luciusrex]

As a hobbyist with no professional background, this was immensely helpful! I was wondering the use case of subcollections were if we could just map them into the top level document (I used them anyway for better organization). Especially since one of the key features of nosql is to denormalized data, this was highlighted in video 1 as well ithink. I am currently using a subdocument (e.g,"users/blogs") to copy a top level document (e.g.,"blogs") into a "users" document but currently need 2 api calls to retrieve user and the user's blogs data, a userDocRef and a userBlogsRef. I will be rewriting my code to make it so that the user's blog data is in the user's document and reduce the number of server calls when retrieving data. Posting/updating data will unlikely be improved. (btw, its not really a blog app im making, it's an operations (management) program but seemed like blogs would be easier to understand).

[@michaeloosthuizen2383]

I wouldn't get to focused on reducing database calls, but if you wanted to make things quicker and reduce calls you could do something like this: Have an array in the Users document that lists basic details of each blog, say the document ID, the title and extract. Then have the full blog documents in a sub collection. That way when you retrieve the User document you get a list of blog posts without the second query but still have enough detail to know exactly which blog document to query if you need to. Then maintain consistency using cloud functions to update the array when ever a blog is created or updated. Here are the docs on those triggers: firebase.google.com/docs/functions/firestore-events

[@marcelomms4021]

When to use simple string to store a reference vs a real reference (type) ?

[@qm3ster]

So, arrays should be used like sets?

[@-no-handle]

I think they just index the arrays as objects. I just tried adding duplicate strings to the array and it worked just fine.

[@acche-rc]

I'm surprised the video didn't cover one big limitation of ArrayContain query: they cannot be chained.

[@adityapalavajjhala1841]

Which among these is more efficient? Updating an array field inside a document or listener on a sub-collection with documents?

[@michaelblum4557]

Great video, thanks

[@Firebase]

Glad you liked it!

[@marcelomms4021]

If I want to store user's tenats, should I create an array of references?

[@vanyvanov]

Thanks, it really helped!

[@Firebase]

You're welcome!

[@zensic]

Thank you so much!

[@rijatru]

If I query a collection with 40 documents, will I get charged for each of those documents reads?

[@Quetzalcoatl0]

I don't know what is the best way of creating a voting system for example for "movies". I want people to vote from 1 to 5 stars but also keep track if they voted or not. I'm showing the vote count number and score of each "movie" so i want it to be in the "movies" collection. I can make an array of users in the "movies" collection, and add the UID there so i can check if the current user voted or not, maybe this is the correct way ? I don't think more than 1000 people are gonna use the web app i'm making, it's for a very specific non popular thing, so i think it should be fine even if there are 20k people in the next 5 years.?

[@cyber3nai1]

Problem with subtitles from 6:23 :(

[@tilakmadichettitheappdeveloper]

use google voice translate

[@steverobertson2732]

Great video. I recently ran into the current limitation of not being able to delete collections (or sub-collections) so deleting a document with sub-collections would orphan the sub-collections. Seems like another reason to use arrays within documents instead of sub-collections - at least for now. Your thoughts?

[@troythompson2]

Right On Time.. Again! Keep it up!

[@benbenzyhome]

Do you have videos for firebase 9

[@alexgu3257]

Caught a small error at 11:24, missing a "s" after "keyword". Thanks for making this btw.

[@qm3ster]

How can I prevent indexing? Should I store a serialized JSON object (as firestore String)?

[@OMARVOLCOM1]

I can't find any documentation to allow me to update my array of maps in Firestore from flutter. Any ideas on how to find this?

[@n00bstude]

Thanks for the quick education!

[@v_iancu]

8:44 Can you please point to an example of how such an index should look like?

[@SingularityLabsAI]

What's the best way to store say I am expecting to store 10m keys. The workflow goes like this If I encounter a key in my app, I check firestore if that key exists, if it does I do some thing if not I put the key in the firestore. I don't really care for the value of the key if there is provision to delete the key or else let the value be true/false, such that I can use false if I wish to delete the key from firestore. I need any of key insertion/lookup/delete in O(1) or O(logN) .

[@GiammaCarioca]

Nice video but it would be great to have an article to accompany it

[@vikasrai1316]

Hey,Please make a video on how to maintain data consistency across all collection and documents.I found write batch for this problem but there is limit of only 500 per one write batch. What if i want to write more than 500 at once.

[@DenisTRUFFAUT]

NoSQL implies you, the dev, handle manually data consistency at write time. Once you edit a record, you have to apply changes on all collections where it is referenced. That said, you really don't want to use foreign keys, which is an artefact of SQL. Generally; you separate stuff and you end up with top level collections such : /users/{userId} describing your user, or /friends/{userId} describing the friends of your user, or /messages/{userId} describing messages of your user.

[@vikasrai1316]

Yeah cool thanks.. But still there is a limit of 500 write batch at once what if there are more than 500 duplicate Data and i want to update them, i can only update 500 at once not more than that.

[@michaeloosthuizen2383]

I found that splitting my triggers across multiple functions helped a lot with that limit. Of course if you still have such large batches then write all updates to an array then step through the array to create separate batches of 500 each and then use Promise.all() to run them.

[@princehamza890]

Absolutely Great Video ! Firebase is Awesome

[@mikemagss]

Is it possible to write security rules for arrays or is a sub collection the only way?

[@michaeloosthuizen2383]

A rule applies to an entire document (ie ALL its fields including arrays) so if you needed to protect data differently to the rest of the document then I would put it in a sub collection.

[@zizaart1199]

Clarified a lot of things, tysm

[@lukecross5439]

If a document has n sub-collections, can this document reach its maximum size?

[@Frank_van_Puffelen]

The size limit on a document applies to individual documents. The subcollections under a document do not count towards the size of that document.

[@lukecross5439]

@@Frank_van_Puffelen So, subCollections can serve as subFolders

[@pkop4]

If array_contains is like map field === true, why can we only query on 1 value at a time, and not multiple ?

[@meerimkubatova225]

How to get number value of nested object, I tried this --> Task task=FirebaseFirestore.getInstance().collection("Users").document(userId).collection("Award").document("Kilograms").get(); DocumentSnapshot documentSnapshot=task.getResult(); myValue= documentSnapshot.getLong("Kilograms").doubleValue(); It did not work, please help me

[@JakeBSHere]

Excellent tutorial

[@Firebase]

Glad you liked it

[@thedevguild7525]

Dear Firebase Team, I have a form which collect the -cafe name -latitude -longitude of its location. I would like to save the entry from my form into firebase firestore (using javascript) but have difficulty saving the lat and long as a geopoint (var name is coordinates). Need your assistance on how to code. form.addEventListener('submit',(e)=>{ e.preventDefault(); db.collection('cafe').add ({ name: form.name.value, coordinates: {Latitude:form.latitude.value, Longitude:form.longitude.value} //unable to save lat and long as a geopoint, need advice. }); });

[@TikTok-iv5fe]

hey guys,is there any one have information on flutter on this firestore concept

[@blessing7695]

I was asking myself this question just yesterday bcz we need to scale up our app.

[@praveenpushpkar57]

If a document can have just 20000 fields max, then its a huge problem to something like comments and replies. A document for comments can have fields for each comment with subfields for comment meta data(eg. time,comment text , comment by,etc). Further for each comment field(which is in the form of a map), can have another map for replies. This would easily and in no time consume those 20000 fields. If we use sub collections for each document or even each reply then it would just cost huge for reads as the number of comments and replies grow. Please tell me if i am missing something and the solution for this problem as till now firestore was my favorite, but now i feel i need to switch to something else :(

[@Frank_van_Puffelen]

Also posted as a question on stackoverflow.com/q/62453703, where Doug already responded. Unless you have a different question here, I'd recommend following up on Stack Overflow if you have further questions.

[@shashikantdurge]

#askfirebase Are arrays ordered as they are sent? Suppose I write [ b , a , c] are they stored in the same order or [ a , b , c ] ?

[@IAmSamuelCharpentier]

How do they do these video edits from the future?

[@youmeandeveryone5893]

Ok, wheres the video about collectionGroup?

[@weilieschok2001]

Could you make some tutorial if i need to store data historically? e.g. employee | date | department | job_title E001 | 1-Jan-2018 | D01 | Trainee E001 | 1-Mar-2019 | D02 | Manager department | date | dept_name D01 | 1-Jan-2000 | Sales D01 | 1-Jan-2017 | Sales & Investment D02 | 1-Jan-2000 | Legal D02 | 1-Jan-2017 | Audit * D01 was first introduced as Sales but later renamed to "Sales & Investment" in 2017, same applied to D02 From the relationship above, i know the employee first hired , 1 Jan 2018 and joined in 'Sales & Investment' department and not 'Sales' Then promoted as manager in 1 Mar 2019 and joined "Audit" department and no longer 'legal'. It's all very easy in SQL/RDBMS. But it's quite tough to model it in NoSQL. Hopefully there is some tutorial on these topic. Thanks

[@Shikschauhan]

Is it possible to update a map in an array ?

[@jasonsebring3983]

I keep thinking just use firestore for realtime updates but use another database for real search capability and return ids for firebase to use. Use the right tool for the job but requires keeping data in sync.

[@mika2666]

🤣 the shark tank refrence at the end

[@LuisSierra42]

ikr, that cracked me up

[@danielbutterfield2309]

I would love to have a pint with Todd!

[@kushalbaldev8490]

Anyone here who are struggling to get firebase working in angular 10? getting this error This likely means that the library (@angular/fire/firestore) which declares AngularFirestore has not been processed correctly by ngcc, or is not compatible with Angular Ivy. Check if a newer version of the library is available, and update if so. Also consider checking with the library's authors to see if the library is expected to be compatible with Ivy.

[@johnbalvin5401]

Good video, now let's wait 3 moths for the next one :O

[@johnbalvin5401]

@@ToddKerpelmanCorp it was all a lie :C

[@johnalto6517]

this guy is awesome

[@stofstik]

Very helpful vid! But only one write per document per second... WAT

[@Algardraug]

So... is the Charles Dickens app market booming yet?

[@orcun481]

@Firebase Where is the documentation of the new array features? I couldn't find in here -> firebase.google.com/docs/firestore/manage-data/delete-data

[@AbhishekKumar-mq1tt]

Thank u for this awesome video

[@Cindorqw]

arrayUnion() still doesnt work properly

[@sveinsoermo9883]

Awesome info...

[@klaudiaadamus3353]

hey, guys, just typing this to say your "subscribe" and "watch more" linked windows are popping up way too early - that may be because of your new content addition ;) anyway, a great video collection, thanks!

[@oghenefokethompson7407]

NICE TEACHING

[@i_am_reshad]

wpw great explanation 4:29