Generally, there's not a hard limit on the size of a partition. DDB will usually keep them around 10GB, but it's not something you need to think about. This is true for main table and GSIs. *However*, if you have a local secondary index (LSI) on your table, then item collections (the set of keys with the same partition key value) cannot exceed 10GB. This is a hard limit and a hard wall to run into! :)

Good question! DynamoDB excels at OLTP-like queries where you're operating on a specific, limited number of rows -- create an Order for this Customer, or view the 10 most recent Orders for this Customer. These queries partition well and are narrowly focused based on fixed parameteres. DynamoDB is not as good at 'table-wide queries', such as 'Give me all the Customers in my application'. There are some ways to do that, but it's tricky. It's usually best done in an OLAP database (if used for internal analytics) or something else like ElasticSearch

Thanks! Glad you liked it :)

I also have the same question!

Thanks, Sanjeev! :)

Thanks, Syed! :)

Are you talking about for a global secondary index (GSI)? The write to the GSI won't happen during the write to your table -- it will happen asynchronously after the write completes. This ensures the initial write is fast and not delayed by propagating to indexes.

@@alexbdebrie how can we model to fetch records based on createddate or enddate. i searched in internet everybody is suggesting to create GSI on dates will it overload the partition ? any suggestions would be great

@@b-or-m-ln6hk Either on your main table or in a GSI you would need to set up a structure where the partition key is what you're grouping by (e.g. CustomerId if customer orders) and the sort key is what you're filtering by (createdDate or endDate). Then, do a Query operation that does an exact match on the partition key and filters on the sort key accordingly.

Email me/ DM me on Twitter! I'm always happy to give a discount :)

Awesome!

Hey Gary! Great question, and glad you're liking the videos. If you do want to implement paging on the related item (e.g., comments), then you probably will want to make separate requests. In that case, you don't need to put them in the same item collection / use the same PK -- you can separate them. In doing so, I'd still try to make it where you can identify the relevant comments w/o needing to fetch the post page first so that you can fetch the two pieces of data in parallel rather than needing to fetch the post first, then follow up for the comments. Let me know if that helps!

@@alexbdebrie it absolutely does and thank you Alex for your generous response! Merry Christmas to you and your family!! I wish the best for you for 2023. You have helped me out immensely with your book, videos etc this year 😊

@@garywaddell6309 Thank you, and Merry Christmas to you and your family as well! You just made my day

That's a pretty tricky pattern. Just to be clear -- you want a list of *all* PK + attribute combinations? To do this, wouldn't you need to see all the data anyway? Table scans aren't efficient, but that's usually b/c you don't want to read all the data. In this case, it seems like you do?

I've got a post coming out on code structure for the Serverless Framework! It overlaps w/ SAM a lot. Will be released in ~2 weeks.

I don't think he does, but you can find him on Twitter here: twitter.com/undef_obj ! :)

Hey Rohit, glad you liked it. Interactions with DynamoDB are over an HTTP API, so your connection will be like other HTTP clients. The big thing here is to ensure you're using HTTP Keep-Alive to reuse the connection across requests.

You're welcome!

Great to hear, Joe! If you want a complete guide to working with DynamoDB, you can check out my book -- dynamodbbook.com :)

Yes, message me! Email or Twitter DM :)

@@alexbdebrie thank you so much Alex! I will!

Thanks, glad you liked it!

Thanks, Bahar! For your question, will these each be different items in your table? If so, the answer is yes -- you can have different items with different sort key patterns. Let me know if I'm misunderstanding -- feel free to email me or DM me on twitter.

For this example, you could use the DynamoDB Scan operation to handle that. In general, Dynamo doesn't want you to fetch data from across your entire table. It's assuming you'll want to break it down by something (e.g. CustomerId, City, etc.), and order the results (e.g. by OrderDate, ZipCode, etc.). Hit me up if you have any examples / questions -- can try to answer those :)

Thanks, Raphael! Glad you liked it :)

Any requests?? Always looking for topic ideas :)

@@alexbdebrie Maybe a video in which you go through how you would structure your dynamodb tables giving some non-trivial (yet a common) situtation (e.g. a dating website). Each user have a list of images, chat messages, comments, e.tc.

@@asdfasdfuhf Good call! I have some like this already. :) You can see an e-commerce example here: kzbin.info/www/bejne/pKSke6CvnJeEi5o&ab_channel=AlexDeBrie Also, that example is from my book, which has a bunch of data modeling techniques plus 5 full walkthrough examples: www.dynamodbbook.com/

Thanks, Rohit! For the migration, make sure you plan out your data model with DynamoDB and how your RDBMS data will get to that state. The number of columns shouldn't be a big deal as long as you're not exceeding the item size limits.

@@alexbdebrie Thanks!

You're welcome!

Thanks, Ori! This is from The DynamoDB Book. You can get the source code with the Plus or Premium package, and this video (plus videos for all chapters) are available in the Premium Package. www.dynamodbbook.com/

Hey, not sure I quite understand your question but yes, the partition key of a GSI generally acts similarly to that of the main table. Thus, you want to avoid using the same partition key for all items.

@@alexbdebrie Yes, thank you a lot!!! That's what I actually mean

Great to hear! I generally recommend a similar philosophy with GraphQL. It can add complexity to your resolvers but will be more performant. That said, I've seen many folks that have opted to use more isolated resolvers and a waterfall strategy for resolving a big graph in a GraphQL query, and that can work as well.

Hey there, it's a good question and one I get a lot. I need to write up the different options on these. It's tricky because you can give them the same partition key in a secondary index, but that can often result in a hot partition that hits partition throughput limits. What is the use case for this? Is this a production, user-facing use case, or more for internal processing?

@@alexbdebrie we are designing it for production from scratch and we had seperate table earlier but now we moving on single table design. We are able to cover almost all use cases but this get all customer, get all orders and there sub tasks are difficult to approach. It would be really great if you can put some light on this. Thanks you so much alex for your prompt reply btw

@@shailendraacharya I probably won't have a blog post in the next week or so. If you send me an email ([email protected]) and tell me the basic overview, I can point you in the right direction.

@@alexbdebrie thank you so much alex and I really appreciate your prompt response for helping me in design db. I will write you email today with all details.

@@alexbdebrie I have sent email with all details.

Thanks, Karim! There's no out-of-the-box way to get all of the primary keys in the table. You could scan over the whole dataset (which could be enormous), or you could maintain a list of the primary keys yourself.

Hey there! The read is only considered one read -- directly from the GSI itself. For the writes, you will pay both for the write to the main table as well as to any secondary index. Thus, if you have two GSIs that your item would be written to, you would pay for 3 writes (main table + 2 secondary indexes).

You're welcome, Dev! Glad you liked it.

Hey David, yep you should be able to model that scenario in DynamoDB. I often don't model 1 to 1 relationships as a separate entity. Can you combine stream + product group into a single entity?

@@alexbdebrie Hi Alex, I believe so since there can only be one product group inside one stream. This is what I actually got so far. My PK is some globally unique streamID (but for my SK I decided to use my groupID (which I think is incorrect since we can have more than one SK with the same ID given the nature of the composite key and having already set a unique streamID) and the product itself ended up as a list (denormalization by using a complex attribute) where each product item is uniquely identified by its own ID. Ended up having a one, massive item that contains it all. Now the problem is that if I want to retrieve say only my groupId and my groupName (those are the only two attributes this entity originally had) I most likley will have to use a GSI. If there are more than a few scenarios where I need to query this table looking for different items I may end up creating a bunch of indices and that can bring the cost of using dynamo up if I understand billig correctly. What do you think? Any suggestions on how you would go about it? Thank you for yout time.

@@davidhodowany8029 Hey David, I think you're on the right track for some parts. There's a lot here so tough to say without really getting into the specifics :) Have you read my book? I think it can help you grok the mental model of DynamoDB and work through some examples. If it doesn't help you, let me know and I'll refund you entirely. dynamodbbook.com

@@alexbdebrie Sure, I'll give it a go. Appreciate your time, and thank you.

Hey Jishnu, you can't use a complex attribute like a map or a list in your primary key, whether for the main table or GSI. It needs to be a scalar value -- string, number, or binary only.

Hey Abhishek, glad you liked it! Check out some of my other videos for more on data modeling, or check out my book -- The DynamoDB Book.

Good question. Under the hood, the Serverless Framework is generating CloudFormation so I consider that the same. I'm not *that* worried about which CloudFormation tool you use (Serverless Framework, SAM, arc.codes, CDK) as long as you're using some infrastructure-as-code tool to deploy.

@@alexbdebrie got it 👍🏻

Thanks, Andriy!

Thanks! Glad you liked it

Good catch! They don't really tell us (we don't need to know!), but I believe DynamoDB doesn't actually remove partitions after they're created. That said, because we don't need to know, they could change that in the future if needed.

Thanks Rick!

Nope, this is all hidden from you. You can sneakily find out the number of partitions in your DynamoDB table by examining DynamoDB streams, but it shouldn't really help you much. Project to find number of partitions here: github.com/zaccharles/dynamodb-table-partitions

@@alexbdebrie no, I'm actually talking about aws Apache keyspace - 4:33 Is it visible.

@@RahulAhire Ahh gotcha. No, I don't think so.

Haha, you know it