How Instacart Optimized Snowflake Costs by 50%

[@olgashevchenko8439]

Cool insights, thanks

[@MattRomano88]

Are you able to post the link to the open source DBT package for monitoring costs at the query level? Specifically the slide that appears at 8:42

[@snowflakedevelopers]

Hi @MattRomano88, thanks for your question! Yes, here's the link to the package: github.com/get-select/dbt-snowflake-monitoring

[@MattRomano88]

@@snowflakedevelopers Awesome thanks for the quick response!

[@colouredlaundry1165]

where can we find the slide deck?

[@cheapdbs]

Can you please provide some context on the 56% savings in 2022 as mentioned on the slide at 03:44? According to the Instacart S-1 filed with the SEC in August 2023, on page 280 it states that Instacart paid "approximately $51 million" to Snowflake in 2022. Does the 56% savings applied to this $51M overall cost of Snowflake in 2022, meaning otherwise it would have been $116M, or does the 56% savings applied to only a single or few warehouses, or somewhere in between? If the latter how many of the 200+ warehouses was the saving applied?

[@nickakincilar3757]

I believe $51M was a multi-year contract where they could burn that amount in any yearly volume as they wanted (for example 17M, 17M, 17M OR 30M, 11M, 10M OR 25M, 15M, 11M & etc.). Fairly certain 56% savings is compared to their initial yearly run rate when they were just building pipelines & products as fast as they to meet the fast growing market due to covid with little time to optimize. Once they were able to scale fast, meet the market demand , capture marketshare & finally stabilize their workloads, they went back to existing pipelines and started optimizing them. Once they were done, they were able to achive 56% savings over their initial data products/pipelines which were deployed very quickly & w/o optimizations to meet the fast growing demand.

[@snowflakedevelopers]

Some additional context may be found in the blog as well: www.snowflake.com/blog/snowflake-and-instacart-the-facts/

[@cheapdbs]

@@snowflakedevelopers Thank you! So then I must adjust this question with the updated information that the actual usage spend was $28 million in the 2022 calendar year. Does the 56% savings of this overall cost in 2022 mean that otherwise the spend would have been $63.6M without the optimization? Therefore the optimization has a huge impact in saving unnecessary costs and we should all have extra resources dedicated to tuning and workload management on every warehouse to keep on top of costs as a priority. Additionally, does the 227% cost impact without optimization represent a common issue that is reflected by the comment made at 00:15 that Snowflake "has a reputation of getting really expensive really fast"?

[@cheapdbs]

@@snowflakedevelopers Actually the context you provided in the referenced blog provides the answer. According to the blog "the actual usage of Snowflake (Instacart’s expenses) was $28 million in the 2021 calendar year, $28 million in the 2022 calendar year" which means that the usage expense was flat from 2021 to 2022, therefore there was no reduction in overall usage costs from 2021 to 2022. So to have reduced the "overall Snowflake cost by 56% in 2022" would mean that the optimization prevented a potential cost of $63.6M (28 / (1 - 0.56)) or cost avoidance of $35.6M (63.6 * 0.56) in 2022 to arrive at an overall usage cost of $28M (63.6 - 35.6) in 2022 to keep actual usage costs flat from 2021 even though there was an expansion of use cases and workloads. Conversely it prevented a 227% cost increase in 2022 over 2021. Can you please confirm that this is how the reduction of overall Snowflake cost by 56% was measured or provide the accounting method on how the 56% reduction was measured?

[@kunalplaysgames]

Can you post the links to "Useful References For the Query Profile" on 33:48?

[@snowflakedevelopers]

Yes! Here they are: teej.ghost.io/understanding-the-snowflake-query-optimizer/ and medium.com/snowflake/introducing-the-snowflake-visual-table-clustering-explorer-6fbb66a15bd5. Thanks for your comment!

[@nini-vd3gy]

5 million active tables? how many of them contain duplicated data?