If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k
@anishninan83742 жыл бұрын
The best part about databricks is that is unifies batch and streaming workloads. Also provides a single source for structured, semi structured and unstructured data.
@SeattleDataGuy2 жыл бұрын
Yes those are all a lot of reasons why I do like databricks. Especially their streaming functionality.
@rachidt27642 жыл бұрын
Would be cool to see a mini project in databricks where you could compare and highlight why you don't see it as a data engineering / BI first tool
@SeattleDataGuy2 жыл бұрын
Yeah I will likely do a comparison between this and foundry
@gardnmi2 жыл бұрын
I've used both. Snowflake is toast. Serverless Clusters will take away the pain of managing clusters and they are making really fast improvement on delta lake which will reduce a lot of the common pains of filter, joins, and updates in spark.
@SeattleDataGuy2 жыл бұрын
Oh boy! I want the competition though. The users win in that world
@Practicalinvestments Жыл бұрын
@@SeattleDataGuy this is all foreign language but I am an investor who’s been very closely watching this company and it sounds top notch from what you say
@jaserogers9978 ай бұрын
This didn't age well.
@BuyNLarge_5 ай бұрын
🎯 Key Takeaways for quick navigation: 01:36 *🚀 Databricks offers managed Spark services along with other tools like Delta Lake and MLflow, providing options for data processing and model deployment.* 03:56 *🏠 Databricks and Snowflake both promote the concept of data lake houses, combining data warehouse and data lake functionalities, but with different emphases on use cases.* 05:37 *🛠️ Key components of Databricks include workspaces, notebooks, tables, clusters, jobs, and libraries, providing an integrated environment for data processing and analysis.* 09:33 *📊 Databricks simplifies the transition from notebooks to production by allowing users to create jobs directly from their notebooks, enabling seamless integration and scheduling.* 11:13 *🌟 Databricks facilitates easier productionization of data science workflows compared to alternatives like Snowflake, with integrated features like job creation and version control.* Made with HARPA AI
@SeattleDataGuy5 ай бұрын
thanks for this!
@kerimsever66742 жыл бұрын
Transitioning into using Databricks and this is a great introduction!
@hughesadam878 ай бұрын
Super helpful - have been using databricks in another system without ever really understanding how much of that other system was simply databricks.
@SeattleDataGuy6 ай бұрын
Glad it helped!
@dahof27896 ай бұрын
Wanted to watch but can't filter that goofy background noise.... Everyone does it and it adds negative value to the listener.
@Skandawin783 ай бұрын
after reading this i can't listen to his voice
@codestrap80312 жыл бұрын
Awesome breakdown Ben. I'm looking forward to a comparison with Foundry. My personal opinion is Databricks has the edge when it comes to their pay version of Spark and Delta. It's hard to do a direct head-to-head though because many of Foundry's overlapping capabilities are not documented. IMO Foundry has much better e2e capabilities with a built-in version control system, online IDE, CI/CD, and Data Apps/ML/AI tools. I'm a huge fan of both companies and really think these are the two players that will be left standing at the end of the Big Data OS wars.
@SeattleDataGuy2 жыл бұрын
Glad you enjoyed it! I think I might finally be able to start filming next week for pltr
@MichaelStephenLau2 жыл бұрын
@@SeattleDataGuy Please do help us understand better from a technical/professional perspective on how Palantir solutions stack up against others in the market (Databricks, Snowflake, AWS, Google, etc.).
@devencareer9 ай бұрын
Your approach in this particular video, is simple and precise, not deep, just right enough. Thanks! Devendra
@SeattleDataGuy9 ай бұрын
Thank you! You're too kind
@jackgolding4235 Жыл бұрын
Had this on while I was working and it just clicked to me, thank you!
@SeattleDataGuy Жыл бұрын
Glad to hear that!
@NroShock2 жыл бұрын
Great video! Would love to see another one on Databricks; moving raw data from blob storage, transforming and storing in databricks tables
@SeattleDataGuy2 жыл бұрын
That would be a great video!
@akshaybaura2 жыл бұрын
great starter !! it'd be interesting to dive a little deeper into delta lake file format and also compare it with iceberg or hudi formats i.e. where they are similar, different, which situations suit one best over the other.
@SeattleDataGuy2 жыл бұрын
Sounds like a great video. Let me see if I can get Ryan Blue in the video.
@ShashankData2 жыл бұрын
🔥 video! Learning so much from your vids man. I’m not really getting the effective difference between DataBricks and Snowflake.
@SeattleDataGuy2 жыл бұрын
Filming that video now :)
@ShashankData2 жыл бұрын
@@SeattleDataGuy woohooo looking forward to it!
@severtone2632 жыл бұрын
You earned my sub. I am all in. Thank you for this, it was a great help!
@SeattleDataGuy Жыл бұрын
Glad you found this helpful!
@artandrock4all2 жыл бұрын
really cool all around video, would love to see more videos on databricks where a more deep dive analysis would be given on each topic ;)
@SeattleDataGuy2 жыл бұрын
I will add it to the list of future data engineering videos
@DatabricksPro8 ай бұрын
This channel is great, but you may also check mine. Cheers.
@alineremy4273Ай бұрын
Super! Thank you so much for the video!
@SeattleDataGuyАй бұрын
You’re welcome!
@hotpeppermovie2 жыл бұрын
Databricks is awesome. Its so easy to work with
@SeattleDataGuy2 жыл бұрын
It really is!
@zonezero32903 ай бұрын
Thank you for sharing this!
@SeattleDataGuy2 ай бұрын
My pleasure!
@edwinfokobo56808 ай бұрын
I love this an would love to learn more and understand what’s needed as a prerequisite before I can get a job
@surfh3r011 ай бұрын
I'm new on the subject, well explained!
@SeattleDataGuy10 ай бұрын
glad you found it helpful!
@JohnNettuno19 күн бұрын
Very clear -- very nice presentation
@mashagalitskaia86425 ай бұрын
a really cool introduction, thanks a lot!
@SeattleDataGuy2 ай бұрын
Thank you! Glad you liked it!
@Tic4554420 күн бұрын
Thanks for your job Ben ;)
@TheAndrewjoynson Жыл бұрын
brilliant intro - well done and and surprisingly i understand a lot of this. (im not a dev nor a data engineer)😄
@keviny9392 Жыл бұрын
Excellent content. Exactly what I needed to get started. Thanks
@aiautoglasscrm Жыл бұрын
Subscribed. When you say productionize your job, I thought I was going to see API endpoints to hit to get results for example a regression price prediction model that gives you a number after inputting variables
@venkateshkothapalli2 жыл бұрын
Love Databricks!
@SeattleDataGuy2 жыл бұрын
Seems to be a decent amount of love for it!
@nujanai2 жыл бұрын
Great overview. Thanks!
@SeattleDataGuy2 жыл бұрын
You're welcome!
@horaciosoldman44812 жыл бұрын
Thanks for this informative video Ben 🙌
@SeattleDataGuy2 жыл бұрын
Glad you found it informative!
@tallalmoshrif6643 Жыл бұрын
Great content as always. Thanks for sharing.
@raphaeldwain78342 жыл бұрын
Very useful. Thanks.
@SeattleDataGuy2 жыл бұрын
You're welcome!
@nguyetdang1115 ай бұрын
Is Databricks different from Azure Databricks?
@femaledeer Жыл бұрын
Video didn't explain what databrick does. When was table show being built ?
@DavidKoleckar4 ай бұрын
that's high quality vid, thx :]
@SeattleDataGuy2 ай бұрын
Glad you liked it!
@mikenashtech Жыл бұрын
Great vid Ben. Like the way, you give tips with commercial thinking behind it. Thanks Mike
@AliTwaij2 жыл бұрын
Thank you
@SeattleDataGuy2 жыл бұрын
You're welcome!
@marklambrecht Жыл бұрын
We need to have you do a video about SAS Viya too!
@stelluspereira9 ай бұрын
Enjoyed our video, thx Do you know how to install databricks &PySpark LOCALLY( on laptop ) & code & test locally Perhaps a video will be appreciated by the community WITHOUT depending on AWS/Azure
@ganonymous84482 жыл бұрын
Great product overall, but sucks that you can’t use Airflow on it.
@SeattleDataGuy2 жыл бұрын
I could have sworn I saw a partnership between them and astronomer
@fenderbender282 жыл бұрын
Their Workflows orchestrator got super powerful and is easier than airflow imo
@wrburggraaf2 жыл бұрын
How would you compare this to SAS Viya? It seems like this is more for building a data lakehouse whereas SAS is primarily for data analysis and analytics (so it might connect to data bricka to get the data). Could you also do data analysis and analyrics well in data bricks?
@manticomar1146 Жыл бұрын
Good work
@zacharythatcher73282 жыл бұрын
To me this just looks like a great way to encourage developers to deploy untested code. Are there any testing pipelines built in that prevent job deployment prior to passing tests?
@gautam3305 Жыл бұрын
Testing in data field is completely not like typical software engineering cicd unit testing
@mikipatel243411 ай бұрын
Great video ruined by the annoying background music. The background music was really distracting and annoying. But very good information. Thank you
@cerberus132111 ай бұрын
Its the same as Domino
@SeattleDataGuy6 ай бұрын
there are some similar features but it is kind of an apples and oranges comparison
@VicusBass Жыл бұрын
Whenever it's Databricks there a Romanian around :)
@JimRohn-u8c2 жыл бұрын
I hate Databricks, would rather use Snowflake.
@michaeld96822 жыл бұрын
Why?
@SeattleDataGuy2 жыл бұрын
I would also love to know why!
@sevegarza2 жыл бұрын
Agreed. Snowflake + Prefect = 😇
@JimRohn-u8c2 жыл бұрын
@@SeattleDataGuy a couple reasons: 1. A lot of times when I’m building a data pipeline there’s a lot of SQL queries I need to write to just analyze the data before I start creating certain metrics and to just start understanding the data. Because of Databricks 1000 row limit this is harder to do. If those tables were in a RDBMS or Snowflake I wouldn’t feel as hindered with regards to this very common task. I know it may seem weird to some people but sometimes just being able to see more of your data and scroll through just helps; maybe this is a Junior Data Engineer thing idk. 2. Idk if this is easier in Snowflake but passing a Parameter from a widget into SparkSQL was a pain in the ass in Databricks, the only reason I figured it out was because a notebook written by someone else did the same thing. We use Azure Synapse notebooks and I like that much more than Databricks as well; it was easier to do some of the same things.
@gautam3305 Жыл бұрын
@@JimRohn-u8c I understand data discovery is key aspect before modeling, but that can be achieved by using groups by, limit, distinct, windowing etc, don't need to print million rows and export to excel for that.