I am interested in data staging and i would love to know how it really works. Do you have like a video or link where you showed how to stage a data ?
@LeapFrogBIАй бұрын
Hi, I don't think we have the content you are looking for, but you might search for "LeapFrogBI An Introduction to Data Solutions". This article has outlines the data solution pyramid which includes staging.
@kiranvenna3 ай бұрын
This is an Amazing series. Great going @leapfrogBI
@kiranvenna3 ай бұрын
Excellent information and very relevant.
@LeapFrogBI3 ай бұрын
Thank you!
@nhatthuonghuynh96044 ай бұрын
Hi, thanks for the informative video. I have a question: You mention that we can track the changes of the customer by connecting primary and mini dimension table to a factless fact table. So that means the data model will have 2 fact tables and 2 dimension tables: the transactional fact and factless fact tables, and both primary and mini dimension tables will be connected to these two fact tables?
@paulbfelix4 ай бұрын
There are multiple ways to design the model, but in the video, I proposed creating a factless fact between the primary and mini dimension. You would end up with a fact table connected to your primary dimension. Then, you have a factless connected to your primary & mini.
@nhatthuonghuynh96044 ай бұрын
@@paulbfelix in this case, isn't the factless table cardinality the same with when you put the primary dimension and mini-dimenstion attributes in the same table, just that it's easier to look at?
@paulbfelix4 ай бұрын
@@nhatthuonghuynh9604 Yes. The factless will match the mini dimension grain. But, it would have 2 foreign keys; one to the primary dim and one to the mini. This approach is only appropriate in situation where the primary dim is wide and the rapidly changing attributes are causing the dimension to grow quickly. By separating the rapidly changing attributes into a mini dim and adding a factless fact, we prevent the wide dim from growing rapidly.
@nhatthuonghuynh96044 ай бұрын
@@paulbfelix thanks a lot for your reply. That raises 3 more questions from me, hope to hear your opinion: 1. If I follow this model, isn't it harder for me to link the changes in mini dimension to fact table. For example, if I want to track how the change in Income level affected the spending amount of a customer? 2. If the factless table matches the mini dimension grain, does it mean the factless table only tracks the changes of mini dimension? Changes in primary dimension are retained in its own table (SCD2)? I think that will be more complicated because we need to make sure tthe effective day of any rows in the factless table overlaps with the effective date in the primary dimension table 3. In this model, why don't we connect the mini dimension to the fact table as well? By doing that, we can track the changes of the mini dimension in the fact table, and also track how it changes with the primary dimension
@Devil_Dog_Ultra4 ай бұрын
In a few years everyone will be dying to go back on prem.
@PaulBFelixAdventures Жыл бұрын
6 years later and KZbin shows me my video in my feed. Ha. Regardless, I still think failure to plan for change is the single biggest reason for DW failure.
@RimKhelifa Жыл бұрын
عمل جيد شكرا لك افدتني كثيرا
@titanium7851 Жыл бұрын
thanks a lot
@gkool46552 жыл бұрын
Bravo ... nice work Bruv. My thanks to you
@oscarestorach61622 жыл бұрын
great content, thanks a lot
@arturbarkou63472 жыл бұрын
Really helpful and easy, thanks!
@junaidsediqi94173 жыл бұрын
MashaAllah, Simple and Nice Video
@LeapFrogBI3 жыл бұрын
Thank you @junaid Sediqi. This an old video, but still relevant.
@Christi8403 жыл бұрын
Very interesting video. Thank you for putting this out there.
@misdeed-cs3 жыл бұрын
you are the best ! thank you
@LeapFrogBI3 жыл бұрын
Thank you for the feedback @pouria shahrabi. Much appreciated.
@adityaverma57643 жыл бұрын
Awesome!
@farjanamammi84953 жыл бұрын
the best lecturer
@LeapFrogBI3 жыл бұрын
Thank you @Farjana Mammi.
@malindupabasara14883 жыл бұрын
thank you sir.❤️
@LeapFrogBI3 жыл бұрын
Thanks for watching
@thepickicool973 жыл бұрын
talk faster, i had to speed up the video
@LeapFrogBI3 жыл бұрын
Thanks for the feedback. I'll try to pick up the pace.
@SagarKumar-db2xy3 жыл бұрын
Great Content.
@EricLaermans3 жыл бұрын
Anno 2021 still holds true! Good video.
@Reginaldpenguins3 жыл бұрын
Thank you very clear, this helped a lot
@LeapFrogBI3 жыл бұрын
Thanks for watching @Nick Hilton. I appreciate the feedback.
@rameshakustagi51893 жыл бұрын
Yes, Agile is best suited for BI development process. but challenge is client need to be trained properly to understand their priorities that fit in Agile
@P_Belle2 жыл бұрын
@Ramesha - training line and mid business side managers who don't "appreciate" data disciplines is an entire project itself!
@jaeycc3 жыл бұрын
This was an excellent series. Concise and easy to follow. Enjoyed every bit of it. Thank you for making this available.
@LeapFrogBI3 жыл бұрын
Thank for watching and thank you for the feedback. I'm glad you found this series useful.
@voiceover21913 жыл бұрын
I disagree with equating data warehouse development to BI. A data warehouse can be a source for BI, but so can streaming data or Big Data for that matter. Data Warehouse development doesn't lend itself easily to an Agile approach as it is a type of project hard to devide into small chunks that will fit into a sprint and also have a working version at the end of each sprint to test. As for testing/deployment, having DevOps CI/CD in place should be a requirement these days, but most companies are not ready for it. A data warehouse development needs in depth documentation, which is not something an Agile approach typically allows for, which will trip you up further down the road as I've experienced many times. The trick obviously is a balancing act of having a waterfall approach but at the same time make the initial phase as iterative as possible from both IT and Business side of things, which, I mean, is a no brainer for any sort of chance of success. The risk that during design phase requirements will have changed by the time you start developing is a real one of course, but Data Warehouse Automation would be a possible solution to that. Starting with a pilot is usually the way it is handled Once BI enters into the fray and I do think indeed that's where I agree with your assessment, an intense iterative approach, preferably by using an Agile/Scrum methodology would be the way to go for the reasons you mentioned. With BI I mean the presentation layer of the system, either Reporting environment or Analytics environment or both, support by an abstraction layer between the data warehouse and the presentation layer, which can be established in various ways, from data marts, to using (materialized) views, data lakes, cubes etc. each having their pros and cons. Besides, at this point data warehouse is boohood in favour of data virtualization, with which I totally disagree, but that is another topic. I'd also advice to have proper data governance in place, including master data and reference data management, data lineage, data quality and data catalog as the lack of these is also a huge cause of problems, especially when it comes to change and incident management, but I realize this cannot be all done at once, but it should be on the roadmap.
@LeapFrogBI3 жыл бұрын
Thanks for the feedback. "Data Warehouse Automation would be a possible solution". Couldn't agree more. It is practically impossible to interatively develop a data warehouse with any measure of efficiency when manually coding. Our DWA platform is key to delivering the value that BI solutions can offer. Business users may not know or care about DWA, but they certainly understand the difference between a change taking 1 day and 1 month. LFBI is focused on BI, but I certainly agree that there are many uses for a DW outside of BI.
@JJ-jy8wc3 жыл бұрын
Type 3 missing ? :)
@LeapFrogBI3 жыл бұрын
Hi @JJ. Yeh, there are quite a few more types that are not covered here. Wiki has a pretty good overview of all types. We just covered the most commonly used types in this short video.
@Flarvain3 жыл бұрын
Hmmm, i've tried to copy the code out through the video's but it looks like there are several methods that aren't shown (WriteJob, CreateJob etc etc) which means i can't finish the application. There's also some changes to System.Data.SqlClient which require you to go through EF6. Is it possible for you to package up and share the source code for the console application? This suits the need I have at the moment to a tee. Happy to reach out via email if you'd prefer. Thanks!
@LeapFrogBI3 жыл бұрын
Hi Flarvain. Please send an email to paul at leapfrogbi dot com.
@VictorSouza-iw9ru3 жыл бұрын
is this video still up to date on the field?
@LeapFrogBI3 жыл бұрын
These concepts are well established and are very much relevant today.
@Flarvain3 жыл бұрын
This will be interesting, i hope this process is still relevant i had a flick through the series and it's in depth. Hoping to follow it and reproduce it from start to finish. Appreciate you taking the time to make such thorough content.
@LeapFrogBI3 жыл бұрын
Hi @Flarvain. We recently stopped using the KZbin API, but up until the time that we turned the feed off the process was running daily with no issues. I suspect that this content remains mostly relavent.
@Flarvain3 жыл бұрын
@@LeapFrogBI Thanks mate, it says the resources are in your for this project. would you be able to link to it, i've tried searching on the video title
@blobbyflobby67523 жыл бұрын
Great interview question! Lol
@joshuac.abella82503 жыл бұрын
This is helpful thank you
@paulbfelix3 жыл бұрын
Thank you for the feedback.
@norpriest5214 жыл бұрын
Great explanation
@paulbfelix4 жыл бұрын
Thanks for watching & commenting @Nor Priest
@olcatgaming8594 жыл бұрын
Me who watched it in 1.25 speed Build a Data Mart in 11 Minutes
@kylebecker50834 жыл бұрын
That was a great explanation! Thank you.
@PaulBFelixAdventures4 жыл бұрын
Thanks Kyle
@vaibhavkumar384 жыл бұрын
If your dimension is growing almost as fast as the fact then we may have a degenerate dimension. Good point Also the DD gets its origin in the source system. not in the data warehousing system
@paulbfelix4 жыл бұрын
Good points. Thank you
@JD-ne1cn4 жыл бұрын
Best explanation yet
@PaulBFelixAdventures4 жыл бұрын
Thank you!
@rugaremukanganga22674 жыл бұрын
Not sure if this question makes sense but for a joined fact and dimension table how is the joined table's primary key chosen when the two tables' primary keys are not the same?
@PaulBFelixAdventures4 жыл бұрын
I'm sorry, Rugare, but I don't understand the question. Can you explain in more detail what you are trying to accomplish, please?
@rugaremukanganga22674 жыл бұрын
Paul Felix let’s say you ran a query that puts together two tables. What determines which column becomes the first column for the new table
@BenWann4 жыл бұрын
Great video. Thank you for putting this out there.
@paulbfelix3 жыл бұрын
Thanks Ben
@vijaypalmanit4 жыл бұрын
"surrogate key has no business meaning" concluded all.
@norpriest5214 жыл бұрын
But says there's fact and dimension tables. Which key is better as a foreign key in the fact table?
@suria33294 жыл бұрын
Your explanation was simple and clear. Thank you 😊
@PaulBFelixAdventures4 жыл бұрын
Thank you for watching @Suri
@titantic91604 жыл бұрын
But isn't the mini dimension no longer SCD2 where before it was?
@PaulBFelixAdventures4 жыл бұрын
The mini dimension itself is not SCD2. However, the fkey in the primary dimension may be tracked SCD2.
@titantic91604 жыл бұрын
@@PaulBFelixAdventures Thanks. But then why move the attributes to a mini dimension. What's the benefit?
@PaulBFelixAdventures4 жыл бұрын
@@titantic9160 Good point. It depends on how many attributes you moved to the mini-dimension and how wide your primary dimension is. If you track the fkey scd2, then at least you are replacing the rapidly changing attributes with an efficient key, but you are correct that this will cause the primary dimension to continue to grow. Not a perfect solution for sure.
@titantic91604 жыл бұрын
@@PaulBFelixAdventures Thanks for your reply. I wasn't going mad then :)
@titantic91604 жыл бұрын
@@PaulBFelixAdventures I suppose the dimension now takes up less blocks so is easier to scan....
@marcosoliveira87314 жыл бұрын
I've really learned somenthing here. Thank you.
@GenerativeAI-Guru4 жыл бұрын
Perfect!
@paulbfelix4 жыл бұрын
Thank you Tushar S
@joeeeee87384 жыл бұрын
Amazing SERIES. It's been a long time but I hope to see more series like this in the future
@paulbfelix4 жыл бұрын
Thank you. I hope to get back to video making as well.
@billybee4 жыл бұрын
this video deserves so much more views... thanks sir!
@paulbfelix4 жыл бұрын
Thanks Billy
@cj007854 жыл бұрын
Very good video, thanks for sharing your knowledge
@trompeteandos4 жыл бұрын
Every time you get the reports, you delete and fill the object table with new videos?
@paulbfelix4 жыл бұрын
Can you elaborate, please? I'm not understanding your questions.
@trompeteandos4 жыл бұрын
@@paulbfelix Thanks for answering me and sorry for the inconvenience. The doubt is the next one: Every day, when you obtain the available reports, you may have uploaded new videos to your channel. So the question is, the "objects" table where you store the titles of the videos "delete" and "fill" each time you request the reports or is there a field that tells you if any element has been modified?
@paulbfelix4 жыл бұрын
@@trompeteandos That is correct. You will need to maintain the objects table. Trunc and load is probably the best way to go. In my case I'm loading a data warehouse which maintains any deleted videos for historical purposes.
@trompeteandos4 жыл бұрын
Hi!! Could you tell me how you control the quota of free requests that you have with the API?
@paulbfelix4 жыл бұрын
Hi, I don't think you have any control over the quota. I think Google manages that part.
@trompeteandos4 жыл бұрын
@@paulbfelix My problem is that in the first tests, only with calls to the DATA API to get video titles and lists I am exceeding the quota .... (about 1,000 videos and 160 playlist)
@paulbfelix4 жыл бұрын
@@trompeteandos I have not run into that issue so I'm not much help here. You will probably need to do some research on the quota limits. Perhaps you can break the request up into smaller pieces? I'm sure there is a way to iterate through the request without exceeding the quote, but I don't have experience with this personally.
@trompeteandos4 жыл бұрын
On your website I do not see the resources of the example followed in the video. Are they public? Thks.
@paulbfelix4 жыл бұрын
Hi, Please let me know which resource you are looking for, and I'll get the exact URL. Thank you.
@trompeteandos4 жыл бұрын
@@paulbfelix I'm doing proof of concept but it only covers the download of the CSV generate by the jobs. And I would like to know more detail about the procedure that you follow to synchronize the request of the report.
@paulbfelix4 жыл бұрын
@@trompeteandos The next video in this series should provide the information you need. Look for KZbin Analytics in Power BI - Retrieve Data.
@BlackmetalSM4 жыл бұрын
I didn't know that the AVGN was also a data expert!
@paulbfelix4 жыл бұрын
Good one!
@alaaalsaiery34914 жыл бұрын
Thank you so much that was helpful. Does the OAuth 2.0 Authorization mean that I need to have the username and password for the youtube account in order to get the report?
@paulbfelix4 жыл бұрын
Yes. You will need authorized credentials. Video 2 of this series will explain in more detail.
@alaaalsaiery34914 жыл бұрын
@@paulbfelix what about the public API ? do we need the authorized credentials to use it ?
@alaaalsaiery34914 жыл бұрын
@@paulbfelix Also is it possible that we use the authenticate credentials at the beginning only? in our case, the team who owns the account can't share the credentials with us for risk management purposes however they can share it for us just to use the API at the beginning. would this work ? will I be using the password at any time in the future?
@paulbfelix4 жыл бұрын
Alaa Alsaiery yes. You will need to have credentials to get setup, but your ongoing access will not require that you login with a user’s credentials.
@alaaalsaiery34914 жыл бұрын
@@paulbfelix Is it possible that the account owner could enter the password remotely through an online meeting (of course with the remote desktop control )? Would it work ?