Why Data Warehouse Projects are a Bad Idea

A data warehouse offers the benefits of fact-based decision making, and these days nearly everyone agrees on their value. But data warehouse project have an alarmingly high failure rate. In this video we explain why and offer a way you can succeed where others have failed.
You can find our article on data lakes here:
www.leapfrogbi.com/archives/1...
You can find our video series on dimensional modeling here:
• DIMENSIONAL MODELING
Follow LeapFrogBI on:
- Facebook / leapfrogbi-38564604483...
- Twitter / leapfrogbi
- LinkedIn / 2524834

[@EricLaermans]

Anno 2021 still holds true! Good video.

[@markteague8889]

I have really enjoyed LeapFrogBI's concise videos on various data warehousing and BI topics immensely and find them to be the best introductory material on the subject that I have found in the absence of any higher educational offerings within university academic programs.

[@LetsGetItStartedNow]

I’ve been a data warehouse engineer for 25 years and most of the problems is in data modeling, not aware of all the nomenclature lingo and completely lose context of the entities. Also, a data-warehouse without data goveranance is sure to fail. Business rules hard coded and buried in ETL that etl dev only knows and forgets what the code does because he doesnt document it. All that should be abstacted away from the ETL where a data steward owns the business rules layer. Datawarehouse can succeed but its missing a lot of necessary disciplines that are missing in the school of Kimball and Inmon.

[@voiceover2191]

One way forward is to use data warehouse automation, where you can centralize your code and automate generic tasks that without automation typically involves endless coding, often even sparsed out over various system layers. I've had a good experience using WhereScape. It covers automation or providing custom templates to do a lot of generic tasks typically custom designed by developers.

[@albertosimeoni7215]

I find myself in this situation... The BI department was under the controlling department ( no brain requests to be done immediately) ...after the IT department was detached from controlling i join the company ...I find many hard coded filters / conditions and no brain choices only to accomodare single users... It was out of control, full of bug and not documented... I work 90% of time in rewrite procedure, extract lookup tables and simplify the logic... Another problem are the changes to business logic in the ERP systems that many times are not shared to BI team...

[@P_Belle]

@lets- you just spoke my world

[@markteague8889]

I suspect that agile methods might work well in an organization that has no current data warehouse and doesn't have a great deal of experience with BI. On the other hand, an organization that has a long standing data warehouse that is well structured with many experienced team members who are familiar with the business needs / requirements will benefit much less from agile methods. Delivering working functionality regularly to the end user in order to receive feedback and adapt is always valuable. One issue the industry as a whole seems to face is that there is little formal education about data warehousing and BI methods in computer science curricula. Data warehousing seems to be a subject about which one can only obtain knowledge through working experience. I think this is evidenced by the popularity of the Kimball Method. Mr. Kimball's organization fills a role that might otherwise be covered in university graduate courses. But, few such courses seem to exist. It's a similar situation to the emergence of design patterns within the object-oriented community. University course treatment on the subject of object-oriented design patterns seem to be very weak.

[@Christi840]

Very interesting video. Thank you for putting this out there.

[@BenWann]

Great video. Thank you for putting this out there.

[@paulbfelix]

Thanks Ben

[@bidw2gobidw2go78]

I' d been developing data warehouses + BI solution for a long time now and I've discovered the agile approach in the course of our development. Have I known it on my earlier projects, maybe we have higher success rates. Kudos to you!

[@LeapFrogBI]

Thanks! Glad to know you found it helpful, and there is one more agile BI practitioner out there.

[@gopallahoti7469]

Did u complete the project

[@P_Belle]

@@LeapFrogBI - another here!

[@hersil2012]

Excellent video. Its name is perfect to attract attencion.

[@LeapFrogBI]

Thank you!

[@rameshakustagi5189]

Yes, Agile is best suited for BI development process. but challenge is client need to be trained properly to understand their priorities that fit in Agile

[@P_Belle]

@Ramesha - training line and mid business side managers who don't "appreciate" data disciplines is an entire project itself!

[@paulbfelix]

So true. Nicely explained.

[@LeapFrogBI]

Thanks Paul

[@jhr3uva90]

I thought the goal of Agile methods was to deliver something fully functional in each Sprint. How do you do that with a data warehouse? Don't you need to have a data warehouse schema designed up front?

[@adamsmithline]

Great question. The goal with agile BI is to start small, with a single subject area from a single data source, and take it all the way through the dev lifecycle from design through report implementation and validation. Even this can be difficult to accomplish in a single sprint, which is where data warehouse automation comes in. For more information on this topic please see our article here: www.leapfrogbi.com/archives/71004

[@voiceover2191]

I disagree with equating data warehouse development to BI. A data warehouse can be a source for BI, but so can streaming data or Big Data for that matter. Data Warehouse development doesn't lend itself easily to an Agile approach as it is a type of project hard to devide into small chunks that will fit into a sprint and also have a working version at the end of each sprint to test. As for testing/deployment, having DevOps CI/CD in place should be a requirement these days, but most companies are not ready for it. A data warehouse development needs in depth documentation, which is not something an Agile approach typically allows for, which will trip you up further down the road as I've experienced many times. The trick obviously is a balancing act of having a waterfall approach but at the same time make the initial phase as iterative as possible from both IT and Business side of things, which, I mean, is a no brainer for any sort of chance of success. The risk that during design phase requirements will have changed by the time you start developing is a real one of course, but Data Warehouse Automation would be a possible solution to that. Starting with a pilot is usually the way it is handled Once BI enters into the fray and I do think indeed that's where I agree with your assessment, an intense iterative approach, preferably by using an Agile/Scrum methodology would be the way to go for the reasons you mentioned. With BI I mean the presentation layer of the system, either Reporting environment or Analytics environment or both, support by an abstraction layer between the data warehouse and the presentation layer, which can be established in various ways, from data marts, to using (materialized) views, data lakes, cubes etc. each having their pros and cons. Besides, at this point data warehouse is boohood in favour of data virtualization, with which I totally disagree, but that is another topic. I'd also advice to have proper data governance in place, including master data and reference data management, data lineage, data quality and data catalog as the lack of these is also a huge cause of problems, especially when it comes to change and incident management, but I realize this cannot be all done at once, but it should be on the roadmap.

[@LeapFrogBI]

Thanks for the feedback. "Data Warehouse Automation would be a possible solution". Couldn't agree more. It is practically impossible to interatively develop a data warehouse with any measure of efficiency when manually coding. Our DWA platform is key to delivering the value that BI solutions can offer. Business users may not know or care about DWA, but they certainly understand the difference between a change taking 1 day and 1 month. LFBI is focused on BI, but I certainly agree that there are many uses for a DW outside of BI.

[@monopoly2170]

Dumb comment, but I'd recommend a wider lens

[@adamsmithline]

Thanks for taking the time to offer constructive feedback! Video production is not my specialty for sure.

[@monopoly2170]

@@adamsmithline No problem. Great video. Best of luck.

[@karimkussainov3889]

The idea is right. The header is bullshit. I was supposed to find out DW drawbacks. Call the video Agile BI or whatever similar, not the "Why Data Warehouse Projects are a Bad Idea".

[@LeapFrogBI]

Hi Karim, thanks for taking the time to comment and offer your feedback. We were trying to emphasize "projects" here, but perhaps it isn't clear enough. We are huge proponents of data warehouses, but if you would like to learn more about them and their drawbacks then we would be glad to speak with you. Please feel free to get in touch via our contact page. www.leapfrogbi.com/about/#contact

[@voiceover2191]

@@LeapFrogBI He is right, the title obviously suggests that building a data warehouse is a bad idea. What you intended unfortunately is not present in the title, but it's good click bait.

[@LeapFrogBI]

@@voiceover2191 Thanks for watching.