Behind the Hype: Is Data a Product?

The idea of producing data using product management principles isn't new, but it had a huge upsurge in popularity with Data Mesh. This has led to a proliferation of definitions, arguments and confusion around what it actually means to deploy Data Products. When are you treating Data as a Product, and when are you developing Data Products? If we're saying a data model can be a Data Product, is there any difference to what we've been doing all along? Is my dashboard a Data Product? Where does it end?
In this video, Simon opens the can of worms that is Data Product thinking - discussing the direction we've taken Data Products and how we talk about them with our clients at AA. This is just the start of the conversation, there are lots of elements to think about, but we thought it was about time to share where we stand on things.
If you're looking at revamping how you engage the business whilst developing new data models or kicking off a new data platform project and want to make sure it's delivering the maximum value - give Advancing Analytics a call!

[@fb-gu2er]

We tech people like to embellish things with fancy words that in the end is just common sense and many are just aliases of other things. For me the biggest issue is the lack of requirements from the owners/business. It’s amazing how people lack the ability to write good requirements while expecting magic without defining what magic means

[@AdvancingAnalytics]

That's the whole point - if a dedicated Product Owner does anything, it's ensure the requirements are very clearly specified. One of the key takeaways is: "We need one of those, but for data"!

[@NeumsFor9]

@@AdvancingAnalytics All true, yet well all know business domain knowledge in IT is crucial, and most folks need to see something before they can articulate it even in this Generative AI era.

[@fishsauce7497]

Business requirements are always at high level and with assumption you know the Industry Process. Biz folks (unless head of data or BI team itself) can never write complete and correct requirements. Someone in the fields of requirement analysis / gathering should be adept in Business Domain. He / She should gather broad level of requirements at high level and detailed business process, then go back and check whether the data from IT systems speaks of same business process or not. In the second iteration, that's when you have engaged discussion with stakeholders during the session of requirement gathering. You start pulling the ropes, ask intriguing questions which Business and IT Maintenance team will start dreading and formulate concrete requirements. This can go upto multiple rounds or with high experience in domain and Business Analysis skill can come down to fewer and concise meetings. Many times I have seen technical people jumping to gather business requirement and end up documenting Primary Key, Unique Key, nulls and not nulls and what not, instead of "atomic business requirements" with "dependency" and "Glossary" as outcome. As an end result, the system is a dead body of technical rules, without a "functional soul".

[@pesetskyps]

Thanks for bringing this up! 🎉 You put some emphasis on business should own, totally agree and it is missing piece at this point. One more thing that you brought up, but Imho didn't highlight enough: if engineering team goes into data as product journey the commitment items like sla, quality, release process adds roughly 30% to the load of the team. So when talking to the business there is hard conversation should happen: we will be doing less products, but with more quality. Without this piece business will tell: sure lets build the products, here is product owner and over time you will have to have this hard conversation anyway

[@alexischicoine2072]

My current organization exposes data products as views in a separate database and data shares for external customers. Using views is very important to have the ability to change the underlying tables without breaking consumers. SLOs, product ownership, documentation, and a commitment not to make breaking changes without first migrating users are key.

[@paulheadey265]

Totally aligned with this- original thinking of a heterogenous data mesh between different technologies and maintaining all the API's data contracts - is one way to burn through $millions. We have created data mesh using data shares including the full product owner model, SDLC/CI-CD methods with data contract guarantees in place using separate tenants on the same platform. In one case the security model drove the architecture decisions where separate tenants were needed due to the highly sensitive nature of the data. Databricks and Snowflake are particularly geared for this. I agree it's not right for everyone, a smaller company needs one central team to build deploy and maintain its data, but they should look to develop data marts and maintain them with a full product lifecycle management approach. Our next project looks really interesting - building a Snow-bricks mesh using Iceberg tables - this is trickier than I first thought, my team are picking through the intricacies as we speak.

[@peterydethomsen8133]

Thanks for another really great video - love your rants even if I do not disagree ;-) Different companies will end up using the term for different things. We usually end up talking about use cases that is supported by data products, where the product sometimes will be a data model and sometimes dashboards and analytical products. What we see internally is, that data models are combined both within and across domains to create analytical products (dashboards/websites/apis). This is usually what we call data products (but backed by support model, SLAs, documentation, training, security and access control) I truly like the idea about have the business help define the value of the products or features within them. Will make some of the discussions a lot simpler 😁

[@user-lg8ki6yk8r]

A lot of food for thought. Thank you!

[@alkiviadesgeorgakis209]

Happy to hear other views and confirm some of my thoughts as well. Dont know about mainstream, i feel if your business started arround data with strong data teams then y, you are at that level if not you probably still trying to convince people to let Excel go...

[@DanKeeley]

yay! here we go!

[@AdvancingAnalytics]

I was nice in this one - need to find a nice spicy topic to get grumpy about next! :D

[@srbasha74]

I would go one step further and call a Semantic Model (preferably an open/universal one) with predefined join, hierarchies and metrics as a data product. That way, I can fairly standardize the outcomes (like the revenue metrics from your dashbaord wont be different to mine). Thoughts??

[@AdvancingAnalytics]

It depends whether that semantic model is the only access layer for your data product. It's can absolutely be an intrinsic part of the model that you stand up, but not every data product has an accompanying semantic model, depending on org, tools, use case etc, so I didn't want to be too prescriptive with that definition

[@srbasha74]

@@AdvancingAnalytics makes sense. Thanks

[@srbasha74]

@@AdvancingAnalytics There are modern semantic layer products which could serve all kinds of consumption like MDX, DAX, API, jdbc etc, taking the headless BI approach. I personally evaluated AtScale and Kvyos. Both seems to meet the promise

[@NeumsFor9]

Call it what you want, but you cannot avoid the cultural shifts needed to do things efficiently. Bottom line is you need to MARKET your successes like a product. It's important to not spend too much time getting lost in word soup. Kimball called the Product Manager the Business Sponsor..... some call it the Data evangelist or Czar. End of the day data can be viewed as an ingredient to MAKE products, but a constructed product mindset is not harmful as long as it is well understood. In all reality, Kimball's Ten Habits Of Effective Business Sponsors really hit on a lot of the same points needed on how to treat data as a product. Like I always like to say, Simon, if you truly love what you're doing and take pride in the output, most of the points about data as a product are really common sense. It's no coincidence that, back in the day, marketing and sales used to drive the funding of data warehouses and lakes.

[@AdvancingAnalytics]

It's true, but as an industry, we generally fail to get people bought into to the business sponsor role. By embracing the terminology and processes from product management, and by being very explicit about what that role entailed, we've had a lot more success getting folks to actually engage with that role and provide the requirements, prioritisation and ongoing engagement thats needed to make these things successful. We've definitely found switching things to run product launch events, market the use case stories to the business, tell people WHY they should care, it's all had a big impact on data culture - if that's down to some simple renaming and repackaging of things, fine, I can accept that!

[@NeumsFor9]

Oftentimes data leaves behind a popcorn trail of who is and is not doing their jobs well. Cover up cultures are a real and undiscussed topic. Start with the passionate pockets of the company and gravitate to those competitive execs that may have ulterior motives ha.....

[@NeumsFor9]

@@AdvancingAnalytics Don't be surprised to see the Chief Data Officer split off into another position of Artificial Intelligence Officer....before the CDO position becomes fully accepted..... lol.

[@NeumsFor9]

@@AdvancingAnalytics Oftentimes, if not almost always, the best move in getting the right data product manager is having someone internal, with an existing track record of success in the company AND who is also heavily involved in data. They must be cultivated. Unfortunately, some of these folks get passed over when their analyses do not paint the picture which some would prefer to hear. So many companies believe in bringing those people in externally or bringing in either overly academic folks or those with previous success in software/IT but no data experience whatsoever. The smart ones bring in Advancing Analytics mentoring!

[@allthingsdata]

It's a bit scary how much your views align with mine over many things but probably a good sign and likely the reason I'm coming back (to the bubble). We have constant debates over data sharing, data marketplace, data mesh etc. When I read the original data mesh article I came to a very similar conclusion like you. One thing I would like to add is that there is probably a reason why it's called data product, data marketplace, etc. These are economic terms so maybe we should apply economic theory which, among many things, asks for a compensation/reward/incentive to provide a product. I haven't read the literature but I wonder how this fits into the traditional IT setup of a cost center. How would incentive or compensation for data products work. Why would anyone invest in making their data a product (to be consumed by s.o. else) if they are not somehow incentivized. Silos are only removed by carrots and/or sticks and usually it's a bit of both.

[@NeumsFor9]

Exactly. Until incentives are built into the job description, a ton of this discussion just academic theory with very few companies out there having the resources to embark on a campaign to change its culture. Here's the cycle: CEO checks LinkedIn >> Talks to buddies about latest trends >> one out of 100 lucky enough to succeed >> 99 others ask "why aren't we doing this" > 80 of them discover they need to change their cultures and drop the idea >> rinse >> repeat. The idea of data contracts are not new at all. They're just attempting to apply its content in a different area.

[@dmitrym_1]

A data product is not (only) curated data set(s), but all related pipelines, infrastructure, CI/CD, code/data versioning, documentation and curated data set(s) in different formats (not only table) as main interface for communication with other components in your system. It's impossible to think about a data product without relationship to whole system therefore you can't release your data product in vacuum. It doesn't matter what approach you use (data mesh or something other) your data product will be always part of some bigger system. Data Mesh could be useful because it is system approach where data products are only components to make whole system work properly.

[@AdvancingAnalytics]

Whilst I agree that all of the surrounding elements (pipelines, infra, ci/cd etc) to get data to the people who need it are a vital part of that product design, that's not the point I was getting at. The real impact here is the people/process elements of product management, regardless of what elements you class as being part of that data product. That's the bit that's really missing in so many orgs we speak to, regardless of their tech maturity.

[@pakistantourneys2023]

If an org has a lot of semantic models or star schemas. Would you clump them all into one data product or each gets its own product pizzazz? I can imagine managing release notes, change management, SLAs, Backlog, Demos and Successes Q/As for each could be borderline overkill? What are your thoughts?

[@AdvancingAnalytics]

It's a lot of work to implement it retrospectively for a whole host of existing models, but each modelled process is essentially it's own product. Of course there are economies of scale where all models/products around a specific domain are owned and managed by the same product owner. If your platform is engineered with these things in mind, the measuring SLAs etc should be fairly minimal work... Once it's implemented!

[@koenverbeeck9514]

Maybe I should rebrand STAR SCHEMA ALL THE THINGS to DATA PRODUCT ALL THE THINGS?