DataOps - Introduction to staging environments

In this quick video I explain the various environments you'll need for a DataOps process from the development environment where you do your work, through the testing environment where automated unit and integration tests get run, and on to QA (pre-production) where data quality tests are run against real data. Finally to production where the business gets access to the data. In addition to these, I explain the experimentation areas where data scientists should work.
0:00 - Introduction to the session
1:44 - Development Environments
4:59 - Testing Environments
7:22 - Quality Assurance/Pre-Production Environments
8:54 - Production Environment
9:06 - Experimentation Area
11:41 - Wrap up
If you're new to data lakes or DataOps/DevOps please ask any questions you have below. Also please comment if you found this workshop series useful or if you'd like to see more of this kind of content.
For all of my other demos, go to davedoesdemos.com or go straight to the GitHub page at github.com/davedoesdemos/Demo.... Also please subscribe to the channel to make sure the latest demos show up in your playlist!

[@jonashoffmann8419]

Thanks Dave. Very insightful. Cheers Jonas

[@adebolaopeyemi1039]

weldone Dave! Quite what i needed.

[@Carlos_Jr1984]

Hi Dave. Nice vid. My studio and its various teams are sharing a pre-prod server for testing, so it has to be booked, scheduled etc before it can be validated in this environment. Any ideas on how to avoid this and have everyone use it at the same time?

[@DaveDoesDemos]

Hi Marcus, thanks for the comment. This would very much depend on what your team do and what's on the server (please feel free to elaborate on this). Sometimes if you have complex requirements scheduling is the only way. Most of the time though, I would recommend using public cloud and scripting the environment as part of your DevOps pipelines. This way you can deploy a pre-prod environment, do your testing and then destroy the environment. With this technique, all of the teams can run testing concurrently because they each have their own environment. This usually also works out cheaper since you're only paying for the environment while the testing is taking place.

[@DaveDoesDemos]

There's a bit of info about this at this link github.com/davedoesdemos/DataDevOps/blob/master/EphemeralTests/EphemeralTests.md

[@greatladyp6632]

Do you do trainings?

[@nathankrasney]

The video is great but I did not get the bottom line - what is the staging environment? you cover dev, test, QA, prod, experiment environments - where is the staging?

[@DaveDoesDemos]

Hi Nathan thanks for the comment. Staging refers to moving code from one environment to the next, and is not a stage in itself. Sometimes people refer to a QA environment as staging or pre-prod but there is no standard for this naming. You should base your stages on your requirements, but generally you need one to do development work (often your laptop!), one for running structured tests, and one for production. Sometimes it's also helpful to have an environment the same as production to allow other people to try out the changes before you release them.

[@nathankrasney]

@@DaveDoesDemos Thanks

[@Ikilledthebanks]

@@DaveDoesDemos excellent explanation

[@Ikilledthebanks]

Dev data should be representative, always missing fields for us.