The IDEAL & Practical CI / CD Pipeline - Concepts Overview

[@burrillgray4812]

This was incredibly informative, logical, and easy to understand and follow. I don’t think I’ve ever seen such a clear, comprehensible step-by-step basic structuring of, as you put it, the ideal CI/CD pipeline. I really appreciate it. Thanks so much and I look forward to your subsequent videos.

[@Vandercox]

Well that will be a great addition to our on-boarding procedure ! You described the ideal CI/CD pipeline in such a smooth yet accurate manner and in a 22 minutes video rather than a 2h one ! Well done!

[@weignerg]

15:30 Errors over an extended period of time may also be indicative of a platform issue rather than an issue with the code. Just because the "canary" detects an issue, this does not mean that a rollback should be performed.

[@MrRobertkhall]

Your videos are great, I recommend these to junior developers on my team all the time. You deserve waaay more views for these man. Thanks again.

[@BeABetterDev]

Thanks so much Robert, really glad you enjoyed and your recommendations :)

[@ChrisCox-wv7oo]

Amazing 30,000 ft view of the CI/CD process, and what the purposes of every step are, not just how to do it but why and what you're going to ac complish strives, love it!

[@AlvinC-sz3li]

From your video, I have 80% confidence you are(or were) Amazon SDE. Some of the terms you mentioned are used by Amazon, but not lot of other places.

[@BeABetterDev]

You caught me! Check out kzbin.info/www/bejne/iqKtm5yGbZ2Ce7M :)

[@EvgenyGoldin]

And Amazon has one of the most advanced Pipelines I've ever seen. Think Route 53 deploying changes all over the world. Coming from R53 team myself.

[@AlvinC-sz3li]

@@BeABetterDev Ahaha, that's why I feel some terms are so familiar. From AWS Identity.

[@arcticape445]

@@EvgenyGoldin How does Amazon Pipelines compare to the systems used by other companies you've worked at?

[@JonathanBerrong]

What terms? Asking because everything in our org use the same terms and we are definitely not Amazon.

[@christophermaisch]

Dude, I've been watching your stuff and I must say that you are a freaking legend for putting so much of yourself into all of this. Thank you! You are the reason why a lot of people will be able to make a better life for themselves, really.

[@BeABetterDev]

Thank you so much Christopher for your kind words !

[@aniceguy241]

Thanks! I can see the content came from a very experienced and down to earth devops engineer. I learn a lot from that, really appreciate. Especially you talked a lot about the developer experience which I also think is very important. Most people talk about devops only as a cloud engineer or only talk about the CI/CD pipeline.

[@resphantom]

Let me help improve that whole ideal pipeline and simplify it: The ideal pipeline has these 4 main stages: Test -> Build -> Scan -> Deploy We have 3 different environments where those 4 main stages take place, which are: Dev, Test, Prod The Stages: *Test* This stage could encompass your: - unit testing - performance/load testing ( kind of difficult to implement/automate ) - integration testing ( kind of difficult to implement/automate ) *Build* This stage could encompass: - building code artifacts - building container images *Scan* This stage could encompass: - scanning for code coverage results - scanning your code with a tool that covers OWASP standard vulnerability scanning (SonarQube, Whitesource, Findbugs, Checkmarx) - Image vulnerability scanning (Clair, Trivy, Snyx) *Deploy* This stage could encompass: - Deploying your service with the correct config and version - Updating your deployment config *General* In general we also include a controller stage that handles some of the pipeline parameters or rules based on which environment you are pointing to, which is usually determined with particular code branches. This controller/preflight stage is usually at the very start or the pipeline. For example we follow the old gitflow model since the team is not yet mature enough, so the branches that follows the rules of each environment are: master (branch) - prod (environment) hotfix (branch) - prod (environment) release (branch) - test (environment) bugfix (branch) - dev (environment) dev (branch) - dev (environment) feature (branch) - dev (environment)

[@BeABetterDev]

Very well said.

[@scottdukes]

Hi, great video. I think you may be conflating Canary and Health Checks. Canary (Deployments) are what you refer to as your 1box approach. The function which you run periodically to check system is a Health Check.

[@AndreaMaruccia]

+1

[@anonymous1943]

Yeah, and I think his 1box example is blue/green deployment.

[@craigfoster9996]

There are also such a thing as "canary alarms" as well and the explanation in the video pretty much matches what I've seen in my experience.

[@vygylate]

Actually there are multiple types of canaries in software development. I've seen this definition of canary in practice, where you have some sort of test running continuously from customer's view. Another version I have seen is "canary deployment" which is where you deploy a small % of the code (similar to 1box here) and rollback based on how it performs in that % rollout. Both are correct, which is why the video explains their definition. I've actually seen more of the first definition, but I don't think it's a standard across the industry.

[@vasiapatov4544]

10:01 No matter how full-blown or sophisticated your CI/CD pipeline is, you should still perform a gradual rollout (what OP refers to as 1box), because production environments almost always have properties that cannot be fully replicated in test environments, so there is always a potential for a surprise in production.

[@BeABetterDev]

Couldn't have said it better myself!

[@Anhar001]

I'm 50/50 on this. If that is the case it means your application code is too tightly coupled to production data and you don't have actually useful tests that model real world use cases, if you have to rely on live users to essentially test your application then something has gone horribly wrong. I go agree there can be edge cases that sometimes happens that literally know one even imagines, and that's fine because in those cases those edges are added as part of the test suite.

[@jake-watkins]

This is a really good overview. Thanks As pointed out several times - your candary testing is a health check. No biggie, obviously people got it. However, the important difference is that a canary test lets you send REAL traffic through your newly released build (say 10%) to see what happens with just a small subset of your user base. If it blows up - it didnt blow up everybody and you can recover. Being able to do this type of testing in production also opens the door to more expirmentation. You can run A/B tests to see which UI your users like better, or if you have a new feature that you want to see if anybody is interested in. All of this stuff (health checks, Canary testing, A/B testing, etc) is enabled by having a functioning CI/CD pipeline.

[@BeABetterDev]

Couldn't agree more with this!

[@onke]

Our pipeline is [Jira task check, version check]-> [unit tests, sonar scan, cherkmarx scan]->[build, publish]->[deploy to dev]->[integration tests, load tests]->[deploy to prod]-[update Jira board]

[@BeABetterDev]

Looks great!

[@jyotikansal191]

Kindly share your code of pipeline

[@BlankFaceAma]

this was sooooo helpful. Thanks! I took thorough notes and am about to share them with the rest of my team. This was awesome.

[@BeABetterDev]

Glad it helped!

[@trekmotion]

One of the best explained videos I have watched on CI/DC pipeline. Liked and subscribed. Now I am just gonne binge watch all of your videos!! thank you.

[@BeABetterDev]

Thanks so much and welcome to the channel!

[@MichaelRicksAherne]

It's missing static analyses, including linting, code quality, security/dependency scanning, etc. It's also missing a stage for a small number of system/smoke tests. And while not technically a part of CI, if you're including monitoring/alerts, then logging should be included as well. (There's a big fuzzy area where CD blends into Ops, so it's not always clear where to draw the line, but your video is a distinctly "dev" perspective -- I wouldn't bisect Ops.) As far as integration tests go -- they test the integration between components. So there's always 2^n-2n possible test combinations. Example: for 3 components a-b-c, there's 2 combos -- ab and bc. (The individual ones a--, -b-, and --c are unit tests, and the whole group "abc" is a system test). For each integration test combo, there are 1 or more components that are "left out". These components are to be mocked/faked/stubbed. **Not every combo should be tested!** Only the ones that make sense or are easy. Most people bend the definition of integration test to mean whatever they want, but that's how you get systematic about them. List out all the components and draw various "system under test" combinations. Do the ones that make sense.

[@derickowens4821]

I agree. Also parallelism in the deploy is essential. Don't want to run all this stuff sequentially on all your environments -- that would take a long time. Parallelism is key for efficient pipelines.

[@chandrashekar-us6ef]

Sir, can you please provide me a link or source where I can learn everything.

[@chandan11choudhury]

I agree. A full CI needs more checks such as lint, code quality, security scan, and even tagging/versioning . CD also has a varied process which depends on the platform where you are deploying - K8, serverless, etc

[@FaizanAnwerAli]

@@chandrashekar-us6ef +1

[@rogerbarrett2172]

Just another dev that doesn’t think about security and compliance

[@DevenSitapara]

Great , to see such videos exists in youtube

[@chrisre2751]

Also you should add a stage/step to CVE check your artifact.

[@BeABetterDev]

Good call out Chris!

[@goodmaninc]

Great video! One additional thing you can add either as part of PR approval or build step is code linters / code analysis like check style or sonarqube.

[@BeABetterDev]

Check out PART 2 - The REVISED CI / CD Pipeline including great additions from the folks in the comments section! kzbin.info/www/bejne/hZTEhoSoh9RqY7M

[@YvonneMcGinnis]

You spoke about canary after I asked the question. Thanks. I think staging occurs prior to canary.

[@DuyTran-ss4lu]

Your channel is pure gold

[@YvonneMcGinnis]

You go from your testing environment to the production environment. In what environments should one include the staging, and canary as pre-production to the production environment, or do you include those two stages in your testing environment?

[@victorcosta1204]

Good explanation of what should be a basic CI/CD pipeline multi-AZ. Thanks man you rock!

[@theyreMineralsMarie]

Only thing you missed was code analysis for vulnerabilities in the build task. Something along the lines of Coverity or Black Duck. Otherwise this almost exactly matches my company's pipeline.

[@BeABetterDev]

This is a really good point Dan! Thanks for calling this out.

[@allahousalami]

1box in azure is deploying to a slot and redirect small percentage of connection to it. It could be also done with blue green deployment so the rollback is just to permut the release version and the previous release

[@severtone263]

Well explained buddy! You are changing lives and thank you

[@DarylMetzler]

I liked this video, but i think it misses a few things to describe a really efficient pipeline: - Review branches are not necessary, but are prescribed here. We can achieve the "for eyeballs" principle in source with pairing and co-signed commits - The "build" step is a perfect place to run linters (and other static analysis tooling). We want to fail hard and fast on not passing linters - The Testing stage is typically the most difficult to implement, and we usually want to run some kind of End-to-end and/or Functional testing there. Deploying a preview/test environment for the test stage is usually one of the more complex parts of a pipeline (how do you handle routing and databases? and cleanup on success or failure?) also, using E2E testing in CI reduces the need for as much pre-prod and canary deployment time. Shift that confidence left. Otherwise there is a lot of good advice here. I like the tech-stack neutral approach.

[@davidpccode]

How about ideal CICD pipeline for monorepos? Thanks

[@BeABetterDev]

Great idea thanks David

[@ManuV-f9c]

Hi! Thanks a lot for this. When are you going share the next part of adding unit test part for lambda to pipeline ?

[@monilpanchal6722]

The point 3) Canary in prod env stage is also called Synthetics monitoring. It simulates traffic real time to the env to ensure the systems are up and behaving as expected. Something similar to New Relic or IBM Cloud Synthetics

[@michaelotto2744]

1box. SO HELPFUL.

[@marcusicaro4563]

I believe the right therm, instead of "Canary" is "Continuous Testing".

[@henokworku4553]

Thank you for this amazing video

[@BeABetterDev]

You're very welcome!

[@salborough2]

Thanks for the great video. In terms of canary deployment and deployment progression a great tool to look at is Argo Rollouts that can handle this for you.

[@ethiohelptube794]

Clear and Concise! Thanks!

[@seekerofsense]

It's very informative for beginners. Could someone help me figure out how integration tests are usually run after an environment is deployed? Are they run automatically or manually?

[@nhuphan7112]

Thanks for the info and loving your video especially as a new junior dev just starting out :)

[@ripplesandleaves]

This was excellent, and you did a fantastic job covering what a pipeline should do in just minutes. Awesome! And thanks for making this. It's super handy to share.

[@BeABetterDev]

Glad you enjoyed it Andrea!

[@SwapanD]

Another masterpiece, appreciations

[@BeABetterDev]

Thank you!!!

[@kumarsaketthedreamer]

Thank you. This is quite informative.

[@rafael.aloizio1769]

Could you share a small version example of this pipe? It could be useful

[@Sn-ik1qx]

Great video. I would also add code quality analysis tools just to catch the bad code/vulnerable plugins etc before going to production.

[@luciangutu]

1 box could be canary deployment strategy

[@aramirezp92]

Hello, how do I make a partial deploy in production if I made databases changes (data and infrastructure) and rollback then. PD. Thank you, I show your videos to my coworkers :)

[@markopavlovic6066]

Do you have a video that covers this throughout the project?

[@brutussegun678]

Great. Please share the video where you build this on AWS.

[@krzysztofszewczyk2969]

Really nice tutorial, just two questions how to handle multiple commits during bake period on 1box prod? Should deployment of next version wait? or always rollout newest version and then downgrade it? Second thing is about db schema changes, as prod have only one db how to handle situation with 1box using different version that other hosts?

[@nvasudeva]

Great explanation! May I know what tool are you using to draw on the screen? Is it through ipad or some sort of a diff tablet ?

[@RichardPerfectKiwi]

That's a good start and you've got all the basics covered but there's a few critical things your missing for a large real-world application. It's a good idea to think about the different major components of your application (Web App, API, Database) and break up those things into different pipelines so that a one line change of your front end doesn't take forever to run. Also I actually hate the word pipeline because it implies there's a strict first-in-first-out approach to code flowing across the architecture. The build process should build Artefacts (npm package, jars, docker containers). The deployment process should deploy Artefacts. These two processes should not be tightly coupled. Any release should be capable of being deployed into any environment. Sure there's a bunch of reasons why you can't deploy a 2.3 API release into a environment with a 1.1 database but once the application is mostly built it becomes quite stable and you'll find there's quite a lot of compatibility between the artefacts being able to work with different versions. Any release into any environment means you can do a couple of things; - If there's an urgent critical production problem then the fix can be built from a branch taken of the production release without interfering with whatever dodgy/unstable major release you might be working on in production. - Performance optimisation usually involves working in a production-like environment (you can't test performance on your Dev machine), so you may need a Performance testing environment, or allocating another environment to performance testing for a period of time. Performance optimisation involves making experimental changes so you'll want these on a branch and you don't want to break the main branch until these are stable. So you need to be able to deploy branch'ed code into any environment = any release into any environment. It does lead to a "version compatibility matrix" problem of knowing which versions will work together, but in practice that's usually pretty easy and things like semantic versioning can help.

[@BeABetterDev]

Hi Richard, Thank you for the detailed reply and added tips. I agree with many of your points especially the about separating out the layers into different pipelines - this is something I definitely should have mentioned.

[@rykou46]

amazing video!!!

[@BeABetterDev]

Glad you liked it!!

[@calbears12345]

Very helpful, thank you.

[@petermanger9047]

Talks about "2" reviewers.... living the dream of a team being properly resourced... :-P

[@vikrantsai7]

Prod 1 box : CUG : Closed user group , where code runs on prod database but only fraction of entire userbase can use that environment.

[@ashishkarnad4059]

1box also can be said as green blue deployment?

[@maroxesen1]

Very nice!

[@BeABetterDev]

Thank you! Cheers!

[@naska8701]

Can you tag me the next video where you started the journey of building this project on AWS like you stated at the end, I really enjoyed this video so I want to continue the next one.

[@LucasCrostarosa]

Curious what your thoughts are on 1box or blue/green deployments when DB schema changes are involved in your release? Especially when some of those schema changes involve data migration and/or breaking changes to previous releases

[@Vali615]

same question

[@GioacchinoPiazzolla]

Is the reason why in my enterprise project never added this. But That's why we never added this to my enterprise project. But it would be nice to find a way to bypass this problem.... in that case we would definitely adopt the B/G deployment

[@fauzrukendworld1633]

If I need to introduce any BC break, I would personally create a new table and write on both tables for the duration of the blue/green deployment while still reading off the old one. When I'm satisfied with the writing part, I would do a new blue green to switch the reading from the new table. Concepts like CQRS probably make this kind of process easier though.

[@HeatherBarron]

I would hope your DB library layer would be architected to be able to navigate the various versions of your db schema gracefully.

[@IntrinDesign]

Thanks!

[@BeABetterDev]

Cheers

[@0x49BBF]

Couldn't have come at a better time. Great video, thank you!

[@carl-fredriksundstrom1814]

No branch protection and pre post merge ci ?

[@VictorVedmich]

What tool do you use for the whiteboarding?

[@BeABetterDev]

Hi Viktor, I'm using Photoshop.

[@Mr767267]

Not sure on the One box, but the Red/Blue method of deployment is the best out there.

[@nadirhamid8948]

Also, as someone else mentioned, I believe you have mistaken Canary with Health checks/probes. The one box environment seems to be using liveness probe to test for validation.

[@BeABetterDev]

Thanks for clarifying this Nadir. I've cleared this up in a recent follow up video.

[@FlorianKnip]

Great stuff

[@nadirhamid8948]

Good overall approach. However each CI/CD pipelines has its own set of needs and this is only good for typical app deployment. If the app is being deployed with more then basic app, meaning it uses specific services, such as Elasticsearch or Kafka, then these environments should also be added to test environment. But I like the overall approach. It is clean and almost bullet proof. Thanks

[@oliviazhai1831]

thanks for the video

[@notaverygoodusername]

Can anyone help me understand how you would do one box with a seperate backend/frontend. I would need the 10% to get both the new frontend and access the new backend. The backend and frontend both have different hostnames, don't know how to carve out groups of users by DNS when there are multiple hostnames in play.

[@beholdergamedesign]

In an ideal CICD linting and other similar static analysis techniques should gate promotion to integretion/cicd testing

[@BeABetterDev]

Good point Ben. I made some amendments in the follow up to this video.

[@skaterjay86]

I love the idea of 1box, its a new concept to me. I dont fully understand how this would fit into the full cycle of CI though. It sounds like if the tests pass in dev, you release to 1box, but when do you release code to full blown production? Would you have it running at a set time, like 9am each day?

[@marcello4258]

can you explain how to compile code for nodejs please ;)

[@andreyklepikov7084]

Thank you!

[@paulwagner1997]

Awesome video! It's really hard to learn these things, if you haven't worked in a large scale company.. Will try to get our startup to implement a similar pipeline!

[@techytalkswithbp]

Great vid !

[@BeABetterDev]

Thanks Brijesh!

[@devinda_me]

Does the canary run only during the bake period ?

[@BeABetterDev]

Hi Devinda, Ideally the canary should run 24/7. Its a useful tool during a deployment, but even when there is no deployment it can notify you very quickly of issues in your application.

[@ericmarcelino4381]

I have a question| How is the 1box is relevant when we rollout a new feature? Isn't the 10% traffic would be a bottleneck in the production? Or does we expect to test the production again?

[@hazemsoussi5309]

Concerning the test part , i think that it would be better to separate it with the production phase

[@ankushtagore4299]

i think using hitting every api per minute will increase server expenses a lot, or is it necessary?

[@paweduda7467]

How do you roll out your change to one box when you've done database changes and the code relies on it? It's a common case.

[@aviadcoppenhagen197]

Thank you for the video.

[@rushengzhang2959]

what if you update frontend and backend, the new front end will not be compatible with old backend, then when automatically roll back, the whole system will crash and trigger the alert, and rollback again, and eventually roll back the first version?

[@omers.143]

How about e2e testing?

[@UFOCurrents]

Awesome video. Thank you.

[@BeABetterDev]

You're very welcome Anthony!

[@zamarinen]

gr8 vid, but im a Infra Engineer, how should i work with CI/CD in the best way?

[@medsmati2063]

Thanks a lot👍

[@BeABetterDev]

You're very welcome Med!

[@bunyaminshabanov7134]

Hi, as you told each developer should have his own local environment, honestly, this did not work for our case. Your project had 6 or so microservices, and we(as FE devs) did not care at all on how they run. However, we had to run local versions which ate almost all ram and disc space in docker. What would you suggest in this kind of situation? Applies to even bigger projects with 10-40 microservices. And if this is possible to solve in cloud(aws/gcp)--please tell the exact mechanism that allows to do that) Thanks in advance, video is great!

[@baotransy7758]

Great video! Thanks for that. Besides, I dont think that we have the word "ideally" in any field

[@VitaliyPitvalo]

Thanks for this great concept. How you suggest handle db migrations ?

[@WelcomeWithinMyDream]

Thanks for your video. 1box? just call it canary release.

[@BeABetterDev]

Good call!

[@trailblazer_nomad]

Why do you think Run Unit Tests should be part of Build step and not Test step?

[@BeABetterDev]

IMO ideally it should be done in both. If a build succeeds but it produces the wrong result (fails the tests), what's the point in moving forward to putting in the test environment? That's my thought process atleast.

[@trailblazer_nomad]

@@BeABetterDev Yes but you said you run integration tests in the test step so why not run the integration tests in build step? What's your motivation for that?

[@johannsebastianbach3411]

Well, integration tests are not functional and not method/class specific, hence the name. And by definition, you need your code built to actually test whether all parts of the code you built integrate well 😂 Tldr unit tests are run while building occurs. Integration tests can only be run after all modules/packages you codebase has have been built

[@AliHassan-iy1eb]

what's the difference between blue green deployment vs 1box?

[@jasonlantz3808]

how about building on a golden image and using things like twistlock for opsec container scanning...

[@niharshiplu]

Nice informative video.I am from QA background.I would like to know why integration tests need to be run on a separate Test/QA Environment and what if we run those tests in DEV environment itself?Is it only to avoid test data dependency for integration or E2E tests.Or there are other reasons behind running the integration tests in a separate environment? Thanks.

[@harshmudhar96]

Is 90%+ code coverage really ideal?

[@caldayham]

what do you mean by "host" when you say roll back "only one host"?

[@caldayham]

nvm, I get it

[@MestreAlbino1]

Great vid. Maybe you could add details on how the prod (1box) and "big" prod work with DB (unified, I assume, any notes on some kind of rollback rules for any incorrect updates?)

[@MrRin_]

Man I love you

[@rahaan365]

I liked your gear.

[@BeABetterDev]

Thanks!

[@arcticape445]

Do you know how to best emulate Amazon Pipelines with public tools?

[@BeABetterDev]

Hi there! This is exactly what I'm trying to do in my recent sets of videos. I'm hoping to get something very similar to what I've described in this video. I've heard good things about Codeflow and Jenkins but havent found anyhting like Amazon's system.

[@arcticape445]

@@BeABetterDev Haha yes, I found that video quickly after. Definitely hoping you can continue that series, and maybe beyond CodePipeline (which IMO, isn't that great lol). It might require a mixture of different technologies, perhaps one service for CI and another service for CD, but I'm not quite sure what is out there to achieve close to what Amazon Pipelines has. One can dream!

[@anasouardini]

What's anomaly detection? AI?