Dagger: The Missing Ingredient for Your Disastrous CI/CD Pipeline

In this video, we will take a look at some of the common mistakes that people make when building CI/CD pipelines, and we will show you how Dagger can help you to avoid these mistakes.
Dagger is a set of libraries enable us to write CI (not CI/CD) pipelines in a variety of languages (NodeJS, Python, Elixir, etc.), that can run anywhere (locally, remotely, in other pipeline tools), and that is based on Docker or other container runtimes.
It replaces many of the tasks we normally write in Jenkins, GitHub Actions, Argo Workflows, Tekton, CircleCI, and other remote pipeline solutions.
#CICD #Dagger #SDLC
Consider joining the channel: / devopstoolkit
▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬
➡ Gist with the commands: gist.github.com/vfarcic/e3a28...
🔗 Dagger: dagger.io
🎬 Your CI/CD Pipelines Are Wrong - From Monoliths To Events: • Your CI/CD Pipelines A...
🎬 Is CUE The Perfect Language For Kubernetes Manifests (Helm Templates Replacement)?: • Is CUE The Perfect Lan...
🎬 Is Timoni With CUE a Helm Replacement?: • Is Timoni With CUE a H...
▬▬▬▬▬▬ 💰 Sponsoships 💰 ▬▬▬▬▬▬
If you are interested in sponsoring this channel, please use calendar.app.google/Q9eaDUHN8... to book a timeslot that suits you, and we'll go over the details. Or feel free to contact me over Twitter or LinkedIn (see below).
▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬
➡ Twitter: / vfarcic
➡ LinkedIn: / viktorfarcic
▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬
🎤 Podcast: www.devopsparadox.com/
💬 Live streams: / devopsparadox
▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬
00:00 introduction
02:02 CI Pipelines Should NOT Be Remote Only
05:56 Pipelines Should NOT Do CI Orchestration
09:20 Pipelines Should NOT Be Declarative
14:50 Pipelines Should NOT Depend On Specific Tools
17:04 Dagger: CI/CD With Code That Runs Everywhere
27:28 Dagger Pros and Cons

[@DevOpsToolkit]

What do you think of Dagger? Can it change how we do CI (NOT CD)?

[@trento8689]

I love this-- I've been looking into Dagger and I'm glad someone made a very digestible intro to Dagger!

[@twistplay122]

We utilize Taskfile, which is a modern equivalent of a makefile. We are delighted with its capabilities, as it covers nearly everything you mentioned in a format combining declarative YAML and Bash. Taskfile provides us with a unified interface for both local and remote CI.

[@DevOpsToolkit]

I think that the last sentence is critical. If whatever you're using works in all variations (e.g. local, remote, etc.), the main problem is solved. Most of the rest are individual preferences.

[@cheebadigga4092]

thanks for the info! didn't know about Taskfile but it looks very cool

[@LeakMemory]

This sounds great in theory (and is super cool!), but 15s cycle time for no changes is too slow. Devs' inner loop cycle time is extremely precious, so while I totally agree it would be great if all those things ran locally, I wouldn't trade 15s to get it. I'm former Netflix DevEx and we initially built something like this based on the mistake of conflating what was good for Platform with what's actually good for app devs -- it's an easy mistake to make and I believe you might've partially fallen into the same trap. We ended up abandoning containers for localdev (except for some sidecar services they didn't touch directly) a few years ago and everyone was happier. Well, everyone except Platform who had to support it!

[@PierfrancescoZamana-jw9ng]

Once again, a fantastic video! I became enamored with Dagger right after watching it. However, I later discovered that it doesn't support rootless, and in my company, privileged containers are akin to shutting down a datacenter, haha. It would be fantastic if Dagger could, in the future, utilize pods instead of containers, especially considering that all developers these days have a local Kubernetes cluster.

[@esantix]

Declarative and imperative are not that distinct concepts. Declarative is not necessarily describing a state. It simply a way of "not describing how". Meanwhile imperative does describe the how. But declarative can define a set of insctructions and not necessarily a state. So YAML can represent a set of instructions to run and not a state. I agree on the discussion about whether a pipeline should be defined by state or by instructions. But its not about declarative or imperative from my point of view.

[@paulogodinho3275]

This reminded me of Nuke, another tool to accomplish similar results. Great video!

[@coocoobau]

"Glorified cronjobs"... made my day. I've been trying to explain this to my fellow devs and devops for some years now :) And yes, we had the huge shell script that did orchestration and was just triggered by basic pipelines in a CI system that eventually became redundant.

[@CostisPanagiotopoulos]

Thanks for the video. Any idea which software Victor uses to create these lovely animated diagrams?

[@DevOpsToolkit]

I think it's Adobe Illustrator. Not 100% sure since it's done by an agency.

[@dgkaba]

Hello, Nice video again thanks, great case on declarative. would you wrap dagger task into cicd task so u could benefit parallelism for example ?

[@DevOpsToolkit]

Depends... Since it's a programming language, running in parallel is not a problem anyways.

[@dgkaba]

@@DevOpsToolkit well yes and no… thanks for your reply 🤗

[@davemeech]

"Today I will continue complaining about CI/CD pipelines..." Already sold.

[@KsaweryKarwacki]

I work in java based environment and used the same principle, but instead I used gradle. Jenkins just uses "sh ./gradlew .." to trigger but whole logic is in gradle and custom plugins. It does not matter if your running on Windows laptop, WSL, virtual machine, Jenkins agent. It always works the same. Heck we even have plugin to bundle JS apps to have consistency. Additionally I do not need to force everyone to learn quirks of Jenkins because most of them are JVM devs so they can just debug gradle runtime on their machine.

[@peterkipkoech2882]

What about system integration, that the microservice I am developing is part of other microservices and I need to test them in a cluster. The microservice already has unit test and if this passes locally I can push it to git and the webhook would trigger the ci build and I can then test it. So far i am not convinced of this use case

[@DevOpsToolkit]

You still need pipelines that accept webhooks. What Dagger brings to the table, among other things, is being able to execute tasks both locally and remotely from inside pipelines (e.g., GitHub Actions).

[@iamunknownperiod3355]

I agree with your point on imperative CI but there's something about seeing the actual code, calling shell via go instead of just writing it in just shell seems too cumbersome. It might be very flexible but at the same time, it seems like not very straightforward and would require a longer time to start when compared to writing it in just shell scripts or lines in a YAML file. YAML is declarative but maybe there is merit on using it for pipelines because of its human-readable nature. I guess it will depend on which you value more.

[@DevOpsToolkit]

That's true. If Shell does it, use it. I also tend to go with Shell scripts for a long as the complexity does not make it worse with Shell than other solutions. That being said, Dagger is more than a replacement for a Shell script. It allows us to work inside containers in a similar way we'd work with containers inside pipelines. Still, if Shell does what you need it to do, there's no need to increase the complexity (or decrease portability) with anything else.

[@MrTheGurra]

Very good points! Use declarative tooling when you are describing an end state. Use imperative tooling when you are describing a process. Declarative pipelines always felt counterintuitive to me

[@scottamolinari]

I thought I watched everything, but I seemed to miss a note about Dagger modules i.e. code reuse for the win! The even cooler thing is, someone who needs to use a module in Go, and they are using Dagger with TypeScript, can still call on the Go module to use it. How awesome is that?

[@DevOpsToolkit]

Yeah. That's one of the great things which i probably did not explain well (or at all).

[@mrcaique1025]

Not sure if i read it correctly, obviously we can run github pipelines locally. The same is with Jenkins, they provide slim runner for that. So in that aspect both solutions are quite similar

[@DevOpsToolkit]

I'm talking about building, testing, and doing whatever else I'm doing while working locally. I tend to switch from writing code to other tasks every minute or even faster and, for that, I do not thing that Jenkins and similar solutions are a good choice. A simple script like test.sh is much better, especially when combined with a tool that will re-run the process every time I make any change to the code. Now, if that's the case, what I need is the ability to run a variation of the same both locally and remotely after I push changes to Git. What's where Dagger comes in. It's a replacement for Shell script that I used locally but also inside Jenkins pipelines.

[@mrcaique1025]

Thanks for clarifications Viktor!

[@danielhd6719]

@@DevOpsToolkit My tests require 32 GB of mem and 12 cores to finish in remotely reasonable time. Let me run that on my 8core 16bg laptop. Trying to solve issues nobody has is the best example of most SaaS solutions.

[@DevOpsToolkit]

@danielhd6719 that does not mean that everyone's tests require that much memory and CPU.

[@kdkulakov]

I don't understand its free or open source tool? I couldn't find information about opensource or community version on the main site

[@DevOpsToolkit]

It's open source apache licence. Here's the repo: github.com/dagger/dagger

[@ZiggleFingers]

What if I told you there is another tool that does everything dagger does but doesn't require using containers. Outputs containers, ami's, kubevirt images, iso's, hyperv, gce, azure, virtualbox, qemu and more. Built-in distributed building, hot rebuild, sandboxed dev environments with all tooling included. Can be run equally locally or remote with one command.

[@DevOpsToolkit]

I'd say, tell me more.

[@CelesteOnYoutube]

LOL

[@ZiggleFingers]

Nix has been doing this for a long time. Take a look!

[@chrisre2751]

Earthly is another interesting solution/tool. Maybe not the best for complex pipelines. It is like Dockerfile und Makefile had a baby child 😂. Easy to use but sometimes limited.

[@marcosnils]

@@ZiggleFingers there's one little but super important difference that nix doesn't solve which is generally one of the strongest aspects of Dagger. You can express your pipelines with *actual code* instead of having to rely on some custom DSL. In the nix story this point is even more relevant since the #1 frustration of users adopting nix is the learning curve.

[@dmitriinadein4733]

Виктор, ты бесспорно профессионал, но тут я не соглашусь - важно не забывать про простоту - KISS метод. Представь, что ты накодил все круто на Go и уволился, а потом компания долго пытается найти девопса который в этом сможет разобраться. И очень повезет, если есть документация! Отчасти поэтому стали везде делать декларативное описание и лично мне это нравится.

[@DevOpsToolkit]

Dagger supports other languages so I would assume that you'd use the one you (and your company) is most comfortable with. Nevertheless, that video is not primary about Dagger but, rather, a discussion whether we should use a declarative format to describe imperative processes. It's clear that declarative formats are great for describing the state of something and the question is whether they should be used as a replacement for designing workflows. P.S. I used Google Translate to translate your message.

[@ZiggleFingers]

dagger seems like an attempt to reimplement a very small part of Nix when they don't understand (or they don't want to fill you in on) the full problem. each step of daggers development will come to the same conclusion Nix has to move it further towards reproducible builds. they will find they need a nixpkgs equivalent (every* software package defined with inputs and outputs) to be able to build their DAG (which Nix has done for decades). then find that it makes sense to use a language to simply define inputs and outputs (hmm Nix?). Why ship a monolithic 1GB container when Nix can build a DAG based container (because it knows each *files* inputs) with upto 128 layers? If all your containers are Nix built, they will share most of the same layers because they're ALL reproduceable from the same inputs! Need to swap out one 500KB shared dependency on 5 pods that run on a 5000 node K8S? One 500KB layer pull - theres no special logic to define this in your bespoke pipeline. Nix knows this.

[@iuliansurugiu7762]

Hi. Have you ever tried batect?

[@DevOpsToolkit]

I haven't 😔 I will... Adding it to my to-do list...

[@coocoobau]

Not maintained anymore...@@DevOpsToolkit

[@iuliansurugiu7762]

Scratch that. Just noticed it's not longer maintained :/

[@ThePapanoob]

This feels like youre searching for a solution at the wrong end. I much rather run pipelines with the same tools i use locally! Thats why i switched to podman, buildah etc.

[@DevOpsToolkit]

Isn't dagger all about being able to run pipelines remotely using the same tool as when running them locally?

[@ThePapanoob]

@@DevOpsToolkit well yea but i rather use the same tools everywhere without adding another tool to do so ;)

[@definitelyno]

The problem with our pipeline is that it uses Jenkins. And it sucks.

[@sharktamer]

Who is Wong?

[@DevOpsToolkit]

Typo... It's fixed now... Thanks for letting me know.

[@julianomoraisbarbosa]

# TIL

[@dirien]

First!

[@haddonist]

Love your content, but the frequent zoom jumps make some of your videos almost unwatchable for me

[@robertkozak]

Big fat NOPE from me. You are wrong on this one. Although I agree Declarative code primarily represent a state it does really work well with pipelines. Pipelines should be simple: Run top to bottom as a DAG in parallel, based on events. Simple conditionals as a property on a yaml section is good enough. It's super easy to read for most people and devs. having imperative code based on language is not inclusive because different teams use different languages.

[@DevOpsToolkit]

There's still the question "how do I run part of it locally?"

[@robertkozak]

@@DevOpsToolkit yes of course. But wouldnt the simplest answer be run an agent on the laptop? like a github actions agent? Maybe even wrap it in a container so you dont even need to install the agent on the laptop and make it ephemeral

[@juanmacoo]

I have to agree here. Declarative is the way to go. As true as it may be that a Pipeline is not particularly a state, you could say that it is a template of a chain of events. I strongly believe that it is possible to build a tool that is declarative and can executed locally (like argo workflows where you define each stage as a container, that could run anywhere, for better or worse). This does sound to me, though, as a perfect “universal” build tool candidate, I’ve used Makefiles for this purpose, but we all know how limiting that is…

[@DevOpsToolkit]

@robertkozak i never managed to make something similar to work as fast and efficient as a simple execution of a script or a binary. I might be wrong though...

[@robertkozak]

@@juanmacoo technically i suppose a pipeline is a state machine

[@dus10dnd]

You lost me at "writing in the programming language of your choice"... Pre-commit is good enough for what you talked about.

[@DevOpsToolkit]

I'm not sure I understood the relation between pre commits and writing pipelines/tasks in any language.

[@dus10dnd]

@@DevOpsToolkit No relation, different thoughts. And not that you're wrong about any of this. Thank you for the content!

[@cloudnativepizza]

Another set of complications to help tech youtubers stay relevant