Argo Workflows and Pipelines - CI/CD, Machine Learning, and Other Kubernetes Workflows

Argo Workflows & Pipelines is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. It is a cloud-native solution designed from ground-up for Kubernetes.
#argo #workflows #pipelines #kubernetes
Timecodes ⏱:
00:00 Intro
01:52 Setup
02:36 Workflows
11:15 Templates
16:14 Submitting workflows
21:28 Final thoughts
➡ Gist with the commands: gist.github.com/28e2adb5946ca...
🔗 Argo Workflows: argoproj.github.io/projects/argo
🎬 Argo CD: • Argo CD - Applying Git...
🎬 Argo Events: • Argo Events - Event-Ba...
🎬 Argo Rollouts: • Argo Rollouts - Canary...
🎬 Kaniko: • Kaniko - Building Cont...
📚 DevOps Catalog, Patterns, And Blueprints: www.devopstoolkitseries.com/p...
📚 Books and courses: www.devopstoolkitseries.com
🎤 Podcast: www.devopsparadox.com/
💬 Live streams: / devopsparadox
➡ Follow me on Twitter: / vfarcic
➡ Follow me on LinkedIn: / viktorfarcic

[@hawk1290]

Thanks for this series, starting to believe that you've already demystified any modern solution I've come across. Keep it up!

[@DevOpsToolkit]

That is the problem with our jobs and tech in general. Things change fast and I am likely going to say in the near future "this is horrible, don't use it" for most of the things I'm claiming are great today.

[@hawk1290]

@@DevOpsToolkit Absolutely, community adoption really makes or breaks that support - so thanks for being part of that. Awesome content for sure.

[@valour.se47]

Yes a video about combining these all things together will definitely help people.

[@DevOpsToolkit]

It's (probably) coming up next week :)

[@shubhendupoothia]

Man this is so awesome!! Thank you so much!

[@cesarvarela4679]

Thank you, very good video. My vote is Yes make the video combine argo tools.

[@DevOpsToolkit]

You got it! It's probably coming up next Thursday.

[@MohammedNoureldin]

Thank you for this! Please make a video comparing Argo Workflows with Jenkins.

[@DevOpsToolkit]

Adding it to my to-do list... 🙂

[@xinitbm9453]

Thank you so much, this is awesome

[@fabiodbr]

Awesome. Didn't know anything about Argo since this video and now I'm almost an expert haha

[@DevOpsToolkit]

Glad I could help!

[@felipeozoski]

Love this channel ❤❤ for real !!

[@wollginator]

Awesome, great introduction and regarding your question: yes, I admit it, I want to use the whole argo stack, so lemme just click your next video! PS: Subscribed!

[@TankaNafaka]

Looks like aws step functions to me. You can also do cmd overrides etc, run job in parallel or serial, and schedule runs via cwatch. Anyways really nice work Viktor, thank you for your effort.

[@bewilderedlearningevolving]

Yes! Thanks Viktor I've been wrestling with the workflows side of argo for months 💵💵💵💵💵

[@DevOpsToolkit]

In that case, I think you'll like the video I'll (probably) publish next Thursday. It's about combining Argo Workflows with Events (and CD, and Rollouts, and SealedSecrets). Workflows really shines only when it is combined with Events, and gets even more boost with other Argo projects.

[@mandolinean3057]

great video!

[@diptochakrabarty2420]

Great content in the argo playlist , one suggestion if you could please add the link of your github repository you use in your demos in the description it would be helpful.

[@DevOpsToolkit]

Argo CD video was one of the first ones I created and it was done in a "weird" way. I adapted the material from my book without creating a Gist that people can follow. I think that all the videos after that one do have a Gist. For example, if you take a look at the description of this video, you'll see the "Gist with the commands" link with all the commands, including `git clone` of the repos I used.

[@mysticaltech]

Hey Victor, thanks! Please make a video of the combination of all Argo tools, as not obvious to beginners!

[@mysticaltech]

Found it kzbin.info/www/bejne/jn-7e6euoLp-m7s !! 🙏

[@DevOpsToolkit]

I already did that in kzbin.info/www/bejne/jn-7e6euoLp-m7s. However, it has been a while since i published it so I will add it to my TODO list to do an updated version.

[@7siwy1]

Victor, what would be the best way to run integration tests during the build phase of CICD? My dev team need few databases to test an app like mariadb, posgress etc. So far we did it by keeping docker-compose in each app repository. Before the tests start we run 'docker-compose up' that provides mocks of all the services needed. How that could work with argo workflows?

[@DevOpsToolkit]

Argo Workflows, just as any other pipeline solution, allows you to execute any command as a step. Since all the steps are executed in containers, you need to specify an image that contains the tools you need. Now, Docker is gone from Kubernetes clusters so you should either connect to Docker running in a VM (you can use environment variables for that) or spin up the DB inside a Kubernetes cluster. I suggest doing the latter. You can, for example, use MariaDB Helm chart. If that DB is temporary, create a new Namespace, spin up MariaDB (or whichever other DB you might need) inside that Namespace, run the tests, and delete that Namespace when you're done. You can also use virtual clusters with, let's say, vCluster if you'd like to have better isolation from the rest of the workloads in that cluster. Also, virtual clusters help with cleanup. If you deploy something that uses cluster-wide resources (e.g., CRDs), deleting the Namespace will not delete everything, but, if that is inside vCluster, you can delete the whole (virtual) cluster.

[@cokegen]

That Marla poster ...

[@DevOpsToolkit]

Oh yeah. My favorite.

[@cokegen]

@@DevOpsToolkit BTW, going through everything argo you published ... it's awesome and being of great help ...

[@raoufmnif6569]

Well done like always Viktor. Can you provide us the installation of argoworkflow with serviceaccount and with the right privilege?

[@DevOpsToolkit]

Adding it to my TODO list...

[@alexanderdo3122]

What is your recommended way of installing a cluster locally on a mac for testing things like this?

[@DevOpsToolkit]

I prefer k3d. It's the fastest and the least resource-intensive local Kubernetes cluster. KinD is a good alternative. You can find videos about both in this channel :)

[@sanjaytiwari3176]

I couldn't find the video where you installed workflow and setup any security context as its failing for the same.

[@DevOpsToolkit]

The instructions with both the commands I executed before and during the video are in a Gist and the link to the Gist is in the description. It includes everything I did to set it up (install it). As for the security context... I used Minikube for the demo so I did not do anything with the context. Most of the time, that's not directly related to a specific app but applicable in general and, sometimes, specific to the k8s platform you chose.

[@maylingtan3716]

Hi Viktor, thank you for the video! I managed to run your example. A question i have is what if the git repo is a private repo? where do i set up the credentials in order to access the git repo? Thanks for the help.

[@DevOpsToolkit]

Something like argo-cd.readthedocs.io/en/stable/user-guide/private-repositories/ should do

[@maylingtan3716]

​@@DevOpsToolkit , thanks for the reply. I think the link you provided is for Argo CD. Using argo workflows to build image referencing a private git repo doesn't seem to work as the the argo workflow UI doesn't have this tab.

[@DevOpsToolkit]

If you need to access a private git repo, you need to define git artifact and provide the secret with the username and password. You can find an example in github.com/argoproj/argo-workflows/blob/master/examples/input-artifact-git.yaml

[@gagyboki799]

Hi Victor, thanks for video.If I understand this workflow is not like Camunda or Spring Activity?This is for orchastration of micro services?Can you build kind of workflow like Camunda workflow and read from MQ?

[@DevOpsToolkit]

Unfortunately, I haven't used either of those so I cannot compare them or say whether they are similar. Argo workflows is a pipeline tool just like Jenkins, GitHub actions, tekton, GitLab ci, etc. As such, it is not limited to microservices but can be used to automate any type of pipeline.

[@gagyboki799]

@@DevOpsToolkit so it is not meant for data flow workflows?

[@DevOpsToolkit]

The initial use case was for data pipelines but, since then, it evolved into pipelines of any kind.

[@gagyboki799]

So can it be used to run data flow like IIB data flow?

[@gagyboki799]

kzbin.info/www/bejne/nXasgHSLj7F9q9E Something like this?

[@akshaymungekar4748]

How can I run python scripts in an Argo workflow? Could you provide any implementation details to achieve this task?

[@DevOpsToolkit]

I do not use python myself. However, that should not matter since Argo workflows allow you to use any container image for any of the steps. If the image you're using is based on python, you should be able to execute any python command inside your workflow.

[@ashutoshtiwari6947]

How can i create volume to have a folder path shared across containers . i.e. all using argo workflow??

[@DevOpsToolkit]

You can found a couple of examples in argoproj.github.io/argo-workflows/examples/#volumes with more detailed info in the spec section of the docs.

[@ashutoshtiwari6947]

@@DevOpsToolkit thanks!! I had a look at it.... But was a bit difficult to comprehend... I am looking for something replica to docker run -v sourcepath: destinationpath container-name.... If you can further explain it that would be great... Thanks in advance

[@DevOpsToolkit]

@@ashutoshtiwari6947 I don't have an example at hand. I'll do my best to create one over the weekend. In the meantime... You need to define a Kubernetes volume (it could be almost any time) in, more or less, the same way you would do it for any k8s resource. From there on, it's all about telling workflows to use that volume.

[@ashutoshtiwari6947]

@@DevOpsToolkit Cool... really looking forward to your next video wherein we can use volumes used via docker run.. or docker artifacts as an alternative to it.

[@DevOpsToolkit]

@@ashutoshtiwari6947 To clarify... When I do such a video (can't promise the date), it will not involve Docker. I do not think that anyone should use Docker inside Kubernetes. There are much better (and safer) alternatives. For building images that could be, for example, Kaniko. As for running containers, that's completely irrelevant since the decision is made by the Kubernetes distro you're using (and is almost certainly not Docker). On top of that, volumes are typically used for temporary storage so that you can pass artifacts from one step to another, not from one pipeline to another. When a permanent storage is needed, it would be a container image or artiface registry like ECR, ACR, Docker Hub, Artifactory, etc.

[@Teej42]

Please enable captions for us folks who can’t hear.

[@DevOpsToolkit]

I think it's one of the things KZbin does without explaining how. When I look at that video, there are no captions, but when I look at, for example, at kzbin.info/www/bejne/gnixdnpjqZ5pm6c, there is the CC button to enable captions. I'm not sure how and why KZbin choose one but not the other. Maybe it's doing it automatically in all the new videos?

[@KanshiTV-INT]

Hey Viktor, I've followed your video and deployed workflows and implemented a Traefik ingressRoute apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: argo-workflow-https spec: entryPoints: - websecure routes: - match: Host(`workflow.xxxx.co.uk`) && PathPrefix(`/`) kind: Rule services: - name: argo-server port: 2746 tls: certResolver: le-staging domains: - main: workflow.xxxx.co.uk but all I get is bad gateway. Any ideas of what could be the problem or how to troubleshoot.

[@KanshiTV-INT]

Hey Viktor, I managed to get it working buy adding the following - server - --secure=false and letting Traefik do the certificates

[@abdullahshaikh6989]

how to pull private git repo in this case? like how to add github secrets in kaniko?

[@DevOpsToolkit]

You need to store auth info as secrets and then mount those secrets inside containers that need them.

[@DevOpsToolkit]

One more thing... Pull repo using Kaniko only if you are not already pulling it for other reasons. More often than not, you will already have the repo cloned for testing or other reasons.

[@primaadi]

please make the combined full flaged aroproc CI CD

[@DevOpsToolkit]

Do you mean something like kzbin.info/www/bejne/jn-7e6euoLp-m7s or you have something else in mind?

[@kubernetestn]

Thanks a lot ! where can i find source code ?

[@DevOpsToolkit]

There is a gist in the description of the video.

[@johndesposito]

Are you available for consultation?

[@DevOpsToolkit]

I am not available for any type of activity that requires dedication. I can spare an hour or two if that can help.

[@johndesposito]

@@DevOpsToolkit - Hi. It would only be one or two hours to help architect an ARGO workflow/CD solution to allow for propagating changes to different environments. Thank You. Please let me know how I can reach your or if you want me to leave my contact information. I greatly appreciate your guidance. Thank You

[@DevOpsToolkit]

@@johndesposito Feel free to pick a time that works for you from calendly.com/vfarcic/meet.

[@johndesposito]

@@DevOpsToolkit Will do .. thank you

[@quant-daddy]

How does this compare to tekton pipelines?

[@DevOpsToolkit]

I haven't been using tekton for some 6 months now so I need to get up-to-speed with what they are doing lately before answering that question :(

[@tigercat123456]

@@DevOpsToolkit .... a bit late to the party ... but I would really like to see a comparison between Argo Workflows and Tekton. Imho Tekton pipelines can be quite flexible but also verbose/tedious to write (aside from a hello-world usecase :), and I am wondering if Argo accomplishes that in a neater way? Also debugging of pipeline runs/workflows/jenkinsfiles/... can be a huge issue and is usually not a built-in feature.

[@SanjeevKumar-nq8td]

It is Tekton principle taken over.

[@DevOpsToolkit]

Actually, the first release of Argo workflows was published a year and a half before the first release of Tekton.

[@DevOpsToolkit]

To be more precise, that was the first release in the current GitHub repo and it was v2.0.0 in November 2017. I did not follow Argo workflows at that time so I'm not sure when was the release release made before it moved to the current repo nor I know how it looked at that time. In any case, I am not sure who copied whom and how Argo evolved back then but it is definitely older than tekton.

[@michaeljuliano8839]

“What it looks like,” or “how it looks,” but never “how it looks like.” “What” is a placeholder for a thing. You need “like” when you say “what” so we know you are comparing to the thing that “what” is standing in for. “It looks like what?” or “What does it look like?” “It looks like this.” “What” stands in for “this.” “How” is a placeholder for a description (e.g. good, quickly, slowly, etc.). because there is nothing to compare to, “like” doesn’t make sense. “It looks how?” or “How does it look?” “It looks good.” “How” stands in for “good.”

[@DevOpsToolkit]

Thank you for that clarification. My English is self-tought and there are too many things i still struggle with.

[@michaeljuliano8839]

⁠​​⁠Don’t get me wrong. Your English is very good. I would even say it’s much better than average for someone from what I’m guessing is Eastern Europe. This is a very common mistake, even among native speakers, and I was just being a bit picky because I was in a bad mood.