I automated EVERYTHING in my HomeLab // Kestra #2

In this video I will show you my ultimate solution for automating everything in my HomeLab, where I integrate all my favorite automation tools and workflows, including Ansible and Terraform, into a unified workflow using Kestra as the platform that enables me to orchestrate all these tools together seamlessly, creating a powerful automation ecosystem. If you're a fan of automation, you won't want to miss this!
References
- Check out Kestra's Github: github.com/kestra-io/kestra?u...
- Join Kestra's slack: kestra.io/slack?Yo...
- Kestra Tutorial: • My NEW HomeLab automat...
________________
💜 Support me and become a Fan!
→ christianlempa.de/patreon
💬 Join our Community!
→ christianlempa.de/discord
👉 Follow me everywhere
→ christianlempa.de
________________
Read my Tech Documentation
christianlempa.de/docs
My Gear and Equipment
christianlempa.de/kit
________________
Timestamps:
00:00 How I’m automating EVERYTHING
00:58 What automation tools I’m using
02:14 The problem with Ansible and Terraform
04:09 My automation pipelines in Kestra
05:26 Build an Ansible Flow in Kestra
09:22 Problems with the Flow execution
10:55 Adding the private SSH key
14:18 Expanding the Ansible Flow
16:07 Some Ansible examples of my HomeLab
21:41 Terraform Flows in Kestra
23:21 Updating my DNS using Terraform and Kestra
26:21 GitHub Actions + Kestra + Terraform + DNS
________________
Links can include affiliate links.

[@jameskirk5914]

this channel is criminally underrated

[@christianlempa]

Thank you :D

[@jasonperry6046]

I completely agree.

[@kylerhella6251]

I’ve basically learned how to be a sysadmin from watching this channel 😅

[@weholmes5315]

Just watched your 2023 Homelab Tour where you shared your 100k Creator Award. And here you are on the brink of 200k! Next Award will be well deserved. Keep up the great work/fun. You are much appreciated 😊

[@christianlempa]

Thank you buddy ❤️ let's celebrate this on the next livestream :D

[@guanche011]

Kestra looks cool, and it could combine a few tools I use professionally and at home. But, having no secrets manager and needing to taking everything down every time I need to add something is not an option. Paying for features is not a problem, but the only other tier next to 'free' is "talk to us". This is a really high barrier, while this is provided and managed in a free AWX setup. Also concurrency requires separate tasks in Kestra and writing extra domain specific YAML, which makes migrating more expensive.. Overall I'm not sold yet. The product looks cool, flashy (slick UI i must say), but these are show stoppers. Would have loved to play with shiny things though ;)

[@BilalInamdar]

Do u have any alternative in mind ? kindly suggest as i am also trying to evaluate the same.

[@dimitriclark9559]

Just what i needed, thank you!

[@christianlempa]

You are so welcome!

[@DreamerCze]

There is Event-drive Ansible now, that might be worth checking out.

[@RealNeuronaut]

you should probably not circumvent permission problems by running the docker container as root... adding the following task after the local file creation does the trick just as well: - id: set_perm type: io.kestra.plugin.scripts.shell.Commands commands: - chmod 600 id_rsa

[@berndeckenfels]

If you automate it anyway, also manage the known hosts file with the server keys and not turn security in ssh off

[@dkryptonut]

Something I'd be keen to see is a video about storing credentials securely, whether it be a key or an actual password - and retrieving it within Kestra.

[@sandeepreddy3994]

Hello Christian, Loved your work…!!! i really appreciate it, It would be a great help, if you can show us how can i control the flow execution based on the exit code of previous tasks in kestra

[@mikephares5104]

Great video. Thanks you for your time. Will you look into Gitea for triggering Kestra?

[@christianlempa]

Maybe, I currently review GitLab self-hosted and Gitea, will take some time until the videos are coming out though

[@nagavenkataphanikumarkanch5827]

Hi can we connect windows servers to run some shell scripts from Kestra? Through ssh or ftp something like that

[@antonkozyk]

How do you use your /app/scripts? What kestra type can be used?

[@mariotubelecce]

it's basically jenkins with a nice interface and a paid option?

[@cheebadigga4092]

kinda yea

[@clarkkentgwapo1]

Deserve subs

[@AlphaX66]

Does Kestra have a vault features to store secrets ? Inject SSH key in the container itself looks ok, but what about some other secret that you want to you use in the ansible playbook ? Do you really need to use the container environment variables each time you need to had a secret ?

[@lvmeijer]

Enterprise only

[@Danielo515]

What I want is a nice way to execute playbooks against custom servers in a nice UI

[@hcjkruse]

Cool, I Didn't know Kestra going to look into it. Would be nice if Kestra also could run helm charts and docker-compose files. Going to look into that. My first thought was, but I have a homelab git repo, but that was anwered quickly. But.... Do you have a separate set of disaster recovery playbooks in a remote backup etc that you can use to setup the machines to run all this just from the shell? All this automation looks cool but bootstrapping it again by hand would not be fun.

[@laurentiusjudhianto6631]

Yes you can run helm and docker compose by utilizing the bash - shell. This is the way to use Kestra if there is no plugin for it.

[@mathesonstep]

I am now trying to decide between using this or Ansible Semaphore

[@Glatze603]

I prefer ansible and cronicle task scheduler for automating tasks in my homelab.

[@piyushanand4451]

Hey Christian, your video is very helpful, but can you tell me how I can pass the password from the inventory only, I don't want to use key based authentication.

[@cheebadigga4092]

very nice! The error about the key files being too open worries me though. The correct permissions of id_rsa is 0600, so you could try to chmod it right before the ansible-playbook command. I'd be interested if that works.

[@christianlempa]

I tried that, but didn't work for me. Maybe I just made a mistake 🙈

[@Mavo1989]

Not sure why your not just using jenkins..... free open source docker image and have full devops CI/CD pipelines and also utlise github/git actions and your branches and have proper devops pipelines.... honestly jenkins is the best answer

[@gautambhatnagar4765]

Top right....

[@johngleeson7919]

What, in your opinion, is the inflection point between time spent doing a task vs time spent automating a task? This video is cool, just wondering what your thoughts are, given in an enterprise environment, you'd typically spend a lot of time automating something that'll be reused many times. In a homelab setting, that's not necessarily the case, but you do get the opportunity to learn without the ramifications of breaking a "real" production environment.

[@christianlempa]

That's a great question, I've recently covered that in my livestream, how useful is automation really in a HomeLab. Because you can argue it's wasting time :) again maybe that recording might be interesting for you

[@Reprint001]

Looks powerful, but a bit OTT for my little homelab. I'm also already heavily invested in NodeRed for automating lots of things (like my Proxmox node backups to PBS). Interesting though. I can only learn so many things and this might be one too far. Also.. YAML makes me shudder.

[@BenReese]

Yeah.... I'm gonna have to watch this one about 6 more times.

[@christianlempa]

Hope you have fun :D

[@corincorvus6679]

So Kestra replaced your Ansible Semaphore Installation in a past Video?

[@christianlempa]

I'm still running Ansible Semaphore for testing, but most things I'm doing in Kestra now. Still some Ansible Semaphore content is planned for the future :)

[@isaacaymerich2741]

I'm tired to see more tools that are not solving anything, this is just another nice UI, you guys need to start learning nix, a real killer solution.

[@avluis86]

I heard you like to automate so I automated your automation 🤖

[@christianlempa]

Haha :D

[@dus10dnd]

Ansible and Terraform are both Infrastructure as Code tools. Ansible is more focused on configuration management and Terraform is more focused on deployment. Terraform + Ansible = Terrible :)

[@RuanBekker]

The only place I really use ansible lately is using Packer with the ansible provisioner to build AWS EC2 AMIs and then using the data resource to retrieve those amis when deoloying ec2. Works not too bad

[@HZGF]

AWX and terraform cloud with a self hosted runner make much more sense to me then trying to „rape“ kestra to do it in a less nice way 🤷‍♂️

[@kimcosmos]

sure if you want to pay for Tower to get the terraform capability in AWX

[@mickolesmana5899]

huh so it is like airflow but more for infra

[@itslemonandrew]

I want to disagree with the first part of the video that you need to run the commands in your project. With Semaphore that you talked about a while ago is made for this scenario.

[@christianlempa]

Semaphore could be an alternative to Kestra, but without anything like it, you have the same problem.

[@keywal]

The more of these videos you make the less respect I have for you. You’ve turned yourself again into an advertiser for your sponsors and in a few weeks it’ll be something else you have been using in your lab. I’m over it - cheerio

[@avluis86]

Personally, I found it very helpful as I've been eyeing this product since recent discovery -- sure we may move away from it in the future but that is the point of a homelab in the first place -- trial all the things!!

[@bubi352]

bye, bye!

[@bluesquadron593]

There is a solution for your problem, skip the content you don't like.

[@maplenerd22]

I don't see this as a problem as long as he clearly states that this video is sponsored by the product.

[@sloraris]

Advertising is why you get sponsored... Kinda the whole point. Good software and content costs time, and time is money. Sorry not everything is free.