How to Speed Up Ansible Playbooks Over 600%

[@toptechskills]

G'day everyone, if you're currently learning Ansible and want to see how all these modules fit together to build something useful, please consider checking out my course and letting me know what you think! It's currently in "early access" while I finish off the last chapter, and the first part of the course is free so you can try it out: learn.toptechskills.com/courses/productive-with-ansible. Thanks for all your support and happy coding!

[@amizednix]

It's 2021 and this is still so useful and helpful! Great videos.

[@toptechskills]

Thank you Amin, very glad it's still useful for you. I know molecule has changed a bit since the recording, but I hope people are able to get what they need to get started from this video.

[@dhir_AND]

Exactly what I was looking for. Well demonstrated !!

[@toptechskills]

Glad the video was what you were looking for, thank you for your kind comment 🙏

[@bab333bab]

Awesome video! Voice, presentation and comparison are perfect, thank you very much!

[@toptechskills]

You're very welcome, thanks for your kind comment 🙏

[@jayeshdhale2226]

Percy This is one of awesome video series of ansible I ever come across, Awesome

[@toptechskills]

Thank you for your kind comment jayesh dhale, glad it was useful for you.

[@toptechskills]

Thanks for watching everyone, I hope you found the video useful! Here's the link to the full article, if you want to check out the detailed benchmarks: bit.ly/2Slzbpa Please comment if you have any questions or feedback.

[@hprangana]

thanks a lot. it helped me a lot

[@binaryfire]

Great video, thanks! Is there a benefit to using both "pipelining = true" and "strategy = mitogen_linear" at the same time?

[@toptechskills]

Great question, _in theory_ pipelining should affect any ssh-based connection plugin, including Ansible's default ssh connection plugin and maybe mitogen too, since it's based on ssh. So yeah, in theory I think there could be some effect, but I didn't test it, so I would be curious to see your results if you end up testing it!

[@alexevinson5267]

Thanks for sharing really good stuff, Percy, your tutorials are extremely useful and very well done! I wish you to grow your channel up to 200k subscribers at least, just because you deserve :) By the way, Ansible playbooks can run even faster with asynchronous mode in tasks (especially multiple asynchronous tasks running concurrently), highest number of forks set, and with facts and connection caching enabled, but the last option is for the huge number of hosts to be served at once. Give it a try!

[@toptechskills]

Hi Alex, thanks so much for those great suggestions. I actually wasn't aware of inventory or fact caching, these could help a lot!

[@italiouine8909]

what command to display (4:21)

[@toptechskills]

Hi Brahim, the window at 4:21 is `iTerm` (www.iterm2.com/) and I have divided the window vertically 3 times with the "Shell > Split vertically" option. Hope that helps, thanks for your comment.

[@lvlupdev]

Hi Percy, thank you very much, you solved a 10-hours problem in 7 minutes.

[@toptechskills]

Awesome, really happy to hear that! Thanks for the comment 🙏

[@ashokpj3807]

Your channel videos are very useful.

[@toptechskills]

So happy the videos are useful for you, thank you for your comment 🙏

[@dialaraBilbob]

hi, does it matter whichpath you extract the tar file? I extracted and move it to /etc/ansible directory ,when I change ansible.cfg strategy=mitogen_linear, and run playbook it gives error.

[@toptechskills]

Hi Dilara, it should not matter where the files are stored, but I think you still might need to set the `strategy_plugins` option to the correct path: ``` # ansible.cfg [defaults] strategy_plugins = /path/to/mitogen-0.2.5/ansible_mitogen/plugins/strategy strategy = mitogen_linear ``` Hope that helps!

[@enneydee]

Hi Percy, have you tried using "strategy = mitogen_free"? Is there a great difference with the default config "strategy = mitogen_linear"? Please provide insights. Thank you!

[@toptechskills]

I generally stick with the default strategy for Ansible (linear) regardless of the connection plugin (ssh, mitogen, etc.) The `linear` (and `mitogen_linear`) strategy both wait for all hosts to complete each task before moving on to the next task, whereas `free` (and `mitogen_free`) will execute tasks as fast as possible on all hosts without waiting for slower hosts. To me, the free strategy is harder to reason about and if you look at your playbook overall, it's not actually going to complete any faster, since your playbook is still going to complete as fast as your _slowest_ host anyway. Some of your hosts might be finished more quickly, but your playbook won't complete until the slowest host is done, so you still need to wait the same length of time. The linear strategy also allows you to deal with hosts that have a dependency on each other, e.g. you want to know that a process has stopped on all hosts in the play before changing the configuration. With the linear strategy, you can know with certainty that when you start a task on any host, that all other hosts in the play have completed the previous task and are at the same stage. All in all, I don't see much value in the `free` strategy. docs.ansible.com/ansible/latest/user_guide/playbooks_strategies.html

[@printjunkee]

Hi Percy, thanks for sharing this. Could you please tell me if can use both pipelining and mitogen plugin together, i.e. in the same configuration file. Also, would it better to just use mitogen plugin or both pipelining and mitogen together to speed up the playbook?

[@toptechskills]

Hi Harshit Chawla, great question, from my understanding the pipelining setting should have no effect on mitogen, but I could be wrong. I will need to try it out later. I know that mitogen operates over most of the transport plugins by establishing a connection to a python agent. In theory, pipelining should have no effect on this. You might want to try enabling and disabling pipelining when using mitogen and see what the results are, I expect there will be no difference.

[@printjunkee]

​@@toptechskills Yes you are right, I tried enabling and disabling pipelining with mitogen and there was no difference in the results.

[@enneydee]

This is my current setup and it doesnt hurt. Makes no upfront noticable difference. What I'd like to know is if someone has tried using "strategy = mitogen_free"...? [defaults] gather_timeout = 30 strategy_plugins = /usr/local/mitogen/ansible_mitogen/plugins/strategy strategy = mitogen_linear callback_whitelist = timer, profile_tasks [ssh_connection] pipelining = True

[@veejeyz]

Hi @Percy.. The contents you teach are extremely helpful and I'd love if you could make one to better understand the deploy_helper module in ansible? Thanks again!!!!

[@toptechskills]

Thanks for the heads up about `deploy_helper`, I hadn't seen that module before but it looks super useful. Thanks again for the comment 🙏

[@infantusgodfrey967]

My controller is centos7 and when connecting with ubuntu16 it shows the following issue as ubuntu16 has ansible_python_interpreter=/usr/bin/python3 EOF on stream; last 100 lines received: ansible@192.168.0.105's password: bash: /usr/bin/python: No such file or directory

[@toptechskills]

I have an article for something similar (www.toptechskills.com/ansible-tutorials-courses/how-to-fix-usr-bin-python-not-found-error-tutorial/), but it looks like your issue is slightly different. Can you share the playbook?

[@mikecmw8492]

Hi, do you have any ansible videos showing how to use the cron module? Need help trying to get this to work.

[@toptechskills]

Hi Mike, unfortunately I haven't had time to record a video for the cron module yet. What's the problem you're having? I'm happy to help you get it working.

[@mikecmw8492]

@@toptechskills the interesting part is I got it working. I did not realize how easy the cron module is to use. Been starting ansible last month or two. There are some hangups. Maybe I will ask as I get to them. I will watching your videos going forward. Thanks

[@toptechskills]

@@mikecmw8492 great that you got it working. I find the best way to get familiar with a new module is to look at the example commands at the bottom of the module docs on docs.ansible.com. If you run into any roadblocks feel free to hit me up at [e-mail]. Good luck!

[@tjwatts100]

Amazing. I'm.going to test mitogen at work

[@sandeeputb]

Thanks for your video. Can u tell me the ide name u are using for yml.

[@toptechskills]

Hi Sandeep, my editor is VS Code (code.visualstudio.com/). Highly recommend this editor as your IDE.

[@boopathi.kkanagaraj9055]

does mitogen work for windows hosts

[@toptechskills]

No, I don't think so

[@compumatterllc]

Ecstatic I stumbled upon this post. I have a very wide variety of tasks being run including downloads, software installs, apache and nginx server setups and much more. So it's a lot more complex than just a few text-oriented tasks. My results were not where near as impressive: Only a 40 second improvement. With pipelining turned on in both cases, it took my sizeable playbook down from: * Playbook run took 0 days, 0 hours, 10 minutes, 0 seconds TO * Playbook run took 0 days, 0 hours, 9 minutes, 20 seconds Do you know of a way to debug / troubleshoot this? Wondering if Mitogen is actually working. Followed your instructions to a 'T'

[@toptechskills]

I have more details in the article I wrote that the video is based on: www.toptechskills.com/ansible-tutorials-courses/speed-up-ansible-playbooks-pipelining-mitogen/#initial-run. In this section I showed that there was little time benefit of mitogen vs pipelining for the _initial_ run of Ansible for a given set of roles. I hypothesize that this is primarily due to the fact that most of the time in the initial run of the playbook is spent actually doing the installation tasks, rather than in "Ansible overhead". In subsequent runs, and in runs on hosts where the ping is high, "Ansible overhead" forms a bigger portion of the overall runtime. Since mitogen primarily reduces the time associated with overhead, but does nothing to speed up the actual tasks (i.e. running "apt install" will be just as fast regardless of whether you did it with mitogen or pipelining). I'm assuming in your case that your results are from a "fresh" run of a playbook (i.e. none of the software is installed already), the ping to the server is very low or it's running locally. In all of those cases I would expect the overhead to be low and for mitogen to give only limited improvements.

[@jaylepore5420]

@@toptechskills Thanks for the write back. Yes, of little value from what I've experienced on my initial run. Will try it again on the update side of life and see how it goes. Thanks.

[@ramodhyarathnayake8035]

Awesome Teaching. Great Video, Please do more videos about Ansible. Thanks

[@toptechskills]

Thanks for your kind comment Ramodhya Rathnayake, very happy you enjoyed it and I will release more videos in future.

[@AnCacaMilis]

Great vid. Very concise

[@toptechskills]

Thanks for your comment @AnCacaMilis, glad you enjoyed it.

[@sidweb3.2]

Great video Percy! May I know which tool are you using to parallelly writing + executing the yml in single window ??

[@toptechskills]

Hi Sidharth paul, I'm using Visual Studio Code as my editor (code.visualstudio.com/) with the built-in terminal. I'm a big fan of this editor, hope you can give it a go. Thanks a lot for your comment.

[@ts-vq7gc]

thanks! great video

[@toptechskills]

You're welcome, hope it was helpful, thank you for your comment 🙏

[@narasimhangopinath1559]

Thanks a lot bro.. Mitogen works cool 👍

[@toptechskills]

Thanks Narasimhan Sargur, I hope it improves your Ansible performance!

[@ahsan-li7sh]

Great video

[@toptechskills]

Thank you for watching Ahsan!

[@johnclarkon369]

Thx. Nfs

[@account-pending-deletion]

what kind of madman uses vs code in any color theme other than Monokai Dimmed? especially a LIGHT theme? what the actual hell, man

[@toptechskills]

Haha, I did some experiments before making videos and found that dark text on a light background is easier to read at small sizes or when compressed, the contrast seemed better to me. When viewing the video minimized or on a phone screen it was clearer.

[@DaVince21]

Hey now, bright themes are excellent when you're doing coding in your backyard!

[@DaVince21]

Seven ex? The x stands for times, you know. :P Anyway, thanks for this informative video, this is helping our team make decisions with regards to pipelining.

[@toptechskills]

Thanks for the comment, glad the information was helpful for you and your team.