Rapidly Build & Test Ansible Roles with Molecule + Docker

[@toptechskills]

G'day everyone, if you're currently learning Ansible and want to see how all these modules fit together to build something useful, please consider checking out my course and letting me know what you think! It's currently in "early access" while I finish off the last chapter, and the first part of the course is free so you can try it out: learn.toptechskills.com/courses/productive-with-ansible. Thanks for all your support and happy coding!

[@creatldd1creatldd165]

Thanks for this great video, well edited, that offers a guided tour of Molecule with Docker. I also appreciate the quick pace of the tour, with no wasted sec.

[@Croswent]

This was awesome, very well explained! 😁 Cheers mate

[@toptechskills]

Thanks for your comment Croswent, hope the video helped!

[@timothyn4699]

well explained and I appreciate you breaking down the steps, helps demystify Molecule and give me a better gist and outline of what's happening

[@toptechskills]

Hi Timothy, thank you and I'm glad the video was helpful for you. Happy you're giving molecule a try, I think it's a great tool. All the best!

[@phil7587]

Thank you, it was exactly what I needed for a first clear and simple introduction

[@toptechskills]

Very glad it was useful for you Philippe, thank you for your comment 🙏

[@muninmatt]

Hey Percy! I'm just now learning molecule myself, and I realized that since making this video, they've released Molecule 3! Some of the syntax and things are different, I would LOVE to see a video that shows how to get started in the new version. Thanks!

[@toptechskills]

Hi muninmatt, thanks a lot for the feedback, I'll need to take a look at the exact differences. Molecule has been a little lax in the upgrade guide for v3 haha: molecule.readthedocs.io/en/latest/upgrade.html. Anyway, hope this video was enough to at least get you started in v3. As I understand, the main differences are that the verify driver has been changed to be Ansible (instead of testinfra) and the default names for the playbooks have changed somewhat. Will do my best to get on it if I have time. Thanks again for the comment.

[@gabugmola]

I'm also interested but I guess I'll investigate it myself.

[@dhruva023]

This tutorial has been very helpful. Thanks you!

[@toptechskills]

Thank you for the kind comment, dhaval vyas

[@sherrychang951]

Hi Percy 😁 Well done! !!

[@toptechskills]

Hi Sherry, thank you! 😅

[@ahsan-li7sh]

Excellent tutorial

[@kevinbates5763]

Super well explained! Thanks a lot!

[@toptechskills]

Glad you liked it Kevin, thank you.

[@medelkhalki2748]

awesome and perfect tutorial nice work

[@toptechskills]

Thank you so much 🙏

[@TechWorldwithPankaj]

Awesome👍

[@houxizhu]

this is nice and really helpful

[@toptechskills]

Thank you very much for the positive comment Xizhu, I'm glad the video was helpful.

[@artgilbert5182]

I found that setting up molecule outside of virtualenv is very buggy. When I use virtualenv, everything works as expected.

[@tutunak]

I think it's because you have problem with installed python packages. I'm using molecule + tox.

[@6006133]

Same experience here. Happens every time with python. Its toolchain is such a mess.

[@chikosan99]

wowww!! you are amazing!

[@toptechskills]

Thank you for your very kind comment Shai Chikorel 🙏

[@slimaloui920]

Perfect. Thank you so much.

[@toptechskills]

You're very welcome, thank you for your comment 🙏

[@ilGhepa]

Does molecule works ok with mitogen? I know the connection is local to your docker container but as you may know ansible creates and kills the connection for each task, instead mitogen changes this behaviour to one single stable connection per each target. So when testing multiple tasks/hosts this might speed up the process

[@toptechskills]

Good question, I'm not really sure about that. I have certainly noticed that even though the docker connection is local, molecule runs can be quite slow. If you're able to get this working I'd be really interested to see your results.

[@Emily-fm3no]

sir any videos like this for ansible test kitchen? how to write test cases for my roles using test kitchen .

[@toptechskills]

Sorry Sunitha AR, I don't have any videos to recommend, I hope you can find what you're looking for.

[@D1Ck3n]

Do you plan to update this tutorial? because the steps does not work anymore on the latest molecule version (molecule 3.1.5) :(

[@toptechskills]

I will need to update it for version 3 of Molecule, but I haven't had time. Hopefully I can do it in future. Sorry this video is no longer useful!

[@jayeshdhale2226]

Hi Percy,hope you are doing great Just wanted to know is that mitogen is a open source tools or do we have to pay for it

[@toptechskills]

Mitogen is a free and open source tool: github.com/dw/mitogen/ I don't see a license on the Github page, though, so I would contact the author via Github if you're worried about that.

[@D1Ck3n]

Thanks for the great video! I would be interested to know how far you can push this? Would it be possible to test a shell script with it? I have one that calls local ansilbe playbooks.

[@toptechskills]

From my understanding of Molecule's use case, it's more of a unit testing framework for individual roles only. I think from the Molecule perspective on your use case the advice would be to refactor your playbooks into roles and then test those roles. I think having a shell script calling multiple playbooks suggests to me that roles might better suit your use case anyway. Hope that's somewhat helpful, thanks for your comment.

[@artgilbert5182]

Very well done. Thanks.

[@michaeljones9229]

When running " molecule init role --role-name ansible-role-redis --drivername docker" i get the error "Usage: molecule init role [OPTIONS] ROLE_NAME Try 'molecule init role --help' for help. Error: No such option: --role-name (Possible options: --driver-name, --lint-name, --provisioner-name): Not sure what to do from here. I had a workaround where I ran "molecule init role acme.moltest --driver-name=docker" and it works but it does not have the tests directory you have and other code.

[@toptechskills]

This is most likely because this tutorial was made with an older version of molecule and the signatures of the commands have changed since then. Hopefully most of the video is still useful even if some of the commands need to be updated.

[@Faruqisan]

Hi, how to run an integration test with Molecule? let's say I have a role that setup database replication and I need one container run as a master DB before my molecule test run?.

[@toptechskills]

Hi Ikhsan Faruqi, You can definitely achieve this by adding more `platforms` in your `molecule.yml` to take the place of these machines. You can put your hosts in `platforms` in different groups and then configure them in the `playbook.yml`. For example: ``` # molecule.yml platforms: - name: leader image: ubuntu:18.04 groups: ["leader"] - name: follower image: ubuntu:18.04 groups: ["follower"] ``` ``` # playbook.yml --- - name: Converge hosts: all roles: - role: db_base - role: db_leader when: "'leader' in group_names" - role: db_follower when: "'follower' in group_names" ``` Hope this helps, thanks for your comment!

[@Faruqisan]

@@toptechskills Hi Percy, Thank you for your answer this is really helpful!

[@suniljamadar1566]

I'm running Ansible on Amazon-linux2 and roles under it, I installed molecule using pip and docker-py as well. When I say molecule test it's giving me an error saying some apiversion error, can anyone pls help

[@toptechskills]

Hi Sunil, can you share the exact command and error message? I know that Molecule has released a new major version since I released this video and certain settings have changed. More details at molecule.readthedocs.io/en/latest/.

[@suniljamadar1566]

@@toptechskills sure and thank you for your time

[@suniljamadar1566]

@@toptechskills it's resolved and I reinstalled docker

[@crytom3915]

How do I write tests if I have different tasks for different instances ? Just one testing file for all instances ? If yes, how to define the testings for each instance in the testing file ?

[@toptechskills]

Hi Cry Tom, what's your specific use case? Why are you trying to run different tests against different instances? Without knowing your specific use cases, one thing that might be helpful is to use "scenarios" (molecule.readthedocs.io/en/latest/getting-started.html#molecule-scenarios). Each scenario has its own molecule.yml and can run a different playbook and tests. You could create a scenario for each group of servers you want to test. You can run different molecule scenarios using the -s or --scenario-name option: molecule test --scenario-name web Let me know if this helps at all. Thanks a lot for the comment.

[@mikecmw8492]

Hi Percy. What is the best place to include ansible script which are part of a git project? Meaning I have a code base whether its scripts of compiled code and I want to install this after building. So Project123 has my code. Where would I include Project1234-ansible.yml? Should that go into the Project123 repo or some other outside project?

[@toptechskills]

Hi Mike, thanks for your message. I have seen 2 conventions on the projects I've worked on, and I think both are logical and effective for different use cases. Convention 1: put all Ansible-related files in an "./ansible" directory relative to the root of your source code . ├── LICENSE ├── README.md ├── ansible │ ├── README.md │ ├── ansible.cfg │ ├── ... ├── ... In my experience this convention has served me really well with smaller projects since it allows the code and the infrastructure to live in the same repo, which makes changes to configuration and code easy to track in a single place in Git. I have followed this convention when adding Terraform to projects too, with similar success. Convention 2: have a completely separate repo that has the Ansible/IAC code for multiple applications/projects, with directories for each application/project and potentially shared directories with common roles, files and playbooks. This has worked well when managing multiple very large applications for a single org. Allows for DRYness because you can easily share roles and other things without duplicating them. The only issue I have found with this convention is that Ansible can be a bit unintuitive about relative directories, so referencing common files relatively can be a little unclear. Based on your description of your project, I'd say convention 1 sounds more suitable. Let me know if you have any more questions about this.

[@mikecmw8492]

@@toptechskills Wow, the longest answer I have received by any YT'r and makes sense. Thanks a ton!

[@Oswee]

Nice intro! But how do you manage SystemD services in rootless containers within Molecule?

[@toptechskills]

The docker images I reference have `systemd` as their `CMD` by default: github.com/percygrunwald/docker-ubuntu1804-ansible/blob/master/Dockerfile, so it should be running as PID 1 unless you are passing a custom command. This is not normal for a production Docker container, but is useful for molecule testing. You should be able to use these containers in a similar way to how you would use a full Ubuntu VM with respect to how it treats systemd units. Did you have any problem working with systemd using these container images?

[@Oswee]

@@toptechskills Thank you for your response. Was not expecting that. :) Well... first, Docker in no-go for me there. Podman is. I want to test HAProxy role which in real-world will be managed by Systemd. On top of that... I need to set SELinux booleans for HAProxy to be able make some connections etc... With Systemd it's a less pain i think... but... on SELinux things i failed totally for now. Sure.. i can skip the SELinux tasks, but its not "close to real-world". So i went other route... by using Vagrant driver with Libvirt plugin. But... as it is at 2021.. Vagrant still has no 1st class Libvirt support. There are many issues out of the box. Why would i even consider VirtualBox on the KVM machine? :) So... Not really you can help me there much... I know that majority of audience are OK to turn off SELinux at all and run containers as root. But that's not for me. :) Even for testing. Anyway.. really enjoying many of your videos. Thank you for your contributions. :)

[@toptechskills]

Yeah, this is a more complex problem that seems better suited to a full VM driver or some cloud-based compute driver (e.g. EC2). This may be slightly slower, but it will likely be closer to your actual use case.

[@Oswee]

@@toptechskills Exactly. This is what i thought as well. But... there have you some funny idea for the video. You definitely know the term "Yak shaving". Right? So... my story is right about that. It started yesterday, when i decided to share the role across several infra projects. :D Instead just copy/pasting i am spending almost 2 days on tinkering with tooling. Sure.. i am learning new stuff alongside... and yet... i am shaving the yak. :)

[@italiouine8909]

I have a problem with molecule test.... Please help, thinks --- dependency: name: galaxy driver: name: docker lint: name: yamllint platforms: image: centos:7 name: instance provisioner: lint: name: ansible-lint name: ansible scenario: name: default verifier: lint: name: flake8 name: testinfra ERROR: Failed to pre-validate. {'lint': ['must be of string type'], 'platforms': ['must be of list type']}

[@toptechskills]

Regarding the `lint` issue, in version 3 of molecule, the `lint` configuration item has changed to accept a string instead of an object, please see molecule.readthedocs.io/en/latest/configuration.html#lint for more details. Regarding the `platforms` key, you need to set this to a list of items. You should change the `platforms` key to this in your example: ``` platforms: - image: centos:7 name: instance ``` This is the YAML list syntax: docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html#yaml-basics Hope that helps

[@hubstrangers3450]

thank for the content, however would like to know do you have a git repos?

[@toptechskills]

Hi hub strangers, I don't have git repos for any of my current videos, but you can check out my github profile at github.com/percygrunwald

[@omegabk]

hello, thanks for this great video. Could you please fork and add geerlingguy centos8 and debian10 docker image for non root user ? thanks

[@toptechskills]

Thank you for your kind comment, I'll try to do it if I have time. If anyone else has done this, please share the link, thanks!

[@suniljamadar1566]

Docker should installed and docker-py also required to run this right ??

[@toptechskills]

That's correct

[@suniljamadar1566]

@@toptechskills thank you for the response.😊

[@suniljamadar1566]

I'm trying this on only one machine, which contains Ansible, docker, docker-py, molecule on Amazon Linux 2 will it work on only single host I mean localhost and also how to use molecule for existing roles that you haven't show in this video. Can u please tell us how to do for already existing roles??

[@toptechskills]

If you use the docker driver for Molecule it is designed to test on just the local host. It will create docker containers based on the `platforms` definition in your `molecule.yml` to be the target hosts for your role under test. For existing roles, you can follow the instructions here: molecule.readthedocs.io/en/latest/usage.html#molecule.command.init.scenario.Scenario Hope that helps, thanks!

[@THIRSTYGNOME]

Is there a recommended example for adopting existing roles into molecule?

[@toptechskills]

Hi Daniel, sorry, it appears my reply did not make it through after submitting it. I'll try again: The command to add molecule to an existing role is `molecule init scenario`. For example, if you have a role called ansible-role-redis that doesn't have molecule, you can add it like this: $ cd ansible-role-redis $ molecule init scenario --role-name ansible-role-redis This will add the molecule directory complete with all the boilerplate.

[@THIRSTYGNOME]

@@toptechskills Wow, that's pretty simple!

[@iMeGAMaN227]

I’ve tried molecule but can’t figure out a way to use it for my job. Basically our software suite is automation of building a suite of virtual machines. We use VMware to create these vms. So I would need molecule to not create a docker image but instead run our build vm playbook. Do you know how this would work?

[@toptechskills]

Hi Sean, thanks for your comment. Based on my reading of the docs, I don't think Molecule fits your use case particularly well. It might make sense for testing some of the individual Ansible roles that make up your suite though.

[@vincenieuwenburg2397]

I understand your use-case, but you are talking about a full deployment. You should use a ci/cd pipeline for that, such as Jenkins. The purpose of molecule is only to perform unit tests on playbooks. Meaning to check the quality of the code and making sure no bugs are introduced. Molecule focuses on testing that on your local laptop, and docker is a great tool for that. However, molecule does have support for different drivers. You should check the documentation, perhaps it supports VMware!

[@toptechskills]

@@vincenieuwenburg2397 Thanks for the reply Vince, I totally agree with what you said.

[@anatolylyssak7326]

I have following error running molecule test ..... TASK [Wait for instance(s) deletion to complete] ******************************* failed: [localhost] (item=None) => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} fatal: [localhost]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} .... Any idea? Thank you.

[@toptechskills]

Hi Anatoly, you can run `molecule --debug test` to get more verbose output to find the cause of the error. I think it's probably something related to Docker, maybe permissions or the daemon is not running. Let me know what happens when you run with --debug.

[@anatolylyssak7326]

@@toptechskills Hi Percy. I've fixed this problem by 1. Adding my user (non-root) to group 'docker'; 2. Checking that docker server is running; Thank you very much for your work. I've seen most of you Ansible stuff. It is very helpful.

[@michaelventarola7100]

I seem to get the following FAILED error. Is there a way to stop this from happening? TASK [Wait for instance(s) creation to complete] ******************************* FAILED - RETRYING: Wait for instance(s) creation to complete (300 retries left). changed: [localhost] => (item=None) changed: [localhost] => (item=None) changed: [localhost]

[@toptechskills]

Hi Michael, this is the normal behavior, Ansible is polling Docker and waiting for the image creation to complete. It's normal for it to fail a few times before going to "changed" (meaning the task is complete). It may seem like something is wrong, but I can assure you there were no errors if the execution continues with a "changed" status.

[@JerryOmann]

Can you update this Video the driver-name docker is not supported anymore. Error: Invalid value for "--driver-name" / "-d": invalid choice: docker. (choose from delegated) But awesome Video. I liked it very much!!

[@toptechskills]

Thanks for your comment Jerry, I definitely need to post an updated video for Molecule 3.x 🙏

[@CaHeoMapMap]

Very nice

[@toptechskills]

Thanks Linh!

[@CaHeoMapMap]

It is exactly what I need now, hehehe.

[@markwork5062]

Reedus instance... but seriously . Nice video

[@sqanto1990]

Nice My CI/CD say thanks

[@vincenieuwenburg2397]

The video is great, but I'm just not sure if I can forgive the absence of a night theme in VScode.

[@toptechskills]

Hahaha, thanks for the comment. I tend to use both dark and light themes day to day, but I found the light theme contrast better for videos..

[@suniljamadar1566]

Molecule doesn't work on python2.7 it required python3*..

[@toptechskills]

That's correct, since the update to Molecule 3 they have bumped the requirements for Python and Ansible (molecule.readthedocs.io/en/latest/installation.html#requirements): Python >= 3.6 with Ansible >= 2.8 At the time this video was made, Molecule was still in version 2 and the requirements were different.