Testing Strategies

Integrating Testing With Ansible Playbooks

Many times, people ask, “how can I best integrate testing with Ansible playbooks?” There are many options. Ansible is actually designedto be a “fail-fast” and ordered system, therefore it makes it easy to embed testing directly in Ansible playbooks. In this chapter,we’ll go into some patterns for integrating tests of infrastructure and discuss the right level of testing that may be appropriate.

Note

This is a chapter about testing the application you are deploying, not the chapter on how to test Ansible modules during development. For that content, please hop over to the Development section.

By incorporating a degree of testing into your deployment workflow, there will be fewer surprises when code hits production and, in many cases,tests can be leveraged in production to prevent failed updates from migrating across an entire installation. Since it’s push-based, it’salso very easy to run the steps on the localhost or testing servers. Ansible lets you insert as many checks and balances into your upgrade workflow as you would like to have.

The Right Level of Testing

Ansible resources are models of desired-state. As such, it should not be necessary to test that services are started, packages areinstalled, or other such things. Ansible is the system that will ensure these things are declaratively true. Instead, assert thesethings in your playbooks.

  1. tasks:
  2.   - service:
  3.       name: foo
  4.       state: started
  5.       enabled: yes

If you think the service may not be started, the best thing to do is request it to be started. If the service fails to start, Ansiblewill yell appropriately. (This should not be confused with whether the service is doing something functional, which we’ll show more about how todo later).

Check Mode As A Drift Test

In the above setup, –check mode in Ansible can be used as a layer of testing as well. If running a deployment playbook against anexisting system, using the –check flag to the ansible command will report if Ansible thinks it would have had to have made any changes tobring the system into a desired state.

This can let you know up front if there is any need to deploy onto the given system. Ordinarily scripts and commands don’t run in check mode, so if youwant certain steps to always execute in check mode, such as calls to the script module, disable check mode for those tasks:

  1. roles:
  2.   - webserver
  3.  
  4. tasks:
  5.   - script: verify.sh
  6.     check_mode: no

Modules That Are Useful for Testing

Certain playbook modules are particularly good for testing. Below is an example that ensures a port is open:

  1. tasks:
  2.  
  3.   - wait_for:
  4.       host: "{{ inventory_hostname }}"
  5.       port: 22
  6.     delegate_to: localhost

Here’s an example of using the URI module to make sure a web service returns:

  1. tasks:
  2.  
  3.   - action: uri url=http://www.example.com return_content=yes
  4.     register: webpage
  5.  
  6.   - fail:
  7.       msg: 'service is not happy'
  8.     when: "'AWESOME' not in webpage.content"

It’s easy to push an arbitrary script (in any language) on a remote host and the script will automatically fail if it has a non-zero return code:

  1. tasks:
  2.  
  3.   - script: test_script1
  4.   - script: test_script2 --parameter value --parameter2 value

If using roles (you should be, roles are great!), scripts pushed by the script module can live in the ‘files/’ directory of a role.

And the assert module makes it very easy to validate various kinds of truth:

  1. tasks:
  2.  
  3.    - shell: /usr/bin/some-command --parameter value
  4.      register: cmd_result
  5.  
  6.    - assert:
  7.        that:
  8.          - "'not ready' not in cmd_result.stderr"
  9.          - "'gizmo enabled' in cmd_result.stdout"

Should you feel the need to test for existence of files that are not declaratively set by your Ansible configuration, the ‘stat’ module is a great choice:

  1. tasks:
  2.  
  3.    - stat:
  4.        path: /path/to/something
  5.      register: p
  6.  
  7.    - assert:
  8.        that:
  9.          - p.stat.exists and p.stat.isdir

As mentioned above, there’s no need to check things like the return codes of commands. Ansible is checking them automatically.Rather than checking for a user to exist, consider using the user module to make it exist.

Ansible is a fail-fast system, so when there is an error creating that user, it will stop the playbook run. You do not haveto check up behind it.

Testing Lifecycle

If writing some degree of basic validation of your application into your playbooks, they will run every time you deploy.

As such, deploying into a local development VM and a staging environment will both validate that things are according to planahead of your production deploy.

Your workflow may be something like this:

  1. - Use the same playbook all the time with embedded tests in development
  2. - Use the playbook to deploy to a staging environment (with the same playbooks) that simulates production
  3. - Run an integration test battery written by your QA team against staging
  4. - Deploy to production, with the same integrated tests.

Something like an integration test battery should be written by your QA team if you are a production webservice. This would includethings like Selenium tests or automated API tests and would usually not be something embedded into your Ansible playbooks.

However, it does make sense to include some basic health checks into your playbooks, and in some cases it may be possible to runa subset of the QA battery against remote nodes. This is what the next section covers.

Integrating Testing With Rolling Updates

If you have read into Delegation, Rolling Updates, and Local Actions it may quickly become apparent that the rolling update pattern can be extended, and youcan use the success or failure of the playbook run to decide whether to add a machine into a load balancer or not.

This is the great culmination of embedded tests:

  1. ---
  2.  
  3. - hosts: webservers
  4.   serial: 5
  5.  
  6.   pre_tasks:
  7.  
  8.     - name: take out of load balancer pool
  9.       command: /usr/bin/take_out_of_pool {{ inventory_hostname }}
  10.       delegate_to: 127.0.0.1
  11.  
  12.   roles:
  13.  
  14.      - common
  15.      - webserver
  16.      - apply_testing_checks
  17.  
  18.   post_tasks:
  19.  
  20.     - name: add back to load balancer pool
  21.       command: /usr/bin/add_back_to_pool {{ inventory_hostname }}
  22.       delegate_to: 127.0.0.1

Of course in the above, the “take out of the pool” and “add back” steps would be replaced with a call to a Ansible load balancermodule or appropriate shell command. You might also have steps that use a monitoring module to start and end an outage windowfor the machine.

However, what you can see from the above is that tests are used as a gate – if the “apply_testing_checks” step is not performed,the machine will not go back into the pool.

Read the delegation chapter about “max_fail_percentage” and you can also control how many failing tests will stop a rolling updatefrom proceeding.

This above approach can also be modified to run a step from a testing machine remotely against a machine:

  1. ---
  2.  
  3. - hosts: webservers
  4.   serial: 5
  5.  
  6.   pre_tasks:
  7.  
  8.     - name: take out of load balancer pool
  9.       command: /usr/bin/take_out_of_pool {{ inventory_hostname }}
  10.       delegate_to: 127.0.0.1
  11.  
  12.   roles:
  13.  
  14.      - common
  15.      - webserver
  16.  
  17.   tasks:
  18.      - script: /srv/qa_team/app_testing_script.sh --server {{ inventory_hostname }}
  19.        delegate_to: testing_server
  20.  
  21.   post_tasks:
  22.  
  23.     - name: add back to load balancer pool
  24.       command: /usr/bin/add_back_to_pool {{ inventory_hostname }}
  25.       delegate_to: 127.0.0.1

In the above example, a script is run from the testing server against a remote node prior to bringing it back intothe pool.

In the event of a problem, fix the few servers that fail using Ansible’s automatically generatedretry file to repeat the deploy on just those servers.

Achieving Continuous Deployment

If desired, the above techniques may be extended to enable continuous deployment practices.

The workflow may look like this:

  1. - Write and use automation to deploy local development VMs
  2. - Have a CI system like Jenkins deploy to a staging environment on every code change
  3. - The deploy job calls testing scripts to pass/fail a build on every deploy
  4. - If the deploy job succeeds, it runs the same deploy playbook against production inventory

Some Ansible users use the above approach to deploy a half-dozen or dozen times an hour without taking all of their infrastructureoffline. A culture of automated QA is vital if you wish to get to this level.

If you are still doing a large amount of manual QA, you should still make the decision on whether to deploy manually as well, butit can still help to work in the rolling update patterns of the previous section and incorporate some basic health checks usingmodules like ‘script’, ‘stat’, ‘uri’, and ‘assert’.

Conclusion

Ansible believes you should not need another framework to validate basic things of your infrastructure is true. This is the casebecause Ansible is an order-based system that will fail immediately on unhandled errors for a host, and prevent further configurationof that host. This forces errors to the top and shows them in a summary at the end of the Ansible run.

However, as Ansible is designed as a multi-tier orchestration system, it makes it very easy to incorporate tests into the end ofa playbook run, either using loose tasks or roles. When used with rolling updates, testing steps can decide whether to put a machineback into a load balanced pool or not.

Finally, because Ansible errors propagate all the way up to the return code of the Ansible program itself, and Ansible by defaultruns in an easy push-based mode, Ansible is a great step to put into a build environment if you wish to use it to roll out systemsas part of a Continuous Integration/Continuous Delivery pipeline, as is covered in sections above.

The focus should not be on infrastructure testing, but on application testing, so we strongly encourage getting together with yourQA team and ask what sort of tests would make sense to run every time you deploy development VMs, and which sort of tests they would liketo run against the staging environment on every deploy. Obviously at the development stage, unit tests are great too. But don’t unittest your playbook. Ansible describes states of resources declaratively, so you don’t have to. If there are cases where you wantto be sure of something though, that’s great, and things like stat/assert are great go-to modules for that purpose.

In all, testing is a very organizational and site-specific thing. Everybody should be doing it, but what makes the most sense for yourenvironment will vary with what you are deploying and who is using it – but everyone benefits from a more robust and reliable deploymentsystem.

See also