Ansible module architecture

This in-depth dive helps you understand Ansible’s program flow to executemodules. It is written for people working on the portions of the Core AnsibleEngine that execute a module. Those writing Ansible Modules may also find thisin-depth dive to be of interest, but individuals simply using Ansible Moduleswill not likely find this to be helpful.

Types of modules

Ansible supports several different types of modules in its code base. Some ofthese are for backwards compatibility and others are to enable flexibility.

Action plugins

Action Plugins look like modules to end users who are writing playbooks butthey’re distinct entities for the purposes of this document. Action Pluginsalways execute on the controller and are sometimes able to do all work there(for instance, the debug Action Plugin which prints some text for the user tosee or the assert Action Plugin which can test whether several values ina playbook satisfy certain criteria.)

More often, Action Plugins set up some values on the controller, then invoke anactual module on the managed node that does something with these values. Aneasy to understand version of this is the template Action Plugin. The template Action Plugin takes values fromthe user to construct a file in a temporary location on the controller usingvariables from the playbook environment. It then transfers the temporary fileto a temporary file on the remote system. After that, it invokes thecopy module which operates on the remote system to move the fileinto its final location, sets file permissions, and so on.

New-style modules

All of the modules that ship with Ansible fall into this category.

New-style modules have the arguments to the module embedded inside of them insome manner. Non-new-style modules must copy a separate file over to themanaged node, which is less efficient as it requires two over-the-wireconnections instead of only one.

Python

New-style Python modules use the Ansiballz framework framework for constructingmodules. All official modules (shipped with Ansible) use either this or thepowershell module framework.

These modules use imports from ansible.module_utils in order to pull inboilerplate module code, such as argument parsing, formatting of returnvalues as JSON, and various file operations.

Note

In Ansible, up to version 2.0.x, the official Python modules used theModule Replacer framework framework. For module authors, Ansiballz framework islargely a superset of Module Replacer framework functionality, so you usuallydo not need to know about one versus the other.

Powershell

New-style powershell modules use the Module Replacer framework framework forconstructing modules. These modules get a library of powershell code embeddedin them before being sent to the managed node.

JSONARGS

Scripts can arrange for an argument string to be placed within them by placingthe string <<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>> somewhere inside of thefile. The module typically sets a variable to that value like this:

json_arguments = """<<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>>"""

Which is expanded as:

json_arguments = """{"param1": "test's quotes", "param2": "\"To be or not to be\" - Hamlet"}"""

Note

Ansible outputs a JSON string with bare quotes. Double quotes areused to quote string values, double quotes inside of string values arebackslash escaped, and single quotes may appear unescaped inside ofa string value. To use JSONARGS, your scripting language must have a wayto handle this type of string. The example uses Python’s triple quotedstrings to do this. Other scripting languages may have a similar quotecharacter that won’t be confused by any quotes in the JSON or it mayallow you to define your own start-of-quote and end-of-quote characters.If the language doesn’t give you any of these then you’ll need to writea non-native JSON module orOld-style module instead.

The module typically parses the contents of json_arguments using a JSONlibrary and then use them as native variables throughout the rest of its code.

Non-native want JSON modules

If a module has the string WANT_JSON in it anywhere, Ansible treatsit as a non-native module that accepts a filename as its only command lineparameter. The filename is for a temporary file containing a JSONstring containing the module’s parameters. The module needs to open the file,read and parse the parameters, operate on the data, and print its return dataas a JSON encoded dictionary to stdout before exiting.

These types of modules are self-contained entities. As of Ansible 2.1, Ansibleonly modifies them to change a shebang line if present.

See also

Examples of Non-native modules written in ruby are in the Ansiblefor Rubyists repository.

Binary modules

From Ansible 2.2 onwards, modules may also be small binary programs. Ansibledoesn’t perform any magic to make these portable to different systems so theymay be specific to the system on which they were compiled or require otherbinary runtime dependencies. Despite these drawbacks, a site may sometimeshave no choice but to compile a custom module against a specific binarylibrary if that’s the only way they have to get access to certain resources.

Binary modules take their arguments and will return data to Ansible in the sameway as want JSON modules.

See also

One example of a binary modulewritten in go.

Old-style modules

Old-style modules are similar towant JSON modules, except that the file thatthey take contains key=value pairs for their parameters instead ofJSON.

Ansible decides that a module is old-style when it doesn’t have any of themarkers that would show that it is one of the other types.

How modules are executed

When a user uses ansible or ansible-playbook, theyspecify a task to execute. The task is usually the name of a module alongwith several parameters to be passed to the module. Ansible takes thesevalues and processes them in various ways before they are finally executed onthe remote machine.

Executor/task_executor

The TaskExecutor receives the module name and parameters that were parsed fromthe playbook (or from the command line in the case of/usr/bin/ansible). It uses the name to decide whether it’s lookingat a module or an Action Plugin. If it’sa module, it loads the Normal Action Pluginand passes the name, variables, and other information about the task and playto that Action Plugin for further processing.

Normal action plugin

The normal action plugin executes the module on the remote host. It isthe primary coordinator of much of the work to actually execute the module onthe managed machine.

• It takes care of creating a connection to the managed machine byinstantiating a Connection class according to the inventoryconfiguration for that host.
• It adds any internal Ansible variables to the module’s parameters (forinstance, the ones that pass along no_log to the module).
• It takes care of creating any temporary files on the remote machine andcleans up afterwards.
• It does the actual work of pushing the module and module parameters to theremote host, although the module_commoncode described in the next section does the work of deciding which formatthose will take.
• It handles any special cases regarding modules (for instance, variouscomplications around Windows modules that must have the same names as Pythonmodules, so that internal calling of modules from other Action Plugins work.)

Much of this functionality comes from the BaseAction class,which lives in plugins/action/init.py. It makes use ofConnection and Shell objects to do its work.

Note

When tasks are run with the async: parameter, Ansibleuses the async Action Plugin instead of the normal Action Pluginto invoke it. That program flow is currently not documented. Read thesource for information on how that works.

Executor/module_common.py

Code in executor/modulecommon.py takes care of assembling the moduleto be shipped to the managed node. The module is first read in, then examinedto determine its type. PowerShell andJSON-args modules are passed throughModule Replacer. New-stylePython modules are assembled by Ansiballz framework.Non-native-want-JSON,Binary modules, andOld-Style modules aren’t touched by either ofthese and pass through unchanged. After the assembling step, one finalmodification is made to all modules that have a shebang line. Ansible checkswhether the interpreter in the shebang line has a specific path configured viaan ansible$X_interpreter inventory variable. If it does, Ansiblesubstitutes that path for the interpreter path given in the module. Afterthis, Ansible returns the complete module data and the module type to theNormal Action which continues execution ofthe module.

Next we’ll go into some details of the two assembler frameworks.

Module Replacer framework

The Module Replacer framework is the original framework implementing new-stylemodules. It is essentially a preprocessor (like the C Preprocessor for thosefamiliar with that programming language). It does straight substitutions ofspecific substring patterns in the module file. There are two types ofsubstitutions:

• Replacements that only happen in the module file. These are publicreplacement strings that modules can utilize to get helpful boilerplate oraccess to arguments.
  • from ansible.module_utils.MOD_LIB_NAME import * is replaced with thecontents of the ansible/module_utils/MOD_LIB_NAME.py These shouldonly be used with new-style Python modules.
  • #<<INCLUDE_ANSIBLE_MODULE_COMMON>> is equivalent tofrom ansible.module_utils.basic import * and should also only applyto new-style Python modules.
  • # POWERSHELL_COMMON substitutes the contents ofansible/module_utils/powershell.ps1. It should only be used withnew-style Powershell modules.
• Replacements that are used by ansible.module_utils code. These are internalreplacement patterns. They may be used internally, in the above publicreplacements, but shouldn’t be used directly by modules.
  • "<<ANSIBLEVERSION>>" is substituted with the Ansible version. Innew-style Python modules under theAnsiballz framework framework the proper way is to instead instantiate an_AnsibleModule and then access the version from:attr:AnsibleModule.ansible_version.
  • "<<INCLUDEANSIBLE_MODULE_COMPLEX_ARGS>>" is substituted witha string which is the Python repr of the JSON encoded moduleparameters. Using repr on the JSON string makes it safe to embed ina Python file. In new-style Python modules under the Ansiballz frameworkthis is better accessed by instantiating an _AnsibleModule andthen using AnsibleModule.params.
  • <<SELINUXSPECIAL_FILESYSTEMS>> substitutes a string which isa comma separated list of file systems which have a file system dependentsecurity context in SELinux. In new-style Python modules, if you reallyneed this you should instantiate an _AnsibleModule and then useAnsibleModule._selinux_special_fs. The variable has also changedfrom a comma separated string of file system names to an actual pythonlist of filesystem names.
  • <<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>> substitutes the moduleparameters as a JSON string. Care must be taken to properly quote thestring as JSON data may contain quotes. This pattern is not substitutedin new-style Python modules as they can get the module parameters anotherway.
  • The string syslog.LOGUSER is replaced wherever it occurs with thesyslog_facility which was named in ansible.cfg or anyansible_syslog_facility inventory variable that applies to this host. Innew-style Python modules this has changed slightly. If you really need toaccess it, you should instantiate an _AnsibleModule and then useAnsibleModule._syslog_facility to access it. It is no longer theactual syslog facility and is now the name of the syslog facility. Seethe documentation on internal argumentsfor details.

Ansiballz framework

Ansible 2.1 switched from the Module Replacer framework framework to theAnsiballz framework for assembling modules. The Ansiballz framework differsfrom module replacer in that it uses real Python imports of things inansible/module_utils instead of merely preprocessing the module. Itdoes this by constructing a zipfile – which includes the module file, filesin ansible/module_utils that are imported by the module, and someboilerplate to pass in the module’s parameters. The zipfile is then Base64encoded and wrapped in a small Python script which decodes the Base64 encodingand places the zipfile into a temp directory on the managed node. It thenextracts just the ansible module script from the zip file and places that inthe temporary directory as well. Then it sets the PYTHONPATH to find pythonmodules inside of the zip file and invokes python on the extractedansible module.

Note

Ansible wraps the zipfile in the Python script for two reasons:

• for compatibility with Python 2.6 which has a lessfunctional version of Python’s -m command line switch.
• so that pipelining will function properly. Pipelining needs to pipe thePython module into the Python interpreter on the remote node. Pythonunderstands scripts on stdin but does not understand zip files.

In Ansiballz, any imports of Python modules from theansible.module_utils package trigger inclusion of that Python fileinto the zipfile. Instances of #<<INCLUDE_ANSIBLE_MODULE_COMMON>> inthe module are turned into from ansible.module_utils.basic import *and ansible/module-utils/basic.py is then included in the zipfile.Files that are included from module_utils are themselves scanned forimports of other Python modules from module_utils to be included inthe zipfile as well.

Warning

At present, the Ansiballz Framework cannot determine whether an importshould be included if it is a relative import. Always use an absoluteimport that has ansible.module_utils in it to allow Ansiballz todetermine that the file should be included.

Passing args

In Module Replacer framework, module arguments are turned into a JSON-ifiedstring and substituted into the combined module file. In Ansiballz framework,the JSON-ified string is passed into the module via stdin. Whena ansible.module_utils.basic.AnsibleModule is instantiated,it parses this string and places the args intoAnsibleModule.params where it can be accessed by the module’sother code.

Note

Internally, the AnsibleModule uses the helper function,ansible.module_utils.basic._load_params(), to load the parametersfrom stdin and save them into an internal global variable. Very dynamiccustom modules which need to parse the parameters prior to instantiatingan AnsibleModule may use _load_params to retrieve theparameters. Be aware that _load_params is an internal function andmay change in breaking ways if necessary to support changes in the code.However, we’ll do our best not to break it gratuitously, which is notsomething that can be said for either the way parameters are passed orthe internal global variable.

Internal arguments

Both Module Replacer framework and Ansiballz framework send additional arguments tothe module beyond those which the user specified in the playbook. Theseadditional arguments are internal parameters that help implement globalAnsible features. Modules often do not need to know about these explicitly asthe features are implemented in ansible.module_utils.basic but certainfeatures need support from the module so it’s good to know about them.

_ansible_no_log

This is a boolean. If it’s True then the playbook specified nolog (ina task’s parameters or as a play parameter). This automatically affects callsto AnsibleModule.log(). If a module implements its own logging thenit needs to check this value. The best way to look at this is for the moduleto instantiate an _AnsibleModule and then check the value ofAnsibleModule.no_log.

Note

no_log specified in a module’s argument_spec are handled by a different mechanism.

_ansible_debug

This is a boolean that turns on more verbose logging. If a module usesAnsibleModule.debug() rather than AnsibleModule.log() thenthe messages are only logged if this is True. This also turns on logging ofexternal commands that the module executes. This can be changed viathe debug setting in ansible.cfg or the environment variableANSIBLE_DEBUG. If, for some reason, a module must access this, itshould do so by instantiating an AnsibleModule and accessingAnsibleModule._debug.

_ansible_diff

This boolean is turned on via the —diff command line option. If a modulesupports it, it will tell the module to show a unified diff of changes to bemade to templated files. The proper way for a module to access this is byinstantiating an AnsibleModule and accessingAnsibleModule._diff.

_ansible_verbosity

This value could be used for finer grained control over logging. However, itis currently unused.

_ansible_selinux_special_fs

This is a list of names of filesystems which should have a special selinuxcontext. They are used by the AnsibleModule methods which operate onfiles (changing attributes, moving, and copying). The list of names is setvia a comma separated string of filesystem names from ansible.cfg:

# ansible.cfg
[selinux]
special_context_filesystems=nfs,vboxsf,fuse,ramfs

If a module cannot use the builtin AnsibleModule methods to manipulatefiles and needs to know about these special context filesystems, it shouldinstantiate an AnsibleModule and then examine the list inAnsibleModule._selinux_special_fs.

This replaces ansible.module_utils.basic.SELINUX_SPECIAL_FS fromModule Replacer framework. In module replacer it was a comma separated string offilesystem names. Under Ansiballz it’s an actual list.

New in version 2.1.

_ansible_syslog_facility

This parameter controls which syslog facility ansible module logs to. It maybe set by changing the syslogfacility value in ansible.cfg. Mostmodules should just use AnsibleModule.log() which will then make use ofthis. If a module has to use this on its own, it should instantiate an_AnsibleModule and then retrieve the name of the syslog facility fromAnsibleModule._syslog_facility. The code will look slightly differentthan it did under Module Replacer framework due to how hacky the old way was

# Old way
import syslog
syslog.openlog(NAME, 0, syslog.LOG_USER)
 
# New way
import syslog
facility_name = module._syslog_facility
facility = getattr(syslog, facility_name, syslog.LOG_USER)
syslog.openlog(NAME, 0, facility)

New in version 2.1.

_ansible_version

This parameter passes the version of ansible that runs the module. To accessit, a module should instantiate an AnsibleModule and then retrieve itfrom AnsibleModule.ansible_version. This replacesansible.module_utils.basic.ANSIBLE_VERSION fromModule Replacer framework.

New in version 2.1.

Special considerations

Pipelining

Ansible can transfer a module to a remote machine in one of two ways:

• it can write out the module to a temporary file on the remote host and thenuse a second connection to the remote host to execute it with theinterpreter that the module needs
• or it can use what’s known as pipelining to execute the module by piping itinto the remote interpreter’s stdin.

Pipelining only works with modules written in Python at this time becauseAnsible only knows that Python supports this mode of operation. Supportingpipelining means that whatever format the module payload takes before beingsent over the wire must be executable by Python via stdin.

Why pass args over stdin?

Passing arguments via stdin was chosen for the following reasons:

• When combined with ANSIBLE_PIPELINING, this keeps the module’s arguments fromtemporarily being saved onto disk on the remote machine. This makes itharder (but not impossible) for a malicious user on the remote machine tosteal any sensitive information that may be present in the arguments.
• Command line arguments would be insecure as most systems allow unprivilegedusers to read the full commandline of a process.
• Environment variables are usually more secure than the commandline but somesystems limit the total size of the environment. This could lead totruncation of the parameters if we hit that limit.

AnsibleModule

Argument spec

The argument_spec provided to AnsibleModule defines the supported arguments for a module, as well as their type, defaults and more.

Example argument_spec:

module = AnsibleModule(argument_spec=dict(
    top_level=dict(
        type='dict',
        options=dict(
            second_level=dict(
                default=True,
                type='bool',
            )
        )
    )
))

This section will discuss the behavioral attributes for arguments:

type

type allows you to define the type of the value accepted for the argument. The default value for type is str. Possible values are:

• str
• list
• dict
• bool
• int
• float
• path
• raw
• jsonarg
• json
• bytes
• bits

The raw type, performs no type validation or type casing, and maintains the type of the passed value.

elements

elements works in combination with type when type='list'. elements can then be defined as elements='int' or any other type, indicating that each element of the specified list should be of that type.

default

The default option allows sets a default value for the argument for the scenario when the argument is not provided to the module. When not specified, the default value is None.

fallback

fallback accepts a tuple where the first argument is a callable (function) that will be used to perform the lookup, based on the second argument. The second argument is a list of values to be accepted by the callable.

The most common callable used is env_fallback which will allow an argument to optionally use an environment variable when the argument is not supplied.

Example:

username=dict(fallback=(env_fallback, ['ANSIBLE_NET_USERNAME']))

choices

choices accepts a list of choices that the argument will accept. The types of choices should match the type.

required

required accepts a boolean, either True or False that indicates that the argument is required. This should not be used in combination with default.

no_log

no_log indicates that the value of the argument should not be logged or displayed.

aliases

aliases accepts a list of alternative argument names for the argument, such as the case where the argument is name but the module accepts aliases=['pkg'] to allow pkg to be interchangably with name

options

options implements the ability to create a sub-argument_spec, where the sub options of the top level argument are also validated using the attributes discussed in this section. The example at the top of this section demonstrates use of options. type or elements should be dict is this case.

apply_defaults

apply_defaults works alongside options and allows the default of the sub-options to be applied even when the top-level argument is not supplied.

In the example of the argument_spec at the top of this section, it would allow module.params['top_level']['second_level'] to be defined, even if the user does not provide top_level when calling the module.

removed_in_version

removed_in_version indicates which version of Ansible a deprecated argument will be removed in.