Ansible module architecture
- Types of modules
- How modules are executed

Ansible module architecture

If you’re working on Ansible’s Core code, writing an Ansible module, or developing an action plugin, this deep dive helps you understand how Ansible’s program flow executes. If you’re just using Ansible Modules in playbooks, you can skip this section.

Types of modules

Ansible supports several different types of modules in its code base. Some ofthese are for backwards compatibility and others are to enable flexibility.

Action plugins

Action plugins look like modules to anyone writing a playbook. Usage documentation for most action plugins lives inside a module of the same name. Some action plugins do all the work, with the module providing only documentation. Some action plugins execute modules. The normal action plugin executes modules that don’t have special action plugins. Action plugins always execute on the controller.

Some action plugins do all their work on the controller. Forexample, the debug action plugin (which prints text forthe user to see) and the assert action plugin (whichtests whether values in a playbook satisfy certain criteria) execute entirely on the controller.

Most action plugins set up some values on the controller, then invoke anactual module on the managed node that does something with these values. For example, the template action plugin takes values fromthe user to construct a file in a temporary location on the controller usingvariables from the playbook environment. It then transfers the temporary fileto a temporary file on the remote system. After that, it invokes thecopy module which operates on the remote system to move the fileinto its final location, sets file permissions, and so on.

New-style modules

All of the modules that ship with Ansible fall into this category. While you can write modules in any language, all official modules (shipped with Ansible) use either Python or PowerShell.

New-style modules have the arguments to the module embedded inside of them insome manner. Old-style modules must copy a separate file over to themanaged node, which is less efficient as it requires two over-the-wireconnections instead of only one.

Python

New-style Python modules use the Ansiballz framework framework for constructingmodules. These modules use imports from ansible.module_utils to pull inboilerplate module code, such as argument parsing, formatting of returnvalues as JSON, and various file operations.

Note

In Ansible, up to version 2.0.x, the official Python modules used theModule Replacer framework framework. For module authors, Ansiballz framework islargely a superset of Module Replacer framework functionality, so you usuallydo not need to know about one versus the other.

PowerShell

New-style PowerShell modules use the Module Replacer framework framework forconstructing modules. These modules get a library of PowerShell code embeddedin them before being sent to the managed node.

JSONARGS modules

These modules are scripts that include the string<<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>> in their body.This string is replaced with the JSON-formatted argument string. These modules typically set a variable to that value like this:

json_arguments = """<<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>>"""

Which is expanded as:

json_arguments = """{"param1": "test's quotes", "param2": "\"To be or not to be\" - Hamlet"}"""

Note

Ansible outputs a JSON string with bare quotes. Double quotes areused to quote string values, double quotes inside of string values arebackslash escaped, and single quotes may appear unescaped inside ofa string value. To use JSONARGS, your scripting language must have a wayto handle this type of string. The example uses Python’s triple quotedstrings to do this. Other scripting languages may have a similar quotecharacter that won’t be confused by any quotes in the JSON or it mayallow you to define your own start-of-quote and end-of-quote characters.If the language doesn’t give you any of these then you’ll need to writea non-native JSON module orOld-style module instead.

These modules typically parse the contents of json_arguments using a JSONlibrary and then use them as native variables throughout the code.

Non-native want JSON modules

If a module has the string WANT_JSON in it anywhere, Ansible treatsit as a non-native module that accepts a filename as its only command lineparameter. The filename is for a temporary file containing a JSONstring containing the module’s parameters. The module needs to open the file,read and parse the parameters, operate on the data, and print its return dataas a JSON encoded dictionary to stdout before exiting.

These types of modules are self-contained entities. As of Ansible 2.1, Ansibleonly modifies them to change a shebang line if present.

Binary modules

From Ansible 2.2 onwards, modules may also be small binary programs. Ansibledoesn’t perform any magic to make these portable to different systems so theymay be specific to the system on which they were compiled or require otherbinary runtime dependencies. Despite these drawbacks, you may haveto compile a custom module against a specific binarylibrary if that’s the only way to get access to certain resources.

Binary modules take their arguments and return data to Ansible in the sameway as want JSON modules.

Old-style modules

Old-style modules are similar towant JSON modules, except that the file thatthey take contains key=value pairs for their parameters instead ofJSON. Ansible decides that a module is old-style when it doesn’t haveany of the markers that would show that it is one of the other types.

How modules are executed

When a user uses ansible or ansible-playbook, theyspecify a task to execute. The task is usually the name of a module alongwith several parameters to be passed to the module. Ansible takes thesevalues and processes them in various ways before they are finally executed onthe remote machine.

Executor/task_executor

The TaskExecutor receives the module name and parameters that were parsed fromthe playbook (or from the command line in the case of/usr/bin/ansible). It uses the name to decide whether it’s lookingat a module or an Action Plugin. If it’sa module, it loads the Normal Action Pluginand passes the name, variables, and other information about the task and playto that Action Plugin for further processing.

The normal action plugin

The normal action plugin executes the module on the remote host. It isthe primary coordinator of much of the work to actually execute the module onthe managed machine.

It loads the appropriate connection plugin for the task, which then transfersor executes as needed to create a connection to that host.
It adds any internal Ansible properties to the module’s parameters (forinstance, the ones that pass along no_log to the module).
It works with other plugins (connection, shell, become, other action plugins)to create any temporary files on the remote machine andcleans up afterwards.
It pushes the module and module parameters to theremote host, although the module_commoncode described in the next section decides which formatthose will take.
It handles any special cases regarding modules (for instance, asyncexecution, or complications around Windows modules that must have the same names as Python modules, so that internal calling of modules from other Action Plugins work.)

Much of this functionality comes from the BaseAction class,which lives in plugins/action/init.py. It uses theConnection and Shell objects to do its work.

Note

When tasks are run with the async: parameter, Ansibleuses the async Action Plugin instead of the normal Action Pluginto invoke it. That program flow is currently not documented. Read thesource for information on how that works.

Executor/module_common.py

Code in executor/module_common.py assembles the moduleto be shipped to the managed node. The module is first read in, then examinedto determine its type:

PowerShell and JSON-args modules are passed through Module Replacer.
New-style Python modules are assembled by Ansiballz framework.
Non-native-want-JSON, Binary modules, and Old-Style modules aren’t touched by either of these and pass through unchanged.

After the assembling step, one finalmodification is made to all modules that have a shebang line. Ansible checkswhether the interpreter in the shebang line has a specific path configured viaan ansible_$X_interpreter inventory variable. If it does, Ansiblesubstitutes that path for the interpreter path given in the module. Afterthis, Ansible returns the complete module data and the module type to theNormal Action which continues execution ofthe module.

Assembler frameworks

Ansible supports two assembler frameworks: Ansiballz and the older Module Replacer.

Module Replacer framework

The Module Replacer framework is the original framework implementing new-stylemodules, and is still used for PowerShell modules. It is essentially a preprocessor (like the C Preprocessor for thosefamiliar with that programming language). It does straight substitutions ofspecific substring patterns in the module file. There are two types ofsubstitutions:

Replacements that only happen in the module file. These are publicreplacement strings that modules can utilize to get helpful boilerplate oraccess to arguments.
- from ansible.module_utils.MOD_LIB_NAME import * is replaced with thecontents of the ansible/module_utils/MOD_LIB_NAME.py These shouldonly be used with new-style Python modules.
- #<<INCLUDE_ANSIBLE_MODULE_COMMON>> is equivalent tofrom ansible.module_utils.basic import * and should also only applyto new-style Python modules.
- # POWERSHELL_COMMON substitutes the contents ofansible/module_utils/powershell.ps1. It should only be used withnew-style Powershell modules.
Replacements that are used by ansible.module_utils code. These are internal replacement patterns. They may be used internally, in the above public replacements, but shouldn’t be used directly by modules.
- "<<ANSIBLEVERSION>>" is substituted with the Ansible version. Innew-style Python modules under theAnsiballz framework framework the proper way is to instead instantiate an_AnsibleModule and then access the version from:attr:AnsibleModule.ansible_version.
- "<<INCLUDEANSIBLE_MODULE_COMPLEX_ARGS>>" is substituted witha string which is the Python repr of the JSON encoded moduleparameters. Using repr on the JSON string makes it safe to embed ina Python file. In new-style Python modules under the Ansiballz frameworkthis is better accessed by instantiating an _AnsibleModule andthen using AnsibleModule.params.
- <<SELINUXSPECIAL_FILESYSTEMS>> substitutes a string which isa comma separated list of file systems which have a file system dependentsecurity context in SELinux. In new-style Python modules, if you reallyneed this you should instantiate an _AnsibleModule and then useAnsibleModule._selinux_special_fs. The variable has also changedfrom a comma separated string of file system names to an actual pythonlist of filesystem names.
- <<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>> substitutes the moduleparameters as a JSON string. Care must be taken to properly quote thestring as JSON data may contain quotes. This pattern is not substitutedin new-style Python modules as they can get the module parameters anotherway.
- The string syslog.LOGUSER is replaced wherever it occurs with thesyslog_facility which was named in ansible.cfg or anyansible_syslog_facility inventory variable that applies to this host. Innew-style Python modules this has changed slightly. If you really need toaccess it, you should instantiate an _AnsibleModule and then useAnsibleModule._syslog_facility to access it. It is no longer theactual syslog facility and is now the name of the syslog facility. Seethe documentation on internal argumentsfor details.

Ansiballz framework

The Ansiballz framework was adopted in Ansible 2.1 and is used for all new-style Python modules. Unlike the Module Replacer, Ansiballz uses real Python imports of things inansible/moduleutils instead of merely preprocessing the module. Itdoes this by constructing a zipfile – which includes the module file, filesin ansible/moduleutils that are imported by the module, and someboilerplate to pass in the module’s parameters. The zipfile is then Base64encoded and wrapped in a small Python script which decodes the Base64 encodingand places the zipfile into a temp directory on the managed node. It thenextracts just the Ansible module script from the zip file and places that inthe temporary directory as well. Then it sets the PYTHONPATH to find Pythonmodules inside of the zip file and imports the Ansible module as the special name, main.Importing it as __main causes Python to think that it is executing a script rather than simplyimporting a module. This lets Ansible run both the wrapper script and the module code in a single copy of Python on the remote machine.

Note

Ansible wraps the zipfile in the Python script for two reasons:

for compatibility with Python 2.6 which has a lessfunctional version of Python’s -m command line switch.
so that pipelining will function properly. Pipelining needs to pipe thePython module into the Python interpreter on the remote node. Pythonunderstands scripts on stdin but does not understand zip files.

Prior to Ansible 2.7, the module was executed via a second Python interpreter instead of beingexecuted inside of the same process. This change was made once Python-2.4 support was droppedto speed up module execution.

In Ansiballz, any imports of Python modules from theansible.module_utils package trigger inclusion of that Python fileinto the zipfile. Instances of #<<INCLUDE_ANSIBLE_MODULE_COMMON>> inthe module are turned into from ansible.module_utils.basic import *and ansible/module-utils/basic.py is then included in the zipfile.Files that are included from module_utils are themselves scanned forimports of other Python modules from module_utils to be included inthe zipfile as well.

Warning

At present, the Ansiballz Framework cannot determine whether an importshould be included if it is a relative import. Always use an absoluteimport that has ansible.module_utils in it to allow Ansiballz todetermine that the file should be included.

Passing args

Arguments are passed differently by the two frameworks:

In Module Replacer framework, module arguments are turned into a JSON-ified string and substituted into the combined module file.
In Ansiballz framework, the JSON-ified string is part of the script which wraps the zipfile. Just before the wrapper script imports the Ansible module as main, it monkey-patches the private, _ANSIBLE_ARGS variable in basic.py with the variable values. When a ansible.module_utils.basic.AnsibleModule is instantiated, it parses this string and places the args into AnsibleModule.params where it can be accessed by the module’s other code.

Warning

If you are writing modules, remember that the way we pass arguments is an internal implementation detail: it has changed in the past and will change again as soon as changes to the common module_utilscode allow Ansible modules to forgo using ansible.module_utils.basic.AnsibleModule. Do not rely on the internal global _ANSIBLE_ARGS variable.

Very dynamic custom modules which need to parse arguments before theyinstantiate an AnsibleModule may use _load_params to retrieve those parameters.Although _load_params may change in breaking ways if necessary to supportchanges in the code, it is likely to be more stable than either the way we pass parameters or the internal global variable.

Note

Prior to Ansible 2.7, the Ansible module was invoked in a second Python interpreter and thearguments were then passed to the script over the script’s stdin.

Internal arguments

Both Module Replacer framework and Ansiballz framework send additional arguments tothe module beyond those which the user specified in the playbook. Theseadditional arguments are internal parameters that help implement globalAnsible features. Modules often do not need to know about these explicitly asthe features are implemented in ansible.module_utils.basic but certainfeatures need support from the module so it’s good to know about them.

The internal arguments listed here are global. If you need to add a local internal argument to a custom module, create an action plugin for that specific module - see _original_basename in the copy action plugin for an example.

_ansible_no_log

Boolean. Set to True whenever a parameter in a task or play specifies no_log. Any module that calls AnsibleModule.log() handles this automatically. If a module implements its own logging thenit needs to check this value. To access in a module, instantiate anAnsibleModule and then check the value of AnsibleModule.no_log.

Note

no_log specified in a module’s argument_spec is handled by a different mechanism.

_ansible_debug

Boolean. Turns more verbose logging on or off and turns on logging ofexternal commands that the module executes. If a module usesAnsibleModule.debug() rather than AnsibleModule.log() thenthe messages are only logged if _ansible_debug is set to True.To set, add debug: True to ansible.cfg or set the environmentvariable ANSIBLE_DEBUG. To access in a module, instantiate anAnsibleModule and access AnsibleModule._debug.

_ansible_diff

Boolean. If a module supports it, tells the module to show a unified diff ofchanges to be made to templated files. To set, pass the —diff command lineoption. To access in a module, instantiate an AnsibleModule and accessAnsibleModule._diff.

_ansible_verbosity

Unused. This value could be used for finer grained control over logging.

_ansible_selinux_special_fs

List. Names of filesystems which should have a special SELinuxcontext. They are used by the AnsibleModule methods which operate onfiles (changing attributes, moving, and copying). To set, add a comma separated string of filesystem names in ansible.cfg:

# ansible.cfg
[selinux]
special_context_filesystems=nfs,vboxsf,fuse,ramfs,vfat

Most modules can use the built-in AnsibleModule methods to manipulatefiles. To access in a module that needs to know about these special context filesystems, instantiate an AnsibleModule and examine the list inAnsibleModule._selinux_special_fs.

This replaces ansible.module_utils.basic.SELINUX_SPECIAL_FS fromModule Replacer framework. In module replacer it was a comma separated string offilesystem names. Under Ansiballz it’s an actual list.

New in version 2.1.

_ansible_syslog_facility

This parameter controls which syslog facility Ansible module logs to. To set, change the syslogfacility value in ansible.cfg. Mostmodules should just use AnsibleModule.log() which will then make use ofthis. If a module has to use this on its own, it should instantiate an_AnsibleModule and then retrieve the name of the syslog facility fromAnsibleModule._syslog_facility. The Ansiballz code is less hacky than the old Module Replacer framework code:

# Old module_replacer way
import syslog
syslog.openlog(NAME, 0, syslog.LOG_USER)
 
# New Ansiballz way
import syslog
facility_name = module._syslog_facility
facility = getattr(syslog, facility_name, syslog.LOG_USER)
syslog.openlog(NAME, 0, facility)

New in version 2.1.

_ansible_version

This parameter passes the version of Ansible that runs the module. To accessit, a module should instantiate an AnsibleModule and then retrieve itfrom AnsibleModule.ansible_version. This replacesansible.module_utils.basic.ANSIBLE_VERSION fromModule Replacer framework.

New in version 2.1.

Module return values & Unsafe strings

At the end of a module’s execution, it formats the data that it wants to return as a JSON string and prints the string to its stdout. The normal action plugin receives the JSON string, parses it into a Python dictionary, and returns it to the executor.

If Ansible templated every string return value, it would be vulnerable to an attack from users with access to managed nodes. If an unscrupulous user disguised malicious code as Ansible return value strings, and if those strings were then templated on the controller, Ansible could execute arbitrary code. To prevent this scenario, Ansible marks all strings inside returned data as Unsafe, emitting any Jinja2 templates in the strings verbatim, not expanded by Jinja2.

Strings returned by invoking a module through ActionPlugin._execute_module() are automatically marked as Unsafe by the normal action plugin. If another action plugin retrieves information from a module through some other means, it must mark its return data as Unsafe on its own.

In case a poorly-coded action plugin fails to mark its results as “Unsafe,” Ansible audits the results again when they are returned to the executor,marking all strings as Unsafe. The normal action plugin protects itself and any other code that it calls with the result data as a parameter. The check inside the executor protects the output of all other action plugins, ensuring that subsequent tasks run by Ansible will not template anything from those results either.

Special considerations

Pipelining

Ansible can transfer a module to a remote machine in one of two ways:

it can write out the module to a temporary file on the remote host and thenuse a second connection to the remote host to execute it with theinterpreter that the module needs
or it can use what’s known as pipelining to execute the module by piping itinto the remote interpreter’s stdin.

Pipelining only works with modules written in Python at this time becauseAnsible only knows that Python supports this mode of operation. Supportingpipelining means that whatever format the module payload takes before beingsent over the wire must be executable by Python via stdin.

Why pass args over stdin?

Passing arguments via stdin was chosen for the following reasons:

When combined with ANSIBLE_PIPELINING, this keeps the module’s arguments fromtemporarily being saved onto disk on the remote machine. This makes itharder (but not impossible) for a malicious user on the remote machine tosteal any sensitive information that may be present in the arguments.
Command line arguments would be insecure as most systems allow unprivilegedusers to read the full commandline of a process.
Environment variables are usually more secure than the commandline but somesystems limit the total size of the environment. This could lead totruncation of the parameters if we hit that limit.

AnsibleModule

Argument spec

The argument_spec provided to AnsibleModule defines the supported arguments for a module, as well as their type, defaults and more.

Example argument_spec:

module = AnsibleModule(argument_spec=dict(
    top_level=dict(
        type='dict',
        options=dict(
            second_level=dict(
                default=True,
                type='bool',
            )
        )
    )
))

This section will discuss the behavioral attributes for arguments:

type

type allows you to define the type of the value accepted for the argument. The default value for type is str. Possible values are:

str
list
dict
bool
int
float
path
raw
jsonarg
json
bytes
bits

The raw type, performs no type validation or type casing, and maintains the type of the passed value.

elements

elements works in combination with type when type='list'. elements can then be defined as elements='int' or any other type, indicating that each element of the specified list should be of that type.

default

The default option allows sets a default value for the argument for the scenario when the argument is not provided to the module. When not specified, the default value is None.

fallback

fallback accepts a tuple where the first argument is a callable (function) that will be used to perform the lookup, based on the second argument. The second argument is a list of values to be accepted by the callable.

The most common callable used is env_fallback which will allow an argument to optionally use an environment variable when the argument is not supplied.

Example:

username=dict(fallback=(env_fallback, ['ANSIBLE_NET_USERNAME']))

choices

choices accepts a list of choices that the argument will accept. The types of choices should match the type.

required

required accepts a boolean, either True or False that indicates that the argument is required. This should not be used in combination with default.

no_log

no_log indicates that the value of the argument should not be logged or displayed.

aliases

aliases accepts a list of alternative argument names for the argument, such as the case where the argument is name but the module accepts aliases=['pkg'] to allow pkg to be interchangeably with name

options

options implements the ability to create a sub-argument_spec, where the sub options of the top level argument are also validated using the attributes discussed in this section. The example at the top of this section demonstrates use of options. type or elements should be dict is this case.

apply_defaults

apply_defaults works alongside options and allows the default of the sub-options to be applied even when the top-level argument is not supplied.

In the example of the argument_spec at the top of this section, it would allow module.params['top_level']['second_level'] to be defined, even if the user does not provide top_level when calling the module.

removed_in_version

removed_in_version indicates which version of Ansible a deprecated argument will be removed in.