Failed Tasks

Sometimes tasks can fail. Let’s see how to deal with failed tasks in nornir.

Let’s start as usual with the needed boilerplate:

  1. [1]:
  1. import logging
  3. from nornir import InitNornir
  4. from nornir.core.task import Task, Result
  5. from nornir_utils.plugins.functions import print_result
  7. # instantiate the nr object
  8. nr = InitNornir(config_file="config.yaml")
  9. # let's filter it down to simplify the output
  10. cmh = nr.filter(site="cmh", type="host")
  12. def count(task: Task, number: int) -> Result:
  13.     return Result(
  14.         host=task.host,
  15.         result=f"{[n for n in range(0, number)]}"
  16.     )
  18. def say(task: Task, text: str) -> Result:
  19.     if task.host.name == "host2.cmh":
  20.         raise Exception("I can't say anything right now")
  21.     return Result(
  22.         host=task.host,
  23.         result=f"{task.host.name} says {text}"
  24.     )

Now, as an example we are going to use a similar task group like the one we used in the previous tutorial:

  1. [2]:
  1. def greet_and_count(task: Task, number: int):
  2.     task.run(
  3.         name="Greeting is the polite thing to do",
  4.         severity_level=logging.DEBUG,
  5.         task=say,
  6.         text="hi!",
  7.     )
  9.     task.run(
  10.         name="Counting beans",
  11.         task=count,
  12.         number=number,
  13.     )
  14.     task.run(
  15.         name="We should say bye too",
  16.         severity_level=logging.DEBUG,
  17.         task=say,
  18.         text="bye!",
  19.     )
  21.     # let's inform if we counted even or odd times
  22.     even_or_odds = "even" if number % 2 == 1 else "odd"
  23.     return Result(
  24.         host=task.host,
  25.         result=f"{task.host} counted {even_or_odds} times!",
  26.     )

Remember there is a hardcoded error on host2.cmh, let’s see what happens when we run the task:

  1. [3]:
  1. result = cmh.run(
  2.     task=greet_and_count,
  3.     number=5,
  4. )

Let’s inspect the object:

  1. [4]:
  1. result.failed
  1. [4]:
  1. True
  1. [5]:
  1. result.failed_hosts
  1. [5]:
  1. {'host2.cmh': MultiResult: [Result: "greet_and_count", Result: "Greeting is the polite thing to do"]}
  1. [6]:
  1. result['host2.cmh'].exception
  1. [6]:
  1. nornir.core.exceptions.NornirSubTaskError()
  1. [7]:
  1. result['host2.cmh'][1].exception
  1. [7]:
  1. Exception("I can't say anything right now")

As you can see, the result object is aware something went wrong and you can inspect the errors if you so desire.

You can also using the print_result function on it:

  1. [8]:
  1. print_result(result)
  1. greet_and_count*****************************************************************
  2. * host1.cmh ** changed : False *************************************************
  3. vvvv greet_and_count ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv INFO
  4. host1.cmh counted even times!
  5. ---- Counting beans ** changed : False ----------------------------------------- INFO
  6. [0, 1, 2, 3, 4]
  7. ^^^^ END greet_and_count ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  8. * host2.cmh ** changed : False *************************************************
  9. vvvv greet_and_count ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv ERROR
  10. Subtask: Greeting is the polite thing to do (failed)
  12. ---- Greeting is the polite thing to do ** changed : False --------------------- ERROR
  13. Traceback (most recent call last):
  14.   File "/opt/conda/lib/python3.8/site-packages/nornir/core/task.py", line 99, in start
  15.     r = self.task(self, **self.params)
  16.   File "<ipython-input-1-3ab8433d31a3>", line 20, in say
  17.     raise Exception("I can't say anything right now")
  18. Exception: I can't say anything right now
  20. ^^^^ END greet_and_count ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

There is also a method that will raise an exception if the task had an error:

  1. [9]:
  1. from nornir.core.exceptions import NornirExecutionError
  2. try:
  3.     result.raise_on_error()
  4. except NornirExecutionError:
  5.     print("ERROR!!!")
  1. ERROR!!!

Skipped hosts

Nornir will keep track of hosts that failed and won’t run future tasks on them:

  1. [10]:
  1. from nornir.core.task import Result
  3. def hi(task: Task) -> Result:
  4.     return Result(host=task.host, result=f"{task.host.name}: Hi, I am still here!")
  6. result = cmh.run(task=hi)
  1. [11]:
  1. print_result(result)
  1. hi******************************************************************************
  2. * host1.cmh ** changed : False *************************************************
  3. vvvv hi ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv INFO
  4. host1.cmh: Hi, I am still here!
  5. ^^^^ END hi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You can force the execution of tasks on failed hosts by passing the argument on_failed=True:

  1. [12]:
  1. result = cmh.run(task=hi, on_failed=True)
  2. print_result(result)
  1. hi******************************************************************************
  2. * host1.cmh ** changed : False *************************************************
  3. vvvv hi ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv INFO
  4. host1.cmh: Hi, I am still here!
  5. ^^^^ END hi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  6. * host2.cmh ** changed : False *************************************************
  7. vvvv hi ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv INFO
  8. host2.cmh: Hi, I am still here!
  9. ^^^^ END hi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You can also exclude the hosts that are “good” if you want to with the on_good flag:

  1. [13]:
  1. result = cmh.run(task=hi, on_failed=True, on_good=False)
  2. print_result(result)
  1. hi******************************************************************************
  2. * host2.cmh ** changed : False *************************************************
  3. vvvv hi ** changed : False vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv INFO
  4. host2.cmh: Hi, I am still here!
  5. ^^^^ END hi ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To achieve this nornir keeps a set of failed hosts in it’s shared data object:

  1. [14]:
  1. nr.data.failed_hosts
  1. [14]:
  1. {'host2.cmh'}

If you want to mark some hosts as succeeded and make them back eligible for future tasks you can do it individually per host with the function recover_host or reset the list completely with reset_failed_hosts:

  1. [15]:
  1. nr.data.reset_failed_hosts()
  2. nr.data.failed_hosts
  1. [15]:
  1. set()

Raise on error automatically

Alternatively, you can configure nornir to raise the exception automatically in case of error with the raise_on_error configuration option:

  1. [16]:
  1. nr = InitNornir(config_file="config.yaml", core={"raise_on_error": True})
  2. cmh = nr.filter(site="cmh", type="host")
  3. try:
  4.     result = cmh.run(
  5.         task=greet_and_count,
  6.         number=5,
  7.     )
  8. except NornirExecutionError:
  9.     print("ERROR!!!")
  1. ERROR!!!

Workflows

The default workflow should work for most use cases as hosts with errors are skipped and the print_result should give enough information to understand what’s going on. For more complex workflows this framework should give you enough room to easily implement them regardless of the complexity.