Event Notification

An event is essentially a significant or meaningful change in the state of both virtual and physical resources associated with a cloud environment. Events are used by monitoring systems, usage and billing systems, or any other event-driven workflow systems to discern a pattern and make the right business decision. In CloudStack an event could be a state change of virtual or physical resources, an action performed by an user (action events), or policy based events (alerts).

Event Logs

There are two types of events logged in the CloudStack Event Log. Standard events log the success or failure of an event and can be used to identify jobs or processes that have failed. There are also long running job events. Events for asynchronous jobs log when a job is scheduled, when it starts, and when it completes. Other long running synchronous jobs log when a job starts, and when it completes. Long running synchronous and asynchronous event logs can be used to gain more information on the status of a pending job or can be used to identify a job that is hanging or has not started. The following sections provide more information on these events..

Notification

Event notification framework provides a means for the Management Server components to publish and subscribe to CloudStack events. Event notification is achieved by implementing the concept of event bus abstraction in the Management Server.

A new event for state change, resource state change, is introduced as part of Event notification framework. Every resource, such as user VM, volume, NIC, network, public IP, snapshot, and template, is associated with a state machine and generates events as part of the state change. That implies that a change in the state of a resource results in a state change event, and the event is published in the corresponding state machine on the event bus. All the CloudStack events (alerts, action events, usage events) and the additional category of resource state change events, are published on to the events bus.

Implementations

An event bus is introduced in the Management Server that allows the CloudStack components and extension plug-ins to subscribe to the events by using the Advanced Message Queuing Protocol (AMQP) client. In CloudStack, a default implementation of event bus is provided as a plug-in that uses the RabbitMQ AMQP client. The AMQP client pushes the published events to a compatible AMQP server. Therefore all the CloudStack events are published to an exchange in the AMQP server.

Additionally, both an in-memory implementation and an Apache Kafka implementation are also available.

Use Cases

The following are some of the use cases:

  • Usage or Billing Engines: A third-party cloud usage solution can implement a plug-in that can connects to CloudStack to subscribe to CloudStack events and generate usage data. The usage data is consumed by their usage software.
  • AMQP plug-in can place all the events on the a message queue, then a AMQP message broker can provide topic-based notification to the subscribers.
  • Publish and Subscribe notification service can be implemented as a pluggable service in CloudStack that can provide rich set of APIs for event notification, such as topics-based subscription and notification. Additionally, the pluggable service can deal with multi-tenancy, authentication, and authorization issues.

AMQP Configuration

As a CloudStack administrator, perform the following one-time configuration to enable event notification framework. At run time no changes can control the behaviour.

  1. Create the folder /etc/cloudstack/management/META-INF/cloudstack/core

  2. Inside that folder, open spring-event-bus-context.xml.

  3. Define a bean named eventNotificationBus as follows:

    • name : Specify a name for the bean.

    • server : The name or the IP address of the RabbitMQ AMQP server.

    • port : The port on which RabbitMQ server is running.

    • username : The username associated with the account to access the RabbitMQ server.

    • password : The password associated with the username of the account to access the RabbitMQ server.

    • exchange : The exchange name on the RabbitMQ server where CloudStack events are published.

      A sample bean is given below:

      1. <beans xmlns="http://www.springframework.org/schema/beans"
      2. xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      3. xmlns:context="http://www.springframework.org/schema/context"
      4. xmlns:aop="http://www.springframework.org/schema/aop"
      5. xsi:schemaLocation="http://www.springframework.org/schema/beans
      6. http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
      7. http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
      8. http://www.springframework.org/schema/context
      9. http://www.springframework.org/schema/context/spring-context-3.0.xsd">
      10.    <bean id="eventNotificationBus" class="org.apache.cloudstack.mom.rabbitmq.RabbitMQEventBus">
      11.       <property name="name" value="eventNotificationBus"/>
      12.       <property name="server" value="127.0.0.1"/>
      13.       <property name="port" value="5672"/>
      14.       <property name="username" value="guest"/>
      15.       <property name="password" value="guest"/>
      16.       <property name="exchange" value="cloudstack-events"/>
      17.    </bean>
      18. </beans>

      The eventNotificationBus bean represents the org.apache.cloudstack.mom.rabbitmq.RabbitMQEventBus class.

      If you want to use encrypted values for the username and password, you have to include a bean to pass those as variables from a credentials file.

      A sample is given below

      1. <beans xmlns="http://www.springframework.org/schema/beans"
      2.        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      3.        xmlns:context="http://www.springframework.org/schema/context"
      4.        xmlns:aop="http://www.springframework.org/schema/aop"
      5.        xsi:schemaLocation="http://www.springframework.org/schema/beans
      6.         http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
      7.         http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
      8.         http://www.springframework.org/schema/context
      9.         http://www.springframework.org/schema/context/spring-context-3.0.xsd"
      10. >
      12.    <bean id="eventNotificationBus" class="org.apache.cloudstack.mom.rabbitmq.RabbitMQEventBus">
      13.       <property name="name" value="eventNotificationBus"/>
      14.       <property name="server" value="127.0.0.1"/>
      15.       <property name="port" value="5672"/>
      16.       <property name="username" value="${username}"/>
      17.       <property name="password" value="${password}"/>
      18.       <property name="exchange" value="cloudstack-events"/>
      19.    </bean>
      21.    <bean id="environmentVariablesConfiguration" class="org.jasypt.encryption.pbe.config.EnvironmentStringPBEConfig">
      22.       <property name="algorithm" value="PBEWithMD5AndDES" />
      23.       <property name="passwordEnvName" value="APP_ENCRYPTION_PASSWORD" />
      24.    </bean>
      26.    <bean id="configurationEncryptor" class="org.jasypt.encryption.pbe.StandardPBEStringEncryptor">
      27.       <property name="config" ref="environmentVariablesConfiguration" />
      28.    </bean>
      30.    <bean id="propertyConfigurer" class="org.jasypt.spring3.properties.EncryptablePropertyPlaceholderConfigurer">
      31.       <constructor-arg ref="configurationEncryptor" />
      32.       <property name="location" value="classpath:/cred.properties" />
      33.    </bean>
      34. </beans>

      Create a new file in the same folder called cred.properties and the specify the values for username and password as jascrypt encrypted strings

      Sample, with guest as values for both fields:

      1. username=nh2XrM7jWHMG4VQK18iiBQ==
      2. password=nh2XrM7jWHMG4VQK18iiBQ==

  4. Restart the Management Server.

Kafka Configuration

As a CloudStack administrator, perform the following one-time configuration to enable event notification framework. At run time no changes can control the behaviour.

  1. Create an appropriate configuration file in /etc/cloudstack/management/kafka.producer.properties which contains valid kafka configuration properties as documented in http://kafka.apache.org/documentation.html#newproducerconfigs The properties may contain an additional topic property which if not provided will default to cloudstack. While key.serializer and value.serializer are usually required for a producer to correctly start, they may be omitted and will default to org.apache.kafka.common.serialization.StringSerializer.

  2. Create the folder /etc/cloudstack/management/META-INF/cloudstack/core

  3. Inside that folder, open spring-event-bus-context.xml.

  4. Define a bean named eventNotificationBus with a single name attribute, A sample bean is given below:

    1. <beans xmlns="http://www.springframework.org/schema/beans"
    2.        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    3.        xmlns:context="http://www.springframework.org/schema/context"
    4.        xmlns:aop="http://www.springframework.org/schema/aop"
    5.        xsi:schemaLocation="http://www.springframework.org/schema/beans
    6.                            http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
    7.                            http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
    8.                            http://www.springframework.org/schema/context
    9.                            http://www.springframework.org/schema/context/spring-context-3.0.xsd">
    10.    <bean id="eventNotificationBus" class="org.apache.cloudstack.mom.kafka.KafkaEventBus">
    11.      <property name="name" value="eventNotificationBus"/>
    12.    </bean>
    13.  </beans>

  5. Restart the Management Server.

Standard Events

The events log records three types of standard events.

  • INFO. This event is generated when an operation has been successfully performed.
  • WARN. This event is generated in the following circumstances.
    • When a network is disconnected while monitoring a template download.
    • When a template download is abandoned.
    • When an issue on the storage server causes the volumes to fail over to the mirror storage server.
  • ERROR. This event is generated when an operation has not been successfully performed

Long Running Job Events

The events log records three types of standard events.

  • INFO. This event is generated when an operation has been successfully performed.
  • WARN. This event is generated in the following circumstances.
    • When a network is disconnected while monitoring a template download.
    • When a template download is abandoned.
    • When an issue on the storage server causes the volumes to fail over to the mirror storage server.
  • ERROR. This event is generated when an operation has not been successfully performed

Event Log Queries

Database logs can be queried from the user interface. The list of events captured by the system includes:

  • Virtual machine creation, deletion, and on-going management operations
  • Virtual router creation, deletion, and on-going management operations
  • Template creation and deletion
  • Network/load balancer rules creation and deletion
  • Storage volume creation and deletion
  • User login and logout

Deleting and Archiving Events and Alerts

CloudStack provides you the ability to delete or archive the existing alerts and events that you no longer want to implement. You can regularly delete or archive any alerts or events that you cannot, or do not want to resolve from the database.

You can delete or archive individual alerts or events either directly by using the Quickview or by using the Details page. If you want to delete multiple alerts or events at the same time, you can use the respective context menu. You can delete alerts or events by category for a time period. For example, you can select categories such as USER.LOGOUT, VM.DESTROY, VM.AG.UPDATE, CONFIGURATION.VALUE.EDI, and so on. You can also view the number of events or alerts archived or deleted.

In order to support the delete or archive alerts, the following global parameters have been added:

  • alert.purge.delay: The alerts older than specified number of days are purged. Set the value to 0 to never purge alerts automatically.
  • alert.purge.interval: The interval in seconds to wait before running the alert purge thread. The default is 86400 seconds (one day).

Note

Archived alerts or events cannot be viewed in the UI or by using the API. They are maintained in the database for auditing or compliance purposes.

Permissions

Consider the following:

  • The root admin can delete or archive one or multiple alerts or events.
  • The domain admin or end user can delete or archive one or multiple events.

Procedure

  1. Log in as administrator to the CloudStack UI.
  2. In the left navigation, click Events.
  3. Perform either of the following:
    • To archive events, click Archive Events, and specify event type and date.
    • To archive events, click Delete Events, and specify event type and date.
  4. Click OK.