3 Low-level discovery

Overview

Low-level discovery provides a way to automatically create items, triggers, and graphs for different entities on a computer. For instance, Zabbix can automatically start monitoring file systems or network interfaces on your machine, without the need to create items for each file system or network interface manually. Additionally, it is possible to configure Zabbix to remove unneeded entities automatically based on actual results of periodically performed discovery.

A user can define their own types of discovery, provided they follow a particular JSON protocol.

The general architecture of the discovery process is as follows.

First, a user creates a discovery rule in “Configuration” → “Templates” → “Discovery” column. A discovery rule consists of (1) an item that discovers the necessary entities (for instance, file systems or network interfaces) and (2) prototypes of items, triggers, and graphs that should be created based on the value of that item.

An item that discovers the necessary entities is like a regular item seen elsewhere: the server asks a Zabbix agent (or whatever the type of the item is set to) for a value of that item, the agent responds with a textual value. The difference is that the value the agent responds with should contain a list of discovered entities in a JSON format. While the details of this format are only important for implementers of custom discovery checks, it is necessary to know that the returned value contains a list of macro → value pairs. For instance, item “net.if.discovery” might return two pairs: “{#IFNAME}” → “lo” and “{#IFNAME}” → “eth0”.

These macros are used in names, keys and other prototype fields where they are then substituted with the received values for creating real items, triggers, graphs or even hosts for each discovered entity. See the full list of options for using LLD macros.

When the server receives a value for a discovery item, it looks at the macro → value pairs and for each pair generates real items, triggers, and graphs, based on their prototypes. In the example with “net.if.discovery” above, the server would generate one set of items, triggers, and graphs for the loopback interface “lo”, and another set for interface “eth0”.

Note that since Zabbix 4.2, the format of the JSON returned by low-level discovery rules has been changed. It is no longer expected that the JSON will contain the “data” object. Low-level discovery will now accept a normal JSON containing an array, in order to support new features such as the item value preprocessing and custom paths to low-level discovery macro values in a JSON document.

Built-in discovery keys have been updated to return an array of LLD rows at the root of JSON document. Zabbix will automatically extract a macro and value if an array field uses the {#MACRO} syntax as a key. Any new native discovery checks will use the new syntax without the “data” elements. When processing a low-level discovery value first the root is located (array at $. or $.data).

While the “data” element has been removed from all native items related to discovery, for backward compatibility Zabbix will still accept the JSON notation with a “data” element, though its use is discouraged. If the JSON contains an object with only one “data” array element, then it will automatically extract the content of the element using JSONPath $.data. Low-level discovery now accepts optional user-defined LLD macros with a custom path specified in JSONPath syntax.

As a result of the changes above, newer agents no longer will be able to work with an older Zabbix server.

See also: Discovered entities

Configuring low-level discovery

We will illustrate low-level discovery based on an example of file system discovery.

To configure the discovery, do the following:

• Go to: Configuration → Templates or Hosts

• Click on Discovery in the row of an appropriate template/host

• Click on Create discovery rule in the upper right corner of the screen

• Fill in the discovery rule form with the required details

Discovery rule

The discovery rule form contains five tabs, representing, from left to right, the data flow during discovery:

• Discovery rule - specifies, most importantly, the built-in item or custom script to retrieve discovery data

• Preprocessing - applies some preprocessing to the discovered data

• LLD macros - allows to extract some macro values to use in discovered items, triggers, etc

• Filters - allows to filter the discovered values

• Overrides - allows to modify items, triggers, graphs or host prototypes when applying to specific discovered objects

The Discovery rule tab contains the item key to use for discovery (as well as some general discovery rule attributes):

All mandatory input fields are marked with a red asterisk.

Discovery rule history is not preserved.

Preprocessing

The Preprocessing tab allows to define transformation rules to apply to the result of discovery. One or several transformations are possible in this step. Transformations are executed in the order in which they are defined. All preprocessing is done by Zabbix server.

See also:

• Preprocessing details

• Preprocessing testing

Note that if the discovery rule has been applied to the host via template then the content of this tab is read-only.

Custom macros

The LLD macros tab allows to specify custom low-level discovery macros.

Custom macros are useful in cases when the returned JSON does not have the required macros already defined. So, for example:

• The native vfs.fs.discovery key for filesystem discovery returns a JSON with some pre-defined LLD macros such as {#FSNAME}, {#FSTYPE}. These macros can be used in item, trigger prototypes (see subsequent sections of the page) directly; defining custom macros is not needed;

• The [vfs.fs.get]($af1d5b75df0abf50.md "manual:discovery:low_level_discovery:mounted_filesystems") agent item also returns a JSON with filesystem data, but without any pre-defined LLD macros. In this case you may define the macros yourself, and map them to the values in the JSON using JSONPath:

The extracted values can be used in discovered items, triggers, etc. Note that values will be extracted from the result of discovery and any preprocessing steps so far.

Filter

The Filters tab contains discovery rule filter definitions allowing to filter discovery values:

for f in ext2 nfs reiserfs smbfs; do echo $f | grep -E ‘^ext|^reiserfs’ || echo “SKIP: $f”; done

A mistake or typo in the regular expression used in LLD rule may cause deleting thousands of configuration elements, historical values and events for many hosts. For example, an incorrect “File systems for discovery” regular expression may cause deleting thousands of items, triggers, historical values and events.

Zabbix database in MySQL must be created as case-sensitive if file system names that differ only by case are to be discovered correctly.

Override

The Override tab allows to set rules to modify the list of item, trigger, graph and host prototypes or their attributes for discovered objects that meet given criteria.

Overrides (if any) are displayed in a reorderable drag-and-drop list and executed in the order in which they are defined. To configure details of a new override, click on in the Overrides block. To edit an existing override, click on the override name. A popup window will open allowing to edit the override rule details.

All mandatory parameters are marked with red asterisks.

Configuring an operation

To configure details of a new operation, click on in the Operations block. To edit an existing operation, click on next to the operation. A popup window where you can edit the operation details will open.

Parameter				Description
Object				Four types of objects are available: Item prototype Trigger prototype Graph prototype Host prtotype
Condition				Allows to filter entities to which the opperation should be applied.
	Operator			Supported operators: equals - apply to this prototype does not equal - apply to all prototypes, except this contains - apply, if prototype name contains this string does not contain - apply, if prototype name does not contain this string matches - apply, if prototype name matches regular expression does not match - apply, if prototype name does not match regular expression
	Pattern			A regular expression or a string to search for.
	Object: Item prototype
	Create enabled			When the checkbox is marked, the buttons will appear, allowing to override original item prototype settings: Yes - the item will be added in an enabled state. No - the item will be added to a discovered entity, but in a disabled state.
	Discover			When the checkbox is marked, the buttons will appear, allowing to override original item prototype settings: Yes - the item will be added. No - the item will not be added.
	Update interval			When the checkbox is marked, two options will appear, allowing to set different interval for the item: Delay - Item update interval. User macros and time suffixes (e.g. 30s, 1m, 2h, 1d) are supported. Should be set to 0 if Custom interval is used. Custom interval - click to specify flexible/sheduling intervals. For detailed information see Custom intervals.
	History storage period			When the checkbox is marked, the buttons will appear, allowing to set different history storage period for the item: Do not keep history - if selected, the history will not be stored. Storage period - if selected, an input field for specifying storage period will appear to the right. User macros and LLD macros are supported.
	Trend storage period			When the checkbox is marked, the buttons will appear, allowing to set different trend storage period for the item: Do not keep trends - if selected, the trends will not be stored. Storage period - if selected, an input field for specifying storage period will appear to the right. User macros and LLD macros are supported.
	Object: Trigger prototype
	Create enabled			When the checkbox is marked, the buttons will appear, allowing to override original trigger prototype settings: Yes - the trigger will be added in an enabled state. No - the trigger will be added to a discovered entity, but in a disabled state.
	Discover			When the checkbox is marked, the buttons will appear, allowing to override original trigger prototype settings: Yes - the trigger will be added. No - the trigger will not be added.
	Severity			When the checkbox is marked, trigger severity buttons will appear, allowing to modify trigger severity.
	Tags			When the checkbox is marked, a new block will appear, allowing to specify tag-value pairs. All trigger prototype tags will be replaced by tags from this override.
	Object: Graph prototype
	Discover			When the checkbox is marked, the buttons will appear, allowing to override original graph prototype settings: Yes - the graph will be added. No - the graph will not be added.
	Object: Host prototype
	Create enabled			When the checkbox is marked, the buttons will appear, allowing to override original host prototype settings: Yes - the host will be created in an enabled state. No - the host will be created in a disabled state.
	Discover			When the checkbox is marked, the buttons will appear, allowing to override original host prototype settings: Yes - the host will be discovered. No - the host will not be discovered.
	Link templates			When the checkbox is marked, an input field for specifying templates will appear. Start typing the template name or click on Select next to the field and select templates from the list in a popup window. All templates linked to a host prototype will be replaced by templates from this override.
	Host inventory			When the checkbox is marked, the buttons will appear, allowing to select different inventory mode for the host prototype: Disabled - do not populate host inventory Manual - provide details manually Automated - auto-fill host inventory data based on collected metrics.

Form buttons

Buttons at the bottom of the form allow to perform several operations.

	Add a discovery rule. This button is only available for new discovery rules.
	Update the properties of a discovery rule. This button is only available for existing discovery rules.
	Create another discovery rule based on the properties of the current discovery rule.
	Perform discovery based on the discovery rule immediately. The discovery rule must already exist. See more details. Note that when performing discovery immediately, configuration cache is not updated, thus the result will not reflect very recent changes to discovery rule configuration.
	Delete the discovery rule.
	Cancel the editing of discovery rule properties.

Item prototypes

Once a rule is created, go to the items for that rule and press “Create prototype” to create an item prototype. Note how macro {#FSNAME} is used where a file system name is required. When the discovery rule is processed, this macro will be substituted with the discovered file system.

Low-level discovery macros and user macros may be used in item prototype configuration and item value preprocessing parameters. Note that when used in update intervals, a single macro has to fill the whole field. Multiple macros in one field or macros mixed with text are not supported.

Context-specific escaping of low-level discovery macros is performed for safe use in regular expression and XPath preprocessing parameters.

Attributes that are specific for item prototypes:

We can create several item prototypes for each file system metric we are interested in:

Mass update option is available if you want to update properties of several item prototypes at once.

Trigger prototypes

We create trigger prototypes in a similar way as item prototypes:

Attributes that are specific for trigger prototypes:

When real triggers are created from the prototypes, there may be a need to be flexible as to what constant (‘20’ in our example) is used for comparison in the expression. See how user macros with context can be useful to accomplish such flexibility.

You can define dependencies between trigger prototypes as well (supported since Zabbix 3.0). To do that, go to the Dependencies tab. A trigger prototype may depend on another trigger prototype from the same low-level discovery (LLD) rule or on a regular trigger. A trigger prototype may not depend on a trigger prototype from a different LLD rule or on a trigger created from trigger prototype. Host trigger prototype cannot depend on a trigger from a template.

Graph prototypes

We can create graph prototypes, too:

Attributes that are specific for trigger prototypes:

Finally, we have created a discovery rule that looks as shown below. It has five item prototypes, two trigger prototypes, and one graph prototype.

Note: For configuring host prototypes, see the section about host prototype configuration in virtual machine monitoring.

Discovered entities

The screenshots below illustrate how discovered items, triggers, and graphs look like in the host’s configuration. Discovered entities are prefixed with an orange link to a discovery rule they come from.

Note that discovered entities will not be created in case there are already existing entities with the same uniqueness criteria, for example, an item with the same key or graph with the same name. An error message is displayed in this case in the frontend that the low-level discovery rule could not create certain entities. The discovery rule itself, however, will not turn unsupported because some entity could not be created and had to be skipped. The discovery rule will go on creating/updating other entities.

Items (similarly, triggers and graphs) created by a low-level discovery rule will be deleted automatically if a discovered entity (file system, interface, etc) stops being discovered (or does not pass the filter anymore). In this case the items, triggers and graphs will be deleted after the days defined in the Keep lost resources period field pass.

When discovered entities become ‘Not discovered anymore’, a lifetime indicator is displayed in the item list. Move your mouse pointer over it and a message will be displayed indicating how many days are left until the item is deleted.

If entities were marked for deletion, but were not deleted at the expected time (disabled discovery rule or item host), they will be deleted the next time the discovery rule is processed.

Entities containing other entities, which are marked for deletion, will not update if changed on the discovery rule level. For example, LLD-based triggers will not update if they contain items that are marked for deletion.

Other types of discovery

More detail and how-tos on other types of out-of-the-box discovery is available in the following sections:

• discovery of network interfaces;

• discovery of CPUs and CPU cores;

• discovery of SNMP OIDs;

• discovery of JMX objects;

• discovery using ODBC SQL queries;

• discovery of Windows services;

• discovery of host interfaces in Zabbix.

For more detail on the JSON format for discovery items and an example of how to implement your own file system discoverer as a Perl script, see creating custom LLD rules.

Data limits for return values

There is no limit for low-level discovery rule JSON data if it is received directly by Zabbix server, because return values are processed without being stored in a database. There’s also no limit for custom low-level discovery rules, however, if it is intended to acquire custom LLD data using a user parameter, then user parameter return value limit applies (512 KB).

If data has to go through Zabbix proxy it has to store this data in database so database limits apply.

Multiple LLD rules for same item

Since Zabbix agent version 3.2 it is possible to define several low-level discovery rules with the same discovery item.

To do that you need to define the Alias agent parameter, allowing to use altered discovery item keys in different discovery rules, for example vfs.fs.discovery[foo], vfs.fs.discovery[bar], etc.

Creating custom LLD rules

It is also possible to create a completely custom LLD rule, discovering any type of entities - for example, databases on a database server.

To do so, a custom item should be created that returns JSON, specifying found objects and optionally - some properties of them. The amount of macros per entity is not limited - while the built-in discovery rules return either one or two macros (for example, two for filesystem discovery), it is possible to return more.

The required JSON format is best illustrated with an example. Suppose we are running an old Zabbix 1.8 agent (one that does not support “vfs.fs.discovery”), but we still need to discover file systems. Here is a simple Perl script for Linux that discovers mounted file systems and outputs JSON, which includes both file system name and type. One way to use it would be as a UserParameter with key “vfs.fs.discovery_perl”:

#!/usr/bin/perl
$first = 1;
print "[\n";
for (`cat /proc/mounts`)
{
    ($fsname, $fstype) = m/\S+ (\S+) (\S+)/;
    print "\t,\n" if not $first;
    $first = 0;
    print "\t{\n";
    print "\t\t\"{#FSNAME}\":\"$fsname\",\n";
    print "\t\t\"{#FSTYPE}\":\"$fstype\"\n";
    print "\t}\n";
}
print "]\n";

Allowed symbols for LLD macro names are 0-9 , A-Z , _ , .

Lowercase letters are not supported in the names.

An example of its output (reformatted for clarity) is shown below. JSON for custom discovery checks has to follow the same format.

[
    { "{#FSNAME}":"/",                           "{#FSTYPE}":"rootfs"   },
    { "{#FSNAME}":"/sys",                        "{#FSTYPE}":"sysfs"    },
    { "{#FSNAME}":"/proc",                       "{#FSTYPE}":"proc"     },
    { "{#FSNAME}":"/dev",                        "{#FSTYPE}":"devtmpfs" },
    { "{#FSNAME}":"/dev/pts",                    "{#FSTYPE}":"devpts"   },
    { "{#FSNAME}":"/lib/init/rw",                "{#FSTYPE}":"tmpfs"    },
    { "{#FSNAME}":"/dev/shm",                    "{#FSTYPE}":"tmpfs"    },
    { "{#FSNAME}":"/home",                       "{#FSTYPE}":"ext3"     },
    { "{#FSNAME}":"/tmp",                        "{#FSTYPE}":"ext3"     },
    { "{#FSNAME}":"/usr",                        "{#FSTYPE}":"ext3"     },
    { "{#FSNAME}":"/var",                        "{#FSTYPE}":"ext3"     },
    { "{#FSNAME}":"/sys/fs/fuse/connections",    "{#FSTYPE}":"fusectl"  }
]

In previous example it is required that the keys match the LLD macro names used in prototypes, the alternative is to extract LLD macro values using JSONPath {#FSNAME} → $.fsname and {#FSTYPE} → $.fstype, thus making such script possible:

#!/usr/bin/perl
$first = 1;
print "[\n";
for (`cat /proc/mounts`)
{
    ($fsname, $fstype) = m/\S+ (\S+) (\S+)/;
    print "\t,\n" if not $first;
    $first = 0;
    print "\t{\n";
    print "\t\t\"fsname\":\"$fsname\",\n";
    print "\t\t\"fstype\":\"$fstype\"\n";
    print "\t}\n";
}
print "]\n";

An example of its output (reformatted for clarity) is shown below. JSON for custom discovery checks has to follow the same format.

[
    { "fsname":"/",                           "fstype":"rootfs"   },
    { "fsname":"/sys",                        "fstype":"sysfs"    },
    { "fsname":"/proc",                       "fstype":"proc"     },
    { "fsname":"/dev",                        "fstype":"devtmpfs" },
    { "fsname":"/dev/pts",                    "fstype":"devpts"   },
    { "fsname":"/lib/init/rw",                "fstype":"tmpfs"    },
    { "fsname":"/dev/shm",                    "fstype":"tmpfs"    },
    { "fsname":"/home",                       "fstype":"ext3"     },
    { "fsname":"/tmp",                        "fstype":"ext3"     },
    { "fsname":"/usr",                        "fstype":"ext3"     },
    { "fsname":"/var",                        "fstype":"ext3"     },
    { "fsname":"/sys/fs/fuse/connections",    "fstype":"fusectl"  }
]

Then, in the discovery rule’s “Filter” field, we could specify “{#FSTYPE}” as a macro and “rootfs|ext3” as a regular expression.

You don’t have to use macro names FSNAME/FSTYPE with custom LLD rules, you are free to use whatever names you like. In case JSONPath is used then LLD row will be an array element that can be an object, but it can be also another array or a value.

Note that, if using a user parameter, the return value is limited to 512 KB. For more details, see data limits for LLD return values.

Using LLD macros in user macro contexts

LLD macros may be used inside user macro context, for example, in trigger prototypes.