3 Elasticsearch 配置

3 Elasticsearch setup

Elasticsearch的支持是实验性的！本节中考虑的配置步骤适用于以下Elasticsearch版本： 5.0.x - > 6.1.x 。 如果使用早期或更高版本的Elasticsearch，某些功能可能无法按预期工作。

Elasticsearch support is experimental!
Setup procedure considered in this section is applicable to the following Elasticsearch versions: 5.0.x –> 6.1.x. In case an earlier or later version of Elasticsearch is used, some functionality may not work as intended.

Zabbix最近开始支持通过使用Elasticsearch而不是数据库来存储历史数据。 现在，用户可以在兼容数据库和Elasticsearch之间选择历史数据的存储位置。

Zabbix has recently started to support storage of historical data by means of Elasticsearch instead of a database. Users are now given the possibility to choose the storage place for historical data between a compatible database and Elasticsearch.

配置

Configuration

正确的设置Zabbix server配置文件和前端配置文件中的参数，以保证所有元素之间的正确通信。 To ensure proper communication between all elements involved make sure server configuration file and frontend configuration file parameters are properly configured.

Zabbix server 和前端
Zabbix server and frontend

已经更新参数的Zabbix server配置文件示例如下： Zabbix server configuration file draft with parameters to be updated:

  1. ### Option: HistoryStorageURL
  2. #    History storage HTTP[S] URL.
  3. #
  4. # Mandatory: no
  5. # Default:
  6. # HistoryStorageURL= 
  7. ### Option: HistoryStorageTypes
  8. #    Comma separated list of value types to be sent to the history storage.
  9. #
  10. # Mandatory: no
  11. # Default:
  12. # HistoryStorageTypes=uint,dbl,str,log,text

用于设置Zabbix server配置文件的示例参数值如下： Example parameter values to fill the Zabbix server configuration file with:

  1. HistoryStorageURL=http://test.elasticsearch.lan:9200
  2. HistoryStorageTypes=str,log,text

此配置文件使Zabbix Server将相应数据库中的数值类型和文本类型的历史数据存储到Elasticsearch中。 This configuration forces Zabbix Server to store history values of numeric types in the corresponding database and textual history data in Elasticsearch.

Elasticsearch支持以下几种监控项类型： Elasticsearch supports the following item types:

  1. uint,dbl,str,log,text

支持的监控项类型说明如下： Supported item type explanation:

Item value typeDatabase tableElasticsearch type
Numeric (unsigned)history_uintuint
Numeric (float)historydbl
Characterhistory_strstr
Loghistory_loglog
Texthistory_texttext

已经更新参数的Zabbix前端配置文件(conf/zabbix.conf.php) 示例如下： Zabbix frontend configuration file (conf/zabbix.conf.php) draft with parameters to be updated:

  1. // Elasticsearch url (can be string if same url is used for all types).
  2. $HISTORY['url']   = [
  3.     'uint' => 'http://localhost:9200',
  4.     'text' => 'http://localhost:9200'
  5. ];
  6. // Value types stored in Elasticsearch.
  7. $HISTORY['types'] = ['uint', 'text'];

用于设置Zabbix前端配置文件的示例参数值如下： Example parameter values to fill the Zabbix frontend configuration file with:

  1. $HISTORY['url']   = 'http://test.elasticsearch.lan:9200';
  2. $HISTORY['types'] = ['str', 'text', 'log'];

此配置文件将文本、字符、日志类型的历史数据存储到Elasticsearch中。 This configuration forces to store Text, Character and Log history values in Elasticsearch.

还需要在conf / zabbix.conf.php文件中配置$HISTORY为全局参数，以确保一切正常工作（请参阅conf / zabbix.conf.php.example以了解如何配置）：

It is also required to make $HISTORY global in conf/zabbix.conf.php to ensure everything is working properly (see conf/zabbix.conf.php.example for how to do it):

  1. // Zabbix GUI configuration file.
  2. global $DB, $HISTORY;
Elasticsearch配置和创建映射
Installing Elasticsearch and creating mapping

正常配置包括安装Elasticsearch和创建映射两个步骤。 Final two steps of making things work are installing Elasticsearch itself and creating mapping process.

安装Elasticsearch请参考Elasticsearch安装指南

To install Elasticsearch please refer to Elasticsearch installation guide.

映射是Elasticsearch中的数据结构（类似于数据库中的表）。 此处提供了所有历史数据类型的映射：database / elasticsearch / elasticsearch.map。

Mapping is a data structure in Elasticsearch (similar to a table in a database). Mapping for all history data types is available here: database/elasticsearch/elasticsearch.map.

必须创建映射。 如果未按照要求创建映射，则某些功能将无法正常使用。

Creating mapping is mandatory. Some functionality will be broken if mapping is not created according to the instruction.

创建text类型的映射可以发送如下请求到Elasticsearch： To create mapping for text type send the following request to Elasticsearch:

  1. curl -X PUT \
  2.  http://your-elasticsearch.here:9200/text \
  3.  -H 'content-type:application/json' \
  4.  -d '{
  5.    "settings" : {
  6.       "index" : {
  7.          "number_of_replicas" : 1,
  8.          "number_of_shards" : 5
  9.       }
  10.    },
  11.    "mappings" : {
  12.       "values" : {
  13.          "properties" : {
  14.             "itemid" : {
  15.                "type" : "long"
  16.             },
  17.             "clock" : {
  18.                "format" : "epoch_second",
  19.                "type" : "date"
  20.             },
  21.             "value" : {
  22.                "fields" : {
  23.                   "analyzed" : {
  24.                      "index" : true,
  25.                      "type" : "text",
  26.                      "analyzer" : "standard"
  27.                   }
  28.                },
  29.                "index" : false,
  30.                "type" : "text"
  31.             }
  32.          }
  33.       }
  34.    }
  35. }'

对于创建“字符”和“日志”类型的历史数据映射，需要执行类似的请求，请求内容需要进行相应的修改。 Similar request is required to be executed for Character and Log history values mapping creation with corresponding type correction.

要使用Elasticsearch，请参阅 Requirement page以获取更多信息。

To work with Elasticsearch please refer to Requirement page for additional information.

Housekeeper 不会删除任何Elasticsearch中的数据

Housekeeper is not deleting any data from Elasticsearch.

历史数据存储于多个基于时间的索引

Storing history data in multiple date-based indices

本节介绍使用pipeline和ingest节点所需的其他配置步骤。 This section describes additional steps required to work with pipelines and ingest nodes.

首先必须为索引创建一个模板。创建uint模板的请求示例如下： To begin with, you must create templates for indices. The following example shows a request for creating uint template:

  1. curl -X PUT \
  2.  http://your-elasticsearch.here:9200/_template/uint_template \
  3.  -H 'content-type:application/json' \
  4.  -d '{
  5.    "template": "uint*",
  6.    "index_patterns": ["uint*"],
  7.    "settings" : {
  8.       "index" : {
  9.          "number_of_replicas" : 1,
  10.          "number_of_shards" : 5
  11.       }
  12.    },
  13.    "mappings" : {
  14.       "values" : {
  15.          "properties" : {
  16.             "itemid" : {
  17.                "type" : "long"
  18.             },
  19.             "clock" : {
  20.                "format" : "epoch_second",
  21.                "type" : "date"
  22.             },
  23.             "value" : {
  24.                "type" : "long"
  25.             }
  26.          }
  27.       }
  28.    }
  29. }'

要创建其他模板，用户应更改请求URL（最后一部分是模板名称），更改“template”和“index_patterns”字段以匹配索引名称并设置可从“database / elasticsearch / elasticsearch”中获取的有效映射 。例如，下面的命令能为一个文本索引创建一个模板：

To create other templates, user should change the URL (last part is the name of template), change “template” and “index_patterns” fields to match index name and to set valid mapping that can be taken from database/elasticsearch/elasticsearch.map. For example, the following command can be used to create a template for text index:

  1. curl -X PUT \
  2.  http://your-elasticsearch.here:9200/_template/text_template \
  3.  -H 'content-type:application/json' \
  4.  -d '{
  5.    "template": "text*",
  6.    "index_patterns": ["text*"],
  7.    "settings" : {
  8.       "index" : {
  9.          "number_of_replicas" : 1,
  10.          "number_of_shards" : 5
  11.       }
  12.    },
  13.    "mappings" : {
  14.       "values" : {
  15.          "properties" : {
  16.             "itemid" : {
  17.                "type" : "long"
  18.             },
  19.             "clock" : {
  20.                "format" : "epoch_second",
  21.                "type" : "date"
  22.             },
  23.             "value" : {
  24.                "fields" : {
  25.                   "analyzed" : {
  26.                      "index" : true,
  27.                      "type" : "text",
  28.                      "analyzer" : "standard"
  29.                   }
  30.                },
  31.                "index" : false,
  32.                "type" : "text"
  33.             }
  34.          }
  35.       }
  36.    }
  37. }'

这是允许Elasticsearch为自动创建的索引设置有效的映射所必需做的。 然后需要创建pipeline定义。 在将数据放入索引之前，pipeline能对数据进行多种预处理操作。 以下命令可用于为uint索引创建pipeline：

This is required to allow Elasticsearch to set valid mapping for indices created automatically. Then it is required to create the pipeline definition. Pipeline is some sort of preprocessing of data before putting data in indices. The following command can be used to create pipeline for uint index:

  1. curl -X PUT \
  2.  http://your-elasticsearch.here:9200/_ingest/pipeline/uint-pipeline \
  3.  -H 'content-type:application/json' \
  4.  -d '{
  5.   "description": "daily uint index naming",
  6.   "processors": [
  7.     {
  8.       "date_index_name": {
  9.         "field": "clock",
  10.         "date_formats": ["UNIX"],
  11.         "index_name_prefix": "uint-",
  12.         "date_rounding": "d"
  13.       }
  14.     }
  15.   ]
  16. }'

用户可以修改参数（“date_rounding”）来设置特定的索引循环周期。 要创建其他pipeline，用户应更改请求URL（最后一部分是pipeline名称）并更改“index_name_prefix”字段以匹配索引名称。

User can change the rounding parameter (“date_rounding”) to set a specific index rotation period. To create other pipelines, user should change the URL (last part is the name of pipeline) and change “index_name_prefix” field to match index name.

可以参考 Elasticsearch文档.

See also Elasticsearch documentation.

另外，可以通过Zabbix server配置文件中新加的参数来配置将历史数据存储于基于时间的多个索引。 Additionally, storing history data in multiple date-based indices should also be enabled in the new parameter in Zabbix server configuration:

  1. ### Option: HistoryStorageDateIndex
  2. #    Enable preprocessing of history values in history storage to store values in different indices based on date.
  3. #    0 - disable
  4. #    1 - enable
  5. #
  6. # Mandatory: no
  7. # Default:
  8. # HistoryStorageDateIndex=0

故障诊断

Troubleshooting

以下步骤可帮助您解决Elasticsearch的配置问题： The following steps may help you troubleshoot problems with Elasticsearch setup:

  1. 检查映射是否正确(通过URL的发送GET请求获取索引信息，例如：http://localhost:9200/uint)。

  2. 检查shards状态是否正常 (不正常时重启Elasticsearch可能解决问题)。

  3. 检查Elasticsearch配置文件，配置文件应允许从Zabbix前端主机和Zabbix server主机进行访问。

  4. 检查Elasticsearch日志。

  1. Check if the mapping is correct (GET request to required index URL like http://localhost:9200/uint).

  2. Check if shards are not in failed state (restart of Elasticsearch should help).

  3. Check the configuration of Elasticsearch. Configuration should allow access from the Zabbix frontend host and the Zabbix server host.

  4. Check Elasticsearch logs.

如果您仍然遇到配置问题，请创建一个错误报告，其中包含映射，错误日志，配置，版本等信息。 If you are still experiencing problems with your installation then please create a bug report with all the information from this list (mapping, error logs, configuration, version, etc.)