TCollector writing

TCollector is part of openTSDB and collects client computer’s logs to send to the database.

You can write the data collected by TCollector to TDengine by simply changing the configuration of TCollector to point to the domain name (or IP address) and corresponding port of the server running taosAdapter. It can take full advantage of TDengine’s efficient storage query performance and clustering capability for time-series data.

Prerequisites

To write data to the TDengine via TCollector requires the following preparations.

  • The TDengine cluster has been deployed and is working properly
  • taosAdapter is installed and running properly. Please refer to the taosAdapter manual for details.
  • TCollector has been installed. Please refer to official documentation for TCollector installation

Configuration steps

Configuring taosAdapter

To configure taosAdapter to receive TCollector data.

  • Enable the configuration item in the taosAdapter configuration file (default location /etc/taos/taosadapter.toml)
  1. ...
  2. [opentsdb_telnet]
  3. enable = true
  4. maxTCPConnections = 250
  5. tcpKeepAlive = false
  6. dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
  7. ports = [6046, 6047, 6048, 6049]
  8. user = "root"
  9. password = "taosdata"
  10. ...

The taosAdapter writes to the database with the default name tcollector. You can also modify the taosAdapter configuration file dbs entry to specify a different name. Fill in the actual user and password for TDengine. After changing the configuration file, you need to restart the taosAdapter.

  • You can also enable taosAdapter to receive tcollector data by using the taosAdapter command-line parameters or setting environment variables.

Configuring TCollector

To use TCollector, you need to download its source code. Its configuration items are in its source code. Note: TCollector differs significantly from version to version, so here is an example of the latest code for the current master branch (git commit: 37ae920).

Modify the contents of the collectors/etc/config.py and tcollector.py files. Change the address of the OpenTSDB host to the domain name or IP address of the server where taosAdapter is deployed, and change the port to the port on which taosAdapter supports TCollector (default is 6049).

Example of git diff output of source code changes.

  1. index e7e7a1c..ec3e23c 100644
  2. --- a/collectors/etc/config.py
  3. +++ b/collectors/etc/config.py
  4. @@ -59,13 +59,13 @@ def get_defaults():
  5.          'http_password': False,
  6.          'reconnectinterval': 0,
  7.          'http_username': False,
  8. -        'port': 4242,
  9. +        'port': 6049,
  10.          'pidfile': '/var/run/tcollector.pid',
  11.          'http': False,
  12.          'http_api_path': "api/put",
  13.          'tags': [],
  14.          'remove_inactive_collectors': False,
  15. -        'host': '',
  16. +        'host': '127.0.0.1',
  17.          'logfile': '/var/log/tcollector.log',
  18.          'cdir': default_cdir,
  19.          'ssl': False,
  20. diff --git a/tcollector.py b/tcollector.py
  21. index 21f9b23..4c71ba2 100755
  22. --- a/tcollector.py
  23. +++ b/tcollector.py
  24. @@ -64,7 +64,7 @@ ALIVE = True
  25.  # exceptions, something is not right and tcollector will shutdown.
  26.  # Hopefully some kind of supervising daemon will then restart it.
  27.  MAX_UNCAUGHT_EXCEPTIONS = 100
  28. -DEFAULT_PORT = 4242
  29. +DEFAULT_PORT = 6049
  30.  MAX_REASONABLE_TIMESTAMP = 2209212000  # Good until Tue  3 Jan 14:00:00 GMT 2040
  31.  # How long to wait for datapoints before assuming
  32.  # a collector is dead and restarting it
  33. @@ -943,13 +943,13 @@ def parse_cmdline(argv):
  34.              'http_password': False,
  35.              'reconnectinterval': 0,
  36.              'http_username': False,
  37. -            'port': 4242,
  38. +            'port': 6049,
  39.              'pidfile': '/var/run/tcollector.pid',
  40.              'http': False,
  41.              'http_api_path': "api/put",
  42.              'tags': [],
  43.              'remove_inactive_collectors': False,
  44. -            'host': '',
  45. +            'host': '127.0.0.1',
  46.              'logfile': '/var/log/tcollector.log',
  47.              'cdir': default_cdir,
  48.              'ssl': False,

Verification method

Restart taosAdapter:

  1. sudo systemctl restart taosadapter

Run sudo ./tcollector.py:

Wait for a few seconds and then use the TDengine CLI to query whether the corresponding database has been created and data are written in TDengine.

  1. taos> show databases;
  2.               name              |
  3. =================================
  4.  information_schema             |
  5.  performance_schema             |
  6.  tcollector                     |
  7. Query OK, 3 rows in database (0.001647s)
  9. taos> use tcollector;
  10. Database changed.
  12. taos> show stables;
  13.               name              |
  14. =================================
  15.  proc.meminfo.hugepages_rsvd    |
  16.  proc.meminfo.directmap1g       |
  17.  proc.meminfo.vmallocchunk      |
  18.  proc.meminfo.hugepagesize      |
  19.  tcollector.reader.lines_dro... |
  20.  proc.meminfo.sunreclaim        |
  21.  proc.stat.ctxt                 |
  22.  proc.meminfo.swaptotal         |
  23.  proc.uptime.total              |
  24.  tcollector.collector.lines_... |
  25.  proc.meminfo.vmallocused       |
  26.  proc.meminfo.memavailable      |
  27.  sys.numa.foreign_allocs        |
  28.  proc.meminfo.committed_as      |
  29.  proc.vmstat.pswpin             |
  30.  proc.meminfo.cmafree           |
  31.  proc.meminfo.mapped            |
  32.  proc.vmstat.pgmajfault         |
  33. ...
TCollector - 图1note
  • TDengine will automatically create unique IDs for sub-table names by the rule.