Optimize writes to InfluxDB

Use these tips to optimize performance and system overhead when writing data to InfluxDB.

• Batch writes
• Sort tags by key
• Use the coarsest time precision possible
• Use gzip compression
• Synchronize hosts with NTP
• Write multiple data points in one request

The following tools write to InfluxDB and employ most write optimizations by default:

• Telegraf
• InfluxDB client libraries
• InfluxDB scrapers

Batch writes

Write data in batches to minimize network overhead when writing data to InfluxDB.

The optimal batch size is 5000 lines of line protocol.

Sort tags by key

Before writing data points to InfluxDB, sort tags by key in lexicographic order. Verify sort results match results from the Go bytes.Compare function.

# Line protocol example with unsorted tags
measurement,tagC=therefore,tagE=am,tagA=i,tagD=i,tagB=think fieldKey=fieldValue 1562020262
# Optimized line protocol example with tags sorted by key
measurement,tagA=i,tagB=think,tagC=therefore,tagD=i,tagE=am fieldKey=fieldValue 1562020262

Use the coarsest time precision possible

By default, InfluxDB writes data in nanosecond precision. However if your data isn’t collected in nanoseconds, there is no need to write at that precision. For better performance, use the coarsest precision possible for timestamps.

Specify timestamp precision when writing to InfluxDB.

Use gzip compression

Use gzip compression to speed up writes to InfluxDB and reduce network bandwidth. Benchmarks have shown up to a 5x speed improvement when data is compressed.

Telegraf Client libraries InfluxDB API

Enable gzip compression in Telegraf

In the influxdb_v2 output plugin configuration in your telegraf.conf, set the content_encoding option to gzip:

[[outputs.influxdb_v2]]
  urls = ["http://localhost:8086"]
  # ...
  content_encoding = "gzip"

Enable gzip compression in InfluxDB client libraries

Each InfluxDB client library provides options for compressing write requests or enforces compression by default. The method for enabling compression is different for each library. For specific instructions, see the InfluxDB client libraries documentation.

Use gzip compression with the InfluxDB API

When using the InfluxDB API /api/v2/write endpoint to write data, compress the data with gzip and set the Content-Encoding header to gzip.

echo "airSensors,sensor_id=TLM0201 temperature=73.97038159354763,humidity=35.23103248356096,co=0.48445310567793615 1630525358 
  airSensors,sensor_id=TLM0202 temperature=75.30007505999716,humidity=35.651929918691714,co=0.5141876544505826 1630525358" | gzip > air-sensors.gzip
curl --request POST \
"http://localhost:8086/api/v2/write?org=YOUR_ORG&bucket=YOUR_BUCKET&precision=ns" \
  --header "Authorization: Token YOUR_API_TOKEN" \
  --header "Content-Encoding: gzip" \
  --header "Content-Type: text/plain; charset=utf-8" \
  --header "Accept: application/json" \
  --data-binary @air-sensors.gzip

Synchronize hosts with NTP

Use the Network Time Protocol (NTP) to synchronize time between hosts. If a timestamp isn’t included in line protocol, InfluxDB uses its host’s local time (in UTC) to assign timestamps to each point. If a host’s clocks isn’t synchronized with NTP, timestamps may be inaccurate.

Write multiple data points in one request

To write multiple lines in one request, each line of line protocol must be delimited by a new line (\n).

Rate limiting

Use the influx write --rate-limit flag to control the rate of writes. Use one of the following string formats to specify the rate limit:

• COUNT(B|kB|MB), or
• COUNT(B|kB|MB)/TIME(s|sec|m|min)

where COUNT is a decimal number and TIME is a positive whole number. Spaces in the value are ignored. For example: “5MB / 5min” can be also expressed as 17476.266666667Bs, 1MB/1min, 1MB/min, 1MBmin or 1MBm. If the rate limit format is invalid, influx write prints out the format and an exact regular expression. The --rate-limit flag can be also used with influx write dryrun.

By default, the free tier rate limit in InfluxDB Cloud is 1MB/min.

best practices write