Migrate from InfluxDB

This guide will help you understand the differences between the data models of GreptimeDB and InfluxDB, and guide you through the migration process.

Data model in difference

While you may already be familiar with InfluxDB key concepts, the data model of GreptimeDB is something new to explore. Let’s start with similarities and differences:

  • Both solutions are schemaless, eliminating the need to define a schema before writing data.
  • In InfluxDB, a point represents a single data record with a measurement, tag set, field set, and a timestamp. In GreptimeDB, it is represented as a row of data in the time-series table, where the table name aligns with the measurement, and the columns are divided into three types: Tag, Field, and Timestamp.
  • GreptimeDB uses TimestampMillisecond as the data type for timestamp data from the InfluxDB line protocol API.
  • GreptimeDB uses Float64 as the data type for numeric data from the InfluxDB line protocol API.

Consider the following sample data borrowed from InfluxDB docs as an example:

_time_measurementlocationscientist_field_value
2019-08-18T00:00:00Zcensusklamathandersonbees23
2019-08-18T00:00:00Zcensusportlandmullenants30
2019-08-18T00:06:00Zcensusklamathandersonbees28
2019-08-18T00:06:00Zcensusportlandmullenants32

The data mentioned above is formatted as follows in the InfluxDB line protocol:

  1. census,location=klamath,scientist=anderson bees=23 1566086400000000000
  2. census,location=portland,scientist=mullen ants=30 1566086400000000000
  3. census,location=klamath,scientist=anderson bees=28 1566086760000000000
  4. census,location=portland,scientist=mullen ants=32 1566086760000000000

In the GreptimeDB data model, the data is represented as follows in the census table:

  1. +---------------------+----------+-----------+------+------+
  2. | ts                  | location | scientist | bees | ants |
  3. +---------------------+----------+-----------+------+------+
  4. | 2019-08-18 00:00:00 | klamath  | anderson  |   23 | NULL |
  5. | 2019-08-18 00:06:00 | klamath  | anderson  |   28 | NULL |
  6. | 2019-08-18 00:00:00 | portland | mullen    | NULL |   30 |
  7. | 2019-08-18 00:06:00 | portland | mullen    | NULL |   32 |
  8. +---------------------+----------+-----------+------+------+

The schema of the census table is as follows:

  1. +-----------+----------------------+------+------+---------+---------------+
  2. | Column    | Type                 | Key  | Null | Default | Semantic Type |
  3. +-----------+----------------------+------+------+---------+---------------+
  4. | location  | String               | PRI  | YES  |         | TAG           |
  5. | scientist | String               | PRI  | YES  |         | TAG           |
  6. | bees      | Float64              |      | YES  |         | FIELD         |
  7. | ts        | TimestampMillisecond | PRI  | NO   |         | TIMESTAMP     |
  8. | ants      | Float64              |      | YES  |         | FIELD         |
  9. +-----------+----------------------+------+------+---------+---------------+

Database connection information

Before you begin writing or querying data, it’s crucial to comprehend the differences in database connection information between InfluxDB and GreptimeDB.

  • Token: The InfluxDB API token, used for authentication, aligns with the GreptimeDB authentication. When interacting with GreptimeDB using InfluxDB’s client libraries or HTTP API, you can use <greptimedb_user:greptimedb_password> as the token.
  • Organization: Unlike InfluxDB, GreptimeDB does not require an organization for connection.
  • Bucket: In InfluxDB, a bucket serves as a container for time series data, which is equivalent to the database name in GreptimeDB.

Write data

GreptimeDB is compatible with both v1 and v2 of InfluxDB’s line protocol format, facilitating a seamless migration from InfluxDB to GreptimeDB.

HTTP API

To write a measurement to GreptimeDB, you can use the following HTTP API request:

  • InfluxDB line protocol v2
  • InfluxDB line protocol v1
  1. curl -X POST 'http://<greptimedb-host>:4000/v1/influxdb/api/v2/write?db=<db-name>' \
  2.   -H 'authorization: token <greptime_user:greptimedb_password>' \
  3.   -d 'census,location=klamath,scientist=anderson bees=23 1566086400000000000'
  1. curl 'http://<greptimedb-host>:4000/v1/influxdb/write?db=<db-name>&u=<greptime_user>&p=<greptimedb_password>' \
  2.   -d 'census,location=klamath,scientist=anderson bees=23 1566086400000000000'

Telegraf

GreptimeDB’s support for the Influxdb line protocol ensures its compatibility with Telegraf. To configure Telegraf, simply add http://<greptimedb-host>:4000 URL into Telegraf configurations:

  • InfluxDB line protocol v2
  • InfluxDB line protocol v1
  1. [[outputs.influxdb_v2]]
  2.   urls = ["http://<greptimedb-host>:4000/v1/influxdb"]
  3.   token = "<greptime_user>:<greptimedb_password>"
  4.   bucket = "<db-name>"
  5.   ## Leave empty
  6.   organization = ""
  1. [[outputs.influxdb]]
  2.   urls = ["http://<greptimedb-host>:4000/v1/influxdb"]
  3.   database = "<db-name>"
  4.   username = "<greptime_user>"
  5.   password = "<greptimedb_password>"

Client libraries

Writing data to GreptimeDB is a straightforward process when using InfluxDB client libraries. Simply include the URL and authentication details in the client configuration.

For example:

  • Node.js
  • Python
  • Go
  • Java
  • PHP
  1. 'use strict'
  2. /** @module write
  3. **/
  5. import { InfluxDB, Point } from '@influxdata/influxdb-client'
  7. /** Environment variables **/
  8. const url = 'http://<greptimedb-host>:4000/v1/influxdb'
  9. const token = '<greptime_user>:<greptimedb_password>'
  10. const org = ''
  11. const bucket = '<db-name>'
  13. const influxDB = new InfluxDB({ url, token })
  14. const writeApi = influxDB.getWriteApi(org, bucket)
  15. writeApi.useDefaultTags({ region: 'west' })
  16. const point1 = new Point('temperature')
  17.   .tag('sensor_id', 'TLM01')
  18.   .floatField('value', 24.0)
  19. writeApi.writePoint(point1)
  1. import influxdb_client
  2. from influxdb_client.client.write_api import SYNCHRONOUS
  4. bucket = "<db-name>"
  5. org = ""
  6. token = "<greptime_user>:<greptimedb_password>"
  7. url="http://<greptimedb-host>:4000/v1/influxdb"
  9. client = influxdb_client.InfluxDBClient(
  10.     url=url,
  11.     token=token,
  12.     org=org
  13. )
  15. # Write script
  16. write_api = client.write_api(write_options=SYNCHRONOUS)
  18. p = influxdb_client.Point("my_measurement").tag("location", "Prague").field("temperature", 25.3)
  19. write_api.write(bucket=bucket, org=org, record=p)
  1. bucket := "<db-name>"
  2. org := ""
  3. token := "<greptime_user>:<greptimedb_password>"
  4. url := "http://<greptimedb-host>:4000/v1/influxdb"
  5. client := influxdb2.NewClient(url, token)
  6. writeAPI := client.WriteAPIBlocking(org, bucket)
  8. p := influxdb2.NewPoint("stat",
  9.     map[string]string{"unit": "temperature"},
  10.     map[string]interface{}{"avg": 24.5, "max": 45},
  11.     time.Now())
  12. writeAPI.WritePoint(context.Background(), p)
  13. client.Close()
  1. private static String url = "http://<greptimedb-host>:4000/v1/influxdb";
  2. private static String org = "";
  3. private static String bucket = "<db-name>";
  4. private static char[] token = "<greptime_user>:<greptimedb_password>".toCharArray();
  6. public static void main(final String[] args) {
  8.     InfluxDBClient influxDBClient = InfluxDBClientFactory.create(url, token, org, bucket);
  9.     WriteApiBlocking writeApi = influxDBClient.getWriteApiBlocking();
  10.     Point point = Point.measurement("temperature")
  11.             .addTag("location", "west")
  12.             .addField("value", 55D)
  13.             .time(Instant.now().toEpochMilli(), WritePrecision.MS);
  15.     writeApi.writePoint(point);
  16.     influxDBClient.close();
  17. }
  1. $client = new Client([
  2.     "url" => "http://<greptimedb-host>:4000/v1/influxdb",
  3.     "token" => "<greptime_user>:<greptimedb_password>",
  4.     "bucket" => "<db-name>",
  5.     "org" => "",
  6.     "precision" => InfluxDB2\Model\WritePrecision::S
  7. ]);
  9. $writeApi = $client->createWriteApi();
  11. $dateTimeNow = new DateTime('NOW');
  12. $point = Point::measurement("weather")
  13.         ->addTag("location", "Denver")
  14.         ->addField("temperature", rand(0, 20))
  15.         ->time($dateTimeNow->getTimestamp());
  16. $writeApi->write($point);

In addition to the languages previously mentioned, GreptimeDB also accommodates client libraries for other languages supported by InfluxDB. You can code in your language of choice by referencing the connection information and code snippets provided earlier.

Query data

GreptimeDB does not support Flux and InfluxQL, opting instead for SQL and PromQL.

SQL is a universal language designed for managing and manipulating relational databases. With flexible capabilities for data retrieval, manipulation, and analytics, it is also reduce the learning curve for users who are already familiar with SQL.

PromQL (Prometheus Query Language) allows users to select and aggregate time series data in real time, The result of an expression can either be shown as a graph, viewed as tabular data in Prometheus’s expression browser, or consumed by external systems via the HTTP API.

Suppose you are querying the maximum cpu usage from the monitor table, recorded over the past 24 hours. In influxQL, the query might look something like this:

  1. SELECT 
  2.    MAX("cpu") 
  3. FROM 
  4.    "monitor" 
  5. WHERE 
  6.    time > now() - 24h 
  7. GROUP BY 
  8.    time(1h)

This InfluxQL query computes the maximum value of the cpu field from the monitor table, considering only the data where the time is within the last 24 hours. The results are then grouped into one-hour intervals.

In Flux, the query might look something like this:

  1. from(bucket: "public")
  2.   |> range(start: -24h)
  3.   |> filter(fn: (r) => r._measurement == "monitor")
  4.   |> aggregateWindow(every: 1h, fn: max)

The similar query in GreptimeDB SQL would be:

  1. SELECT
  2.     ts,
  3.     host,
  4.     AVG(cpu) RANGE '1h' as mean_cpu
  5. FROM
  6.     monitor
  7. WHERE
  8.     ts > NOW() - INTERVAL '24 hours'
  9. ALIGN '1h' TO NOW
  10. ORDER BY ts DESC;

In this SQL query, the RANGE clause determines the time window for the AVG(cpu) aggregation function, while the ALIGN clause sets the alignment time for the time series data. For more information on time window grouping, please refer to the Aggregate data by time window document.

The similar query in PromQL would be something like:

  1. avg_over_time(monitor[1h])

To query time series data from the last 24 hours, you need to execute this PromQL, using the start and end parameters of the HTTP API to define the time range. For more information on PromQL, please refer to the PromQL document.

Visualize data

It is recommended using Grafana to visualize data in GreptimeDB. Please refer to the Grafana documentation for details on configuring GreptimeDB.

Migrate data

For a seamless migration of data from InfluxDB to GreptimeDB, you can follow these steps:

Double write to GreptimeDB and InfluxDB

  1. Write data to both GreptimeDB and InfluxDB to avoid data loss during migration.
  2. Export all historical data from InfluxDB and import the data into GreptimeDB.
  3. Stop writing data to InfluxDB and remove the InfluxDB server.

Write data to both GreptimeDB and InfluxDB simultaneously

Writing data to both GreptimeDB and InfluxDB simultaneously is a practical strategy to avoid data loss during migration. By utilizing InfluxDB’s client libraries, you can set up two client instances - one for GreptimeDB and another for InfluxDB. For guidance on writing data to GreptimeDB using the InfluxDB line protocol, please refer to the write data section.

If retaining all historical data isn’t necessary, you can simultaneously write data to both GreptimeDB and InfluxDB for a specific period to accumulate the required recent data. Subsequently, cease writing to InfluxDB and continue exclusively with GreptimeDB. If a complete migration of all historical data is needed, please proceed with the following steps.

Export data from InfluxDB v1 Server

Create a temporary directory to store the exported data of InfluxDB.

  1. mkdir -p /path/to/export

Use the influx_inspect export command of InfluxDB to export data.

  1. influx_inspect export \
  2.   -database <db-name> \ 
  3.   -end <end-time> \
  4.   -lponly \
  5.   -datadir /var/lib/influxdb/data \
  6.   -waldir /var/lib/influxdb/wal \
  7.   -out /path/to/export/data
  • The -database flag specifies the database to be exported.
  • The -end flag specifies the end time of the data to be exported. Must be in RFC3339 format, such as 2024-01-01T00:00:00Z. You can use the timestamp when simultaneously writing data to both GreptimeDB and InfluxDB as the end time.
  • The -lponly flag specifies that only the Line Protocol data should be exported.
  • The -datadir flag specifies the path to the data directory, as configured in the InfluxDB data settings.
  • The -waldir flag specifies the path to the WAL directory, as configured in the InfluxDB data settings.
  • The -out flag specifies the output directory.

The exported data in InfluxDB line protocol looks like the following:

  1. disk,device=disk1s5s1,fstype=apfs,host=bogon,mode=ro,path=/ inodes_used=356810i 1714363350000000000
  2. diskio,host=bogon,name=disk0 iops_in_progress=0i 1714363350000000000
  3. disk,device=disk1s6,fstype=apfs,host=bogon,mode=rw,path=/System/Volumes/Update inodes_used_percent=0.0002391237988702021 1714363350000000000
  4. ...

Export Data from InfluxDB v2 Server

Create a temporary directory to store the exported data of InfluxDB.

  1. mkdir -p /path/to/export

Use the influx inspect export-lp command of InfluxDB to export data in the bucket to line protocol.

  1. influxd inspect export-lp \
  2.   --bucket-id <bucket-id> \
  3.   --engine-path /var/lib/influxdb2/engine/ \
  4.   --end <end-time> \
  5.   --output-path /path/to/export/data
  • The --bucket-id flag specifies the bucket ID to be exported.
  • The --engine-path flag specifies the path to the engine directory, as configured in the InfluxDB data settings.
  • The --end flag specifies the end time of the data to be exported. Must be in RFC3339 format, such as 2024-01-01T00:00:00Z. You can use the timestamp when simultaneously writing data to both GreptimeDB and InfluxDB as the end time.
  • The --output-path flag specifies the output directory.

The outputs look like the following:

  1. {"level":"info","ts":1714377321.4795408,"caller":"export_lp/export_lp.go:219","msg":"exporting TSM files","tsm_dir":"/var/lib/influxdb2/engine/data/307013e61d514f3c","file_count":1}
  2. {"level":"info","ts":1714377321.4940555,"caller":"export_lp/export_lp.go:315","msg":"exporting WAL files","wal_dir":"/var/lib/influxdb2/engine/wal/307013e61d514f3c","file_count":1}
  3. {"level":"info","ts":1714377321.4941633,"caller":"export_lp/export_lp.go:204","msg":"export complete"}

The exported data in InfluxDB line protocol looks like the following:

  1. cpu,cpu=cpu-total,host=bogon usage_idle=80.4448912910468 1714376180000000000
  2. cpu,cpu=cpu-total,host=bogon usage_idle=78.50167052182304 1714376190000000000
  3. cpu,cpu=cpu-total,host=bogon usage_iowait=0 1714375700000000000
  4. cpu,cpu=cpu-total,host=bogon usage_iowait=0 1714375710000000000
  5. ...

Import Data to GreptimeDB

Before importing data to GreptimeDB, if the data file is too large, it’s recommended to split the data file into multiple slices:

  1. split -l 100000 -d -a 10 data data.
  2. # -l [line_count]    Create split files line_count lines in length.
  3. # -d                 Use a numeric suffix instead of a alphabetic suffix.
  4. # -a [suffix_length] Use suffix_length letters to form the suffix of the file name.

You can import data using the HTTP API as described in the write data section. The Python script provided below will help you in reading data from the files and importing it into GreptimeDB.

Create a Python file named ingest.py, ensuring you’re using Python version 3.9 or later, and then copy and paste the following code into it.

  1. import os
  2. import sys
  3. import subprocess
  5. def process_file(file_path, url, token):
  6.     print("Ingesting file:", file_path)
  7.     curl_command = ['curl', '-i',
  8.                     '-H', "authorization: token {}".format(token),
  9.                     '-X', "POST",
  10.                     '--data-binary', "@{}".format(file_path),
  11.                     url]
  12.     print(" ".join(curl_command))
  14.     attempts = 0
  15.     while attempts < 3:  # Retry up to 3 times
  16.         result = subprocess.run(curl_command, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  17.         print(result)
  18.         # Check if there are any warnings or errors in the curl command output
  19.         output = result.stderr.lower()
  20.         if "warning" in output or "error" in output:
  21.             print("Warnings or errors detected. Retrying...")
  22.             attempts += 1
  23.         else:
  24.             break
  26.     if attempts == 3:
  27.         print("Request failed after 3 attempts. Giving up.")
  28.         sys.exit(1)
  30. def process_directory(directory, url, token):
  31.     file_names = []
  33.     # Walk through the directory
  34.     for root, dirs, files in os.walk(directory):
  35.         for file in files:
  36.             file_path = os.path.join(root, file)
  37.             file_names.append(file_path)
  39.     # Sort the file names array
  40.     file_names.sort()
  42.     # Process each file
  43.     for file_name in file_names:
  44.         process_file(file_name, url, token)
  46. # Check if the arguments are provided
  47. if len(sys.argv) < 4:
  48.     print("Please provide the directory path as the first argument, the url as the second argument and the token as the third argument.")
  49.     sys.exit(1)
  51. directory_path = sys.argv[1]
  52. url = sys.argv[2]
  53. token = sys.argv[3]
  55. # Call the function to process the directory
  56. process_directory(directory_path, url, token)

Suppose your dictionary tree is as following:

  1. .
  2. ├── ingest.py
  3. └── slices
  4.     ├── data.0000000000
  5.     ├── data.0000000001
  6.     ├── data.0000000002

Execute the Python script in the current directory and wait for the data import to complete.

  1. python3 ingest.py slices http://<greptimedb-host>:4000/v1/influxdb/write?db=<db-name> <token>