Upgrade

From v0.4, we provide a built-in utility to help upgrade your previous GreptimeDB deployment to the latest version if there are some breaking changes. It’s recommended to use this method to upgrade your GreptimeDB with different versions.

The lowest version that can be upgraded by this tool is v0.3.0.

CLI

This tool is in the greptime binary. You need to prepare the corresponding binary of target version before start.

  1. greptime cli export --help

And the help text is as follows:

  1. greptime-cli-export 
  3. USAGE:
  4.     greptime cli export [OPTIONS] --addr <ADDR> --output-dir <OUTPUT_DIR> --target <TARGET>
  6. OPTIONS:
  7.       --addr <ADDR>
  8.           Server address to connect
  10.       --output-dir <OUTPUT_DIR>
  11.           Directory to put the exported data. E.g.: /tmp/greptimedb-export
  13.       --database <DATABASE>
  14.           The name of the catalog to export
  16.           [default: greptime-*]
  18.   -j, --export-jobs <EXPORT_JOBS>
  19.           Parallelism of the export
  21.           [default: 1]
  23.       --max-retry <MAX_RETRY>
  24.           Max retry times for each job
  26.           [default: 3]
  28.   -t, --target <TARGET>
  29.           Things to export
  31.           [default: all]
  33.           Possible values:
  34.           - schema: Export all table schemas, corresponding to `SHOW CREATE TABLE`
  35.           - data:   Export all table data, corresponding to `COPY DATABASE TO`
  36.           - all:    Export all table schemas and data at once
  38.       --log-dir <LOG_DIR>
  41.       --start-time <START_TIME>
  42.           A half-open time range: [start_time, end_time). The start of the time range (time-index column) for data export
  44.       --end-time <END_TIME>
  45.           A half-open time range: [start_time, end_time). The end of the time range (time-index column) for data export
  47.       --log-level <LOG_LEVEL>
  50.       --auth-basic <AUTH_BASIC>
  51.           The basic authentication for connecting to the server
  53.   -h, --help
  54.           Print help (see a summary with '-h')
  56.   -V, --version
  57.           Print version

Here explains the meaning of some important options

  • --addr: The server address of the Frontend node or Standalone process.
  • --output-dir: The directory to put the exported data. Give a path at your current machine. The exported SQL files will be put in that directory.
  • -target: Specifies the data to export. The schema option exports the CREATE TABLE clause for each table. The data option exports the data of each database along with the COPY FROM clause. By default, all data is exported for both schema and data. It is recommended not to specify this option that use the default value to export all data.

For a complete upgrade, you will need to execute this tools twice with each target options.

Upgrade from 0.8.x

Here is a complete example for upgrading from v0.8.x to v0.9.x.

Export CREATE TABLEs and table data at once

Assuming the HTTP service port of the old database is 4000.

  1. greptime cli export --addr '127.0.0.1:4000' --output-dir /tmp/greptimedb-export

If success, you will see something like this

  1. 2024-08-01T06:32:26.547809Z  INFO cmd: Starting app: greptime-cli
  2. 2024-08-01T06:32:27.239639Z  INFO cmd::cli::export: Finished exporting greptime.greptime_private with 0 table schemas to path: /tmp/greptimedb-export/greptime/greptime_private/
  3. 2024-08-01T06:32:27.540696Z  INFO cmd::cli::export: Finished exporting greptime.pg_catalog with 0 table schemas to path: /tmp/greptimedb-export/greptime/pg_catalog/
  4. 2024-08-01T06:32:27.832018Z  INFO cmd::cli::export: Finished exporting greptime.public with 0 table schemas to path: /tmp/greptimedb-export/greptime/public/
  5. 2024-08-01T06:32:28.272054Z  INFO cmd::cli::export: Finished exporting greptime.test with 1 table schemas to path: /tmp/greptimedb-export/greptime/test/
  6. 2024-08-01T06:32:28.272166Z  INFO cmd::cli::export: Success 4/4 jobs, cost: 1.724222791s
  7. 2024-08-01T06:32:28.416532Z  INFO cmd::cli::export: Executing sql: COPY DATABASE "greptime"."greptime_private" TO '/tmp/greptimedb-export/greptime/greptime_private/' WITH (FORMAT='parquet');
  8. 2024-08-01T06:32:28.556017Z  INFO cmd::cli::export: Finished exporting greptime.greptime_private data into path: /tmp/greptimedb-export/greptime/greptime_private/
  9. 2024-08-01T06:32:28.556330Z  INFO cmd::cli::export: Finished exporting greptime.greptime_private copy_from.sql
  10. 2024-08-01T06:32:28.556424Z  INFO cmd::cli::export: Executing sql: COPY DATABASE "greptime"."pg_catalog" TO '/tmp/greptimedb-export/greptime/pg_catalog/' WITH (FORMAT='parquet');
  11. 2024-08-01T06:32:28.738719Z  INFO cmd::cli::export: Finished exporting greptime.pg_catalog data into path: /tmp/greptimedb-export/greptime/pg_catalog/
  12. 2024-08-01T06:32:28.738998Z  INFO cmd::cli::export: Finished exporting greptime.pg_catalog copy_from.sql
  13. 2024-08-01T06:32:28.739098Z  INFO cmd::cli::export: Executing sql: COPY DATABASE "greptime"."public" TO '/tmp/greptimedb-export/greptime/public/' WITH (FORMAT='parquet');
  14. 2024-08-01T06:32:28.875600Z  INFO cmd::cli::export: Finished exporting greptime.public data into path: /tmp/greptimedb-export/greptime/public/
  15. 2024-08-01T06:32:28.875888Z  INFO cmd::cli::export: Finished exporting greptime.public copy_from.sql
  16. 2024-08-01T06:32:28.876005Z  INFO cmd::cli::export: Executing sql: COPY DATABASE "greptime"."test" TO '/tmp/greptimedb-export/greptime/test/' WITH (FORMAT='parquet');
  17. 2024-08-01T06:32:29.053681Z  INFO cmd::cli::export: Finished exporting greptime.test data into path: /tmp/greptimedb-export/greptime/test/
  18. 2024-08-01T06:32:29.054104Z  INFO cmd::cli::export: Finished exporting greptime.test copy_from.sql
  19. 2024-08-01T06:32:29.054162Z  INFO cmd::cli::export: Success 4/4 jobs, costs: 781.98875ms
  20. 2024-08-01T06:32:29.054181Z  INFO cmd: Goodbye!

And now the output directory structure is

  1. /tmp/greptimedb-export/
  2. ├── greptime/public
  3. │   ├── copy_from.sql
  4. │   ├── create_tables.sql
  5. │   ├── up.parquet
  6. │   └── other-tables.parquet

The content includes create_tables.sql, copy_from.sql, and the parquet files for each table in the DB greptime-public. The create_tables.sql contains the create table statements for all tables in the current DB, while copy_from.sql includes a single COPY DATABASE FROM statement used to copy data files to the target DB. The remaining parquet files are the data files for each table.

Import table schema and data

Then you need to execute SQL files generated by the previous step. First is create_tables.sql. SQL generated in previous step is in PostgreSQL dialect, we will use PostgreSQL protocol in the following steps. This document assumes the client is psql.

Upgrade - 图1NOTICE

From this step, all the operation is done in the new version of GreptimeDB.

The default port of PostgreSQL protocol is 4003.

Before executing the following command, you need first to create the corresponding database in your new deployment (but in this example, the database greptime-public is the default one).

This command will create all the tables in the new version of GreptimeDB.

  1. psql -h 127.0.0.1 -p 4003 -d public -f /tmp/greptimedb-export/greptime/public/create_tables.sql

And then import the data

  1. psql -h 127.0.0.1 -p 4003 -d public -f /tmp/greptimedb-export/greptime/public/copy_from.sql

Clean up

At this step all the data is migrated. You can check the data in the new cluster.

After confirming that the data is correct, you can clean up the old cluster and the temporary --output-dir. /tmp/greptimedb-export at this example.

This section gives a recommended overall process for upgrading GreptimeDB smoothly. You can skip this section if your environment can go offline on the upgrade progress.

  1. Create a brand new v0.9.x cluster.
  2. Use the v0.9.x CLI tool to export and import create-table.
  3. Switch the workload to the new cluster.
  4. Use the v0.9.x CLI tool to export and import database-data.

Caveats

  • Changes to table structure between step 2 and 3 will be lost
  • Old data is invisible in new cluster until step 4 is done