Backup and Restore

EMQX adopts a distributed storage schema and also introduces a cluster transfer feature to ensure the system’s high availability.

This chapter will discuss how to backup your operating data and configuration files to prevent data losses in case of system malfunctions.

Data Import and Export

EMQX 5.1 introduces a user-friendly command-line tool for data import and export. While similar to the one available in EMQX 4.x, it has several significant differences and is not compatible with it.

In EMQX 4.x, a single JSON file was used to store all the necessary data from both the EMQX configuration and the built-in database. However, in EMQX 5.1, an opaque compressed tar archive is used. This allows for more efficient and structured handling of potentially large amounts of user data.

The data supported for import and export in EMQX 5.1 is analogous to that of EMQX 4.x. It includes:

  • Contents of EMQX configuration rewrite file:
    • Rules and bridges
    • Authentication and authorization configuration
    • Listeners, gateways configuration
    • Other EMQX settings
  • Built-in database (Mnesia) data:
    • Dashboard users and API keys
    • Authentication user credentials (both simple and enhanced)
    • PSK authentication data
    • Authorization rules
    • Banned clients
  • SSL/TLS certificates stored in EMQX data directory (node.data_dir)
  • Authorization acl.conf file stored in EMQX data directory

Warning

Any SSL/TLS certificate or acl.conf files stored outside the EMQX data directory are not included in the exported archive. Therefore, when exporting and importing data, it’s important to note that these files should be provisioned on all EMQX target nodes before importing the data. This ensures that the necessary SSL/TLS certificates and acl.conf files are available for proper functionality.

Export

Data can be exported from any running cluster node.

Import

To import data, the EMQX node must be running, and certain conditions must be met for a successful import operation:

  • If Core + Replicant mode is enabled, the data import can only be initiated on core nodes. This does not affect the actual import behavior as the data will be copied to all cluster nodes, both core and replicant. Initiating the import on a core node makes sure that data is imported correctly.
  • Data exported from EMQX Enterprise Edition cluster cannot be imported to EMQX Open Source Edition cluster.
  • The data file must not be renamed.

If any of the above conditions are not satisfied, the import process will be aborted, and an appropriate error message will be displayed.

During the data import operation, the data will either be inserted (if it was not present in the target EMQX cluster) or updated (in case of a conflict) in EMQX. The import process does not remove any existing data from the EMQX cluster.

Warning

In rare circumstances, the existing data may not be compatible with the imported data.

For example, if the EMQX cluster uses the built-in database authentication mechanism with a ‘suffix’ salt position, and the imported data defines the same authentication source with a ‘prefix’ salt position, the old user credentials created before the import will no longer be operational.

Therefore, importing data to a non-clean EMQX cluster may require extra caution.

Example

This section shows how to import and export data using the command Line Interface.

  1. Export data. The file name format of the exported file is emqx-export-YYYY-MM-DD-HH-mm-ss.sss.tar.gz, and export directory is <EMQX data directory>/backup:

    1. $ ./emqx_ctl data export
    2. Exporting data to "data/backup/emqx-export-2023-06-19-15-14-19.947.tar.gz"...
    3. Exporting cluster configuration...
    4. Exporting additional files from EMQX data_dir: "data"...
    5. Exporting built-in database...
    6. Exporting emqx_admin database table...
    7. Exporting emqx_authn_mnesia database table...
    8. Exporting emqx_enhanced_authn_scram_mnesia database table...
    9. Exporting emqx_app database table...
    10. Exporting emqx_acl database table...
    11. Exporting emqx_psk database table...
    12. Exporting emqx_banned database table...
    13. Data has been successfully exported to data/backup/emqx-export-2023-06-19-15-14-19.947.tar.gz.

  2. Import data. The name of the imported file can be specified as an absolute path or a relative path. If the file resides in <EMQX data directory>/backup directory, its basename without a path can also be used, e.g.:

    1. # import the file by the absolute path
    2. $ ./emqx_ctl data import /tmp/emqx-export-2023-06-19-15-14-19.947.tar.gz
    3. Importing data from "/tmp/emqx-export-2023-06-19-15-14-19.947.tar.gz"...
    4. Importing cluster configuration...
    5. Importing built-in database...
    6. Importing emqx_banned database table...
    7. Importing emqx_psk database table...
    8. Importing emqx_acl database table...
    9. Importing emqx_app database table...
    10. Importing emqx_enhanced_authn_scram_mnesia database table...
    11. Importing emqx_authn_mnesia database table...
    12. Importing emqx_admin database table...
    13. Data has been imported successfully.
    15. # import the file by the path relative to EMQX root directory:
    16. $ ./emqx_ctl data import ../../../tmp/emqx-export-2023-06-21-13-28-06.418.tar.gz
    17. Importing data from "../../../tmp/emqx-export-2023-06-21-13-28-06.418.tar.gz"...
    18. Importing cluster configuration...
    19. Importing built-in database...
    20. Importing emqx_enhanced_authn_scram_mnesia database table...
    21. Importing emqx_authn_mnesia database table...
    22. Importing emqx_admin database table...
    23. Importing emqx_acl database table...
    24. Importing emqx_banned database table...
    25. Importing emqx_psk database table...
    26. Importing emqx_app database table...
    27. Data has been imported successfully.
    29. # import the file from `<EMQX data directory>/backup` directory:
    30. $ cp /tmp/emqx-export-2023-06-21-13-28-06.418.tar.gz /opt/emqx/data/backup/
    31. $ ./emqx_ctl data import emqx-export-2023-06-21-13-28-06.418.tar.gz
    32. Importing data from "data/backup/emqx-export-2023-06-21-13-28-06.418.tar.gz"...
    33. Importing cluster configuration...
    34. Importing built-in database...
    35. Importing emqx_enhanced_authn_scram_mnesia database table...
    36. Importing emqx_authn_mnesia database table...
    37. Importing emqx_admin database table...
    38. Importing emqx_acl database table...
    39. Importing emqx_banned database table...
    40. Importing emqx_psk database table...
    41. Importing emqx_app database table...
    42. Data has been imported successfully.