Fast Cluster Restore

The Fast Cluster Restore procedure documented in this page is recommended to speed-up the performance of arangorestore in a Cluster environment.

It is assumed that a Cluster environment is running and a logical backup with arangodump has already been taken.

The procedure described in this page is particularly useful for ArangoDB version 3.3, but can be used in 3.4 and later versions as well. Note that from v3.4, arangorestore includes the option --threads which can be a first good step already in achieving restore parallelization and its speed benefit. In ArangoDB 3.8 and later, you should create dumps with arangodump in the non-enveloped format, as this will allow arangorestore to restore single collections with multiple threads. However, the procedure below allows for even further parallelization (making use of different Coordinators), and the part regarding temporarily setting replication factor to 1 is still useful in 3.4 and later versions.

The speed improvement obtained by the procedure below is achieved by:

  1. Restoring into a Cluster that has replication factor 1, thus reducing number of network hops needed during the restore operation (replication factor is reverted to initial value at the end of the procedure - steps #2, #3 and #6).
  2. Restoring in parallel multiple collections on different Coordinators (steps #4 and #5).

Please refer to the arangorestore examples for further context on the factors affecting restore speed when restoring using arangorestore in a Cluster.

Step 1: Copy the dump directory to all Coordinators

The first step is to copy the directory that contains the dump to all machines where Coordinators are running.

This step is not strictly required as the backup can be restored over the network. However, if the restore is executed locally the restore speed is significantly improved.

Step 2: Restore collection structures

The collection structures have to be restored from exactly one Coordinator (any Coordinator can be used) with a command similar to the following one. Please add any additional needed option for your specific use case, e.g. --create-database if the database where you want to restore does not exist yet:

  1. arangorestore
  2.   --server.endpoint <endpoint-of-a-coordinator>
  3.   --server.database <database-name>
  4.   --server.password <password>
  5.   --import-data false
  6.   --input-directory <dump-directory>

If you are using v3.3.22 or higher, or v3.4.2 or higher, please also add in the command above the option --replication-factor 1.

The option --import-data false tells arangorestore to restore only the collection structure and no data.

Step 3: Set Replication Factor to 1

This step is not needed if you are using v3.3.22 or higher or v3.4.2 or higher and you have used in the previous step the option --replication-factor 1.

To speed up restore, it is possible to set the replication factor to 1 before importing any data. Run the following command from exactly one Coordinator (any Coordinator can be used):

  1. echo 'db._collections().filter(function(c) { return c.name()[0] !== "_"; })
  2. .forEach(function(c) { print("collection:", c.name(), "replicationFactor:",
  3. c.properties().replicationFactor); c.properties({ replicationFactor: 1 }); });'
  4. | arangosh
  5.   --server.endpoint <endpoint-of-a-coordinator>
  6.   --server.database <database-name>
  7.   --server.username <user-name>
  8.   --server.password <password>

Step 4: Create parallel restore scripts

Now that the Cluster is prepared, the parallelRestore script will be used.

Please create the below parallelRestore script in any of your Coordinators.

When executed (see below for further details), this script will create other scripts that can be then copied and executed on each Coordinator.

  1. #!/bin/sh
  2. #
  3. # Version: 0.3
  4. #
  5. # Release Notes:
  6. # - v0.3: fixed a bug that was happening when the collection name included an underscore 
  7. # - v0.2: compatibility with version 3.4: now each coordinator_<number-of-coordinator>.sh
  8. #         includes a single restore command (instead of one for each collection)
  9. #         which allows making using of the --threads option in v.3.4.0 and later
  10. # - v0.1: initial version
  12. if test -z "$ARANGOSH" ; then
  13.   export ARANGOSH=arangosh
  14. fi
  15. cat > /tmp/parallelRestore$$.js <<'EOF'
  16. var fs = require("fs");
  17. var print = require("internal").print;
  18. var exit = require("internal").exit;
  19. var arangorestore = "arangorestore";
  20. var env = require("internal").env;
  21. if (env.hasOwnProperty("ARANGORESTORE")) {
  22.   arangorestore = env["ARANGORESTORE"];
  23. }
  25. // Check ARGUMENTS: dumpDir coordinator1 coordinator2 ...
  27. if (ARGUMENTS.length < 2) {
  28.   print("Need at least two arguments DUMPDIR and COORDINATOR_ENDPOINTS!");
  29.   exit(1);
  30. }
  32. var dumpDir = ARGUMENTS[0];
  33. var coordinators = ARGUMENTS[1].split(",");
  34. var otherArgs = ARGUMENTS.slice(2);
  36. // Quickly check the dump dir:
  37. var files = fs.list(dumpDir).filter(f => !fs.isDirectory(f));
  38. var found = files.indexOf("ENCRYPTION");
  39. if (found === -1) {
  40.   print("This directory does not have an ENCRYPTION entry.");
  41.   exit(2);
  42. }
  43. // Remove ENCRYPTION entry:
  44. files = files.slice(0, found).concat(files.slice(found+1));
  46. for (let i = 0; i < files.length; ++i) {
  47.   if (files[i].slice(-5) !== ".json") {
  48.     print("This directory has files which do not end in '.json'!");
  49.     exit(3);
  50.   }
  51. }
  52. files = files.map(function(f) {
  53.   var fullName = fs.join(dumpDir, f);
  54.   var collName = "";
  55.   if (f.slice(-10) === ".data.json") {
  56.     var pos;
  57.     if (f.slice(0, 1) === "_") {  // system collection
  58.       pos = f.slice(1).indexOf("_") + 1;
  59.       collName = "_" + f.slice(1, pos);
  60.     } else {
  61.       pos = f.lastIndexOf("_")
  62.       collName = f.slice(0, pos);
  63.     }
  64.   }
  65.   return {name: fullName, collName, size: fs.size(fullName)};
  66. });
  67. files = files.sort(function(a, b) { return b.size - a.size; });
  68. var dataFiles = [];
  69. for (let i = 0; i < files.length; ++i) {
  70.   if (files[i].name.slice(-10) === ".data.json") {
  71.     dataFiles.push(i);
  72.   }
  73. }
  75. // Produce the scripts, one for each Coordinator:
  76. var scripts = [];
  77. var collections = [];
  78. for (let i = 0; i < coordinators.length; ++i) {
  79.   scripts.push([]);
  80.   collections.push([]);
  81. }
  83. var cnum = 0;
  84. var temp = '';
  85. var collections = [];
  86. for (let i = 0; i < dataFiles.length; ++i) {
  87.   var f = files[dataFiles[i]];
  88.   if (typeof collections[cnum] == 'undefined') {
  89.     collections[cnum] = (`--collection ${f.collName}`);
  90.   } else {
  91.     collections[cnum] += (` --collection ${f.collName}`);
  92.   }
  93.   cnum += 1;
  94.   if (cnum >= coordinators.length) {
  95.     cnum = 0;
  96.   }
  97. }
  99. var cnum = 0;
  100. for (let i = 0; i < coordinators.length; ++i) {
  101.   scripts[i].push(`${arangorestore} --input-directory ${dumpDir} --server.endpoint ${coordinators[i]} ` + collections[i] + ' ' + otherArgs.join(" "));
  102. }
  104. for (let i = 0; i < coordinators.length; ++i) {
  105.   let f = "coordinator_" + i + ".sh";
  106.   print("Writing file", f, "...");
  107.   fs.writeFileSync(f, scripts[i].join("\n"));
  108. }
  109. EOF
  111. ${ARANGOSH} --javascript.execute /tmp/parallelRestore$$.js -- "$@"
  112. rm /tmp/parallelRestore$$.js

To run this script, all Coordinator endpoints of the Cluster have to be provided. The script accepts all options of the tool arangorestore.

The command below can for instance be used on a Cluster with three Coordinators:

  1. ./parallelRestore <dump-directory>
  2.   tcp://<ip-of-coordinator1>:<port of coordinator1>,
  3.   tcp://<ip-of-coordinator2>:<port of coordinator2>,
  4.   tcp://<ip-of-coordinator3>:<port of coordinator3>
  5.   --server.username <username>
  6.   --server.password <password>
  7.   --server.database <database_name>
  8.   --create-collection false

Notes:

  • The option --create-collection false is passed since the collection structures were created already in the previous step.
  • Starting from v3.4.0 the arangorestore option --threads N can be passed to the command above, where N is an integer, to further parallelize the restore (default is --threads 2).

The above command will create three scripts, where three corresponds to the amount of listed Coordinators.

The resulting scripts are named coordinator_<number-of-coordinator>.sh (e.g. coordinator_0.sh, coordinator_1.sh, coordinator_2.sh).

Step 5: Execute parallel restore scripts

The coordinator_<number-of-coordinator>.sh scripts, that were created in the previous step, now have to be executed on each machine where a Coordinator is running. This will start a parallel restore of the dump.

Step 6: Revert to the initial Replication Factor

Once the arangorestore process on every Coordinator is completed, the replication factor has to be set to its initial value.

Run the following command from exactly one Coordinator (any Coordinator can be used). Please adjust the replicationFactor value to your specific case (2 in the example below):

  1. echo 'db._collections().filter(function(c) { return c.name()[0] !== "_"; })
  2. .forEach(function(c) { print("collection:", c.name(), "replicationFactor:",
  3. c.properties().replicationFactor); c.properties({ replicationFactor: 2 }); });'
  4. | arangosh
  5.   --server.endpoint <endpoint-of-a-coordinator>
  6.   --server.database <database-name>
  7.   --server.username <user-name>
  8.   --server.password <password>