Fast Cluster Restore

The Fast Cluster Restore procedure documented in this page is recommendedto speed-up the performance of arangorestorein a Cluster environment.

It is assumed that a Cluster environment is running and a logical backupwith arangodump has already been taken.

The procedure described in this page is particularly useful for ArangoDBversion 3.3, but can be used in 3.4 and later versions as well. Note thatfrom v3.4, arangorestore includes the option —threads which can be a firstgood step already in achieving restore parallelization and its speed benefit.However, the procedure below allows for even further parallelization (makinguse of different Coordinators), and the part regarding temporarily settingreplication factor to 1 is still useful in 3.4 and later versions.

The speed improvement obtained by the procedure below is achieved by:

  • Restoring into a Cluster that has replication factor 1, thus reducingnumber of network hops needed during the restore operation (_replication factor_is reverted to initial value at the end of the procedure - steps #2, #3 and #6).
  • Restoring in parallel multiple collections on different Coordinators(steps #4 and #5).

Please refer tothissection for further context on the factors affecting restore speed when restoringusing arangorestore in a Cluster.

Step 1: Copy the dump directory to all Coordinators

The first step is to copy the directory that contains the dump to all machineswhere Coordinators are running.

This step is not strictly required as the backup can be restored over thenetwork. However, if the restore is executed locally the restore speed issignificantly improved.

Step 2: Restore collection structures

The collection structures have to be restored from exactly one Coordinator (anyCoordinator can be used) with a command similar to the following one. Please addany additional needed option for your specific use case, e.g. —create-databaseif the database where you want to restore does not exist yet:

  1. arangorestore
  2.   --server.endpoint <endpoint-of-a-coordinator>
  3.   --server.database <database-name>
  4.   --server.password <password>
  5.   --import-data false
  6.   --input-directory <dump-directory>

If you are using v3.3.22 or higher, or v3.4.2 or higher, please also add in thecommand above the option —replication-factor 1.

The option —import-data false tells arangorestore to restore only thecollection structure and no data.

Step 3: Set Replication Factor to 1

This step is not needed if you are using v3.3.22 or higher or v3.4.2 or higherand you have used in the previous step the option —replication-factor 1.

To speed up restore, it is possible to set the replication factor to 1 beforeimporting any data. Run the following command from exactly one Coordinator (anyCoordinator can be used):

  1. echo 'db._collections().filter(function(c) { return c.name()[0] !== "_"; })
  2. .forEach(function(c) { print("collection:", c.name(), "replicationFactor:",
  3. c.properties().replicationFactor); c.properties({ replicationFactor: 1 }); });'
  4. | arangosh
  5.   --server.endpoint <endpoint-of-a-coordinator>
  6.   --server.database <database-name>
  7.   --server.username <user-name>
  8.   --server.password <password>

Step 4: Create parallel restore scripts

Now that the Cluster is prepared, the parallelRestore script will be used.

Please create the below parallelRestore script in any of your Coordinators.

When executed (see below for further details), this script will create other scriptsthat can be then copied and executed on each Coordinator.

  1. #!/bin/sh
  2. #
  3. # Version: 0.3
  4. #
  5. # Release Notes:
  6. # - v0.3: fixed a bug that was happening when the collection name included an underscore 
  7. # - v0.2: compatibility with version 3.4: now each coordinator_<number-of-coordinator>.sh
  8. #         includes a single restore command (instead of one for each collection)
  9. #         which allows making using of the --threads option in v.3.4.0 and later
  10. # - v0.1: initial version
  12. if test -z "$ARANGOSH" ; then
  13.   export ARANGOSH=arangosh
  14. fi
  15. cat > /tmp/parallelRestore$$.js <<'EOF'
  16. var fs = require("fs");
  17. var print = require("internal").print;
  18. var exit = require("internal").exit;
  19. var arangorestore = "arangorestore";
  20. var env = require("internal").env;
  21. if (env.hasOwnProperty("ARANGORESTORE")) {
  22.   arangorestore = env["ARANGORESTORE"];
  23. }
  25. // Check ARGUMENTS: dumpDir coordinator1 coordinator2 ...
  27. if (ARGUMENTS.length < 2) {
  28.   print("Need at least two arguments DUMPDIR and COORDINATOR_ENDPOINTS!");
  29.   exit(1);
  30. }
  32. var dumpDir = ARGUMENTS[0];
  33. var coordinators = ARGUMENTS[1].split(",");
  34. var otherArgs = ARGUMENTS.slice(2);
  36. // Quickly check the dump dir:
  37. var files = fs.list(dumpDir).filter(f => !fs.isDirectory(f));
  38. var found = files.indexOf("ENCRYPTION");
  39. if (found === -1) {
  40.   print("This directory does not have an ENCRYPTION entry.");
  41.   exit(2);
  42. }
  43. // Remove ENCRYPTION entry:
  44. files = files.slice(0, found).concat(files.slice(found+1));
  46. for (let i = 0; i < files.length; ++i) {
  47.   if (files[i].slice(-5) !== ".json") {
  48.     print("This directory has files which do not end in '.json'!");
  49.     exit(3);
  50.   }
  51. }
  52. files = files.map(function(f) {
  53.   var fullName = fs.join(dumpDir, f);
  54.   var collName = "";
  55.   if (f.slice(-10) === ".data.json") {
  56.     var pos;
  57.     if (f.slice(0, 1) === "_") {  // system collection
  58.       pos = f.slice(1).indexOf("_") + 1;
  59.       collName = "_" + f.slice(1, pos);
  60.     } else {
  61.       pos = f.lastIndexOf("_")
  62.       collName = f.slice(0, pos);
  63.     }
  64.   }
  65.   return {name: fullName, collName, size: fs.size(fullName)};
  66. });
  67. files = files.sort(function(a, b) { return b.size - a.size; });
  68. var dataFiles = [];
  69. for (let i = 0; i < files.length; ++i) {
  70.   if (files[i].name.slice(-10) === ".data.json") {
  71.     dataFiles.push(i);
  72.   }
  73. }
  75. // Produce the scripts, one for each coordinator:
  76. var scripts = [];
  77. var collections = [];
  78. for (let i = 0; i < coordinators.length; ++i) {
  79.   scripts.push([]);
  80.   collections.push([]);
  81. }
  83. var cnum = 0;
  84. var temp = '';
  85. var collections = [];
  86. for (let i = 0; i < dataFiles.length; ++i) {
  87.   var f = files[dataFiles[i]];
  88.   if (typeof collections[cnum] == 'undefined') {
  89.     collections[cnum] = (`--collection ${f.collName}`);
  90.   } else {
  91.     collections[cnum] += (` --collection ${f.collName}`);
  92.   }
  93.   cnum += 1;
  94.   if (cnum >= coordinators.length) {
  95.     cnum = 0;
  96.   }
  97. }
  99. var cnum = 0;
  100. for (let i = 0; i < coordinators.length; ++i) {
  101.   scripts[i].push(`${arangorestore} --input-directory ${dumpDir} --server.endpoint ${coordinators[i]} ` + collections[i] + ' ' + otherArgs.join(" "));
  102. }
  104. for (let i = 0; i < coordinators.length; ++i) {
  105.   let f = "coordinator_" + i + ".sh";
  106.   print("Writing file", f, "...");
  107.   fs.writeFileSync(f, scripts[i].join("\n"));
  108. }
  109. EOF
  111. ${ARANGOSH} --javascript.execute /tmp/parallelRestore$$.js -- "$@"
  112. rm /tmp/parallelRestore$$.js

To run this script, all Coordinator endpoints of the Cluster have to beprovided. The script accepts all options of the tool arangorestore.

The command below can for instance be used on a Cluster with threeCoordinators:

  1. ./parallelRestore <dump-directory>
  2.   tcp://<ip-of-coordinator1>:<port of coordinator1>,
  3.   tcp://<ip-of-coordinator2>:<port of coordinator2>,
  4.   tcp://<ip-of-coordinator3>:<port of coordinator3>
  5.   --server.username <username>
  6.   --server.password <password>
  7.   --server.database <database_name>
  8.   --create-collection false

Notes:

  • The option —create-collection false is passed since the collectionstructures were created already in the previous step.
  • Starting from v3.4.0 the arangorestore option —threads N can bepassed to the command above, where N is an integer, to further parallelizethe restore (default is —threads 2).The above command will create three scripts, where three corresponds tothe amount of listed Coordinators.

The resulting scripts are named coordinator_<number-of-coordinator>.sh (e.g.coordinator_0.sh, coordinator_1.sh, coordinator_2.sh).

Step 5: Execute parallel restore scripts

The coordinator_<number-of-coordinator>.sh scripts, that were created in theprevious step, now have to be executed on each machine where a _Coordinator_is running. This will start a parallel restore of the dump.

Step 6: Revert to the initial Replication Factor

Once the arangorestore process on every Coordinator is completed, thereplication factor has to be set to its initial value.

Run the following command from exactly one Coordinator (any Coordinator can beused). Please adjust the replicationFactor value to your specific case (2 in theexample below):

  1. echo 'db._collections().filter(function(c) { return c.name()[0] !== "_"; })
  2. .forEach(function(c) { print("collection:", c.name(), "replicationFactor:",
  3. c.properties().replicationFactor); c.properties({ replicationFactor: 2 }); });'
  4. | arangosh
  5.   --server.endpoint <endpoint-of-a-coordinator>
  6.   --server.database <database-name>
  7.   --server.username <user-name>
  8.   --server.password <password>