Foxx queues

const queues = require('@arangodb/foxx/queues')

Foxx allows defining job queues that let you perform slow or expensive actionsasynchronously. These queues can be used to send e-mails, call external APIs orperform other actions that you do not want to perform directly or want to retryon failure.

Foxx queue jobs can be any script named in the manifest of a service in the same database.

Please note that Foxx queues are database-specific. Queues and jobs are alwaysrelative to the database in which they are created or accessed.

For disabling the Foxx queues feature or adjusting the polling interval see thefoxx.queues and foxx.queues-poll-interval options.

For the low-level functionality see the chapter on thetask management module.

Managing queues

queues.create

queues.create(name, [maxWorkers]): Queue

Returns the queue for the given name. If the queue does not exist, a new queuewith the given name will be created. If a queue with the given name already existsand maxWorkers is set, the queue’s maximum number of workers will be updated.The queue will be created in the current database.

Arguments

  • name: string

Name of the queue to create.

  • maxWorkers: number (Default: 1)

The maximum number of workers.

Examples

  1. // Create a queue with the default number of workers (i.e. one)
  2. const queue1 = queues.create("my-queue");
  3. // Create a queue with a given number of workers
  4. const queue2 = queues.create("another-queue", 2);
  5. // Update the number of workers of an existing queue
  6. const queue3 = queues.create("my-queue", 10);
  7. // queue1 and queue3 refer to the same queue
  8. assertEqual(queue1, queue3);

queues.get

queues.get(name): Queue

Returns the queue for the given name. If the queue does not exist an exceptionis thrown instead.

The queue will be looked up in the current database.

Arguments

  • name: string

Name of the queue to fetch.

Examples

If the queue does not yet exist an exception is thrown:

  1. queues.get("some-queue");
  2. // Error: Queue does not exist: some-queue
  3. //     at ...

Otherwise the queue will be returned:

  1. const queue1 = queues.create("some-queue");
  2. const queue2 = queues.get("some-queue");
  3. assertEqual(queue1, queue2);

queues.delete

queues.delete(name): boolean

Returns true if the queue was deleted successfully.If the queue did not exist, it returns false instead.The queue will be looked up and deleted in the current database.

When a queue is deleted, jobs on that queue will no longer be executed.

Deleting a queue will not delete any jobs on that queue.

Arguments

  • name: string

Name of the queue to delete.

Examples

  1. const queue = queues.create("my-queue");
  2. queues.delete("my-queue"); // true
  3. queues.delete("my-queue"); // false

Queue API

queue.push

queue.push(script, data, [opts]): string

The job will be added to the specified queue in the current database.

Returns the job id.

Arguments

  • script: object

A job type definition, consisting of an object with the following properties:

  • name: string

Name of the script that will be invoked.

  • mount: string

Mount path of the service that defines the script.

  • backOff: Function | number (Default: 1000)

Either a function that takes the number of times the job has failed beforeas input and returns the number of milliseconds to wait before trying thejob again, or the delay to be used to calculate anexponential back-off,or 0 for no delay.

  • maxFailures: number | Infinity (Default: 0):

Number of times a single run of a job will be re-tried before it is markedas "failed". A negative value or Infinity means that the job will bere-tried on failure indefinitely.

  • schema: Schema (optional)

Schema to validate a job’s data against before enqueuing the job.

  • preprocess: Function (optional)

Function to pre-process a job’s (validated) data before serializing it in the queue.

  • data: any

Job data of the job; must be serializable to JSON.

  • opts: object (optional)

Object with any of the following properties:

  • success: Function (optional)

Function to be called after the job has been completed successfully.

  • failure: Function (optional)

Function to be called after the job has failed too many times.

  • delayUntil: number | Date (Default: Date.now())

Timestamp in milliseconds (or Date instance) until which the execution ofthe job should be delayed.

  • backOff: Function | number (Default: 1000)

See script.backOff.

  • maxFailures: number | Infinity (Default: 0):

See script.maxFailures.

  • repeatTimes: number (Default: 0)

If set to a positive number, the job will be repeated this many times(not counting recovery when using maxFailures).If set to a negative number or Infinity, the job will be repeatedindefinitely. If set to 0 the job will not be repeated.

  • repeatUntil: number | Date (optional)

If the job is set to automatically repeat, this can be set to a timestampin milliseconds (or Date instance) after which the job will no longer repeat.Setting this value to zero, a negative value or Infinity has no effect.

  • repeatDelay: number (Default: 0)

If the job is set to automatically repeat, this can be set to a non-negativevalue to set the number of milliseconds for which the job will be delayedbefore it is started again.

Note that if you pass a function for the backOff calculation, success_callback or _failure callback options the function will be serialized tothe database as a string and therefore must not rely on any external scopeor external variables.

When the job is set to automatically repeat, the failure callback will onlybe executed when a run of the job has failed more than maxFailures times.Note that if the job fails and maxFailures is set, it will be rescheduledaccording to the backOff until it has either failed too many times orcompleted successfully before being scheduled according to the repeatDelay_again. Recovery attempts by _maxFailures do not count towards repeatTimes.

The success and failure callbacks receive the following arguments:

  • result: any

The return value of the script for the current run of the job.

  • jobData: any

The data passed to this method.

  • job: object

ArangoDB document representing the job’s current state.

Examples

Let’s say we have an service mounted at /mailer that provides a script called send-mail:

  1. 'use strict';
  2. const queues = require('@arangodb/foxx/queues');
  3. const queue = queues.create('my-queue');
  4. queue.push(
  5.   {mount: '/mailer', name: 'send-mail'},
  6.   {to: 'hello@example.com', body: 'Hello world'}
  7. );

This will not work, because log was defined outside the callback function(the callback must be serializable to a string):

  1. // WARNING: THIS DOES NOT WORK!
  2. 'use strict';
  3. const queues = require('@arangodb/foxx/queues');
  4. const queue = queues.create('my-queue');
  5. const log = require('console').log; // outside the callback's function scope
  6. queue.push(
  7.   {mount: '/mailer', name: 'send-mail'},
  8.   {to: 'hello@example.com', body: 'Hello world'},
  9.   {success: function () {
  10.     log('Yay!'); // throws 'log is not defined'
  11.   }}
  12. );

Here’s an example of a job that will be executed every 5 seconds until tomorrow:

  1. 'use strict';
  2. const queues = require('@arangodb/foxx').queues;
  3. const queue = queues.create('my-queue');
  4. queue.push(
  5.   {mount: '/mailer', name: 'send-mail'},
  6.   {to: 'hello@example.com', body: 'Hello world'},
  7.   {
  8.     repeatTimes: Infinity,
  9.     repeatUntil: Date.now() + (24 * 60 * 60 * 1000),
  10.     repeatDelay: 5 * 1000
  11.   }
  12. );

queue.get

queue.get(jobId): Job

Creates a proxy object representing a job with the given job id.

The job will be looked up in the specified queue in the current database.

Returns the job for the given jobId. Properties of the job object will befetched whenever they are referenced and can not be modified.

Arguments

  • jobId: string

The id of the job to create a proxy object for.

Examples

  1. const jobId = queue.push({mount: '/logger', name: 'log'}, 'Hello World!');
  2. const job = queue.get(jobId);
  3. assertEqual(job.id, jobId);

queue.delete

queue.delete(jobId): boolean

Deletes a job with the given job id.The job will be looked up and deleted in the specified queue in the current database.

Arguments

  • jobId: string

The id of the job to delete.

Returns true if the job was deleted successfully. If the job did not existit returns false instead.

queue.pending

queue.pending([script]): Array<string>

Returns an array of job ids of jobs in the given queue with the status"pending", optionally filtered by the given job type.The jobs will be looked up in the specified queue in the current database.

Arguments

  • script: object (optional)

An object with the following properties:

  • name: string

Name of the script.

  • mount: string

Mount path of the service defining the script.

Examples

  1. const logScript = {mount: '/logger', name: 'log'};
  2. queue.push(logScript, 'Hello World!', {delayUntil: Date.now() + 50});
  3. assertEqual(queue.pending(logScript).length, 1);
  4. // 50 ms later...
  5. assertEqual(queue.pending(logScript).length, 0);
  6. assertEqual(queue.progress(logScript).length, 1);
  7. // even later...
  8. assertEqual(queue.progress(logScript).length, 0);
  9. assertEqual(queue.complete(logScript).length, 1);

queue.progress

queue.progress([script])

Returns an array of job ids of jobs in the given queue with the status"progress", optionally filtered by the given job type.The jobs will be looked up in the specified queue in the current database.

Arguments

  • script: object (optional)

An object with the following properties:

  • name: string

Name of the script.

  • mount: string

Mount path of the service defining the script.

queue.complete

queue.complete([script]): Array<string>

Returns an array of job ids of jobs in the given queue with the status"complete", optionally filtered by the given job type.The jobs will be looked up in the specified queue in the current database.

Arguments

  • script: object (optional)

An object with the following properties:

  • name: string

Name of the script.

  • mount: string

Mount path of the service defining the script.

queue.failed

queue.failed([script]): Array<string>

Returns an array of job ids of jobs in the given queue with the status"failed", optionally filtered by the given job type.The jobs will be looked up in the specified queue in the current database.

Arguments

  • script: object (optional)

An object with the following properties:

  • name: string

Name of the script.

  • mount: string

Mount path of the service defining the script.

queue.all

queue.all([script]): Array<string>

Returns an array of job ids of all jobs in the given queue,optionally filtered by the given job type.The jobs will be looked up in the specified queue in the current database.

Arguments

  • script: object (optional)

An object with the following properties:

  • name: string

Name of the script.

  • mount: string

Mount path of the service defining the script.

Job API

job.abort

job.abort(): void

Aborts a non-completed job.

Sets a job’s status to "failed" if it is not already "complete",without calling the job’s onFailure callback.