Features and Improvements in ArangoDB 3.1

The following list shows in detail which features have been added or improved inArangoDB 3.1. ArangoDB 3.1 also contains several bugfixes that are not listedhere.

SmartGraphs

ArangoDB 3.1 adds a first major Enterprise Edition feature called SmartGraphs.SmartGraphs form an addition to the already existing graph features and allow toscale graphs beyond a single machine while keeping almost the same query performance.The SmartGraph feature is suggested for all graph database use cases that requirea cluster of DB-Servers for what ever reason.You can either have a graph that is too large to be stored on a single machine only.Or you can have a small graph, but at the same time need additional data with has to besharded and you want to keep all of them in the same environment.Or you simply use the cluster for high-availability.In all the above cases SmartGraphs will significantly increase the performance ofgraph operations.For more detailed information read the SmartGraphs section.

Data format

The format of the revision values stored in the _rev attribute of documentshas been changed in 3.1. Up to 3.0 they were strings containing largish decimal numbers. With 3.1, revision values are still strings, but are actually encoded time stamps of the creation date of the revision of the document. The time stamps are acquired using a hybrid logical clock (HLC) on the DB-Server that holds therevision (for the concept of a hybrid logical clock seethis paper).See this manual section for details.

ArangoDB >= 3.1 can ArangoDB 3.0 database directories and will simply continueto use the old _rev attribute values. New revisions will be written withthe new time stamps.

It is highly recommended to backup all your data before loading a databasedirectory that was written by ArangoDB <= 3.0 into an ArangoDB >= 3.1.

Communication Layer

ArangoDB up to 3.0 used libev forthe communication layer. ArangoDB starting from 3.1 usesBoost ASIO.

Starting with ArangoDB 3.1 we begin to provide the VelocyStream Protocol (vst) asa addition to the established http protocol.

A few options have changed concerning communication, please checkoutIncompatible changes in 3.1.

Cluster

For its internal cluster communication a (bundled version) of curl is now beingused. This enables asynchronous operation throughout the cluster and shouldimprove general performance slightly.

Authentication is now supported within the cluster.

Document revisions cache

The ArangoDB server now provides an in-memory cache for frequently accesseddocument revisions. Documents that are accessed during read/write operationsare loaded into the revisions cache automatically, and subsequently served fromthere.

The cache has a total target size, which can be controlled with the startupoption —database.revision-cache-target-size. Once the cache reaches thetarget size, older entries may be evicted from the cache to free memory. Note thatthe target size currently is a high water mark that will trigger cache memorygarbage collection if exceeded. However, if all cache chunks are still in usewhen the high water mark is reached, the cache may still grow and allocate morechunks until cache entries become unused and are allowed to be garbage-collected.

The cache is maintained on a per-collection basis, that is, memory for the cacheis allocated on a per-collection basis in chunks. The size for the cache memorychunks can be controlled via the startup option —database.revision-cache-chunk-size.The default value is 4 MB per chunk.Bigger chunk sizes allow saving more documents per chunk, which can lead to moreefficient chunk allocation and lookups, but will also lead to memory waste if manychunks are allocated and not fully used. The latter will be the case if there existmany small collections which all allocate their own chunks but not fully utilize thembecause of the low number of documents.

AQL

Functions added

The following AQL functions have been added in 3.1:

  • OUTERSECTION(array1, array2, …, arrayn): returns the values that occuronly once across all arrays specified.

  • DISTANCE(lat1, lon1, lat2, lon2): returns the distance between the twocoordinates specified by (lat1, lon1) and (lat2, lon2). The distance iscalculated using the haversine formula.

  • JSON_STRINGIFY(value): returns a JSON string representation of the value.

  • JSON_PARSE(value): converts a JSON-encoded string into a regular object

Index usage in traversals

3.1 allows AQL traversals to use other indexes than just the edge index.Traversals with filters on edges can now make use of more specific indexes. Forexample, the query

  1. FOR v, e, p IN 2 OUTBOUND @start @@edge
  2.   FILTER p.edges[0].foo == "bar"
  3.   RETURN [v, e, p]

may use a hash index on ["_from", "foo"] instead of the edge index on just["_from"].

Optimizer improvements

Make the AQL query optimizer inject filter condition expressions referred toby variables during filter condition aggregation. For example, in the followingquery

  1. FOR doc IN collection
  2.   LET cond1 = (doc.value == 1)
  3.   LET cond2 = (doc.value == 2)
  4.   FILTER cond1 || cond2
  5.   RETURN { doc, cond1, cond2 }

the optimizer will now inject the conditions for cond1 and cond2 into thefilter condition cond1 || cond2, expanding it to (doc.value == 1) || (doc.value == 2)and making these conditions available for index searching.

Note that the optimizer previously already injected some conditions into otherconditions, but only if the variable that defined the condition was not used elsewhere.For example, the filter condition in the query

  1. FOR doc IN collection
  2.   LET cond = (doc.value == 1)
  3.   FILTER cond
  4.   RETURN { doc }

already got optimized before because cond was only used once in the query and theoptimizer decided to inject it into the place where it was used.

This only worked for variables that were referred to once in the query. When a variablewas used multiple times, the condition was not injected as in the following query

  1. FOR doc IN collection
  2.   LET cond = (doc.value == 1)
  3.   FILTER cond
  4.   RETURN { doc, cond }

3.1 allows using this condition so that the query can use an index on doc.value(if such index exists).

Miscellaneous improvements

The performance of the [*] operator was improved for cases in which this operatordid not use any filters, projections and/or offset/limits.

The AQL query executor can now report the time required for loading and locking thecollections used in an AQL query. When profiling is enabled, it will report the totalloading and locking time for the query in the loading collections sub-attribute of theextra.profile value of the result. The loading and locking time can also be view in theAQL query editor in the web interface.

Audit Log

Audit logging has been added, see Auditing.

Client tools

Added option —skip-lines for arangoimpThis allows skipping the first few lines from the import file in case the CSV or TSVimport are used and some initial lines should be skipped from the input.

Web Admin Interface

The usability of the AQL editor significantly improved. In addition to the standard JSONoutput, the AQL Editor is now able to render query results as a graph preview or a table.Furthermore the AQL editor displays query profiling information.

Added a new Graph Viewer in order to exchange the technically obsolete version. The new GraphViewer is based on Canvas but does also include a first WebGL implementation (limitedfunctionality - will change in the future). The new Graph Viewer offers a smooth way todiscover and visualize your graphs.

The shard view in cluster mode now displays a progress indicator while moving shards.

Authentication

Up to ArangoDB 3.0 authentication of client requests was only possible with HTTP basicauthentication.

Starting with 3.1 it is now possible to also use a JSON Web Tokens(JWT) for authenticating incoming requests.

For details check the HTTP authentication chapter. Both authentication methods arevalid and will be supported in the near future. Use whatever suits you best.

Foxx

GraphQL

It is now easy to get started with providing GraphQL APIs in Foxx, see Foxx GraphQL.

OAuth2

Foxx now officially provides a module for implementing OAuth2 clients, see Foxx OAuth2.

Per-route middleware

It’s now possible to specify middleware functions for a route when defining a route handler. These middleware functions only apply to the single route and share the route’s parameter definitions. Check out the Foxx Router documentation for more information.