Incompatible changes in ArangoDB 2.3

It is recommended to check the following list of incompatible changes before upgrading to ArangoDB 2.3, and adjust any client programs if necessary.

Configuration file changes

Threads and contexts

The number of server threads specified is now the minimum of threadsstarted. There are situation in which threads are waiting for results ofdistributed database servers. In this case the number of threads is dynamicallyincreased.

With ArangoDB 2.3, the number of server threads can be configured independentlyof the number of V8 contexts. The configuration option—javascript.v8-contexts was added to arangod to provide better control overthe number of V8 contexts created in arangod.

Previously, the number of V8 contexts arangod created at startup was equal to the number of server threads (as specified by option —server.threads).

In some situations it may be more sensible to create different amounts of threads and V8 contexts. This is because each V8 contexts created will consume memoryand requires CPU resources for periodic garbage collection. Contrary, serverthreads do not have such high memory or CPU footprint.

If the option —javascript.v8-contexts is not specified, the number of V8contexts created at startup will remain equal to the number of server threads.Thus no change in configuration is required to keep the same behavior as inprevious ArangoDB versions.

If you are using the default config files or merge them with your local configfiles, please review if the default number of server threads is okay in yourenvironment. Additionally you should verify that the number of V8 contextscreated (as specified in option —javascript.v8-contexts) is okay.

Syslog

The command-line option —log.syslog was used in previous versions ofArangoDB to turn logging to syslog on or off: when setting to a non-emptystring, syslog logging was turned on, otherwise turned off.When syslog logging was turned on, logging was done with the applicationname specified in —log.application, which defaulted to triagens.There was also a command-line option —log.hostname which could be setbut did not have any effect.

This behavior turned out to be unintuitive and was changed in 2.3 as follows:

  • the command-line option —log.syslog is deprecated and does not have anyeffect when starting ArangoDB.
  • to turn on syslog logging in 2.3, the option —log.facility has to be setto a non-empty string. The value for facility is OS-dependent (possiblevalues can be found in /usr/include/syslog.h or the like - user shouldbe available on many systems).
  • the default value for —log.application has been changed from triagens toarangod.
  • the command-line option —log.hostname is deprecated and does not have anyeffect when starting ArangoDB. Instead, the host name will be set by syslogautomatically.
  • when logging to syslog, ArangoDB now omits the datetime prefix and the processid, because they’ll be added by syslog automatically.

AQL

AQL queries throw less exceptions

ArangoDB 2.3 contains a completely rewritten AQL query optimizer and execution engine. This means that AQL queries will be executed with a different engine thanin ArangoDB 2.2 and earlier. Parts of AQL queries might be executed in different order than before because the AQL optimizer has more freedom to move things around in a query.

In previous versions of ArangoDB, AQL queries aborted with an exception in manysituations and threw a runtime exception. Exceptions were thrown when trying tofind a value using the IN operator in a non-array element, when trying to usenon-boolean values with the logical operands && or || or !, when using non-numericvalues in arithmetic operations, when passing wrong parameters into functions etc.

In ArangoDB 2.3 this has been changed in many cases to make AQL more user-friendlyand to allow the optimization to perform much more query optimizations.

Here is a summary of changes:

  • when a non-array value is used on the right-hand side of the IN operator, the result will be false in ArangoDB 2.3, and no exception will be thrown.
  • the boolean operators && and || do not throw in ArangoDB 2.3 if any of theoperands is not a boolean value. Instead, they will perform an implicit cast ofthe values to booleans. Their result will be as follows:
    • lhs && rhs will return lhs if it is false or would be false when convertedinto a boolean. If lhs is true or would be true when converted to a boolean,rhs will be returned.
    • lhs || rhs will return lhs if it is true or would be true when convertedinto a boolean. If lhs is false or would be false when converted to a boolean,rhs will be returned.
    • ! value will return the negated value of value converted into a boolean
  • the arithmetic operators (+, -, *, /, %) can be applied to any value and will not throw exceptions when applied to non-numeric values. Instead, any value usedin these operators will be casted to a numeric value implicitly. If no numeric resultcan be produced by an arithmetic operator, it will return null in ArangoDB 2.3. Thisis also true for division by zero.
  • passing arguments of invalid types into AQL functions does not throw a runtimeexception in most cases, but may produce runtime warnings. Built-in AQL functions that receive invalid arguments will then return null.

Nested FOR loop execution order

The query optimizer in 2.3 may permute the order of nested FOR loops in AQL queries,provided that exchanging the loops will not alter a query result. However, a changein the order of returned values is allowed because no sort order is guaranteed by AQL(and was never) unless an explicit SORT statement is used in a query.

Changed return values of ArangoQueryCursor.getExtra()

The return value of ArangoQueryCursor.getExtra() has been changed in ArangoDB 2.3.It now contains a stats attribute with statistics about the query previously executed.It also contains a warnings attribute with warnings that happened during queryexecution. The return value structure has been unified in 2.3 for both read-only anddata-modification queries.

The return value looks like this for a read-only query:

  1. arangosh> stmt = db._createStatement("FOR i IN mycollection RETURN i"); stmt.execute().getExtra()
  2. { 
  3.   "stats" : { 
  4.     "writesExecuted" : 0, 
  5.     "writesIgnored" : 0, 
  6.     "scannedFull" : 2600, 
  7.     "scannedIndex" : 0 
  8.   }, 
  9.   "warnings" : [ ] 
  10. }

For data-modification queries, ArangoDB 2.3 returns a result with the same structure:

  1. arangosh> stmt = db._createStatement("FOR i IN xx REMOVE i IN xx"); stmt.execute().getExtra()
  2. { 
  3.   "stats" : { 
  4.     "writesExecuted" : 2600, 
  5.     "writesIgnored" : 0, 
  6.     "scannedFull" : 2600, 
  7.     "scannedIndex" : 0 
  8.   }, 
  9.   "warnings" : [ ] 
  10. }

In ArangoDB 2.2, the return value of ArangoQueryCursor.getExtra() was empty for read-onlyqueries and contained an attribute operations with two sub-attributes for data-modification queries:

  1. arangosh> stmt = db._createStatement("FOR i IN mycollection RETURN i"); stmt.execute().getExtra()
  2. { 
  3. }
  1. arangosh> stmt = db._createStatement("FOR i IN mycollection REMOVE i IN mycollection"); stmt.execute().getExtra()
  2. { 
  3.   "operations" : { 
  4.     "executed" : 2600, 
  5.     "ignored" : 0 
  6.   } 
  7. }

Changed return values in HTTP method POST /_api/cursor

The previously mentioned change also leads to the statistics being returned in theHTTP REST API method POST /_api/cursor. Previously, the return value containedan optional extra attribute that was filled only for data-modification queries and in some other cases as follows:

  1. {
  2.   "result" : [ ],
  3.   "hasMore" : false,
  4.   "extra" : {
  5.     "operations" : {
  6.       "executed" : 2600,
  7.       "ignored" : 0
  8.     }
  9.   }
  10. }

With the changed result structure in ArangoDB 2.3, the extra attribute in the resultwill look like this:

  1. { 
  2.   "result" : [],
  3.   "hasMore" : false,
  4.   "extra" : {
  5.     "stats" : {
  6.       "writesExecuted" : 2600,
  7.       "writesIgnored" : 0,
  8.       "scannedFull" : 0,
  9.       "scannedIndex" : 0
  10.     },
  11.     "warnings" : [ ]
  12.   }
  13. }

If the query option fullCount is requested, the fullCount result value will alsobe returned inside the stats attribute of the extra attribute, and not directlyas an attribute inside the extra attribute as in 2.2. Note that a fullCount willonly be present in extra.stats if it was requested as an option for the query.

The result in ArangoDB 2.3 will also contain a warnings attribute with the array of warnings that happened during query execution.

Changed return values in ArangoStatement.explain()

The return value of ArangoStatement.explain() has changed significantly in ArangoDB 2.3. The new return value structure is not compatible with the structurereturned by 2.2.

In ArangoDB 2.3, the full execution plan for an AQL query is returned alongside all applied optimizer rules, optimization warnings etc. It is also possible to have the optimizer return all execution plans. This required a new data structure.

Client programs that use ArangoStatement.explain() or the HTTP REST API methodPOST /_api/explain may need to be adjusted to use the new return format.

The return value of ArangoStatement.parse() has been extended in ArangoDB 2.3.In addition to the existing attributes, ArangoDB 2.3 will also return an ast attributecontaining the abstract syntax tree of the statement. This extra attribute cansafely be ignored by client programs.

Variables not updatable in queries

Previous versions of ArangoDB allowed the modification of variables inside AQL queries, e.g.

  1. LET counter = 0
  2. FOR i IN 1..10 
  3.   LET counter = counter + 1
  4.   RETURN counter

While this is admittedly a convenient feature, the new query optimizer design did notallow to keep it. Additionally, updating variables inside a query would prevent a lotof optimizations to queries that we would like the optimizer to make. Additionally,updating variables in queries that run on different nodes in a cluster would like causenon-deterministic behavior because queries are not executed linearly.

Changed return value of TO_BOOL

The AQL function TOBOOL now always returns _true if its argument is an array or an object.In previous versions of ArangoDB, the function returned false for empty arrays or forobjects without attributes.

Changed return value of TO_NUMBER

The AQL function TONUMBER now returns _null if its argument is an object or anarray with more than one member. In previous version of ArangoDB, the returnvalue in these cases was 0. TO_NUMBER will return 0 for empty array, and the numericequivalent of the array member’s value for arrays with a single member.

New AQL keywords

The following keywords have been added to AQL in ArangoDB 2.3:

  • NOT
  • AND
  • ORUnquoted usage of these keywords for attribute names in AQL queries will likelyfail in ArangoDB 2.3. If any such attribute name needs to be used in a query, itshould be enclosed in backticks to indicate the usage of a literal attributename.

Removed features

Bitarray indexes

Bitarray indexes were only half-way documented and integrated in previous versions of ArangoDB so their benefit was limited. The support for bitarray indexes hasthus been removed in ArangoDB 2.3. It is not possible to create indexes of type“bitarray” with ArangoDB 2.3.

When a collection is opened that contains a bitarray index definition created with a previous version of ArangoDB, ArangoDB will ignore it and log the followingwarning:

  1. index type 'bitarray' is not supported in this version of ArangoDB and is ignored

Future versions of ArangoDB may automatically remove such index definitions so thewarnings will eventually disappear.

Other removed features

The HTTP REST API method at POST /_admin/modules/flush has been removed.

Known issues

In ArangoDB 2.3.0, AQL queries containing filter conditions with an IN expression will not yet use an index:

  1. FOR doc IN collection FILTER doc.indexedAttribute IN [ ... ] RETURN doc
  3. FOR doc IN collection
  4.   FILTER doc.indexedAttribute IN [ ... ]
  5.   RETURN doc

We’re currently working on getting the IN optimizations done, and will ship them in a 2.3 maintenance release soon (e.g. 2.3.1 or 2.3.2).