Data wire formats

This section describes the data wire format of standard EdgeDB types.

Sets and array<>

The set and array values are represented as the following structure:

  1. struct SetOrArrayValue {
  2.     // Number of dimensions, currently must
  3.     // always be 0 or 1. 0 indicates an empty set or array.
  4.     int32       ndims;
  6.     // Reserved.
  7.     int32       reserved0;
  9.     // Reserved.
  10.     int32       reserved1;
  12.     // Dimension data.
  13.     Dimension   dimensions[ndims];
  15.     // Element data, the number of elements
  16.     // in this array is the sum of dimension sizes:
  17.     // sum((d.upper - d.lower + 1) for d in dimensions)
  18.     Element     elements[];
  19. };
  21. struct Dimension {
  22.     // Upper dimension bound, inclusive,
  23.     // number of elements in the dimension
  24.     // relative to the lower bound.
  25.     int32       upper;
  27.     // Lower dimension bound, always 1.
  28.     int32       lower;
  29. };
  31. struct Element {
  32.     // Encoded element data length in bytes.
  33.     int32       length;
  35.     // Element data.
  36.     uint8       data[length];
  37. };

Note: zero-length arrays (and sets) are represented as a 12-byte value where dims equal to zero regardless of the shape in type descriptor.

Sets of arrays are a special case. Every array within a set is wrapped in an Envelope. The full structure follows:

  1. struct SetOfArrayValue {
  2.     // Number of dimensions, currently must
  3.     // always be 0 or 1. 0 indicates an empty set.
  4.     int32 ndims;
  6.     // Reserved.
  7.     int32 reserved0;
  9.     // Reserved.
  10.     int32 reserved1;
  12.     // Dimension data. Same layout as above.
  13.     Dimension dimensions[ndims];
  15.     // Envelope data, the number of elements
  16.     // in this array is the sum of dimension sizes:
  17.     // sum((d.upper - d.lower + 1) for d in dimensions)
  18.     Envelope elements[];
  19. };
  21. struct Envelope {
  22.     // Encoded envelope element length in bytes.
  23.     int32 length;
  25.     // Number of elements, currently must
  26.     // always be 1.
  27.     int32 nelems;
  29.     // Reserved.
  30.     int32 reserved
  32.     // Element data. Same layout as above.
  33.     Element element[nelems];
  34. };

tuple<>, namedtuple<>, and object<>

The values are represented as the following structure:

  1. struct TupleOrNamedTupleOrObjectValue {
  2.     // Number of elements
  3.     int32       nelems;
  5.     // Element data.
  6.     Element     elements[nelems];
  7. };
  9. struct Element {
  10.     // Reserved.
  11.     int32       reserved;
  13.     // Encoded element data length in bytes.
  14.     int32       length;
  16.     // Element data.
  17.     uint8       data[length];
  18. };

Note that for objects, Element.length can be set to -1, which means an empty set.

Sparse Objects

The values are represented as the following structure:

  1. struct SparseObjectValue {
  2.     // Number of elements
  3.     int32       nelems;
  5.     // Element data.
  6.     Element     elements[nelems];
  7. };
  9. struct Element {
  10.     // Index of the element in the input shape.
  11.     int32       index;
  13.     // Encoded element data length in bytes.
  14.     int32       length;
  16.     // Element data.
  17.     uint8       data[length];
  18. };

Ranges

The ranges are represented as the following structure:

  1. struct Range {
  2.     // A bit mask of range definition.
  3.     uint8<RangeFlag> flags;
  5.     // Lower boundary data.
  6.     Boundary         lower;
  8.     // Upper boundary data.
  9.     Boundary         upper;
  10. };
  12. struct Boundary {
  13.     // Encoded boundary data length in bytes.
  14.     int32       length;
  16.     // Boundary data.
  17.     uint8       data[length];
  18. };
  20. enum RangeFlag {
  21.     // Empty range.
  22.     EMPTY   = 0x0001;
  24.     // Included lower boundary.
  25.     LB_INC  = 0x0002;
  27.     // Included upper boundary.
  28.     UB_INC  = 0x0004;
  30.     // Inifinity (excluded) lower boundary.
  31.     LB_INF  = 0x0008;
  33.     // Infinity (excluded) upper boundary.
  34.     UB_INF  = 0x0010;
  35. };

std::uuid

The std::uuid values are represented as a sequence of 16 unsigned byte values.

For example, the UUID value b9545c35-1fe7-485f-a6ea-f8ead251abd3 is represented as:

  1. 0xb9 0x54 0x5c 0x35 0x1f 0xe7 0x48 0x5f
  2. 0xa6 0xea 0xf8 0xea 0xd2 0x51 0xab 0xd3

std::str

The std::str values are represented as a UTF-8 encoded byte string. For example, the str value 'Hello! 🙂' is encoded as:

  1. 0x48 0x65 0x6c 0x6c 0x6f 0x21 0x20 0xf0 0x9f 0x99 0x82

std::bytes

The std::bytes values are represented as-is.

std::int16

The std::int16 values are represented as two bytes, most significant byte first.

For example, the int16 value 6556 is represented as:

  1. 0x19 0x9c

std::int32

The std::int32 values are represented as four bytes, most significant byte first.

For example, the int32 value 655665 is represented as:

  1. 0x00 0x0a 0x01 0x31

std::int64

The std::int64 values are represented as eight bytes, most significant byte first.

For example, the int64 value 123456789987654321 is represented as:

  1. 0x01 0xb6 0x9b 0x4b 0xe0 0x52 0xfa 0xb1

std::float32

The std::float32 values are represented as a IEEE 754-2008 binary 32-bit value, most significant byte first.

For example, the float32 value -15.625 is represented as:

  1. 0xc1 0x7a 0x00 0x00

std::float64

The std::float32 values are represented as a IEEE 754-2008 binary 64-bit value, most significant byte first.

For example, the float64 value -15.625 is represented as:

  1. 0xc0 0x2f 0x40 0x00 0x00 0x00 0x00 0x00

std::decimal

The std::decimal values are represented as the following structure:

  1. struct Decimal {
  2.     // Number of digits in digits[], can be 0.
  3.     uint16               ndigits;
  5.     // Weight of first digit.
  6.     int16                weight;
  8.     // Sign of the value
  9.     uint16<DecimalSign>  sign;
  11.     // Value display scale.
  12.     uint16               dscale;
  14.     // base-10000 digits.
  15.     uint16                digits[ndigits];
  16. };
  18. enum DecimalSign {
  19.     // Positive value.
  20.     POS     = 0x0000;
  22.     // Negative value.
  23.     NEG     = 0x4000;
  24. };

The decimal values are represented as a sequence of base-10000 digits. The first digit is assumed to be multiplied by weight * 10000, i.e. there might be up to weight + 1 digits before the decimal point. Trailing zeros can be absent. It is possible to have negative weight.

dscale, or display scale, is the nominal precision expressed as number of base-10 digits after the decimal point. It is always non-negative. dscale may be more than the number of physically present fractional digits, implying significant trailing zeroes. The actual number of digits physically present in the digits array contains trailing zeros to the next 4-byte increment (meaning that integer and fractional part are always distinc base-10000 digits).

For example, the decimal value -15000.6250000 is represented as:

  1. // ndigits
  2. 0x00 0x04
  4. // weight
  5. 0x00 0x01
  7. // sign
  8. 0x40 0x00
  10. // dscale
  11. 0x00 0x07
  13. // digits
  14. 0x00 0x01 0x13 0x88 0x18 0x6a 0x00 0x00

std::bool

The std::bool values are represented as an int8 with only two valid values: 0x01 for true and 0x00 for false.

std::datetime

The std::datetime values are represented as a 64-bit integer, most sigificant byte first. The value is the number of microseconds between the encoded datetime and January 1st 2000, 00:00 UTC. A Unix timestamp can be converted into an EdgeDB datetime value using this formula:

  1. edb_datetime = (unix_ts + 946684800) * 1000000

For example, the datetime value '2019-05-06T12:00+00:00' is encoded as:

  1. 0x00 0x02 0x2b 0x35 0x9b 0xc4 0x10 0x00

cal::local_datetime

The cal::local_datetime values are represented as a 64-bit integer, most sigificant byte first. The value is the number of microseconds between the encoded datetime and January 1st 2000, 00:00.

For example, the local_datetime value '2019-05-06T12:00' is encoded as:

  1. 0x00 0x02 0x2b 0x35 0x9b 0xc4 0x10 0x00

cal::local_date

The cal::local_date values are represented as a 32-bit integer, most sigificant byte first. The value is the number of days between the encoded date and January 1st 2000.

For example, the local_date value '2019-05-06' is encoded as:

  1. 0x00 0x00 0x1b 0x99

cal::local_time

The cal::local_time values are represented as a 64-bit integer, most sigificant byte first. The value is the number of microseconds since midnight.

For example, the local_time value '12:10' is encoded as:

  1. 0x00 0x00 0x00 0x0a 0x32 0xae 0xf6 0x00

std::duration

The std::duration values are represented as the following structure:

  1. struct Duration {
  2.     int64   microseconds;
  4.     // deprecated, is always 0
  5.     int32   days;
  7.     // deprecated, is always 0
  8.     int32   months;
  9. };

For example, the duration value '48 hours 45 minutes 7.6 seconds' is encoded as:

  1. // microseconds
  2. 0x00 0x00 0x00 0x28 0xdd 0x11 0x72 0x80
  4. // days
  5. 0x00 0x00 0x00 0x00
  7. // months
  8. 0x00 0x00 0x00 0x00

cal::relative_duration

The cal::relative_duration values are represented as the following structure:

  1. struct Duration {
  2.     int64   microseconds;
  3.     int32   days;
  4.     int32   months;
  5. };

For example, the cal::relative_duration value '2 years 7 months 16 days 48 hours 45 minutes 7.6 seconds' is encoded as:

  1. // microseconds
  2. 0x00 0x00 0x00 0x28 0xdd 0x11 0x72 0x80
  4. // days
  5. 0x00 0x00 0x00 0x10
  7. // months
  8. 0x00 0x00 0x00 0x1f

std::json

The std::json values are represented as the following structure:

  1. struct JSON {
  2.     uint8   format;
  3.     uint8   jsondata[];
  4. };

format is currently always 1, and jsondata is a UTF-8 encoded JSON string.

std::bigint

The std::bigint values are represented as the following structure:

  1. struct BigInt {
  2.     // Number of digits in digits[], can be 0.
  3.     uint16               ndigits;
  5.     // Weight of first digit.
  6.     int16                weight;
  8.     // Sign of the value
  9.     uint16<DecimalSign>  sign;
  11.     // Reserved value, must be zero
  12.     uint16               reserved;
  14.     // base-10000 digits.
  15.     uint16                digits[ndigits];
  16. };
  18. enum BigIntSign {
  19.     // Positive value.
  20.     POS     = 0x0000;
  22.     // Negative value.
  23.     NEG     = 0x4000;
  24. };

The decimal values are represented as a sequence of base-10000 digits. The first digit is assumed to be multiplied by weight * 10000, i.e. there might be up to weight + 1 digits. Trailing zeros can be absent.

For example, the bigint value -15000 is represented as:

  1. // ndigits
  2. 0x00 0x02
  4. // weight
  5. 0x00 0x01
  7. // sign
  8. 0x40 0x00
  10. // reserved
  11. 0x00 0x00
  13. // digits
  14. 0x00 0x01 0x13 0x88

cfg::memory

The cfg::memory values are represented as a number of bytes encoded as a 64-bit integer, most sigificant byte first.

For example, the cfg::memory value 123MiB is represented as:

  1. 0x00 0x00 0x00 0x00 0x07 0xb0 0x00 0x00