- 29. Typed Arrays: handling binary data (Advanced)
- 29.1. The basics of the API
- 29.2. Foundations of the Typed Array API
- 29.3. ArrayBuffers
- 29.4. Typed Arrays
- 29.4.1. Typed Arrays vs. normal Arrays
- 29.4.2. Typed Arrays are iterable
- 29.4.3. Converting Typed Arrays to and from normal Arrays
- 29.4.4. The class hierarchy of Typed Arrays
- 29.4.5. Static methods of TypedArray
- 29.4.6. Properties of TypedArray<T>.prototype
- 29.4.7. new «ElementType»Array()
- 29.4.8. Static properties of «ElementType»Array
- 29.4.9. Properties of «ElementType»Array.prototype
- 29.4.10. Concatenating Typed Arrays
- 29.5. DataViews
- 29.6. Further reading
29. Typed Arrays: handling binary data (Advanced)
29.1. The basics of the API
Much data on the web is text: JSON files, HTML files, CSS files, JavaScript code, etc. JavaScript handles such data well, via its built-in strings.
However, before 2011, it did not handle binary data well. The Typed Array Specification 1.0 was introduced on 8 February 2011 and provides tools for working with binary data. With ECMAScript 6, Typed Arrays were added to the core language and gained methods that were previously only available for normal Arrays (.map()
, .filter()
, etc.).
29.1.1. Use cases for Typed Arrays
The main uses cases for Typed Arrays are:
- Processing binary data: manipulating image data in HTML Canvas elements, parsing binary files, handling binary network protocols, etc.
- Interacting with native APIs: Native APIs often receive and return data in a binary format, which you could neither store nor manipulate well in pre-ES6 JavaScript. That meant that, whenever you were communicating with such an API, data had to be converted from JavaScript to binary and back, for every call. Typed Arrays eliminate this bottleneck. One example of communicating with native APIs is WebGL, for which Typed Arrays were initially created. Section “History of Typed Arrays” of the article “Typed Arrays: Binary Data in the Browser” (by Ilmari Heikkinen for HTML5 Rocks) has more information.
29.1.2. Browser APIs that support Typed Arrays
The following browser APIs support Typed Arrays:
29.1.3. The core classes: ArrayBuffer, Typed Arrays, DataView
The Typed Array API stores binary data in instances of ArrayBuffer
:
An ArrayBuffer itself is opaque. If you want to access its data, you must wrap it in another object – a view object. Two kinds of view objects are available:
- Typed Arrays: let you access the data as an indexed sequence of elements that all have the same type. Examples include:
Uint8Array
: Elements are unsigned 8-bit integers. Unsigned means that their ranges start at zero.Int16Array
: Elements are signed 16-bit integers. Signed means that they have a sign and can be negative, zero, or positive.Float32Array
: Elements are 32-bit floating point numbers.
- DataViews: let you interpret the data as various types (
Uint8
,Int16
,Float32
, etc.) that you can read and write at any byte offset.
Fig. 19 shows a class diagram of the API.
29.1.4. Using Typed Arrays
Typed Arrays are used much like normal Arrays, with a few notable differences:
- Typed Arrays store their data in ArrayBuffers.
- All elements are initialized with zeros.
- All elements have the same type. Writing values to a Typed Array coerces them to that type. Reading values produces normal numbers.
- Once a Typed Array is created, its length never changes.
- Typed Arrays can’t have holes.
This is an example of using a Typed Array:
const typedArray = new Uint8Array(2); // 2 elements
assert.equal(typedArray.length, 2);
// The wrapped ArrayBuffer
assert.deepEqual(
typedArray.buffer, new ArrayBuffer(2)); // 2 bytes
// Getting and setting elements:
assert.equal(typedArray[1], 0); // initialized with 0
typedArray[1] = 72;
assert.equal(typedArray[1], 72);
Other ways of creating Typed Arrays:
29.1.5. Using DataViews
This is how DataViews are used:
29.2. Foundations of the Typed Array API
29.2.1. Element types
The following element types are supported by the API:
Element type | Bytes | Description | C type |
---|---|---|---|
Int8 | 1 | 8-bit signed integer | signed char |
Uint8 | 1 | 8-bit unsigned integer | unsigned char |
Uint8C | 1 | 8-bit unsigned integer (clamped conversion) | unsigned char |
Int16 | 2 | 16-bit signed integer | short |
Uint16 | 2 | 16-bit unsigned integer | unsigned short |
Int32 | 4 | 32-bit signed integer | int |
Uint32 | 4 | 32-bit unsigned integer | unsigned int |
Float32 | 4 | 32-bit floating point | float |
Float64 | 8 | 64-bit floating point | double |
The element type Uint8C
is special: it is not supported by DataView
and only exists to enable Uint8ClampedArray
. This Typed Array is used by the canvas
element (where it replaces CanvasPixelArray
). The only difference between Uint8C
and Uint8
is how overflow and underflow are handled (as explained in the next section). It is recommended to avoid the former – quoting Brendan Eich:
Just to be super-clear (and I was around when it was born),Uint8ClampedArray
is totally a historical artifact (of the HTML5 canvas element). Avoid unless you really are doing canvas-y things.
29.2.2. Handling overflow and underflow
Normally, when a value is out of the range of the element type, modulo arithmetic is used to convert it to a value within range. For signed and unsigned integers that means that:
- The highest value plus one is converted to the lowest value (0 for unsigned integers).
- The lowest value minus one is converted to the highest value.
The following functions helps illustrate how conversion works:
Modulo conversion for unsigned 8-bit integers:
Modulo conversion for signed 8-bit integers:
Clamped conversion is different:
- All underflowing values are converted to the lowest value.
- All overflowing values are converted to the highest value.
29.2.3. Endianness
Whenever a type (such as Uint16
) is stored as a sequence of multiple bytes, endianness matters:
- Big endian: the most significant byte comes first. For example, the
Uint16
value 0x4321 is stored as two bytes – first 0x43, then 0x21. - Little endian: the least significant byte comes first. For example, the
Uint16
value 0x4321 is stored as two bytes – first 0x21, then 0x43.
Endianness tends to be fixed per CPU architecture and consistent across native APIs. Typed Arrays are used to communicate with those APIs, which is why their endianness follows the endianness of the platform and can’t be changed.
On the other hand, the endianness of protocols and binary files varies and is fixed across platforms. Therefore, we must be able to access data with either endianness. DataViews serve this use case and let you specify endianness when you get or set a value.
Quoting Wikipedia on Endianness:
- Big-endian representation is the most common convention in data networking; fields in the protocols of the Internet protocol suite, such as IPv4, IPv6, TCP, and UDP, are transmitted in big-endian order. For this reason, big-endian byte order is also referred to as network byte order.
- Little-endian storage is popular for microprocessors in part due to significant historical influence on microprocessor designs by Intel Corporation.
You can use the following function to determine the endianness of a platform.
const BIG_ENDIAN = Symbol('BIG_ENDIAN');
const LITTLE_ENDIAN = Symbol('LITTLE_ENDIAN');
function getPlatformEndianness() {
const arr32 = Uint32Array.of(0x87654321);
const arr8 = new Uint8Array(arr32.buffer);
if (compare(arr8, [0x87, 0x65, 0x43, 0x21])) {
return BIG_ENDIAN;
} else if (compare(arr8, [0x21, 0x43, 0x65, 0x87])) {
return LITTLE_ENDIAN;
} else {
throw new Error('Unknown endianness');
}
}
function compare(arr1, arr2) {
if (arr1.length !== arr2.length) {
return false;
}
for (let i=0; i<arr1.length; i++) {
if (arr1[i] !== arr2[i]) {
return false;
}
}
return true;
}
Other orderings are also possible. Those are generically called middle-endian or mixed-endian.
29.2.4. Indices and offsets
For Typed Arrays, we distinguish:
Indices for the bracket operator
[ ]
: You can only use non-negative indices (starting at 0).Indices for methods of ArrayBuffers, Typed Arrays and DataViews: Every index can be negative. If it is, it is added to the length of the entity, to produce the actual index. Therefore
-1
refers to the last element,-2
to the second-last, etc. Methods of normal Arrays work the same way.
- Offsets passed to methods of Typed Arrays and DataViews: must be non-negative. For example:
29.3. ArrayBuffers
ArrayBuffers store binary data, which is meant to be accessed via Typed Arrays and DataViews.
29.3.1. new ArrayBuffer()
The type signature of the constructor is:
Invoking this constructor via new
creates an instance whose capacity is length
bytes. Each of those bytes is initially 0.
You can’t change the length of an ArrayBuffer, you can only create a new one with a different length.
29.3.2. Static methods of ArrayBuffer
ArrayBuffer.isView(arg: any)
Returnstrue
ifarg
is an object and a view for an ArrayBuffer (a Typed Array or a DataView).
29.3.3. Properties of ArrayBuffer.prototype
get .byteLength(): number
Returns the capacity of this ArrayBuffer in bytes.
.slice(startIndex: number, endIndex=this.byteLength)
Creates a new ArrayBuffer that contains the bytes of this ArrayBuffer whose indices are greater than or equal to startIndex
and less than endIndex
. start
and endIndex
can be negative (see section “Indices and offsets”).
29.4. Typed Arrays
The various kinds of Typed Arrays are only different w.r.t. the types of their elements:
- Typed Arrays whose elements are integers:
Int8Array
,Uint8Array
,Uint8ClampedArray
,Int16Array
,Uint16Array
,Int32Array
,Uint32Array
- Typed Arrays whose elements are floats:
Float32Array
,Float64Array
29.4.1. Typed Arrays vs. normal Arrays
Typed Arrays are much like normal Arrays: they have a .length
, elements can be accessed via the bracket operator [ ]
and they have most of the standard Array methods. They differ from normal Arrays in the following ways:
- Typed Arrays have buffers. The elements of a Typed Array
ta
are not stored inta
, they are stored in an associated ArrayBuffer that can be accessed viata.buffer
:
Typed Arrays are initialized with zeros:
new Array(4)
creates a normal Array without any elements. It only has 4 holes (indices less than the.length
that have no associated elements).new Uint8Array(4)
creates a Typed Array whose 4 elements are all 0.
All of the elements of a Typed Array have the same type:
- Setting elements converts values to that type.
- Getting elements returns numbers.
The
.length
of a Typed Array is derived from its ArrayBuffer and never changes (unless you switch to a different ArrayBuffer).Normal Arrays can have holes; Typed Arrays can’t.
29.4.2. Typed Arrays are iterable
Typed Arrays are iterable. That means that you can use the for-of
loop and similar mechanisms:
ArrayBuffers and DataViews are not iterable.
29.4.3. Converting Typed Arrays to and from normal Arrays
To convert a normal Array to a Typed Array, you pass it to a Typed Array constructor. For example:
To convert a Typed Array to a normal Array, you can use spreading or Array.from()
(because Typed Arrays are iterable):
29.4.4. The class hierarchy of Typed Arrays
The properties of the various Typed Array objects are introduced in two steps:
TypedArray
: First, we look at the common superclass of all Typed Array classes (which was shown in the class diagram at the beginning of this chapter). I’m calling that superclassTypedArray
, but it is not directly accessible from JavaScript.TypedArray.prototype
houses all methods of Typed Arrays.«ElementType»Array
: The actual Typed Array classes are calledUint8Array
,Int16Array
,Float32Array
, etc.
29.4.5. Static methods of TypedArray
Both static TypedArray
methods are inherited by its subclasses (Uint8Array
etc.).
29.4.5.1. TypedArray.of()
This method has the type signature:
The notation of the return type is my invention: .of()
returns an instance of this
(the class on which of()
was invoked). The elements of the instance are the arguments of of()
.
You can think of of()
as a custom literal for Typed Arrays:
29.4.5.2. TypedArray.from()
This method has the type signature:
It converts source
into an instance of this
(a Typed Array). Once again, the syntax instanceof this
is my invention.
For example, normal Arrays are iterable and can be converted with this method:
Typed Arrays are also iterable:
The source
can also be an Array-like object:
The optional mapfn
lets you transform the elements of source
before they become elements of the result. Why perform the two steps mapping and conversion in one go? Compared to mapping separately via .map()
, there are two advantages:
- No intermediate Array or Typed Array is needed.
- When converting between Typed Arrays with different precisions, less can go wrong.
To illustrate the second advantage, let’s first convert a Typed Array to a Typed Array with a higher precision. If we use.from()
to map, the result is automatically correct. Otherwise, you must first convert and then map.
const typedArray = Int8Array.of(127, 126, 125);
assert.deepEqual(
Int16Array.from(typedArray, x => x * 2),
Int16Array.of(254, 252, 250));
assert.deepEqual(
Int16Array.from(typedArray).map(x => x * 2),
Int16Array.of(254, 252, 250)); // OK
assert.deepEqual(
Int16Array.from(typedArray.map(x => x * 2)),
Int16Array.of(-2, -4, -6)); // wrong
If we go from a Typed Array to a Typed Array with a lower precision, mapping via .from()
produces the correct result. Otherwise, we must first map and then convert.
assert.deepEqual(
Int8Array.from(Int16Array.of(254, 252, 250), x => x / 2),
Int8Array.of(127, 126, 125));
assert.deepEqual(
Int8Array.from(Int16Array.of(254, 252, 250).map(x => x / 2)),
Int8Array.of(127, 126, 125)); // OK
assert.deepEqual(
Int8Array.from(Int16Array.of(254, 252, 250)).map(x => x / 2),
Int8Array.of(-1, -2, -3)); // wrong
The problem is that, if we map via .map()
, then input type and output type are the same (if we work with Typed Arrays). In contrast, .from()
goes from an arbitrary input type to an output type that you specify via its receiver.
According to Allen Wirfs-Brock, mapping between Typed Arrays was what motivated the mapfn
parameter of .from()
.
29.4.6. Properties of TypedArray<T>.prototype
Indices accepted by Typed Array methods can be negative (they work like traditional Array methods that way). Offsets must be non-negative. For details, see section “Indices and offsets”.
29.4.6.1. Properties specific to Typed Arrays
The following properties are specific to Typed Arrays; normal Arrays don’t have them:
get .buffer(): ArrayBuffer
Returns the buffer backing this Typed Array.
get .length(): number
Returns the length in elements of this Typed Array’s buffer. Note that the length of normal Arrays is not a getter, it is a special property that instances have.
get .byteLength(): number
Returns the size in bytes of this Typed Array’s buffer.
get .byteOffset(): number
Returns the offset where this Typed Array “starts” inside its ArrayBuffer.
.set(arrayLike: ArrayLike<number>, offset=0): void
.set(typedArray: TypedArray, offset=0): void
Copies all elements of the first parameter to this Typed Array. The element at index 0 of the parameter is written to indexoffset
of this Typed Array (etc.).- First parameter is
arrayLike
: its elements are converted to numbers, which are then converted to the element typeT
of this Typed Array. - First parameter is
typedArray
: each of its elements is converted directly to the appropriate type for this Typed Array. If both Typed Arrays have the same element type then faster, byte-wise copying is used.
- First parameter is
.subarray(startIndex=0, end=this.length): TypedArray<T>
Returns a new Typed Array that has the same buffer as this Typed Array, but a (generally) smaller range. If startIndex
is non-negative then the first element of the resulting Typed Array is this[startIndex]
, the second this[startIndex+1]
(etc.). If startIndex
in negative, it is converted appropriately.
29.4.6.2. Array methods
The following methods are basically the same as the methods of normal Arrays:
.copyWithin(target: number, start: number, end=this.length): this
[W, ES6].entries(): Iterable<[number, T]>
[R, ES6].every(callback: (value: T, index: number, array: TypedArray<T>) => boolean, thisArg?: any): boolean
[R, ES5].fill(value: T, start=0, end=this.length): this
[W, ES6].filter(callback: (value: T, index: number, array: TypedArray<T>) => any, thisArg?: any): T[]
[R, ES5].find(predicate: (value: T, index: number, obj: T[]) => boolean, thisArg?: any): T | undefined
[R, ES6].findIndex(predicate: (value: T, index: number, obj: T[]) => boolean, thisArg?: any): number
[R, ES6].forEach(callback: (value: T, index: number, array: TypedArray<T>) => void, thisArg?: any): void
[R, ES5].includes(searchElement: T, fromIndex=0): boolean
[R, ES2016].indexOf(searchElement: T, fromIndex=0): number
[R, ES5].join(separator = ','): string
[R, ES1].keys(): Iterable<number>
[R, ES6].lastIndexOf(searchElement: T, fromIndex=this.length-1): number
[R, ES5].map<U>(mapFunc: (value: T, index: number, array: TypedArray<T>) => U, thisArg?: any): U[]
[R, ES5].reduce<U>(callback: (accumulator: U, element: T, index: number, array: T[]) => U, init?: U): U
[R, ES5].reduceRight<U>(callback: (accumulator: U, element: T, index: number, array: T[]) => U, init?: U): U
[R, ES5].reverse(): this
[W, ES1].slice(start=0, end=this.length): T[]
[R, ES3].some(callback: (value: T, index: number, array: TypedArray<T>) => boolean, thisArg?: any): boolean
[R, ES5].sort(compareFunc?: (a: T, b: T) => number): this
[W, ES1].toString(): string
[R, ES1].values(): Iterable<number>
[R, ES6]
For details on how these methods work, please consult the chapter on normal Arrays.
29.4.7. new «ElementType»Array()
Each Typed Array constructor has a name that follows the pattern «ElementType»Array
, where «ElementType»
is one of the element types in the table at the beginning. That means that there are 9 constructors for Typed Arrays: Int8Array
, Uint8Array
, Uint8ClampedArray
(element type Uint8C
), Int16Array
, Uint16Array
, Int32Array
, Uint32Array
, Float32Array
, Float64Array
.
Each constructor has four overloaded versions – it behaves differently depending on how many arguments it receives and what their types are:
new «ElementType»Array(buffer: ArrayBuffer, byteOffset=0, length=0)
Creates a new «ElementType»Array
whose buffer is buffer
. It starts accessing the buffer at the given byteOffset
and will have the given length
. Note that length
counts elements of the Typed Array (with 1–4 bytes each), not bytes.
new «ElementType»Array(length=0)
Creates a new «ElementType»Array
with the given length
and the appropriate buffer (whose size in bytes is length * «ElementType»Array.BYTES_PER_ELEMENT
).
new «ElementType»Array(source: TypedArray)
Creates a new instance of «ElementType»Array
whose elements have the same values as the elements of source
, but coerced to ElementType
.
new «ElementType»Array(source: ArrayLike<number>)
Creates a new instance of «ElementType»Array
whose elements have the same values as the elements of source
, but coerced to ElementType
. (For more information on Array-like objects, consult the chapter on Arrays.)
The following code shows three different ways of creating the same Typed Array:
29.4.8. Static properties of «ElementType»Array
«ElementType»Array.BYTES_PER_ELEMENT: number
Counts how many bytes are needed to store a single element:
29.4.9. Properties of «ElementType»Array.prototype
.BYTES_PER_ELEMENT: number
The same as «ElementType»Array.BYTES_PER_ELEMENT
.
29.4.10. Concatenating Typed Arrays
Typed Arrays don’t have a method .concat()
, like normal Arrays do. The work-around is to use the method
That method copies an existing Typed Array into typedArray
at index offset
. Then you only have to make sure that typedArray
is big enough to hold all (Typed) Arrays you want to concatenate:
function concatenate(resultConstructor, ...arrays) {
let totalLength = 0;
for (const arr of arrays) {
totalLength += arr.length;
}
const result = new resultConstructor(totalLength);
let offset = 0;
for (const arr of arrays) {
result.set(arr, offset);
offset += arr.length;
}
return result;
}
assert.deepEqual(
concatenate(Uint8Array,
Uint8Array.of(1, 2), [3, 4]),
Uint8Array.of(1, 2, 3, 4)
);
29.5. DataViews
29.5.1. new DataView()
new DataView(buffer: ArrayBuffer, byteOffset=0, byteLength=buffer.byteLength-byteOffset)
Creates a new DataView whose data is stored in the ArrayBuffer buffer
. By default, the new DataView can access all of buffer
, the last two parameters allow you to change that.
29.5.2. Properties of DataView.prototype
«ElementType»
can be: Float32
, Float64
, Int8
, Int16
, Int32
, Uint8
, Uint16
, Uint32
.
get .buffer()
Returns the ArrayBuffer of this DataView.
get .byteLength()
Returns how many bytes can be accessed by this DataView.
get .byteOffset()
Returns at which offset this DataView starts accessing the bytes in its buffer.
.get«ElementType»(byteOffset: number, littleEndian=false)
Reads a value from the buffer of this DataView.
.set«ElementType»(byteOffset: number, value: number, littleEndian=false)
Writes value
to the buffer of this DataView.
29.6. Further reading
The chapter on Typed Arrays in “Exploring ES6” has some additional content:
- More details on browser APIs that support Typed Arrays
- A real-world example
- And a few more technical details.