MAP Complex Type (Impala 2.3 or higher only)

A complex data type representing an arbitrary set of key-value pairs. The key part is a scalar type, while the value part can be a scalar or another complex type (ARRAY, STRUCT, or MAP).

Syntax:

  1. column_name MAP < primitive_type, type >
  3. type ::= primitive_type | complex_type

Usage notes:

Because complex types are often used in combination, for example an ARRAY of STRUCT elements, if you are unfamiliar with the Impala complex types, start with Complex Types (Impala 2.3 or higher only) for background information and usage examples.

The MAP complex data type represents a set of key-value pairs. Each element of the map is indexed by a primitive type such as BIGINT or STRING, letting you define sequences that are not continuous or categories with arbitrary names. You might find it convenient for modelling data produced in other languages, such as a Python dictionary or Java HashMap, where a single scalar value serves as the lookup key.

In a big data context, the keys in a map column might represent a numeric sequence of events during a manufacturing process, or TIMESTAMP values corresponding to sensor observations. The map itself is inherently unordered, so you choose whether to make the key values significant (such as a recorded TIMESTAMP) or synthetic (such as a random global universal ID).

Note: Behind the scenes, the MAP type is implemented in a similar way as the ARRAY type. Impala does not enforce any uniqueness constraint on the KEY values, and the KEY values are processed by looping through the elements of the MAP rather than by a constant-time lookup. Therefore, this type is primarily for ease of understanding when importing data and algorithms from non-SQL contexts, rather than optimizing the performance of key lookups.

You can pass a multi-part qualified name to DESCRIBE to specify an ARRAY, STRUCT, or MAP column and visualize its structure as if it were a table. For example, if table T1 contains an ARRAY column A1, you could issue the statement DESCRIBE t1.a1. If table T1 contained a STRUCT column S1, and a field F1 within the STRUCT was a MAP, you could issue the statement DESCRIBE t1.s1.f1. An ARRAY is shown as a two-column table, with ITEM and POS columns. A STRUCT is shown as a table with each field representing a column in the table. A MAP is shown as a two-column table, with KEY and VALUE columns.

Added in: Impala 2.3.0

Restrictions:

  • Columns with this data type can only be used in tables or partitions with the Parquet file format.

  • Columns with this data type cannot be used as partition key columns in a partitioned table.

  • The COMPUTE STATS statement does not produce any statistics for columns of this data type.

  • The maximum length of the column definition for any complex type, including declarations for any nested types, is 4000 characters.

  • See Limitations and Restrictions for Complex Types for a full list of limitations and associated guidelines about complex type columns.

Kudu considerations:

Currently, the data types CHAR, VARCHAR, ARRAY, MAP, and STRUCT cannot be used with Kudu tables.

Examples:

Note: Many of the complex type examples refer to tables such as CUSTOMER and REGION adapted from the tables used in the TPC-H benchmark. See Sample Schema and Data for Experimenting with Impala Complex Types for the table definitions.

The following example shows a table with various kinds of MAP columns, both at the top level and nested within other complex types. Each row represents information about a specific country, with complex type fields of various levels of nesting to represent different information associated with the country: factual measurements such as area and population, notable people in different categories, geographic features such as cities, points of interest within each city, and mountains with associated facts. Practice the CREATE TABLE and query notation for complex type columns using empty tables, until you can visualize a complex data structure and construct corresponding SQL statements reliably.

  1. create TABLE map_demo
  2. (
  3.   country_id BIGINT,
  5. -- Numeric facts about each country, looked up by name.
  6. -- For example, 'Area':1000, 'Population':999999.
  7. -- Using a MAP instead of a STRUCT because there could be
  8. -- a different set of facts for each country.
  9.   metrics MAP <STRING, BIGINT>,
  11. -- MAP whose value part is an ARRAY.
  12. -- For example, the key 'Famous Politicians' could represent an array of 10 elements,
  13. -- while the key 'Famous Actors' could represent an array of 20 elements.
  14.   notables MAP <STRING, ARRAY <STRING>>,
  16. -- MAP that is a field within a STRUCT.
  17. -- (The STRUCT is inside another ARRAY, because it is rare
  18. -- for a STRUCT to be a top-level column.)
  19. -- For example, city #1 might have points of interest with key 'Zoo',
  20. -- representing an array of 3 different zoos.
  21. -- City #2 might have completely different kinds of points of interest.
  22. -- Because the set of field names is potentially large, and most entries could be blank,
  23. -- a MAP makes more sense than a STRUCT to represent such a sparse data structure.
  24.   cities ARRAY < STRUCT <
  25.     name: STRING,
  26.     points_of_interest: MAP <STRING, ARRAY <STRING>>
  27.   >>,
  29. -- MAP that is an element within an ARRAY. The MAP is inside a STRUCT field to associate
  30. -- the mountain name with all the facts about the mountain.
  31. -- The "key" of the map (the first STRING field) represents the name of some fact whose value
  32. -- can be expressed as an integer, such as 'Height', 'Year First Climbed', and so on.
  33.   mountains ARRAY < STRUCT < name: STRING, facts: MAP <STRING, INT > > >
  34. )
  35. STORED AS PARQUET;
  1. DESCRIBE map_demo;
  2. +------------+------------------------------------------------+
  3. | name       | type                                           |
  4. +------------+------------------------------------------------+
  5. | country_id | bigint                                         |
  6. | metrics    | map<string,bigint>                             |
  7. | notables   | map<string,array<string>>                      |
  8. | cities     | array<struct<                                  |
  9. |            |   name:string,                                 |
  10. |            |   points_of_interest:map<string,array<string>> |
  11. |            | >>                                             |
  12. | mountains  | array<struct<                                  |
  13. |            |   name:string,                                 |
  14. |            |   facts:map<string,int>                        |
  15. |            | >>                                             |
  16. +------------+------------------------------------------------+
  18. DESCRIBE map_demo.metrics;
  19. +-------+--------+
  20. | name  | type   |
  21. +-------+--------+
  22. | key   | string |
  23. | value | bigint |
  24. +-------+--------+
  26. DESCRIBE map_demo.notables;
  27. +-------+---------------+
  28. | name  | type          |
  29. +-------+---------------+
  30. | key   | string        |
  31. | value | array<string> |
  32. +-------+---------------+
  34. DESCRIBE map_demo.notables.value;
  35. +------+--------+
  36. | name | type   |
  37. +------+--------+
  38. | item | string |
  39. | pos  | bigint |
  40. +------+--------+
  42. DESCRIBE map_demo.cities;
  43. +------+------------------------------------------------+
  44. | name | type                                           |
  45. +------+------------------------------------------------+
  46. | item | struct<                                        |
  47. |      |   name:string,                                 |
  48. |      |   points_of_interest:map<string,array<string>> |
  49. |      | >                                              |
  50. | pos  | bigint                                         |
  51. +------+------------------------------------------------+
  53. DESCRIBE map_demo.cities.item.points_of_interest;
  54. +-------+---------------+
  55. | name  | type          |
  56. +-------+---------------+
  57. | key   | string        |
  58. | value | array<string> |
  59. +-------+---------------+
  61. DESCRIBE map_demo.cities.item.points_of_interest.value;
  62. +------+--------+
  63. | name | type   |
  64. +------+--------+
  65. | item | string |
  66. | pos  | bigint |
  67. +------+--------+
  69. DESCRIBE map_demo.mountains;
  70. +------+-------------------------+
  71. | name | type                    |
  72. +------+-------------------------+
  73. | item | struct<                 |
  74. |      |   name:string,          |
  75. |      |   facts:map<string,int> |
  76. |      | >                       |
  77. | pos  | bigint                  |
  78. +------+-------------------------+
  80. DESCRIBE map_demo.mountains.item.facts;
  81. +-------+--------+
  82. | name  | type   |
  83. +-------+--------+
  84. | key   | string |
  85. | value | int    |
  86. +-------+--------+

The following example shows a table that uses a variety of data types for the MAP “key” field. Typically, you use BIGINT or STRING to use numeric or character-based keys without worrying about exceeding any size or length constraints.

  1. CREATE TABLE map_demo_obscure
  2. (
  3.   id BIGINT,
  4.   m1 MAP <INT, INT>,
  5.   m2 MAP <SMALLINT, INT>,
  6.   m3 MAP <TINYINT, INT>,
  7.   m4 MAP <TIMESTAMP, INT>,
  8.   m5 MAP <BOOLEAN, INT>,
  9.   m6 MAP <CHAR(5), INT>,
  10.   m7 MAP <VARCHAR(25), INT>,
  11.   m8 MAP <FLOAT, INT>,
  12.   m9 MAP <DOUBLE, INT>,
  13.   m10 MAP <DECIMAL(12,2), INT>
  14. )
  15. STORED AS PARQUET;
  1. CREATE TABLE celebrities (name STRING, birth_year MAP < STRING, SMALLINT >) STORED AS PARQUET;
  2. -- A typical row might represent values with 2 different birth years, such as:
  3. -- ("Joe Movie Star", { "real": 1972, "claimed": 1977 })
  5. CREATE TABLE countries (name STRING, famous_leaders MAP < INT, STRING >) STORED AS PARQUET;
  6. -- A typical row might represent values with different leaders, with key values corresponding to their numeric sequence, such as:
  7. -- ("United States", { 1: "George Washington", 3: "Thomas Jefferson", 16: "Abraham Lincoln" })
  9. CREATE TABLE airlines (name STRING, special_meals MAP < STRING, MAP < STRING, STRING > >) STORED AS PARQUET;
  10. -- A typical row might represent values with multiple kinds of meals, each with several components:
  11. -- ("Elegant Airlines",
  12. --   {
  13. --     "vegetarian": { "breakfast": "pancakes", "snack": "cookies", "dinner": "rice pilaf" },
  14. --     "gluten free": { "breakfast": "oatmeal", "snack": "fruit", "dinner": "chicken" }
  15. --   } )

Related information:

Complex Types (Impala 2.3 or higher only), ARRAY Complex Type (Impala 2.3 or higher only), STRUCT Complex Type (Impala 2.3 or higher only)

Parent topic: Data Types