Group Aggregation

Batch Streaming

Like most data systems, Apache Flink supports aggregate functions; both built-in and user-defined. User-defined functions must be registered in a catalog before use.

An aggregate function computes a single result from multiple input rows. For example, there are aggregates to compute the COUNT, SUM, AVG (average), MAX (maximum) and MIN (minimum) over a set of rows.

  1. SELECT COUNT(*) FROM Orders

For streaming queries, it is important to understand that Flink runs continuous queries that never terminate. Instead, they update their result table according to the updates on its input tables. For the above query, Flink will output an updated count each time a new row is inserted into the Orders table.

Apache Flink supports the standard GROUP BY clause for aggregating data.

  1. SELECT COUNT(*)
  2. FROM Orders
  3. GROUP BY order_id

For streaming queries, the required state for computing the query result might grow infinitely. State size depends on the number of groups and the number and type of aggregation functions. For example MIN/MAX are heavy on state size while COUNT is cheap. You can provide a query configuration with an appropriate state time-to-live (TTL) to prevent excessive state size. Note that this might affect the correctness of the query result. See query configuration for details.

Apache Flink provides a set of performance tuning ways for Group Aggregation, see more Performance Tuning.

DISTINCT Aggregation

Distinct aggregates remove duplicate values before applying an aggregation function. The following example counts the number of distinct order_ids instead of the total number of rows in the Orders table.

  1. SELECT COUNT(DISTINCT order_id) FROM Orders

For streaming queries, the required state for computing the query result might grow infinitely. State size is mostly depends on the number of distinct rows and the time that a group is maintained, short lived group by windows are not a problem. You can provide a query configuration with an appropriate state time-to-live (TTL) to prevent excessive state size. Note that this might affect the correctness of the query result. See query configuration for details.

GROUPING SETS

Grouping sets allow for more complex grouping operations than those describable by a standard GROUP BY. Rows are grouped separately by each specified grouping set and aggregates are computed for each group just as for simple GROUP BY clauses.

  1. SELECT supplier_id, rating, COUNT(*) AS total
  2. FROM (VALUES
  3.     ('supplier1', 'product1', 4),
  4.     ('supplier1', 'product2', 3),
  5.     ('supplier2', 'product3', 3),
  6.     ('supplier2', 'product4', 4))
  7. AS Products(supplier_id, product_id, rating)
  8. GROUP BY GROUPING SETS ((supplier_id, rating), (supplier_id), ())

Results:

  1. +-------------+--------+-------+
  2. | supplier_id | rating | total |
  3. +-------------+--------+-------+
  4. |   supplier1 |      4 |     1 |
  5. |   supplier1 | (NULL) |     2 |
  6. |      (NULL) | (NULL) |     4 |
  7. |   supplier1 |      3 |     1 |
  8. |   supplier2 |      3 |     1 |
  9. |   supplier2 | (NULL) |     2 |
  10. |   supplier2 |      4 |     1 |
  11. +-------------+--------+-------+

Each sublist of GROUPING SETS may specify zero or more columns or expressions and is interpreted the same way as though it was used directly in the GROUP BY clause. An empty grouping set means that all rows are aggregated down to a single group, which is output even if no input rows were present.

References to the grouping columns or expressions are replaced by null values in result rows for grouping sets in which those columns do not appear.

For streaming queries, the required state for computing the query result might grow infinitely. State size depends on number of group sets and type of aggregation functions. You can provide a query configuration with an appropriate state time-to-live (TTL) to prevent excessive state size. Note that this might affect the correctness of the query result. See query configuration for details.

ROLLUP

ROLLUP is a shorthand notation for specifying a common type of grouping set. It represents the given list of expressions and all prefixes of the list, including the empty list.

For example, the following query is equivalent to the one above.

  1. SELECT supplier_id, rating, COUNT(*)
  2. FROM (VALUES
  3.     ('supplier1', 'product1', 4),
  4.     ('supplier1', 'product2', 3),
  5.     ('supplier2', 'product3', 3),
  6.     ('supplier2', 'product4', 4))
  7. AS Products(supplier_id, product_id, rating)
  8. GROUP BY ROLLUP (supplier_id, rating)

CUBE

CUBE is a shorthand notation for specifying a common type of grouping set. It represents the given list and all of its possible subsets - the power set.

For example, the following two queries are equivalent.

  1. SELECT supplier_id, rating, product_id, COUNT(*)
  2. FROM (VALUES
  3.     ('supplier1', 'product1', 4),
  4.     ('supplier1', 'product2', 3),
  5.     ('supplier2', 'product3', 3),
  6.     ('supplier2', 'product4', 4))
  7. AS Products(supplier_id, product_id, rating)
  8. GROUP BY CUBE (supplier_id, rating, product_id)
  10. SELECT supplier_id, rating, product_id, COUNT(*)
  11. FROM (VALUES
  12.     ('supplier1', 'product1', 4),
  13.     ('supplier1', 'product2', 3),
  14.     ('supplier2', 'product3', 3),
  15.     ('supplier2', 'product4', 4))
  16. AS Products(supplier_id, product_id, rating)
  17. GROUP BY GROUPING SET (
  18.     ( supplier_id, product_id, rating ),
  19.     ( supplier_id, product_id         ),
  20.     ( supplier_id,             rating ),
  21.     ( supplier_id                     ),
  22.     (              product_id, rating ),
  23.     (              product_id         ),
  24.     (                          rating ),
  25.     (                                 )
  26. )

HAVING

HAVING eliminates group rows that do not satisfy the condition. HAVING is different from WHERE: WHERE filters individual rows before the GROUP BY while HAVING filters group rows created by GROUP BY. Each column referenced in condition must unambiguously reference a grouping column unless it appears within an aggregate function.

  1. SELECT SUM(amount)
  2. FROM Orders
  3. GROUP BY users
  4. HAVING SUM(amount) > 50

The presence of HAVING turns a query into a grouped query even if there is no GROUP BY clause. It is the same as what happens when the query contains aggregate functions but no GROUP BY clause. The query considers all selected rows to form a single group, and the SELECT list and HAVING clause can only reference table columns from within aggregate functions. Such a query will emit a single row if the HAVING condition is true, zero rows if it is not true.