Set Operations

Batch Streaming

UNION

UNION and UNION ALL return the rows that are found in either table. UNION takes only distinct rows while UNION ALL does not remove duplicates from the result rows.

  1. Flink SQL> create view t1(s) as values ('c'), ('a'), ('b'), ('b'), ('c');
  2. Flink SQL> create view t2(s) as values ('d'), ('e'), ('a'), ('b'), ('b');
  4. Flink SQL> (SELECT s FROM t1) UNION (SELECT s FROM t2);
  5. +---+
  6. |  s|
  7. +---+
  8. |  c|
  9. |  a|
  10. |  b|
  11. |  d|
  12. |  e|
  13. +---+
  15. Flink SQL> (SELECT s FROM t1) UNION ALL (SELECT s FROM t2);
  16. +---+
  17. |  c|
  18. +---+
  19. |  c|
  20. |  a|
  21. |  b|
  22. |  b|
  23. |  c|
  24. |  d|
  25. |  e|
  26. |  a|
  27. |  b|
  28. |  b|
  29. +---+

INTERSECT

INTERSECT and INTERSECT ALL return the rows that are found in both tables. INTERSECT takes only distinct rows while INTERSECT ALL does not remove duplicates from the result rows.

  1. Flink SQL> (SELECT s FROM t1) INTERSECT (SELECT s FROM t2);
  2. +---+
  3. |  s|
  4. +---+
  5. |  a|
  6. |  b|
  7. +---+
  9. Flink SQL> (SELECT s FROM t1) INTERSECT ALL (SELECT s FROM t2);
  10. +---+
  11. |  s|
  12. +---+
  13. |  a|
  14. |  b|
  15. |  b|
  16. +---+

EXCEPT

EXCEPT and EXCEPT ALL return the rows that are found in one table but not the other. EXCEPT takes only distinct rows while EXCEPT ALL does not remove duplicates from the result rows.

  1. Flink SQL> (SELECT s FROM t1) EXCEPT (SELECT s FROM t2);
  2. +---+
  3. | s |
  4. +---+
  5. | c |
  6. +---+
  8. Flink SQL> (SELECT s FROM t1) EXCEPT ALL (SELECT s FROM t2);
  9. +---+
  10. | s |
  11. +---+
  12. | c |
  13. | c |
  14. +---+

IN

Returns true if an expression exists in a given table sub-query. The sub-query table must consist of one column. This column must have the same data type as the expression.

  1. SELECT user, amount
  2. FROM Orders
  3. WHERE product IN (
  4.     SELECT product FROM NewProducts
  5. )

The optimizer rewrites the IN condition into a join and group operation. For streaming queries, the required state for computing the query result might grow infinitely depending on the number of distinct input rows. You can provide a query configuration with an appropriate state time-to-live (TTL) to prevent excessive state size. Note that this might affect the correctness of the query result. See query configuration for details.

EXISTS

  1. SELECT user, amount
  2. FROM Orders
  3. WHERE product EXISTS (
  4.     SELECT product FROM NewProducts
  5. )

Returns true if the sub-query returns at least one row. Only supported if the operation can be rewritten in a join and group operation.

The optimizer rewrites the EXISTS operation into a join and group operation. For streaming queries, the required state for computing the query result might grow infinitely depending on the number of distinct input rows. You can provide a query configuration with an appropriate state time-to-live (TTL) to prevent excessive state size. Note that this might affect the correctness of the query result. See query configuration for details.