LIVE_PROGRESS Query Option (Impala 2.3 or higher only)

In Impala 3.4.0 and higher when you start the Impala shell in interactive mode, the LIVE_PROGRESS query option is set to TRUE by default. When you run a query, the live progress bar appears in the output of a query. The bar shows roughly the percentage of completed processing. When the query finishes, the live progress bar disappears from the console output.

Starting in Impala 3.1, the summary output also includes the queuing status consisting of whether the query was queued and what was the latest queuing reason.

Type: Boolean

Default: TRUE (0)

Command-line equivalent:

You can turn off LIVE_PROGRESS within the shell using the SET command. You can also turn off LIVE_PROGRESS by starting the shell with the --disable_live_progress command-line option.

Usage notes:

The output from this query option is printed to standard error. The output is displayed only in interactive mode, not when the -q or -f options are used.

For tracking the progress of an interactive query through all phases of processing, see LIVE_SUMMARY Query Option (Impala 2.3 or higher only).

Restrictions:

Because the percentage complete figure is calculated using the number of issued and completed “scan ranges”, which occur while reading the table data, the progress bar might reach 100% before the query is entirely finished. For example, the query might do work to perform aggregations after all the table data has been read. If many of your queries fall into this category, consider using the LIVE_SUMMARY option for more granular progress reporting.

The LIVE_PROGRESS and LIVE_SUMMARY query options currently do not produce any output during COMPUTE STATS operations.

Because the LIVE_PROGRESS and LIVE_SUMMARY query options are available only within the impala-shell interpreter:

  • You cannot change these query options through the SQL SET statement using the JDBC or ODBC interfaces. The SET command in impala-shell recognizes these names as shell-only options.

  • Be careful when using impala-shell on a pre-Impala 2.3 system to connect to a system running Impala 2.3 or higher. The older impala-shell does not recognize these query option names. Upgrade impala-shell on the systems where you intend to use these query options.

  • Likewise, the impala-shell command relies on some information only available in Impala 2.3 and higher to prepare live progress reports and query summaries. The LIVE_PROGRESS and LIVE_SUMMARY query options have no effect when impala-shell connects to a cluster running an older version of Impala.

Added in: Impala 2.3.0

Examples:

  1. [localhost:21000] > select count(*) from customer;
  2. +----------+
  3. | count(*) |
  4. +----------+
  5. | 150000   |
  6. +----------+
  7. [localhost:21000] > select count(*) from customer t1 cross join customer t2;
  8. [###################################                                   ] 50%
  9. [######################################################################] 100%

To see how the LIVE_PROGRESS and LIVE_SUMMARY query options work in real time, see this animated demo.

Parent topic: Query Options for the SET Statement