Connecting to impalad through impala-shell

Within an impala-shell session, you can only issue queries while connected to an instance of the impalad daemon. You can specify the connection information:

  • Through command-line options when you run the impala-shell command.
  • Through a configuration file that is read when you run the impala-shell command.
  • During an impala-shell session, by issuing a CONNECT command.

See impala-shell Configuration Options for the command-line and configuration file options you can use.

You can connect to any DataNode where an instance of impalad is running, and that host coordinates the execution of all queries sent to it.

For simplicity during development, you might always connect to the same host, perhaps running impala-shell on the same host as impalad and specifying the hostname as localhost.

In a production environment, you might enable load balancing, in which you connect to specific host/port combination but queries are forwarded to arbitrary hosts. This technique spreads the overhead of acting as the coordinator node among all the DataNodes in the cluster. See Using Impala through a Proxy for High Availability for details.

To connect the Impala shell during shell startup:

  1. Locate the hostname of a DataNode within the cluster that is running an instance of the impalad daemon. If that DataNode uses a non-default port (something other than port 21000) for impala-shell connections, find out the port number also.

  2. Use the -i option to the impala-shell interpreter to specify the connection information for that instance of impalad:

    1. # When you are logged into the same machine running impalad.
    2. # The prompt will reflect the current hostname.
    3. $ impala-shell
    5. # When you are logged into the same machine running impalad.
    6. # The host will reflect the hostname 'localhost'.
    7. $ impala-shell -i localhost
    9. # When you are logged onto a different host, perhaps a client machine
    10. # outside the Hadoop cluster.
    11. $ impala-shell -i some.other.hostname
    13. # When you are logged onto a different host, and impalad is listening
    14. # on a non-default port. Perhaps a load balancer is forwarding requests
    15. # to a different host/port combination behind the scenes.
    16. $ impala-shell -i some.other.hostname:port_number

To connect the Impala shell after shell startup:

  1. Start the Impala shell with no connection:

    1. $ impala-shell

    You should see a prompt like the following:

    1. Welcome to the Impala shell. Press TAB twice to see a list of available commands.
    2. ...
    3. (Shell
    4.       build version: Impala Shell v3.0.x (hash) built on
    5.       date)
    6. [Not connected] >

  2. Locate the hostname of a DataNode within the cluster that is running an instance of the impalad daemon. If that DataNode uses a non-default port (something other than port 21000) for impala-shell connections, find out the port number also.

  3. Use the connect command to connect to an Impala instance. Enter a command of the form:

    1. [Not connected] > connect impalad-host
    2. [impalad-host:21000] >

    Note: Replace impalad-host with the hostname you have configured for any DataNode running Impala in your environment. The changed prompt indicates a successful connection.

To start impala-shell in a specific database:

You can use all the same connection options as in previous examples. For simplicity, these examples assume that you are logged into one of the DataNodes that is running the impalad daemon.

  1. Find the name of the database containing the relevant tables, views, and so on that you want to operate on.

  2. Use the -d option to the impala-shell interpreter to connect and immediately switch to the specified database, without the need for a USE statement or fully qualified names:

    1. # Subsequent queries with unqualified names operate on
    2. # tables, views, and so on inside the database named 'staging'.
    3. $ impala-shell -i localhost -d staging
    5. # It is common during development, ETL, benchmarking, and so on
    6. # to have different databases containing the same table names
    7. # but with different contents or layouts.
    8. $ impala-shell -i localhost -d parquet_snappy_compression
    9. $ impala-shell -i localhost -d parquet_gzip_compression

To run one or several statements in non-interactive mode:

You can use all the same connection options as in previous examples. For simplicity, these examples assume that you are logged into one of the DataNodes that is running the impalad daemon.

  1. Construct a statement, or a file containing a sequence of statements, that you want to run in an automated way, without typing or copying and pasting each time.

  2. Invoke impala-shell with the -q option to run a single statement, or the -f option to run a sequence of statements from a file. The impala-shell command returns immediately, without going into the interactive interpreter.

    1. # A utility command that you might run while developing shell scripts
    2. # to manipulate HDFS files.
    3. $ impala-shell -i localhost -d database_of_interest -q 'show tables'
    5. # A sequence of CREATE TABLE, CREATE VIEW, and similar DDL statements
    6. # can go into a file to make the setup process repeatable.
    7. $ impala-shell -i localhost -d database_of_interest -f recreate_tables.sql

Parent topic: Using the Impala Shell (impala-shell Command)