Troubleshooting Impala

Troubleshooting for Impala requires being able to diagnose and debug problems with performance, network connectivity, out-of-memory conditions, disk space usage, and crash or hang conditions in any of the Impala-related daemons.

The following sections describe the general troubleshooting procedures to diagnose different kinds of problems:

• Impala Web User Interface for Debugging

• Breakpad Minidumps for Impala (Impala 2.6 or higher only)

Troubleshooting Impala SQL Syntax Issues

In general, if queries issued against Impala fail, you can try running these same queries against Hive.

• If a query fails against both Impala and Hive, it is likely that there is a problem with your query or other elements of your environment:
  • Review the Language Reference to ensure your query is valid.
  • Check Impala Reserved Words to see if any database, table, column, or other object names in your query conflict with Impala reserved words. Quote those names with backticks (`` ) if so.
  • Check Impala Built-In Functions to confirm whether Impala supports all the built-in functions being used by your query, and whether argument and return types are the same as you expect.
  • Review the contents of the Impala logs for any information that may be useful in identifying the source of the problem.
• If a query fails against Impala but not Hive, it is likely that there is a problem with your Impala installation.

Troubleshooting Crashes Caused by Memory Resource Limit

Under very high concurrency, Impala could encounter a serious error due to usage of various operating system resources. Errors similar to the following may be caused by operating system resource exhaustion:

F0629 08:20:02.956413 29088 llvm-codegen.cc:111] LLVM hit fatal error: Unable to allocate section memory!
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::thread_resource_error> >'

The KRPC implementation in Impala 2.12 / 3.0 greatly reduces thread counts and the chances of hitting a resource limit.

If you still get an error similar to the above in Impala 3.0 and higher, try increasing the max_map_count OS virtual memory parameter. max_map_count defines the maximum number of memory map areas that a process can use. Configure each host running an impalad daemon with the command to increase max_map_count to 8 GB.

echo 8000000 > /proc/sys/vm/max_map_count

To make the above settings durable, refer to your OS documentation. For example, on RHEL 6.x:

1. Add the following line to /etc/sysctl.conf:

   vm.max_map_count=8000000

2. Run the following command:

   sysctl -p

Troubleshooting I/O Capacity Problems

Impala queries are typically I/O-intensive. If there is an I/O problem with storage devices, or with HDFS itself, Impala queries could show slow response times with no obvious cause on the Impala side. Slow I/O on even a single Impala daemon could result in an overall slowdown, because queries involving clauses such as ORDER BY, GROUP BY, or JOIN do not start returning results until all executor Impala daemons have finished their work.

To test whether the Linux I/O system itself is performing as expected, run Linux commands like the following on each host Impala daemon is running:

$ sudo sysctl -w vm.drop_caches=3 vm.drop_caches=0
vm.drop_caches = 3
vm.drop_caches = 0
$ sudo dd if=/dev/sda bs=1M of=/dev/null count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.60373 s, 192 MB/s
$ sudo dd if=/dev/sdb bs=1M of=/dev/null count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.51145 s, 195 MB/s
$ sudo dd if=/dev/sdc bs=1M of=/dev/null count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.58096 s, 192 MB/s
$ sudo dd if=/dev/sdd bs=1M of=/dev/null count=1k
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 5.43924 s, 197 MB/s

On modern hardware, a throughput rate of less than 100 MB/s typically indicates a performance issue with the storage device. Correct the hardware problem before continuing with Impala tuning or benchmarking.

Impala Troubleshooting Quick Reference

The following table lists common problems and potential solutions.