Managing Disk Space for Impala Data

Although Impala typically works with many large files in an HDFS storage system with plenty of capacity, there are times when you might perform some file cleanup to reclaim space, or advise developers on techniques to minimize space consumption and file duplication.

• Use compact binary file formats where practical. Numeric and time-based data in particular can be stored in more compact form in binary data files. Depending on the file format, various compression and encoding features can reduce file size even further. You can specify the STORED AS clause as part of the CREATE TABLE statement, or ALTER TABLE with the SET FILEFORMAT clause for an existing table or partition within a partitioned table. See How Impala Works with Hadoop File Formats for details about file formats, especially Using the Parquet File Format with Impala Tables. See CREATE TABLE Statement and ALTER TABLE Statement for syntax details.

• You manage underlying data files differently depending on whether the corresponding Impala table is defined as an internal or external table:

  • Use the DESCRIBE FORMATTED statement to check if a particular table is internal (managed by Impala) or external, and to see the physical location of the data files in HDFS. See DESCRIBE Statement for details.
  • For Impala-managed (“internal”) tables, use DROP TABLE statements to remove data files. See DROP TABLE Statement for details.
  • For tables not managed by Impala (“external” tables), use appropriate HDFS-related commands such as hadoop fs, hdfs dfs, or distcp, to create, move, copy, or delete files within HDFS directories that are accessible by the impala user. Issue a REFRESH table_name statement after adding or removing any files from the data directory of an external table. See REFRESH Statement for details.
  • Use external tables to reference HDFS data files in their original location. With this technique, you avoid copying the files, and you can map more than one Impala table to the same set of data files. When you drop the Impala table, the data files are left undisturbed. See External Tables for details.
  • Use the LOAD DATA statement to move HDFS files into the data directory for an Impala table from inside Impala, without the need to specify the HDFS path of the destination directory. This technique works for both internal and external tables. See LOAD DATA Statement for details.

• Make sure that the HDFS trashcan is configured correctly. When you remove files from HDFS, the space might not be reclaimed for use by other files until sometime later, when the trashcan is emptied. See DROP TABLE Statement for details. See User Account Requirements for permissions needed for the HDFS trashcan to operate correctly.

• Drop all tables in a database before dropping the database itself. See DROP DATABASE Statement for details.

• Clean up temporary files after failed INSERT statements. If an INSERT statement encounters an error, and you see a directory named .impala_insert_staging or _impala_insert_staging left behind in the data directory for the table, it might contain temporary data files taking up space in HDFS. You might be able to salvage these data files, for example if they are complete but could not be moved into place due to a permission error. Or, you might delete those files through commands such as hadoop fs or hdfs dfs, to reclaim space before re-trying the INSERT. Issue DESCRIBE FORMATTED table_name to see the HDFS path where you can check for temporary files.

• If you use the Amazon Simple Storage Service (S3) as a place to offload data to reduce the volume of local storage, Impala 2.2.0 and higher can query the data directly from S3. See Using Impala with Amazon S3 Object Store for details.

Configuring Scratch Space for Spilling to Disk

Impala uses intermediate files during large sort, join, aggregation, or analytic function operations The files are removed when the operation finishes. You can specify locations of the intermediate files by starting the impalad daemon with the ‑‑scratch_dirs="path_to_directory" configuration option. By default, intermediate files are stored in the directory /tmp/impala-scratch.

• You can specify a single directory or a comma-separated list of directories.

• You can specify an optional a capacity quota per scratch directory using the colon (:) as the delimiter.

  The capacity quota of -1 or 0 is the same as no quota for the directory.

• The scratch directories must be on the local filesystem, not in HDFS.

• You might specify different directory paths for different hosts, depending on the capacity and speed of the available storage devices.

If there is less than 1 GB free on the filesystem where that directory resides, Impala still runs, but writes a warning message to its log.

Impala successfully starts (with a warning written to the log) if it cannot create or read and write files in one of the scratch directories.

The following are examples for specifying scratch directories.

Allocation from a scratch directory will fail if the specified limit for the directory is exceeded.

If Impala encounters an error reading or writing files in a scratch directory during a query, Impala logs the error, and the query fails.

Increasing Scratch Capacity

You can compress the data spilled to disk to increase the effective scratch capacity. You typically more than double capacity using compression and reduce spilling to disk. Use the —disk_spill_compression_codec and –-disk_spill_punch_holes startup options. The —disk_spill_compression_codec takes any value supported by the COMPRESSION_CODEC query option. The value is not case-sensitive. A value of ZSTD or LZ4 is recommended (default is NONE).

For example:

--disk_spill_compression_codec=LZ4
--disk_spill_punch_holes=true

If you set --disk_spill_compression_codec to a value other than NONE, you must set --disk_spill_punch_holes to true.

The hole punching feature supported by many filesystems is used to reclaim space in scratch files during execution of a query that spills to disk. This results in lower scratch space requirements in many cases, especially when combined with disk spill compression. When this option is not enabled, scratch space is still recycled by a query, but less effectively in many cases.

You can specify a compression level for ZSTD only. For example:

--disk_spill_compression_codec=ZSTD:10
--disk_spill_punch_holes=true

Compression levels from 1 up to 22 (default 3) are supported for ZSTD. The lower the compression level, the faster the speed at the cost of compression ratio.

Parent topic: Impala Administration