lzma —- 用 LZMA 算法压缩

3.3 新版功能.

源代码：Lib/lzma.py

This module provides classes and convenience functions for compressing anddecompressing data using the LZMA compression algorithm. Also included is a fileinterface supporting the .xz and legacy .lzma file formats used by thexz utility, as well as raw compressed streams.

The interface provided by this module is very similar to that of the bz2module. However, note that LZMAFile is not thread-safe, unlikebz2.BZ2File, so if you need to use a single LZMAFile instancefrom multiple threads, it is necessary to protect it with a lock.

  • exception lzma.LZMAError
  • This exception is raised when an error occurs during compression ordecompression, or while initializing the compressor/decompressor state.

读写压缩文件

  • lzma.open(filename, mode="rb", *, format=None, check=-1, preset=None, filters=None, encoding=None, errors=None, newline=None)
  • Open an LZMA-compressed file in binary or text mode, returning a fileobject.

The filename argument can be either an actual file name (given as astr, bytes or path-like object), inwhich case the named file is opened, or it can be an existing file objectto read from or write to.

The mode argument can be any of "r", "rb", "w", "wb","x", "xb", "a" or "ab" for binary mode, or "rt","wt", "xt", or "at" for text mode. The default is "rb".

When opening a file for reading, the format and filters arguments havethe same meanings as for LZMADecompressor. In this case, the check_and _preset arguments should not be used.

When opening a file for writing, the format, check, preset andfilters arguments have the same meanings as for LZMACompressor.

For binary mode, this function is equivalent to the LZMAFileconstructor: LZMAFile(filename, mode, …). In this case, the encoding,errors and newline arguments must not be provided.

For text mode, a LZMAFile object is created, and wrapped in anio.TextIOWrapper instance with the specified encoding, errorhandling behavior, and line ending(s).

在 3.4 版更改: Added support for the "x", "xb" and "xt" modes.

在 3.6 版更改: 接受一个 path-like object。

  • class lzma.LZMAFile(filename=None, mode="r", *, format=None, check=-1, preset=None, filters=None)
  • Open an LZMA-compressed file in binary mode.

An LZMAFile can wrap an already-open file object, or operatedirectly on a named file. The filename argument specifies either the fileobject to wrap, or the name of the file to open (as a str,bytes or path-like object). When wrapping anexisting file object, the wrapped file will not be closed when theLZMAFile is closed.

The mode argument can be either "r" for reading (default), "w" foroverwriting, "x" for exclusive creation, or "a" for appending. Thesecan equivalently be given as "rb", "wb", "xb" and "ab"respectively.

If filename is a file object (rather than an actual file name), a mode of"w" does not truncate the file, and is instead equivalent to "a".

When opening a file for reading, the input file may be the concatenation ofmultiple separate compressed streams. These are transparently decoded as asingle logical stream.

When opening a file for reading, the format and filters arguments havethe same meanings as for LZMADecompressor. In this case, the check_and _preset arguments should not be used.

When opening a file for writing, the format, check, preset andfilters arguments have the same meanings as for LZMACompressor.

LZMAFile supports all the members specified byio.BufferedIOBase, except for detach() and truncate().Iteration and the with statement are supported.

也提供以下方法：

  • peek(size=-1)
  • Return buffered data without advancing the file position. At least onebyte of data will be returned, unless EOF has been reached. The exactnumber of bytes returned is unspecified (the size argument is ignored).

注解

While calling peek() does not change the file position ofthe LZMAFile, it may change the position of the underlyingfile object (e.g. if the LZMAFile was constructed by passing afile object for filename).

在 3.4 版更改: Added support for the "x" and "xb" modes.

在 3.5 版更改: The read() method now accepts an argument ofNone.

在 3.6 版更改: 接受一个 path-like object。

Compressing and decompressing data in memory

  • class lzma.LZMACompressor(format=FORMAT_XZ, check=-1, preset=None, filters=None)
  • Create a compressor object, which can be used to compress data incrementally.

For a more convenient way of compressing a single chunk of data, seecompress().

The format argument specifies what container format should be used.Possible values are:

    • FORMAT_XZ: The .xz container format.
    • 这是默认格式。
    • FORMAT_ALONE: The legacy .lzma container format.
    • This format is more limited than .xz — it does not support integritychecks or multiple filters.
    • FORMAT_RAW: A raw data stream, not using any container format.
    • This format specifier does not support integrity checks, and requires thatyou always specify a custom filter chain (for both compression anddecompression). Additionally, data compressed in this manner cannot bedecompressed using FORMAT_AUTO (see LZMADecompressor).

The check argument specifies the type of integrity check to include in thecompressed data. This check is used when decompressing, to ensure that thedata has not been corrupted. Possible values are:

  • CHECK_NONE: No integrity check.This is the default (and the only acceptable value) forFORMAT_ALONE and FORMAT_RAW.

  • CHECK_CRC32: 32-bit Cyclic Redundancy Check.

  • CHECK_CRC64: 64-bit Cyclic Redundancy Check.This is the default for FORMAT_XZ.

  • CHECK_SHA256: 256-bit Secure Hash Algorithm.

If the specified check is not supported, an LZMAError is raised.

The compression settings can be specified either as a preset compressionlevel (with the preset argument), or in detail as a custom filter chain(with the filters argument).

The preset argument (if provided) should be an integer between 0 and9 (inclusive), optionally OR-ed with the constantPRESETEXTREME. If neither _preset nor filters are given, thedefault behavior is to use PRESET_DEFAULT (preset level 6).Higher presets produce smaller output, but make the compression processslower.

注解

In addition to being more CPU-intensive, compression with higher presetsalso requires much more memory (and produces output that needs more memoryto decompress). With preset 9 for example, the overhead for anLZMACompressor object can be as high as 800 MiB. For this reason,it is generally best to stick with the default preset.

The filters argument (if provided) should be a filter chain specifier.See Specifying custom filter chains for details.

  • compress(data)

  • Compress data (a bytes object), returning a bytesobject containing compressed data for at least part of the input. Some ofdata may be buffered internally, for use in later calls tocompress() and flush(). The returned data should beconcatenated with the output of any previous calls to compress().

  • flush()

  • Finish the compression process, returning a bytes objectcontaining any data stored in the compressor's internal buffers.

The compressor cannot be used after this method has been called.

  • class lzma.LZMADecompressor(format=FORMAT_AUTO, memlimit=None, filters=None)
  • Create a decompressor object, which can be used to decompress dataincrementally.

For a more convenient way of decompressing an entire compressed stream atonce, see decompress().

The format argument specifies the container format that should be used. Thedefault is FORMAT_AUTO, which can decompress both .xz and.lzma files. Other possible values are FORMAT_XZ,FORMAT_ALONE, and FORMAT_RAW.

The memlimit argument specifies a limit (in bytes) on the amount of memorythat the decompressor can use. When this argument is used, decompression willfail with an LZMAError if it is not possible to decompress the inputwithin the given memory limit.

The filters argument specifies the filter chain that was used to createthe stream being decompressed. This argument is required if format isFORMAT_RAW, but should not be used for other formats.See Specifying custom filter chains for more information about filter chains.

注解

This class does not transparently handle inputs containing multiplecompressed streams, unlike decompress() and LZMAFile. Todecompress a multi-stream input with LZMADecompressor, you mustcreate a new decompressor for each stream.

  • decompress(data, max_length=-1)
  • Decompress data (a bytes-like object), returninguncompressed data as bytes. Some of data may be bufferedinternally, for use in later calls to decompress(). Thereturned data should be concatenated with the output of anyprevious calls to decompress().

If max_length is nonnegative, returns at most max_length_bytes of decompressed data. If this limit is reached and furtheroutput can be produced, the needs_input attribute willbe set to False. In this case, the next call todecompress() may provide _data as b'' to obtainmore of the output.

If all of the input data was decompressed and returned (eitherbecause this was less than max_length bytes, or becausemax_length was negative), the needs_input attributewill be set to True.

Attempting to decompress data after the end of stream is reachedraises an EOFError. Any data found after the end of thestream is ignored and saved in the unused_data attribute.

在 3.5 版更改: Added the max_length parameter.

  • check

  • The ID of the integrity check used by the input stream. This may beCHECK_UNKNOWN until enough of the input has been decoded todetermine what integrity check it uses.

  • eof

  • 若达到了数据流末尾标识符则为 True。

  • unused_data

  • 压缩数据流的末尾还有数据。

Before the end of the stream is reached, this will be b"".

  • needs_input
  • False if the decompress() method can provide moredecompressed data before requiring new uncompressed input.

3.5 新版功能.

  • lzma.compress(data, format=FORMAT_XZ, check=-1, preset=None, filters=None)
  • Compress data (a bytes object), returning the compressed data as abytes object.

See LZMACompressor above for a description of the format, check,preset and filters arguments.

  • lzma.decompress(data, format=FORMAT_AUTO, memlimit=None, filters=None)
  • Decompress data (a bytes object), returning the uncompressed dataas a bytes object.

If data is the concatenation of multiple distinct compressed streams,decompress all of these streams, and return the concatenation of the results.

See LZMADecompressor above for a description of the format,memlimit and filters arguments.

杂项

  • lzma.ischeck_supported(_check)
  • Return True if the given integrity check is supported on this system.

CHECK_NONE and CHECK_CRC32 are always supported.CHECK_CRC64 and CHECK_SHA256 may be unavailable if you areusing a version of liblzma that was compiled with a limitedfeature set.

Specifying custom filter chains

A filter chain specifier is a sequence of dictionaries, where each dictionarycontains the ID and options for a single filter. Each dictionary must containthe key "id", and may contain additional keys to specify filter-dependentoptions. Valid filter IDs are as follows:

    • Compression filters:

      • FILTER_LZMA1 (for use with FORMAT_ALONE)

      • FILTER_LZMA2 (for use with FORMAT_XZ and FORMAT_RAW)

    • Delta filter:
      • FILTER_DELTA
    • Branch-Call-Jump (BCJ) filters:

      • FILTER_X86

      • FILTER_IA64

      • FILTER_ARM

      • FILTER_ARMTHUMB

      • FILTER_POWERPC

      • FILTER_SPARC

A filter chain can consist of up to 4 filters, and cannot be empty. The lastfilter in the chain must be a compression filter, and any other filters must bedelta or BCJ filters.

Compression filters support the following options (specified as additionalentries in the dictionary representing the filter):

  • preset: A compression preset to use as a source of default values foroptions that are not specified explicitly.

  • dict_size: Dictionary size in bytes. This should be between 4 KiB and1.5 GiB (inclusive).

  • lc: Number of literal context bits.

  • lp: Number of literal position bits. The sum lc + lp must be atmost 4.

  • pb: Number of position bits; must be at most 4.

  • mode: MODE_FAST or MODE_NORMAL.

  • nice_len: What should be considered a "nice length" for a match.This should be 273 or less.

  • mf: What match finder to use — MF_HC3, MF_HC4,MF_BT2, MF_BT3, or MF_BT4.

  • depth: Maximum search depth used by match finder. 0 (default) means toselect automatically based on other filter options.

The delta filter stores the differences between bytes, producing more repetitiveinput for the compressor in certain circumstances. It supports one option,dist. This indicates the distance between bytes to be subtracted. Thedefault is 1, i.e. take the differences between adjacent bytes.

The BCJ filters are intended to be applied to machine code. They convertrelative branches, calls and jumps in the code to use absolute addressing, withthe aim of increasing the redundancy that can be exploited by the compressor.These filters support one option, start_offset. This specifies the addressthat should be mapped to the beginning of the input data. The default is 0.

例子

Reading in a compressed file:

  1. import lzma
  2. with lzma.open("file.xz") as f:
  3.     file_content = f.read()

创建一个压缩文件:

  1. import lzma
  2. data = b"Insert Data Here"
  3. with lzma.open("file.xz", "w") as f:
  4.     f.write(data)

在内存中压缩文件:

  1. import lzma
  2. data_in = b"Insert Data Here"
  3. data_out = lzma.compress(data_in)

Incremental compression:

  1. import lzma
  2. lzc = lzma.LZMACompressor()
  3. out1 = lzc.compress(b"Some data\n")
  4. out2 = lzc.compress(b"Another piece of data\n")
  5. out3 = lzc.compress(b"Even more data\n")
  6. out4 = lzc.flush()
  7. # Concatenate all the partial results:
  8. result = b"".join([out1, out2, out3, out4])

Writing compressed data to an already-open file:

  1. import lzma
  2. with open("file.xz", "wb") as f:
  3.     f.write(b"This data will not be compressed\n")
  4.     with lzma.open(f, "w") as lzf:
  5.         lzf.write(b"This *will* be compressed\n")
  6.     f.write(b"Not compressed\n")

Creating a compressed file using a custom filter chain:

  1. import lzma
  2. my_filters = [
  3.     {"id": lzma.FILTER_DELTA, "dist": 5},
  4.     {"id": lzma.FILTER_LZMA2, "preset": 7 | lzma.PRESET_EXTREME},
  5. ]
  6. with lzma.open("file.xz", "w", filters=my_filters) as f:
  7.     f.write(b"blah blah blah")