v1.54 experimental

Compress

Warning

This remote is currently experimental. Things may break and data may be lost. Anything you do with this remote is at your own risk. Please understand the risks associated with using experimental code and don’t use this remote in critical applications.

The Compress remote adds compression to another remote. It is best used with remotes containing many large compressible files.

Configuration

To use this remote, all you need to do is specify another remote and a compression mode to use:

  1. Current remotes:
  3. Name                 Type
  4. ====                 ====
  5. remote_to_press      sometype
  7. e) Edit existing remote
  8. $ rclone config
  9. n) New remote
  10. d) Delete remote
  11. r) Rename remote
  12. c) Copy remote
  13. s) Set configuration password
  14. q) Quit config
  15. e/n/d/r/c/s/q> n
  16. name> compress
  17. ...
  18.  8 / Compress a remote
  19.    \ "compress"
  20. ...
  21. Storage> compress
  22. ** See help for compress backend at: https://rclone.org/compress/ **
  24. Remote to compress.
  25. Enter a string value. Press Enter for the default ("").
  26. remote> remote_to_press:subdir 
  27. Compression mode.
  28. Enter a string value. Press Enter for the default ("gzip").
  29. Choose a number from below, or type in your own value
  30.  1 / Gzip compression balanced for speed and compression strength.
  31.    \ "gzip"
  32. compression_mode> gzip
  33. Edit advanced config? (y/n)
  34. y) Yes
  35. n) No (default)
  36. y/n> n
  37. Remote config
  38. --------------------
  39. [compress]
  40. type = compress
  41. remote = remote_to_press:subdir
  42. compression_mode = gzip
  43. --------------------
  44. y) Yes this is OK (default)
  45. e) Edit this remote
  46. d) Delete this remote
  47. y/e/d> y

Compression Modes

Currently only gzip compression is supported. It provides a decent balance between speed and size and is well supported by other applications. Compression strength can further be configured via an advanced setting where 0 is no compression and 9 is strongest compression.

File types

If you open a remote wrapped by compress, you will see that there are many files with an extension corresponding to the compression algorithm you chose. These files are standard files that can be opened by various archive programs, but they have some hidden metadata that allows them to be used by rclone. While you may download and decompress these files at will, do not manually delete or rename files. Files without correct metadata files will not be recognized by rclone.

File names

The compressed files will be named *.###########.gz where * is the base file and the # part is base64 encoded size of the uncompressed file. The file names should not be changed by anything other than the rclone compression backend.

Standard options

Here are the Standard options specific to compress (Compress a remote).

--compress-remote

Remote to compress.

Properties:

  • Config: remote
  • Env Var: RCLONE_COMPRESS_REMOTE
  • Type: string
  • Required: true

--compress-mode

Compression mode.

Properties:

  • Config: mode
  • Env Var: RCLONE_COMPRESS_MODE
  • Type: string
  • Default: “gzip”
  • Examples:
    • “gzip”
      • Standard gzip compression with fastest parameters.

Advanced options

Here are the Advanced options specific to compress (Compress a remote).

--compress-level

GZIP compression level (-2 to 9).

Generally -1 (default, equivalent to 5) is recommended. Levels 1 to 9 increase compression at the cost of speed. Going past 6 generally offers very little return.

Level -2 uses Huffman encoding only. Only use if you know what you are doing. Level 0 turns off compression.

Properties:

  • Config: level
  • Env Var: RCLONE_COMPRESS_LEVEL
  • Type: int
  • Default: -1

--compress-ram-cache-limit

Some remotes don’t allow the upload of files with unknown size. In this case the compressed file will need to be cached to determine it’s size.

Files smaller than this limit will be cached in RAM, files larger than this limit will be cached on disk.

Properties:

  • Config: ram_cache_limit
  • Env Var: RCLONE_COMPRESS_RAM_CACHE_LIMIT
  • Type: SizeSuffix
  • Default: 20Mi

--compress-description

Description of the remote.

Properties:

  • Config: description
  • Env Var: RCLONE_COMPRESS_DESCRIPTION
  • Type: string
  • Required: false

Metadata

Any metadata supported by the underlying remote is read and written.

See the metadata docs for more info.