Seafile GC

Seafile uses storage de-duplication technology to reduce storage usage. The underlying data blocks will not be removed immediately after you delete a file or a library. As a result, the number of unused data blocks will increase on Seafile server.

To release the storage space occupied by unused blocks, you have to run a
“garbage collection” program to clean up unused blocks on your server.

The GC program cleans up two types of unused blocks:

  1. Blocks that no library references to, that is, the blocks belong to deleted libraries;
  2. If you set history length limit on some libraries, the out-dated blocks in those libraries will also be removed.

Before running GC, you must shutdown the Seafile program on your server if you use the community edition. For professional edition, from version 3.1.11, online GC operation is supported. If you use Professional edition, you don’t need to shutdown the Seafile program if you are using MySQL or PostgreSQL as database.
This is because new blocks written into Seafile while GC is running may be mistakenly deleted by the GC program.

Community version - GC cleanup Script

At the bottom of the page there is a script that you can use to run the cleanup manually or e.g. once a week with as cronjob.

Run GC in version 4.1.1 and later

In community edition 4.1.1 and Pro edition 4.1.0, GC program’s command line and output are improved.

Dry-run Mode

To see how much garbage can be collected without actually removing any garbage, use the dry-run option:

  1. seaf-gc.sh --dry-run [repo-id1] [repo-id2] ...

The output should look like:

  1. [03/19/15 19:41:49] seafserv-gc.c(115): GC version 1 repo My Library(ffa57d93)
  2. [03/19/15 19:41:49] gc-core.c(394): GC started. Total block number is 265.
  3. [03/19/15 19:41:49] gc-core.c(75): GC index size is 1024 Byte.
  4. [03/19/15 19:41:49] gc-core.c(408): Populating index.
  5. [03/19/15 19:41:49] gc-core.c(262): Populating index for repo ffa57d93.
  6. [03/19/15 19:41:49] gc-core.c(308): Traversed 5 commits, 265 blocks.
  7. [03/19/15 19:41:49] gc-core.c(440): Scanning unused blocks.
  8. [03/19/15 19:41:49] gc-core.c(472): GC finished. 265 blocks total, about 265 reachable blocks, 0 blocks can be removed.
  10. [03/19/15 19:41:49] seafserv-gc.c(115): GC version 1 repo aa(f3d0a8d0)
  11. [03/19/15 19:41:49] gc-core.c(394): GC started. Total block number is 5.
  12. [03/19/15 19:41:49] gc-core.c(75): GC index size is 1024 Byte.
  13. [03/19/15 19:41:49] gc-core.c(408): Populating index.
  14. [03/19/15 19:41:49] gc-core.c(262): Populating index for repo f3d0a8d0.
  15. [03/19/15 19:41:49] gc-core.c(308): Traversed 8 commits, 5 blocks.
  16. [03/19/15 19:41:49] gc-core.c(264): Populating index for sub-repo 9217622a.
  17. [03/19/15 19:41:49] gc-core.c(308): Traversed 4 commits, 4 blocks.
  18. [03/19/15 19:41:49] gc-core.c(440): Scanning unused blocks.
  19. [03/19/15 19:41:49] gc-core.c(472): GC finished. 5 blocks total, about 9 reachable blocks, 0 blocks can be removed.
  21. [03/19/15 19:41:49] seafserv-gc.c(115): GC version 1 repo test2(e7d26d93)
  22. [03/19/15 19:41:49] gc-core.c(394): GC started. Total block number is 507.
  23. [03/19/15 19:41:49] gc-core.c(75): GC index size is 1024 Byte.
  24. [03/19/15 19:41:49] gc-core.c(408): Populating index.
  25. [03/19/15 19:41:49] gc-core.c(262): Populating index for repo e7d26d93.
  26. [03/19/15 19:41:49] gc-core.c(308): Traversed 577 commits, 507 blocks.
  27. [03/19/15 19:41:49] gc-core.c(440): Scanning unused blocks.
  28. [03/19/15 19:41:49] gc-core.c(472): GC finished. 507 blocks total, about 507 reachable blocks, 0 blocks can be removed.
  30. [03/19/15 19:41:50] seafserv-gc.c(124): === Repos deleted by users ===
  31. [03/19/15 19:41:50] seafserv-gc.c(145): === GC is finished ===
  33. [03/19/15 19:41:50] Following repos have blocks to be removed:
  34. repo-id1
  35. repo-id2
  36. repo-id3

If you give specific library ids, only those libraries will be checked; otherwise all libraries will be checked.

Notice that at the end of the output there is a “repos have blocks to be removed” section. It contains the list of librareis that have garbage blocks. Later when you run GC without —dry-run option, you can use these libraris ids as input arguments to GC program.

Removing Garbage

To actually remove garbage blocks, run without the —dry-run option:

  1. seaf-gc.sh [repo-id1] [repo-id2] ...

If libraries ids are specified, only those libraries will be checked for garbage.

As described before, there are two types of garbage blocks to be removed. Sometimes just removing the first type of blocks (those that belong to deleted libraries) is good enough. In this case, the GC program won’t bother to check the libraries for outdated historic blocks. The “-r” option implements this feature:

  1. seaf-gc.sh -r

In Seafile version 4.1.1 and later, libraries deleted by the users are not immediately removed from the system. Instead, they’re moved into a “trash” in the system admin page. Before they’re cleared from the trash, their blocks won’t be garbage collected.

Using Multiple Threads in GC

Since Pro server 5.1.0, you can specify the thread number in GC. By default,

  • If storage backend is S3/Swift/Ceph, 10 threads are started to do the GC work.
  • If storage backend is file system, only 1 thread is started.

You can specify the thread number in with “-t” option. “-t” option can be used together with all other options. Each thread will do GC on one library. For example, the following command will use 20 threads to GC all libraries:

  1. seaf-gc.sh -t 20

Since the threads are concurrent, the output of each thread may mix with each others. Library ID is printed in each line of output.

Run GC in older versions (before 4.1.1)

To run GC program

  1. ./seaf-gc.sh run

If you want to do sanity check before actually removing any data, you can use the —dry-run option

  1. ./seaf-gc.sh dry-run

It will show you the total block number vs. the number of blocks to be removed.

To check data integrity after running GC, you can use seaf-fsck

GC cleanup script for Community Version

To use this script you need:

  • Setup the seafile-service file at ‘/etc/init.d/seafile-server’
  • Files of seafile setup need to be owner by ‘seafile:nogroup’ or ‘seafile:seafile’
  • Run the script with sudo or as root
  • Put the script into crontab of a root user

Create the script file (change the location to your liking):

  1. touch /opt/haiwen/seafile/cleanupScript.sh

Use your favorite text editor and paste the following code:

  1. #!/bin/bash
  3. #####
  4. # Uncomment the following line if you rather want to run the script manually.
  5. # Display usage if the script is not run as root user
  6. #        if [[ $USER != "root" ]]; then
  7. #                echo "This script must be run as root user!"
  8. #                exit 1
  9. #        fi
  10. #
  11. # echo "Super User detected!!"
  12. # read -p "Press [ENTER] to start the procedure, this will stop the seafile server!!"
  13. #####
  15. # stop the server
  16. echo Stopping the Seafile-Server...
  17. systemctl stop seafile.service
  18. systemctl stop seahub.service
  20. echo Giving the server some time to shut down properly....
  21. sleep 20
  23. # run the cleanup
  24. echo Seafile cleanup started...
  25. sudo -u seafile $pathtoseafile/seafile-server-latest/seaf-gc.sh
  27. echo Giving the server some time....
  28. sleep 10
  30. # start the server again
  31. echo Starting the Seafile-Server...
  32. systemctl start seafile.service
  33. systemctl start seahub.service
  35. echo Seafile cleanup done!

Make sure that the script has been given execution rights, to do that run this command.

  1. sudo chmod +x /path/to/yourscript.sh

Then open crontab with the root user

  1. crontab -e

Add the following line (change the location of your script accordingly!)

  1. 0 2 * * Sun /opt/haiwen/seafile/cleanupScript.sh

The script will then run every Sunday at 2:00 AM.

GC in the seafile docker container

To perform garbage collection inside the seafile docker container, you must run the /scripts/gc.sh script. Simply run docker exec <whatever-your-seafile-container-is-called> /scripts/gc.sh.