Understanding Synchronization

This article describes the mechanisms Syncthing uses to bring files in sync on a high level.

Blocks

Files are divided into blocks. The blocks making up a file are all the same size (except the last one in the file which may be smaller). The block size is dependent on the file size and varies from 128 KiB up to 16 MiB. Each file is sliced into a number of these blocks, and the SHA256 hash of each block is computed. This results in a block list containing the offset, size and hash of all blocks in the file.

To update a file, Syncthing compares the block list of the current version of the file to the block list of the desired version of the file. It then tries to find a source for each block that differs. This might be locally, if another file already has a block with the same hash, or it may be from another device in the cluster. In the first case the block is simply copied on disk, in the second case it is requested over the network from the other device.

When a block is copied or received from another device, its SHA256 hash is computed and compared with the expected value. If it matches the block is written to a temporary copy of the file, otherwise it is discarded and Syncthing tries to find another source for the block.

Scanning

There are two methods how Syncthing detects changes: By regular full scans and by notifications received from the filesystem (“watcher”). By default the watcher is enabled and full scans are done once per hour. This behaviour can be changed by folder. Increasing the full scan interval uses less resources and is useful for example on large folders that change infrequently. To make sure that not all folders are rescanned at the same time, the actual scan interval is a random time between 3/4 and 5/4 of the given scan interval. Even with watcher enabled it is advised to keep regular full scans enabled, as it is possible that some changes aren’t picked up by it.

During a rescan (regardless whether full or from watcher) the existing files are checked for changes to their modification time, size or permission bits. The file is “rehashed” if a change is detected based on those attributes, that is a new block list is calculated for the file. It is not possible to know which parts of a file have changed without reading the file and computing new SHA256 hashes for each block.

The watcher does not immediately schedule a scan when a change is detected. It accumulates changes for 10s (adjustable by fsWatcherDelayS) and deleted files are further delayed for 1min. Therefore it is expected that you experience a slight delay between making the change and it appearing on another device.

Changes that were detected and hashed are transmitted to the other devices after each rescan.

Syncing

Syncthing keeps track of several versions of each file - the version that it currently has on disk, called the local version, the versions announced by all other connected devices, and the “best” (usually the most recent) version of the file. This version is called the global version and is the one that each device strives to be up to date with.

This information is kept in the index database, which is stored in the configuration directory and called index-vx.y.z.db (for some version x.y.z which may not be exactly the version of Syncthing you’re running).

When new index data is received from other devices Syncthing recalculates which version for each file should be the global version, and compares this to the current local version. When the two differ, Syncthing needs to synchronize the file. The block lists are compared to build a list of needed blocks, which are then requested from the network or copied locally, as described above.

Temporary Files

Syncthing never writes directly to a destination file. Instead all changes are made to a temporary copy which is then moved in place over the old version. If an error occurs during the copying or syncing, such as a necessary block not being available, the temporary file is kept around for up to a day. This is to avoid needlessly requesting data over the network.

The temporary files are named .syncthing.original-filename.ext.tmp or, on Windows, ~syncthing~original-filename.ext.tmp where original-filename.ext is the destination filename. The temporary file is normally hidden. If the temporary file name would be too long due to the addition of the prefix and extra extension, a hash of the original file name is used instead of the actual original file name.