Debugging Syncthing

There’s a lot that happens behind the covers, and Syncthing is generally quite silent about it. A number of environment variables can be used to set the logging to verbose for various parts of the program, and to enable profiling.

Enabling any of the STTRACE facilities will also change the log format to include microsecond timestamps and file names plus line numbers. This can be used to enable this extra information on the normal logging level, without enabling any debugging: STTRACE=somethingnonexistent for example.

Under Unix (including Mac) the easiest way to run Syncthing with an environment variable set is to prepend the variable to the command line. I.e:

$ STTRACE=model syncthing

On windows, it needs to be set prior to running Syncthing.

  1. C:\> set STTRACE=model
  2. C:\> syncthing

Environment Variables

STTRACE

Used to increase the debugging verbosity in specific or all facilities, generally mapping to a Go package. Enabling any of these also enables microsecond timestamps, file names plus line numbers. Enter a comma-separated string of facilities to trace. syncthing -help always outputs an up-to-date list. The valid facility strings are:

  • Main and operational facilities:

    • config

      Configuration loading and saving.

      db

      The database layer.

      main

      Main package.

      model

      The root hub; the largest chunk of the system. File pulling, index transmission and requests for chunks.

      scanner

      File change detection and hashing.

      versioner

      File versioning.

  1. Networking facilities:
  3. -   beacon
  5.     Multicast and broadcast UDP discovery packets: Selected interfaces and addresses.
  7.     connections
  9.     Connection handling.
  11.     dialer
  13.     Dialing connections.
  15.     discover
  17.     Remote device discovery requests, replies and registration of devices.
  19.     nat
  21.     NAT discovery and port mapping.
  23.     pmp
  25.     NAT-PMP discovery and port mapping.
  27.     protocol
  29.     The BEP protocol.
  31.     relay
  33.     Relay interaction (`strelaysrv`).
  35.     upnp
  37.     UPnP discovery and port mapping.
  40. Other facilities:
  42. -   fs
  44.     Filesystem access.
  46.     events
  48.     Event generation and logging.
  50.     http
  52.     REST API.
  54.     sha256
  56.     SHA256 hashing package (this facility currently unused).
  58.     stats
  60.     Persistent device and folder statistics.
  62.     sync
  64.     Mutexes. Used for debugging race conditions and deadlocks.
  66.     upgrade
  68.     Binary upgrades.
  70.     walkfs
  72.     Filesystem access while walking.
  74.     all
  76.     All of the above.

STBLOCKPROFILE

Write block profiles to block-$pid-$timestamp.pprof every 20 seconds.

STCPUPROFILE

Write a CPU profile to cpu-$pid.pprof on exit.

STDEADLOCKTIMEOUT

Used for debugging internal deadlocks; sets debug sensitivity. Use only under direction of a developer.

STDEADLOCKTHRESHOLD

Used for debugging internal deadlocks; sets debug sensitivity. Use only under direction of a developer.

STGUIASSETS

Directory to load GUI assets from. Overrides compiled in assets. Useful for developing webgui, commonly use STGUIASSETS=gui bin/syncthing.

STHASHING

Specify which hashing package to use. Defaults to automatic based on performance. Specify “minio” (compatibility) or “standard” for the default Go implementation.

STHEAPPROFILE

Write heap profiles to heap-$pid-$timestamp.pprof each time heap usage increases.

STNODEFAULTFOLDER

Don’t create a default folder when starting for the first time. This variable will be ignored anytime after the first run.

STNORESTART

Equivalent to the -no-restart flag. Disable the Syncthing monitor process which handles restarts for some configuration changes, upgrades, crashes and also log file writing (stdout is still written).

STNOUPGRADE

Disable automatic upgrades.

STPROFILER

Set to a listen address such as “127.0.0.1:9090” to start the profiler with HTTP access, which then can be reached at http://localhost:9090/debug/pprof. See go tool pprof for more information.

STPERFSTATS

Write running performance statistics to perf-$pid.csv. Not supported on Windows.

STRECHECKDBEVERY

Time before folder statistics (file, dir, … counts) are recalculated from scratch. The given duration must be parseable by Go’s time.ParseDuration. If missing or not parseable, the default value of 1 month is used. To force recalculation on every startup, set it to 1s.

GOMAXPROCS

Set the maximum number of CPU cores to use. Defaults to all available CPU cores.

GOGC

Percentage of heap growth at which to trigger GC. Default is 100. Lower numbers keep peak memory usage down, at the price of CPU usage (i.e. performance).

Stepping with breakpoints

If you like to step through the running program, build a non-optimized binary and run with https://github.com/derekparker/delve.

Follow these steps:

  1. $ go run build.go -debug-binary build
  2. $ STNODEFAULTFOLDER=1 STNOUPGRADE=1  STNORESTART=1 dlv --listen=:2345 --headless=true --api-version=2 exec ./syncthing -- -home=./_test_config -no-browser

For installing and using delve itself see: