Syncthing Development

Controlling Syncthing from External Applications

People all over the world have developed a number of useful applications that build around the Syncthing core, such as tray notifications and Android support. These are made possible using two APIs:

• A long polling interface for exposing events from the core utility to an external party. This Event API is useful for being notified of when changes to files, network connections or sync status occur.

• A REST API for controlling the operation of Syncthing and directly querying for current status.

If this covers what you need to do, there is no need to delve deeper. However, if you would like to add functionality to Syncthing itself, or correct a bug or two in there, please read on.

Contributing to the Syncthing Core

First of all, follow Building Syncthing to get your workspace set up correctly. Syncthing is written mainly in Go which has some fairly specific opinions on the required directory layout. If you’re new to Go, don’t fear – it’s a small language and easy to learn. There’s a wealth of resources on the web to help you get up to speed, and many people joining the project have done so with it being their first contact with Go.

When you are ready to start hacking, take a quick glance at the Contribution Guidelines to know what to expect and to make the process smoother. The main take away is to keep the code clean, base it on the main branch, and we’ll sort out the rest once you file a pull request.

Source Code Layout

In the source repository you’ll find a tree of various packages and directories. There is some Go code at the top level, but it’s basically scripts for the build system. The actual code lives in the cmd/syncthing and lib directories. The web GUI lives in gui. The rest is as follows.

assets/

Various graphical assets – the logo.

cmd/

Commands either built as end products or used by the build process itself.

• genassets/

  Generates asset files that are compiled into syncthing as part of the build process (build utility).

  stcompdirs/

  Compares two directories (debugging utility).

  stevents/

  Displays event trace from a remote syncthing using the API (debugging utility).

  stfileinfo/

  Shows information about a file, in the same manner syncthing would see it (debugging utility).

  stfinddevice/

  Looks up a device on a global discovery server (debugging utility).

  stindex/

  Prints index (database) contents (debugging utility).

  syncthing/

  Synchronizes files between devices…

  todos/

  Converts line endings from Unix to DOS standard (build utility).

  transifexdl/

  Downloads translations from Transifex (build utility).

  translate/

  Generates translation source for Transifex based on the HTML source (build utility).

etc/

Startup scripts and integration files. Included as-is in the release packages.

gui/

The web GUI source. Gets compiled into the syncthing binary by way of genassets and the build process.

lib/

Contains all packages that make up the parts of syncthing.

• auto/

  Auto generated asset data, created by genassets based on the contents of the gui directory.

  beacon/

  Multicast and broadcast UDP beacons. Used by the local discovery system.

  config/

  Parses, validates and saves configuration files.

  db/

  Stores and processes file index information in a database on disk.

  discover/

  The local and global device discovery – maps device IDs to IP and port tuples.

  events/

  The event subsystem, handles emitting of and subscribing to events across the other packages.

  fnmatch/

  Matches strings to glob patterns, used by the ignore package.

  ignore/

  Parses the .stignore file and matches it against file paths.

  model/

  Ties together many parts of syncthing and handles the main logic of synchronizing files with other devices.

  osutil/

  Abstracts away certain OS specific quirks.

  rc/

  Remote controls a Syncthing process over the REST API.

  protocol/

  Implementation of the BEP protocol.

  scanner/

  Looks for changes to files and hashes them as appropriate.

  stats/

  Records statistics about devices and folders.

  symlinks/

  Handles symlinks in a platform independent manner.

  sync/

  Provides optional debugging on top of the regular Mutex / RWMutex primitives.

  upgrade/

  Downloads and performs upgrade of the running binary.

  upnp/

  Discovers UPnP devices and sets up port mappings for incoming connections.

  versioner/

  Provides file versioning algorithms; simple, staggered and external.

man/

Manual pages, generated from the documentation.

pkg/

Compiled packages, generated by the build process.

protocol/

Legacy location of the protocol package.

script/

Various utility scripts for auto generating stuff and so on.

test/

The integration test suite.

vendor/

Locally vendored copies of external dependencies.

Why are you being so hard on my pull request?

A pull request looks a little different depending on whether you’re on the “contributor” or “maintainer” side. The contributor says:

I implemented a new feature in your project for you!

However, the maintainer hears:

I wrote some code. I’d like you to test, support, document and maintain it for me forever.

The maintainer will want to make sure that the code is something we feel comfortable taking that responsibility for. That means well tested, clear implementation, fits into the overall architecture, etc.

But perhaps the existing code doesn’t fulfill this to start with; is it then fair to expect it from a change in a pull request? For example asking for a test or documentation, where there is none before. Well, the existing code has some advantage just by being legacy;

• Perhaps there isn’t a test, but we know this code works because it’s been running in production for a long time without complaints. Then it’s fair to expect tests from code replacing it.

• Perhaps there isn’t a test, and your code fixes a bug with the code. That just highlights that there should have been a test to start with, and this is the optimal time to add one.

• Perhaps how the code works (or what exactly it does) isn’t clear to the reviewer. A test will clarify and lock this down, and also prevent us from inadvertently breaking it later.

Another thing that the maintainer might be hard about is whether the code actually solves the entire problem, or at least enough of it to stand on its own. This will be more relevant to new features than bugfixes and includes questions like;

• Is the feature general enough to be used by other users? If not, do we really need it or can it be implemented as part of something more general?

• Is the feature completely implemented? That is, if a new feature is added it should be available in the GUI, emit relevant trace information to enable debugging, be correctly saved in the configuration, etc. If components of this are missing, that’s work the maintainer will have to do after accepting the pull request.

All in all, a great pull request creates less work for the maintainer, not more.