Architecture

Async Replication to another Filer - 图1

Requirements

There should be two Seaweed File Systems running, possibly across data centers. Each should have its filer, master, and volume servers.

Configuration

  • Configure notification. use "weed scaffold -config=filer" to see the notification section.
  1. [notification.log]
  2. enabled = false
  4. [notification.kafka]
  5. enabled = true
  6. hosts = [
  7.   "localhost:9092"
  8. ]
  9. topic = "seaweedfs_filer1_to_filer2"

  • Setup Kafka. Possibly you need to create the Kafka topic if auto topic creation is not enabled.

  • Configure replication. use "weed scaffold -config=replication" to see the notification section.

  1. [source.filer]
  2. enabled = true
  3. grpcAddress = "localhost:18888"
  4. directory = "/buckets"    # all files under this directory tree are replicated
  6. [notification.kafka]
  7. enabled = true
  8. hosts = [
  9.   "localhost:9092"
  10. ]
  11. topic = "seaweedfs_filer1_to_filer2"
  13. [sink.filer]
  14. enabled = true
  15. grpcAddress = "localhost:18888"
  16. directory = "/backup"    # all replicated files are under this directory tree
  17. replication = ""
  18. collection = ""
  19. ttlSec = 0

Please read the source.filer and sink.filer configurations carefully and make sure you fully understand its intention. This is actually using one filer and replicate files from one folder to another folder.

  • Start the Kafka.
  • Start the replication. "weed filer.replicate"
  • Start the filer. "weed filer"

Replicate existing files

The weed filer.replicate only replicates files that have been updated. If there are already a lot of files, very likely you would need to copy them too.

To do this, use echo 'fs.meta.notify' | weed shell, which will iterate all files and generate one "file creation" event for each file to the message queue. Thus, all files will be replicated.