Cloud storage options, such as Amazon S3, Google Cloud Storage, Azure, Backblaze B2, etc, are ideal for backup purpose.

For example, for Amazon S3, the upload is free. You only pay for the storage.So you have the benefit of:

  • Extremely fast access to local SeaweedFS Filer
  • Near-Real-Time Backup to Amazon S3 with zero-cost upload network traffic.

Architecture

Every file changes in Filer will trigger a notification sent to a message queue.A "weed replicate" process will read from the message queue, read the actual file content, and send the update to the cloud sink.

  • Message Queue can be: Kafka, AWS SQS, Google Cloud Platform Pub/Sub.
  • Cloud Sink can be: AWS S3, Google Cloud Storage, Microsoft Azure, Backblaze B2.

Backup to Cloud - 图1

Configuration

  • Configure notification. use "weed scaffold -config=notification" to see the notification section.
  1. [notification.kafka]
  2. enabled = true
  3. hosts = [
  4.   "localhost:9092"
  5. ]
  6. topic = "seaweedfs_filer_to_s3"

  • Setup Kafka. Possibly you need to create the Kafka topic if auto topic creation is not enabled.

  • Configure replication. use "weed scaffold -config=replication" to see the notification section.

  1. [source.filer]
  2. enabled = true
  3. grpcAddress = "localhost:18888"
  4. directory = "/buckets"    # all files under this directory tree are replicated
  6. [sink.s3]
  7. # read credentials doc at https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/sessions.html
  8. # default loads credentials from the shared credentials file (~/.aws/credentials).
  9. enabled = false
  10. aws_access_key_id     = ""     # if empty, loads from the shared credentials file (~/.aws/credentials).
  11. aws_secret_access_key = ""     # if empty, loads from the shared credentials file (~/.aws/credentials).
  12. region = "us-west-1"
  13. bucket = "your_bucket_name"    # an existing bucket
  14. directory = "/"                 # destination directory
  • Start the Kafka.
  • Start the replication. "weed filer.replicate"
  • Start the filer. "weed filer"

Replicate existing files

See Async-Replication-to-another-Filer#replicate-existing-files