In order to achieve high concurrency, SeaweedFS tries to read and write the whole file into memory. But this would not work for large files.

The following is implemented in "weed upload" command. For 3rd party clients, here is the spec.

To support large files, SeaweedFS supports these two kinds of files:

  • Chunk File. Each chunk file is actually just normal files to SeaweedFS.
  • Chunk Manifest. A simple json file with the list of all the chunks.

This piece of code shows the json file structure:

https://github.com/chrislusf/seaweedfs/blob/master/weed/operation/chunked_file.go#L24

  1. type ChunkInfo struct {
  2.     Fid    string `json:"fid"`
  3.     Offset int64  `json:"offset"`
  4.     Size   int64  `json:"size"`
  5. }
  7. type ChunkList []*ChunkInfo
  9. type ChunkManifest struct {
  10.     Name   string    `json:"name,omitempty"`
  11.     Mime   string    `json:"mime,omitempty"`
  12.     Size   int64     `json:"size,omitempty"`
  13.     Chunks ChunkList `json:"chunks,omitempty"`
  14. }

When reading Chunk Manifest files, the SeaweedFS will find and send the data file based on the list of ChunkInfo.

Create new large file

SeaweedFS delegates the effort to the client side. The steps are:

  • split large files into chunks
  • upload each file chunk as usual. Save the related info into ChunkInfo struct. Each chunk can be spread onto different volumes, possibly giving faster parallel access.
  • upload the manifest file with mime type "application/json", and add url parameter "cm=true". The FileId to store the manifest file is the entry point of the large file.

Update large file

Usually we just append large files. Updating a specific chunk of file is almost the same.

The steps to append a large file:

  • upload the new file chunks as usual. Save the related info into ChunkInfo struct.
  • update the updated manifest file with mime type "application/json", and add url parameter "cm=true".

Example

  1. # split -n 2 sy.jpg   // split the file into two: xxa and xxb
  3. # curl -X POST -F "file=@xaa" http://localhost:9333/submit?pretty=yes
  4. {
  5.   "eTag": "809b2add",
  6.   "fid": "6,1b70e99bcd",
  7.   "fileName": "xaa",
  8.   "fileUrl": "10.34.254.62:8080/6,1b70e99bcd",
  9.   "size": 73433
  10. }
  12. # curl -X POST -F "file=@xab" http://localhost:9333/submit?pretty=yes
  13. {
  14.   "eTag": "9c6ca661",
  15.   "fid": "3,1c863b4563",
  16.   "fileName": "xab",
  17.   "fileUrl": "10.34.254.62:8080/3,1c863b4563",
  18.   "size": 73433
  19. }
  21. // get one fid for manifest file
  22. # curl "10.34.254.62:9333/dir/assign?pretty=yes"
  23. {
  24.   "fid": "5,1ea9c7d93e",
  25.   "url": "10.34.254.62:8080",
  26.   "publicUrl": "10.34.254.62:8080",
  27.   "count": 1
  28. }
  30. # cat manifest.json 
  31. {
  32.         "name": "sy.jpg",
  33.         "mime": "image/jpeg",
  34.         "size": 146866,
  35.         "chunks": [{
  36.                 "fid": "6,0100711ab7",
  37.                 "offset": 0,
  38.                 "size": 73433
  39.         }, {
  40.                 "fid": "3,1c863b4563",
  41.                 "offset": 73433,
  42.                 "size": 73433
  43.         }]
  44. }
  46. // upload the manifest file
  47. # curl -v -F "file=@manifest.json" "http://10.34.254.62:8080/5,1ea9c7d93e?cm=true&pretty=yes"
  48. *   Trying 10.34.254.62...
  49. * Connected to 10.34.254.62 (10.34.254.62) port 8080 (#0)
  50. > POST /5,1ea9c7d93e?cm=true&pretty=yes HTTP/1.1
  51. > Host: 10.34.254.62:8080
  52. > User-Agent: curl/7.47.0
  53. > Accept: */*
  54. > Content-Length: 418
  55. > Expect: 100-continue
  56. > Content-Type: multipart/form-data; boundary=------------------------a872064c8f40903c
  57. > 
  58. < HTTP/1.1 100 Continue
  59. < HTTP/1.1 201 Created
  60. < Content-Type: application/json
  61. < Etag: "2229f9b4"
  62. < Date: Wed, 15 Jan 2020 03:12:18 GMT
  63. < Content-Length: 66
  64. < 
  65. {
  66.   "name": "manifest.json",
  67.   "size": 213,
  68.   "eTag": "2229f9b4"
  69. }
  70. * Connection #0 to host 10.34.254.62 left intact
  72. // download the full file
  73. # curl -v "http://10.34.254.62:8080/5,1ff0fb46c9" -o out.data
  74. *   Trying 10.34.254.62...
  75.   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
  76.                                  Dload  Upload   Total   Spent    Left  Speed
  77.   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 10.34.254.62 (10.34.254.62) port 8080 (#0)
  78. > GET /5,1ff0fb46c9 HTTP/1.1
  79. > Host: 10.34.254.62:8080
  80. > User-Agent: curl/7.47.0
  81. > Accept: */*
  82. > 
  83. < HTTP/1.1 200 OK
  84. < Accept-Ranges: bytes
  85. < Content-Disposition: inline; filename="sy.jpg"
  86. < Content-Length: 146866
  87. < Content-Type: image/jpeg
  88. < Etag: "3e8ef528"
  89. < Last-Modified: Wed, 15 Jan 2020 03:32:47 GMT
  90. < X-File-Store: chunked
  91. < Date: Wed, 15 Jan 2020 03:33:04 GMT
  92. < 
  93. { [7929 bytes data]
  94. 100  143k  100  143k    0     0  47.2M      0 --:--:-- --:--:-- --:--:-- 70.0M
  95. * Connection #0 to host 10.34.254.62 left intact
  97. // check md5 of the downloaded files
  98. # md5sum out.data 
  99. 836eababc392419580641a7b65370e82  out.data
  101. # md5sum sy.jpg 
  102. 836eababc392419580641a7b65370e82  sy.jpg

Notes

There are no particular limit in terms of chunk file size. Each chunk size does not need to be the same, even in the same file. The rule of thumb is to just being able to keep the whole chunk file in memory, and not to have too many small chunk files.

weed filer and weed mount

The filer server and the FUSE implementation that uses filer server are automatically chunking large files into smaller chunks.

The list of chunks are stored in filer storage, and managed by filer or weed mount client.