Create a File Server for Imports and Backups

Create a File Server for Imports and Backups

If you need a location to store files for the IMPORT process or CockroachDB enterprise backups, but do not have access to (or simply cannot use) cloud storage providers, you can run a local file server. You can then use this file server by leveraging support for our HTTP Export Storage API.

This is especially useful for:

Implementing a compatibility layer in front of custom or proprietary storage providers for which CockroachDB does not yet have native support
Using on-premises storage

HTTP export storage API

CockroachDB tasks that require reading or writing external files (such as IMPORT and BACKUP) can use the HTTP Export Storage API by prefacing the address with http, e.g., http://fileserver/mnt/cockroach-exports.

This API uses the GET, PUT and DELETE methods. This behaves like you would expect typical HTTP requests to work. After a PUT request to some path, a subsequent GET request should return the content sent in the PUT request body, at least until a DELETE request is received for that path.

Examples

You can use any file server software that supports GET, PUT and DELETE methods, but we've included code samples for common ones:

Note:
We do not recommend using any machines running cockroach as file servers. Using machines that are running cockroach as file servers could negatively impact performance if I/O operations exceed capacity.

Using PHP with IMPORT

The PHP language has an HTTP server built in. You can serve local files using the commands below. For more information about how to import these locally served files, see the documentation for the IMPORT statement.

$ cd /path/to/data
$ php -S 127.0.0.1:3000 # files available at e.g., 'http://localhost:3000/data.sql'

Using Python with IMPORT

The Python language has an HTTP server included in the standard library. You can serve local files using the commands below. For more information about how to import these locally served files, see the documentation for the IMPORT statement.

$ cd /path/to/data
$ python -m SimpleHTTPServer 3000 # files available at e.g., 'http://localhost:3000/data.sql'

If you use Python 3, try:

$ cd /path/to/data
$ python -m http.server 3000

Using Ruby with IMPORT

The Ruby language has an HTTP server included in the standard library. You can serve local files using the commands below. For more information about how to import these locally served files, see the documentation for the IMPORT statement.

$ cd /path/to/data
$ ruby -run -ehttpd . -p3000 # files available at e.g., 'http://localhost:3000/data.sql'

Using Caddy as a file server

Download the Caddy web server. Before downloading, in the Customize your build step, open the list of Plugins and make sure to check the http.upload option.
Copy the caddy binary to the directory containing the files you want to serve, and run it with an upload directive, either in the command line or via Caddyfile.
Command line example (with no TLS):

$ caddy -root /mnt/cockroach-exports "upload / {" 'to "/mnt/cockroach-exports"' 'yes_without_tls' "}"

Caddyfile example (using a key and cert):

tls key cert
root "/mnt/cockroach-exports"
upload / {
  to "/mnt/cockroach-exports"
}

For more information about Caddy, see its documentation.

Using nginx as a file server

Install nginx with the webdav module (often included in -full or similarly named packages in various distributions).
In the nginx.conf file, add a dav_methods PUT DELETE directive. For example:

events {
    worker_connections  1024;
}
http {
  server {
    listen 20150;
    location / {
      dav_methods  PUT DELETE;
      root /mnt/cockroach-exports;
      sendfile           on;
      sendfile_max_chunk 1m;
    }
  }
}

Create a File Server for IMPORT/BACKUP