Load Balancing with Varnish

Varnish is a powerful HTTP load balancer (reverse proxy), which is also very good at caching. When running multiple TSDs, Varnish comes in handy to distribute the HTTP traffic across the TSDs. Bear in mind that write traffic doesn’t use the HTTP protocol by default, and as such you can only use Varnish for read queries. Using Varnish will help you easily scale the amount of read capacity of your TSD cluster.

The following is a sample Varnish configuration recommended for use with OpenTSDB. It uses a slightly custom load balancing strategy to achieve optimal cache hit rate at the TSD level. This configuration requires at least Varnish 2.1.0 to run, but using Varnish 3.0 or above is strongly recommended.

This sample configuration is for 2 backends, named foo and bar. You need to substitute at least the host names.

  1. # VCL configuration for OpenTSDB.
  3. backend foo {
  4.     .host = "foo";
  5.     .port = "4242";
  6.     .probe = {
  7.         .url = "/version";
  8.         .interval = 30s;
  9.         .timeout = 10s;
  10.         .window = 5;
  11.         .threshold = 3;
  12.     }
  13. }
  15. backend bar {
  16.     .host = "bar";
  17.     .port = "4242";
  18.     .probe = {
  19.         .url = "/version";
  20.         .interval = 30s;
  21.         .timeout = 10s;
  22.         .window = 5;
  23.         .threshold = 3;
  24.     }
  25. }
  27. # The `client' director will select a backend based on `client.identity'.
  28. # It's normally used to implement session stickiness but here we abuse it
  29. # to try to send pairs of requests to the same TSD, in order to achieve a
  30. # higher cache hit rate.  The UI sends queries first with a "&json" at the
  31. # end, in order to get meta-data back about the results, and then it sends
  32. # the same query again with "&png".  If the second query goes to a different
  33. # TSD, then that TSD will have to fetch the data from HBase again.  Whereas
  34. # if it goes to the same TSD that served the "&json" query, it'll hit the
  35. # cache of that TSD and produce the PNG directly without using HBase.
  36. #
  37. # Note that we cannot use the `hash' director here, because otherwise Varnish
  38. # would hash both the "&json" and the "&png" requests identically, and it
  39. # would thus serve a cached JSON response to a "&png" request.
  40. director tsd client {
  41.     { .backend = foo; .weight = 100; }
  42.     { .backend = bar; .weight = 100; }
  43. }
  45. sub vcl_recv {
  46.     set req.backend = tsd;
  47.     # Make sure we hit the same backend based on the URL requested,
  48.     # but ignore some parameters before hashing the URL.
  49.     set client.identity = regsuball(req.url, "&(o|ignore|png|json|html|y2?range|y2?label|y2?log|key|nokey)\b(=[^&]*)?", "");
  50. }
  52. sub vcl_hash {
  53.     # Remove the `ignore' parameter from the URL we hash, so that two
  54.     # identical requests modulo that parameter will hit Varnish's cache.
  55.     hash_data(regsuball(req.url, "&ignore\b(=[^&]*)?", ""));
  56.     if (req.http.host) {
  57.         hash_data(req.http.host);
  58.     } else {
  59.         hash_data(server.ip);
  60.     }
  61.     return (hash);
  62. }

On many Linux distros (including Debian and Ubuntu), you need to put the configuration above in /etc/varnish/default.vcl. We also recommend tweaking the command-line parameters of varnishd in order to use a memory-backed cache of about 1GB if you can afford it. On Debian/Ubuntu systems, this is done by editing /etc/default/varnish to make sure that -s malloc,1G is passed to varnishd.

Read more about Varnish:

Note

if you’re using Varnish 2.x (which is not recommended as we would strongly encourage you to migrate to 3.x) you have to replace each function call hash_data(foo); to set req.hash += foo; in the VCL configuration above.