Deprecated and removed since RocksDB 6.0

RocksDB CompactionFilter offers a way to remove / filter expired key / value pairs based on custom logic in background. Now we have implemented an extension on top of it which allows users to implement their custom CompactionFilter in Lua! This feature is available in RocksDB 5.0.

Benefits

Developing CompactionFilter in Lua has the following benefits:

  • On-the-fly change: updating Lua Compaction Filter can be done on the fly without shutting down your RocksDB instance.
  • Individual binary unit: as updating Lua Compaction Filter can be done on the fly, it no longer requires rebuilding the binary as required by the C++ compaction filter. This is a huge benefit for services built on top of RocksDB who maintains custom CompactionFilters for their customers.
  • Safer: errors in CompactionFilter will no longer result in core dump. Lua engine captures all the exceptions.

How to Use?

Using RocksLuaCompactionFilter is simple. All you need to do is the following steps:

  • Build RocksDB with LUA_PATH set to the root directory of Lua.
  • Config RocksLuaCompactionFilterOptions with your Lua script. (more details are described in the next section)
  • Use your RocksLuaCompactionFilterOptions in step 1 to construct a RocksLuaCompactionFilterFactory.
  • Pass your RocksLuaCompactionFilterFactory in step 2 to ColumnFamilyOptions::compaction_filter_factory.

Example

Here’s a simple RocksLuaCompactionFilter that filter out any keys whose initial is less than r:

  1.   lua::RocksLuaCompactionFilterOptions lua_opt;
  2.   // removes all keys whose initial is less than 'r'
  3.   lua_opt.lua_script =
  4.       "function Filter(level, key, existing_value)\n"
  5.       "  if key:sub(1,1) < 'r' then\n"
  6.       "    return true, false, \"\"\n"
  7.       "  end\n"
  8.       "  return false, false, \"\"\n"
  9.       "end\n"
  10.       "\n"
  11.       "function FilterMergeOperand(level, key, operand)\n"
  12.       "  return false\n"
  13.       "end\n"
  14.       "function Name()\n"
  15.       "  return \"KeepsAll\"\n"
  16.       "end\n";
  18.   // specify error log.
  19.   auto* rocks_logger = new facebook::rocks::RocksLogger(
  20.       "RocksLuaTest",
  21.       true,  // print error message in GLOG
  22.       true,  // print error message to scribe, available in LogView `RocksDB ERROR`
  23.       nullptr);
  25.   // Create RocksLuaCompactionFilter with the above lua_opt and pass it to Options
  26.   rocksdb::Options options;
  27.   options.compaction_filter_factory =
  28.       std::make_shared<rocksdb::lua::RocksLuaCompactionFilterFactory>(lua_opt);
  29.   ...
  30.   // open FbRocksDB with the above options
  31.   rocksdb::DB* db;
  32.   auto status = openRocksDB(options, "RocksDBWithLua", &db);

Config RocksLuaCompactionFilterOptions

Here we introduce how to config RocksLuaCompactionFilterOptions in more detail. The definition of RocksLuaCompactionFilterOptions can be found in include/rocksdb/utilities/lua/rocks_lua_compaction_filter.h.

Config the Lua Script (RocksLuaCompactionFilter::script)

The first and the most important parameter is RocksLuaCompactionFilterOptions::script, which is where your Lua compaction filter will be implemented. Your Lua script must implement the required functions, which are Name() and Filter().

  1.   // The lua script in string that implements all necessary CompactionFilter
  2.   // virtual functions.  The specified lua_script must implement the following
  3.   // functions, which are Name and Filter, as described below.
  4.   //
  5.   // 0. The Name function simply returns a string representing the name of
  6.   //    the lua script.  If there's any erorr in the Name function, an
  7.   //    empty string will be used.
  8.   //    --- Example
  9.   //      function Name()
  10.   //        return "DefaultLuaCompactionFilter"
  11.   //      end
  12.   //
  13.   //
  14.   // 1. The script must contains a function called Filter, which implements
  15.   //    CompactionFilter::Filter() , takes three input arguments, and returns
  16.   //    three values as the following API:
  17.   //
  18.   //   function Filter(level, key, existing_value)
  19.   //     ...
  20.   //     return is_filtered, is_changed, new_value
  21.   //   end
  22.   //
  23.   //   Note that if ignore_value is set to true, then Filter should implement
  24.   //   the following API:
  25.   //
  26.   //   function Filter(level, key)
  27.   //     ...
  28.   //     return is_filtered
  29.   //   end
  30.   //
  31.   //   If there're any error in the Filter() function, then it will keep
  32.   //   the input key / value pair.
  33.   //
  34.   //   -- Input
  35.   //   The function must take three arguments (integer, string, string),
  36.   //   which map to "level", "key", and "existing_value" passed from
  37.   //   RocksDB.
  38.   //
  39.   //   -- Output
  40.   //   The function must return three values (boolean, boolean, string).
  41.   //     - is_filtered: if the first return value is true, then it indicates
  42.   //       the input key / value pair should be filtered.
  43.   //     - is_changed: if the second return value is true, then it indicates
  44.   //       the existing_value needs to be changed, and the resulting value
  45.   //       is stored in the third return value.
  46.   //     - new_value: if the second return value is true, then this third
  47.   //       return value stores the new value of the input key / value pair.
  48.   //
  49.   //   -- Examples
  50.   //     -- a filter that keeps all key-value pairs
  51.   //     function Filter(level, key, existing_value)
  52.   //       return false, false, ""
  53.   //     end
  54.   //
  55.   //     -- a filter that keeps all keys and change their values to "Rocks"
  56.   //     function Filter(level, key, existing_value)
  57.   //       return false, true, "Rocks"
  58.   //     end
  60.   std::string lua_script;

An optimization without using value (RocksLuaCompactionFilter::ignore_value) In case your CompactionFilter never uses value to determine whether to keep or discard a key / value pair, then setting RocksLuaCompactionFilterOptions::ignore_value=true and implement the simplified Filter() API. Our result shows that this optimization can save up to 40% CPU overhead introduced by LuaCompactionFilter:

  1.   // If set to true, then existing_value will not be passed to the Filter
  2.   // function, and the Filter function only needs to return a single boolean
  3.   // flag indicating whether to filter out this key or not.
  4.   //
  5.   //   function Filter(level, key)
  6.   //     ...
  7.   //     return is_filtered
  8.   //   end
  9.   bool ignore_value = false;

The simplified Filter() API only takes two input arguments and returns only one boolean flag indicating whether to keep or discard the input key.

Error Log Configuration (RocksLuaCompactionFilterOptions::error_log)

When RocksLuaCompactionFilter hit any error, it will act as no-op and always return false for any key / value pair that cause errors. Developers can config RocksLuaCompactionFilterOptions::error_log to log any Lua errors:

  1.  // When specified a non-null pointer, the first "error_limit_per_filter"
  2.  // errors of each CompactionFilter that is lua related will be included
  3.  // in this log.
  4.  std::shared_ptr<Logger> error_log;

Note that for each compaction job, we only log the first few Lua errors in error_log to avoid generating too many error messages, and the number of errors it reports per compaction job can be configured via error_limit_per_filter. The default number is one.

  1.  // The number of errors per CompactionFilter will be printed
  2.  // to error_log.
  3.  int error_limit_per_filter = 1;

Dynamic Updating Lua Script

To update the Lua script while the RocksDB database is running, simply call the SetScript() API of your RocksLuaCompactionFilterFactory:

  1.   // Change the Lua script so that the next compaction after this
  2.   // function call will use the new Lua script.
  3.   void SetScript(const std::string& new_script);