Query Tracing and Replaying

The trace_replay APIs allow the user to track down the query information to a trace file. In the current implementation, Get, WriteBatch (Put, Delete, Merge, SingleDelete, and DeleteRange), Iterator (Seek and SeekForPrev) and MultiGet are the queries that tracked by the trace-replay API. Key, query timestamp, value (if applied), column family ID*(s) form one trace record. For Iterator, we also track the lower and upper bound if they are specified in the Iterator. For MultiGet, we serialize the whole multi-get information (a vector or timestamp, keys, and column family IDs) in one record.

Since one lock is used to protect the tracing instance and there will be extra I/Os for the trace file, the performance of DB will be influenced. According to the current test on the MyRocks and ZippyDB shadow server, the performance hasn’t been a concern in those shadows. The trace records from the same DB instance are written to a binary trace file. User can specify the path of trace file (e.g., store in different storage devices to reduce the I/O influence).

Currently, the trace file can be replayed via Replayer or by using the db_bench. The queries records in the traces file can be replayed to the target DB instance according to the time stamps. It can replay the workload nearly the same as the workload being collected, which will provide a more production-like testing case. If you want to replay the workload in a precise way. Right before you start tracing, you can create a hard link of the DB directory to another location (snapshot), delete the manifest in the copied directory, make a real copy of the manifest to the copied directory. Now, you can safely copy the snapshot to the location you want to replay.

An simple example to use the tracing APIs to generate a trace file:

  1. Options opt;
  2. Env* env = rocksdb::Env::Default();
  3. EnvOptions env_options(opt);
  4. std::string trace_path = "/tmp/trace_test_example";
  5. std::unique_ptr<TraceWriter> trace_writer;
  6. DB* db = nullptr;
  7. std::string db_name = "/tmp/rocksdb";
  9. // Create the trace file writer
  10. NewFileTraceWriter(env, env_options, trace_path, &trace_writer);
  11. DB::Open(opt, db_name, &db);
  13. // Start tracing
  14. TraceOptions trace_opt;
  15. db->StartTrace(trace_opt, std::move(trace_writer));
  17. /* your call of RocksDB APIs */
  19. // End tracing
  20. db->EndTrace()

To automatically replay the trace via Replayer:

  1. Status s;
  2. DB* db;
  3. std::vector<ColumnFamilyHandle*> handles;
  5. /* Open db and get db handles */
  7. // Create TraceReader
  8. std::string trace_path = "/tmp/trace_test_example";
  9. std::unique_ptr<TraceReader> trace_reader;
  10. s = NewFileTraceReader(db->GetEnv(), EnvOptions(), trace_path, &trace_reader);
  12. // Create Replayer
  13. std::unique_ptr<Replayer> replayer;
  14. s = db->NewDefaultReplayer(handles, std::move(trace_reader), &replayer);
  16. // Prepare (read trace header)
  17. s = replayer->Prepare();
  19. // Automatically replay (4 threads, 2x spped)
  20. s = replayer->Replay(ReplayOptions(4, 2.0), nullptr);
  22. /* To restart replay, Prepare() first, then Replay() */

To manually replay the trace via Replayer:

  1. /* Open db, create TraceReader and Replayer */
  3. // Prepare (read trace header)
  4. s = replayer->Prepare();
  6. // Skip any unsupported traces
  7. while (s.ok() || s.IsNotSupported()) {
  8.   std::unique_ptr<TraceRecord> record;
  9.   s = replayer->Next(&record);
  10.   if (s.ok()) {
  11.     // Execute the TraceRecord
  12.     s = replayer->Execute(std::move(record), nullptr);
  13.   }
  14. }
  16. // The current trace end is represented by Status::Incomplete()
  17. assert(s.IsIncomplete());

Example of replaying traces and reporting execution results:

  1. class SimpleResultHandler : public TraceRecordResult::Handler {
  2.  public:
  3.   SimpleResultHandler() {}
  4.   ~SimpleResultHandler() override {}
  6.   Status Handle(const StatusOnlyTraceExecutionResult& result) override {
  7.     cnt_++;
  8.     latency_ += result.GetLatency();
  9.     std::cout << "Status: " << result.GetStatus().ToString() << std::endl;
  10.   }
  12.   Status Handle(const SingleValueTraceExecutionResult& result) override {
  13.     cnt_++;
  14.     latency_ += result.GetLatency();
  15.     std::cout << "Status: " << result.GetStatus().ToString()
  16.               << ", value: " << result.GetValue() << std::endl;
  17.   }
  19.   Status Handle(const MultiValuesTraceExecutionResult& result) override {
  20.     cnt_++;
  21.     latency_ += result.GetLatency();
  22.     size_t size = result.GetMultiStatus().size();
  23.     for (size_t i = 0; i < size; i++) {
  24.       std::cout << i << ", status: " << result.GetMultiStatus()[i].ToString()
  25.                 << ", value: " << result.GetValues()[i] << std::endl;
  26.     }
  27.   }
  29.   Status Handle(const IteratorTraceExecutionResult& result) override {
  30.     cnt_++;
  31.     latency_ += result.GetLatency();
  32.     if (result.GetValid()) {
  33.       std::cout << "Valid"
  34.                 << ", status: " << result.GetStatus().ToString()
  35.                 << ", key: " << result.GetKey().ToString()
  36.                 << ", value: " << result.GetValue().ToString() << std::endl;
  37.     } else {
  38.       std::cout << "Invalid"
  39.                 << ", status: " << result.GetStatus().ToString() << std::endl;
  40.     }
  41.   }
  43.   uint64_t GetCount() const { return cnt_; }
  45.   double GetAvgLatency() const { return 1.0 * latency_ / cnt_; }
  47.   void Reset() {
  48.     cnt_ = 0;
  49.     latency_ = 0;
  50.   }
  52.  private:
  53.   std::atomic<uint64_t> cnt_{0};
  54.   std::atomic<uint64_t> latency_{0};
  55. };
  57. /* Open db, create TraceReader and Replayer */
  59. // Prepare (read trace header)
  60. s = replayer->Prepare();
  62. SimpleResultHandler handler;
  64. /* For auto-replay */
  65. // Result callback for auto-replay
  66. auto res_cb = [&handler](Status st, std::unique_ptr<TraceRecordResult>&& res) {
  67.   // res == nullptr if !st.ok()
  68.   if (res != nullptr) {
  69.     res->Accept(&handler);
  70.   }
  71. };
  73. s = replayer->Replay(ReplayOptions(4, 2.0), res_cb);
  75. std::cout << "Total traces: " << handler.GetCount() << std::endl;
  76. std::cout << "Average latency: " << handler.GetAvgLatency() << " us"
  77.           << std::endl;
  79. /* For manual-replay */
  80. // handler.Reset();
  82. s = replayer->Prepare();
  84. // Skip any unsupported traces
  85. while (s.ok() || s.IsNotSupported()) {
  86.   std::unique_ptr<TraceRecord> record;
  87.   s = replayer->Next(&record);
  88.   if (s.ok()) {
  89.     // Execute the TraceRecord
  90.     std::unique_ptr<TraceRecordResult> res;
  91.     s = replayer->Execute(std::move(record), &res);
  92.     // res == nullptr if !s.ok()
  93.     if (res != nullptr) {
  94.       res->Accept(&handler);
  95.     }
  96.   }
  97. }
  99. std::cout << "Total traces: " << handler.GetCount() << std::endl;
  100. std::cout << "Average latency: " << handler.GetAvgLatency() << " us"
  101.           << std::endl;

To replay the trace via db_bench:

  1. # You may also set --trace_replay_threads and --trace_replay_fast_forward
  2. ./db_bench --benchmarks=replay --trace_file=/tmp/trace_test_example --num_column_families=5

Trace Analyzing, Visualizing, and Modeling

After the user finishes the tracing steps by using the trace_replay APIs, the user will get one binary trace file. In the trace file, Get, Seek, and SeekForPrev are tracked with separate trace record, while queries of Put, Merge, Delete, SingleDelete, and DeleteRange are packed into WriteBatches. One tool is needed to 1) interpret the trace into the human readable format for further analyzing, 2) provide rich and powerful in-memory processing options to analyze the trace and output the corresponding results, and 3) be easy to add new analyzing options and query types to the tool.

The RocksDB team developed the initial version of the tool: trace_analyzer. It provides the following analyzing options and output results.

Note that most of the generated analyzing results output files will be separated in different column families and different query types, which means, one query type in one column family will have its own output files. Usually, one specified output option will generate one output file.

Analyze The Trace

The trace analyzer options

  1.     -analyze_delete (Analyze the Delete query.) type: bool default: false
  2.     -analyze_get (Analyze the Get query.) type: bool default: false
  3.     -analyze_iterator ( Analyze the iterate query like Seek() and
  4.       SeekForPrev().) type: bool default: false
  5.     -analyze_merge (Analyze the Merge query.) type: bool default: false
  6.     -analyze_multiget ( Analyze the MultiGet query. NOTE: for MultiGet, we
  7.       analyze each KV-pair read in one MultiGet query. Therefore, the total
  8.       queries and QPS are calculated based on the number of KV-pairs being
  9.       accessed not the number of MultiGet.It can be improved in the future if
  10.       needed) type: bool default: false
  11.     -analyze_put (Analyze the Put query.) type: bool default: false
  12.     -analyze_range_delete (Analyze the DeleteRange query.) type: bool
  13.       default: false
  14.     -analyze_single_delete (Analyze the SingleDelete query.) type: bool
  15.       default: false
  16.     -convert_to_human_readable_trace (Convert the binary trace file to a human
  17.       readable txt file for further processing. This file will be extremely
  18.       large (similar size as the original binary trace file). You can specify
  19.       'no_key' to reduce the size, if key is not needed in the next step.
  20.       File name: <prefix>_human_readable_trace.txt
  21.       Format:[type_id cf_id value_size time_in_micorsec <key>].) type: bool
  22.       default: false
  23.     -key_space_dir (<the directory stores full key space files> 
  24.       The key space files should be: <column family id>.txt) type: string
  25.       default: ""
  26.     -no_key ( Does not output the key to the result files to make smaller.)
  27.       type: bool default: false
  28.     -no_print (Do not print out any result) type: bool default: false
  29.     -output_access_count_stats (Output the access count distribution statistics
  30.       to file.
  31.       File name: 
  32.       <prefix>-<query_type>-<cf_id>-accessed_key_count_distribution.txt 
  33.       Format:[access_count number_of_access_count]) type: bool default: false
  34.     -output_dir (The directory to store the output files.) type: string
  35.       default: ""
  36.     -output_ignore_count (<threshold>, ignores the access count <= this value,
  37.       it will shorter the output.) type: int32 default: 0
  38.     -output_key_distribution (Print the key size distribution.) type: bool
  39.       default: false
  40.     -output_key_stats (Output the key access count statistics to file
  41.       for accessed keys:
  42.       file name: <prefix>-<query_type>-<cf_id>-accessed_key_stats.txt
  43.       Format:[cf_id value_size access_keyid access_count]
  44.       for the whole key space keys:
  45.       File name: <prefix>-<query_type>-<cf_id>-whole_key_stats.txt
  46.       Format:[whole_key_space_keyid access_count]) type: bool default: false
  47.     -output_prefix (The prefix used for all the output files.) type: string
  48.       default: "trace"
  49.     -output_prefix_cut (The number of bytes as prefix to cut the keys.
  50.       If it is enabled, it will generate the following:
  51.       For accessed keys:
  52.       File name: <prefix>-<query_type>-<cf_id>-accessed_key_prefix_cut.txt 
  53.       Format:[acessed_keyid access_count_of_prefix number_of_keys_in_prefix
  54.       average_key_access prefix_succ_ratio prefix]
  55.       For whole key space keys:
  56.       File name: <prefix>-<query_type>-<cf_id>-whole_key_prefix_cut.txt
  57.       Format:[start_keyid_in_whole_keyspace prefix]
  58.       if 'output_qps_stats' and 'top_k' are enabled, it will output:
  59.       File name:
  60.       <prefix>-<query_type>-<cf_id>-accessed_top_k_qps_prefix_cut.txt
  61.       Format:[the_top_ith_qps_time QPS], [prefix qps_of_this_second].)
  62.       type: int32 default: 0
  63.     -output_qps_stats (Output the query per second(qps) statistics 
  64.       For the overall qps, it will contain all qps of each query type. The time
  65.       is started from the first trace record
  66.       File name: <prefix>_qps_stats.txt
  67.       Format: [qps_type_1 qps_type_2 ...... overall_qps]
  68.       For each cf and query, it will have its own qps output.
  69.       File name: <prefix>-<query_type>-<cf_id>_qps_stats.txt 
  70.       Format:[query_count_in_this_second].) type: bool default: false
  71.     -output_time_series (Output the access time in second of each key, such
  72.       that we can have the time series data of the queries 
  73.       File name: <prefix>-<query_type>-<cf_id>-time_series.txt
  74.       Format:[type_id time_in_sec access_keyid].) type: bool default: false
  75.     -output_value_distribution (Out put the value size distribution, only
  76.       available for Put and Merge.
  77.       File name:
  78.       <prefix>-<query_type>-<cf_id>-accessed_value_size_distribution.txt
  79.       Format:[Number_of_value_size_between x and x+value_interval is: <the
  80.       count>]) type: bool default: false
  81.     -print_correlation (intput format: [correlation pairs][.,.]
  82.       Output the query correlations between the pairs of query types listed in
  83.       the parameter, input should select the operations from:
  84.       get, put, delete, single_delete, rangle_delete, merge. No space between
  85.       the pairs separated by comma. Example: =[get,get]... It will print out
  86.       the number of pairs of 'A after B' and the average time interval between
  87.       the two query.) type: string default: ""
  88.     -print_overall_stats ( Print the stats of the whole trace, like total
  89.       requests, keys, and etc.) type: bool default: true
  90.     -print_top_k_access (<top K of the variables to be printed> Print the top k
  91.       accessed keys, top k accessed prefix and etc.) type: int32 default: 1
  92.     -sample_ratio (If the trace size is extremely huge or user want to sample
  93.       the trace when analyzing, sample ratio can be set (0, 1.0]) type: double
  94.       default: 1
  95.     -trace_path (The trace file path.) type: string default: ""
  96.     -try_process_corrupted_trace (In default, trace_analyzer will exit if the
  97.       trace file is corrupted due to the unexpected tracing cases. If this
  98.       option is enabled, trace_analyzer will stop reading the trace file, and
  99.       start analyzing the read-in data.) type: bool default: false
  100.     -value_interval (To output the value distribution, we need to set the value
  101.       intervals and make the statistic of the value size distribution in
  102.       different intervals. The default is 8.) type: int32 default: 8

One Example

  1. ./trace_analyzer \
  2.   -analyze_get \
  3.   -output_access_count_stats \
  4.   -output_dir=/data/trace/result \
  5.   -output_key_stats \
  6.   -output_qps_stats \
  7.   -convert_to_human_readable_trace \
  8.   -output_value_distribution \
  9.   -output_key_distribution \
  10.   -print_overall_stats \
  11.   -print_top_k_access=3 \
  12.   -output_prefix=test \
  13.   -trace_path=/data/trace/trace

Another Example

  1. ./trace_analyzer -analyze_get -analyze_put -analyze_merge -analyze_delete \
  2. -analyze_single_delete -analyze_iterator -analyze_multiget \
  3. -output_access_count_stats -output_dir=./result -output_key_stats -output_qps_stats \
  4. -output_value_distribution -output_key_distribution -output_time_series -print_overall_stats \
  5. -print_top_k_access=3 -value_interval=1 -output_prefix=trace_test -trace_path=./trace_example

Query Type Options

User can specify which type queries that should be analyzed and use “-analyze_”.

Output Human Readable Traces

The original binary trace stores the encoded data structures and content, to interpret the trace, the tool should use the RocksDB library. Thus, to simplify the further analyzing of the trace, user can specify

  1. -convert_to_human_readable_trace

The original trace will be converted to a txt file, the content is “[type_id cf_id value_size time_in_micorsec ]”. If the key is not needed, user can specify “-no_key” to reduce the file size. This option is independent to all other option, once it is specified, the converted trace will be generated. If the original key is included, the txt file size might be similar or even larger than the original trace file size.

Input and Output Options

To analyze a trace file, user need to indicate the path to the trace file by

  1. -trace_path=<path to the trace>

To store the output files, user can specify a directory (make sure the directory exist before running the analyzer) to store these files

  1. -output_dir=<the path to the output directory>

If user wants to analyze the accessed keys together with the existing keyspace. User needs to specify a directory that stores the keyspace files. The file should be in the name of “.txt” and each line is one key. Usually, user can use the “./ldb scan” of the LDB tool to dump out all the existing keys. To specify the directory

  1. -key_space_dir=<the path to the key space directory>

To collect the output files more easily, user can specify the “prefix” for all the output files

  1. -output_prefix=<the prefix, like "trace1">

If user does not want to print out the general statistics to the screen, user can specify

  1. -no_print

The Analyzing Options

Currently, the trace_analyzer tool provides several different analyzing options to characterize the workload. Some of the results are directly printed out (options with prefix “-print”) others will output to the files (options with prefix “-output”). User can specify the combination of the options to analyze the trace. Note that, some of the analyzing options are very memory intensive (e.g., -output_time_series, -print_correlation, and -key_space_dir). If the memory is not enough, try to run them in different time.

The general information of the workloads like the total number of keys, the number of analyzed queries of each column family, the key and value size statistics (average and medium), the number of keys being accessed are printed to screen when this option is specified

  1. -print_overall_stats

To get the total access count of each key and the size of the value, user can specify

  1. -output_key_stats

It will output the access count of each key to a file and the keys are sorted in lexicographical order. Each key will be assigned with an integer as the ID for further processing

In some workloads, the composition of the keys has some common part. For example, in MyRocks, the first X bytes of the key is the table index_num. We can use the first X bytes to cut the keys into different prefix range. By specifying the number of bytes to cut the key space, the trace_analyzer will generate a file. One record in the file represents a cut of prefix, the corresponding KeyID, and the prefix content are stored. There will be two separate files if the -key_space_dir is specified. One file is for the accessed keys, the other one is for the whole key space. Usually, the prefix cut file is used together with the accessed_key_stats.txt and whole_key_stats.txt respectively.

  1. -output_prefix_cut=<number of bytes as prefix>

If user wants to visualize the accesses of the keys in the tracing timeline, user can specify:

  1. -output_time_series

Each access to one key will be stored as one record in the time series file.

If the user is interested to know about the detailed QPS changing during the tracing time, user can specify:

  1. -output_qps_stats

For each query type of one column family, one file with the query number per second will be generated. Also, one file with the QPS of each query type on all column families as well as the overall QPS are output to a separate file. The average QPS and peak QPS will be printed out.

Sometimes, user might be interested to know about the TOP statistics. user can specify

  1. -print_top_k_access

The top K accessed keys, the access number will be printed. Also, if the prefix_cut option is specified, the top K accessed prefix with their total access count are printed. At the same time, the top K prefix with the highest average access is printed.

If the user is interested to know about the value size distribution (only applicable for Put and Merge ) user can specify

  1. -output_value_distribution
  2. -value_interval

Since the value size varies a lot, User might just want to know how many values are in each value size range. User can specify the value_interval=x to generate the number of values between [0,x), [x,2x)……

The key size distribution is output to the file if user specify

  1. -output_key_distribution

Visualize the workload

After processing the trace with trace_analyzer, user will get a couple of output files. Some of the files can be used to visualize the workload such as the heatmap (the hotness or coldness of keys), time series graph (the overview of the key access in timeline), the QPS of the analyzed queries.

Here, we use the open source plot tool GNUPLOT as an example to generate the graphs. More details about the GNUPLOT can be found here (http://gnuplot.info/). User can directly write the GNUPLOT command to draw the graph, or to make it simple, user can use the following shell script to generate the GNUPLOT source file (before using the script, make sure the file name and some of the content are replaced with the effective ones).

To draw the heat map of accessed keys (note that, the following scripts need to be customized based on the file name, the arguments of your workloads, and the plotting style you want. They are just examples)

  1. #!/bin/bash
  3. # The query type
  4. ops="iterator"
  6. # The column family ID
  7. cf="9"
  9. # The column family name if known, if not, replace it with some prefix
  10. cf_name="rev:cf-assoc-deleter-id1-type"
  12. form="accessed"
  14. # The column number that will be plotted
  15. use="4"
  17. # The higher bound of Y-axis
  18. y=2
  20. # The higher bound of X-axis
  21. x=29233
  22. echo "set output '${cf_name}-${ops}-${form}-key_heatmap.png'" > plot-${cf_name}-${ops}-${form}-heatmap.gp
  23. echo "set term png size 2000,500" >>plot-${cf_name}-${ops}-${form}-heatmap.gp
  24. echo "set title 'CF: ${cf_name} ${form} Key Space Heat Map'">>plot-${cf_name}-${ops}-${form}-heatmap.gp
  25. echo "set xlabel 'Key Sequence'">>plot-${cf_name}-${ops}-${form}-heatmap.gp
  26. echo "set ylabel 'Key access count'">>plot-${cf_name}-${ops}-${form}-heatmap.gp
  27. echo "set yrange [0:$y]">>plot-${cf_name}-${ops}-${form}-heatmap.gp
  28. echo "set xrange [0:$x]">>plot-${cf_name}-${ops}-${form}-heatmap.gp
  30. # If the preifx cut is avialable, it will draw the prefix cut
  31. while read f1 f2
  32. do
  33.     echo "set arrow from $f1,0 to $f1,$y nohead lc rgb 'red'" >> plot-${cf_name}-${ops}-${form}-heatmap.gp
  34. done < "trace.1532381594728669-${ops}-${cf}-${form}_key_prefix_cut.txt"
  35. echo "plot 'trace.1532381594728669-${ops}-${cf}-${form}_key_stats.txt' using ${use} notitle w dots lt 2" >>plot-${cf_name}-${ops}-${form}-heatmap.gp
  36. gnuplot plot-${cf_name}-${ops}-${form}-heatmap.gp

Or you can directly us GNUPLOT script to plot out the key access distribution. Here is one example (plot_get_heat_map.gp):

  1. set term png size 2000,600 enhanced font 'Times New Roman, 40'
  2. set output '24-0-get-accessed-key_heatmap.png'
  3. set title 'Default' offset 0,-1
  4. set ylabel 'Key access count'
  5. set yrange [0:400]
  6. set xrange [0:60000000]
  7. plot 'trace.x-get-0-accessed_key_stats.txt' using 4 notitle w dots lt 2

To draw the time series map of accessed keys

  1. #!/bin/bash
  3. # The query type
  4. ops="iterator"
  6. # The higher bound of X-axis
  7. x=29233
  9. # The column family ID
  10. cf="8"
  12. # The column family name if known, if not, replace it with some prefix
  13. cf_name="rev:cf-assoc-deleter-id1-type"
  15. # The type of the output file
  16. form="time_series"
  18. # The column number that will be plotted
  19. use="3:2"
  21. # The total time of the tracing duration, in seconds
  22. y=88000
  23. echo "set output '${cf_name}-${ops}-${form}-key_heatmap.png'" > plot-${cf_name}-${ops}-${form}-heatmap.gp
  24. echo "set term png size 3000,3000" >>plot-${cf_name}-${ops}-${form}-heatmap.gp
  25. echo "set title 'CF: ${cf_name} time series'">>plot-${cf_name}-${ops}-${form}-heatmap.gp
  26. echo "set xlabel 'Key Sequence'">>plot-${cf_name}-${ops}-${form}-heatmap.gp
  27. echo "set ylabel 'Key access count'">>plot-${cf_name}-${ops}-${form}-heatmap.gp
  28. echo "set yrange [0:$y]">>plot-${cf_name}-${ops}-${form}-heatmap.gp
  29. echo "set xrange [0:$x]">>plot-${cf_name}-${ops}-${form}-heatmap.gp
  31. # If the preifx cut is avialable, it will draw the prefix cut
  32. while read f1 f2
  33. do
  34.     echo "set arrow from $f1,0 to $f1,$y nohead lc rgb 'red'" >> plot-${cf_name}-${ops}-${form}-heatmap.gp
  35. done < "trace.1532381594728669-${ops}-${cf}-accessed_key_prefix_cut.txt"
  36. echo "plot 'trace.1532381594728669-${ops}-${cf}-${form}.txt' using ${use} notitle w dots lt 2" >>plot-${cf_name}-${ops}-${form}-heatmap.gp
  37. gnuplot plot-${cf_name}-${ops}-${form}-heatmap.gp

Or you can use the GNUPLOT script as follows (plot_time_series.gp):

  1. set output 'new-trace_1-get-0-time_series.png'
  2. set term png size 1000,800 enhanced font 'Times New Roman, 32'
  3. set xlabel 'Key Sequence'
  4. set ylabel 'Time(second)' offset 1,0,0
  5. set yrange [0:89000]
  6. set xrange [0:500000]
  7. set format x '%.0tx10^%T'
  8. set xtics 100000
  9. set mxtics
  10. plot 'trace_1-get-0-time_series.txt' using 3:2 notitle w dots lt 2

To plot out the QPS

  1. #!/bin/bash
  3. # The query type
  4. ops="iterator"
  6. # The higher bound of the QPS
  7. y=5
  9. # The column family ID
  10. cf="9"
  12. # The column family name if known, if not, replace it with some prefix
  13. cf_name="rev:cf-assoc-deleter-id1-type"
  15. # The type of the output file
  16. form="qps_stats"
  18. # The column number that will be plotted
  19. use="1"
  21. # The total time of the tracing duration, in seconds
  22. x=88000
  23. echo "set output '${cf_name}-${ops}-${form}-IO_per_second.png'" > plot-${cf_name}-${ops}-${form}-heatmap.gp
  24. echo "set term png size 2000,1200" >>plot-${cf_name}-${ops}-${form}-heatmap.gp
  25. echo "set title 'CF: ${cf_name} QPS Over Time'">>plot-${cf_name}-${ops}-${form}-heatmap.gp
  26. echo "set xlabel 'Time in second'">>plot-${cf_name}-${ops}-${form}-heatmap.gp
  27. echo "set ylabel 'QPS'">>plot-${cf_name}-${ops}-${form}-heatmap.gp
  28. echo "set yrange [0:$y]">>plot-${cf_name}-${ops}-${form}-heatmap.gp
  29. echo "set xrange [0:$x]">>plot-${cf_name}-${ops}-${form}-heatmap.gp
  30. echo "plot 'trace.1532381594728669-${ops}-${cf}-${form}.txt' using ${use} notitle with linespoints" >>plot-${cf_name}-${ops}-${form}-heatmap.gp
  31. gnuplot plot-${cf_name}-${ops}-${form}-heatmap.gp

Or you can use the following GNUPLOT script (plot_qps_all.gp): First, you can use the following python script to preprocess the Get, Put, Merge QPS raw file. Here is one example to preprocess the raw file:

  1. import numpy as np
  2. s1 = "trace_2-qps_stats.txt"
  3. s2 = "./result_sample/"
  4. s3 = s2+s1
  5. l1 = []
  6. l2 = []
  7. l3 = []
  8. l4 = []
  9. l5 = []
  10. l6 = []
  11. f = open(s3,'r')
  12. start=1539923212
  13. sum = 0
  14. result = list()
  15. for line in open(s3):
  16.     line = f.readline()
  17.     r1,r2,r3,r4,r5,r6,r7,r8,r9 = [str(i) for i in line.split()]
  18.     l1.append(int(r1)*30)
  19.     l2.append(int(r2)*30)
  20.     l3.append(int(r3)*30)
  21.     l4.append(int(r4)*30)
  22.     l5.append((int(r7)+int(r8))*30)
  23.     l6.append(int(r9)*30)
  24. f.close()
  25. s4 = s2+"new-"+s1
  26. fw = open(s4,'w')
  27. for i in range(0,len(l1)):
  28.     fw.write("%d %d %d %d %d %d %d\n"%(start, l1[i],l2[i],l3[i],l4[i],l5[i],l6[i]))
  29.     start = start+1
  30. print len(l1)

Then, you can use gnuplot to plot out the results:

  1. set term png size 2000,1000 enhanced font 'Times New Roman, 40'
  2. set output 'all_qps_100.png'
  3. set ylabel 'Queries/sec'
  4. set yrange [0:100]
  5. plot 'new-trace_2-get-0-qps_stats.txt' title 'Get' with lines lw 3 lt 6,\
  6.      'new-trace_2-put-0-qps_stats.txt' title 'Put' with lines lw 3 lt -1,\
  7.      'new-trace_2-merge-0-qps_stats.txt' title 'Merge' with lines linestyle 3 lw 3 lt 5

Here is the GNUPLOT script for key and value size distribution plot (plot_get_key_size_distribution.gp): First, you can use the following python script to pre-process the raw file:

  1. import numpy as np
  2. s1 = "trace_2-get-0-accessed_key_size_distribution.txt"
  3. s2 = "./result_sample/"
  4. s3 = s2+s1
  5. l1 = []
  6. l2 = []
  7. f = open(s3,'r')
  8. sum = 0
  9. result = list()
  10. for line in open(s3):
  11.     line = f.readline()
  12.     r1,r2= [str(i) for i in line.split()]
  13.     l1.append(int(r1))
  14.     l2.append(int(r2))
  15.     sum = sum + int(r2)
  16. f.close()
  17. s4 = s2+"new-"+s1
  18. fw = open(s4,'w')
  19. cur = 0.0
  20. count = 0
  21. for i in range(0,200):
  22.     if count < len(l1) and l1[count] == i:
  23.         cur = float(l2[count]) / sum
  24.         count = count+1
  25.         fw.write("%d %.12f\n"%(i,cur))
  26.     else:
  27.         fw.write("%d %.12f\n"%(i,0))
  28. print sum

Then, you can plot them out

  1. set term png size 1000,600 enhanced font 'Times New Roman, 36'
  2. set output 'key-size-distribution.png'
  3. set xlabel 'Key size (bytes)'
  4. set yrange [0:1.01]
  5. set xrange [0:180]
  6. set key right bottom
  7. plot 'new-trace_2-get-0-accessed_key_size_distribution.txt' using 1:2 title 'Get' with linespoints lw 5 lt 6 pointtype 2 pointsize 3,\
  8.  'new-trace_2-put-0-accessed_key_size_distribution.txt' using 1:2 title 'Put' with linespoints lw 5 lt -1 pointtype 7 pointsize 3,\
  9. 'new-trace_2-merge-0-accessed_key_size_distribution.txt' using 1:2 title 'Merge' with linespoints lw 5 lt 1 pointtype 1 pointsize 3

Model the Workloads

We can use different tools, script, and models to fit the workload statistics. Typically, user can use the distributions of key access count and key-range access count to fit in the model. Also, QPS can be modeled. Here, we use Matlab as an example to fit the key access count, key-range access count, and QPS. Note that, before we model the key-range hotness, we need to decide how to cut the whole key-space into key-ranges. Our suggestion is that the key-range size can be similar to the number of KV-pairs in a SST file.

User can try different distributions to fit the model. In this example, we use two-term exponential model to fit the access distribution, and two-term sin() to fit the QPS

To fit the key access statistics to the model and get the statistics, user can run the following script:

  1. % This script is used to fit the key access count distribution
  2. % to the two-term exponential distirbution and get the parameters
  4. % The input file with surfix: accessed_key_stats.txt
  5. fileID = fopen('trace.1531329742187378-get-4-accessed_key_stats.txt');
  6. txt = textscan(fileID,'%f %f %f %f');
  7. fclose(fileID);
  9. % Get the number of keys that has access count x
  10. t2=sort(txt{4},'descend');
  12. % The number of access count that is used to fit the data
  13. % The value depends on the accuracy demond of your model fitting
  14. % and the value of count should be always not greater than
  15. % the size of t2
  16. count=30000;
  18. % Generate the access count x
  19. x=1:1:count;
  20. x=x';
  22. % Adjust the matrix and uniformed
  23. y=t2(1:count);
  24. y=y/(sum(y));
  25. figure;
  27. % fitting the data to the exp2 model
  28. f=fit(x,y,'exp2')
  30. %plot out the original data and fitted line to compare
  31. plot(f,x,y);

To fit the key access count distribution to the model, user can run the following script:

  1. % This script is used to fit the key access count distribution
  2. % to the two-term exponential distirbution and get the parameters
  4. % The input file with surfix: key_count_distribution.txt
  5. fileID = fopen('trace-get-9-accessed_key_count_distribution.txt');
  6. input = textscan(fileID,'%s %f %s %f');
  7. fclose(fileID);
  9. % Get the number of keys that has access count x
  10. t2=sort(input{4},'descend');
  12. % The number of access count that is used to fit the data
  13. % The value depends on the accuracy demond of your model fitting
  14. % and the value of count should be always not greater than
  15. % the size of t2
  16. count=100;
  18. % Generate the access count x
  19. x=1:1:count;
  20. x=x';
  21. y=t2(1:count);
  23. % Adjust the matrix and uniformed
  24. y=y/(sum(y));
  25. y=y(1:count);
  26. x=x(1:count);
  28. figure;
  29. % fitting the data to the exp2 model
  30. f=fit(x,y,'exp2')
  32. %plot out the original data and fitted line to compare
  33. plot(f,x,y);

To fit the key-range average access count to the model, user can run the following script:

  1. % This script is used to fit the prefix average access count distribution
  2. % to the two-term exponential distirbution and get the parameters
  4. % The input file with surfix: accessed_key_prefix_cut.txt
  5. fileID = fopen('trace-get-4-accessed_key_prefix_cut.txt');
  6. txt = textscan(fileID,'%f %f %f %f %s');
  7. fclose(fileID);
  9. % The per key access (average) of each prefix, sorted
  10. t2=sort(txt{4},'descend');
  12. % The number of access count that is used to fit the data
  13. % The value depends on the accuracy demond of your model fitting
  14. % and the value of count should be always not greater than
  15. % the size of t2
  16. count=1000;
  18. % Generate the access count x
  19. x=1:1:count;
  20. x=x';
  22. % Adjust the matrix and uniformed
  23. y=t2(0:count);
  24. y=y/(sum(y));
  25. x=x(1:count);
  27. % fitting the data to the exp2 model
  28. figure;
  29. f=fit(x,y,'exp2')
  31. %plot out the original data and fitted line to compare
  32. plot(f,x,y);

To fit the QPS to the model, user can run the following script:

  1. % This script is used to fit the qps of the one query in one of the column
  2. % family to the sin'x' model. 'x' can be 1 to 10. With the higher value
  3. % of the 'x', you can get more accurate fitting of the qps. However,
  4. % the model will be more complex and some times will be overfitted.
  5. % The suggestion is to use sin1 or sin2
  7. % The input file shoud with surfix: qps_stats.txt
  8. fileID = fopen('trace-get-4-io_stats.txt');
  9. txt = textscan(fileID,'%f');
  10. fclose(fileID);
  11. t1=txt{1};
  13. % The input is the queries per second. If you directly use the qps
  14. % you may got a high value of noise. Here, 'n' is the number of qps
  15. % that you want to combined to one average value, such that you can
  16. % reduce it to queries per n*seconds.
  17. n=10;
  18. s1 = size(t1, 1);
  19. M  = s1 - mod(s1, n);
  20. t2  = reshape(t1(1:M), n, []);
  21. y = transpose(sum(t2, 1) / n);
  23. % Up to this point, you need to move the data down to the x-axis,
  24. % the offset is the ave. So the model will be 
  25. % s(x) =  a1*sin(b1*x+c1) + a2*sin(b2*x+c2) + ave
  26. ave = mean(y);
  27. y=y-ave;
  29. % Adjust the matrix
  30. count = size(y,1);
  31. x=1:1:count;
  32. x=x';
  34. % Fit the model to 'sin2' in this example and draw the point and
  35. % fitted line to compare
  36. figure;
  37. s = fit(x,y,'sin2')
  38. plot(s,x,y);

Users can use the model for further analyzing or use it to generate the synthetic workload.

Synthetic Workload Generation based on Models

In the previous section, users can use the fitting functions of Matlab to fit the traced workload to different models, such that we can use a set of parameters and functions to profile the workload. We focus on the 4 variables to profile the workload: 1) value size; 2) KV-pair access hotness; 3) QPS; 4) Iterator scan length. According to our current research, the value size and Iterator scan length follows the Generalized Pareto Distribution. The probability density function is: f(x) = (1/sigma) * (1+k * (x-theta)\sigma)^(-1-1/k). The KV-pair access follows power-law, in which about 80% of the KV-pair has an access count less than 4. We sort the keys based on the access count in descending order and fit them to the models. The two-term power model fit the KV-pair access distribution best. The probability density function is: f(x) = ax^b+c. The Sine function fits the QPS best. F(x) = A * sin(B * x + C) + D.

Here is one example of the parameters we get from the workload collected in Facebook social graph:

  1. Value Size: sigma = 226.409, k = 0.923$, theta = 0
  2. KV-pair Access: a = 0.001636, b = -0.7094 , and c = 3.217*10^-9
  3. QPS: $A = 147.9, B = 8.3*10^-5, C = -1.734, D = 1064.2
  4. Iterator scan length: sigma = 1.747, k = 0.0819, theta = 0

We developed a benchmark called “mixgraph” in db_bench, which can use the four set of parameters to generate the synthetic workload. The workload is statistically similar to the original one. Note that, only the workload that can be fit to the models used for the four variables can be used in the mixgraph. For example, if the value size follows the power distribution instead of Generalized Pareto Distribution, we cannot use the mixgraph to generate the workloads.

More details about the workloads, benchmarks, and models, please refer to the published paper at FAST2020 (here)

To enable the “mixgraph” benchmark, user needs to specify:

  1. ./db_bench —benchmarks="mixgraph"

To set the parameters of the value size distribution (Generalized Pareto Distribution only), user needs to specify:

  1. -value_k=<> -value_sigma=<> -value_theta=<>

To set the parameters of the KV-pair access distribution (power distribution only and C==0), user needs to specify:

  1. -key_dist_a=<> -key_dist_b=<>

To set the parameters of key-range distributions which follows the two-term exponential distribution (f(x)=a_exp(b_x)+c_exp(d_x)), user needs to specify both the key-range hotness distribution parameters and the number of key-ranges (keyrange_num):

  1. -keyrange_dist_a=<> -keyrange_dist_b=<> -keyrange_dist_c=<> -keyrange_dist_d=<> -keyrange_num=<>

To set the parameters of the QPS (Sine), user needs to specify:

  1. -sine_a=<> -sine_b=<> -sine_c=<> -sine_d=<> -sine_mix_rate_interval_milliseconds=<>

The mix rate is used to set the time interval that how lone we should correct the rate according to the distribution, the smaller it is, the better it will fit.

To set the parameters of the iterator scan length distribution (Generalized Pareto Distribution only), user needs to specify:

  1. -iter_k=<> -iter_sigma=<> -iter_theta=<>

User need to specify the query ratio between the Get, Put, and Seek. Such that we can generate the mixed workload that can be similar to the social graph workload (so called mix graph). Make sure that the sum of the ratio is 1.

  1. -mix_get_ratio=<> -mix_put_ratio=<> -mix_seek_ratio=<>

Finally, user need to specify how many queries they want to execute:

  1. -reads=<>

and what’s the total KV-pairs are in the current testing DB

  1. -num=<>

The num together with the aforementioned distributions decided the queries.

Here is one example that can be directly used to generate the workload which simulate the queries of ZippyDB described in the published paper link. The workloads has 0.42 billion queries and 50 million KV-pairs in total. We use 30 key-ranges. Note that, if user runs the benchmark following the 24 hours Sine period, it will take about 22-24 hours. In order to speedup the benchmarking, user can increase the sine_d to a larger value such as 45000 to increase the workload intensiveness and also reduce the sine_b accordingly.

  1. ./db_bench \
  2.   --benchmarks="mixgraph" \
  3.   -use_direct_io_for_flush_and_compaction=true \
  4.   -use_direct_reads=true \
  5.   -cache_size=268435456 \
  6.   -keyrange_dist_a=14.18 \
  7.   -keyrange_dist_b=-2.917 \
  8.   -keyrange_dist_c=0.0164 \
  9.   -keyrange_dist_d=-0.08082 \
  10.   -keyrange_num=30 \
  11.   -value_k=0.2615 \
  12.   -value_sigma=25.45 \
  13.   -iter_k=2.517 \
  14.   -iter_sigma=14.236 \
  15.   -mix_get_ratio=0.85 \
  16.   -mix_put_ratio=0.14 \
  17.   -mix_seek_ratio=0.01 \
  18.   -sine_mix_rate_interval_milliseconds=5000 \
  19.   -sine_a=1000 \
  20.   -sine_b=0.000073 \
  21.   -sine_d=4500 \
  22.   --perf_level=2 \
  23.   -reads=420000000 \
  24.   -num=50000000 \
  25.   -key_size=48