Debug Point

Debug point is a piece of code, inserted into FE or BE code, when program running into this code,

it can change variables or behaviors of the program.

It is mainly used for unit test or regression test when it is impossible to trigger some exceptions through normal means.

Each debug point has a name, the name can be whatever you want, there are swithes to enable and disable debug points,

and you can also pass data to debug points.

Both FE and BE support debug point, and after inserting debug point code, recompilation of FE or BE is needed.

Code Example

FE example

  1. private Status foo() {
  2.     // dbug_fe_foo_do_nothing is the debug point name
  3.     // when it's active, DebugPointUtil.isEnable("dbug_fe_foo_do_nothing") returns true
  4.     if (DebugPointUtil.isEnable("dbug_fe_foo_do_nothing")) {
  5.         return Status.Nothing;
  6.     }
  8.     do_foo_action();
  10.     return Status.Ok;
  11. }

BE example

  1. void Status foo() {
  2.      // dbug_be_foo_do_nothing is the debug point name
  3.      // when it's active, DBUG_EXECUTE_IF will execute the code block
  4.      DBUG_EXECUTE_IF("dbug_be_foo_do_nothing",  { return Status.Nothing; });
  6.      do_foo_action();
  8.      return Status.Ok;
  9. }

Global Config

To enable debug points globally, we need to set enable_debug_points to true,

enable_debug_points is located in FE’s fe.conf and BE’s be.conf.

Activate A Specified Debug Point

After debug points are enabled globally, a http request with a debug point name should be send to FE or BE node,
only after that, when the program running into the specified debug point, related code can be executed.

API

  1. POST /api/debug_point/add/{debug_point_name}[?timeout=<int>&execute=<int>]

Query Parameters

  • debug_point_name Debug point name. Mandatory parameter.

  • timeout Timeout in seconds. When timeout, the debug point will be deactivated. Default is -1, never timeout. Optional.

  • execute After activating, the max times the debug point can be executed. Default is -1, unlimited times. Optional.

Request body

None

Response

  1. {
  2.     msg: "OK",
  3.     code: 0
  4. }

Examples

After activating debug point foo, executed no more than five times.

  1. curl -X POST "http://127.0.0.1:8030/api/debug_point/add/foo?execute=5"

Pass Custom Parameters

When activating debug point, besides “timeout” and “execute” mentioned above, passing custom parameters is also allowed.
A parameter is a key-value pair in the form of “key=value” in url path, after debug point name glued by character ‘?’.
See examples below.

API

  1. POST /api/debug_point/add/{debug_point_name}[?k1=v1&k2=v2&k3=v3...]
  • k1=v1
    k1 is parameter name
    v1 is parameter value
    multiple key-value pairs are concatenated by &

Request body

None

Response

  1. {
  2.     msg: "OK",
  3.     code: 0
  4. }

Examples

Assuming a FE node with configuration http_port=8030 in fe.conf,
the following http request activates a debug point named foo in FE node and passe parameter percent and duration:

NOTE: User name and password may be needed.

  1. curl -u root: -X POST "http://127.0.0.1:8030/api/debug_point/add/foo?percent=0.5&duration=3"
  1. NOTE:
  2. 1. Inside FE and BE code, names and values of parameters are taken as strings.
  3. 2. Parameter names and values are case sensitive in http request and FE/BE code.
  4. 3. FE and BE share same url paths of REST API, it's just their IPs and Ports are different.

Use parameters in FE and BE code

Following request activates debug point OlapTableSink.write_random_choose_sink in FE and passes parameter needCatchUp and sinkNum:

  1. curl -u root: -X POST "http://127.0.0.1:8030/api/debug_point/add/OlapTableSink.write_random_choose_sink?needCatchUp=true&sinkNum=3"

The code in FE checks debug point OlapTableSink.write_random_choose_sink and gets parameter values:

  1. private void debugWriteRandomChooseSink(Tablet tablet, long version, Multimap<Long, Long> bePathsMap) {
  2.     DebugPoint debugPoint = DebugPointUtil.getDebugPoint("OlapTableSink.write_random_choose_sink");
  3.     if (debugPoint == null) {
  4.         return;
  5.     }
  6.     boolean needCatchup = debugPoint.param("needCatchUp", false);
  7.     int sinkNum = debugPoint.param("sinkNum", 0);
  8.     ...
  9. }

Following request activates debug point TxnManager.prepare_txn.random_failed in BE and passes parameter percent:

  1. curl -X POST "http://127.0.0.1:8040/api/debug_point/add/TxnManager.prepare_txn.random_failed?percent=0.7

The code in BE checks debug point TxnManager.prepare_txn.random_failed and gets parameter value:

  1. DBUG_EXECUTE_IF("TxnManager.prepare_txn.random_failed",
  2.         {if (rand() % 100 < (100 * dp->param("percent", 0.5))) {
  3.                 LOG_WARNING("TxnManager.prepare_txn.random_failed random failed");
  4.                 return Status::InternalError("debug prepare txn random failed");
  5.         }}
  6. );

Disable Debug Point

API

  1.     POST /api/debug_point/remove/{debug_point_name}

Query Parameters

  • debug_point_name Debug point name. Mandatory parameter.

Request body

None

Response

  1. {
  2.     msg: "OK",
  3.     code: 0
  4. }

Examples

Disable debug point foo.

  1. curl -X POST "http://127.0.0.1:8030/api/debug_point/remove/foo"

Clear Debug Points

API

  1. POST /api/debug_point/clear

Request body

None

Response

  1. {
  2.     msg: "OK",
  3.     code: 0
  4. }

Examples

  1. curl -X POST "http://127.0.0.1:8030/api/debug_point/clear"

Debug Points in Regression Test

In community’s CI system, enable_debug_points configuration of FE and BE are true by default.

The Regression test framework also provides methods to activate and deactivate a particular debug point,
they are declared as below:

  1. // "name" is the debug point to activate, "params" is a list of key-value pairs passed to debug point
  2. def enableDebugPointForAllFEs(String name, Map<String, String> params = null);
  3. def enableDebugPointForAllBEs(String name, Map<String, String> params = null);
  4. // "name" is the debug point to deactivate
  5. def disableDebugPointForAllFEs(String name);
  6. def disableDebugPointForAllFEs(String name);

enableDebugPointForAllFEs() or enableDebugPointForAllBEs() needs to be called before the test actions you want to generate error,
and disableDebugPointForAllFEs() or disableDebugPointForAllBEs() needs to be called afterward.

Concurrent Issue

Enabled debug points affects FE or BE globally, which could cause other concurrent tests to fail unexpectedly in your pull request.
To avoid this, there’s a convention that regression tests using debug points must be in directory regression-test/suites/fault_injection_p0,
and their group name must be “nonConcurrent”, as these regression tests will be executed serially by pull request workflow.

Examples

  1. // .groovy file of the test case must be in regression-test/suites/fault_injection_p0
  2. // and the group name must be 'nonConcurrent'
  3. suite('debugpoint_action', 'nonConcurrent') {
  4.     try {
  5.         // Activate debug point named "PublishVersionDaemon.stop_publish" in all FE
  6.         // and pass parameter "timeout"
  7.         // "execute" and "timeout" are pre-existing parameters, usage is mentioned above
  8.         GetDebugPoint().enableDebugPointForAllFEs('PublishVersionDaemon.stop_publish', [timeout:1])
  10.         // Activate debug point named "Tablet.build_tablet_report_info.version_miss" in all BE
  11.         // and pass parameter "tablet_id", "version_miss" and "timeout"
  12.         GetDebugPoint().enableDebugPointForAllBEs('Tablet.build_tablet_report_info.version_miss',
  13.                                                   [tablet_id:'12345', version_miss:true, timeout:1])
  15.         // Test actions which will run into debug point and generate error
  16.         sql """CREATE TABLE tbl_1 (k1 INT, k2 INT)
  17.                DUPLICATE KEY (k1)
  18.                DISTRIBUTED BY HASH(k1)
  19.                BUCKETS 3
  20.                PROPERTIES ("replication_allocation" = "tag.location.default: 1");
  21.             """
  22.         sql "INSERT INTO tbl_1 VALUES (1, 10)"
  23.         sql "INSERT INTO tbl_1 VALUES (2, 20)"
  24.         order_qt_select_1_1 'SELECT * FROM tbl_1'
  26.     } finally {
  27.         // Deactivate debug points
  28.         GetDebugPoint().disableDebugPointForAllFEs('PublishVersionDaemon.stop_publish')
  29.         GetDebugPoint().disableDebugPointForAllBEs('Tablet.build_tablet_report_info.version_miss')
  30.     }
  31. }