Transforms APIs

Aside from using OpenSearch Dashboards, you can also use the REST API to create, start, stop, and complete other operations relative to transform jobs.

Table of contents

Create a transform job

Creates a transform job.

Sample Request

  1. PUT _plugins/_transform/<transform_id>
  3. {
  4.   "transform": {
  5.   "enabled": true,
  6.   "schedule": {
  7.     "interval": {
  8.     "period": 1,
  9.     "unit": "Minutes",
  10.     "start_time": 1602100553
  11.     }
  12.   },
  13.   "description": "Sample transform job",
  14.   "source_index": "sample_index",
  15.   "target_index": "sample_target",
  16.   "data_selection_query": {
  17.     "match_all": {}
  18.   },
  19.   "page_size": 1,
  20.   "groups": [
  21.     {
  22.     "terms": {
  23.       "source_field": "customer_gender",
  24.       "target_field": "gender"
  25.     }
  26.     },
  27.     {
  28.     "terms": {
  29.       "source_field": "day_of_week",
  30.       "target_field": "day"
  31.     }
  32.     }
  33.   ],
  34.   "aggregations": {
  35.     "quantity": {
  36.     "sum": {
  37.       "field": "total_quantity"
  38.     }
  39.     }
  40.   }
  41.   }
  42. }

Sample Response

  1. {
  2.   "_id": "sample",
  3.   "_version": 7,
  4.   "_seq_no": 13,
  5.   "_primary_term": 1,
  6.   "transform": {
  7.     "transform_id": "sample",
  8.     "schema_version": 7,
  9.     "schedule": {
  10.       "interval": {
  11.         "start_time": 1621467964243,
  12.         "period": 1,
  13.         "unit": "Minutes"
  14.       }
  15.     },
  16.     "metadata_id": null,
  17.     "updated_at": 1621467964243,
  18.     "enabled": true,
  19.     "enabled_at": 1621467964243,
  20.     "description": "Sample transform job",
  21.     "source_index": "sample_index",
  22.     "data_selection_query": {
  23.       "match_all": {
  24.         "boost": 1.0
  25.       }
  26.     },
  27.     "target_index": "sample_target",
  28.     "roles": [],
  29.     "page_size": 1,
  30.     "groups": [
  31.       {
  32.         "terms": {
  33.           "source_field": "customer_gender",
  34.           "target_field": "gender"
  35.         }
  36.       },
  37.       {
  38.         "terms": {
  39.           "source_field": "day_of_week",
  40.           "target_field": "day"
  41.         }
  42.       }
  43.     ],
  44.     "aggregations": {
  45.       "quantity": {
  46.         "sum": {
  47.           "field": "total_quantity"
  48.         }
  49.       }
  50.     }
  51.   }
  52. }

You can specify the following options in the HTTP request body:

OptionData TypeDescriptionRequired
enabledBooleanIf true, the transform job is enabled at creation.No
scheduleJSONThe schedule the transform job runs on.Yes
start_timeIntegerThe Unix epoch time of the transform job’s start time.Yes
descriptionStringDescribes the transform job.No
metadata_idStringAny metadata to be associated with the transform job.No
source_indexStringThe source index whose data to transform.Yes
target_indexStringThe target index the newly transformed data is added into. You can create a new index or update an existing one.Yes
data_selection_queryJSONThe query DSL to use to filter a subset of the source index for the transform job. See query DSL for more information.Yes
page_sizeIntegerThe number of fields to transform at a time. Higher number means higher performance but requires more memory and can cause higher latency. (Default: 1)Yes
groupsArraySpecifies the grouping(s) to use in the transform job. Supported groups are terms, histogram, and date_histogram. For more information, see Bucket Aggregations.Yes if not using aggregations
source_fieldStringThe field(s) to transformYes
aggregationsJSONThe aggregations to use in the transform job. Supported aggregations are: sum, max, min, value_count, avg, scripted_metric, and percentiles. For more information, see Metric Aggregations.Yes if not using groups

Update a transform job

Updates a transform job if transform_id already exists.

Sample Request

  1. PUT _plugins/_transform/<transform_id>
  3. {
  4.   "transform": {
  5.   "enabled": true,
  6.   "schedule": {
  7.     "interval": {
  8.     "period": 1,
  9.     "unit": "Minutes",
  10.     "start_time": 1602100553
  11.     }
  12.   },
  13.   "description": "Sample transform job",
  14.   "source_index": "sample_index",
  15.   "target_index": "sample_target",
  16.   "data_selection_query": {
  17.     "match_all": {}
  18.   },
  19.   "page_size": 1,
  20.   "groups": [
  21.     {
  22.     "terms": {
  23.       "source_field": "customer_gender",
  24.       "target_field": "gender"
  25.     }
  26.     },
  27.     {
  28.     "terms": {
  29.       "source_field": "day_of_week",
  30.       "target_field": "day"
  31.     }
  32.     }
  33.   ],
  34.   "aggregations": {
  35.     "quantity": {
  36.     "sum": {
  37.       "field": "total_quantity"
  38.     }
  39.     }
  40.   }
  41.   }
  42. }

Sample Response

  1. {
  2.   "_id": "sample",
  3.   "_version": 2,
  4.   "_seq_no": 14,
  5.   "_primary_term": 1,
  6.   "transform": {
  7.     "transform_id": "sample",
  8.     "schema_version": 7,
  9.     "schedule": {
  10.       "interval": {
  11.         "start_time": 1602100553,
  12.         "period": 1,
  13.         "unit": "Minutes"
  14.       }
  15.     },
  16.     "metadata_id": null,
  17.     "updated_at": 1621889843874,
  18.     "enabled": true,
  19.     "enabled_at": 1621889843874,
  20.     "description": "Sample transform job",
  21.     "source_index": "sample_index",
  22.     "data_selection_query": {
  23.       "match_all": {
  24.         "boost": 1.0
  25.       }
  26.     },
  27.     "target_index": "sample_target",
  28.     "roles": [],
  29.     "page_size": 1,
  30.     "groups": [
  31.       {
  32.         "terms": {
  33.           "source_field": "customer_gender",
  34.           "target_field": "gender"
  35.         }
  36.       },
  37.       {
  38.         "terms": {
  39.           "source_field": "day_of_week",
  40.           "target_field": "day"
  41.         }
  42.       }
  43.     ],
  44.     "aggregations": {
  45.       "quantity": {
  46.         "sum": {
  47.           "field": "total_quantity"
  48.         }
  49.       }
  50.     }
  51.   }
  52. }

The Update operation supports the following URL parameters:

ParameterDescriptionRequired
if_seq_noOnly perform the transform operation if the last operation that changed the transform job has the specified sequence number.No
if_primary_termOnly perform the transform operation if the last operation that changed the transform job has the specified sequence term.No

Get a transform job’s details

Returns a transform job’s details.

Sample Request

  1. GET _plugins/_transform/<transform_id>

Sample Response

  1. {
  2.   "_id": "sample",
  3.   "_version": 7,
  4.   "_seq_no": 13,
  5.   "_primary_term": 1,
  6.   "transform": {
  7.     "transform_id": "sample",
  8.     "schema_version": 7,
  9.     "schedule": {
  10.       "interval": {
  11.         "start_time": 1621467964243,
  12.         "period": 1,
  13.         "unit": "Minutes"
  14.       }
  15.     },
  16.     "metadata_id": null,
  17.     "updated_at": 1621467964243,
  18.     "enabled": true,
  19.     "enabled_at": 1621467964243,
  20.     "description": "Sample transform job",
  21.     "source_index": "sample_index",
  22.     "data_selection_query": {
  23.       "match_all": {
  24.         "boost": 1.0
  25.       }
  26.     },
  27.     "target_index": "sample_target",
  28.     "roles": [],
  29.     "page_size": 1,
  30.     "groups": [
  31.       {
  32.         "terms": {
  33.           "source_field": "customer_gender",
  34.           "target_field": "gender"
  35.         }
  36.       },
  37.       {
  38.         "terms": {
  39.           "source_field": "day_of_week",
  40.           "target_field": "day"
  41.         }
  42.       }
  43.     ],
  44.     "aggregations": {
  45.       "quantity": {
  46.         "sum": {
  47.           "field": "total_quantity"
  48.         }
  49.       }
  50.     }
  51.   }
  52. }

You can also get details of all transform jobs by omitting transform_id.

Sample Request

  1. GET _plugins/_transform/

Sample Response

  1. {
  2.   "total_transforms": 1,
  3.   "transforms": [
  4.     {
  5.       "_id": "sample",
  6.       "_seq_no": 13,
  7.       "_primary_term": 1,
  8.       "transform": {
  9.         "transform_id": "sample",
  10.         "schema_version": 7,
  11.         "schedule": {
  12.           "interval": {
  13.             "start_time": 1621467964243,
  14.             "period": 1,
  15.             "unit": "Minutes"
  16.           }
  17.         },
  18.         "metadata_id": null,
  19.         "updated_at": 1621467964243,
  20.         "enabled": true,
  21.         "enabled_at": 1621467964243,
  22.         "description": "Sample transform job",
  23.         "source_index": "sample_index",
  24.         "data_selection_query": {
  25.           "match_all": {
  26.             "boost": 1.0
  27.           }
  28.         },
  29.         "target_index": "sample_target",
  30.         "roles": [],
  31.         "page_size": 1,
  32.         "groups": [
  33.           {
  34.             "terms": {
  35.               "source_field": "customer_gender",
  36.               "target_field": "gender"
  37.             }
  38.           },
  39.           {
  40.             "terms": {
  41.               "source_field": "day_of_week",
  42.               "target_field": "day"
  43.             }
  44.           }
  45.         ],
  46.         "aggregations": {
  47.           "quantity": {
  48.             "sum": {
  49.               "field": "total_quantity"
  50.             }
  51.           }
  52.         }
  53.       }
  54.     }
  55.   ]
  56. }

You can specify these options as the GET API operation’s URL parameters to filter results:

ParameterDescriptionRequired
fromThe starting index to search from. (Default: 0)No
sizeSpecifies the amount of results to return (Default: 10)No
searchThe search term to use to filter results.No
sortFieldThe field to sort results with.No
sortDirectionSpecifies the direction to sort results in. Can be ASC or DESC. (Default: ASC)No

For example, this request returns two results starting from the eighth index.

Sample Request

  1. GET _plugins/_transform?size=2&from=8

Sample Response

  1. {
  2.   "total_transforms": 18,
  3.   "transforms": [
  4.     {
  5.       "_id": "sample8",
  6.       "_seq_no": 93,
  7.       "_primary_term": 1,
  8.       "transform": {
  9.         "transform_id": "sample8",
  10.         "schema_version": 7,
  11.         "schedule": {
  12.           "interval": {
  13.             "start_time": 1622063596812,
  14.             "period": 1,
  15.             "unit": "Minutes"
  16.           }
  17.         },
  18.         "metadata_id": "y4hFAB2ZURQ2dzY7BAMxWA",
  19.         "updated_at": 1622063657233,
  20.         "enabled": false,
  21.         "enabled_at": null,
  22.         "description": "Sample transform job",
  23.         "source_index": "sample_index3",
  24.         "data_selection_query": {
  25.           "match_all": {
  26.             "boost": 1.0
  27.           }
  28.         },
  29.         "target_index": "sample_target3",
  30.         "roles": [],
  31.         "page_size": 1,
  32.         "groups": [
  33.           {
  34.             "terms": {
  35.               "source_field": "customer_gender",
  36.               "target_field": "gender"
  37.             }
  38.           },
  39.           {
  40.             "terms": {
  41.               "source_field": "day_of_week",
  42.               "target_field": "day"
  43.             }
  44.           }
  45.         ],
  46.         "aggregations": {
  47.           "quantity": {
  48.             "sum": {
  49.               "field": "total_quantity"
  50.             }
  51.           }
  52.         }
  53.       }
  54.     },
  55.     {
  56.       "_id": "sample9",
  57.       "_seq_no": 98,
  58.       "_primary_term": 1,
  59.       "transform": {
  60.         "transform_id": "sample9",
  61.         "schema_version": 7,
  62.         "schedule": {
  63.           "interval": {
  64.             "start_time": 1622063598065,
  65.             "period": 1,
  66.             "unit": "Minutes"
  67.           }
  68.         },
  69.         "metadata_id": "x8tCIiYMTE3veSbIJkit5A",
  70.         "updated_at": 1622063658388,
  71.         "enabled": false,
  72.         "enabled_at": null,
  73.         "description": "Sample transform job",
  74.         "source_index": "sample_index4",
  75.         "data_selection_query": {
  76.           "match_all": {
  77.             "boost": 1.0
  78.           }
  79.         },
  80.         "target_index": "sample_target4",
  81.         "roles": [],
  82.         "page_size": 1,
  83.         "groups": [
  84.           {
  85.             "terms": {
  86.               "source_field": "customer_gender",
  87.               "target_field": "gender"
  88.             }
  89.           },
  90.           {
  91.             "terms": {
  92.               "source_field": "day_of_week",
  93.               "target_field": "day"
  94.             }
  95.           }
  96.         ],
  97.         "aggregations": {
  98.           "quantity": {
  99.             "sum": {
  100.               "field": "total_quantity"
  101.             }
  102.           }
  103.         }
  104.       }
  105.     }
  106.   ]
  107. }

Start a transform job

Transform jobs created using the API are automatically enabled, but if you ever need to enable a job, you can use the start API operation.

Sample Request

  1. POST _plugins/_transform/<transform_id>/_start

Sample Response

  1. {
  2.   "acknowledged": true
  3. }

Stop a transform job

Stops/disables a transform job.

Sample Request

  1. POST _plugins/_transform/<transform_id>/_stop

Sample Response

  1. {
  2.   "acknowledged": true
  3. }

Get the status of a transform job

Returns the status and metadata of a transform job.

Sample Request

  1. GET _plugins/_transform/<transform_id>/_explain

Sample Response

  1. {
  2.   "sample": {
  3.     "metadata_id": "PzmjweME5xbgkenl9UpsYw",
  4.     "transform_metadata": {
  5.       "transform_id": "sample",
  6.       "last_updated_at": 1621883525873,
  7.       "status": "finished",
  8.       "failure_reason": "null",
  9.       "stats": {
  10.         "pages_processed": 0,
  11.         "documents_processed": 0,
  12.         "documents_indexed": 0,
  13.         "index_time_in_millis": 0,
  14.         "search_time_in_millis": 0
  15.       }
  16.     }
  17.   }
  18. }

Preview a transform job’s results

Returns a preview of what a transformed index would look like.

Sample Request

  1. POST _plugins/_transform/_preview
  3. {
  4.   "transform": {
  5.   "enabled": false,
  6.   "schedule": {
  7.     "interval": {
  8.     "period": 1,
  9.     "unit": "Minutes",
  10.     "start_time": 1602100553
  11.     }
  12.   },
  13.   "description": "test transform",
  14.   "source_index": "sample_index",
  15.   "target_index": "sample_target",
  16.   "data_selection_query": {
  17.     "match_all": {}
  18.   },
  19.   "page_size": 10,
  20.   "groups": [
  21.     {
  22.     "terms": {
  23.       "source_field": "customer_gender",
  24.       "target_field": "gender"
  25.     }
  26.     },
  27.     {
  28.     "terms": {
  29.       "source_field": "day_of_week",
  30.       "target_field": "day"
  31.     }
  32.     }
  33.   ],
  34.   "aggregations": {
  35.     "quantity": {
  36.     "sum": {
  37.       "field": "total_quantity"
  38.     }
  39.     }
  40.   }
  41.   }
  42. }

Sample Response

  1. {
  2.   "documents" : [
  3.   {
  4.     "quantity" : 862.0,
  5.     "gender" : "FEMALE",
  6.     "day" : "Friday"
  7.   },
  8.   {
  9.     "quantity" : 682.0,
  10.     "gender" : "FEMALE",
  11.     "day" : "Monday"
  12.   },
  13.   {
  14.     "quantity" : 772.0,
  15.     "gender" : "FEMALE",
  16.     "day" : "Saturday"
  17.   },
  18.   {
  19.     "quantity" : 669.0,
  20.     "gender" : "FEMALE",
  21.     "day" : "Sunday"
  22.   },
  23.   {
  24.     "quantity" : 887.0,
  25.     "gender" : "FEMALE",
  26.     "day" : "Thursday"
  27.   }
  28.   ]
  29. }

Delete a transform job

Deletes a transform job. This operation does not delete the source or target indices.

Sample Request

  1. DELETE _plugins/_transform/<transform_id>

Sample Response

  1. {
  2.   "took": 205,
  3.   "errors": false,
  4.   "items": [
  5.     {
  6.       "delete": {
  7.         "_index": ".opensearch-ism-config",
  8.         "_type": "_doc",
  9.         "_id": "sample",
  10.         "_version": 4,
  11.         "result": "deleted",
  12.         "forced_refresh": true,
  13.         "_shards": {
  14.           "total": 2,
  15.           "successful": 1,
  16.           "failed": 0
  17.         },
  18.         "_seq_no": 6,
  19.         "_primary_term": 1,
  20.         "status": 200
  21.       }
  22.     }
  23.   ]
  24. }