Text to visualization

This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, join the discussion on the OpenSearch forum.

The OpenSearch Dashboards Assistant can create visualizations using natural language instructions.

Configuration

To configure text to visualization, use the following steps.

Step 1: Enable text to visualization

To enable text to visualization, configure the following opensearch_dashboards.yml setting:

  1. assistant.text2viz.enabled: true

copy

Step 2: Create the agents

To orchestrate text to visualization, you’ll need to create the necessary agents. Create a workflow template for creating all necessary text-to-visualization agents by sending the following request:

Request

  1. POST /_plugins/_flow_framework/workflow
  2. {
  3.   "name": "Text to visualization agents",
  4.   "description": "This template is to create all Agents required for text to visualization",
  5.   "use_case": "REGISTER_AGENTS",
  6.   "version": {
  7.     "template": "1.0.0",
  8.     "compatibility": [
  9.       "2.18.0",
  10.       "3.0.0"
  11.     ]
  12.   },
  13.   "workflows": {
  14.     "provision": {
  15.       "user_params": {},
  16.       "nodes": [
  17.         {
  18.           "id": "create_claude_connector",
  19.           "type": "create_connector",
  20.           "previous_node_inputs": {},
  21.           "user_inputs": {
  22.             "credential": {
  23.               "access_key": "<YOUR_ACCESS_KEY>",
  24.               "secret_key": "<YOUR_SECRET_KEY>",
  25.               "session_token": "<YOUR_SESSION_TOKEN>"
  26.             },
  27.             "parameters": {
  28.               "endpoint": "bedrock-runtime.us-east-1.amazonaws.com",
  29.               "content_type": "application/json",
  30.               "auth": "Sig_V4",
  31.               "max_tokens_to_sample": "8000",
  32.               "service_name": "bedrock",
  33.               "temperature": "0.0000",
  34.               "response_filter": "$.content[0].text",
  35.               "region": "us-east-1",
  36.               "anthropic_version": "bedrock-2023-05-31"
  37.             },
  38.             "version": "1",
  39.             "name": "Claude haiku runtime Connector",
  40.             "protocol": "aws_sigv4",
  41.             "description": "The connector to BedRock service for claude model",
  42.             "actions": [
  43.               {
  44.                 "action_type": "predict",
  45.                 "method": "POST",
  46.                 "url": "https://bedrock-runtime.us-east-1.amazonaws.com/model/anthropic.claude-3-haiku-20240307-v1:0/invoke",
  47.                 "headers": {
  48.                   "content-type": "application/json",
  49.                   "x-amz-content-sha256": "required"
  50.                 },
  51.                 "request_body": "{\"messages\":[{\"role\":\"user\",\"content\":[{\"type\":\"text\",\"text\":\"${parameters.prompt}\"}]}],\"anthropic_version\":\"${parameters.anthropic_version}\",\"max_tokens\":${parameters.max_tokens_to_sample}}"
  52.               }
  53.             ]
  54.           }
  55.         },
  56.         {
  57.           "id": "register_claude_model",
  58.           "type": "register_remote_model",
  59.           "previous_node_inputs": {
  60.             "create_claude_connector": "connector_id"
  61.           },
  62.           "user_inputs": {
  63.             "name": "claude-haiku",
  64.             "description": "Claude model",
  65.             "deploy": true
  66.           }
  67.         },
  68.         {
  69.           "id": "create_t2vega_tool",
  70.           "type": "create_tool",
  71.           "previous_node_inputs": {
  72.             "register_claude_model": "model_id"
  73.           },
  74.           "user_inputs": {
  75.             "parameters": {
  76.               "prompt": "You're an expert at creating vega-lite visualization. No matter what the user asks, you should reply with a valid vega-lite specification in json.\nYour task is to generate Vega-Lite specification in json based on the given sample data, the schema of the data, the PPL query to get the data and the user's input.\nLet's start from dimension and metric/date. Now I have a question, I already transfer it to PPL and query my Opensearch cluster. \nThen I get data. For the PPL, it will do aggregation like \"stats AVG(field_1) as avg, COUNT(field_2) by field_3, field_4, field_5\". \nIn this aggregation, the metric is [avg, COUNT(field_2)] , and then we judge the type of field_3,4,5. If only field_5 is type related to date, the dimension is [field_3, field_4], and date is [field_5]\nFor example, stats SUM(bytes) by span(timestamp, 1w), machine.os, response, then SUM(bytes) is metric and span(timestamp, 1w) is date, while machine.os, response are dimensions.\nNotice: Some fields like 'span()....' will be the date, but not metric and dimension. \nAnd one field will only count once in dimension count. You should always pick field name from schema\nTo summarize,\nA dimension is a categorical variable that is used to group, segment, or categorize data. It is typically a qualitative attribute that provides context for metrics and is used to slice and dice data to see how different categories perform in relation to each other.\nThe dimension is not date related fields. The dimension and date are very closed. The only difference is date is related to datetime, while dimension is not.\nA metric is a quantitative measure used to quantify or calculate some aspect of the data. Metrics are numerical and typically represent aggregated values like sums, averages, counts, or other statistical calculations.\n\nIf a ppl doesn't have aggregation using 'stats', then each field in output is dimension.\nOtherwise, if a ppl use aggregation using 'stats' but doesn't group by using 'by', then each field in output is metric.\n\nThen for each given PPL, you could give the metric and dimension and date. One field will in only one of the metric, dimension or date.\n\nThen according to the metric number and dimension number of PPL result, you should first format the entrance code by metric_number, dimension_number, and date_number. For example, if metric_number = 1, dimension_number = 2, date_number=1, then the entrance code is  121.\nI define several use case categories here according to the entrance code.\nFor each category, I will define the entrance condition (number of metric and dimension)\nI will also give some defined attribute of generated vega-lite. Please refer to it to generate vega-lite.\n\nType 1:\nEntrance code: <1, 1, 0>\nDefined Attributes:\n      {\n      \"title\": \"<title>\",\n      \"description\": \"<description>\",\n      \"mark\": \"bar\",\n      \"encoding\": {\n        \"x\": {\n          \"field\": \"<metric name>\",\n          \"type\": \"quantitative\"\n        },\n        \"y\": {\n          \"field\": \"<dimension name>\",\n          \"type\": \"nominal\"\n        }\n      },\n    }\n\nType 2:\nEntrance code: <1, 2, 0>\nDefined Attributes:\n{\n      \"mark\": \"bar\",\n      \"encoding\": {\n        \"x\": {\n          \"field\": \"<metric 1>\",\n          \"type\": \"quantitative\"\n        },\n        \"y\": {\n          \"field\": \"<dimension 1>\",\n          \"type\": \"nominal\"\n        },\n        \"color\": {\n          \"field\": \"<dimension 2>\",\n          \"type\": \"nominal\"\n        }\n      }\n    }\n\n\nType 3\nEntrance code: <3, 1, 0>\nDefined Attributes:\n{\n    \"mark\": \"point\",\n    \"encoding\": {\n        \"x\": {\n            \"field\": \"<metric 1>\",\n            \"type\": \"quantitative\"\n        },\n        \"y\": {\n            \"field\": \"<metric 2>\",\n            \"type\": \"quantitative\"\n        },\n        \"size\": {\n            \"field\": \"<metric 3>\",\n            \"type\": \"quantitative\"\n        },\n        \"color\": {\n            \"field\": \"<dimension 1>\",\n            \"type\": \"nominal\"\n        }\n    }\n}\n\nType 4\nEntrance code: <2, 1, 0>\nDefined Attributes:\n{\n    \"mark\": \"point\",\n    \"encoding\": {\n        \"x\": {\n            \"field\": \"<mtric 1>\",\n            \"type\": \"quantitative\"\n        },\n        \"y\": {\n            \"field\": \"<metric 2>\",\n            \"type\": \"quantitative\"\n        },\n        \"color\": {\n            \"field\": \"<dimension 1>\",\n            \"type\": \"nominal\"\n        }\n    }\n}\n\nType 5:\nEntrance code: <2, 1, 1>\nDefined Attributes:\n{\n      \"layer\": [\n        {\n          \"mark\": \"bar\",\n          \"encoding\": {\n            \"x\": {\n              \"field\": \"<date 1>\",\n              \"type\": \"temporal\"\n            },\n            \"y\": {\n              \"field\": \"<metric 1>\",\n              \"type\": \"quantitative\",\n              \"axis\": {\n                \"title\": \"<metric 1 name>\"\n              }\n            },\n            \"color\": {\n              \"field\": \"<dimension 1>\",\n              \"type\": \"nominal\"\n            }\n          }\n        },\n        {\n          \"mark\": {\n            \"type\": \"line\",\n            \"color\": \"red\"\n          },\n          \"encoding\": {\n            \"x\": {\n              \"field\": \"<date 1>\",\n              \"type\": \"temporal\"\n            },\n            \"y\": {\n              \"field\": \"<metric 2>\",\n              \"type\": \"quantitative\",\n              \"axis\": {\n                \"title\": \"<metric 2 name>\",\n                \"orient\": \"right\"\n              }\n            },\n            \"color\": {\n              \"field\": \"<dimension 1>\",\n              \"type\": \"nominal\"\n            }\n          }\n        }\n      ],\n      \"resolve\": {\n        \"scale\": {\n          \"y\": \"independent\"\n        }\n      }\n    }\n\nType 6:\nEntrance code: <2, 0, 1>\nDefined Attributes:\n{\n      \"title\": \"<title>\",\n      \"description\": \"<description>\",\n      \"layer\": [\n        {\n          \"mark\": \"area\",\n          \"encoding\": {\n            \"x\": {\n              \"field\": \"<date 1>\",\n              \"type\": \"temporal\"\n            },\n            \"y\": {\n              \"field\": \"<metric 1>\",\n              \"type\": \"quantitative\",\n              \"axis\": {\n                \"title\": \"<metric 1 name>\"\n              }\n            }\n          }\n        },\n        {\n          \"mark\": {\n            \"type\": \"line\",\n            \"color\": \"black\"\n          },\n          \"encoding\": {\n            \"x\": {\n              \"field\": \"date\",\n              \"type\": \"temporal\"\n            },\n            \"y\": {\n              \"field\": \"metric 2\",\n              \"type\": \"quantitative\",\n              \"axis\": {\n                \"title\": \"<metric 2 name>\",\n                \"orient\": \"right\"\n              }\n            }\n          }\n        }\n      ],\n      \"resolve\": {\n        \"scale\": {\n          \"y\": \"independent\"\n        }\n      }\n    }\n    \nType 7:\nEntrance code: <1, 0, 1>\nDefined Attributes:\n{\n      \"title\": \"<title>\",\n      \"description\": \"<description>\",\n      \"mark\": \"line\",\n      \"encoding\": {\n        \"x\": {\n          \"field\": \"<date 1>\",\n          \"type\": \"temporal\",\n          \"axis\": {\n            \"title\": \"<date name>\"\n          }\n        },\n        \"y\": {\n          \"field\": \"<metric 1>\",\n          \"type\": \"quantitative\",\n          \"axis\": {\n            \"title\": \"<metric name>\"\n          }\n        }\n      }\n    }\n\nType 8:\nEntrance code: <1, 1, 1>\nDefined Attributes:\n{\n      \"title\": \"<title>\",\n      \"description\": \"<description>\",\n      \"mark\": \"line\",\n      \"encoding\": {\n        \"x\": {\n          \"field\": \"<date 1>\",\n          \"type\": \"temporal\",\n          \"axis\": {\n            \"title\": \"<date name>\"\n          }\n        },\n        \"y\": {\n          \"field\": \"<metric 1>\",\n          \"type\": \"quantitative\",\n          \"axis\": {\n            \"title\": \"<metric name>\"\n          }\n        },\n        \"color\": {\n          \"field\": \"<dimension 1>\",\n          \"type\": \"nominal\",\n          \"legend\": {\n            \"title\": \"<dimension name>\"\n          }\n        }\n      }\n    }\n\nType 9:\nEntrance code: <1, 2, 1>\nDefined Attributes:\n{\n      \"title\": \"<title>\",\n      \"description\": \"<description>\",\n      \"mark\": \"line\",\n      \"encoding\": {\n        \"x\": {\n          \"field\": \"<date 1>\",\n          \"type\": \"temporal\",\n          \"axis\": {\n            \"title\": \"<date name>\"\n          }\n        },\n        \"y\": {\n          \"field\": \"<metric 1>\",\n          \"type\": \"quantitative\",\n          \"axis\": {\n            \"title\": \"<metric 1>\"\n          }\n        },\n        \"color\": {\n          \"field\": \"<dimension 1>\",\n          \"type\": \"nominal\",\n          \"legend\": {\n            \"title\": \"<dimension 1>\"\n          }\n        },\n        \"facet\": {\n          \"field\": \"<dimension 2>\",\n          \"type\": \"nominal\",\n          \"columns\": 2\n        }\n      }\n    }\n\nType 10:\nEntrance code: all other code\nAll others type.\nUse a table to show the result\n\n\nBesides, here are some requirements:\n1. Do not contain the key called 'data' in vega-lite specification.\n2. If mark.type = point and shape.field is a field of the data, the definition of the shape should be inside the root \"encoding\" object, NOT in the \"mark\" object, for example, {\"encoding\": {\"shape\": {\"field\": \"field_name\"}}}\n3. Please also generate title and description\n\nThe sample data in json format:\n${parameters.sampleData}\n\nThis is the schema of the data:\n${parameters.dataSchema}\n\nThe user used this PPL query to get the data: ${parameters.ppl}\n\nThe user's question is: ${parameters.input_question}\n\nNotice: Some fields like 'span()....' will be the date, but not metric and dimension. \nAnd one field will only count once in dimension count.  You should always pick field name from schema.\n And when you code is <2, 1, 0>, it belongs type 4.\n  And when you code is <1, 2, 0>, it belongs type 9.\n\n\nNow please reply a valid vega-lite specification in json based on above instructions.\nPlease return the number of dimension, metric and date. Then choose the type. \nPlease also return the type.\nFinally return the vega-lite specification according to the type.\nPlease make sure all the key in the schema matches the word I given. \nYour answer format should be:\nNumber of metrics:[list the metric name here, Don't use duplicate name]  <number of metrics {a}>  \nNumber of dimensions:[list the dimension name here]  <number of dimension {b}> \nNumber of dates:[list the date name here]  <number of dates {c}> \nThen format the entrance code by: <Number of metrics, Number of dimensions, Number of dates>\nType and its entrance code: <type number>: <its entrance code>\nThen apply the vega-lite requirements of the type.\n<vega-lite> {here is the vega-lite json} </vega-lite>\n\nAnd don't use 'transformer' in your vega-lite and wrap your vega-lite json in <vega-lite> </vega-lite> tags\n"
  77.             },
  78.             "name": "Text2Vega",
  79.             "type": "MLModelTool"
  80.           }
  81.         },
  82.         {
  83.           "id": "create_instruction_based_t2vega_tool",
  84.           "type": "create_tool",
  85.           "previous_node_inputs": {
  86.             "register_claude_model": "model_id"
  87.           },
  88.           "user_inputs": {
  89.             "parameters": {
  90.               "prompt": "You're an expert at creating vega-lite visualization. No matter what the user asks, you should reply with a valid vega-lite specification in json.\nYour task is to generate Vega-Lite specification in json based on the given sample data, the schema of the data, the PPL query to get the data and the user's input.\n\nBesides, here are some requirements:\n1. Do not contain the key called 'data' in vega-lite specification.\n2. If mark.type = point and shape.field is a field of the data, the definition of the shape should be inside the root \"encoding\" object, NOT in the \"mark\" object, for example, {\"encoding\": {\"shape\": {\"field\": \"field_name\"}}}\n3. Please also generate title and description\n\nThe sample data in json format:\n${parameters.sampleData}\n\nThis is the schema of the data:\n${parameters.dataSchema}\n\nThe user used this PPL query to get the data: ${parameters.ppl}\n\nThe user's input question is: ${parameters.input_question}\nThe user's instruction on the visualization is: ${parameters.input_instruction}\n\nNow please reply a valid vega-lite specification in json based on above instructions.\nPlease only contain vega-lite in your response.\n"
  91.             },
  92.             "name": "Text2Vega",
  93.             "type": "MLModelTool"
  94.           }
  95.         },
  96.         {
  97.           "id": "t2vega_agent",
  98.           "type": "register_agent",
  99.           "previous_node_inputs": {
  100.             "create_t2vega_tool": "tools"
  101.           },
  102.           "user_inputs": {
  103.             "parameters": {},
  104.             "type": "flow",
  105.             "name": "t2vega agent",
  106.             "description": "this is the t2vega agent that has a set of rules to generate the visualizations"
  107.           }
  108.         },
  109.         {
  110.           "id": "t2vega_instruction_based_agent",
  111.           "type": "register_agent",
  112.           "previous_node_inputs": {
  113.             "create_instruction_based_t2vega_tool": "tools"
  114.           },
  115.           "user_inputs": {
  116.             "parameters": {},
  117.             "type": "flow",
  118.             "name": "t2vega instruction based agent",
  119.             "description": "this is the t2vega agent that supports instructions"
  120.           }
  121.         }
  122.       ]
  123.     }
  124.   }
  125. }

copy

Use the workflow ID returned in the response to provision the resources:

  1. POST /_plugins/_flow_framework/workflow/<workflow_id>/_provision

copy

To view the status of the workflow and all created resources, send the following request:

  1. GET /_plugins/_flow_framework/workflow/<workflow_id>/_status

copy

Step 3: Configure the root agent

Next, configure a root agent for text to visualization:

  1. POST /.plugins-ml-config/_doc/os_text2vega
  2. {
  3.   "type": "os_chat_root_agent",
  4.   "configuration": {
  5.     "agent_id": "<ROOT_AGENT_ID>"
  6.   }
  7. }

copy

Configure the agent to receive user instructions for creating visualizations:

  1. POST /.plugins-ml-config/_doc/os_text2vega_with_instructions
  2. {
  3.   "type": "os_chat_root_agent",
  4.   "configuration": {
  5.     "agent_id": "<ROOT_AGENT_ID>"
  6.   }
  7. }

copy

This example demonstrates a system index. In security-enabled domains, only superadmins have permissions to execute this code. For information about making superadmin calls, see System indexes. For access permissions, contact your system administrator.

Step 4: Test the agent

You can verify that the agent was created successfully by calling the agent with an example payload:

  1. POST /_plugins/_ml/agents/<ROOT_AGENT_ID>/_execute
  2. {
  3.   "parameters": {
  4.     "input_question": "find unique visitors and average bytes every 3 hours",
  5.     "input_instruction": "display with different layers, use independent scale for different layers, display unique visitors with light blue bar chart",
  6.     "ppl": "source=opensearch_dashboards_sample_data_ecommerce| stats DISTINCT_COUNT(user) as unique_visitors, AVG(taxful_total_price) as avg_bytes by span(order_date, 3h)",
  7.     "sampleData": """[{\"unique_visitors\":15,\"avg_bytes\":90.98684210526316,\"span(order_date,3h)\":\"2024-04-25 00:00:00\"},{\"unique_visitors\":14,\"avg_bytes\":72.72083333333333,\"span(order_date,3h)\":\"2024-04-25 03:00:00\"}]""",
  8.     "dataSchema": """[{\"name\":\"unique_visitors\",\"type\":\"integer\"},{\"name\":\"avg_bytes\",\"type\":\"double\"},{\"name\":\"span(order_date,3h)\",\"type\":\"timestamp\"}]"""
  9.   }
  10. }

copy

Generating a visualization from text

You can generate a visualization from text by calling the /api/assistant/text2vega API endpoint. The input_instruction parameter is optional:

  1. POST /api/assistant/text2vega
  2. {
  3.   "input_instruction": "<input_instruction>",
  4.   "input_question": "<input_question>",
  5.   "ppl": "<ppl_query>",
  6.   "dataSchema": "<data_schema_of_ppl_response>",
  7.   "sampleData": "<sample_data_of_ppl_response>"
  8. }

copy

The following table describes the Text to Visualization API parameters.

ParameterRequired/OptionalDescription
input_questionRequiredThe user’s original question used to generate the corresponding Piped Processing Language (PPL) query.
pplRequiredThe generated PPL query that retrieves the data required for the visualization.
dataSchemaRequiredDescribes the structure and types of the data fields in the visualization output, based on the PPL response.
sampleDataRequiredProvides sample entries from the data that will populate the visualization.
input_instructionOptionalSpecifies the styling instructions, such as colors, for the visualization.

Generating visualizations from text in OpenSearch Dashboards

To generate visualizations from text in OpenSearch Dashboards, use the following steps:

  1. On the top menu bar, go to OpenSearch Dashboards > Visualize and then select Create visualization.

  2. In the New Visualization dialog, select Natural language, as shown in the following image.

    Create a visualization by selecting natural language

  3. From the data sources dropdown list, select a data source, as shown in the following image.

    Create a visualization by selecting natural language

  4. In the text box on the upper right, enter a question using natural language. A new visualization is generated, as shown in the following image.

    Create a visualization by selecting natural language

  5. To modify the generated visualization, select Edit visual. In the Edit visual dialog, enter the desired modifications and then select Apply, as shown in the following image.

    Create a visualization by selecting natural language

    The visualization is updated, as shown in the following image.

    Create a visualization by selecting natural language