Agents and tools tutorial

Introduced 2.13

The following tutorial illustrates creating a flow agent for retrieval-augmented generation (RAG). A flow agent runs its configured tools sequentially, in the order specified. In this example, you’ll create an agent with two tools:

  1. VectorDBTool: The agent will use this tool to retrieve OpenSearch documents relevant to the user question. You’ll ingest supplementary information into an OpenSearch index. To facilitate vector search, you’ll deploy a text embedding model that translates text into vector embeddings. OpenSearch will translate the ingested documents into embeddings and store them in the index. When you provide a user question to the agent, the agent will construct a query from the question, run vector search on the OpenSearch index, and pass the relevant retrieved documents to the MLModelTool.
  2. MLModelTool: The agent will run this tool to connect to a large language model (LLM) and send the user query augmented with OpenSearch documents to the model. In this example, you’ll use the Anthropic Claude model hosted on Amazon Bedrock. The LLM will then answer the question based on its knowledge and the provided documents.

Prerequisites

To use the memory feature, first configure the following cluster settings. This tutorial assumes that you have no dedicated machine learning (ML) nodes:

  1. PUT _cluster/settings
  2. {
  3.   "persistent": {
  4.     "plugins.ml_commons.only_run_on_ml_node": "false",
  5.     "plugins.ml_commons.memory_feature_enabled": "true"
  6.   }
  7. }

copy

For more information, see ML Commons cluster settings.

Step 1: Register and deploy a text embedding model

You need a text embedding model to facilitate vector search. For this tutorial, you’ll use one of the OpenSearch-provided pretrained models. When selecting a model, note its dimensionality because you’ll need to provide it when creating an index.

In this tutorial, you’ll use the huggingface/sentence-transformers/all-MiniLM-L12-v2 model, which generates 384-dimensional dense vector embeddings. To register and deploy the model, send the following request:

  1. POST /_plugins/_ml/models/_register?deploy=true
  2. {
  3.   "name": "huggingface/sentence-transformers/all-MiniLM-L12-v2",
  4.   "version": "1.0.1",
  5.   "model_format": "TORCH_SCRIPT"
  6. }

copy

Registering a model is an asynchronous task. OpenSearch returns a task ID for this task:

  1. {
  2.   "task_id": "aFeif4oB5Vm0Tdw8yoN7",
  3.   "status": "CREATED"
  4. }

You can check the status of the task by calling the Tasks API:

  1. GET /_plugins/_ml/tasks/aFeif4oB5Vm0Tdw8yoN7

copy

Once the task is complete, the task state changes to COMPLETED and the Tasks API response includes a model ID for the deployed model:

  1. {
  2.   "model_id": "aVeif4oB5Vm0Tdw8zYO2",
  3.   "task_type": "REGISTER_MODEL",
  4.   "function_name": "TEXT_EMBEDDING",
  5.   "state": "COMPLETED",
  6.   "worker_node": [
  7.     "4p6FVOmJRtu3wehDD74hzQ"
  8.   ],
  9.   "create_time": 1694358489722,
  10.   "last_update_time": 1694358499139,
  11.   "is_async": true
  12. }

Step 2: Create an ingest pipeline

To translate text into vector embeddings, you’ll set up an ingest pipeline. The pipeline translates the text field and writes the resulting vector embeddings into the embedding field. Create the pipeline by specifying the model_id from the previous step in the following request:

  1. PUT /_ingest/pipeline/test-pipeline-local-model
  2. {
  3.   "description": "text embedding pipeline",
  4.   "processors": [
  5.     {
  6.       "text_embedding": {
  7.         "model_id": "aVeif4oB5Vm0Tdw8zYO2",
  8.         "field_map": {
  9.           "text": "embedding"
  10.         }
  11.       }
  12.     }
  13.   ]
  14. }

Step 3: Create a k-NN index and ingest data

Now you’ll ingest supplementary data into an OpenSearch index. In OpenSearch, vectors are stored in a k-NN index. You can create a k-NN index by sending the following request:

  1. PUT my_test_data
  2. {
  3.   "mappings": {
  4.     "properties": {
  5.       "text": {
  6.         "type": "text"
  7.       },
  8.       "embedding": {
  9.         "type": "knn_vector",
  10.         "dimension": 384
  11.       }
  12.     }
  13.   },
  14.   "settings": {
  15.     "index": {
  16.       "knn.space_type": "cosinesimil",
  17.       "default_pipeline": "test-pipeline-local-model",
  18.       "knn": "true"
  19.     }
  20.   }
  21. }

copy

Then, ingest data into the index by using a bulk request:

  1. POST _bulk
  2. {"index": {"_index": "my_test_data", "_id": "1"}}
  3. {"text": "Chart and table of population level and growth rate for the Ogden-Layton metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\nThe current metro area population of Ogden-Layton in 2023 is 750,000, a 1.63% increase from 2022.\nThe metro area population of Ogden-Layton in 2022 was 738,000, a 1.79% increase from 2021.\nThe metro area population of Ogden-Layton in 2021 was 725,000, a 1.97% increase from 2020.\nThe metro area population of Ogden-Layton in 2020 was 711,000, a 2.16% increase from 2019."}
  4. {"index": {"_index": "my_test_data", "_id": "2"}}
  5. {"text": "Chart and table of population level and growth rate for the New York City metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of New York City in 2023 is 18,937,000, a 0.37% increase from 2022.\\nThe metro area population of New York City in 2022 was 18,867,000, a 0.23% increase from 2021.\\nThe metro area population of New York City in 2021 was 18,823,000, a 0.1% increase from 2020.\\nThe metro area population of New York City in 2020 was 18,804,000, a 0.01% decline from 2019."}
  6. {"index": {"_index": "my_test_data", "_id": "3"}}
  7. {"text": "Chart and table of population level and growth rate for the Chicago metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Chicago in 2023 is 8,937,000, a 0.4% increase from 2022.\\nThe metro area population of Chicago in 2022 was 8,901,000, a 0.27% increase from 2021.\\nThe metro area population of Chicago in 2021 was 8,877,000, a 0.14% increase from 2020.\\nThe metro area population of Chicago in 2020 was 8,865,000, a 0.03% increase from 2019."}
  8. {"index": {"_index": "my_test_data", "_id": "4"}}
  9. {"text": "Chart and table of population level and growth rate for the Miami metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Miami in 2023 is 6,265,000, a 0.8% increase from 2022.\\nThe metro area population of Miami in 2022 was 6,215,000, a 0.78% increase from 2021.\\nThe metro area population of Miami in 2021 was 6,167,000, a 0.74% increase from 2020.\\nThe metro area population of Miami in 2020 was 6,122,000, a 0.71% increase from 2019."}
  10. {"index": {"_index": "my_test_data", "_id": "5"}}
  11. {"text": "Chart and table of population level and growth rate for the Austin metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Austin in 2023 is 2,228,000, a 2.39% increase from 2022.\\nThe metro area population of Austin in 2022 was 2,176,000, a 2.79% increase from 2021.\\nThe metro area population of Austin in 2021 was 2,117,000, a 3.12% increase from 2020.\\nThe metro area population of Austin in 2020 was 2,053,000, a 3.43% increase from 2019."}
  12. {"index": {"_index": "my_test_data", "_id": "6"}}
  13. {"text": "Chart and table of population level and growth rate for the Seattle metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of Seattle in 2023 is 3,519,000, a 0.86% increase from 2022.\\nThe metro area population of Seattle in 2022 was 3,489,000, a 0.81% increase from 2021.\\nThe metro area population of Seattle in 2021 was 3,461,000, a 0.82% increase from 2020.\\nThe metro area population of Seattle in 2020 was 3,433,000, a 0.79% increase from 2019."}

copy

Step 4: Create a connector to an externally hosted model

You’ll need an LLM to generate responses to user questions. An LLM is too large for an OpenSearch cluster, so you’ll create a connection to an externally hosted LLM. For this example, you’ll create a connector to the Anthropic Claude model hosted on Amazon Bedrock:

  1. POST /_plugins/_ml/connectors/_create
  2. {
  3.   "name": "BedRock test claude Connector",
  4.   "description": "The connector to BedRock service for claude model",
  5.   "version": 1,
  6.   "protocol": "aws_sigv4",
  7.   "parameters": {
  8.       "region": "us-east-1",
  9.       "service_name": "bedrock",
  10.       "anthropic_version": "bedrock-2023-05-31",
  11.       "endpoint": "bedrock.us-east-1.amazonaws.com",
  12.       "auth": "Sig_V4",
  13.       "content_type": "application/json",
  14.       "max_tokens_to_sample": 8000,
  15.       "temperature": 0.0001,
  16.       "response_filter": "$.completion"
  17.   },
  18.   "credential": {
  19.       "access_key": "<bedrock_access_key>",
  20.       "secret_key": "<bedrock_secret_key>",
  21.       "session_token": "<bedrock_session_token>"
  22.   },
  23.   "actions": [
  24.     {
  25.       "action_type": "predict",
  26.       "method": "POST",
  27.       "url": "https://bedrock-runtime.us-east-1.amazonaws.com/model/anthropic.claude-v2/invoke",
  28.       "headers": { 
  29.         "content-type": "application/json",
  30.         "x-amz-content-sha256": "required"
  31.       },
  32.       "request_body": "{\"prompt\":\"${parameters.prompt}\", \"max_tokens_to_sample\":${parameters.max_tokens_to_sample}, \"temperature\":${parameters.temperature},  \"anthropic_version\":\"${parameters.anthropic_version}\" }"
  33.     }
  34.   ]
  35. }

copy

The response contains the connector ID for the newly created connector:

  1. {
  2.   "connector_id": "a1eMb4kBJ1eYAeTMAljY"
  3. }

Step 5: Register and deploy the externally hosted model

Like the text embedding model, an LLM needs to be registered and deployed to OpenSearch. To set up the externally hosted model, first create a model group for this model:

  1. POST /_plugins/_ml/model_groups/_register
  2. {
  3.     "name": "test_model_group_bedrock",
  4.     "description": "This is a public model group"
  5. }

copy

The response contains the model group ID that you’ll use to register a model to this model group:

  1. {
  2.  "model_group_id": "wlcnb4kBJ1eYAeTMHlV6",
  3.  "status": "CREATED"
  4. }

Next, register and deploy the externally hosted Claude model:

  1. POST /_plugins/_ml/models/_register?deploy=true
  2. {
  3.     "name": "Bedrock Claude V2 model",
  4.     "function_name": "remote",
  5.     "model_group_id": "wlcnb4kBJ1eYAeTMHlV6",
  6.     "description": "test model",
  7.     "connector_id": "a1eMb4kBJ1eYAeTMAljY"
  8. }

copy

Similarly to Step 1, the response contains a task ID that you can use to check the status of the deployment. Once the model is deployed, the status changes to COMPLETED and the response includes the model ID for the Claude model:

  1. {
  2.   "model_id": "NWR9YIsBUysqmzBdifVJ",
  3.   "task_type": "REGISTER_MODEL",
  4.   "function_name": "remote",
  5.   "state": "COMPLETED",
  6.   "worker_node": [
  7.     "4p6FVOmJRtu3wehDD74hzQ"
  8.   ],
  9.   "create_time": 1694358489722,
  10.   "last_update_time": 1694358499139,
  11.   "is_async": true
  12. }

To test the LLM, send the following predict request:

  1. POST /_plugins/_ml/models/NWR9YIsBUysqmzBdifVJ/_predict
  2. {
  3.   "parameters": {
  4.     "prompt": "\n\nHuman:hello\n\nAssistant:"
  5.   }
  6. }

copy

Step 6: Register and execute an agent

Finally, you’ll use the text embedding model created in Step 1 and the Claude model created in Step 5 to create a flow agent. This flow agent will run a VectorDBTool and then an MLModelTool. The VectorDBTool is configured with the model ID for the text embedding model created in Step 1 for vector search. The MLModelTool is configured with the Claude model created in step 5:

  1. POST /_plugins/_ml/agents/_register
  2. {
  3.   "name": "Test_Agent_For_RAG",
  4.   "type": "flow",
  5.   "description": "this is a test agent",
  6.   "tools": [
  7.     {
  8.       "type": "VectorDBTool",
  9.       "parameters": {
  10.         "model_id": "aVeif4oB5Vm0Tdw8zYO2",
  11.         "index": "my_test_data",
  12.         "embedding_field": "embedding",
  13.         "source_field": ["text"],
  14.         "input": "${parameters.question}"
  15.       }
  16.     },
  17.     {
  18.       "type": "MLModelTool",
  19.       "description": "A general tool to answer any question",
  20.       "parameters": {
  21.         "model_id": "NWR9YIsBUysqmzBdifVJ",
  22.         "prompt": "\n\nHuman:You are a professional data analyst. You will always answer a question based on the given context first. If the answer is not directly shown in the context, you will analyze the data and find the answer. If you don't know the answer, just say you don't know. \n\n Context:\n${parameters.VectorDBTool.output}\n\nHuman:${parameters.question}\n\nAssistant:"
  23.       }
  24.     }
  25.   ]
  26. }

copy

OpenSearch returns an agent ID for the newly created agent:

  1. {
  2.   "agent_id": "879v9YwBjWKCe6Kg12Tx"
  3. }

You can inspect the agent by sending a request to the agents endpoint and providing the agent ID:

  1. GET /_plugins/_ml/agents/879v9YwBjWKCe6Kg12Tx

copy

To execute the agent, send the following request. When registering the agent, you configured it to take in parameters.question, so you need to provide this parameter in the request. This parameter represents a human-generated user question:

  1. POST /_plugins/_ml/agents/879v9YwBjWKCe6Kg12Tx/_execute
  2. {
  3.   "parameters": {
  4.     "question": "what's the population increase of Seattle from 2021 to 2023"
  5.   }
  6. }

copy

The LLM does not have the recent information in its knowledge base, so it infers the response to the question based on the ingested data, demonstrating RAG:

  1. {
  2.   "inference_results": [
  3.     {
  4.       "output": [
  5.         {
  6.           "result": """ Based on the given context, the key information is:
  8. The metro area population of Seattle in 2021 was 3,461,000.
  9. The metro area population of Seattle in 2023 is 3,519,000.
  11. To calculate the population increase from 2021 to 2023:
  13. Population in 2023 (3,519,000) - Population in 2021 (3,461,000) = 58,000
  15. Therefore, the population increase of Seattle from 2021 to 2023 is 58,000."""
  16.         }
  17.       ]
  18.     }
  19.   ]
  20. }

Hidden agents

Introduced 2.13

To hide agent details from end users, including the cluster admin, you can register a hidden agent. If an agent is hidden, non-superadmin users don’t have permission to call any Agent APIs except for the Execute API, on the agent.

Only superadmin users can register a hidden agent. To register a hidden agent, you first need to authenticate with an admin certificate:

  1. curl -k --cert ./kirk.pem --key ./kirk-key.pem -XGET 'https://localhost:9200/.opendistro_security/_search'

All agents created by a superadmin user are automatically registered as hidden. Only the superadmin user can view hidden agent details and delete hidden agents. To register a hidden agent, send a request to the _register endpoint:

  1. curl -k --cert ./kirk.pem --key ./kirk-key.pem -X POST 'https://localhost:9200/_plugins/_ml/agents/_register' -H 'Content-Type: application/json' -d '
  2. {
  3.   "name": "Test_Agent_For_RAG",
  4.   "type": "flow",
  5.   "description": "this is a test agent",
  6.   "tools": [
  7.     {
  8.       "name": "vector_tool",
  9.       "type": "VectorDBTool",
  10.       "parameters": {
  11.         "model_id": "zBRyYIsBls05QaITo5ex",
  12.         "index": "my_test_data",
  13.         "embedding_field": "embedding",
  14.         "source_field": [
  15.           "text"
  16.         ],
  17.         "input": "${parameters.question}"
  18.       }
  19.     },
  20.     {
  21.       "type": "MLModelTool",
  22.       "description": "A general tool to answer any question",
  23.       "parameters": {
  24.         "model_id": "NWR9YIsBUysqmzBdifVJ",
  25.         "prompt": "\n\nHuman:You are a professional data analyst. You will always answer question based on the given context first. If the answer is not directly shown in the context, you will analyze the data and find the answer. If you don't know the answer, just say don't know. \n\n Context:\n${parameters.vector_tool.output}\n\nHuman:${parameters.question}\n\nAssistant:"
  26.       }
  27.     }
  28.   ]
  29. }'