Conversational search

Conversational search allows you to ask questions in natural language and refine the answers by asking follow-up questions. Thus, the conversation becomes a dialog between you and a large language model (LLM). For this to happen, instead of answering each question individually, the model needs to remember the context of the entire conversation.

Conversational search is implemented with the following components:

Conversation history

Conversation history consists of a simple CRUD-like API comprising two resources: memories and messages. All messages for the current conversation are stored within one conversation memory. A message represents a question/answer pair: a human-input question and an AI answer. Messages do not exist by themselves; they must be added to a memory.

RAG

RAG retrieves data from the index and history and sends all the information as context to the LLM. The LLM then supplements its static knowledge base with the dynamically retrieved data. In OpenSearch, RAG is implemented through a search pipeline containing a retrieval-augmented generation processor. The processor intercepts OpenSearch query results, retrieves previous messages in the conversation from the conversation memory, and sends a prompt to the LLM. After the processor receives a response from the LLM, it saves the response in conversation memory and returns both the original OpenSearch query results and the LLM response.

As of OpenSearch 2.11, the RAG technique has only been tested with OpenAI models and the Anthropic Claude model on Amazon Bedrock.

When the Security plugin is enabled, all memories exist in a private security mode. Only the user who created a memory can interact with that memory. No user can see another user’s memory.

Prerequisites

To begin using conversational search, enable conversation memory and RAG pipeline features:

  1. PUT /_cluster/settings
  2. {
  3.   "persistent": {
  4.     "plugins.ml_commons.memory_feature_enabled": true,
  5.     "plugins.ml_commons.rag_pipeline_feature_enabled": true
  6.   }
  7. }

copy

To use conversational search, follow these steps:

  1. Create a connector to a model.
  2. Register and deploy the model
  3. Create a search pipeline.
  4. Ingest RAG data into an index.
  5. Create a conversation memory.
  6. Use the pipeline for RAG.

Step 1: Create a connector to a model

RAG requires an LLM in order to function. To connect to an LLM, create a connector. The following request creates a connector for the OpenAI GPT 3.5 model:

  1. POST /_plugins/_ml/connectors/_create
  2. {
  3.   "name": "OpenAI Chat Connector",
  4.   "description": "The connector to public OpenAI model service for GPT 3.5",
  5.   "version": 2,
  6.   "protocol": "http",
  7.   "parameters": {
  8.     "endpoint": "api.openai.com",
  9.     "model": "gpt-3.5-turbo",
  10.     "temperature": 0
  11.   },
  12.   "credential": {
  13.     "openAI_key": "<YOUR_OPENAI_KEY>"
  14.   },
  15.   "actions": [
  16.     {
  17.       "action_type": "predict",
  18.       "method": "POST",
  19.       "url": "https://${parameters.endpoint}/v1/chat/completions",
  20.       "headers": {
  21.         "Authorization": "Bearer ${credential.openAI_key}"
  22.       },
  23.       "request_body": """{ "model": "${parameters.model}", "messages": ${parameters.messages}, "temperature": ${parameters.temperature} }"""
  24.     }
  25.   ]
  26. }

copy

OpenSearch responds with a connector ID for the connector:

  1. {
  2.   "connector_id": "u3DEbI0BfUsSoeNTti-1"
  3. }

For example requests that connect to other services and models, see Connector blueprints.

Step 2: Register and deploy the model

Register the LLM for which you created a connector in the previous step. To register the model with OpenSearch, provide the connector_id returned in the previous step:

  1. POST /_plugins/_ml/models/_register
  2. {
  3.   "name": "openAI-gpt-3.5-turbo",
  4.   "function_name": "remote",
  5.   "description": "test model",
  6.   "connector_id": "u3DEbI0BfUsSoeNTti-1"
  7. }

copy

OpenSearch returns a task ID for the register task and a model ID for the registered model:

  1. {
  2.   "task_id": "gXDIbI0BfUsSoeNT_jAb",
  3.   "status": "CREATED",
  4.   "model_id": "gnDIbI0BfUsSoeNT_jAw"
  5. }

To verify that the registration is complete, call the Tasks API:

  1. GET /_plugins/_ml/tasks/gXDIbI0BfUsSoeNT_jAb

copy

The state changes to COMPLETED in the response:

  1. {
  2.   "model_id": "gnDIbI0BfUsSoeNT_jAw",
  3.   "task_type": "REGISTER_MODEL",
  4.   "function_name": "REMOTE",
  5.   "state": "COMPLETED",
  6.   "worker_node": [
  7.     "kYv-Z5-mQ4uCUy_cRC6LXA"
  8.   ],
  9.   "create_time": 1706927128091,
  10.   "last_update_time": 1706927128125,
  11.   "is_async": false
  12. }

To deploy the model, provide the model_id to the Deploy API:

  1. POST /_plugins/_ml/models/gnDIbI0BfUsSoeNT_jAw/_deploy

copy

OpenSearch acknowledges that the model is deployed:

  1. {
  2.   "task_id": "cnDObI0BfUsSoeNTDzGd",
  3.   "task_type": "DEPLOY_MODEL",
  4.   "status": "COMPLETED"
  5. }

Step 3: Create a search pipeline

Next, create a search pipeline with a retrieval_augmented_generation processor:

  1. PUT /_search/pipeline/rag_pipeline
  2. {
  3.   "response_processors": [
  4.     {
  5.       "retrieval_augmented_generation": {
  6.         "tag": "openai_pipeline_demo",
  7.         "description": "Demo pipeline Using OpenAI Connector",
  8.         "model_id": "gnDIbI0BfUsSoeNT_jAw",
  9.         "context_field_list": ["text"],
  10.         "system_prompt": "You are a helpful assistant",
  11.         "user_instructions": "Generate a concise and informative answer in less than 100 words for the given question"
  12.       }
  13.     }
  14.   ]
  15. }

copy

For information about the processor fields, see Retrieval-augmented generation processor.

Step 4: Ingest RAG data into an index

RAG augments the LLM’s knowledge with some supplementary data.

First, create an index in which to store this data and set the default search pipeline to the pipeline created in the previous step:

  1. PUT /my_rag_test_data
  2. {
  3.   "settings": {
  4.     "index.search.default_pipeline" : "rag_pipeline"
  5.   },
  6.   "mappings": {
  7.     "properties": {
  8.       "text": {
  9.         "type": "text"
  10.       }
  11.     }
  12.   }
  13. }

copy

Next, ingest the supplementary data into the index:

  1. POST _bulk
  2. {"index": {"_index": "my_rag_test_data", "_id": "1"}}
  3. {"text": "Abraham Lincoln was born on February 12, 1809, the second child of Thomas Lincoln and Nancy Hanks Lincoln, in a log cabin on Sinking Spring Farm near Hodgenville, Kentucky.[2] He was a descendant of Samuel Lincoln, an Englishman who migrated from Hingham, Norfolk, to its namesake, Hingham, Massachusetts, in 1638. The family then migrated west, passing through New Jersey, Pennsylvania, and Virginia.[3] Lincoln was also a descendant of the Harrison family of Virginia; his paternal grandfather and namesake, Captain Abraham Lincoln and wife Bathsheba (née Herring) moved the family from Virginia to Jefferson County, Kentucky.[b] The captain was killed in an Indian raid in 1786.[5] His children, including eight-year-old Thomas, Abraham's father, witnessed the attack.[6][c] Thomas then worked at odd jobs in Kentucky and Tennessee before the family settled in Hardin County, Kentucky, in the early 1800s."}
  4. {"index": {"_index": "my_rag_test_data", "_id": "2"}}
  5. {"text": "Chart and table of population level and growth rate for the New York City metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of New York City in 2023 is 18,937,000, a 0.37% increase from 2022.\\nThe metro area population of New York City in 2022 was 18,867,000, a 0.23% increase from 2021.\\nThe metro area population of New York City in 2021 was 18,823,000, a 0.1% increase from 2020.\\nThe metro area population of New York City in 2020 was 18,804,000, a 0.01% decline from 2019."}

copy

RAG pipeline

RAG is a technique that retrieves documents from an index, passes them through a seq2seq model, such as an LLM, and then supplements the static LLM information with the dynamically retrieved data in context.

As of OpenSearch 2.12, the RAG technique has only been tested with OpenAI models, the Anthropic Claude model on Amazon Bedrock, and Cohere Command models.

Configuring the Cohere Command model to enable RAG requires using a post-processing function to transform the model output. For more information, see the Cohere RAG Tutorial.

Enabling RAG

Step 5: Create a conversation memory

You’ll need to create a conversation memory that will store all messages from a conversation. To make the memory easily identifiable, provide a name for the memory in the optional name field, as shown in the following example. Because the name parameter is not updatable, this is your only opportunity to name your conversation.

  1. POST /_plugins/_ml/memory/
  2. {
  3.   "name": "Conversation about NYC population"
  4. }

copy

OpenSearch responds with a memory ID for the newly created memory:

  1. {
  2.   "memory_id": "znCqcI0BfUsSoeNTntd7"
  3. }

You’ll use the memory_id to add messages to the memory.

Step 6: Use the pipeline for RAG

To use the RAG pipeline, send a query to OpenSearch and provide additional parameters in the ext.generative_qa_parameters object.

The generative_qa_parameters object supports the following parameters.

ParameterRequiredDescription
llm_questionYesThe question that the LLM must answer.
llm_modelNoOverrides the original model set in the connection in cases where you want to use a different model (for example, GPT 4 instead of GPT 3.5). This option is required if a default model is not set during pipeline creation.
memory_idNoIf you provide a memory_id, the pipeline retrieves the 10 most recent messages in the specified memory and adds them to the LLM prompt. If you don’t specify a memory_id, the prior context is not added to the LLM prompt.
context_sizeNoThe number of search results sent to the LLM. This is typically needed in order to meet the token size limit, which can vary by model. Alternatively, you can use the size parameter in the Search API to control the number of search results sent to the LLM.
message_sizeNoThe number of messages sent to the LLM. Similarly to the number of search results, this affects the total number of tokens received by the LLM. When not set, the pipeline uses the default message size of 10.
timeoutNoThe number of seconds that the pipeline waits for the remote model using a connector to respond. Default is 30.

If your LLM includes a set token limit, set the size field in your OpenSearch query to limit the number of documents used in the search response. Otherwise, the RAG pipeline will send every document in the search results to the LLM.

If you ask an LLM a question about the present, it cannot provide an answer because it was trained on data from a few years ago. However, if you add current information as context, the LLM is able to generate a response. For example, you can ask the LLM about the population of the New York City metro area in 2023. You’ll construct a query that includes an OpenSearch match query and an LLM query. Provide the memory_id so that the message is stored in the appropriate memory object:

  1. GET /my_rag_test_data/_search
  2. {
  3.   "query": {
  4.     "match": {
  5.       "text": "What's the population of NYC metro area in 2023"
  6.     }
  7.   },
  8.   "ext": {
  9.     "generative_qa_parameters": {
  10.       "llm_model": "gpt-3.5-turbo",
  11.       "llm_question": "What's the population of NYC metro area in 2023",
  12.       "memory_id": "znCqcI0BfUsSoeNTntd7",
  13.       "context_size": 5,
  14.       "message_size": 5,
  15.       "timeout": 15
  16.     }
  17.   }
  18. }

copy

Because the context included a document containing information about the population of New York City, the LLM was able to correctly answer the question (though it included the word “projected” because it was trained on data from previous years). The response contains the matching documents from the supplementary RAG data and the LLM response:

Response

  1. {
  2.   "took": 1,
  3.   "timed_out": false,
  4.   "_shards": {
  5.     "total": 1,
  6.     "successful": 1,
  7.     "skipped": 0,
  8.     "failed": 0
  9.   },
  10.   "hits": {
  11.     "total": {
  12.       "value": 2,
  13.       "relation": "eq"
  14.     },
  15.     "max_score": 5.781642,
  16.     "hits": [
  17.       {
  18.         "_index": "my_rag_test_data",
  19.         "_id": "2",
  20.         "_score": 5.781642,
  21.         "_source": {
  22.           "text": """Chart and table of population level and growth rate for the New York City metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\nThe current metro area population of New York City in 2023 is 18,937,000, a 0.37% increase from 2022.\nThe metro area population of New York City in 2022 was 18,867,000, a 0.23% increase from 2021.\nThe metro area population of New York City in 2021 was 18,823,000, a 0.1% increase from 2020.\nThe metro area population of New York City in 2020 was 18,804,000, a 0.01% decline from 2019."""
  23.         }
  24.       },
  25.       {
  26.         "_index": "my_rag_test_data",
  27.         "_id": "1",
  28.         "_score": 0.9782871,
  29.         "_source": {
  30.           "text": "Abraham Lincoln was born on February 12, 1809, the second child of Thomas Lincoln and Nancy Hanks Lincoln, in a log cabin on Sinking Spring Farm near Hodgenville, Kentucky.[2] He was a descendant of Samuel Lincoln, an Englishman who migrated from Hingham, Norfolk, to its namesake, Hingham, Massachusetts, in 1638. The family then migrated west, passing through New Jersey, Pennsylvania, and Virginia.[3] Lincoln was also a descendant of the Harrison family of Virginia; his paternal grandfather and namesake, Captain Abraham Lincoln and wife Bathsheba (née Herring) moved the family from Virginia to Jefferson County, Kentucky.[b] The captain was killed in an Indian raid in 1786.[5] His children, including eight-year-old Thomas, Abraham's father, witnessed the attack.[6][c] Thomas then worked at odd jobs in Kentucky and Tennessee before the family settled in Hardin County, Kentucky, in the early 1800s."
  31.         }
  32.       }
  33.     ]
  34.   },
  35.   "ext": {
  36.     "retrieval_augmented_generation": {
  37.       "answer": "The population of the New York City metro area in 2023 is projected to be 18,937,000.",
  38.       "message_id": "x3CecI0BfUsSoeNT9tV9"
  39.     }
  40.   }
  41. }

Now you’ll ask an LLM a follow-up question as part of the same conversation. Again, provide the memory_id in the request:

  1. GET /my_rag_test_data/_search
  2. {
  3.   "query": {
  4.     "match": {
  5.       "text": "What was it in 2022"
  6.     }
  7.   },
  8.   "ext": {
  9.     "generative_qa_parameters": {
  10.       "llm_model": "gpt-3.5-turbo",
  11.       "llm_question": "What was it in 2022",
  12.       "memory_id": "znCqcI0BfUsSoeNTntd7",
  13.       "context_size": 5,
  14.       "message_size": 5,
  15.       "timeout": 15
  16.     }
  17.   }
  18. }

copy

The LLM correctly identifies the subject of the conversation and returns a relevant response:

  1. {
  2.   ...
  3.   "ext": {
  4.     "retrieval_augmented_generation": {
  5.       "answer": "The population of the New York City metro area in 2022 was 18,867,000.",
  6.       "message_id": "p3CvcI0BfUsSoeNTj9iH"
  7.     }
  8.   }
  9. }

To verify that both messages were added to the memory, provide the memory_ID to the Get Messages API:

  1. GET /_plugins/_ml/memory/znCqcI0BfUsSoeNTntd7/messages

The response contains both messages:

Response

  1. {
  2.   "messages": [
  3.     {
  4.       "memory_id": "znCqcI0BfUsSoeNTntd7",
  5.       "message_id": "x3CecI0BfUsSoeNT9tV9",
  6.       "create_time": "2024-02-03T20:33:50.754708446Z",
  7.       "input": "What's the population of NYC metro area in 2023",
  8.       "prompt_template": """[{"role":"system","content":"You are a helpful assistant"},{"role":"user","content":"Generate a concise and informative answer in less than 100 words for the given question"}]""",
  9.       "response": "The population of the New York City metro area in 2023 is projected to be 18,937,000.",
  10.       "origin": "retrieval_augmented_generation",
  11.       "additional_info": {
  12.         "metadata": """["Chart and table of population level and growth rate for the New York City metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of New York City in 2023 is 18,937,000, a 0.37% increase from 2022.\\nThe metro area population of New York City in 2022 was 18,867,000, a 0.23% increase from 2021.\\nThe metro area population of New York City in 2021 was 18,823,000, a 0.1% increase from 2020.\\nThe metro area population of New York City in 2020 was 18,804,000, a 0.01% decline from 2019.","Abraham Lincoln was born on February 12, 1809, the second child of Thomas Lincoln and Nancy Hanks Lincoln, in a log cabin on Sinking Spring Farm near Hodgenville, Kentucky.[2] He was a descendant of Samuel Lincoln, an Englishman who migrated from Hingham, Norfolk, to its namesake, Hingham, Massachusetts, in 1638. The family then migrated west, passing through New Jersey, Pennsylvania, and Virginia.[3] Lincoln was also a descendant of the Harrison family of Virginia; his paternal grandfather and namesake, Captain Abraham Lincoln and wife Bathsheba (née Herring) moved the family from Virginia to Jefferson County, Kentucky.[b] The captain was killed in an Indian raid in 1786.[5] His children, including eight-year-old Thomas, Abraham's father, witnessed the attack.[6][c] Thomas then worked at odd jobs in Kentucky and Tennessee before the family settled in Hardin County, Kentucky, in the early 1800s."]"""
  13.       }
  14.     },
  15.     {
  16.       "memory_id": "znCqcI0BfUsSoeNTntd7",
  17.       "message_id": "p3CvcI0BfUsSoeNTj9iH",
  18.       "create_time": "2024-02-03T20:36:10.24453505Z",
  19.       "input": "What was it in 2022",
  20.       "prompt_template": """[{"role":"system","content":"You are a helpful assistant"},{"role":"user","content":"Generate a concise and informative answer in less than 100 words for the given question"}]""",
  21.       "response": "The population of the New York City metro area in 2022 was 18,867,000.",
  22.       "origin": "retrieval_augmented_generation",
  23.       "additional_info": {
  24.         "metadata": """["Chart and table of population level and growth rate for the New York City metro area from 1950 to 2023. United Nations population projections are also included through the year 2035.\\nThe current metro area population of New York City in 2023 is 18,937,000, a 0.37% increase from 2022.\\nThe metro area population of New York City in 2022 was 18,867,000, a 0.23% increase from 2021.\\nThe metro area population of New York City in 2021 was 18,823,000, a 0.1% increase from 2020.\\nThe metro area population of New York City in 2020 was 18,804,000, a 0.01% decline from 2019.","Abraham Lincoln was born on February 12, 1809, the second child of Thomas Lincoln and Nancy Hanks Lincoln, in a log cabin on Sinking Spring Farm near Hodgenville, Kentucky.[2] He was a descendant of Samuel Lincoln, an Englishman who migrated from Hingham, Norfolk, to its namesake, Hingham, Massachusetts, in 1638. The family then migrated west, passing through New Jersey, Pennsylvania, and Virginia.[3] Lincoln was also a descendant of the Harrison family of Virginia; his paternal grandfather and namesake, Captain Abraham Lincoln and wife Bathsheba (née Herring) moved the family from Virginia to Jefferson County, Kentucky.[b] The captain was killed in an Indian raid in 1786.[5] His children, including eight-year-old Thomas, Abraham's father, witnessed the attack.[6][c] Thomas then worked at odd jobs in Kentucky and Tennessee before the family settled in Hardin County, Kentucky, in the early 1800s."]"""
  25.       }
  26.     }
  27.   ]
  28. }

Next steps

  • To learn more about connecting to models on external platforms, see Connectors.
  • For supported APIs, see Memory APIs.
  • To learn more about search pipelines and processors, see Search pipelines.
  • For available OpenSearch queries, see Query DSL.