Configuring model guardrails

Introduced 2.13

Guardrails can guide a large language model (LLM) toward desired behavior. They act as a filter, preventing the LLM from generating output that is harmful or violates ethical principles and facilitating safer use of AI. Guardrails also cause the LLM to produce more focused and relevant output.

To configure guardrails for your LLM, you can provide a list of words to be prohibited in the input or output of the model. Alternatively, you can provide a regular expression against which the model input or output will be matched.

Prerequisites

Before you start, make sure you have fulfilled the prerequisites for connecting to an externally hosted model.

Step 1: Create a guardrail index

To start, create an index that will store the excluded words (stopwords). In the index settings, specify a title field, which will contain excluded words, and a query field of the percolator type. The percolator query will be used to match the LLM input or output:

  1. PUT /words0
  2. {
  3.   "mappings": {
  4.     "properties": {
  5.       "title": {
  6.         "type": "text"
  7.       },
  8.       "query": {
  9.         "type": "percolator"
  10.       }
  11.     }
  12.   }
  13. }

copy

Step 2: Index excluded words or phrases

Next, index a query string query that will be used to match excluded words in the model input or output:

  1. PUT /words0/_doc/1?refresh
  2. {
  3.   "query": {
  4.     "query_string": {
  5.       "query": "title: blacklist"
  6.     }
  7.   }
  8. }

copy

  1. PUT /words0/_doc/2?refresh
  2. {
  3.   "query": {
  4.     "query_string": {
  5.       "query": "title: \"Master slave architecture\""
  6.     }
  7.   }
  8. }

copy

For more query string options, see Query string query.

Step 3: Register a model group

To register a model group, send the following request:

  1. POST /_plugins/_ml/model_groups/_register
  2. {
  3.     "name": "bedrock",
  4.     "description": "This is a public model group."
  5. }

copy

The response contains the model group ID that you’ll use to register a model to this model group:

  1. {
  2.  "model_group_id": "wlcnb4kBJ1eYAeTMHlV6",
  3.  "status": "CREATED"
  4. }

To learn more about model groups, see Model access control.

Step 4: Create a connector

Now you can create a connector for the model. In this example, you’ll create a connector to the Anthropic Claude model hosted on Amazon Bedrock:

  1. POST /_plugins/_ml/connectors/_create
  2. {
  3.   "name": "BedRock test claude Connector",
  4.   "description": "The connector to BedRock service for claude model",
  5.   "version": 1,
  6.   "protocol": "aws_sigv4",
  7.   "parameters": {
  8.       "region": "us-east-1",
  9.       "service_name": "bedrock",
  10.       "anthropic_version": "bedrock-2023-05-31",
  11.       "endpoint": "bedrock.us-east-1.amazonaws.com",
  12.       "auth": "Sig_V4",
  13.       "content_type": "application/json",
  14.       "max_tokens_to_sample": 8000,
  15.       "temperature": 0.0001,
  16.       "response_filter": "$.completion"
  17.   },
  18.   "credential": {
  19.       "access_key": "<YOUR_ACCESS_KEY>",
  20.       "secret_key": "<YOUR_SECRET_KEY>"
  21.   },
  22.   "actions": [
  23.     {
  24.       "action_type": "predict",
  25.       "method": "POST",
  26.       "url": "https://bedrock-runtime.us-east-1.amazonaws.com/model/anthropic.claude-v2/invoke",
  27.       "headers": { 
  28.         "content-type": "application/json",
  29.         "x-amz-content-sha256": "required"
  30.       },
  31.       "request_body": "{\"prompt\":\"${parameters.prompt}\", \"max_tokens_to_sample\":${parameters.max_tokens_to_sample}, \"temperature\":${parameters.temperature},  \"anthropic_version\":\"${parameters.anthropic_version}\" }"
  32.     }
  33.   ]
  34. }

copy

The response contains the connector ID for the newly created connector:

  1. {
  2.   "connector_id": "a1eMb4kBJ1eYAeTMAljY"
  3. }

Step 5: Register and deploy the model with guardrails

To register an externally hosted model, provide the model group ID from step 3 and the connector ID from step 4 in the following request. To configure guardrails, include the guardrails object:

  1. POST /_plugins/_ml/models/_register?deploy=true
  2. {
  3.   "name": "Bedrock Claude V2 model",
  4.   "function_name": "remote",
  5.   "model_group_id": "wlcnb4kBJ1eYAeTMHlV6",
  6.   "description": "test model",
  7.   "connector_id": "a1eMb4kBJ1eYAeTMAljY",
  8.   "guardrails": {
  9.     "type": "local_regex",
  10.     "input_guardrail": {
  11.       "stop_words": [
  12.         {
  13.           "index_name": "words0",
  14.           "source_fields": [
  15.             "title"
  16.           ]
  17.         }
  18.       ],
  19.       "regex": [
  20.         ".*abort.*",
  21.         ".*kill.*"
  22.       ]
  23.     },
  24.     "output_guardrail": {
  25.       "stop_words": [
  26.         {
  27.           "index_name": "words0",
  28.           "source_fields": [
  29.             "title"
  30.           ]
  31.         }
  32.       ],
  33.       "regex": [
  34.         ".*abort.*",
  35.         ".*kill.*"
  36.       ]
  37.     }
  38.   }
  39. }

copy

For more information, see The guardrails parameter.

OpenSearch returns the task ID of the register operation:

  1. {
  2.   "task_id": "cVeMb4kBJ1eYAeTMFFgj",
  3.   "status": "CREATED"
  4. }

To check the status of the operation, provide the task ID to the Tasks API:

  1. GET /_plugins/_ml/tasks/cVeMb4kBJ1eYAeTMFFgj

copy

When the operation is complete, the state changes to COMPLETED:

  1. {
  2.   "model_id": "cleMb4kBJ1eYAeTMFFg4",
  3.   "task_type": "DEPLOY_MODEL",
  4.   "function_name": "REMOTE",
  5.   "state": "COMPLETED",
  6.   "worker_node": [
  7.     "n-72khvBTBi3bnIIR8FTTw"
  8.   ],
  9.   "create_time": 1689793851077,
  10.   "last_update_time": 1689793851101,
  11.   "is_async": true
  12. }

Step 6 (Optional): Test the model

To demonstrate how guardrails are applied, first run the predict operation that does not contain any excluded words:

  1. POST /_plugins/_ml/models/p94dYo4BrXGpZpgPp98E/_predict
  2. {
  3.   "parameters": {
  4.     "prompt": "\n\nHuman:this is a test\n\nnAssistant:"
  5.   }
  6. }

copy

The response contains inference results:

  1. {
  2.   "inference_results": [
  3.     {
  4.       "output": [
  5.         {
  6.           "name": "response",
  7.           "dataAsMap": {
  8.             "response": " Thank you for the test, I appreciate you taking the time to interact with me. I'm an AI assistant created by Anthropic to be helpful, harmless, and honest."
  9.           }
  10.         }
  11.       ],
  12.       "status_code": 200
  13.     }
  14.   ]
  15. }

Then run the predict operation that contains excluded words:

  1. POST /_plugins/_ml/models/p94dYo4BrXGpZpgPp98E/_predict
  2. {
  3.   "parameters": {
  4.     "prompt": "\n\nHuman:this is a test of Master slave architecture\n\nnAssistant:"
  5.   }
  6. }

copy

The response contains an error message because guardrails were triggered:

  1. {
  2.   "error": {
  3.     "root_cause": [
  4.       {
  5.         "type": "illegal_argument_exception",
  6.         "reason": "guardrails triggered for user input"
  7.       }
  8.     ],
  9.     "type": "illegal_argument_exception",
  10.     "reason": "guardrails triggered for user input"
  11.   },
  12.   "status": 400
  13. }

Guardrails are also triggered when a prompt matches the supplied regular expression.

Next steps