Connecting to externally hosted models

Introduced 2.9

Integrations with machine learning (ML) models hosted on third-party platforms allow system administrators and data scientists to run ML workloads outside of their OpenSearch cluster. Connecting to externally hosted models enables ML developers to create integrations with other ML services, such as Amazon SageMaker or OpenAI.

To integrate a model hosted on a third-party platform, choose from the following options:

  • If you’re an ML developer wanting to create integrations with your specific ML services, see Connector blueprints.
  • If you’re a system administrator or data scientist wanting to create a connection to an ML service, see Connectors.

Prerequisites

If you’re an admin deploying an ML connector, make sure that the target model of the connector has already been deployed on your chosen platform. Furthermore, make sure that you have permissions to send and receive data to the third-party API for your connector.

When access control is enabled on your third-party platform, you can enter your security settings using the authorization or credential settings inside the connector API.

Adding trusted endpoints

To configure connectors in OpenSearch, add the trusted endpoints to your cluster settings by using the plugins.ml_commons.trusted_connector_endpoints_regex setting, which supports Java regex expressions:

  1. PUT /_cluster/settings
  2. {
  3.     "persistent": {
  4.         "plugins.ml_commons.trusted_connector_endpoints_regex": [
  5.           "^https://runtime\\.sagemaker\\..*[a-z0-9-]\\.amazonaws\\.com/.*$",
  6.           "^https://api\\.openai\\.com/.*$",
  7.           "^https://api\\.cohere\\.ai/.*$",
  8.           "^https://bedrock-runtime\\..*[a-z0-9-]\\.amazonaws\\.com/.*$"
  9.         ]
  10.     }
  11. }

copy

Setting up connector access control

If you plan on using a remote connector, make sure to use an OpenSearch cluster with the Security plugin enabled. Using the Security plugin gives you access to connector access control, which is required when using a remote connector.

If you require granular access control for your connectors, use the following cluster setting:

  1. PUT /_cluster/settings
  2. {
  3.     "persistent": {
  4.         "plugins.ml_commons.connector_access_control_enabled": true
  5.     }
  6. }

copy

When access control is enabled, you can install the Security plugin. This makes the backend_roles, add_all_backend_roles, or access_model options required in order to use the connector API. If successful, OpenSearch returns the following response:

  1. {
  2.   "acknowledged": true,
  3.   "persistent": {
  4.     "plugins": {
  5.       "ml_commons": {
  6.         "connector_access_control_enabled": "true"
  7.       }
  8.     }
  9.   },
  10.   "transient": {}
  11. }

Step 1: Register a model group

To register a model, you have the following options:

  • You can use model_group_id to register a model version to an existing model group.
  • If you do not use model_group_id, ML Commons creates a model with a new model group.

To register a model group, send the following request:

  1. POST /_plugins/_ml/model_groups/_register
  2. {
  3.   "name": "remote_model_group",
  4.   "description": "A model group for external models"
  5. }

copy

The response contains the model group ID that you’ll use to register a model to this model group:

  1. {
  2.  "model_group_id": "wlcnb4kBJ1eYAeTMHlV6",
  3.  "status": "CREATED"
  4. }

To learn more about model groups, see Model access control.

Step 2: Create a connector

You can create a standalone connector that can be reused for multiple models. Alternatively, you can specify a connector when creating a model so that it can be used only for that model. For more information and example connectors, see Connectors.

The Connectors Create API, /_plugins/_ml/connectors/_create, creates connectors that facilitate registering and deploying external models in OpenSearch. Using the endpoint parameter, you can connect ML Commons to any supported ML tool by using its specific API endpoint. For example, you can connect to a ChatGPT model by using the api.openai.com endpoint:

  1. POST /_plugins/_ml/connectors/_create
  2. {
  3.     "name": "OpenAI Chat Connector",
  4.     "description": "The connector to public OpenAI model service for GPT 3.5",
  5.     "version": 1,
  6.     "protocol": "http",
  7.     "parameters": {
  8.         "endpoint": "api.openai.com",
  9.         "model": "gpt-3.5-turbo"
  10.     },
  11.     "credential": {
  12.         "openAI_key": "..."
  13.     },
  14.     "actions": [
  15.         {
  16.             "action_type": "predict",
  17.             "method": "POST",
  18.             "url": "https://${parameters.endpoint}/v1/chat/completions",
  19.             "headers": {
  20.                 "Authorization": "Bearer ${credential.openAI_key}"
  21.             },
  22.             "request_body": "{ \"model\": \"${parameters.model}\", \"messages\": ${parameters.messages} }"
  23.         }
  24.     ]
  25. }

copy

The response contains the connector ID for the newly created connector:

  1. {
  2.   "connector_id": "a1eMb4kBJ1eYAeTMAljY"
  3. }

Step 3: Register an externally hosted model

To register an externally hosted model to the model group created in step 1, provide the model group ID from step 1 and the connector ID from step 2 in the following request. You must specify the function_name as remote:

  1. POST /_plugins/_ml/models/_register
  2. {
  3.     "name": "openAI-gpt-3.5-turbo",
  4.     "function_name": "remote",
  5.     "model_group_id": "1jriBYsBq7EKuKzZX131",
  6.     "description": "test model",
  7.     "connector_id": "a1eMb4kBJ1eYAeTMAljY"
  8. }

copy

OpenSearch returns the task ID of the register operation:

  1. {
  2.   "task_id": "cVeMb4kBJ1eYAeTMFFgj",
  3.   "status": "CREATED"
  4. }

To check the status of the operation, provide the task ID to the Tasks API:

  1. GET /_plugins/_ml/tasks/cVeMb4kBJ1eYAeTMFFgj

copy

When the operation is complete, the state changes to COMPLETED:

  1. {
  2.   "model_id": "cleMb4kBJ1eYAeTMFFg4",
  3.   "task_type": "REGISTER_MODEL",
  4.   "function_name": "REMOTE",
  5.   "state": "COMPLETED",
  6.   "worker_node": [
  7.     "XPcXLV7RQoi5m8NI_jEOVQ"
  8.   ],
  9.   "create_time": 1689793598499,
  10.   "last_update_time": 1689793598530,
  11.   "is_async": false
  12. }

Take note of the returned model_id because you’ll need it to deploy the model.

Step 4: Deploy the model

Starting with OpenSearch version 2.13, externally hosted models are deployed automatically by default when you send a Predict API request for the first time. To disable automatic deployment for an externally hosted model, set plugins.ml_commons.model_auto_deploy.enable to false:

  1. PUT _cluster/settings
  2. {
  3.   "persistent": {
  4.     "plugins.ml_commons.model_auto_deploy.enable" : "false"
  5.   }
  6. }

copy

To undeploy the model, use the Undeploy API.

  1. POST /_plugins/_ml/models/cleMb4kBJ1eYAeTMFFg4/_deploy

copy

The response contains the task ID that you can use to check the status of the deploy operation:

  1. {
  2.   "task_id": "vVePb4kBJ1eYAeTM7ljG",
  3.   "status": "CREATED"
  4. }

As in the previous step, check the status of the operation by calling the Tasks API:

  1. GET /_plugins/_ml/tasks/vVePb4kBJ1eYAeTM7ljG

copy

When the operation is complete, the state changes to COMPLETED:

  1. {
  2.   "model_id": "cleMb4kBJ1eYAeTMFFg4",
  3.   "task_type": "DEPLOY_MODEL",
  4.   "function_name": "REMOTE",
  5.   "state": "COMPLETED",
  6.   "worker_node": [
  7.     "n-72khvBTBi3bnIIR8FTTw"
  8.   ],
  9.   "create_time": 1689793851077,
  10.   "last_update_time": 1689793851101,
  11.   "is_async": true
  12. }

Step 5 (Optional): Test the model

Use the Predict API to test the model:

  1. POST /_plugins/_ml/models/cleMb4kBJ1eYAeTMFFg4/_predict
  2. {
  3.   "parameters": {
  4.     "messages": [
  5.       {
  6.         "role": "system",
  7.         "content": "You are a helpful assistant."
  8.       },
  9.       {
  10.         "role": "user",
  11.         "content": "Hello!"
  12.       }
  13.     ]
  14.   }
  15. }

copy

To learn more about chat functionality within OpenAI, see the OpenAI Chat API.

The response contains the inference results provided by the OpenAI model:

  1. {
  2.   "inference_results": [
  3.     {
  4.       "output": [
  5.         {
  6.           "name": "response",
  7.           "dataAsMap": {
  8.             "id": "chatcmpl-7e6s5DYEutmM677UZokF9eH40dIY7",
  9.             "object": "chat.completion",
  10.             "created": 1689793889,
  11.             "model": "gpt-3.5-turbo-0613",
  12.             "choices": [
  13.               {
  14.                 "index": 0,
  15.                 "message": {
  16.                   "role": "assistant",
  17.                   "content": "Hello! How can I assist you today?"
  18.                 },
  19.                 "finish_reason": "stop"
  20.               }
  21.             ],
  22.             "usage": {
  23.               "prompt_tokens": 19,
  24.               "completion_tokens": 9,
  25.               "total_tokens": 28
  26.             }
  27.           }
  28.         }
  29.       ]
  30.     }
  31.   ]
  32. }

Step 6: Use the model for batch ingestion

To learn how to use the model for batch ingestion in order to improve ingestion performance, see Using externally hosted ML models for batch ingestion.

To learn how to use the model for vector search, see Using an ML model for neural search.

Step 8 (Optional): Undeploy the model

You can undeploy the model automatically by defining a TTL in the model settings or by using the Undeploy API to undeploy the model manually. For more information, see Undeploy API.

Next steps