Creating connectors for third-party ML platforms

Machine Learning (ML) connectors provide the ability to integrate OpenSearch ML capabilities with third-party ML tools and platforms. Through connectors, OpenSearch can invoke these third-party endpoints to enrich query results and data pipelines.

You can provision connectors in two ways:

  1. An external connector, saved in a connector index, which can be reused and shared with multiple remote models but requires access to both the model, the connector inside of OpenSearch, and the third party being accessed by the connector, such as OpenAI or SageMaker.

  2. A local connector, saved in the model index, which can only be used with one remote model. Unlike a standalone connector, users only need access to the model itself to access an internal connector because the connection is established inside the model.

Supported connectors

As of OpenSearch 2.9, connectors have been tested for the following ML services, though it is possible to create connectors for other platforms not listed here:

  • Amazon SageMaker allows you to host and manage the lifecycle of text-embedding models, powering semantic search queries in OpenSearch. When connected, Amazon SageMaker hosts your models and OpenSearch is used to query inferences. This benefits Amazon SageMaker users who value its functionality, such as model monitoring, serverless hosting, and workflow automation for continuous training and deployment.
  • OpenAI ChatGPT enables you to invoke an OpenAI chat model from inside an OpenSearch cluster.
  • Cohere allows you to use data from OpenSearch to power Cohere’s large language models.

All connectors consist of a JSON blueprint created by machine learning (ML) developers. The blueprint allows administrators and data scientists to make connections between OpenSearch and an AI service or model-serving technology.

You can find blueprints for each connector in the ML Commons repository.

If you want to build your own blueprint, see Building blueprints.

External connector

Admins are only required to enter their credential settings, such as "openAI_key", for the service they are connecting to. All other parameters are defined within the blueprint.

The connector creation API, /_plugins/_ml/connectors/_create, creates connections that allow users to deploy and register external models through OpenSearch. Using the endpoint parameter, you can connect ML Commons to any supported ML tool using its specific API endpoint. For example, to connect to a ChatGPT model, you can connect using api.openai.com, as shown in the following example:

  1. POST /_plugins/_ml/connectors/_create
  2. {
  3.     "name": "OpenAI Chat Connector",
  4.     "description": "The connector to public OpenAI model service for GPT 3.5",
  5.     "version": 1,
  6.     "protocol": "http",
  7.     "parameters": {
  8.         "endpoint": "api.openai.com",
  9.         "model": "gpt-3.5-turbo"
  10.     },
  11.     "credential": {
  12.         "openAI_key": "..."
  13.     },
  14.     "actions": [
  15.         {
  16.             "action_type": "predict",
  17.             "method": "POST",
  18.             "url": "https://${parameters.endpoint}/v1/chat/completions",
  19.             "headers": {
  20.                 "Authorization": "Bearer ${credential.openAI_key}"
  21.             },
  22.             "request_body": "{ \"model\": \"${parameters.model}\", \"messages\": ${parameters.messages} }"
  23.         }
  24.     ]
  25. }

copy

If successful, the connector API responds with the connector_id for the connection:

  1. {
  2.   "connector_id": "a1eMb4kBJ1eYAeTMAljY"
  3. }
  1. POST /_plugins/_ml/models/_register
  2. {
  3.     "name": "openAI-gpt-3.5-turbo",
  4.     "function_name": "remote",
  5.     "model_group_id": "lEFGL4kB4ubqQRzegPo2",
  6.     "description": "test model",
  7.     "connector": {
  8.         "name": "OpenAI Connector",
  9.         "description": "The connector to public OpenAI model service for GPT 3.5",
  10.         "version": 1,
  11.         "protocol": "http",
  12.         "parameters": {
  13.             "endpoint": "api.openai.com",
  14.             "max_tokens": 7,
  15.             "temperature": 0,
  16.             "model": "text-davinci-003"
  17.         },
  18.         "credential": {
  19.             "openAI_key": "..."
  20.         },
  21.         "actions": [
  22.             {
  23.                 "action_type": "predict",
  24.                 "method": "POST",
  25.                 "url": "https://${parameters.endpoint}/v1/completions",
  26.                 "headers": {
  27.                     "Authorization": "Bearer ${credential.openAI_key}"
  28.                 },
  29.                 "request_body": "{ \"model\": \"${parameters.model}\", \"prompt\": \"${parameters.prompt}\", \"max_tokens\": ${parameters.max_tokens}, \"temperature\": ${parameters.temperature} }"
  30.             }
  31.         ]
  32.     }
  33. }

Local connector

Admins are only required to enter their credential settings, such as "openAI_key", for the service they are connecting to. All other parameters are defined within the blueprint.

To create an internal connector, add the connector parameter to the Register model API, as shown in the following example:

  1. POST /_plugins/_ml/models/_register
  2. {
  3.     "name": "openAI-GPT-3.5: internal connector",
  4.     "function_name": "remote",
  5.     "model_group_id": "lEFGL4kB4ubqQRzegPo2",
  6.     "description": "test model",
  7.     "connector": {
  8.         "name": "OpenAI Connector",
  9.         "description": "The connector to public OpenAI model service for GPT 3.5",
  10.         "version": 1,
  11.         "protocol": "http",
  12.         "parameters": {
  13.             "endpoint": "api.openai.com",
  14.             "max_tokens": 7,
  15.             "temperature": 0,
  16.             "model": "text-davinci-003"
  17.         },
  18.         "credential": {
  19.             "openAI_key": "..."
  20.         },
  21.         "actions": [
  22.             {
  23.                 "action_type": "predict",
  24.                 "method": "POST",
  25.                 "url": "https://${parameters.endpoint}/v1/completions",
  26.                 "headers": {
  27.                     "Authorization": "Bearer ${credential.openAI_key}"
  28.                 },
  29.                 "request_body": "{ \"model\": \"${parameters.model}\", \"prompt\": \"${parameters.prompt}\", \"max_tokens\": ${parameters.max_tokens}, \"temperature\": ${parameters.temperature} }"
  30.             }
  31.         ]
  32.     }
  33. }
  34. }

Registering and deploying a connected model

After a connection has been created, use the connector_id from the response to register and deploy a connected model.

To register a model, you have the following options:

  • You can use model_group_id to register a model version to an existing model group.
  • If you do not use model_group_id, ML Commons creates a model with a new model group.

If you want to create a new model_group, use the following example:

  1. POST /_plugins/_ml/model_groups/_register
  2. {
  3.   "name": "remote_model_group",
  4.   "description": "This is an example description"
  5. }

ML Commons returns the following response:

  1. {
  2.   "model_group_id": "wlcnb4kBJ1eYAeTMHlV6",
  3.   "status": "CREATED"
  4. }

The following example registers a model named openAI-gpt-3.5-turbo:

  1. POST /_plugins/_ml/models/_register
  2. {
  3.     "name": "openAI-gpt-3.5-turbo",
  4.     "function_name": "remote",
  5.     "model_group_id": "wlcnb4kBJ1eYAeTMHlV6",
  6.     "description": "test model",
  7.     "connector_id": "a1eMb4kBJ1eYAeTMAljY"
  8. }

ML Commons returns the task_id and registration status of the model:

  1. {
  2.   "task_id": "cVeMb4kBJ1eYAeTMFFgj",
  3.   "status": "CREATED"
  4. }

You can use the task_id to find the model_id, as shown the following example:

GET task request

  1. GET /_plugins/_ml/tasks/cVeMb4kBJ1eYAeTMFFgj

GET task response

  1. {
  2.   "model_id": "cleMb4kBJ1eYAeTMFFg4",
  3.   "task_type": "REGISTER_MODEL",
  4.   "function_name": "REMOTE",
  5.   "state": "COMPLETED",
  6.   "worker_node": [
  7.     "XPcXLV7RQoi5m8NI_jEOVQ"
  8.   ],
  9.   "create_time": 1689793598499,
  10.   "last_update_time": 1689793598530,
  11.   "is_async": false
  12. }

Lastly, use the model_id to deploy the model:

Deploy model request

  1. POST /_plugins/_ml/models/cleMb4kBJ1eYAeTMFFg4/_deploy

Deploy model response

  1. {
  2.   "task_id": "vVePb4kBJ1eYAeTM7ljG",
  3.   "status": "CREATED"
  4. }

Use the task_id from the deploy model response to make sure the model deployment completes:

Verify deploy completion request

  1. GET /_plugins/_ml/tasks/vVePb4kBJ1eYAeTM7ljG

Verify deploy completion response

  1. {
  2.   "model_id": "cleMb4kBJ1eYAeTMFFg4",
  3.   "task_type": "DEPLOY_MODEL",
  4.   "function_name": "REMOTE",
  5.   "state": "COMPLETED",
  6.   "worker_node": [
  7.     "n-72khvBTBi3bnIIR8FTTw"
  8.   ],
  9.   "create_time": 1689793851077,
  10.   "last_update_time": 1689793851101,
  11.   "is_async": true
  12. }

After a successful deployment, you can test the model using the Predict API set in the connector’s action settings, as shown in the following example:

  1. POST /_plugins/_ml/models/cleMb4kBJ1eYAeTMFFg4/_predict
  2. {
  3.   "parameters": {
  4.     "messages": [
  5.       {
  6.         "role": "system",
  7.         "content": "You are a helpful assistant."
  8.       },
  9.       {
  10.         "role": "user",
  11.         "content": "Hello!"
  12.       }
  13.     ]
  14.   }
  15. }

The Predict API returns inference results for the connected model, as shown in the following example response:

  1. {
  2.   "inference_results": [
  3.     {
  4.       "output": [
  5.         {
  6.           "name": "response",
  7.           "dataAsMap": {
  8.             "id": "chatcmpl-7e6s5DYEutmM677UZokF9eH40dIY7",
  9.             "object": "chat.completion",
  10.             "created": 1689793889,
  11.             "model": "gpt-3.5-turbo-0613",
  12.             "choices": [
  13.               {
  14.                 "index": 0,
  15.                 "message": {
  16.                   "role": "assistant",
  17.                   "content": "Hello! How can I assist you today?"
  18.                 },
  19.                 "finish_reason": "stop"
  20.               }
  21.             ],
  22.             "usage": {
  23.               "prompt_tokens": 19,
  24.               "completion_tokens": 9,
  25.               "total_tokens": 28
  26.             }
  27.           }
  28.         }
  29.       ]
  30.     }
  31.   ]
  32. }

Examples

The following example connector requests show how to create a connector with supported third-party tools.

OpenAI chat connector

The following example creates a standalone OpenAI chat connector. The same options can be used for an internal connector under the connector parameter:

  1. POST /_plugins/_ml/connectors/_create
  2. {
  3.     "name": "OpenAI Chat Connector",
  4.     "description": "The connector to public OpenAI model service for GPT 3.5",
  5.     "version": 1,
  6.     "protocol": "http",
  7.     "parameters": {
  8.         "endpoint": "api.openai.com",
  9.         "model": "gpt-3.5-turbo"
  10.     },
  11.     "credential": {
  12.         "openAI_key": "..."
  13.     },
  14.     "actions": [
  15.         {
  16.             "action_type": "predict",
  17.             "method": "POST",
  18.             "url": "https://${parameters.endpoint}/v1/chat/completions",
  19.             "headers": {
  20.                 "Authorization": "Bearer ${credential.openAI_key}"
  21.             },
  22.             "request_body": "{ \"model\": \"${parameters.model}\", \"messages\": ${parameters.messages} }"
  23.         }
  24.     ]
  25. }

After creating the connector, you can retrieve the task_id and connector_id to register and deploy the model and then use the Predict API, similarly to a standalone connector.

Amazon SageMaker

The following example creates a standalone Amazon SageMaker connector. The same options can be used for an internal connector under the connector parameter:

  1. POST /_plugins/_ml/connectors/_create
  2. {
  3.     "name": "sagemaker: embedding",
  4.     "description": "Test connector for Sagemaker embedding model",
  5.     "version": 1,
  6.     "protocol": "aws_sigv4",
  7.     "credential": {
  8.         "access_key": "...",
  9.         "secret_key": "...",
  10.         "session_token": "..."
  11.     },
  12.     "parameters": {
  13.         "region": "us-west-2",
  14.         "service_name": "sagemaker"
  15.     },
  16.     "actions": [
  17.         {
  18.             "action_type": "predict",
  19.             "method": "POST",
  20.             "headers": {
  21.                 "content-type": "application/json"
  22.             },
  23.             "url": "https://runtime.sagemaker.${parameters.region}.amazonaws.com/endpoints/lmi-model-2023-06-24-01-35-32-275/invocations",
  24.             "request_body": "[\"${parameters.inputs}\"]"
  25.         }
  26.     ]
  27. }

The credential parameter contains the following options reserved for aws_sigv4 authentication:

  • access_key: Required. Provides the access key for the AWS instance.
  • secret_key: Required. Provides the secret key for the AWS instance.
  • session_token: Optional. Provides a temporary set of credentials for the AWS instance.

The parameters section requires the following options when using aws_sigv4 authentication:

  • region: The AWS Region in which the AWS instance is located.
  • service_name: The name of the AWS service for the connector.

Cohere

The following example request creates a standalone Cohere connection:

  1. POST /_plugins/_ml/connectors/_create
  2. {
  3.   "name": "YOUR CONNECTOR NAME",
  4.   "description": "YOUR CONNECTOR DESCRIPTION",
  5.   "version": "YOUR CONNECTOR VERSION",
  6.   "protocol": "http",
  7.   "credential": {
  8.     "cohere_key": "ADD YOUR Cohere API KEY HERE"
  9.   },
  10.   "parameters": {
  11.     "model": "embed-english-v2.0",
  12.     "truncate": "END"
  13.   },
  14.   "actions": [
  15.     {
  16.       "action_type": "predict",
  17.       "method": "POST",
  18.       "url": "https://api.cohere.ai/v1/embed",
  19.       "headers": {
  20.         "Authorization": "Bearer ${credential.cohere_key}"
  21.       },
  22.       "request_body": "{ \"texts\": ${parameters.texts}, \"truncate\": \"${parameters.truncate}\", \"model\": \"${parameters.model}\" }"
  23.     }
  24.   ]
  25. }

copy

Next steps