This guide walks you through setting up the AI Proxy plugin with Amazon Bedrock.

Prerequisites

Configure the AI Proxy plugin

  1. Create a service in Kong Gateway that will represent the Bedrock API:

    1. curl -X POST http://localhost:8001/services \
    2.  --data "name=bedrock-service" \
    3.  --data "url=https://bedrock-runtime.us-east-1.amazonaws.com"

    Be sure to replace us-east-1 with the region for your environment.

  2. Create a route that maps to the service you defined:

    1. curl -i -X POST http://localhost:8001/routes \
    2.  --data "paths[]=/bedrock" \
    3.  --data "service.id=$(curl -s http://localhost:8001/services/bedrock-service | jq -r '.id')"

  3. Use the Kong Admin API to configure the AI Proxy Plugin to route requests to Amazon Bedrock:

    1. curl -X POST http://localhost:8001/services/bedrock-service/plugins \
    2.   --header 'Content-Type: application/json' \
    3.   --header 'accept: application/json' \
    4.   --data '{
    5.  "name": "ai-proxy",
    6.  "instance_name": "ai-proxy-bedrock",
    7.  "config": {
    8.    "route_type": "llm/v1/chat",
    9.    "auth": {
    10.      "allow_override": false,
    11.      "aws_access_key_id": "<YOUR_AWS_ACCESS_KEY_ID>",
    12.      "aws_secret_access_key": "<YOUR_AWS_SECRET_ACCESS_KEY>"
    13.    },
    14.    "model": {
    15.      "provider": "bedrock",
    16.      "name": "meta.llama3-70b-instruct-v1:0",
    17.      "options": {
    18.        "bedrock": {
    19.          "aws_region": "us-east-1"
    20.        }
    21.      }
    22.    }
    23.  }
    24.   }'

Be sure to replace the following values:

  • YOUR_AWS_ACCESS_KEY_ID
  • YOUR_AWS_SECRET_ACCESS_KEY
  • Configure the correct provider model (config.model.name) and AWS region (config.model.options.bedrock.aws_region) for your environment.

Test the configuration

Make an llm/v1/chat type request to test your new endpoint:

  1. curl -X POST http://localhost:8000/bedrock \
  2. -H 'Content-Type: application/json' \
  3. --data-raw '{ "messages": [ { "role": "system", "content": "You are a mathematician" }, { "role": "user", "content": "What is 1+1?"} ] }'

Previous Set up AI Proxy with OpenAI

Next Set up AI Proxy with Gemini