OpenSearch Service

Amazon OpenSearch Service (successor to Amazon Elasticsearch Service)

The OpenSearch Service in LocalStack lets you create one or more single-node OpenSearch clusters that behave like the Amazon OpenSearch Service. This service is, like its AWS counterpart, heavily linked with the Elasticsearch Service. Any cluster created with the OpenSearch Service will show up in the Elasticsearch Service and vice versa.

Creating an OpenSearch cluster

You can go ahead and use awslocal to create a new OpenSearch domain via the aws opensearch create-domain command.

Note: Every time when you create a cluster with a version of OpenSearch you haven’t used before, the OpenSearch binary for the respective version needs to be downloaded, which may take a while.

Note: The default OpenSearch version used is 1.1.0.

  1. $ awslocal opensearch create-domain --domain-name my-domain
  2. {
  3.     "DomainStatus": {
  4.         "DomainId": "000000000000/my-domain",
  5.         "DomainName": "my-domain",
  6.         "ARN": "arn:aws:es:us-east-1:000000000000:domain/my-domain",
  7.         "Created": true,
  8.         "Deleted": false,
  9.         "Endpoint": "my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566",
  10.         "Processing": true,
  11.         "UpgradeProcessing": false,
  12.         "EngineVersion": "OpenSearch_1.1",
  13.         "ClusterConfig": {
  14.             "InstanceType": "m3.medium.search",
  15.             "InstanceCount": 1,
  16.             "DedicatedMasterEnabled": true,
  17.             "ZoneAwarenessEnabled": false,
  18.             "DedicatedMasterType": "m3.medium.search",
  19.             "DedicatedMasterCount": 1,
  20.             "WarmEnabled": false,
  21.             "ColdStorageOptions": {
  22.                 "Enabled": false
  23.             }
  24.         },
  25.         "EBSOptions": {
  26.             "EBSEnabled": true,
  27.             "VolumeType": "gp2",
  28.             "VolumeSize": 10,
  29.             "Iops": 0
  30.         },
  31.         "AccessPolicies": "",
  32.         "SnapshotOptions": {
  33.             "AutomatedSnapshotStartHour": 0
  34.         },
  35.         "CognitoOptions": {
  36.             "Enabled": false
  37.         },
  38.         "EncryptionAtRestOptions": {
  39.             "Enabled": false
  40.         },
  41.         "NodeToNodeEncryptionOptions": {
  42.             "Enabled": false
  43.         },
  44.         "AdvancedOptions": {
  45.             "override_main_response_version": "false",
  46.             "rest.action.multi.allow_explicit_index": "true"
  47.         },
  48.         "ServiceSoftwareOptions": {
  49.             "CurrentVersion": "",
  50.             "NewVersion": "",
  51.             "UpdateAvailable": false,
  52.             "Cancellable": false,
  53.             "UpdateStatus": "COMPLETED",
  54.             "Description": "There is no software update available for this domain.",
  55.             "AutomatedUpdateDate": 0.0,
  56.             "OptionalDeployment": true
  57.         },
  58.         "DomainEndpointOptions": {
  59.             "EnforceHTTPS": false,
  60.             "TLSSecurityPolicy": "Policy-Min-TLS-1-0-2019-07",
  61.             "CustomEndpointEnabled": false
  62.         },
  63.         "AdvancedSecurityOptions": {
  64.             "Enabled": false,
  65.             "InternalUserDatabaseEnabled": false
  66.         },
  67.         "AutoTuneOptions": {
  68.             "State": "ENABLE_IN_PROGRESS"
  69.         }
  70.     }
  71. }

In the LocalStack log you will see something like, where you can see the cluster starting up in the background.

  1. 2022-01-13T10:36:29.436:INFO:localstack.services.opensearch.cluster: starting opensearch: /var/lib/localstack/libs/opensearch/1.1.0/bin/opensearch -E http.port=35403 -E http.publish_port=35403 -E transport.port=0 -E network.host=127.0.0.1 -E http.compression=false -E path.data="/var/lib/localstack/opensearch/arn:aws:es:us-east-1:000000000000:domain/my-domain/data" -E path.repo="/var/lib/localstack/opensearch/arn:aws:es:us-east-1:000000000000:domain/my-domain/backup" -E plugins.security.disabled=true with env {'OPENSEARCH_JAVA_OPTS': '-Xms200m -Xmx600m', 'OPENSEARCH_TMPDIR': '/var/lib/localstack/opensearch/arn:aws:es:us-east-1:000000000000:domain/my-domain/tmp'}
  2. 2022-01-13T10:36:29.437:INFO:localstack.services.opensearch.cluster: registering an endpoint proxy for http://my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566 => http://127.0.0.1:35403
  3. 2022-01-13T10:36:32.803:INFO:localstack.services.opensearch.cluster: [2022-01-13T10:36:32,800][INFO ][o.o.n.Node               ] [host-pc] version[1.1.0], pid[231895], build[tar/15e9f137622d878b79103df8f82d78d782b686a1/2021-10-04T21:29:03.079792Z], OS[Linux/5.11.0-46-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
  4. 2022-01-13T10:36:32.805:INFO:localstack.services.opensearch.cluster: [2022-01-13T10:36:32,805][INFO ][o.o.n.Node               ] [host-pc] JVM home [/var/lib/localstack/libs/opensearch/1.1.0/jdk], using bundled JDK [true]
  5. 2022-01-13T10:36:32.806:INFO:localstack.services.opensearch.cluster: [2022-01-13T10:36:32,805][INFO ][o.o.n.Node               ] [host-pc] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/lib/localstack/opensearch/arn:aws:es:us-east-1:000000000000:domain/my-domain/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms200m, -Xmx600m, -XX:MaxDirectMemorySize=314572800, -Dopensearch.path.home=/var/lib/localstack/libs/opensearch/1.1.0, -Dopensearch.path.conf=/var/lib/localstack/libs/opensearch/1.1.0/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
  6. ...

and after some time, you should see that the Processing state of the domain is set to false:

  1. $ awslocal opensearch describe-domain --domain-name my-domain | jq ".DomainStatus.Processing"
  2. false

Creating an Elasticsearch cluster

Like in AWS, the OpenSearch service can create Elasticsearch clusters and manage them. To do so, you can use awslocal and select an Elasticsearch version with the --engine-version parameter of the awslocal opensearch create-domain command. For an overview of existing Elasticsearch versions you can use awslocal opensearch list-versions.

Interact with the cluster

You can now interact with the cluster at the cluster API endpoint for the domain, in this case http://my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566.

For example:

  1. $ curl http://my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566
  2. {
  3.   "name" : "host-pc",
  4.   "cluster_name" : "opensearch",
  5.   "cluster_uuid" : "DMN-2TlwRkuhMH4aRRqrkA",
  6.   "version" : {
  7.     "distribution" : "opensearch",
  8.     "number" : "1.1.0",
  9.     "build_type" : "tar",
  10.     "build_hash" : "15e9f137622d878b79103df8f82d78d782b686a1",
  11.     "build_date" : "2021-10-04T21:29:03.079792Z",
  12.     "build_snapshot" : false,
  13.     "lucene_version" : "8.9.0",
  14.     "minimum_wire_compatibility_version" : "6.8.0",
  15.     "minimum_index_compatibility_version" : "6.0.0-beta1"
  16.   },
  17.   "tagline" : "The OpenSearch Project: https://opensearch.org/"
  18. }

Or the health endpoint:

  1. $ curl -s http://my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566/_cluster/health | jq .
  2. {
  3.   "cluster_name": "opensearch",
  4.   "status": "green",
  5.   "timed_out": false,
  6.   "number_of_nodes": 1,
  7.   "number_of_data_nodes": 1,
  8.   "discovered_master": true,
  9.   "active_primary_shards": 0,
  10.   "active_shards": 0,
  11.   "relocating_shards": 0,
  12.   "initializing_shards": 0,
  13.   "unassigned_shards": 0,
  14.   "delayed_unassigned_shards": 0,
  15.   "number_of_pending_tasks": 0,
  16.   "number_of_in_flight_fetch": 0,
  17.   "task_max_waiting_in_queue_millis": 0,
  18.   "active_shards_percent_as_number": 100
  19. }

Advanced topics

Endpoints

There are two configurable strategies that govern how domain endpoints are created. The strategy can be configured via the OPENSEARCH_ENDPOINT_STRATEGY environment variable.

ValueFormatDescription
domain<domain-name>.<region>.<engine-type>.localhost.localstack.cloud:4566This is the default strategy that uses the localhost.localstack.cloud domain to route to your localhost
pathlocalhost:4566/<engine-type>/<region>/<domain-name>An alternative that can be useful if you cannot resolve LocalStack’s localhost domain
portlocalhost:<port-from-range>Exposes the cluster(s) directly with ports from the external service port range

Regardless of the service from which the clusters were created, the domain of the cluster always corresponds to the engine type (OpenSearch or Elasticsearch) of the cluster. OpenSearch cluster therefore have opensearch in their domain (e.g. my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566) and Elasticsearch clusters have es in their domain (e.g. my-domain.us-east-1.es.localhost.localstack.cloud:4566)

Custom Endpoints

LocalStack allows you to set arbitrary custom endpoints for your clusters in the domain endpoint options. This can be used to overwrite the behavior of the endpoint strategies described above. You can also choose custom domains, however it is important to add the edge port (80/443 or by default 4566).

  1. $ awslocal opensearch create-domain --domain-name my-domain \
  2.       --domain-endpoint-options '{ "CustomEndpoint": "http://localhost:4566/my-custom-endpoint", "CustomEndpointEnabled": true }'

Once the domain processing is complete, you can access the cluster:

  1. $ curl http://localhost:4566/my-custom-endpoint/_cluster/health

Re-using a single cluster instance

In some cases, you may not want to create a new cluster instance for each domain, for example when you are only interested in testing API interactions instead of actual OpenSearch functionality. In this case, you can set OPENSEARCH_MULTI_CLUSTER=0, which will multiplex all domains to the same cluster. This can however lead to unexpected behavior when persisting data into OpenSearch, or creating clusters with different versions, so we do not recommend it.

Storage Layout

OpenSearch will be organized in your state directory as follows:

  1. localstack@machine % tree -L 4 ./volume/state
  2. ./volume/state
  3. ├── opensearch
  4. │   └── arn:aws:es:us-east-1:000000000000:domain
  5. │       ├── my-cluster-1
  6. │       │   ├── backup
  7. │       │   ├── data
  8. │       │   └── tmp
  9. │       ├── my-cluster-2
  10. │       │   ├── backup
  11. │       │   ├── data
  12. │       │   └── tmp

Custom OpenSearch backends

LocalStack downloads OpenSearch asynchronously the first time you run the aws opensearch create-domain, so you will get the response from LocalStack first and then (after download/install) you will have your OpenSearch cluster running locally. You may not want this, and instead use your already running OpenSearch cluster. This can also be useful when you want to run a cluster with a custom configuration that LocalStack does not support.

To customize the OpenSearch backend, you can start your own OpenSearch cluster locally and point LocalStack to it using the OPENSEARCH_CUSTOM_BACKEND environment variable. Note that only a single backend can be configured, meaning that you will get a similar behavior as when you re-use a single cluster instance.

Example

The following shows a sample docker-compose.yaml file that contains a single-node OpenSearch cluster and a basic LocalStack setup.

  1. version: "3.9"
  3. services:
  4.   opensearch:
  5.     container_name: opensearch
  6.     image: opensearchproject/opensearch:1.1.0
  7.     environment:
  8.       - node.name=opensearch
  9.       - cluster.name=opensearch-docker-cluster
  10.       - discovery.type=single-node
  11.       - bootstrap.memory_lock=true
  12.       - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
  13.       - "DISABLE_SECURITY_PLUGIN=true"
  14.     ports:
  15.       - "9200:9200"
  16.     ulimits:
  17.       memlock:
  18.         soft: -1
  19.         hard: -1
  20.     volumes:
  21.       - data01:/usr/share/opensearch/data
  23.   localstack:
  24.     container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
  25.     image: localstack/localstack
  26.     ports:
  27.       - "4566:4566"
  28.     depends_on:
  29.       - opensearch
  30.     environment:
  31.       - OPENSEARCH_CUSTOM_BACKEND=http://opensearch:9200
  32.       - DEBUG=${DEBUG- }
  33.       - PERSISTENCE=${PERSISTENCE- }
  34.       - DOCKER_HOST=unix:///var/run/docker.sock
  35.     volumes:
  36.       - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
  37.       - "/var/run/docker.sock:/var/run/docker.sock"
  39. volumes:
  40.   data01:
  41.     driver: local

  1. Run docker compose:

    1. $ docker-compose up -d

  2. Create the OpenSearch domain:

    1. $ awslocal opensearch create-domain --domain-name my-domain
    2. {
    3.     "DomainStatus": {
    4.         "DomainId": "000000000000/my-domain",
    5.         "DomainName": "my-domain",
    6.         "ARN": "arn:aws:es:us-east-1:000000000000:domain/my-domain",
    7.         "Created": true,
    8.         "Deleted": false,
    9.         "Endpoint": "my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566",
    10.         "Processing": false,
    11.         "UpgradeProcessing": false,
    12.         "EngineVersion": "OpenSearch_1.1",
    13.         "ClusterConfig": {
    14.             "InstanceType": "m3.medium.search",
    15.             "InstanceCount": 1,
    16.             "DedicatedMasterEnabled": true,
    17.             "ZoneAwarenessEnabled": false,
    18.             "DedicatedMasterType": "m3.medium.search",
    19.             "DedicatedMasterCount": 1,
    20.             "WarmEnabled": false,
    21.             "ColdStorageOptions": {
    22.                 "Enabled": false
    23.             }
    24.         },
    25.         "EBSOptions": {
    26.             "EBSEnabled": true,
    27.             "VolumeType": "gp2",
    28.             "VolumeSize": 10,
    29.             "Iops": 0
    30.         },
    31.         "AccessPolicies": "",
    32.         "SnapshotOptions": {
    33.             "AutomatedSnapshotStartHour": 0
    34.         },
    35.         "CognitoOptions": {
    36.             "Enabled": false
    37.         },
    38.         "EncryptionAtRestOptions": {
    39.             "Enabled": false
    40.         },
    41.         "NodeToNodeEncryptionOptions": {
    42.             "Enabled": false
    43.         },
    44.         "AdvancedOptions": {
    45.             "override_main_response_version": "false",
    46.             "rest.action.multi.allow_explicit_index": "true"
    47.         },
    48.         "ServiceSoftwareOptions": {
    49.             "CurrentVersion": "",
    50.             "NewVersion": "",
    51.             "UpdateAvailable": false,
    52.             "Cancellable": false,
    53.             "UpdateStatus": "COMPLETED",
    54.             "Description": "There is no software update available for this domain.",
    55.             "AutomatedUpdateDate": 0.0,
    56.             "OptionalDeployment": true
    57.         },
    58.         "DomainEndpointOptions": {
    59.             "EnforceHTTPS": false,
    60.             "TLSSecurityPolicy": "Policy-Min-TLS-1-0-2019-07",
    61.             "CustomEndpointEnabled": false
    62.         },
    63.         "AdvancedSecurityOptions": {
    64.             "Enabled": false,
    65.             "InternalUserDatabaseEnabled": false
    66.         },
    67.         "AutoTuneOptions": {
    68.             "State": "ENABLE_IN_PROGRESS"
    69.         }
    70.     }
    71. }

  3. If the Processing status is true, it means that the cluster is not yet healthy. You can run decribe-domain to receive the status:

    1. $ awslocal opensearch describe-domain --domain-name my-domain

  4. Check the cluster health endpoint and create indices:

    1. $ curl my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566/_cluster/health | jq
    2. {
    3.   "name": "host-pc",
    4.   "cluster_name": "opensearch",
    5.   "cluster_uuid": "DMN-2TlwRkuhMH4aRRqrkA",
    6.   "version": {
    7.     "distribution": "opensearch",
    8.     "number": "1.1.0",
    9.     "build_type": "tar",
    10.     "build_hash": "15e9f137622d878b79103df8f82d78d782b686a1",
    11.     "build_date": "2021-10-04T21:29:03.079792Z",
    12.     "build_snapshot": false,
    13.     "lucene_version": "8.9.0",
    14.     "minimum_wire_compatibility_version": "6.8.0",
    15.     "minimum_index_compatibility_version": "6.0.0-beta1"
    16.   },
    17.   "tagline": "The OpenSearch Project: https://opensearch.org/"
    18. }

  5. Create an example index:

    1. $ curl -X PUT my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566/my-index
    2. {"acknowledged":true,"shards_acknowledged":true,"index":"my-index"}

Differences to AWS

  • By default, AWS only sets the Endpoint attribute of the cluster status once the cluster is up. LocalStack will return the endpoint immediately, but keep Processing = "true" until the cluster has been started.
  • The CustomEndpointOptions allows arbitrary endpoint URLs, which is not allowed in AWS.

Troubleshooting

If you are using the OPENSEARCH_ENDPOINT_STRATEGY=domain (which is the default) and are having issues with resolving the subdomains, please check if your DNS blocks rebind queries.

Last modified July 26, 2022: fix some typos (#214) (6ab8502d)