Cloud Discovery

Table of Contents

Amazon EC2 Discovery

CrateDB has native discovery support when running a cluster with Amazon Web Services (AWS). The discovery mechanism uses the Amazon EC2 API to to generate the list of hosts for the unicast host discovery (see Unicast Host Discovery).

There is a best practice how to configure and run a CrateDB cluster on Amazon EC2.

Microsoft Azure Discovery

CrateDB has native discovery support when running a cluster on Microsoft Azure infrastructure. The discovery mechanism uses the Azure Resource Management (ARM) API to generate the list of hosts for the unicast host discovery (see Unicast Host Discovery).

The discovery mechanism is implemented as a plugin and resides within the plugin folder of the CrateDB installation. However, since the Azure Java SDKs does have a lot of dependencies and we want to keep the plugin jar small in size the dependencies are not included in the CrateDB distribution.

Requirements

To make the plugin work you have to add the following Java libraries to the $CRATE_HOME/plugins/crate-azure-discovery folder:

  1. activation-1.1.jar
  2. adal4j-1.0.0.jar
  3. azure-core-0.9.3.jar
  4. azure-mgmt-compute-0.9.3.jar
  5. azure-mgmt-network-0.9.3.jar
  6. azure-mgmt-resources-0.9.3.jar
  7. azure-mgmt-storage-0.9.3.jar
  8. azure-mgmt-utility-0.9.3.jar
  9. bcprov-jdk15on-1.51.jar
  10. commons-io-2.4.jar
  11. commons-lang-2.6.jar
  12. commons-lang3-3.3.1.jar
  13. gson-2.2.4.jar
  14. jackson-core-asl-1.9.2.jar
  15. jackson-jaxrs-1.9.2.jar
  16. jackson-mapper-asl-1.9.2.jar
  17. jackson-xc-1.9.2.jar
  18. javax.inject-1.jar
  19. jaxb-api-2.2.2.jar
  20. jaxb-impl-2.2.3-1.jar
  21. jcip-annotations-1.0.jar
  22. jersey-client-1.13.jar
  23. jersey-core-1.13.jar
  24. jersey-json-1.13.jar
  25. jettison-1.1.jar
  26. json-smart-1.1.1.jar
  27. lang-tag-1.4.jar
  28. mail-1.4.7.jar
  29. nimbus-jose-jwt-3.1.2.jar
  30. oauth2-oidc-sdk-4.5.jar
  31. stax-api-1.0-2.jar

You can download the libraries using a simple build.gradle file:

  1. apply plugin: "java"
  3. repositories {
  4.   jcenter()
  5. }
  7. dependencies {
  8.     compile('com.microsoft.azure:azure-mgmt-utility:0.9.3') {
  9.         exclude group: 'stax', module: 'stax-api'
  10.         exclude group: 'org.slf4j', module: 'slf4j-api'
  11.         exclude group: 'commons-logging', module: 'commons-logging'
  12.         exclude group: 'commons-codec', module: 'commons-codec'
  13.         exclude group: 'com.fasterxml.jackson.core', module: 'jackson-core'
  14.         exclude group: 'org.apache.httpcomponents', module: 'httpclient'
  15.     }
  16. }
  18. task azureLibs (type: Copy, dependsOn: ['compileJava']) {
  19.   from configurations.testRuntime
  20.   into "libs"
  21. }

Running gradle azureLibs will fetch the required jars and put them into the libs/ folder from where you can copy them into the plugin folder.

Basic Configuration

To enable Azure discovery simply change the discovery.zen.hosts_provider setting to azure:

  1. discovery.zen.hosts_provider: azure

The discovery mechanism can discover CrateDB instances within the same vnet or the same subnet of the same resource group. By default it will the vnet.

You can change the behaviour using the discovery.azure.method setting:

  1. discovery.azure.method: subnet

The used resource group also needs to be provided:

  1. cloud.azure.management.resourcegroup.name: production

Authentication

The discovery plugin requires authentication as service principle. To do so, you have to create an Active Directory application with a password. We recommened to follow the AD Application Guide .

The configuration settings for authentication are as follows:

  1. cloud.azure.management:
  2.   subscription.id: my-id
  3.   tenant.id: my-tenant
  4.   app:
  5.     id: my-app
  6.     secret: my-secret

For a complete list of settings please refer to Discovery on Microsoft Azure.