Elastic GraphQL connector reference

Elastic GraphQL connector reference

The Elastic GraphQL connector is written in Python using the Elastic connector framework. View the source code for this connector.

Availability and prerequisites

This connector was introduced in Elastic 8.14.0, available as a self-managed self-managed connector.

To use this connector, satisfy all self-managed connector prerequisites. Importantly, you must deploy the connectors service on your own infrastructure. You have two deployment options:

This connector is in technical preview and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Technical preview features are not subject to the support SLA of official GA features.

Usage

To set up this connector in the UI, select the GraphQL tile when creating a new connector under Search → Connectors.

If you’re already familiar with how connectors work, you can also use the Connector APIs.

For additional operations, see Connectors UI in Kibana.

Deploy with Docker

You can deploy the GraphQL connector as a self-managed connector using Docker. Follow these instructions.

Step 1: Download sample configuration file

Download the sample configuration file. You can either download it manually or run the following command:

  1. curl https://raw.githubusercontent.com/elastic/connectors/main/config.yml.example --output ~/connectors-config/config.yml

Remember to update the --output argument value if your directory name is different, or you want to use a different config file name.

Step 2: Update the configuration file for your self-managed connector

Update the configuration file with the following settings to match your environment:

  • elasticsearch.host
  • elasticsearch.api_key
  • connectors

If you’re running the connector service against a Dockerized version of Elasticsearch and Kibana, your config file will look like this:

  1. # When connecting to your cloud deployment you should edit the host value
  2. elasticsearch.host: http://host.docker.internal:9200
  3. elasticsearch.api_key: <ELASTICSEARCH_API_KEY>
  5. connectors:
  6.   -
  7.     connector_id: <CONNECTOR_ID_FROM_KIBANA>
  8.     service_type: graphql
  9.     api_key: <CONNECTOR_API_KEY_FROM_KIBANA> # Optional. If not provided, the connector will use the elasticsearch.api_key instead

Using the elasticsearch.api_key is the recommended authentication method. However, you can also use elasticsearch.username and elasticsearch.password to authenticate with your Elasticsearch instance.

Note: You can change other default configurations by simply uncommenting specific settings in the configuration file and modifying their values.

Step 3: Run the Docker image

Run the Docker image with the Connector Service using the following command:

  1. docker run \
  2. -v ~/connectors-config:/config \
  3. --network "elastic" \
  4. --tty \
  5. --rm \
  6. docker.elastic.co/enterprise-search/elastic-connectors:8.17.0.0 \
  7. /app/bin/elastic-ingest \
  8. -c /config/config.yml

Refer to DOCKER.md in the elastic/connectors repo for more details.

Find all available Docker images in the official registry.

We also have a quickstart self-managed option using Docker Compose, so you can spin up all required services at once: Elasticsearch, Kibana, and the connectors service. Refer to this README in the elastic/connectors repo for more information.

Configuration

Configure GraphQL connector

Note the following configuration fields:

http_endpoint (required)

Base URL of the GraphQL endpoint. Example: https://api.xyz.com/graphql

http_method (required)

GET or POST.

authentication_method(required)

Select from No Auth, Basic Auth, and Bearer Token.

username

Required when using basic authentication.

password

Required when using basic authentication.

token

Required when using bearer token authentication.

graphql_query (required)

Query used to fetch data from the source. Can contain variables provided in the graphql_variables field. The connector will substitute the variables in the query with values from graphql_variables and make a GraphQL query to the source.

Example:

  1. query getUser($id: ID!) {
  2.     user(id: $id) {
  3.         name
  4.         email
  5.     }
  6. }

graphql_variables

A JSON object of key/value pairs containing variables used in the GraphQL query. The connector will substitute the variables in the query with the values provided here and make a GraphQL query to the source.

Example:

For the GraphQL query query getUser($id: ID!) { user(id: $id) { name } }

  • Where the value of graphql_variables is {"id": "123"}
  • The connector will execute query getUser { user(id: "123") { name } } to fetch data from the source

graphql_object_to_id_map (required)

A JSON mapping between GraphQL response objects to index and their ID fields. The connector will fetch data for each object (JSON key) and use the provided ID field (JSON value) to index the object into Elasticsearch. The connector will index all fields for each object specified in the mapping. Use dot (.) notation to specify the full path from the root of the GraphQL response to the desired object.

Example:

The GraphQL query query getUser { organization { users{ user_id name email} } } fetches all available users from the source. To index every user as a separate document configure this field as below.

  1. {
  2.     "organization.users": "user_id"
  3. }

In this example user_id is unique in every user document. Therefore, we set user_id as the value for organization.users.

The path provided in this field should only contain JSON objects and not lists.

headers

JSON object containing custom headers to be sent with each GraphQL request:

  1. {
  2.     "content-type": "Application/json"
  3. }

pagination_model (required)

This field specifies the pagination model to be used by the connector. The connector supports No pagination and Cursor-based pagination pagination models.

For cursor-based pagination, add pageInfo {endCursor hasNextPage} and an after argument variable in your query at the desired node (Pagination key). Use the after query argument with a variable to iterate through pages. The default value for this field is No pagination. Example:

For Cursor-based pagination, the query should look like this example:

  1. query getUsers($cursor: String!) {
  2.     sampleData {
  3.         users(after: $cursor) {
  4.             pageInfo {
  5.                 endCursor
  6.                 hasNextPage
  7.             }
  8.             nodes {
  9.                 first_name
  10.                 last_name
  11.                 address
  12.             }
  13.         }
  14.     }
  15. }

The value of pagination_key is sampleData.users so it must contain:

  • pageInfo {endCursor hasNextPage}
  • the after argument with a variable when using cursor-based pagination

pagination_key (required)

Specifies which GraphQL object is used for pagination. Use . to provide the full path of the object from the root of the response.

Example:

  • organization.users

connection_timeout

Specifies the maximum time in seconds to wait for a response from the GraphQL source. Default value is 30 seconds.

Documents and syncs

The connector syncs the objects and entities based on GraphQL Query and GraphQL Object List.

Sync types

Full syncs are supported by default for all connectors.

This connector currently does not support incremental syncs.

Sync rules

Basic sync rules are identical for all connectors and are available by default.

Advanced Sync Rules

Advanced sync rules are not available for this connector in the present version.

Connector Client operations

End-to-end Testing

The connector framework enables operators to run functional tests against a real data source, using Docker Compose. You don’t need a running Elasticsearch instance or GraphQL source to run this test.

Refer to Connector testing for more details.

To perform E2E testing for the GraphQL connector, run the following command:

  1. $ make ftest NAME=graphql

For faster tests, add the DATA_SIZE=small flag:

  1. make ftest NAME=graphql DATA_SIZE=small

By default, DATA_SIZE=MEDIUM.

Known issues

  • Every document will be updated in every sync.
  • If the same field name exists with different types across different objects, the connector might raise a mapping parser exception.

Refer to Known issues for a list of known issues for all connectors.

Troubleshooting

See Troubleshooting.

Security

See Security.