Cloud

The following describes parameters used to sync indexes with cloud storage. Cloud object storage, the Hugging Face Hub and custom providers are all supported.

Parameters are set via the embeddings.load and embeddings.save methods.

provider

  1. provider: string

Cloud provider. Can be one of the following:

  • Cloud object storage. Set to one of these providers.

  • Hugging Face Hub. Set to huggingface-hub.

  • Custom providers. Set to the full class path of the custom provider.

container

  1. container: string

Container/bucket/directory/repository name. Embeddings will be stored in the container with the filename specified by the path configuration.

Cloud object storage configuration

In addition to the above common configuration, the cloud object storage provider has the following additional configuration parameters. Note that some cloud providers do not need any of these parameters and can use implicit authentication with service accounts.

See the libcloud documentation for more information on these parameters.

key

  1. key: string

Provider-specific access key. Can also be set via ACCESS_KEY environment variable. Ensure the configuration file is secured if added to the file. When using implicit authentication, set this to a value such as ‘using-implicit-auth’.

secret

  1. secret: string

Provider-specific access secret. Can also be set via ACCESS_SECRET environment variable. Ensure the configuration file is secured if added to the file. When using implicit authentication, this option is not required.

host

  1. host: string

Optional server host name. Set when using a local cloud storage server.

port

  1. port: int

Optional server port. Set when using a local cloud storage server.

token

  1. token: string

Optional temporary session token

region

  1. region: string

Optional parameter to specify the storage region, provider-specific.

Hugging Face Hub configuration

The huggingface-hub provider supports the following additional configuration parameters. More on these parameters can be found in the Hugging Face Hub’s documentation.

revision

  1. revision: string

Optional Git revision id which can be a branch name, a tag, or a commit hash

cache

  1. cache: string

Path to the folder where cached files are stored

token

  1. token: string|boolean

Token to be used for the download. If set to True, the token will be read from the Hugging Face config folder.