description: Official and Microsoft Certified Azure Storage Blob connector

Azure Blob

The Azure Blob output plugin allows ingesting your records into Azure Blob Storage service. This connector is designed to use the Append Blob and Block Blob API.

Our plugin works with the official Azure Service and also can be configured to be used with a service emulator such as Azurite.

Azure Storage Account

Before getting started, make sure you already have an Azure Storage account. As a reference, the following link explains step-by-step how to set up your account:

Configuration Parameters

We expose different configuration properties. The following table lists all the options available, and the next section has specific configuration details for the official service or the emulator.

Key Description default
account_name Azure Storage account name. This configuration property is mandatory
shared_key Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory.
container_name Name of the container that will contain the blobs. This configuration property is mandatory
blob_type Specify the desired blob type. Fluent Bit supports appendblob and blockblob. appendblob
auto_create_container If container_name does not exist in the remote service, enabling this option will handle the exception and auto-create the container. on
path Optional path to store your blobs. If your blob name is myblob, you can specify sub-directories where to store it using path, so setting path to /logs/kubernetes will store your blob in /logs/kubernetes/myblob.
emulator_mode If you want to send data to an Azure emulator service like Azurite, enable this option so the plugin will format the requests to the expected format. off
endpoint If you are using an emulator, this option allows you to specify the absolute HTTP address of such service. e.g: http://127.0.0.1:10000.
tls Enable or disable TLS encryption. Note that Azure service requires this to be turned on. off
compress Compression type for Azure blob objects. ‘gzip’ is currently the only supported value. The Content-Encoding HTTP Header will be set to ‘gzip’. none

Getting Started

As mentioned above, you can either deliver records to the official service or an emulator. Below we have an example for each use case.

Configuration for Azure Storage Service

The following configuration example generates a random message with a custom tag:

  1. [SERVICE]
  2.     flush     1
  3.     log_level info
  5. [INPUT]
  6.     name      dummy
  7.     dummy     {"name": "Fluent Bit", "year": 2020}
  8.     samples   1
  9.     tag       var.log.containers.app-default-96cbdef2340.log
  11. [OUTPUT]
  12.     name                  azure_blob
  13.     match                 *
  14.     account_name          YOUR_ACCOUNT_NAME
  15.     shared_key            YOUR_SHARED_KEY
  16.     path                  kubernetes
  17.     container_name        logs
  18.     auto_create_container on
  19.     tls                   on

After you run the configuration file above, you will be able to query the data using the Azure Storage Explorer. The example above will generate the following content in the explorer:

Azure Blob - 图1

Configuring and using Azure Emulator: Azurite

Install and run Azurite

The quickest way to get started is to install Azurite using npm:

  1. $ npm install -g azurite

then run the service:

  1. $ azurite
  2. Azurite Blob service is starting at http://127.0.0.1:10000
  3. Azurite Blob service is successfully listening at http://127.0.0.1:10000
  4. Azurite Queue service is starting at http://127.0.0.1:10001
  5. Azurite Queue service is successfully listening at http://127.0.0.1:10001

Configuring Fluent Bit for Azurite

Azurite comes with a default account_name and shared_key, so make sure to use the specific values provided in the example below (do an exact copy/paste):

  1. [SERVICE]
  2.     flush     1
  3.     log_level info
  5. [INPUT]
  6.     name      dummy
  7.     dummy     {"name": "Fluent Bit", "year": 2020}
  8.     samples   1
  9.     tag       var.log.containers.app-default-96cbdef2340.log
  11. [OUTPUT]
  12.     name                  azure_blob
  13.     match                 *
  14.     account_name          devstoreaccount1
  15.     shared_key            Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
  16.     path                  kubernetes
  17.     container_name        logs
  18.     auto_create_container on
  19.     tls                   off
  20.     emulator_mode         on
  21.     endpoint              http://127.0.0.1:10000

after running that Fluent Bit configuration you will see the data flowing into Azurite:

  1. $ azurite
  2. Azurite Blob service is starting at http://127.0.0.1:10000
  3. Azurite Blob service is successfully listening at http://127.0.0.1:10000
  4. Azurite Queue service is starting at http://127.0.0.1:10001
  5. Azurite Queue service is successfully listening at http://127.0.0.1:10001
  6. 127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "GET /devstoreaccount1/logs?restype=container HTTP/1.1" 404 -
  7. 127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs?restype=container HTTP/1.1" 201 -
  8. 127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log?comp=appendblock HTTP/1.1" 404 -
  9. 127.0.0.1 - - [03/Sep/2020:17:40:03 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log HTTP/1.1" 201 -
  10. 127.0.0.1 - - [03/Sep/2020:17:40:04 +0000] "PUT /devstoreaccount1/logs/kubernetes/var.log.containers.app-default-96cbdef2340.log?comp=appendblock HTTP/1.1" 201 -