Loki Storage

Loki needs to store two different types of data: chunks and indexes.

Loki receives logs in separate streams, where each stream is uniquely identifiedby its tenant ID and its set of labels. As log entries from a stream arrive,they are GZipped as “chunks” and saved in the chunks store. See chunkformat for how chunks are stored internally.

The index stores each stream’s label set and links them to the individualchunks.

Refer to Loki’s configuration for details onhow to configure the storage and the index.

For more information:

  1. Table Manager
  2. Retention

Supported Stores

The following are supported for the index:

The following are supported for the chunks:

Cloud Storage Permissions

S3

When using S3 as object storage, the following permissions are needed:

  • s3:ListBucket
  • s3:PutObject
  • s3:GetObject

DynamoDB

When using DynamoDB for the index, the following permissions are needed:

  • dynamodb:BatchGetItem
  • dynamodb:BatchWriteItem
  • dynamodb:DeleteItem
  • dynamodb:DescribeTable
  • dynamodb:GetItem
  • dynamodb:ListTagsOfResource
  • dynamodb:PutItem
  • dynamodb:Query
  • dynamodb:TagResource
  • dynamodb:UntagResource
  • dynamodb:UpdateItem
  • dynamodb:UpdateTable

Chunk Format

  1. -------------------------------------------------------------------
  2. | | |
  3. | MagicNumber(4b) | version(1b) |
  4. | | |
  5. -------------------------------------------------------------------
  6. | block-1 bytes | checksum (4b) |
  7. -------------------------------------------------------------------
  8. | block-2 bytes | checksum (4b) |
  9. -------------------------------------------------------------------
  10. | block-n bytes | checksum (4b) |
  11. -------------------------------------------------------------------
  12. | #blocks (uvarint) |
  13. -------------------------------------------------------------------
  14. | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  15. -------------------------------------------------------------------
  16. | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  17. -------------------------------------------------------------------
  18. | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  19. -------------------------------------------------------------------
  20. | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
  21. -------------------------------------------------------------------
  22. | checksum(from #blocks) |
  23. -------------------------------------------------------------------
  24. | metasOffset - offset to the point with #blocks |
  25. -------------------------------------------------------------------