Go Config

Go Config is a pluggable dynamic config library

Overview

Most config in applications are statically configured or include complex logic to load from multiple sources. Go-config makes this easy, pluggable and mergeable. You’ll never have to deal with config in the same way again.

Features

• Dynamic Loading - Load configuration from multiple source as and when needed. Go Config manages watching config sourcesin the background and automatically merges and updates an in memory view.

• Pluggable Sources - Choose from any number of sources to load and merge config. The backend source is abstracted away into a standard format consumed internally and decoded via encoders. Sources can be env vars, flags, file, etcd, k8s configmap, etc.

• Mergeable Config - If you specify multiple sources of config, regardless of format, they will be merged and presented in a single view. This massively simplifies priority order loading and changes based on environment.

• Observe Changes - Optionally watch the config for changes to specific values. Hot reload your app using Go Config’s watcher.You don’t have to handle ad-hoc hup reloading or whatever else, just keep reading the config and watch for changes if you need to be notified.

• Safe Recovery - In case config loads badly or is completely wiped away for some unknown reason, you can specify fallback values when accessing any config values directly. This ensures you’ll always be reading some sane default in the event of a problem.

Getting Started

• Source - A backend from which config is loaded
• Encoder - Handles encoding/decoding source config
• Reader - Merges multiple encoded sources as a single format
• Config - Config manager which manages multiple sources
• Usage - Example usage of go-config
• FAQ - General questions and answers
• TODO - TODO tasks/features

Sources

A Source is a backend from which config is loaded. Multiple sources can be used at the same time.

The following sources are supported:

• cli - read from parsed CLI flags
• consul - read from consul
• env - read from environment variables
• etcd - read from etcd v3
• file - read from file
• flag - read from flags
• memory - read from memory

There are also community-supported plugins, which support the following sources:

• configmap - read from k8s configmap
• grpc - read from grpc server
• runtimevar - read from Go Cloud Development Kit runtime variable
• url - read from URL
• vault - read from Vault server

TODO:

• git url

ChangeSet

Sources return config as a ChangeSet. This is a single internal abstraction for multiple backends.

type ChangeSet struct {
    // Raw encoded config data
    Data      []byte
    // MD5 checksum of the data
    Checksum  string
    // Encoding format e.g json, yaml, toml, xml
    Format    string
    // Source of the config e.g file, consul, etcd
    Source    string
    // Time of loading or update
    Timestamp time.Time
}

Encoder

An Encoder handles source config encoding/decoding. Backend sources may store config in many different formats. Encoders give us the ability to handle any format. If an Encoder is not specified it defaults to json.

The following encoding formats are supported:

• json
• yaml
• toml
• xml
• hcl

Reader

A Reader represents multiple changesets as a single merged and queryable set of values.

type Reader interface {
    // Merge multiple changeset into a single format
    Merge(...*source.ChangeSet) (*source.ChangeSet, error)
    // Return return Go assertable values
    Values(*source.ChangeSet) (Values, error)
    // Name of the reader e.g a json reader
    String() string
}

The reader makes use of Encoders to decode changesets into map[string]interface{} then merge them into a single changeset. It looks at the Format field to determine the Encoder. The changeset is then represented as a set of Values with the ability to retrive Go types and fallback where values cannot be loaded.

// Values is returned by the reader
type Values interface {
    // Return raw data
        Bytes() []byte
    // Retrieve a value
        Get(path ...string) Value
    // Return values as a map
        Map() map[string]interface{}
    // Scan config into a Go type
        Scan(v interface{}) error
}

The Value interface allows casting/type asserting to go types with fallback defaults.

type Value interface {
    Bool(def bool) bool
    Int(def int) int
    String(def string) string
    Float64(def float64) float64
    Duration(def time.Duration) time.Duration
    StringSlice(def []string) []string
    StringMap(def map[string]string) map[string]string
    Scan(val interface{}) error
    Bytes() []byte
}

Config

Config manages all config, abstracting away sources, encoders and the reader.

It manages reading, syncing, watching from multiple backend sources and represents them as a single merged and queryable source.

// Config is an interface abstraction for dynamic configuration
type Config interface {
        // provide the reader.Values interface
        reader.Values
    // Stop the config loader/watcher
    Close() error
    // Load config sources
    Load(source ...source.Source) error
    // Force a source changeset sync
    Sync() error
    // Watch a value for changes
    Watch(path ...string) (Watcher, error)
}

Usage

• Sample Config
• New Config
• Load File
• Read Config
• Read Values
• Watch Path
• Multiple Sources
• Set Source Encoder
• Add Reader Encoder

Sample Config

A config file can be of any format as long as we have an Encoder to support it.

Example json config:

{
    "hosts": {
        "database": {
            "address": "10.0.0.1",
            "port": 3306
        },
        "cache": {
            "address": "10.0.0.2",
            "port": 6379
        }
    }
}

New Config

Create a new config (or just make use of the default instance)

import "github.com/micro/go-micro/v2/config"
conf := config.NewConfig()

Load File

Load config from a file source. It uses the file extension to determine config format.

import (
    "github.com/micro/go-micro/v2/config"
)
// Load json config file
config.LoadFile("/tmp/config.json")

Load a yaml, toml or xml file by specifying a file with the appropriate file extension

// Load yaml config file
config.LoadFile("/tmp/config.yaml")

If an extension does not exist, specify the encoder

import (
    "github.com/micro/go-micro/v2/config"
    "github.com/micro/go-micro/v2/config/source/file"
)
enc := toml.NewEncoder()
// Load toml file with encoder
config.Load(file.NewSource(
        file.WithPath("/tmp/config"),
    source.WithEncoder(enc),
))

Read Config

Read the entire config as a map

// retrieve map[string]interface{}
conf := config.Map()
// map[cache:map[address:10.0.0.2 port:6379] database:map[address:10.0.0.1 port:3306]]
fmt.Println(conf["hosts"])

Scan the config into a struct

type Host struct {
        Address string `json:"address"`
        Port int `json:"port"`
}
type Config struct{
    Hosts map[string]Host `json:"hosts"`
}
var conf Config
config.Scan(&conf)
// 10.0.0.1 3306
fmt.Println(conf.Hosts["database"].Address, conf.Hosts["database"].Port)

Read Values

Scan a value from the config into a struct

type Host struct {
    Address string `json:"address"`
    Port int `json:"port"`
}
var host Host
config.Get("hosts", "database").Scan(&host)
// 10.0.0.1 3306
fmt.Println(host.Address, host.Port)

Read individual values as Go types

// Get address. Set default to localhost as fallback
address := config.Get("hosts", "database", "address").String("localhost")
// Get port. Set default to 3000 as fallback
port := config.Get("hosts", "database", "port").Int(3000)

Watch Path

Watch a path for changes. When the file changes the new value will be made available.

w, err := config.Watch("hosts", "database")
if err != nil {
    // do something
}
// wait for next value
v, err := w.Next()
if err != nil {
    // do something
}
var host Host
v.Scan(&host)

Multiple Sources

Multiple sources can be loaded and merged. Merging priority is in reverse order.

config.Load(
    // base config from env
    env.NewSource(),
    // override env with flags
    flag.NewSource(),
    // override flags with file
    file.NewSource(
        file.WithPath("/tmp/config.json"),
    ),
)

Set Source Encoder

A source requires an encoder to encode/decode data and specify the changeset format.

The default encoder is json. To change the encoder to yaml, xml, toml specify as an option.

e := yaml.NewEncoder()
s := consul.NewSource(
    source.WithEncoder(e),
)

Add Reader Encoder

The reader uses encoders to decode data from sources with different formats.

The default reader supports json, yaml, xml, toml and hcl. It represents the merged config as json.

Add a new encoder by specifying it as an option.

e := yaml.NewEncoder()
r := json.NewReader(
    reader.WithEncoder(e),
)

FAQ

How is this different from Viper?

Viper and go-config are solving the same problem. Go-config provides a different interface and is part of the larger micro ecosystem of tooling.

What’s the difference between Encoder and Reader?

The encoder is used by a backend source to encode/decode it’s data. The reader uses encoders to decode data from multiple sources with different formats, it then merges them into a single encoding format.

In the case of a file source , we use the file extension to determine the config format so the encoder is not used.

In the case of consul, etcd or similar key-value source we may load from a prefix containing multiple keys which means the source needs to understand the encoding so it can return a single changeset.

In the case of environment variables and flags we also need a way to encode the values as bytes and specify the format so it can later be merged by the reader.

Why is changeset data not represented as map[string]interface{}?

In some cases source data may not actually be key-value so it’s easier to represent it as bytes and defer decoding to the reader.