Configuration

The best practice for configuring microservices or cloud-native applications is to separate configuration files from application code - do not put configuration files in code repositories or package them into container images, but mount the configuration files or load them directly from the configuration center at runtime. Kratos’ config component is used to help applications load configurations from various sources.

Design Philosophy

1. Support for Multiple Configuration Sources

Kratos defines standardized Source and Watcher interfaces to adapt to various configuration sources.

The framework comes with built-in implementations of local file (file) and environment variable (env).

In addition, in contrib/config, we also provide adapters for the following configuration centers:

If the above configuration loading methods do not cover your environment, you can also implement the interface to adapt your own configuration loading method.

2. Support for Multiple Configuration Formats

The config component reuses the deserialization logic in encoding as the configuration parsing. It supports the following formats by default:

  • JSON
  • Protobuf
  • XML
  • YAML

The framework will parse the configuration file based on its type by matching the corresponding codec. You can also implement Codec and register it with the encoding.RegisterCodec method to parse other formats of configuration files.

The extraction of configuration file types varies slightly depending on the specific implementation of the configuration source. The built-in file source uses the file extension as the file type. Please refer to the documentation of the other configuration source plugins for their specific logic.

3. Hot Reloading

Kratos’ config component supports hot reloading of configurations. You can use the configuration center to update the configuration of a service online without re-deploying, stopping, or restarting the service, and modify some behaviors of the service.

4. Configuration Merge

In the config component, the configurations (files) from all configuration sources will be read one by one, parsed into maps, and merged into one map. Therefore, after loading, you don’t need to consider the file names or search for configurations by file names. Instead, you can use the structure of the contents to index the values of the configurations. When designing and writing configuration files, please note that the root-level keys in different configuration files should not be duplicated, otherwise they may be overwritten.

For example, if we have the following two configuration files:

  1. # File 1
  2. foo:
  3.   baz: "2"
  4.   biu: "example"
  5. hello:
  6.   a: b
  1. # File 2
  2. foo:
  3.   bar: 3
  4.   baz: aaaa
  5. hey:
  6.   good: bad
  7.   qux: quux

After calling .Load, they will be merged into the following structure:

  1. {
  2.   "foo": {
  3.     "baz": "aaaa",
  4.     "bar": 3,
  5.     "biu": "example"
  6.   },
  7.   "hey": {
  8.     "good": "bad",
  9.     "qux": "quux"
  10.   },
  11.   "hello": {
  12.     "a": "b"
  13.   }
  14. }

As we can see, the configurations from different files are merged separately, and when there is a key conflict, the values will be overwritten. The specific order of overwriting will be determined by the read order in the configuration source implementation. Therefore, I would like to remind you again that the root-level keys in different configuration files should not be duplicated, and do not rely on this overwrite feature to avoid problems caused by the overlapping of contents in different configuration files.

When using the configuration, you can use .Value("foo.bar") to directly get the value of a specific field, or use the .Scan method to read the entire map into a specific structure. Please refer to the following sections for specific usage.

Usage

1. Initialize Configuration Sources

Use file, which loads from a local file: Here, the path is the path to the configuration file. You can also specify a directory name, and all files in the directory will be parsed and loaded into the same map.

  1. import (
  2.     "github.com/go-kratos/kratos/v2/config"
  3.     "github.com/go-kratos/kratos/v2/config/file"
  4. )
  6. path := "configs/config.yaml"
  7. c := config.New(
  8.     config.WithSource(
  9.         file.NewSource(path),
  10.     ),
  11. )

If you want to use an external configuration center, you can find one in contrib/config. Taking Consul as an example:

  1. import (
  2.     "github.com/go-kratos/kratos/contrib/config/consul/v2"
  3.     "github.com/hashicorp/consul/api"
  4. )
  6. consulClient, err := api.NewClient(&api.Config{
  7.   Address: "127.0.0.1:8500",
  8. })
  9. if err != nil {
  10.   panic(err)
  11. }
  12. cs, err := consul.New(consulClient, consul.WithPath("app/cart/configs/"))
  13. if err != nil {
  14.   panic(err)
  15. }
  16. c := config.New(config.WithSource(cs))

Different configuration source plugins have slightly different usage methods. You can refer to their respective documentation or examples.

2. Read Configuration

First, define a structure to parse the fields of the configuration file. If you are using a project created with kratos-layout, you can refer to the section on kratos-layout below, which uses a .proto file to define the configuration and generate a struct.

Here, we demonstrate manually defining the structure. You need to use JSON tags to define the fields in your configuration file.

  1. var v struct {
  2.   Service struct {
  3.     Name    string `json:"name"`
  4.     Version string `json:"version"`
  5.   } `json:"service"`
  6. }

Using the initialized config instance, call the .Scan method to read the configuration file into the structure. This method is suitable for obtaining the entire content of the configuration file.

  1. // Unmarshal the config to struct
  2. if err := c.Scan(&v); err != nil {
  3.   panic(err)
  4. }
  5. fmt.Printf("config: %+v", v)

You can use the .Value method of the config instance to get the content of a specific field.

  1. name, err := c.Value("service.name").String()
  2. if err != nil {
  3.   panic(err)
  4. }
  5. fmt.Printf("service: %s", name)

3. Watch Configuration Changes

You can use the .Watch method to listen for changes to a specific field in the configuration. When there are configuration file changes in the local or remote configuration center, the callback function will be executed for custom processing.

  1. if err := c.Watch("service.name", func(key string, value config.Value) {
  2.   fmt.Printf("config changed: %s = %v\n", key, value)
  3.   // Write your callback logic here
  4. }); err != nil {
  5.   log.Error(err)
  6. }

4. Read Environment Variables

If there are configurations that need to be read from environment variables, please use the following method:

Configure the environment variable source env:

  1. c := config.New(
  2.     config.WithSource(
  3.         // Add environment variables with the prefix KRATOS_, or set it to an empty string if not needed
  4.         env.NewSource("KRATOS_"),
  5.         // Add configuration files
  6.         file.NewSource(path),
  7.     ))
  9. // Load configuration sources:
  10. if err := c.Load(); err != nil {
  11.     log.Fatal(err)
  12. }
  14. // Get the value of environment variable KRATOS_PORT, here we read it using the name without the prefix
  15. port, err := c.Value("PORT").String()

In addition to using the Value method mentioned above to read directly, you can also use placeholders in the configuration file to render the values from environment variables:

  1. service:
  2.   name: "kratos_app"
  3. http:
  4.   server:
  5.     # Use the value of service.name
  6.     name: "${service.name}"
  7.     # Replace with the environment variable PORT, if it does not exist, use the default value 8080
  8.     port: "${PORT:8080}"
  9.     # This format is not supported and will be treated as a regular string
  10.     timeout: "$TIMEOUT"

5. Configure the Decoder

The Decoder is used to parse the content of the configuration file using a specific deserialization method. By default, the default decoder automatically recognizes and parses the type of the file based on its type. In general, you do not need to customize this. You can register more file types by implementing the Codec as described in the following section.

You can override the Decoder behavior by adding the WithDecoder parameter when initializing the config. The following code shows how to configure a custom Decoder. Here, we use the yaml library to parse all configuration files. You can use this method to specify a specific configuration file parsing method, but it is recommended to implement the Codec as described in the following section to support multiple formats of parsing at the same time.

  1. import "gopkg.in/yaml.v2"
  3. c := config.New(
  4.   config.WithSource(
  5.     file.NewSource(flagconf),
  6.   ),
  7.   config.WithDecoder(func(kv *config.KeyValue, v map[string]interface{}) error {
  8.     return yaml.Unmarshal(kv.Value, v)
  9.   }),
  10. )

6. Configure the Resolver

The Resolver is used to further process the parsed map structure. The default resolver fills in placeholders in the configuration. You can override the behavior of the resolver by adding the WithResolver parameter when initializing the config.

  1. c := config.New(
  2.   config.WithSource(
  3.     file.NewSource(flagconf),
  4.   ),
  5.   config.WithResolver(func (input map[string]interface{}) (err error)  {
  6.     // Process the input here
  7.     // You may need to define a recursive function to process nested map structures
  8.     return 
  9.   }),
  10. )

7. Support for Other Configuration File Formats

First, implement the Codec. Here, we take YAML as an example.

  1. import (
  2.     "github.com/go-kratos/kratos/v2/encoding"
  3.     "gopkg.in/yaml.v3"
  4. )
  6. const Name = "myyaml"
  8. func init() {
  9.     encoding.RegisterCodec(codec{})
  10. }
  12. // codec is a Codec implementation with yaml.
  13. type codec struct{}
  15. func (codec) Marshal(v interface{}) ([]byte, error) {
  16.     return yaml.Marshal(v)
  17. }
  19. func (codec) Unmarshal(data []byte, v interface{}) error {
  20.     return yaml.Unmarshal(data, v)
  21. }
  23. func (codec) Name() string {
  24.     return Name
  25. }

Then, register the Codec. Since we put the registration code encoding.RegisterCodec(codec{}) in the init function of the package, it will be executed when the package is imported, which means it will be registered. Therefore, you can register it in the entry point of the code (e.g., main.go).

  1. import _ "path/to/your/codec"

After that, the config component will use the const Name = "myyaml" in the above code as the format type name and call the Codec to parse the file.

kratos-layout

Philosophy

1. Project Structure

The layout includes the following parts related to configuration files, and we will briefly introduce their roles:

  • cmd/server/main.go: This is the entry point of the service. We use the built-in config/file component to load the configuration file from the local file system by default. It will read the configs directory by default. You can modify the configuration source used in the config.New() parameter to load configurations from other sources (such as a configuration center). The configurations will be loaded into the conf.Bootstrap struct, and the content of this struct can be injected into other layers of the service, such as server or data, so that each layer can read the configurations it needs and complete its initialization.

  • configs/config.yaml: This is an example configuration file. The content in the configs directory is usually not used in the production environment of the service. You can use it to load configuration files for local development, so that the application can run locally for debugging. Do not put production environment configurations here.

  • internal/conf: Here, you put the structure definitions of the configuration files. We use .proto files to define the configurations here, and then use make config in the root directory to generate the corresponding .pb.go files in the same directory for use. In the initial state, the structure defined in the conf.proto is the same as the structure of configs/config.yaml, so please keep them consistent.

  • make config: This command in the Makefile is used to generate the .pb.go files corresponding to the .proto definitions (which actually calls protoc) in the internal directory. Remember to execute this command every time you modify the definitions to regenerate the go files.

2. Configuration Generation Command

We have already included the command for generating structs based on proto in the Makefile. You can generate the files by executing make config in the project root directory. It actually calls the protoc tool to scan the proto files in the internal directory and generate the corresponding .pb.go files.

3. Usage

The usage of reading configuration items, listening for configuration changes, and other advanced usages are the same as mentioned above, so we will not repeat them here.

Further Reading