Logging Control

Overview

Logging in the peer and orderer is provided by the common/flogging package. This package supports

  • Logging control based on the severity of the message

  • Logging control based on the software logger generating the message

  • Different pretty-printing options based on the severity of the message

All logs are currently directed to stderr. Global and logger-level control of logging by severity is provided for both users and developers. There are currently no formalized rules for the types of information provided at each severity level. When submitting bug reports, developers may want to see full logs down to the DEBUG level.

In pretty-printed logs the logging level is indicated both by color and by a four-character code, e.g, “ERRO” for ERROR, “DEBU” for DEBUG, etc. In the logging context a logger is an arbitrary name (string) given by developers to groups of related messages. In the pretty-printed example below, the loggers ledgermgmt, kvledger, and peer are generating logs.

  1. 2018-11-01 15:32:38.268 UTC [ledgermgmt] initialize -> INFO 002 Initializing ledger mgmt
  2. 2018-11-01 15:32:38.268 UTC [kvledger] NewProvider -> INFO 003 Initializing ledger provider
  3. 2018-11-01 15:32:38.342 UTC [kvledger] NewProvider -> INFO 004 ledger provider Initialized
  4. 2018-11-01 15:32:38.357 UTC [ledgermgmt] initialize -> INFO 005 ledger mgmt initialized
  5. 2018-11-01 15:32:38.357 UTC [peer] func1 -> INFO 006 Auto-detected peer address: 172.24.0.3:7051
  6. 2018-11-01 15:32:38.357 UTC [peer] func1 -> INFO 007 Returning peer0.org1.example.com:7051

An arbitrary number of loggers can be created at runtime, therefore there is no “global list” of loggers, and logging control constructs can not check whether logging loggers actually do or will exist.

Logging specification

The logging levels of the peer and orderer commands are controlled by a logging specification, which is set via the FABRIC_LOGGING_SPEC environment variable.

The full logging level specification is of the form

  1. [<logger>[,<logger>...]=]<level>[:[<logger>[,<logger>...]=]<level>...]

Logging severity levels are specified using case-insensitive strings chosen from

  1. FATAL | PANIC | ERROR | WARNING | INFO | DEBUG

A logging level by itself is taken as the overall default. Otherwise, overrides for individual or groups of loggers can be specified using the

  1. <logger>[,<logger>...]=<level>

syntax. Examples of specifications:

  1. info                                        - Set default to INFO
  2. warning:msp,gossip=warning:chaincode=info   - Default WARNING; Override for msp, gossip, and chaincode
  3. chaincode=info:msp,gossip=warning:warning   - Same as above

Note

Logging specification terms are separated by a colon. If a term does not include a specific logger, for example info: then it is applied as the default log level across all loggers on the component. The string info:dockercontroller,endorser,chaincode,chaincode.platform=debug sets the default log level to INFO for all loggers and then the dockercontroller, endorser, chaincode, and chaincode.platform loggers are set to DEBUG. The order of the terms does not matter. In the examples above, the second and third options produce the same result although the order of the terms is reversed.

Logging format

The logging format of the peer and orderer commands is controlled via the FABRIC_LOGGING_FORMAT environment variable. This can be set to a format string, such as the default

  1. "%{color}%{time:2006-01-02 15:04:05.000 MST} [%{module}] %{shortfunc} -> %{level:.4s} %{id:03x}%{color:reset} %{message}"

to print the logs in a human-readable console format. It can be also set to json to output logs in JSON format.

Typical debug levels

This section provides some typical log levels to debug various areas of a peer node or ordering service node by setting the FABRIC_LOGGING_SPEC environment variable.

Peer smart contract debug:

  1. FABRIC_LOGGING_SPEC=info:dockercontroller,endorser,chaincode,chaincode.platform=debug

Peer private data debug:

  1. FABRIC_LOGGING_SPEC=info:kvledger,ledgerstorage,transientstore,pvtdatastorage,gossip.privdata=debug

Peer ledger and state database debug:

  1. FABRIC_LOGGING_SPEC=info:kvledger,lockbasedtxmgr,ledgerstorage,stateleveldb,statecouchdb,couchdb=debug

Peer full debug, with the noisy components set to info:

  1. FABRIC_LOGGING_SPEC=debug:cauthdsl,policies,msp,grpc,peer.gossip.mcs,gossip,leveldbhelper=info

Ordering node full debug, with the noisy components set to info:

  1. FABRIC_LOGGING_SPEC=debug:cauthdsl,policies,msp,grpc,orderer.consensus.etcdraft,orderer.common.cluster,orderer.common.cluster.step,common.configtx,blkstorage=info

Chaincode

Chaincode logging is the responsibility of the chaincode developer.

As independently executed programs, user-provided chaincodes may technically also produce output on stdout/stderr. While naturally useful for “devmode”, these channels are normally disabled on a production network to mitigate abuse from broken or malicious code. However, it is possible to enable this output even for peer-managed containers (e.g. “netmode”) on a per-peer basis via the CORE_VM_DOCKER_ATTACHSTDOUT=true configuration option.

Once enabled, each chaincode will receive its own logging channel keyed by its container-id. Any output written to either stdout or stderr will be integrated with the peer’s log on a per-line basis. It is not recommended to enable this for production.

Stdout and stderr not forwarded to the peer container can be viewed from the chaincode container using standard commands for your container platform.

  1. docker logs <chaincode_container_id>
  2. kubectl logs -n <namespace> <pod_name>
  3. oc logs -n <namespace> <pod_name>