The Operations Service

The peer and the orderer host an HTTP server that offers a RESTful “operations” API. This API is unrelated to the Fabric network services and is intended to be used by operators, not administrators or “users” of the network.

The API exposes the following capabilities:

  • Log level management

  • Health checks

  • Prometheus target for operational metrics (when configured)

  • Endpoint for retrieving version information

Configuring the Operations Service

The operations service requires two basic pieces of configuration:

  • The address and port to listen on.

  • The TLS certificates and keys to use for authentication and encryption. Note, these certificates should be generated by a separate and dedicated CA. Do not use a CA that has generated certificates for any organizations in any channels.

Peer

For each peer, the operations server can be configured in the operations section of core.yaml:

  1. operations:
  2.   # host and port for the operations server
  3.   listenAddress: 127.0.0.1:9443
  5.   # TLS configuration for the operations endpoint
  6.   tls:
  7.     # TLS enabled
  8.     enabled: true
  10.     # path to PEM encoded server certificate for the operations server
  11.     cert:
  12.       file: tls/server.crt
  14.     # path to PEM encoded server key for the operations server
  15.     key:
  16.       file: tls/server.key
  18.     # most operations service endpoints require client authentication when TLS
  19.     # is enabled. clientAuthRequired requires client certificate authentication
  20.     # at the TLS layer to access all resources.
  21.     clientAuthRequired: false
  23.     # paths to PEM encoded ca certificates to trust for client authentication
  24.     clientRootCAs:
  25.       files: []

The listenAddress key defines the host and port that the operation server will listen on. If the server should listen on all addresses, the host portion can be omitted.

The tls section is used to indicate whether or not TLS is enabled for the operations service, the location of the service’s certificate and private key, and the locations of certificate authority root certificates that should be trusted for client authentication. When enabled is true, most of the operations service endpoints require client authentication, therefore clientRootCAs.files must be set. When clientAuthRequired is true, the TLS layer will require clients to provide a certificate for authentication on every request. See Operations Security section below for more details.

Orderer

For each orderer, the operations server can be configured in the Operations section of orderer.yaml:

  1. Operations:
  2.   # host and port for the operations server
  3.   ListenAddress: 127.0.0.1:8443
  5.   # TLS configuration for the operations endpoint
  6.   TLS:
  7.     # TLS enabled
  8.     Enabled: true
  10.     # PrivateKey: PEM-encoded tls key for the operations endpoint
  11.     PrivateKey: tls/server.key
  13.     # Certificate governs the file location of the server TLS certificate.
  14.     Certificate: tls/server.crt
  16.     # Paths to PEM encoded ca certificates to trust for client authentication
  17.     ClientRootCAs: []
  19.     # Most operations service endpoints require client authentication when TLS
  20.     # is enabled. ClientAuthRequired requires client certificate authentication
  21.     # at the TLS layer to access all resources.
  22.     ClientAuthRequired: false

The ListenAddress key defines the host and port that the operations server will listen on. If the server should listen on all addresses, the host portion can be omitted.

The TLS section is used to indicate whether or not TLS is enabled for the operations service, the location of the service’s certificate and private key, and the locations of certificate authority root certificates that should be trusted for client authentication. When Enabled is true, most of the operations service endpoints require client authentication, therefore RootCAs must be set. When ClientAuthRequired is true, the TLS layer will require clients to provide a certificate for authentication on every request. See Operations Security section below for more details.

Operations Security

As the operations service is focused on operations and intentionally unrelated to the Fabric network, it does not use the Membership Services Provider for access control. Instead, the operations service relies entirely on mutual TLS with client certificate authentication.

When TLS is disabled, authorization is bypassed and any client that can connect to the operations endpoint will be able to use the API.

When TLS is enabled, a valid client certificate must be provided in order to access the logging and metrics services. The health check and version services only require a valid client certificate when clientAuthRequired is enabled, since these services are often used by network operators and only provide read-only information.

When clientAuthRequired is enabled, the TLS layer will also require a valid client certificate regardless of the resource being accessed.

Log Level Management

The operations service provides a /logspec resource that operators can use to manage the active logging spec for a peer or orderer. The resource is a conventional REST resource and supports GET and PUT requests.

When a GET /logspec request is received by the operations service, it will respond with a JSON payload that contains the current logging specification:

  1. {"spec":"info"}

When a PUT /logspec request is received by the operations service, it will read the body as a JSON payload. The payload must consist of a single attribute named spec.

  1. {"spec":"chaincode=debug:info"}

If the spec is activated successfully, the service will respond with a 204 "No Content" response. If an error occurs, the service will respond with a 400 "Bad Request" and an error payload:

  1. {"error":"error message"}

When TLS is enabled, a valid client certificate is required to use this service regardless of whether clientAuthRequired is set to true at the TLS level.

Metrics

Some components of the Fabric peer and orderer expose metrics that can help provide insight into the behavior of the system. Operators and administrators can use this information to better understand how the system is performing over time.

Configuring Metrics

Fabric provides two ways to expose metrics: a pull model based on Prometheus and a push model based on StatsD.

Prometheus

A typical Prometheus deployment scrapes metrics by requesting them from an HTTP endpoint exposed by instrumented targets. As Prometheus is responsible for requesting the metrics, it is considered a pull system.

When configured, a Fabric peer or orderer will present a /metrics resource on the operations service.

When TLS is enabled, a valid client certificate is required to use this service regardless of whether clientAuthRequired is set to true at the TLS level.

Peer

A peer can be configured to expose a /metrics endpoint for Prometheus to scrape by setting the metrics provider to prometheus in the metrics section of core.yaml.

  1. metrics:
  2.   provider: prometheus

Orderer

An orderer can be configured to expose a /metrics endpoint for Prometheus to scrape by setting the metrics provider to prometheus in the Metrics section of orderer.yaml.

  1. Metrics:
  2.   Provider: prometheus

StatsD

StatsD is a simple statistics aggregation daemon. Metrics are sent to a statsd daemon where they are collected, aggregated, and pushed to a backend for visualization and alerting. As this model requires instrumented processes to send metrics data to StatsD, this is considered a push system.

Peer

A peer can be configured to send metrics to StatsD by setting the metrics provider to statsd in the metrics section of core.yaml. The statsd subsection must also be configured with the address of the StatsD daemon, the network type to use (tcp or udp), and how often to send the metrics. An optional prefix may be specified to help differentiate the source of the metrics — for example, differentiating metrics coming from separate peers — that would be prepended to all generated metrics.

  1. metrics:
  2.   provider: statsd
  3.   statsd:
  4.     network: udp
  5.     address: 127.0.0.1:8125
  6.     writeInterval: 10s
  7.     prefix: peer-0

Orderer

An orderer can be configured to send metrics to StatsD by setting the metrics provider to statsd in the Metrics section of orderer.yaml. The Statsd subsection must also be configured with the address of the StatsD daemon, the network type to use (tcp or udp), and how often to send the metrics. An optional prefix may be specified to help differentiate the source of the metrics.

  1. Metrics:
  2.     Provider: statsd
  3.     Statsd:
  4.       Network: udp
  5.       Address: 127.0.0.1:8125
  6.       WriteInterval: 30s
  7.       Prefix: org-orderer

For a look at the different metrics that are generated, check out Metrics Reference.

Health Checks

The operations service provides a /healthz resource that operators can use to help determine the liveness and health of peers and orderers. The resource is a conventional REST resource that supports GET requests. The implementation is intended to be compatible with the liveness probe model used by Kubernetes but can be used in other contexts.

When a GET /healthz request is received, the operations service will call all registered health checkers for the process to ensure all registered services and dependencies are available. When all of the health checkers return successfully, the operations service will respond with a 200 "OK" and a JSON body:

  1. {
  2.   "status": "OK",
  3.   "time": "2009-11-10T23:00:00Z"
  4. }

If one or more of the health checkers returns an error, the operations service will respond with a 503 "Service Unavailable" and a JSON body that includes information about which health checker failed:

  1. {
  2.   "status": "Service Unavailable",
  3.   "time": "2009-11-10T23:00:00Z",
  4.   "failed_checks": [
  5.     {
  6.       "component": "docker",
  7.       "reason": "failed to connect to Docker daemon: invalid endpoint"
  8.     }
  9.   ]
  10. }

The peer has the following health checks available:

  • Docker daemon health check (if a Docker endpoint is configured for chaincodes)

  • CouchDB health check (if CouchDB is configured as the state database)

When TLS is enabled, a valid client certificate is not required to use this service unless clientAuthRequired is set to true.

Version

The orderer and peer both expose a /version endpoint. This endpoint serves a JSON document containing the orderer or peer version and the commit SHA on which the release was created.

When TLS is enabled, a valid client certificate is not required to use this service unless clientAuthRequired is set to true.