Checklist for a production ordering node

As you prepare to build a production ordering service (or a single ordering node), you need to customize the configuration by editing the orderer.yaml file, which is copied into the /config directory when downloading the Fabric binaries, and available within the Fabric ordering node image at /etc/hyperledger/fabric/orderer.yaml.

While in a production environment you could override the environment variables in the orderer.yaml file in your Docker container or your Kubernetes job, these instructions show how to edit orderer.yaml instead. It’s important to understand the parameters in the configuration file and their dependencies on other parameter settings in the file. Blindly overriding one setting using an environment variable could affect the functionality of another setting. Therefore, the recommendation is that before starting the ordering node, you make the modifications to the settings in the configuration file to become familiar with the available settings and how they work. Afterwards, you may choose to override these parameters using environment variables.

This checklist covers key configuration parameters for setting up a production ordering service. Of course, you can always refer to the orderer.yaml file for additional parameters or more information. It also provides guidance on which parameters should be overridden. The list of parameters that you need to understand and that are described in this topic include:

General.ListenAddress

  1. # Listen address: The IP on which to bind to listen.
  2. ListenAddress: 127.0.0.1
  • ListenAddress: (default value should be overridden) This is the location where the orderer will listen, for example, 0.0.0.0. Note: unlike the peer, the orderer.yaml separates the address and the port, hence the General.ListenPort parameter.

General.ListenPort

  1. # Listen port: The port on which to bind to listen.
  2. ListenPort: 7050
  • ListenPort: (default value should be overridden) This is the port that the orderer listens on.

General.TLS

  1. # Require server-side TLS
  2. Enabled: false
  3. # PrivateKey governs the file location of the private key of the TLS certificate.
  4. PrivateKey: tls/server.key
  5. # Certificate governs the file location of the server TLS certificate.
  6. Certificate: tls/server.crt
  7. # RootCAs contains a list additional root certificates used for verifying certificates
  8. # of other orderer nodes during outbound connections.
  9. # It is not required to be set, but can be used to augment the set of TLS CA certificates
  10. # available from the MSPs of each channel’s configuration.
  11. RootCAs:
  12.   - tls/ca.crt
  13. # Require client certificates / mutual TLS for inbound connections.
  14. ClientAuthRequired: false
  15. # If mutual TLS is enabled, ClientRootCAs contains a list of additional root certificates
  16. # used for verifying certificates of client connections.
  17. # It is not required to be set, but can be used to augment the set of TLS CA certificates
  18. # available from the MSPs of each channel’s configuration.
  19. ClientRootCAs:

  • Enabled: (default value should be overridden) In a production network, you should be using TLS-secured communications. This value should be true.

  • PrivateKey: (default value should be overridden). Provide the path to, and filename of, the private key generated by your TLS CA for this node.

  • Certificate: (default value should be overridden) Provide the path to, and filename of, the public certificate (also known as the sign certificate) generated by your TLS CA for this node.

  • RootCAs: (should be commented out) This parameter is typically unset for normal use. It is a list of the paths to additional root certificates used for verifying certificates of other orderer nodes during outbound connections. It can be used to augment the set of TLS CA certificates available from the MSPs of each channel’s configuration.

  • ClientAuthRequired: (Mutual TLS only) Setting this value to “true” will enable mutual TLS on your network, and must be done for the entire network, not just one node.

  • ClientRootCAs: (Mutual TLS only) Can be left blank if mutual TLS is disabled. If mutual TLS is enabled, this is a list of the paths to additional root certificates used for verifying certificates of client connections. It can be used to augment the set of TLS CA certificates available from the MSPs of each channel’s configuration.

General.KeepAlive

The KeepAlive values might need to be tuned for compatibility with any networking devices or software (like firewalls or proxies) in between components of your network. Ideally, these settings would be manipulated if needed in a test or pre-prod environment and then set accordingly for your production environment.

  1. # ServerMinInterval is the minimum permitted time between client pings.
  2. # If clients send pings more frequently, the server will
  3. # disconnect them.
  4. ServerMinInterval: 60s
  5. # ServerInterval is the time between pings to clients.
  6. ServerInterval: 7200s
  7. # ServerTimeout is the duration the server waits for a response from
  8. # a client before closing the connection.
  9. ServerTimeout: 20s

  • ServerMinInterval: (default value should not be overridden, unless determined necessary through testing)

  • ServerInterval: (default value should not be overridden, unless determined necessary through testing)

  • ServerTimeout: (default value should not be overridden, unless determined necessary through testing)

General.Cluster

  1. # SendBufferSize is the maximum number of messages in the egress buffer.
  2. # Consensus messages are dropped if the buffer is full, and transaction
  3. # messages are waiting for space to be freed.
  4. SendBufferSize: 100
  5. # ClientCertificate governs the file location of the client TLS certificate
  6. # If not set, the server General.TLS.Certificate is re-used.
  7. ClientCertificate:
  8. # If not set, the server General.TLS.PrivateKey is re-used.
  9. ClientPrivateKey:
  10. # The below 4 properties should be either set together, or be unset together.
  11. # If they are set, then the orderer node uses a separate listener for intra-cluster
  12. # communication. If they are unset, then the general orderer listener is used.
  13. # This is useful if you want to use a different TLS server certificates on the
  14. # client-facing and the intra-cluster listeners.
  16. # ListenPort defines the port on which the cluster listens to connections.
  17. ListenPort:
  18. # ListenAddress defines the IP on which to listen to intra-cluster communication.
  19. ListenAddress:
  20. # ServerCertificate defines the file location of the server TLS certificate used for intra-cluster
  21. # communication.
  22. ServerCertificate:
  23. # ServerPrivateKey defines the file location of the private key of the TLS certificate.
  24. ServerPrivateKey:

If unset, the ClientCertificate and ClientPrivateKey default to the server General.TLS.Certificate and General.TLS.PrivateKey when the orderer is not configured to use a separate cluster port.

  • ClientCertificate: Provide the path to, and filename of, the public certificate (also known as a signed certificate) generated by your TLS CA for this node.

  • ClientPrivateKey: Provide the path to, and filename of, the private key generated by your TLS CA for this node.

In general, these four parameters would only need to be configured if you want to configure a separate listener and TLS certificates for intra-cluster communication (with other Raft orderers), as opposed to using the listener that peer clients and application clients utilize. This is an advanced deployment option. These four parameters should be set together or left unset, and if they are set, note that the ClientCertificate and ClientPrivateKey must be set as well.

  • ListenPort

  • ListenAddress

  • ServerCertificate

  • ServerPrivateKey

General.BoostrapMethod

  1. # Bootstrap method: The method by which to obtain the bootstrap block
  2. # system channel is specified. The option can be one of:
  3. #   "file" - path to a file containing the genesis block or config block of system channel
  4. #   "none" - allows an orderer to start without a system channel configuration
  5. BootstrapMethod: file
  • BootstrapMethod: If you plan to create this node on a network that is not using a system channel, override this value to none and then ensure that ChannelParticipation.Enabled is set to true, otherwise you will get an error when you attempt to start the node. If you are creating a node to be joined to a system channel, unless you plan to use a file type other than “file”, this value should be left as is.

General.BoostrapFile

  1. # Bootstrap file: The file containing the bootstrap block to use when
  2. # initializing the orderer system channel and BootstrapMethod is set to
  3. # "file".  The bootstrap file can be the genesis block, and it can also be
  4. # a config block for late bootstrap of some consensus methods like Raft.
  5. # Generate a genesis block by updating $FABRIC_CFG_PATH/configtx.yaml and
  6. # using configtxgen command with "-outputBlock" option.
  7. # Defaults to file "genesisblock" (in $FABRIC_CFG_PATH directory) if not specified.
  8. BootstrapFile:
  • BoostrapFile: (if you are creating this node to be joined to a system channel, the default value should be overridden) Specify the location and name of the system channel genesis block to use when this node is created. If you are creating this node without using a system channel, this value will not be used, and can therefore be left blank.

General.LocalMSPDir

  1. # LocalMSPDir is where to find the private crypto material needed by the
  2. # orderer. It is set relative here as a default for dev environments but
  3. # should be changed to the real location in production.
  4. LocalMSPDir: msp

LocalMSPDir: (default value will often be overridden) This is the path to the ordering node’s local MSP, which must be created before it can be deployed. The path can be absolute or relative to FABRIC_CFG_PATH (by default, it is /etc/hyperledger/fabric in the orderer image). Unless an absolute path is specified to a folder named something other than “msp”, the ordering node defaults to looking for a folder called “msp” at the path (in other words, FABRIC_CFG_PATH/msp) and when using the orderer image: /etc/hyperledger/fabric/msp. If you are using the recommended folder structure described in the Registering and enrolling identities with a CA topic, it would be relative to the FABRIC_CFG_PATH as follows: config/organizations/ordererOrganizations/org0.example.com/orderers/orderer0.org0.example.com/msp. The best practice is to store this data in persistent storage. This prevents the MSP from being lost if your orderer containers are destroyed for some reason.

General.LocalMSPID

  1. # LocalMSPID is the identity to register the local MSP material with the MSP
  2. # manager. IMPORTANT: The local MSP ID of an orderer needs to match the MSP
  3. # ID of one of the organizations defined in the orderer system channel's
  4. # /Channel/Orderer configuration. The sample organization defined in the
  5. # sample configuration provided has an MSP ID of "SampleOrg".
  6. LocalMSPID: SampleOrg
  • LocalMSPID: (default value should be overridden) This identifies the organization this ordering node belongs to. The MSP ID must match the orderer organization MSP ID that exists in the configuration of any channel this joined will be joined to.

General.BCCSP.*

  1. # Default specifies the preferred blockchain crypto service provider
  2.         # to use. If the preferred provider is not available, the software
  3.         # based provider ("SW") will be used.
  4.         # Valid providers are:
  5.         #  - SW: a software based crypto provider
  6.         #  - PKCS11: a CA hardware security module crypto provider.
  7.         Default: SW
  9.         # SW configures the software based blockchain crypto provider.
  10.         SW:
  11.             # TODO: The default Hash and Security level needs refactoring to be
  12.             # fully configurable. Changing these defaults requires coordination
  13.             # SHA2 is hardcoded in several places, not only BCCSP
  14.             Hash: SHA2
  15.             Security: 256
  16.             # Location of key store. If this is unset, a location will be
  17.             # chosen using: 'LocalMSPDir'/keystore
  18.             FileKeyStore:
  19.                 KeyStore:

(Optional) This section is used to configure the Blockchain crypto provider.

  • BCCSP.Default: If you plan to use a Hardware Security Module (HSM), then this must be set to PKCS11.
  1. # Settings for the PKCS#11 crypto provider (i.e. when DEFAULT: PKCS11)
  2.         PKCS11:
  3.             # Location of the PKCS11 module library
  4.             Library:
  5.             # Token Label
  6.             Label:
  7.             # User PIN
  8.             Pin:
  9.             Hash:
  10.             Security:
  11.             FileKeyStore:
  12.                 KeyStore:
  • BCCSP.PKCS11.*: Provide this set of parameters according to your HSM configuration. Refer to this example of an HSM configuration for more information.

FileLedger.Location

  1. # Location: The directory to store the blocks in.
  2. Location: /var/hyperledger/production/orderer
  • Location: (default value should be overridden in the unlikely event where two ordering nodes are running on the same node) Every channel on which the node is a consenter will have its own subdirectory at this location. The user running the orderer needs to own and have write access to this directory. The best practice is to store this data in persistent storage. This prevents the ledger from being lost if your orderer containers are destroyed for some reason.

Operations.*

The operations service is used for monitoring the health of the ordering node and relies on mutual TLS to secure its communication. Therefore, you need to set operations.tls.clientAuthRequired to true. When this parameter is set to true, clients attempting to ascertain the health of the node are required to provide a valid certificate for authentication. If the client does not provide a certificate or the service cannot verify the client’s certificate, the request is rejected. This means that the clients will need to register with the ordering node’s TLS CA and provide their TLS signing certificate on the requests. See The Operations Service to learn more.

If you plan to use Prometheus metrics to monitor your ordering node, you must configure the operations service here.

In the unlikely case where two ordering nodes are running on the same node on your infrastructure, you need to modify the addresses for the second ordering node to use a different port. Otherwise, when you start the second ordering node, it will fail to start, reporting that the addresses are already in use.

  1. # host and port for the operations server
  2.     ListenAddress: 127.0.0.1:8443
  4.     # TLS configuration for the operations endpoint
  5.     TLS:
  6.         # TLS enabled
  7.         Enabled: false
  9.         # Certificate is the location of the PEM encoded TLS certificate
  10.         Certificate:
  12.         # PrivateKey points to the location of the PEM-encoded key
  13.         PrivateKey:
  15.         # Most operations service endpoints require client authentication when TLS
  16.         # is enabled. ClientAuthRequired requires client certificate authentication
  17.         # at the TLS layer to access all resources.
  18.         ClientAuthRequired: false
  20.         # Paths to PEM encoded ca certificates to trust for client authentication
  21.         ClientRootCAs: []

  • ListenAddress: (required when using the operations service) Specify the address and port of the operations server.

  • Enabled: (required when using the operations service) Must be true if the operations service is being used.

  • Certificate: (required when using the operations service) Can be the same file as the General.TLS.Certificate.

  • PrivateKey: (required when using the operations service) Can be the same file as the General.TLS.PrivateKey.

  • ClientAuthRequired: (required when using the operations service) Must be set to true to enable mutual TLS between the client and the server.

  • ClientRootCAs: (required when using the operations service) Similar to the client root CA cert file in TLS, it contains a list of client root CA certificates that can be used to verify client certificates. If the client enrolled with the orderer organization CA, then this value is the orderer organization root CA cert.

Metrics.*

By default this is disabled, but if you want to monitor the metrics for the orderer, you need to use StatsD or Prometheus as your metric provider. StatsD uses a “push” model, pushing metrics from the ordering node to a StatsD endpoint. Because of this, it does not require configuration of the operations service itself. Prometheus metrics, by contrast, are pulled from an ordering node.

For more information about the available Prometheus metrics, check out Prometheus

For more information about the available StatsD metrics, check out StatsD.

Because Prometheus utilizes a “pull” model there is not any configuration required, beyond making the operations service available. Rather, Prometheus will send requests to the operations URL to poll for available metrics.

  1.     # The metrics provider is one of statsd, prometheus, or disabled
  2.     Provider: disabled
  4.     # The statsd configuration
  5.     Statsd:
  6.       # network type: tcp or udp
  7.       Network: udp
  9.       # the statsd server address
  10.       Address: 127.0.0.1:8125
  12.       # The interval at which locally cached counters and gauges are pushed
  13.       # to statsd; timings are pushed immediately
  14.       WriteInterval: 30s
  16.       # The prefix is prepended to all emitted statsd metrics
  17.       Prefix:

  • Provider: Set this value to statsd if using StatsD or prometheus if using Prometheus.

  • Statsd.Address: (required to use StatsD metrics for the ordering node) When StatsD is enabled, you will need to configure the hostname and port of the StatsD server so that the ordering node can push metric updates.

Admin.*

  1. Admin:
  2.     # host and port for the admin server
  3.     ListenAddress: 127.0.0.1:9443
  5.     # TLS configuration for the admin endpoint
  6.     TLS:
  7.         # TLS enabled
  8.         Enabled: false
  10.         # Certificate is the location of the PEM encoded TLS certificate
  11.         Certificate:
  13.         # PrivateKey points to the location of the PEM-encoded key
  14.         PrivateKey:
  16.         # Most admin service endpoints require client authentication when TLS
  17.         # is enabled. ClientAuthRequired requires client certificate authentication
  18.         # at the TLS layer to access all resources.
  19.         #
  20.         # NOTE: When TLS is enabled, the admin endpoint requires mutual TLS. The
  21.         # orderer will panic on startup if this value is set to false.
  22.         ClientAuthRequired: true
  24.         # Paths to PEM encoded ca certificates to trust for client authentication
  25.         ClientRootCAs: []

  • ListenAddress: The orderer admin server address (host and port) that can be used by the osnadmin command to configure channels on the ordering service. This value should be a unique host:port combination to avoid conflicts.

  • TLS.Enabled: Technically this can be set to false, but this is not recommended. In general, you should always set this value to true.

  • TLS.Certificate: The path to and file name of the orderer signed certificate issued by the TLS CA.

  • TLS.PrivateKey: The path to and file name of the orderer private key issued by the TLS CA.

  • TLS.ClientAuthRequired: This value must be set to true. Note that while mutual TLS is required for all operations on the orderer Admin endpoint, the entire network is not required to use mutual TLS.

  • TLS.ClientRootCAs: The path to and file name of the admin client TLS CA root certificate.

ChannelParticipation.*

  1. ChannelParticipation:
  2.     # Channel participation API is enabled.
  3.     Enabled: false
  5.     # The maximum size of the request body when joining a channel.
  6.     MaxRequestBodySize: 1 MB

  • Enabled: If you are bootstrapping the ordering node with a system channel genesis block, this value can be set to either true or false (setting the value to true allows you to list channels and to migrate away from the system channel in the future). If you are not bootstrapping the ordering node with a system channel genesis block, this value must be set to true and the General.BoostrapMethod should be set to none.

  • MaxRequestBodySize: (default value should not be overridden) This value controls the maximum size a configuration block can be and be accepted by this ordering node. Most configuration blocks are smaller than 1 MB, but if for some reason a configuration block is too large to be accept, bring down the node, increase this value, and restart the node.

Consensus.*

The values of this section vary by consensus plugin. The values below are for the etcdraft consensus plugin. If you are using a different consensus plugin, refer to its documentation for allowed keys and recommended values.

  1. # The allowed key-value pairs here depend on consensus plugin. For etcd/raft,
  2. # we use following options:
  4. # WALDir specifies the location at which Write Ahead Logs for etcd/raft are
  5. # stored. Each channel will have its own subdir named after channel ID.
  6. WALDir: /var/hyperledger/production/orderer/etcdraft/wal
  8. # SnapDir specifies the location at which snapshots for etcd/raft are
  9. # stored. Each channel will have its own subdir named after channel ID.
  10. SnapDir: /var/hyperledger/production/orderer/etcdraft/snapshot

  • WALDir: (default value should be overridden) This is the path to the write ahead logs on the local filesystem of the ordering node. It can be an absolute path or relative to FABRIC_CFG_PATH. It defaults to /var/hyperledger/production/orderer/etcdraft/wal. Each channel will have its own subdirectory named after the channel ID. The user running the ordering node needs to own and have write access to this directory. The best practice is to store this data in persistent storage. This prevents the write ahead log from being lost if your orderer containers are destroyed for some reason.

  • SnapDir: (default value should be overridden) This is the path to the snapshots on the local filesystem of the ordering node. For more information about how snapshots work in a Raft ordering service, check out Snapshots/ It can be an absolute path or relative to FABRIC_CFG_PATH. It defaults to /var/hyperledger/production/orderer/etcdraft/wal. Each channel will have its own subdirectory named after the channel ID. The user running the ordering node needs to own and have write access to this directory. The best practice is to store this data in persistent storage. This prevents snapshots from being lost if your orderer containers are destroyed for some reason.

For more information about ordering node configuration, including how to set parameters that are not available in orderer.yaml, check out Configuring and operating a Raft ordering service.