Configuring Monitor/OSD Interaction

After you have completed your initial Ceph configuration, you may deploy and runCeph. When you execute a command such as ceph health or ceph -s, theCeph Monitor reports on the current state of the Ceph StorageCluster. The Ceph Monitor knows about the Ceph Storage Cluster by requiringreports from each Ceph OSD Daemon, and by receiving reports from CephOSD Daemons about the status of their neighboring Ceph OSD Daemons. If the CephMonitor doesn’t receive reports, or if it receives reports of changes in theCeph Storage Cluster, the Ceph Monitor updates the status of the CephCluster Map.

Ceph provides reasonable default settings for Ceph Monitor/Ceph OSD Daemoninteraction. However, you may override the defaults. The following sectionsdescribe how Ceph Monitors and Ceph OSD Daemons interact for the purposes ofmonitoring the Ceph Storage Cluster.

OSDs Check Heartbeats

Each Ceph OSD Daemon checks the heartbeat of other Ceph OSD Daemons at randomintervals less than every 6 seconds. If a neighboring Ceph OSD Daemon doesn’tshow a heartbeat within a 20 second grace period, the Ceph OSD Daemon mayconsider the neighboring Ceph OSD Daemon down and report it back to a CephMonitor, which will update the Ceph Cluster Map. You may change this graceperiod by adding an osd heartbeat grace setting under the [mon]and [osd] or [global] section of your Ceph configuration file,or by setting the value at runtime.

Heartbeat Settings - 图1

OSDs Report Down OSDs

By default, two Ceph OSD Daemons from different hosts must report to the CephMonitors that another Ceph OSD Daemon is down before the Ceph Monitorsacknowledge that the reported Ceph OSD Daemon is down. But there is chancethat all the OSDs reporting the failure are hosted in a rack with a bad switchwhich has trouble connecting to another OSD. To avoid this sort of false alarm,we consider the peers reporting a failure a proxy for a potential “subcluster”over the overall cluster that is similarly laggy. This is clearly not true inall cases, but will sometimes help us localize the grace correction to a subsetof the system that is unhappy. mon osd reporter subtree level is used togroup the peers into the “subcluster” by their common ancestor type in CRUSHmap. By default, only two reports from different subtree are required to reportanother Ceph OSD Daemon down. You can change the number of reporters fromunique subtrees and the common ancestor type required to report a Ceph OSDDaemon down to a Ceph Monitor by adding an mon osd min down reportersand mon osd reporter subtree level settings under the [mon] section ofyour Ceph configuration file, or by setting the value at runtime.

Heartbeat Settings - 图2

OSDs Report Peering Failure

If a Ceph OSD Daemon cannot peer with any of the Ceph OSD Daemons defined in itsCeph configuration file (or the cluster map), it will ping a Ceph Monitor forthe most recent copy of the cluster map every 30 seconds. You can change theCeph Monitor heartbeat interval by adding an osd mon heartbeat intervalsetting under the [osd] section of your Ceph configuration file, or bysetting the value at runtime.

Heartbeat Settings - 图3

OSDs Report Their Status

If an Ceph OSD Daemon doesn’t report to a Ceph Monitor, the Ceph Monitor willconsider the Ceph OSD Daemon down after the mon osd report timeoutelapses. A Ceph OSD Daemon sends a report to a Ceph Monitor when a reportableevent such as a failure, a change in placement group stats, a change inup_thru or when it boots within 5 seconds. You can change the Ceph OSDDaemon minimum report interval by adding an osd mon report intervalsetting under the [osd] section of your Ceph configuration file, or bysetting the value at runtime. A Ceph OSD Daemon sends a report to a CephMonitor every 120 seconds irrespective of whether any notable changes occur.You can change the Ceph Monitor report interval by adding an osd mon reportinterval max setting under the [osd] section of your Ceph configurationfile, or by setting the value at runtime.

Heartbeat Settings - 图4

Configuration Settings

When modifying heartbeat settings, you should include them in the [global]section of your configuration file.

Monitor Settings

mon osd min up ratio

  • Description

  • The minimum ratio of up Ceph OSD Daemons before Ceph willmark Ceph OSD Daemons down.

  • Type

  • Double

  • Default

  • .3

mon osd min in ratio

  • Description

  • The minimum ratio of in Ceph OSD Daemons before Ceph willmark Ceph OSD Daemons out.

  • Type

  • Double

  • Default

  • .75

mon osd laggy halflife

  • Description

  • The number of seconds laggy estimates will decay.

  • Type

  • Integer

  • Default

  • 60*60

mon osd laggy weight

  • Description

  • The weight for new samples in laggy estimation decay.

  • Type

  • Double

  • Default

  • 0.3

mon osd laggy max interval

  • Description

  • Maximum value of laggy_interval in laggy estimations (in seconds).Monitor uses an adaptive approach to evaluate the laggy_interval ofa certain OSD. This value will be used to calculate the grace time forthat OSD.

  • Type

  • Integer

  • Default

  • 300

mon osd adjust heartbeat grace

  • Description

  • If set to true, Ceph will scale based on laggy estimations.

  • Type

  • Boolean

  • Default

  • true

mon osd adjust down out interval

  • Description

  • If set to true, Ceph will scaled based on laggy estimations.

  • Type

  • Boolean

  • Default

  • true

mon osd auto mark in

  • Description

  • Ceph will mark any booting Ceph OSD Daemons as inthe Ceph Storage Cluster.

  • Type

  • Boolean

  • Default

  • false

mon osd auto mark auto out in

  • Description

  • Ceph will mark booting Ceph OSD Daemons auto marked outof the Ceph Storage Cluster as in the cluster.

  • Type

  • Boolean

  • Default

  • true

mon osd auto mark new in

  • Description

  • Ceph will mark booting new Ceph OSD Daemons as in theCeph Storage Cluster.

  • Type

  • Boolean

  • Default

  • true

mon osd down out interval

  • Description

  • The number of seconds Ceph waits before marking a Ceph OSD Daemondown and out if it doesn’t respond.

  • Type

  • 32-bit Integer

  • Default

  • 600

mon osd down out subtree limit

  • Description

  • The smallest CRUSH unit type that Ceph will notautomatically mark out. For instance, if set to host and ifall OSDs of a host are down, Ceph will not automatically mark outthese OSDs.

  • Type

  • String

  • Default

  • rack

mon osd report timeout

  • Description

  • The grace period in seconds before declaringunresponsive Ceph OSD Daemons down.

  • Type

  • 32-bit Integer

  • Default

  • 900

mon osd min down reporters

  • Description

  • The minimum number of Ceph OSD Daemons required to report adown Ceph OSD Daemon.

  • Type

  • 32-bit Integer

  • Default

  • 2

mon osd reporter subtree level

  • Description

  • In which level of parent bucket the reporters are counted. The OSDssend failure reports to monitor if they find its peer is not responsive.And monitor mark the reported OSD out and then down after a grace period.

  • Type

  • String

  • Default

  • host

OSD Settings

osd heartbeat address

  • Description

  • An Ceph OSD Daemon’s network address for heartbeats.

  • Type

  • Address

  • Default

  • The host address.

osd heartbeat interval

  • Description

  • How often an Ceph OSD Daemon pings its peers (in seconds).

  • Type

  • 32-bit Integer

  • Default

  • 6

osd heartbeat grace

  • Description

  • The elapsed time when a Ceph OSD Daemon hasn’t shown a heartbeatthat the Ceph Storage Cluster considers it down.This setting has to be set in both the [mon] and [osd] or [global]section so that it is read by both the MON and OSD daemons.

  • Type

  • 32-bit Integer

  • Default

  • 20

osd mon heartbeat interval

  • Description

  • How often the Ceph OSD Daemon pings a Ceph Monitor if it has noCeph OSD Daemon peers.

  • Type

  • 32-bit Integer

  • Default

  • 30

osd mon heartbeat stat stale

  • Description

  • Stop reporting on heartbeat ping times which haven’t been updated forthis many seconds. Set to zero to disable this action.

  • Type

  • 32-bit Integer

  • Default

  • 3600

osd mon report interval

  • Description

  • The number of seconds a Ceph OSD Daemon may waitfrom startup or another reportable event before reportingto a Ceph Monitor.

  • Type

  • 32-bit Integer

  • Default

  • 5

osd mon ack timeout

  • Description

  • The number of seconds to wait for a Ceph Monitor to acknowledge arequest for statistics.

  • Type

  • 32-bit Integer

  • Default

  • 30