Health checks

Overview

There is a finite set of possible health messages that a Ceph cluster canraise – these are defined as health checks which have unique identifiers.

The identifier is a terse pseudo-human-readable (i.e. like a variable name)string. It is intended to enable tools (such as UIs) to make sense ofhealth checks, and present them in a way that reflects their meaning.

This page lists the health checks that are raised by the monitor and managerdaemons. In addition to these, you may also see health checks that originatefrom MDS daemons (see CephFS health messages), and health checksthat are defined by ceph-mgr python modules.

Definitions

Monitor

MON_DOWN

One or more monitor daemons is currently down. The cluster requires amajority (more than 1/2) of the monitors in order to function. Whenone or more monitors are down, clients may have a harder time formingtheir initial connection to the cluster as they may need to try moreaddresses before they reach an operating monitor.

The down monitor daemon should generally be restarted as soon aspossible to reduce the risk of a subsequen monitor failure leading toa service outage.

MON_CLOCK_SKEW

The clocks on the hosts running the ceph-mon monitor daemons are notsufficiently well synchronized. This health alert is raised if thecluster detects a clock skew greater than mon_clock_drift_allowed.

This is best resolved by synchronizing the clocks using a tool likentpd or chrony.

If it is impractical to keep the clocks closely synchronized, themon_clock_drift_allowed threshold can also be increased, but thisvalue must stay significantly below the mon_lease interval inorder for monitor cluster to function properly.

MON_MSGR2_NOT_ENABLED

The ms_bind_msgr2 option is enabled but one or more monitors isnot configured to bind to a v2 port in the cluster’s monmap. Thismeans that features specific to the msgr2 protocol (e.g., encryption)are not available on some or all connections.

In most cases this can be corrected by issuing the command:

ceph mon enable-msgr2

That command will change any monitor configured for the old defaultport 6789 to continue to listen for v1 connections on 6789 and alsolisten for v2 connections on the new default 3300 port.

If a monitor is configured to listen for v1 connections on a non-standard port (not 6789), then the monmap will need to be modified manually.

MON_DISK_LOW

One or more monitors is low on disk space. This alert triggers if theavailable space on the file system storing the monitor database(normally /var/lib/ceph/mon), as a percentage, drops belowmon_data_avail_warn (default: 30%).

This may indicate that some other process or user on the system isfilling up the same file system used by the monitor. It may alsoindicate that the monitors database is large (see MON_DISK_BIGbelow).

If space cannot be freed, the monitor’s data directory may need to bemoved to another storage device or file system (while the monitordaemon is not running, of course).

MON_DISK_CRIT

One or more monitors is critically low on disk space. This alerttriggers if the available space on the file system storing the monitordatabase (normally /var/lib/ceph/mon), as a percentage, dropsbelow mon_data_avail_crit (default: 5%). See MON_DISK_LOW, above.

MON_DISK_BIG

The database size for one or more monitors is very large. This alerttriggers if the size of the monitor’s database is larger thanmon_data_size_warn (default: 15 GiB).

A large database is unusual, but may not necessarily indicate aproblem. Monitor databases may grow in size when there are placementgroups that have not reached an active+clean state in a long time.

This may also indicate that the monitor’s database is not properlycompacting, which has been observed with some older versions ofleveldb and rocksdb. Forcing a compaction with ceph daemon mon.<id>compact may shrink the on-disk size.

This warning may also indicate that the monitor has a bug that ispreventing it from pruning the cluster metadata it stores. If theproblem persists, please report a bug.

The warning threshold may be adjusted with:

ceph config set global mon_data_size_warn <size>

Manager

MGR_DOWN

All manager daemons are currently down. The cluster should normallyhave at least one running manager (ceph-mgr) daemon. If nomanager daemon is running, the cluster’s ability to monitor itself willbe compromised, and parts of the management API will becomeunavailable (for example, the dashboard will not work, and most CLIcommands that report metrics or runtime state will block). However,the cluster will still be able to perform all IO operations andrecover from failures.

The down manager daemon should generally be restarted as soon aspossible to ensure that the cluster can be monitored (e.g., so thatthe ceph -s information is up to date, and/or metrics can bescraped by Prometheus).

MGR_MODULE_DEPENDENCY

An enabled manager module is failing its dependency check. This health checkshould come with an explanatory message from the module about the problem.

For example, a module might report that a required package is not installed:install the required package and restart your manager daemons.

This health check is only applied to enabled modules. If a module isnot enabled, you can see whether it is reporting dependency issues inthe output of ceph module ls.

MGR_MODULE_ERROR

A manager module has experienced an unexpected error. Typically,this means an unhandled exception was raised from the module’s _serve_function. The human readable description of the error may be obscurelyworded if the exception did not provide a useful description of itself.

This health check may indicate a bug: please open a Ceph bug report if youthink you have encountered a bug.

If you believe the error is transient, you may restart your managerdaemon(s), or use ceph mgr fail on the active daemon to prompta failover to another daemon.

OSDs

OSD_DOWN

One or more OSDs are marked down. The ceph-osd daemon may have beenstopped, or peer OSDs may be unable to reach the OSD over the network.Common causes include a stopped or crashed daemon, a down host, or anetwork outage.

Verify the host is healthy, the daemon is started, and network isfunctioning. If the daemon has crashed, the daemon log file(/var/log/ceph/ceph-osd.*) may contain debugging information.

OSD_<crush type>_DOWN

(e.g. OSD_HOST_DOWN, OSD_ROOT_DOWN)

All the OSDs within a particular CRUSH subtree are marked down, for exampleall OSDs on a host.

OSD_ORPHAN

An OSD is referenced in the CRUSH map hierarchy but does not exist.

The OSD can be removed from the CRUSH hierarchy with:

ceph osd crush rm osd.<id>

OSD_OUT_OF_ORDER_FULL

The utilization thresholds for nearfull, backfillfull, full,and/or failsafe_full are not ascending. In particular, we expectnearfull < backfillfull, backfillfull < full, and full <failsafe_full.

The thresholds can be adjusted with:

ceph osd set-nearfull-ratio <ratio>
ceph osd set-backfillfull-ratio <ratio>
ceph osd set-full-ratio <ratio>

OSD_FULL

One or more OSDs has exceeded the full threshold and is preventingthe cluster from servicing writes.

Utilization by pool can be checked with:

ceph df

The currently defined full ratio can be seen with:

ceph osd dump | grep full_ratio

A short-term workaround to restore write availability is to raise the fullthreshold by a small amount:

ceph osd set-full-ratio <ratio>

New storage should be added to the cluster by deploying more OSDs orexisting data should be deleted in order to free up space.

OSD_BACKFILLFULL

One or more OSDs has exceeded the backfillfull threshold, which willprevent data from being allowed to rebalance to this device. This isan early warning that rebalancing may not be able to complete and thatthe cluster is approaching full.

Utilization by pool can be checked with:

ceph df

OSD_NEARFULL

One or more OSDs has exceeded the nearfull threshold. This is an earlywarning that the cluster is approaching full.

Utilization by pool can be checked with:

ceph df

OSDMAP_FLAGS

One or more cluster flags of interest has been set. These flags include:

• full - the cluster is flagged as full and cannot serve writes

• pauserd, pausewr - paused reads or writes

• noup - OSDs are not allowed to start

• nodown - OSD failure reports are being ignored, such that themonitors will not mark OSDs down

• noin - OSDs that were previously marked out will not be markedback in when they start

• noout - down OSDs will not automatically be marked out after theconfigured interval

• nobackfill, norecover, norebalance - recovery or datarebalancing is suspended

• noscrub, nodeep_scrub - scrubbing is disabled

• notieragent - cache tiering activity is suspended

With the exception of full, these flags can be set or cleared with:

ceph osd set <flag>
ceph osd unset <flag>

OSD_FLAGS

One or more OSDs or CRUSH {nodes,device classes} has a flag of interest set.These flags include:

• noup: these OSDs are not allowed to start

• nodown: failure reports for these OSDs will be ignored

• noin: if these OSDs were previously marked out automaticallyafter a failure, they will not be marked in when they start

• noout: if these OSDs are down they will not automatically be markedout after the configured interval

These flags can be set and cleared in batch with:

ceph osd set-group <flags> <who>
ceph osd unset-group <flags> <who>

For example,

ceph osd set-group noup,noout osd.0 osd.1
ceph osd unset-group noup,noout osd.0 osd.1
ceph osd set-group noup,noout host-foo
ceph osd unset-group noup,noout host-foo
ceph osd set-group noup,noout class-hdd
ceph osd unset-group noup,noout class-hdd

OLD_CRUSH_TUNABLES

The CRUSH map is using very old settings and should be updated. Theoldest tunables that can be used (i.e., the oldest client version thatcan connect to the cluster) without triggering this health warning isdetermined by the mon_crush_min_required_version config option.See Tunables for more information.

OLD_CRUSH_STRAW_CALC_VERSION

The CRUSH map is using an older, non-optimal method for calculatingintermediate weight values for straw buckets.

The CRUSH map should be updated to use the newer method(straw_calc_version=1). SeeTunables for more information.

CACHE_POOL_NO_HIT_SET

One or more cache pools is not configured with a hit set to trackutilization, which will prevent the tiering agent from identifyingcold objects to flush and evict from the cache.

Hit sets can be configured on the cache pool with:

ceph osd pool set <poolname> hit_set_type <type>
ceph osd pool set <poolname> hit_set_period <period-in-seconds>
ceph osd pool set <poolname> hit_set_count <number-of-hitsets>
ceph osd pool set <poolname> hit_set_fpp <target-false-positive-rate>

OSD_NO_SORTBITWISE

No pre-luminous v12.y.z OSDs are running but the sortbitwise flag has notbeen set.

The sortbitwise flag must be set before luminous v12.y.z or newerOSDs can start. You can safely set the flag with:

ceph osd set sortbitwise

POOL_FULL

One or more pools has reached its quota and is no longer allowing writes.

Pool quotas and utilization can be seen with:

ceph df detail

You can either raise the pool quota with:

ceph osd pool set-quota <poolname> max_objects <num-objects>
ceph osd pool set-quota <poolname> max_bytes <num-bytes>

or delete some existing data to reduce utilization.

BLUEFS_SPILLOVER

One or more OSDs that use the BlueStore backend have been allocateddb partitions (storage space for metadata, normally on a fasterdevice) but that space has filled, such that metadata has “spilledover” onto the normal slow device. This isn’t necessarily an errorcondition or even unexpected, but if the administrator’s expectationwas that all metadata would fit on the faster device, it indicatesthat not enough space was provided.

This warning can be disabled on all OSDs with:

ceph config set osd bluestore_warn_on_bluefs_spillover false

Alternatively, it can be disabled on a specific OSD with:

ceph config set osd.123 bluestore_warn_on_bluefs_spillover false

To provide more metadata space, the OSD in question could be destroyed andreprovisioned. This will involve data migration and recovery.

It may also be possible to expand the LVM logical volume backing thedb storage. If the underlying LV has been expanded, the OSD daemonneeds to be stopped and BlueFS informed of the device size change with:

ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-$ID

BLUEFS_AVAILABLE_SPACE

To check how much space is free for BlueFS do:

ceph daemon osd.123 bluestore bluefs available

This will output up to 3 values: BDEV_DB free, BDEV_SLOW free andavailable_from_bluestore. BDEV_DB and BDEV_SLOW report amount of space thathas been acquired by BlueFS and is considered free. Value _available_from_bluestore_denotes ability of BlueStore to relinquish more space to BlueFS.It is normal that this value is different from amount of BlueStore free space, asBlueFS allocation unit is typically larger than BlueStore allocation unit.This means that only part of BlueStore free space will be acceptable for BlueFS.

BLUEFS_LOW_SPACE

If BlueFS is running low on available free space and there is littleavailable_from_bluestore one can consider reducing BlueFS allocation unit size.To simulate available space when allocation unit is different do:

ceph daemon osd.123 bluestore bluefs available <alloc-unit-size>

BLUESTORE_FRAGMENTATION

As BlueStore works free space on underlying storage will get fragmented.This is normal and unavoidable but excessive fragmentation will cause slowdown.To inspect BlueStore fragmentation one can do:

ceph daemon osd.123 bluestore allocator score block

Score is given in [0-1] range.[0.0 .. 0.4] tiny fragmentation[0.4 .. 0.7] small, acceptable fragmentation[0.7 .. 0.9] considerable, but safe fragmentation[0.9 .. 1.0] severe fragmentation, may impact BlueFS ability to get space from BlueStore

If detailed report of free fragments is required do:

ceph daemon osd.123 bluestore allocator dump block

In case when handling OSD process that is not running fragmentation can beinspected with ceph-bluestore-tool.Get fragmentation score:

ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-123 --allocator block free-score

And dump detailed free chunks:

ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-123 --allocator block free-dump

BLUESTORE_LEGACY_STATFS

In the Nautilus release, BlueStore tracks its internal usagestatistics on a per-pool granular basis, and one or more OSDs haveBlueStore volumes that were created prior to Nautilus. If all OSDsare older than Nautilus, this just means that the per-pool metrics arenot available. However, if there is a mix of pre-Nautilus andpost-Nautilus OSDs, the cluster usage statistics reported by cephdf will not be accurate.

The old OSDs can be updated to use the new usage tracking scheme by stopping each OSD, running a repair operation, and the restarting it. For example, if osd.123 needed to be updated,:

systemctl stop ceph-osd@123
ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-123
systemctl start ceph-osd@123

This warning can be disabled with:

ceph config set global bluestore_warn_on_legacy_statfs false

BLUESTORE_NO_PER_POOL_OMAP

Starting with the Octopus release, BlueStore tracks omap space utilizationby pool, and one or more OSDs have volumes that were created prior toOctopus. If all OSDs are not running BlueStore with the new trackingenabled, the cluster will report and approximate value for per-pool omap usagebased on the most recent deep-scrub.

The old OSDs can be updated to track by pool by stopping each OSD,running a repair operation, and the restarting it. For example, ifosd.123 needed to be updated,:

systemctl stop ceph-osd@123
ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-123
systemctl start ceph-osd@123

This warning can be disabled with:

ceph config set global bluestore_warn_on_no_per_pool_omap false

BLUESTORE_DISK_SIZE_MISMATCH

One or more OSDs using BlueStore has an internal inconsistency between the sizeof the physical device and the metadata tracking its size. This can lead tothe OSD crashing in the future.

The OSDs in question should be destroyed and reprovisioned. Care should betaken to do this one OSD at a time, and in a way that doesn’t put any data atrisk. For example, if osd $N has the error,:

ceph osd out osd.$N
while ! ceph osd safe-to-destroy osd.$N ; do sleep 1m ; done
ceph osd destroy osd.$N
ceph-volume lvm zap /path/to/device
ceph-volume lvm create --osd-id $N --data /path/to/device

BLUESTORE_NO_COMPRESSION

One or more OSDs is unable to load a BlueStore compression plugin.This can be caused by a broken installation, in which the ceph-osdbinary does not match the compression plugins, or a recent upgradethat did not include a restart of the ceph-osd daemon.

Verify that the package(s) on the host running the OSD(s) in questionare correctly installed and that the OSD daemon(s) have beenrestarted. If the problem persists, check the OSD log for any cluesas to the source of the problem.

Device health

DEVICE_HEALTH

One or more devices is expected to fail soon, where the warningthreshold is controlled by the mgr/devicehealth/warn_thresholdconfig option.

This warning only applies to OSDs that are currently marked “in”, sothe expected response to this failure is to mark the device “out” sothat data is migrated off of the device, and then to remove thehardware from the system. Note that the marking out is normally doneautomatically if mgr/devicehealth/self_heal is enabled based onthe mgr/devicehealth/mark_out_threshold.

Device health can be checked with:

ceph device info <device-id>

Device life expectancy is set by a prediction model run bythe mgr or an by external tool via the command:

ceph device set-life-expectancy <device-id> <from> <to>

You can change the stored life expectancy manually, but that usuallydoesn’t accomplish anything as whatever tool originally set it willprobably set it again, and changing the stored value does not affectthe actual health of the hardware device.

DEVICE_HEALTH_IN_USE

One or more devices is expected to fail soon and has been marked “out”of the cluster based on mgr/devicehealth/mark_out_threshold, but itis still participating in one more PGs. This may be because it wasonly recently marked “out” and data is still migrating, or because datacannot be migrated off for some reason (e.g., the cluster is nearlyfull, or the CRUSH hierarchy is such that there isn’t another suitableOSD to migrate the data too).

This message can be silenced by disabling the self heal behavior(setting mgr/devicehealth/self_heal to false), by adjusting themgr/devicehealth/mark_out_threshold, or by addressing what ispreventing data from being migrated off of the ailing device.

DEVICE_HEALTH_TOOMANY

Too many devices is expected to fail soon and themgr/devicehealth/self_heal behavior is enabled, such that markingout all of the ailing devices would exceed the clustersmon_osd_min_in_ratio ratio that prevents too many OSDs from beingautomatically marked “out”.

This generally indicates that too many devices in your cluster areexpected to fail soon and you should take action to add newer(healthier) devices before too many devices fail and data is lost.

The health message can also be silenced by adjusting parameters likemon_osd_min_in_ratio or mgr/devicehealth/mark_out_threshold,but be warned that this will increase the likelihood of unrecoverabledata loss in the cluster.

Data health (pools & placement groups)

PG_AVAILABILITY

Data availability is reduced, meaning that the cluster is unable toservice potential read or write requests for some data in the cluster.Specifically, one or more PGs is in a state that does not allow IOrequests to be serviced. Problematic PG states include peering,stale, incomplete, and the lack of active (if those conditions do not clearquickly).

Detailed information about which PGs are affected is available from:

ceph health detail

In most cases the root cause is that one or more OSDs is currentlydown; see the discussion for OSD_DOWN above.

The state of specific problematic PGs can be queried with:

ceph tell <pgid> query

PG_DEGRADED

Data redundancy is reduced for some data, meaning the cluster does nothave the desired number of replicas for all data (for replicatedpools) or erasure code fragments (for erasure coded pools).Specifically, one or more PGs:

• has the degraded or undersized flag set, meaning there are notenough instances of that placement group in the cluster;

• has not had the clean flag set for some time.

Detailed information about which PGs are affected is available from:

ceph health detail

In most cases the root cause is that one or more OSDs is currentlydown; see the dicussion for OSD_DOWN above.

The state of specific problematic PGs can be queried with:

ceph tell <pgid> query

PG_RECOVERY_FULL

Data redundancy may be reduced or at risk for some data due to a lackof free space in the cluster. Specifically, one or more PGs has therecovery_toofull flag set, meaning that thecluster is unable to migrate or recover data because one or more OSDsis above the full threshold.

See the discussion for OSD_FULL above for steps to resolve this condition.

PG_BACKFILL_FULL

Data redundancy may be reduced or at risk for some data due to a lackof free space in the cluster. Specifically, one or more PGs has thebackfill_toofull flag set, meaning that thecluster is unable to migrate or recover data because one or more OSDsis above the backfillfull threshold.

See the discussion for OSD_BACKFILLFULL above forsteps to resolve this condition.

PG_DAMAGED

Data scrubbing has discovered some problems with data consistency inthe cluster. Specifically, one or more PGs has the inconsistent orsnaptrim_error flag is set, indicating an earlier scrub operationfound a problem, or that the repair flag is set, meaning a repairfor such an inconsistency is currently in progress.

See Repairing PG inconsistencies for more information.

OSD_SCRUB_ERRORS

Recent OSD scrubs have uncovered inconsistencies. This error is generallypaired with PG_DAMAGED (see above).

See Repairing PG inconsistencies for more information.

LARGE_OMAP_OBJECTS

One or more pools contain large omap objects as determined byosd_deep_scrub_large_omap_object_key_threshold (threshold for number of keysto determine a large omap object) orosd_deep_scrub_large_omap_object_value_sum_threshold (the threshold forsummed size (bytes) of all key values to determine a large omap object) or both.More information on the object name, key count, and size in bytes can be foundby searching the cluster log for ‘Large omap object found’. Large omap objectscan be caused by RGW bucket index objects that do not have automatic reshardingenabled. Please see RGW Dynamic Bucket Index Resharding for more information on resharding.

The thresholds can be adjusted with:

ceph config set osd osd_deep_scrub_large_omap_object_key_threshold <keys>
ceph config set osd osd_deep_scrub_large_omap_object_value_sum_threshold <bytes>

CACHE_POOL_NEAR_FULL

A cache tier pool is nearly full. Full in this context is determinedby the target_max_bytes and target_max_objects properties onthe cache pool. Once the pool reaches the target threshold, writerequests to the pool may block while data is flushed and evictedfrom the cache, a state that normally leads to very high latencies andpoor performance.

The cache pool target size can be adjusted with:

ceph osd pool set <cache-pool-name> target_max_bytes <bytes>
ceph osd pool set <cache-pool-name> target_max_objects <objects>

Normal cache flush and evict activity may also be throttled due to reducedavailability or performance of the base tier, or overall cluster load.

TOO_FEW_PGS

The number of PGs in use in the cluster is below the configurablethreshold of mon_pg_warn_min_per_osd PGs per OSD. This can leadto suboptimal distribution and balance of data across the OSDs inthe cluster, and similarly reduce overall performance.

This may be an expected condition if data pools have not yet beencreated.

The PG count for existing pools can be increased or new pools can be created.Please refer to Choosing the number of Placement Groups for moreinformation.

POOL_PG_NUM_NOT_POWER_OF_TWO

One or more pools has a pg_num value that is not a power of two.Although this is not strictly incorrect, it does lead to a lessbalanced distribution of data because some PGs have roughly twice asmuch data as others.

This is easily corrected by setting the pg_num value for theaffected pool(s) to a nearby power of two:

ceph osd pool set <pool-name> pg_num <value>

This health warning can be disabled with:

ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false

POOL_TOO_FEW_PGS

One or more pools should probably have more PGs, based on the amountof data that is currently stored in the pool. This can lead tosuboptimal distribution and balance of data across the OSDs in thecluster, and similarly reduce overall performance. This warning isgenerated if the pg_autoscale_mode property on the pool is set towarn.

To disable the warning, you can disable auto-scaling of PGs for thepool entirely with:

ceph osd pool set <pool-name> pg_autoscale_mode off

To allow the cluster to automatically adjust the number of PGs,:

ceph osd pool set <pool-name> pg_autoscale_mode on

You can also manually set the number of PGs for the pool to therecommended amount with:

ceph osd pool set <pool-name> pg_num <new-pg-num>

Please refer to Choosing the number of Placement Groups andAutoscaling placement groups for more information.

TOO_MANY_PGS

The number of PGs in use in the cluster is above the configurablethreshold of monmax_pg_per_osd PGs per OSD. If this threshold isexceed the cluster will not allow new pools to be created, pool _pg_num tobe increased, or pool replication to be increased (any of which would lead tomore PGs in the cluster). A large number of PGs can leadto higher memory utilization for OSD daemons, slower peering aftercluster state changes (like OSD restarts, additions, or removals), andhigher load on the Manager and Monitor daemons.

The simplest way to mitigate the problem is to increase the number ofOSDs in the cluster by adding more hardware. Note that the OSD countused for the purposes of this health check is the number of “in” OSDs,so marking “out” OSDs “in” (if there are any) can also help:

ceph osd in <osd id(s)>

Please refer to Choosing the number of Placement Groups for moreinformation.

POOL_TOO_MANY_PGS

One or more pools should probably have more PGs, based on the amountof data that is currently stored in the pool. This can lead to highermemory utilization for OSD daemons, slower peering after cluster statechanges (like OSD restarts, additions, or removals), and higher loadon the Manager and Monitor daemons. This warning is generated if thepg_autoscale_mode property on the pool is set to warn.

To disable the warning, you can disable auto-scaling of PGs for thepool entirely with:

ceph osd pool set <pool-name> pg_autoscale_mode off

To allow the cluster to automatically adjust the number of PGs,:

ceph osd pool set <pool-name> pg_autoscale_mode on

You can also manually set the number of PGs for the pool to therecommended amount with:

ceph osd pool set <pool-name> pg_num <new-pg-num>

Please refer to Choosing the number of Placement Groups andAutoscaling placement groups for more information.

POOL_TARGET_SIZE_RATIO_OVERCOMMITTED

One or more pools have a target_size_ratio property set toestimate the expected size of the pool as a fraction of total storage,but the value(s) exceed the total available storage (either bythemselves or in combination with other pools’ actual usage).

This is usually an indication that the target_size_ratio value forthe pool is too large and should be reduced or set to zero with:

ceph osd pool set <pool-name> target_size_ratio 0

For more information, see Specifying expected pool size.

POOL_TARGET_SIZE_BYTES_OVERCOMMITTED

One or more pools have a target_size_bytes property set toestimate the expected size of the pool,but the value(s) exceed the total available storage (either bythemselves or in combination with other pools’ actual usage).

This is usually an indication that the target_size_bytes value forthe pool is too large and should be reduced or set to zero with:

ceph osd pool set <pool-name> target_size_bytes 0

For more information, see Specifying expected pool size.

TOO_FEW_OSDS

The number of OSDs in the cluster is below the configurablethreshold of osd_pool_default_size.

SMALLER_PGP_NUM

One or more pools has a pgp_num value less than pg_num. Thisis normally an indication that the PG count was increased withoutalso increasing the placement behavior.

This is sometimes done deliberately to separate out the split stepwhen the PG count is adjusted from the data migration that is neededwhen pgp_num is changed.

This is normally resolved by setting pgp_num to match pg_num,triggering the data migration, with:

ceph osd pool set <pool> pgp_num <pg-num-value>

MANY_OBJECTS_PER_PG

One or more pools has an average number of objects per PG that issignificantly higher than the overall cluster average. The specificthreshold is controlled by the mon_pg_warn_max_object_skewconfiguration value.

This is usually an indication that the pool(s) containing most of thedata in the cluster have too few PGs, and/or that other pools that donot contain as much data have too many PGs. See the discussion ofTOO_MANY_PGS above.

The threshold can be raised to silence the health warning by adjustingthe mon_pg_warn_max_object_skew config option on the managers.

POOL_APP_NOT_ENABLED

A pool exists that contains one or more objects but has not beentagged for use by a particular application.

Resolve this warning by labeling the pool for use by an application. Forexample, if the pool is used by RBD,:

rbd pool init <poolname>

If the pool is being used by a custom application ‘foo’, you can also labelvia the low-level command:

ceph osd pool application enable foo

For more information, see Associate Pool to Application.

POOL_FULL

One or more pools has reached (or is very close to reaching) itsquota. The threshold to trigger this error condition is controlled bythe mon_pool_quota_crit_threshold configuration option.

Pool quotas can be adjusted up or down (or removed) with:

ceph osd pool set-quota <pool> max_bytes <bytes>
ceph osd pool set-quota <pool> max_objects <objects>

Setting the quota value to 0 will disable the quota.

POOL_NEAR_FULL

One or more pools is approaching is quota. The threshold to triggerthis warning condition is controlled by themon_pool_quota_warn_threshold configuration option.

Pool quotas can be adjusted up or down (or removed) with:

ceph osd pool set-quota <pool> max_bytes <bytes>
ceph osd pool set-quota <pool> max_objects <objects>

Setting the quota value to 0 will disable the quota.

OBJECT_MISPLACED

One or more objects in the cluster is not stored on the node thecluster would like it to be stored on. This is an indication thatdata migration due to some recent cluster change has not yet completed.

Misplaced data is not a dangerous condition in and of itself; dataconsistency is never at risk, and old copies of objects are neverremoved until the desired number of new copies (in the desiredlocations) are present.

OBJECT_UNFOUND

One or more objects in the cluster cannot be found. Specifically, theOSDs know that a new or updated copy of an object should exist, but acopy of that version of the object has not been found on OSDs that arecurrently online.

Read or write requests to unfound objects will block.

Ideally, a down OSD can be brought back online that has the morerecent copy of the unfound object. Candidate OSDs can be identified from thepeering state for the PG(s) responsible for the unfound object:

ceph tell <pgid> query

If the latest copy of the object is not available, the cluster can betold to roll back to a previous version of the object. SeeUnfound Objects for more information.

SLOW_OPS

One or more OSD requests is taking a long time to process. This canbe an indication of extreme load, a slow storage device, or a softwarebug.

The request queue on the OSD(s) in question can be queried with thefollowing command, executed from the OSD host:

ceph daemon osd.<id> ops

A summary of the slowest recent requests can be seen with:

ceph daemon osd.<id> dump_historic_ops

The location of an OSD can be found with:

ceph osd find osd.<id>

PG_NOT_SCRUBBED

One or more PGs has not been scrubbed recently. PGs are normallyscrubbed every mon_scrub_interval seconds, and this warningtriggers when mon_warn_pg_not_scrubbed_ratio percentage of interval has elapsedwithout a scrub since it was due.

PGs will not scrub if they are not flagged as clean, which mayhappen if they are misplaced or degraded (see PG_AVAILABILITY andPG_DEGRADED above).

You can manually initiate a scrub of a clean PG with:

ceph pg scrub <pgid>

PG_NOT_DEEP_SCRUBBED

One or more PGs has not been deep scrubbed recently. PGs are normallyscrubbed every osd_deep_scrub_interval seconds, and this warningtriggers when mon_warn_pg_not_deep_scrubbed_ratio percentage of interval has elapsedwithout a scrub since it was due.

PGs will not (deep) scrub if they are not flagged as clean, which mayhappen if they are misplaced or degraded (see PG_AVAILABILITY andPG_DEGRADED above).

You can manually initiate a scrub of a clean PG with:

ceph pg deep-scrub <pgid>

PG_SLOW_SNAP_TRIMMING

The snapshot trim queue for one or more PGs has exceeded theconfigured warning threshold. This indicates that either an extremelylarge number of snapshots were recently deleted, or that the OSDs areunable to trim snapshots quickly enough to keep up with the rate ofnew snapshot deletions.

The warning threshold is controlled by themon_osd_snap_trim_queue_warn_on option (default: 32768).

This warning may trigger if OSDs are under excessive load and unableto keep up with their background work, or if the OSDs’ internalmetadata database is heavily fragmented and unable to perform. It mayalso indicate some other performance issue with the OSDs.

The exact size of the snapshot trim queue is reported by thesnaptrimq_len field of ceph pg ls -f json-detail.

Miscellaneous

RECENT_CRASH

One or more Ceph daemons has crashed recently, and the crash has notyet been archived (acknowledged) by the administrator. This mayindicate a software bug, a hardware problem (e.g., a failing disk), orsome other problem.

New crashes can be listed with:

ceph crash ls-new

Information about a specific crash can be examined with:

ceph crash info <crash-id>

This warning can be silenced by “archiving” the crash (perhaps afterbeing examined by an administrator) so that it does not generate thiswarning:

ceph crash archive <crash-id>

Similarly, all new crashes can be archived with:

ceph crash archive-all

Archived crashes will still be visible via ceph crash ls but notceph crash ls-new.

The time period for what “recent” means is controlled by the optionmgr/crash/warn_recent_interval (default: two weeks).

These warnings can be disabled entirely with:

ceph config set mgr/crash/warn_recent_interval 0

TELEMETRY_CHANGED

Telemetry has been enabled, but the contents of the telemetry reporthave changed since that time, so telemetry reports will not be sent.

The Ceph developers periodically revise the telemetry feature toinclude new and useful information, or to remove information found tobe useless or sensitive. If any new information is included in thereport, Ceph will require the administrator to re-enable telemetry toensure they have an opportunity to (re)review what information will beshared.

To review the contents of the telemetry report,:

ceph telemetry show

Note that the telemetry report consists of several optional channelsthat may be independently enabled or disabled. For more information, seeTelemetry Module.

To re-enable telemetry (and make this warning go away),:

ceph telemetry on

To disable telemetry (and make this warning go away),:

ceph telemetry off

AUTH_BAD_CAPS

One or more auth users has capabilities that cannot be parsed by themonitor. This generally indicates that the user will not beauthorized to perform any action with one or more daemon types.

This error is mostly likely to occur after an upgrade if thecapabilities were set with an older version of Ceph that did notproperly validate their syntax, or if the syntax of the capabilitieshas changed.

The user in question can be removed with:

ceph auth rm <entity-name>

(This will resolve the health alert, but obviously clients will not beable to authenticate as that user.)

Alternatively, the capabilities for the user can be updated with:

ceph auth <entity-name> <daemon-type> <caps> [<daemon-type> <caps> ...]

For more information about auth capabilities, see User Management.

OSD_NO_DOWN_OUT_INTERVAL

The mon_osd_down_out_interval option is set to zero, which meansthat the system will not automatically perform any repair or healingoperations after an OSD fails. Instead, an administrator (or someother external entity) will need to manually mark down OSDs as ‘out’(i.e., via ceph osd out <osd-id>) in order to trigger recovery.

This option is normally set to five or ten minutes–enough time for ahost to power-cycle or reboot.

This warning can silenced by setting themon_warn_on_osd_down_out_interval_zero to false:

ceph config global mon mon_warn_on_osd_down_out_interval_zero false