Configuring Ceph
When you start the Ceph service, the initialization process activates a seriesof daemons that run in the background. A Ceph Storage Cluster runsthree types of daemons:
Ceph Monitor (
ceph-mon
)Ceph Manager (
ceph-mgr
)Ceph OSD Daemon (
ceph-osd
)
Ceph Storage Clusters that support the Ceph File System run atleast one Ceph Metadata Server (ceph-mds
). Clusters thatsupport Ceph Object Storage run Ceph Gateway daemons(radosgw
).
Each daemon has a series of configuration options, each of which has adefault values. You may adjust the behavior of the system by changing theseconfiguration options.
Option names
All Ceph configuration options have a unique name consisting of wordsformed with lower-case characters and connected with underscore(_
) characters.
When option names are specified on the command line, either underscore(_
) or dash (-
) characters can be used interchangeable (e.g.,—mon-host
is equivalent to —mon_host
).
When option names appear in configuration files, spaces can also beused in place of underscore or dash.
Config sources
Each Ceph daemon, process, and library will pull its configurationfrom several sources, listed below. Sources later in the list willoverride those earlier in the list when both are present.
the compiled-in default value
the monitor cluster’s centralized configuration database
a configuration file stored on the local host
environment variables
command line arguments
runtime overrides set by an administrator
One of the first things a Ceph process does on startup is parse theconfiguration options provided via the command line, environment, andlocal configuration file. The process will then contact the monitorcluster to retrieve configuration stored centrally for the entirecluster. Once a complete view of the configuration is available, thedaemon or process startup will proceed.
Bootstrap options
Because some configuration options affect the process’s ability tocontact the monitors, authenticate, and retrieve the cluster-storedconfiguration, they may need to be stored locally on the node and setin a local configuration file. These options include:
mon_host
, the list of monitors for the cluster
mon_dns_serv_name
(default: ceph-mon), the name of the DNSSRV record to check to identify the cluster monitors via DNS
mon_data
,osd_data
,mds_data
,mgr_data
, andsimilar options that define which local directory the daemonstores its data in.
keyring
,keyfile
, and/orkey
, which can be used tospecify the authentication credential to use to authenticate withthe monitor. Note that in most cases the default keyring locationis in the data directory specified above.
In the vast majority of cases the default values of these areappropriate, with the exception of the mon_host
option thatidentifies the addresses of the cluster’s monitors. When DNS is usedto identify monitors a local ceph configuration file can be avoidedentirely.
Skipping monitor config
Any process may be passed the option —no-mon-config
to skip thestep that retrieves configuration from the cluster monitors. This isuseful in cases where configuration is managed entirely viaconfiguration files or where the monitor cluster is currently down butsome maintenance activity needs to be done.
Configuration sections
Any given process or daemon has a single value for each configurationoption. However, values for an option may vary across differentdaemon types even daemons of the same type. Ceph options that arestored in the monitor configuration database or in local configurationfiles are grouped into sections to indicate which daemons or clientsthey apply to.
These sections include:
global
- Description
Settings under
global
affect all daemons and clientsin a Ceph Storage Cluster.Example
log_file = /var/log/ceph/$cluster-$type.$id.log
mon
- Description
Settings under
mon
affect allceph-mon
daemons inthe Ceph Storage Cluster, and override the same setting inglobal
.Example
mon_cluster_log_to_syslog = true
mgr
- Description
Settings in the
mgr
section affect allceph-mgr
daemons inthe Ceph Storage Cluster, and override the same setting inglobal
.Example
mgr_stats_period = 10
osd
- Description
Settings under
osd
affect allceph-osd
daemons inthe Ceph Storage Cluster, and override the same setting inglobal
.Example
osd_op_queue = wpq
mds
- Description
Settings in the
mds
section affect allceph-mds
daemons inthe Ceph Storage Cluster, and override the same setting inglobal
.Example
mds_cache_memory_limit = 10G
client
- Description
Settings under
client
affect all Ceph Clients(e.g., mounted Ceph File Systems, mounted Ceph Block Devices,etc.) as well as Rados Gateway (RGW) daemons.Example
objecter_inflight_ops = 512
Sections may also specify an individual daemon or client name. For example,mon.foo
, osd.123
, and client.smith
are all valid section names.
Any given daemon will draw its settings from the global section, thedaemon or client type section, and the section sharing its name.Settings in the most-specific section take precedence, so for exampleif the same option is specified in both global
, mon
, andmon.foo
on the same source (i.e., in the same configurationfile),the mon.foo
value will be used.
If multiple values of the same configuration option are specified in the samesection, the last value wins.
Note that values from the local configuration file always takeprecedence over values from the monitor configuration database,regardless of which section they appear in.
Metavariables
Metavariables simplify Ceph Storage Cluster configurationdramatically. When a metavariable is set in a configuration value,Ceph expands the metavariable into a concrete value at the time theconfiguration value is used. Ceph metavariables are similar to variable expansion in the Bash shell.
Ceph supports the following metavariables:
$cluster
- Description
Expands to the Ceph Storage Cluster name. Useful when runningmultiple Ceph Storage Clusters on the same hardware.
Example
/etc/ceph/$cluster.keyring
Default
ceph
$type
- Description
Expands to a daemon or process type (e.g.,
mds
,osd
, ormon
)Example
/var/lib/ceph/$type
$id
- Description
Expands to the daemon or client identifier. For
osd.0
, this would be0
; formds.a
, it wouldbea
.Example
/var/lib/ceph/$type/$cluster-$id
$host
- Description
- Expands to the host name where the process is running.
$name
- Description
Expands to
$type.$id
.Example
/var/run/ceph/$cluster-$name.asok
$pid
- Description
Expands to daemon pid.
Example
/var/run/ceph/$cluster-$name-$pid.asok
The Configuration File
On startup, Ceph processes search for a configuration file in thefollowing locations:
$CEPHCONF
(_i.e., the path following the$CEPH_CONF
environment variable)-c path/path
(i.e., the-c
command line argument)/etc/ceph/$cluster.conf
~/.ceph/$cluster.conf
./$cluster.conf
(i.e., in the current working directory)On FreeBSD systems only,
/usr/local/etc/ceph/$cluster.conf
where $cluster
is the cluster’s name (default ceph
).
The Ceph configuration file uses an ini style syntax. You can add commentsby preceding comments with a pound sign (#) or a semi-colon (;). For example:
- # <--A number (#) sign precedes a comment.
- ; A comment may be anything.
- # Comments always follow a semi-colon (;) or a pound (#) on each line.
- # The end of the line terminates a comment.
- # We recommend that you provide comments in your configuration file(s).
Config file section names
The configuration file is divided into sections. Each section must begin with avalid configuration section name (see Configuration sections, above)surrounded by square brackets. For example,
- [global]
- debug ms = 0
- [osd]
- debug ms = 1
- [osd.1]
- debug ms = 10
- [osd.2]
- debug ms = 10
Config file option values
The value of a configuration option is a string. If it is too long tofit in a single line, you can put a backslash (\
) at the end of lineas the line continuation marker, so the value of the option will bethe string after =
in current line combined with the string in the nextline:
- [global]
- foo = long long ago\
- long ago
In the example above, the value of “foo
” would be “long long ago long ago
”.
Normally, the option value ends with a new line, or a comment, like
- [global]
- obscure one = difficult to explain # I will try harder in next release
- simpler one = nothing to explain
In the example above, the value of “obscure one
” would be “difficult to explain
”;and the value of “simpler one
would be “nothing to explain
”.
If an option value contains spaces, and we want to make it explicit, wecould quote the value using single or double quotes, like
- [global]
- line = "to be, or not to be"
Certain characters are not allowed to be present in the option values directly.They are =
, #
, ;
and [
. If we have to, we need to escape them,like
- [global]
- secret = "i love \# and \["
Monitor configuration database
The monitor cluster manages a database of configuration options thatcan be consumed by the entire cluster, enabling streamlined centralconfiguration management for the entire system. The vast majority ofconfiguration options can and should be stored here for ease ofadministration and transparency.
A handful of settings may still need to be stored in localconfiguration files because they affect the ability to connect to themonitors, authenticate, and fetch configuration information. In mostcases this is limited to the mon_host
option, although this canalso be avoided through the use of DNS SRV records.
Sections and masks
Configuration options stored by the monitor can live in a globalsection, daemon type section, or specific daemon section, just likeoptions in a configuration file can.
In addition, options may also have a mask associated with them tofurther restrict which daemons or clients the option applies to.Masks take two forms:
type:location
where type is a CRUSH property like rack orhost, and location is a value for that property. For example,host:foo
would limit the option only to daemons or clientsrunning on a particular host.class:device-class
where device-class is the name of a CRUSHdevice class (e.g.,hdd
orssd
). For example,class:ssd
would limit the option only to OSDs backed by SSDs.(This mask has no effect for non-OSD daemons or clients.)
When setting a configuration option, the who may be a section name,a mask, or a combination of both separated by a slash (/
)character. For example, osd/rack:foo
would mean all OSD daemonsin the foo
rack.
When viewing configuration options, the section name and mask aregenerally separated out into separate fields or columns to ease readability.
Commands
The following CLI commands are used to configure the cluster:
ceph config dump
will dump the entire configuration database forthe cluster.ceph config get <who>
will dump the configuration for a specificdaemon or client (e.g.,mds.a
), as stored in the monitors’configuration database.ceph config set <who> <option> <value>
will set a configurationoption in the monitors’ configuration database.ceph config show <who>
will show the reported runningconfiguration for a running daemon. These settings may differ fromthose stored by the monitors if there are also local configurationfiles in use or options have been overridden on the command line orat run time. The source of the option values is reported as partof the output.ceph config assimilate-conf -i <input file> -o <output file>
will ingest a configuration file from input file and move anyvalid options into the monitors’ configuration database. Anysettings that are unrecognized, invalid, or cannot be controlled bythe monitor will be returned in an abbreviated config file stored inoutput file. This command is useful for transitioning from legacyconfiguration files to centralized monitor-based configuration.
Help
You can get help for a particular option with:
- ceph config help <option>
Note that this will use the configuration schema that is compiled into the running monitors. If you have a mixed-version cluster (e.g., during an upgrade), you might also want to query the option schema from a specific running daemon:
- ceph daemon <name> config help [option]
For example,:
- $ ceph config help log_file
- log_file - path to log file
- (std::string, basic)
- Default (non-daemon):
- Default (daemon): /var/log/ceph/$cluster-$name.log
- Can update at runtime: false
- See also: [log_to_stderr,err_to_stderr,log_to_syslog,err_to_syslog]
or:
- $ ceph config help log_file -f json-pretty
- {
- "name": "log_file",
- "type": "std::string",
- "level": "basic",
- "desc": "path to log file",
- "long_desc": "",
- "default": "",
- "daemon_default": "/var/log/ceph/$cluster-$name.log",
- "tags": [],
- "services": [],
- "see_also": [
- "log_to_stderr",
- "err_to_stderr",
- "log_to_syslog",
- "err_to_syslog"
- ],
- "enum_values": [],
- "min": "",
- "max": "",
- "can_update_at_runtime": false
- }
The level
property can be any of basic, advanced, or dev.The dev options are intended for use by developers, generally fortesting purposes, and are not recommended for use by operators.
Runtime Changes
In most cases, Ceph allows you to make changes to the configuration ofa daemon at runtime. This capability is quite useful forincreasing/decreasing logging output, enabling/disabling debugsettings, and even for runtime optimization.
Generally speaking, configuration options can be updated in the usualway via the ceph config set
command. For example, do enable the debug log level on a specific OSD,:
- ceph config set osd.123 debug_ms 20
Note that if the same option is also customized in a localconfiguration file, the monitor setting will be ignored (it has alower priority than the local config file).
Override values
You can also temporarily set an option using the tell or daemon_interfaces on the Ceph CLI. These _override values are ephemeral inthat they only affect the running process and are discarded/lost ifthe daemon or process restarts.
Override values can be set in two ways:
- From any host, we can send a message to a daemon over the network with:
- ceph tell <name> config set <option> <value>
For example,:
- ceph tell osd.123 config set debug_osd 20
The tell command can also accept a wildcard for the daemonidentifier. For example, to adjust the debug level on all OSDdaemons,:
- ceph tell osd.* config set debug_osd 20
- From the host the process is running on, we can connect directly tothe process via a socket in
/var/run/ceph
with:
- ceph daemon <name> config set <option> <value>
For example,:
- ceph daemon osd.4 config set debug_osd 20
Note that in the ceph config show
command output these temporaryvalues will be shown with a source of override
.
Viewing runtime settings
You can see the current options set for a running daemon with the ceph config show
command. For example,:
- ceph config show osd.0
will show you the (non-default) options for that daemon. You can also look at a specific option with:
- ceph config show osd.0 debug_osd
or view all options (even those with default values) with:
- ceph config show-with-defaults osd.0
You can also observe settings for a running daemon by connecting to it from the local host via the admin socket. For example,:
- ceph daemon osd.0 config show
will dump all current settings,:
- ceph daemon osd.0 config diff
will show only non-default settings (as well as where the value came from: a config file, the monitor, an override, etc.), and:
- ceph daemon osd.0 config get debug_osd
will report the value of a single option.
Changes since nautilus
We changed the way the configuration file is parsed in Octopus. The changes arelisted as follows:
- repeated configuration options are allowed, and no warnings will be printed.The value of the last one wins. Before this change, we would print out warningmessages at seeing lines with duplicated values, like:
- warning line 42: 'foo' in section 'bar' redefined
invalid UTF-8 options are ignored with warning messages. But since Octopus,they are treated as fatal errors.
backslash
\
is used as the line continuation marker to combine the nextline with current one. Before Octopus, it was required to follow backslash withnon-empty line. But in Octopus, empty line following backslash is now allowed.In the configuration file, each line specifies an individual configurationoption. The option’s name and its value are separated with
=
. And thevalue can be quoted using single or double quotes. But if an invalidconfiguration is specified, we will treat it as an invalid configurationfile
- bad option ==== bad value
- Before Octopus, if no section name was specified in the configuration file,all options would be grouped into the section of
global
. But this isdiscouraged now. Since Octopus, only a single option is allowed forconfiguration files without a section name.