CephFS Administrative commands

File Systems

These commands operate on the CephFS file systems in your Ceph cluster.Note that by default only one file system is permitted: to enablecreation of multiple file systems use ceph fs flag set enable_multiple true.

  1. fs new <file system name> <metadata pool name> <data pool name>

This command creates a new file system. The file system name and metadata poolname are self-explanatory. The specified data pool is the default data pool andcannot be changed once set. Each file system has its own set of MDS daemonsassigned to ranks so ensure that you have sufficient standby daemons availableto accommodate the new file system.

  1. fs ls

List all file systems by name.

  1. fs dump [epoch]

This dumps the FSMap at the given epoch (default: current) which includes allfile system settings, MDS daemons and the ranks they hold, and the list ofstandby MDS daemons.

  1. fs rm <file system name> [--yes-i-really-mean-it]

Destroy a CephFS file system. This wipes information about the state of thefile system from the FSMap. The metadata pool and data pools are untouched andmust be destroyed separately.

  1. fs get <file system name>

Get information about the named file system, including settings and ranks. Thisis a subset of the same information from the fs dump command.

  1. fs set <file system name> <var> <val>

Change a setting on a file system. These settings are specific to the namedfile system and do not affect other file systems.

  1. fs add_data_pool <file system name> <pool name/id>

Add a data pool to the file system. This pool can be used for file layoutsas an alternate location to store file data.

  1. fs rm_data_pool <file system name> <pool name/id>

This command removes the specified pool from the list of data pools for thefile system. If any files have layouts for the removed data pool, the filedata will become unavailable. The default data pool (when creating the filesystem) cannot be removed.

Settings

  1. fs set <fs name> max_file_size <size in bytes>

CephFS has a configurable maximum file size, and it’s 1TB by default.You may wish to set this limit higher if you expect to store large filesin CephFS. It is a 64-bit field.

Setting max_file_size to 0 does not disable the limit. It wouldsimply limit clients to only creating empty files.

Maximum file sizes and performance

CephFS enforces the maximum file size limit at the point of appending tofiles or setting their size. It does not affect how anything is stored.

When users create a file of an enormous size (without necessarilywriting any data to it), some operations (such as deletes) cause the MDSto have to do a large number of operations to check if any of the RADOSobjects within the range that could exist (according to the file size)really existed.

The max_file_size setting prevents users from creating files thatappear to be eg. exabytes in size, causing load on the MDS as it triesto enumerate the objects during operations like stats or deletes.

Taking the cluster down

Taking a CephFS cluster down is done by setting the down flag:

  1. fs set <fs_name> down true

To bring the cluster back online:

  1. fs set <fs_name> down false

This will also restore the previous value of max_mds. MDS daemons are broughtdown in a way such that journals are flushed to the metadata pool and allclient I/O is stopped.

Taking the cluster down rapidly for deletion or disaster recovery

To allow rapidly deleting a file system (for testing) or to quickly bring thefile system and MDS daemons down, use the fs fail command:

  1. fs fail <fs_name>

This command sets a file system flag to prevent standbys fromactivating on the file system (the joinable flag).

This process can also be done manually by doing the following:

  1. fs set <fs_name> joinable false

Then the operator can fail all of the ranks which causes the MDS daemons torespawn as standbys. The file system will be left in a degraded state.

  1. # For all ranks, 0-N:
  2. mds fail <fs_name>:<n>

Once all ranks are inactive, the file system may also be deleted or left inthis state for other purposes (perhaps disaster recovery).

To bring the cluster back up, simply set the joinable flag:

  1. fs set <fs_name> joinable true

Daemons

Most commands manipulating MDSs take a <role> argument which can take oneof three forms:

  1. <fs_name>:<rank>
  2. <fs_id>:<rank>
  3. <rank>

Commands to manipulate MDS daemons:

  1. mds fail <gid/name/role>

Mark an MDS daemon as failed. This is equivalent to what the clusterwould do if an MDS daemon had failed to send a message to the monfor mds_beacon_grace second. If the daemon was active and a suitablestandby is available, using mds fail will force a failover to the standby.

If the MDS daemon was in reality still running, then using mds failwill cause the daemon to restart. If it was active and a standby wasavailable, then the “failed” daemon will return as a standby.

  1. tell mds.<daemon name> command ...

Send a command to the MDS daemon(s). Use mds. to send a command to alldaemons. Use ceph tell mds. help to learn available commands.

  1. mds metadata <gid/name/role>

Get metadata about the given MDS known to the Monitors.

  1. mds repaired <role>

Mark the file system rank as repaired. Unlike the name suggests, this commanddoes not change a MDS; it manipulates the file system rank which has beenmarked damaged.

Minimum Client Version

It is sometimes desirable to set the minimum version of Ceph that a client must berunning to connect to a CephFS cluster. Older clients may sometimes still berunning with bugs that can cause locking issues between clients (due tocapability release). CephFS provides a mechanism to set the minimumclient version:

  1. fs set <fs name> min_compat_client <release>

For example, to only allow Nautilus clients, use:

  1. fs set cephfs min_compat_client nautilus

Clients running an older version will be automatically evicted.

Global settings

  1. fs flag set <flag name> <flag val> [<confirmation string>]

Sets a global CephFS flag (i.e. not specific to a particular file system).Currently, the only flag setting is ‘enable_multiple’ which allows havingmultiple CephFS file systems.

Some flags require you to confirm your intentions with “–yes-i-really-mean-it”or a similar string they will prompt you with. Consider these actions carefullybefore proceeding; they are placed on especially dangerous activities.

Advanced

These commands are not required in normal operation, and existfor use in exceptional circumstances. Incorrect use of thesecommands may cause serious problems, such as an inaccessiblefile system.

  1. mds compat rm_compat

Removes an compatibility feature flag.

  1. mds compat rm_incompat

Removes an incompatibility feature flag.

  1. mds compat show

Show MDS compatibility flags.

  1. mds rmfailed

This removes a rank from the failed set.

  1. fs reset <file system name>

This command resets the file system state to defaults, except for the name andpools. Non-zero ranks are saved in the stopped set.