v0.56.7 “bobtail”

This bobtail update fixes a range of radosgw bugs (including an easily triggered crash from multi-delete), a possible data corruption issue with power failure on XFS, and several OSD problems, including a memory “leak” that will affect aged clusters.

Notable changes

• ceph-fuse: create finisher flags after fork()

• debian: fix prerm/postinst hooks; do not restart daemons on upgrade

• librados: fix async aio completion wakeup (manifests as rbd hang)

• librados: fix hang when osd becomes full and then not full

• librados: fix locking for aio completion refcounting

• librbd python bindings: fix stripe_unit, stripe_count

• librbd: make image creation default configurable

• mon: fix validation of mds ids in mon commands

• osd: avoid excessive disk updates during peering

• osd: avoid excessive memory usage on scrub

• osd: avoid heartbeat failure/suicide when scrubbing

• osd: misc minor bug fixes

• osd: use fdatasync instead of sync_file_range (may avoid xfs power-loss corruption)

• rgw: escape prefix correctly when listing objects

• rgw: fix copy attrs

• rgw: fix crash on multi delete

• rgw: fix locking/crash when using ops log socket

• rgw: fix usage logging

• rgw: handle deep uri resources

For more detailed information, see the complete changelog.

v0.56.6 “bobtail”

Notable changes

• rgw: fix garbage collection

• rpm: fix package dependencies

For more detailed information, see the complete changelog.

v0.56.5 “bobtail”

Upgrading

• ceph-disk[-prepare,-activate] behavior has changed in various ways. There should not be any compatibility issues, but chef users should be aware.

Notable changes

• mon: fix recording of quorum feature set (important for argonaut -> bobtail -> cuttlefish mon upgrades)

• osd: minor peering bug fixes

• osd: fix a few bugs when pools are renamed

• osd: fix occasionally corrupted pg stats

• osd: fix behavior when broken v0.56[.0] clients connect

• rbd: avoid FIEMAP ioctl on import (it is broken on some kernels)

• librbd: fixes for several request/reply ordering bugs

• librbd: only set STRIPINGV2 feature on new images when needed

• librbd: new async flush method to resolve qemu hangs (requires QEMU update as well)

• librbd: a few fixes to flatten

• ceph-disk: support for dm-crypt

• ceph-disk: many backports to allow bobtail deployments with ceph-deploy, chef

• sysvinit: do not stop starting daemons on first failure

• udev: fixed rules for redhat-based distros

• build fixes for raring

For more detailed information, see the complete changelog.

v0.56.4 “bobtail”

Upgrading

• There is a fix in the syntax for the output of ‘ceph osd tree –format=json’.

• The MDS disk format has changed from prior releases and from v0.57. In particular, upgrades to v0.56.4 are safe, but you cannot move from v0.56.4 to v0.57 if you are using the MDS for CephFS; you must upgrade directly to v0.58 (or later) instead.

Notable changes

• mon: fix bug in bringup with IPv6

• reduce default memory utilization by internal logging (all daemons)

• rgw: fix for bucket removal

• rgw: reopen logs after log rotation

• rgw: fix multipat upload listing

• rgw: don’t copy object when copied onto self

• osd: fix caps parsing for pools with - or _

• osd: allow pg log trimming when degraded, scrubbing, recoverying (reducing memory consumption)

• osd: fix potential deadlock when ‘journal aio = true’

• osd: various fixes for collection creation/removal, rename, temp collections

• osd: various fixes for PG split

• osd: deep-scrub omap key/value data

• osd: fix rare bug in journal replay

• osd: misc fixes for snapshot tracking

• osd: fix leak in recovery reservations on pool deletion

• osd: fix bug in connection management

• osd: fix for op ordering when rebalancing

• ceph-fuse: report file system size with correct units

• mds: get and set directory layout policies via virtual xattrs

• mds: on-disk format revision (see upgrading note above)

• mkcephfs, init-ceph: close potential security issues with predictable filenames

For more detailed information, see the complete changelog.

v0.56.3 “bobtail”

This release has several bug fixes surrounding OSD stability. Most significantly, an issue with OSDs being unresponsive shortly after startup (and occasionally crashing due to an internal heartbeat check) is resolved. Please upgrade.

Upgrading

• A bug was fixed in which the OSDMap epoch for PGs without any IO requests was not recorded. If there are pools in the cluster that are completely idle (for example, the data and metadata pools normally used by CephFS), and a large number of OSDMap epochs have elapsed since the ceph-osd daemon was last restarted, those maps will get reprocessed when the daemon restarts. This process can take a while if there are a lot of maps. A workaround is to ‘touch’ any idle pools with IO prior to restarting the daemons after packages are upgraded:

  rados bench 10 write -t 1 -b 4096 -p {POOLNAME}

  This will typically generate enough IO to touch every PG in the pool without generating significant cluster load, and also cleans up any temporary objects it creates.

Notable changes

• osd: flush peering work queue prior to start

• osd: persist osdmap epoch for idle PGs

• osd: fix and simplify connection handling for heartbeats

• osd: avoid crash on invalid admin command

• mon: fix rare races with monitor elections and commands

• mon: enforce that OSD reweights be between 0 and 1 (NOTE: not CRUSH weights)

• mon: approximate client, recovery bandwidth logging

• radosgw: fixed some XML formatting to conform to Swift API inconsistency

• radosgw: fix usage accounting bug; add repair tool

• radosgw: make fallback URI configurable (necessary on some web servers)

• librbd: fix handling for interrupted ‘unprotect’ operations

• mds, ceph-fuse: allow file and directory layouts to be modified via virtual xattrs

For more detailed information, see the complete changelog.

v0.56.2 “bobtail”

This release has a wide range of bug fixes, stability improvements, and some performance improvements. Please upgrade.

Upgrading

• The meaning of the ‘osd scrub min interval’ and ‘osd scrub max interval’ has changed slightly. The min interval used to be meaningless, while the max interval would only trigger a scrub if the load was sufficiently low. Now, the min interval option works the way the old max interval did (it will trigger a scrub after this amount of time if the load is low), while the max interval will force a scrub regardless of load. The default options have been adjusted accordingly. If you have customized these in ceph.conf, please review their values when upgrading.

• CRUSH maps that are generated by default when calling ceph-mon --mkfs directly now distribute replicas across hosts instead of across OSDs. Any provisioning tools that are being used by Ceph may be affected, although probably for the better, as distributing across hosts is a much more commonly sought behavior. If you use mkcephfs to create the cluster, the default CRUSH rule is still inferred by the number of hosts and/or racks in the initial ceph.conf.

Notable changes

• osd: snapshot trimming fixes

• osd: scrub snapshot metadata

• osd: fix osdmap trimming

• osd: misc peering fixes

• osd: stop heartbeating with peers if internal threads are stuck/hung

• osd: PG removal is friendlier to other workloads

• osd: fix recovery start delay (was causing very slow recovery)

• osd: fix scheduling of explicitly requested scrubs

• osd: fix scrub interval config options

• osd: improve recovery vs client io tuning

• osd: improve ‘slow request’ warning detail for better diagnosis

• osd: default CRUSH map now distributes across hosts, not OSDs

• osd: fix crash on 32-bit hosts triggered by librbd clients

• librbd: fix error handling when talking to older OSDs

• mon: fix a few rare crashes

• ceph command: ability to easily adjust CRUSH tunables

• radosgw: object copy does not copy source ACLs

• rados command: fix omap command usage

• sysvinit script: set ulimit -n properly on remote hosts

• msgr: fix narrow race with message queuing

• fixed compilation on some old distros (e.g., RHEL 5.x)

For more detailed information, see the complete changelog.

v0.56.1 “bobtail”

This release has two critical fixes. Please upgrade.

Upgrading

• There is a protocol compatibility problem between v0.56 and any other version that is now fixed. If your radosgw or RBD clients are running v0.56, they will need to be upgraded too. If they are running a version prior to v0.56, they can be left as is.

Notable changes

• osd: fix commit sequence for XFS, ext4 (or any other non-btrfs) to prevent data loss on power cycle or kernel panic

• osd: fix compatibility for CALL operation

• osd: process old osdmaps prior to joining cluster (fixes slow startup)

• osd: fix a couple of recovery-related crashes

• osd: fix large io requests when journal is in (non-default) aio mode

• log: fix possible deadlock in logging code

For more detailed information, see the complete changelog.

v0.56 “bobtail”

Bobtail is the second stable release of Ceph, named in honor of the Bobtail Squid: https://en.wikipedia.org/wiki/Bobtail_squid.

Key features since v0.48 “argonaut”

• Object Storage Daemon (OSD): improved threading, small-io performance, and performance during recovery

• Object Storage Daemon (OSD): regular “deep” scrubbing of all stored data to detect latent disk errors

• RADOS Block Device (RBD): support for copy-on-write clones of images.

• RADOS Block Device (RBD): better client-side caching.

• RADOS Block Device (RBD): advisory image locking

• Rados Gateway (RGW): support for efficient usage logging/scraping (for billing purposes)

• Rados Gateway (RGW): expanded S3 and Swift API coverage (e.g., POST, multi-object delete)

• Rados Gateway (RGW): improved striping for large objects

• Rados Gateway (RGW): OpenStack Keystone integration

• RPM packages for Fedora, RHEL/CentOS, OpenSUSE, and SLES

• mkcephfs: support for automatically formatting and mounting XFS and ext4 (in addition to btrfs)

Upgrading

Please refer to the document Upgrading from Argonaut to Bobtail for details.

• Cephx authentication is now enabled by default (since v0.55). Upgrading a cluster without adjusting the Ceph configuration will likely prevent the system from starting up on its own. We recommend first modifying the configuration to indicate that authentication is disabled, and only then upgrading to the latest version:

  auth client required = none
  auth service required = none
  auth cluster required = none

• Ceph daemons can be upgraded one-by-one while the cluster is online and in service.

• The ceph-osd daemons must be upgraded and restarted before any radosgw daemons are restarted, as they depend on some new ceph-osd functionality. (The ceph-mon, ceph-osd, and ceph-mds daemons can be upgraded and restarted in any order.)

• Once each individual daemon has been upgraded and restarted, it cannot be downgraded.

• The cluster of ceph-mon daemons will migrate to a new internal on-wire protocol once all daemons in the quorum have been upgraded. Upgrading only a majority of the nodes (e.g., two out of three) may expose the cluster to a situation where a single additional failure may compromise availability (because the non-upgraded daemon cannot participate in the new protocol). We recommend not waiting for an extended period of time between ceph-mon upgrades.

• The ops log and usage log for radosgw are now off by default. If you need these logs (e.g., for billing purposes), you must enable them explicitly. For logging of all operations to objects in the .log pool (see radosgw-admin log ...):

  rgw enable ops log = true

  For usage logging of aggregated bandwidth usage (see radosgw-admin usage ...):

  rgw enable usage log = true

• You should not create or use “format 2” RBD images until after all ceph-osd daemons have been upgraded. Note that “format 1” is still the default. You can use the new ceph osd ls and ceph tell osd.N version commands to doublecheck your cluster. ceph osd ls will give a list of all OSD IDs that are part of the cluster, and you can use that to write a simple shell loop to display all the OSD version strings:

  for i in $(ceph osd ls); do
      ceph tell osd.${i} version
  done

Compatibility changes

• The ‘ceph osd create [<uuid>]’ command now rejects an argument that is not a UUID. (Previously it would take take an optional integer OSD id.) This correct syntax has been ‘ceph osd create [<uuid>]’ since v0.47, but the older calling convention was being silently ignored.

• The CRUSH map root nodes now have type root instead of type pool. This avoids confusion with RADOS pools, which are not directly related. Any scripts or tools that use the ceph osd crush ... commands may need to be adjusted accordingly.

• The ceph osd pool create <poolname> <pgnum> command now requires the pgnum argument. Previously this was optional, and would default to 8, which was almost never a good number.

• Degraded mode (when there fewer than the desired number of replicas) is now more configurable on a per-pool basis, with the min_size parameter. By default, with min_size 0, this allows I/O to objects with N - floor(N/2) replicas, where N is the total number of expected copies. Argonaut behavior was equivalent to having min_size = 1, so I/O would always be possible if any completely up to date copy remained. min_size = 1 could result in lower overall availability in certain cases, such as flapping network partitions.

• The sysvinit start/stop script now defaults to adjusting the max open files ulimit to 16384. On most systems the default is 1024, so this is an increase and won’t break anything. If some system has a higher initial value, however, this change will lower the limit. The value can be adjusted explicitly by adding an entry to the ceph.conf file in the appropriate section. For example:

  [global]
          max open files = 32768

• ‘rbd lock list’ and ‘rbd showmapped’ no longer use tabs as separators in their output.

• There is configurable limit on the number of PGs when creating a new pool, to prevent a user from accidentally specifying a ridiculous number for pg_num. It can be adjusted via the ‘mon max pool pg num’ option on the monitor, and defaults to 65536 (the current max supported by the Linux kernel client).

• The osd capabilities associated with a rados user have changed syntax since 0.48 argonaut. The new format is mostly backwards compatible, but there are two backwards-incompatible changes:

  • specifying a list of pools in one grant, i.e. ‘allow r pool=foo,bar’ is now done in separate grants, i.e. ‘allow r pool=foo, allow r pool=bar’.

  • restricting pool access by pool owner (‘allow r uid=foo’) is removed. This feature was not very useful and unused in practice.

  The new format is documented in the ceph-authtool man page.

• ‘rbd cp’ and ‘rbd rename’ use rbd as the default destination pool, regardless of what pool the source image is in. Previously they would default to the same pool as the source image.

• ‘rbd export’ no longer prints a message for each object written. It just reports percent complete like other long-lasting operations.

• ‘ceph osd tree’ now uses 4 decimal places for weight so output is nicer for humans

• Several monitor operations are now idempotent:

  • ceph osd pool create

  • ceph osd pool delete

  • ceph osd pool mksnap

  • ceph osd rm

  • ceph pg <pgid> revert

Notable changes

• auth: enable cephx by default

• auth: expanded authentication settings for greater flexibility

• auth: sign messages when using cephx

• build fixes for Fedora 18, CentOS/RHEL 6

• ceph: new ‘osd ls’ and ‘osd tell <osd.N> version’ commands

• ceph-debugpack: misc improvements

• ceph-disk-prepare: creates and labels GPT partitions

• ceph-disk-prepare: support for external journals, default mount/mkfs options, etc.

• ceph-fuse/libcephfs: many misc fixes, admin socket debugging

• ceph-fuse: fix handling for .. in root directory

• ceph-fuse: many fixes (including memory leaks, hangs)

• ceph-fuse: mount helper (mount.fuse.ceph) for use with /etc/fstab

• ceph.spec: misc packaging fixes

• common: thread pool sizes can now be adjusted at runtime

• config: $pid is now available as a metavariable

• crush: default root of tree type is now ‘root’ instead of ‘pool’ (to avoid confusiong wrt rados pools)

• crush: fixed retry behavior with chooseleaf via tunable

• crush: tunables documented; feature bit now present and enforced

• libcephfs: java wrapper

• librados: several bug fixes (rare races, locking errors)

• librados: some locking fixes

• librados: watch/notify fixes, misc memory leaks

• librbd: a few fixes to ‘discard’ support

• librbd: fine-grained striping feature

• librbd: fixed memory leaks

• librbd: fully functional and documented image cloning

• librbd: image (advisory) locking

• librbd: improved caching (of object non-existence)

• librbd: ‘flatten’ command to sever clone parent relationship

• librbd: ‘protect’/’unprotect’ commands to prevent clone parent from being deleted

• librbd: clip requests past end-of-image.

• librbd: fixes an issue with some windows guests running in qemu (remove floating point usage)

• log: fix in-memory buffering behavior (to only write log messages on crash)

• mds: fix ino release on abort session close, relative getattr path, mds shutdown, other misc items

• mds: misc fixes

• mkcephfs: fix for default keyring, osd data/journal locations

• mkcephfs: support for formatting xfs, ext4 (as well as btrfs)

• init: support for automatically mounting xfs and ext4 osd data directories

• mon, radosgw, ceph-fuse: fixed memory leaks

• mon: improved ENOSPC, fs error checking

• mon: less-destructive ceph-mon –mkfs behavior

• mon: misc fixes

• mon: more informative info about stuck PGs in ‘health detail’

• mon: information about recovery and backfill in ‘pg <pgid> query’

• mon: new ‘osd crush create-or-move …’ command

• mon: new ‘osd crush move …’ command lets you rearrange your CRUSH hierarchy

• mon: optionally dump ‘osd tree’ in json

• mon: configurable cap on maximum osd number (mon max osd)

• mon: many bug fixes (various races causing ceph-mon crashes)

• mon: new on-disk metadata to facilitate future mon changes (post-bobtail)

• mon: election bug fixes

• mon: throttle client messages (limit memory consumption)

• mon: throttle osd flapping based on osd history (limits osdmap ΄thrashing’ on overloaded or unhappy clusters)

• mon: ‘report’ command for dumping detailed cluster status (e.g., for use when reporting bugs)

• mon: osdmap flags like noup, noin now cause a health warning

• msgr: improved failure handling code

• msgr: many bug fixes

• osd, mon: honor new ‘nobackfill’ and ‘norecover’ osdmap flags

• osd, mon: use feature bits to lock out clients lacking CRUSH tunables when they are in use

• osd: backfill reservation framework (to avoid flooding new osds with backfill data)

• osd: backfill target reservations (improve performance during recovery)

• osd: better tracking of recent slow operations

• osd: capability grammar improvements, bug fixes

• osd: client vs recovery io prioritization

• osd: crush performance improvements

• osd: default journal size to 5 GB

• osd: experimental support for PG “splitting” (pg_num adjustment for existing pools)

• osd: fix memory leak on certain error paths

• osd: fixed detection of EIO errors from fs on read

• osd: major refactor of PG peering and threading

• osd: many bug fixes

• osd: more/better dump info about in-progress operations

• osd: new caps structure (see compatibility notes)

• osd: new ‘deep scrub’ will compare object content across replicas (once per week by default)

• osd: new ‘lock’ rados class for generic object locking

• osd: optional ‘min’ pg size

• osd: recovery reservations

• osd: scrub efficiency improvement

• osd: several out of order reply bug fixes

• osd: several rare peering cases fixed

• osd: some performance improvements related to request queuing

• osd: use entire device if journal is a block device

• osd: use syncfs(2) when kernel supports it, even if glibc does not

• osd: various fixes for out-of-order op replies

• rados: ability to copy, rename pools

• rados: bench command now cleans up after itself

• rados: ‘cppool’ command to copy rados pools

• rados: ‘rm’ now accepts a list of objects to be removed

• radosgw: POST support

• radosgw: REST API for managing usage stats

• radosgw: fix bug in bucket stat updates

• radosgw: fix copy-object vs attributes

• radosgw: fix range header for large objects, ETag quoting, GMT dates, other compatibility fixes

• radosgw: improved garbage collection framework

• radosgw: many small fixes, cleanups

• radosgw: openstack keystone integration

• radosgw: stripe large (non-multipart) objects

• radosgw: support for multi-object deletes

• radosgw: support for swift manifest objects

• radosgw: vanity bucket dns names

• radosgw: various API compatibility fixes

• rbd: import from stdin, export to stdout

• rbd: new ‘ls -l’ option to view images with metadata

• rbd: use generic id and keyring options for ‘rbd map’

• rbd: don’t issue usage on errors

• udev: fix symlink creation for rbd images containing partitions

• upstart: job files for all daemon types (not enabled by default)

• wireshark: ceph protocol dissector patch updated

v0.54

Upgrading

• The osd capabilities associated with a rados user have changed syntax since 0.48 argonaut. The new format is mostly backwards compatible, but there are two backwards-incompatible changes:

  • specifying a list of pools in one grant, i.e. ‘allow r pool=foo,bar’ is now done in separate grants, i.e. ‘allow r pool=foo, allow r pool=bar’.

  • restricting pool access by pool owner (‘allow r uid=foo’) is removed. This feature was not very useful and unused in practice.

  The new format is documented in the ceph-authtool man page.

• Bug fixes to the new osd capability format parsing properly validate the allowed operations. If an existing rados user gets permissions errors after upgrading, its capabilities were probably misconfigured. See the ceph-authtool man page for details on osd capabilities.

• ‘rbd lock list’ and ‘rbd showmapped’ no longer use tabs as separators in their output.