Memory Profiling

Ceph MON, OSD and MDS can generate heap profiles usingtcmalloc. To generate heap profiles, ensure you havegoogle-perftools installed:

  1. sudo apt-get install google-perftools

The profiler dumps output to your log file directory (i.e.,/var/log/ceph). See Logging and Debugging for details.To view the profiler logs with Google’s performance tools, execute thefollowing:

  1. google-pprof --text {path-to-daemon}  {log-path/filename}

For example:

  1. $ ceph tell osd.0 heap start_profiler
  2. $ ceph tell osd.0 heap dump
  3. osd.0 tcmalloc heap stats:------------------------------------------------
  4. MALLOC:        2632288 (    2.5 MiB) Bytes in use by application
  5. MALLOC: +       499712 (    0.5 MiB) Bytes in page heap freelist
  6. MALLOC: +       543800 (    0.5 MiB) Bytes in central cache freelist
  7. MALLOC: +       327680 (    0.3 MiB) Bytes in transfer cache freelist
  8. MALLOC: +      1239400 (    1.2 MiB) Bytes in thread cache freelists
  9. MALLOC: +      1142936 (    1.1 MiB) Bytes in malloc metadata
  10. MALLOC:   ------------
  11. MALLOC: =      6385816 (    6.1 MiB) Actual memory used (physical + swap)
  12. MALLOC: +            0 (    0.0 MiB) Bytes released to OS (aka unmapped)
  13. MALLOC:   ------------
  14. MALLOC: =      6385816 (    6.1 MiB) Virtual address space used
  15. MALLOC:
  16. MALLOC:            231              Spans in use
  17. MALLOC:             56              Thread heaps in use
  18. MALLOC:           8192              Tcmalloc page size
  19. ------------------------------------------------
  20. Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
  21. Bytes released to the OS take up virtual address space but no physical memory.
  22. $ google-pprof --text \
  23.                /usr/bin/ceph-osd  \
  24.                /var/log/ceph/ceph-osd.0.profile.0001.heap
  25.  Total: 3.7 MB
  26.  1.9  51.1%  51.1%      1.9  51.1% ceph::log::Log::create_entry
  27.  1.8  47.3%  98.4%      1.8  47.3% std::string::_Rep::_S_create
  28.  0.0   0.4%  98.9%      0.0   0.6% SimpleMessenger::add_accept_pipe
  29.  0.0   0.4%  99.2%      0.0   0.6% decode_message
  30.  ...

Another heap dump on the same daemon will add another file. It isconvenient to compare to a previous heap dump to show what has grownin the interval. For instance:

  1. $ google-pprof --text --base out/osd.0.profile.0001.heap \
  2.       ceph-osd out/osd.0.profile.0003.heap
  3.  Total: 0.2 MB
  4.  0.1  50.3%  50.3%      0.1  50.3% ceph::log::Log::create_entry
  5.  0.1  46.6%  96.8%      0.1  46.6% std::string::_Rep::_S_create
  6.  0.0   0.9%  97.7%      0.0  26.1% ReplicatedPG::do_op
  7.  0.0   0.8%  98.5%      0.0   0.8% __gnu_cxx::new_allocator::allocate

Refer to Google Heap Profiler for additional details.

Once you have the heap profiler installed, start your cluster andbegin using the heap profiler. You may enable or disable the heapprofiler at runtime, or ensure that it runs continuously. For thefollowing commandline usage, replace {daemon-type} with mon,osd or mds, and replace {daemon-id} with the OSD number orthe MON or MDS id.

Starting the Profiler

To start the heap profiler, execute the following:

  1. ceph tell {daemon-type}.{daemon-id} heap start_profiler

For example:

  1. ceph tell osd.1 heap start_profiler

Alternatively the profile can be started when the daemon startsrunning if the CEPH_HEAP_PROFILER_INIT=true variable is found inthe environment.

Printing Stats

To print out statistics, execute the following:

  1. ceph  tell {daemon-type}.{daemon-id} heap stats

For example:

  1. ceph tell osd.0 heap stats

Note

Printing stats does not require the profiler to be running and doesnot dump the heap allocation information to a file.

Dumping Heap Information

To dump heap information, execute the following:

  1. ceph tell {daemon-type}.{daemon-id} heap dump

For example:

  1. ceph tell mds.a heap dump

Note

Dumping heap information only works when the profiler is running.

Releasing Memory

To release memory that tcmalloc has allocated but which is not being used bythe Ceph daemon itself, execute the following:

  1. ceph tell {daemon-type}{daemon-id} heap release

For example:

  1. ceph tell osd.2 heap release

Stopping the Profiler

To stop the heap profiler, execute the following:

  1. ceph tell {daemon-type}.{daemon-id} heap stop_profiler

For example:

  1. ceph tell osd.0 heap stop_profiler