Network Protocol

This file describes the network protocol used by Ceph. In order to understandthe way the structures are defined it is recommended to read the introductionof Network Encoding first.

Hello

The protocol starts with a handshake that confirms that both nodes are talkingceph and shares some basic information.

Banner

The first action is the server sending banner to the client. The banner isdefined in CEPH_BANNER from src/include/msgr.h. This is followed bythe server’s then client’s address each encoded as a entity_addr_t.

Once the client verifies that the servers banner matches its own it replies withits banner and its address.

Connect

Once the banners have been verified and the addresses exchanged the connectionnegotiation begins. First the client sends a ceph_msg_connect structurewith its information.

  1. // From src/include/msgr.h
  2. struct ceph_msg_connect {
  3.         u64le features;            // Supported features (CEPH_FEATURE_*)
  4.         u32le host_type;           // CEPH_ENTITY_TYPE_*
  5.         u32le global_seq;          // Number of connections initiated by this host.
  6.         u32le connect_seq;         // Number of connections initiated in this session.
  7.         u32le protocol_version;
  8.         u32le authorizer_protocol;
  9.         u32le authorizer_len;
  10.         u8    flags;               // CEPH_MSG_CONNECT_*
  11.         u8    authorizer[authorizer_len];
  12. }

Connect Reply

Once the connect has been sent the connection has effectively been opened,however the first message the server sends must be a connect reply message.

  1. struct ceph_msg_connect_reply {
  2.         u8    tag; // Tag indicating response code.
  3.         u64le features;
  4.         u32le global_seq;
  5.         u32le connect_seq;
  6.         u32le protocol_version;
  7.         u32le authorizer_len;
  8.         u8    flags;
  9.         u8    authorizer[authorizer_len];
  10. }

MSGR Protocol

This is a low level protocol over which messages are delivered. The messagesat this level consist of a tag byte, identifying the type of message, followedby the message data.

  1. // Virtual structure.
  2. struct {
  3.         u8 tag; // CEPH_MSGR_TAG_*
  4.         u8 data[]; // Length depends on tag and data.
  5. }

The length of data is determined by the tag byte and depending on themessage type via information in the data array itself.

Note

There is no way to determine the length of the message if you do notunderstand the type of message.

The message tags are defined in src/include/msgr.h and the current onesare listed below along with the data they include. Note that the definedstructures don’t exist in the source and are merely for representing theprotocol.

CEPH_MSGR_TAG_CLOSE (0x06)

  1. struct ceph_msgr_close {
  2.         u8 tag = 0x06;
  3.         u8 data[0]; // No data.
  4. }

The close message indicates that the connection is being closed.

CEPH_MSGR_TAG_MSG (0x07)

  1. struct ceph_msgr_msg {
  2.         u8 tag = 0x07;
  3.         ceph_msg_header header;
  4.         u8 front [header.front_len ];
  5.         u8 middle[header.middle_len];
  6.         u8 data  [header.data_len  ];
  7.         ceph_msg_footer footer;
  8. }
  9.  
  10. // From src/include/msgr.h
  11. struct ceph_msg_header {
  12.         u64le seq;       // Sequence number.
  13.         u64le tid;       // Transaction ID.
  14.         u16le type;      // Message type (CEPH_MSG_* or MSG_*).
  15.         u16le priority;  // Priority (higher is more important).
  16.         u16le version;   // Version of message encoding.
  17.  
  18.         u32le front_len;  // The size of the front section.
  19.         u32le middle_len; // The size of the middle section.
  20.         u32le data_len;   // The size of the data section.
  21.         u16le data_off;   // The way data should be aligned by the receiver.
  22.  
  23.         ceph_entity_name src; // Information about the sender.
  24.  
  25.         u16le compat_version; // Oldest compatible encoding version.
  26.         u16le reserved;       // Unused.
  27.         u32le crc;            // CRC of header.
  28. }
  29.  
  30. // From src/include/msgr.h
  31. struct ceph_msg_footer {
  32.         u32le front_crc;  // Checksums of the various sections.
  33.         u32le middle_crc; //
  34.         u32le data_crc;   //
  35.         u64le sig; // Crypographic signature.
  36.         u8    flags;
  37. }

Messages are the business logic of Ceph. They are what is used to send data andrequests between nodes. The message header contains the length of the messageso unknown messages can be handled gracefully.

There are two names for the message type constants CEPHMSG and MSG_.The only difference between the two is that the first are considered “public”while the second is for internal use only. There is no protocol-leveldifference.

CEPH_MSGR_TAG_ACK (0x08)

  1. struct ceph_msgr_ack {
  2.         u8    tag = 0x08;
  3.         u64le seq; // The sequence number of the message being acknowledged.
  4. }

CEPH_MSGR_TAG_KEEPALIVE (0x09)

  1. struct ceph_msgr_keepalive {
  2.         u8 tag = 0x09;
  3.         u8 data[0]; // No data.
  4. }

CEPH_MSGR_TAG_KEEPALIVE2 (0x0E)

  1. struct ceph_msgr_keepalive2 {
  2.         u8      tag = 0x0E;
  3.         utime_t timestamp;
  4. }

CEPH_MSGR_TAG_KEEPALIVE2_ACK (0x0F)

  1. struct ceph_msgr_keepalive2_ack {
  2.         u8      tag = 0x0F;
  3.         utime_t timestamp;
  4. }