A Detailed Documentation on How to Set up Ceph Kerberos Authentication

This document provides details on the Kerberos authorization protocol. This isthe 1st draft and we will try to keep it updated along with code changes thatmight take place.

Several free implementations of this protocol are available (MIT, Heimdal,MS…), covering a wide range of operating systems. The MassachusettsInstitute of Technology (MIT), where Kerberos was originally developed,continues to develop their Kerberos package and it is the implementation wechose to work with. MIT Kerberos.

Please, provide feedback to Daniel Oliveira (doliveira@suse.com)

Last update: Dec 3, 2018

Background

Before we get into Kerberos details, let us define a few terms so we canunderstand what to expect from it, what it can and can’t do:

  • Directory Services
  • A directory service is a customizable information store that functions asa single point from which users can locate resources and servicesdistributed throughout the network. This customizable information storealso gives administrators a single point for managing its objects and theirattributes. Although this information store appears as a single point tothe users of the network, it is actually most often stored in a distributedform. A directory service consists of at least one Directory Server and aDirectory Client and are implemented based on X.500 standards.

OpenLDAP, 389 Directory Server, MS Active Directory, NetIQ eDirectory aresome good examples.

A directory service is often characterized as a write-once-read-many-timesservice, meaning the data that would normally be stored in an directoryservice would not be expected to change on every access.

The database that forms a directory service is not designed fortransactional data.

  • LDAP (Lightweight Directory Access Protocol v3)
  • LDAP is a set of LDAP Protocol Exchanges (not an implementation of aserver) that defines the method by which data is accessed. LDAPv3 is astandard defined by the IETF in RFC 2251 and describes how data isrepresented in the Directory Service (the Data Model or DIT).

Finally, it defines how data is loaded into (imported) and saved from(exported) a directory service (using LDIF). LDAP does not define how datais stored or manipulated. Data Store is an ‘automagic’ process as far asthe standard is concerned and is generally handled by back-end modules.

No Directory Service implementation has all the features of LDAP v3protocol implemented. All Directory Server implementations have theirdifferent problems and/or anomalies, and features that may not returnresults as another Directory Server implementation would.

  • Authentication
  • Authentication is about validating credentials (like User Name/ID andpassword) to verify the identity. The system determines whether one is whatthey say they are using their credentials.

Usually, authentication is done by a username and password, and sometimesin conjunction with (single, two, or multi) factors of authentication,which refers to the various ways to be authenticated.

  • Authorization
  • Authorization occurs after the identity is successfully authenticated bythe system, which ultimately gives one full permission to access theresources such as information, files, databases, and so forth, almostanything. It determines the ability to access the system and up to whatextent (what kind of permissions/rights are given and to where/what).
  • Auditing
  • Auditing takes the results from both authentication and authorization andrecords them into an audit log. The audit log records records all actionstaking by/during the authentication and authorization for later review bythe administrators. While authentication and authorization are preventivesystems (in which unauthorized access is prevented), auditing is a reactivesystem (in which it gives detailed log of how/when/where someone accessedthe environment).
  • Kerberos (KRB v5)
  • Kerberos is a network authentication protocol. It is designed to providestrong authentication for client/server applications by using secret-keycryptography (symmetric key). A free implementation of this protocol isavailable from the MIT. However, Kerberos is available in many commercialproducts as well.

It was designed to provide secure authentication to services over aninsecure network. Kerberos uses tickets to authenticate a user, or serviceapplication and never transmits passwords over the network in the clear.So both client and server can prove their identity without sending anyunencrypted secrets over the network.

Kerberos can be used for single sign-on (SSO). The idea behind SSO issimple, we want to login just once and be able to use any service that weare entitled to, without having to login on each of those services.

  • Simple Authentication and Security Layer (SASL)
  • SASL (RFC 4422) is a framework that helps developers to implementdifferent authentication mechanisms (implementing a series of challengesand responses), allowing both clients and servers to negotiate a mutuallyacceptable mechanism for each connection, instead of hard-coding them.

Examples of SASL mechanisms:

  • ANONYMOUS (RFC 4505)

    • For guest access, meaning unauthenticated

  • CRAM-MD5 (RFC 2195)

    • Simple challenge-response scheme based on HMAC-MD5.It does not establish any security layer. Less secure thanDIGEST-MD5 and GSSAPI.

  • DIGEST-MD5 (RFC 2831)

    • HTTP Digest compatible (partially) challenge-response schemebased upon MD5, offering a data security layer. It is preferredover PLAIN text passwords, protecting against plain text attacks.It is a mandatory authentication method for LDAPv3 servers.

  • EXTERNAL (RFCs 4422, 5246, 4301, 2119)

    • Where authentication is implicit in the context (i.e; forprotocols using IPsec or TLS [TLS/SSL to performing certificate-based authentication] already). This method uses public keys forstrong authentication.

  • GS2 (RFC 5801)

    • Family of mechanisms supports arbitrary GSS-API mechanisms inSASL

  • NTLM (MS Proprietary)

    • MS Windows NT LAN Manager authentication mechanism

  • OAuth 1.0/2.0 (RFCs 5849, 6749, 7628)

    • Authentication protocol for delegated resource access

  • OTP (RFC 2444)

    • One-time password mechanism (obsoletes the SKEY mechanism)

  • PLAIN (RFC 4616)

    • Simple Cleartext password mechanism (RFC 4616). This is not apreferred mechanism for most applications because of its relativelack of strength.

  • SCRAM (RFCs 5802, 7677)

    • Modern challenge-response scheme based mechanism with channelbinding support

  • Generic Security Services Application Program Interface (GSSAPI)
  • GSSAPI (RFCs 2078, 2743, 2744, 4121, 4752) is widely used by protocolimplementers as a way to implement Kerberos v5 support in theirapplications. It provides a generic interface and message format that canencapsulate authentication exchanges from any authentication method thathas a GSSAPI-compliant library.

It does not define a protocol, authentication, or security mechanismitself; it instead makes it easier for application programmers to supportmultiple authentication mechanisms by providing a uniform, generic API forsecurity services. It is a set of functions that include both an API and amethodology for approaching authentication, aiming to insulate applicationprotocols from the specifics of security protocols as much as possible.

Microsoft Windows Kerberos implementation does not include GSSAPI supportbut instead includes a Microsoft-specific API, the Security SupportProvider Interface (SSPI). In Windows, an SSPI client can communicate witha GSSAPI server.

Most applications that support GSSAPI also support Kerberos v5.

  • Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO)
  • As we can see, GSSAPI solves the problem of providing a single API todifferent authentication mechanisms. However, it does not solve the problemof negotiating which mechanism to use. In fact for GSSAPI to work, the twoapplications communicating with each other must know in advance whatauthentication mechanism they plan to use, which usually is not a problemif only one mechanism is supported (meaning Kerberos v5).

However, if there are multiple mechanisms to choose from, a method isneeded to securely negotiate an authentication mechanism that is mutuallysupported between both client and server; which is whereSPNEGO (RFC 2478, 4178) makes a difference.

SPNEGO provides a framework for two parties that are engaged inauthentication to select from a set of possible authentication mechanisms,in a manner that preserves the opaque nature of the security protocols tothe application protocol that uses it.

It is a security protocol that uses a GSSAPI authentication mechanism andnegotiates among several available authentication mechanisms in animplementation, selecting one for use to satisfy the authentication needsof the application protocol.

It is a meta protocol that travels entirely in other applicationprotocols; it is never used directly without an application protocol.

Why is this important and why do we care? Like, at all?

Having this background information in mind, we can easily describe thingslike:

1. Ceph Kerberos authentication is based totally on MIT Kerberosimplementation using GSSAPI.

2. At the moment we are still using Kerberos default backenddatabase, however we plan on adding LDAP as a backend which wouldprovide us with authentication with GSSAPI (KRB5) and authorizationwith LDAP (LDAPv3), via SASL mechanism.

Before We Start

We assume the environment already has some external services up and runningproperly:

  • Kerberos needs to be properly configured, which also means (for bothevery server and KDC):

    • Time Synchronization (either using NTP or chrony).

      • Not only Kerberos, but also Ceph depends and relies on timesynchronization.

    • DNS resolution

      • Both (forward and reverse) zones, with fully qualified domainname (fqdn) (hostname + domain.name)

      • KDC discover can be set up to to use DNS (srv resources) asservice location protocol (RFCs 2052, 2782), as well as hostor domain to the appropriate realm (txt record).

      • Even though these DNS entries/settings are not required to run aKerberos realm, they certainly help to eliminate the need formanual configuration on all clients.

      • This is extremely important, once most of the Kerberos issues areusually related to name resolution. Kerberos is very picky whenchecking on systems names and host lookups.

  • Whenever possible, in order to avoid a single point of failure, set upa backup, secondary, or slave, for every piece/part in theinfrastructure (ntp, dns, and kdc servers).

Also, the following Kerberos terminology is important:

  • Ticket

    • Tickets or Credentials, are a set of information that can be used toverify the client’s identity. Kerberos tickets may be stored in afile, or they may exist only in memory.

    • The first ticket obtained is a ticket-granting ticket (TGT), whichallows the clients to obtain additional tickets. These additionaltickets give the client permission for specific services. Therequesting and granting of these additional tickets happenstransparently.

      • The TGT, which expires at a specified time, permits the client toobtain additional tickets, which give permission for specificservices. The requesting and granting of these additional ticketsis user-transparent.

  • Key Distribution Center (KDC).

    • The KDC creates a ticket-granting ticket (TGT) for the client,encrypts it using the client’s password as the key, and sends theencrypted TGT back to the client. The client then attempts to decryptthe TGT, using its password. If the client successfully decrypts theTGT (i.e., if the client gave the correct password), it keeps thedecrypted TGT, which indicates proof of the client’s identity.

    • The KDC is comprised of three components:

      • Kerberos database, which stores all the information about theprincipals and the realm they belong to, among other things.

      • Authentication service (AS)

      • Ticket-granting service (TGS)

  • Client

    • Either a user, host or a service who sends a request for a ticket.

  • Principal

    • It is a unique identity to which Kerberos can assign tickets.Principals can have an arbitrary number of components. Each componentis separated by a component separator, generally /. The lastcomponent is the realm, separated from the rest of the principal bythe realm separator, generally @.

    • If there is no realm component in the principal, then it will beassumed that the principal is in the default realm for the context inwhich it is being used.

    • Usually, a principal is divided into three parts:

      • The primary, the instance, and the realm

      • The format of a typical Kerberos V5 principal isprimary/instance@REALM.

      • The primary is the first part of the principal. In the caseof a user, it’s the same as the username. For a host, theprimary is the word host. For Ceph, will use ceph as aprimary name which makes it easier to organize and identify Cephrelated principals.

      • The instance is an optional string that qualifies theprimary. The instance is separated from the primary by a slash/. In the case of a user, the instance is usually null,but a user might also have an additional principal, with aninstance called admin, which one uses to administrate adatabase.

        The principal johndoe@MYDOMAIN.COM is completely separatefrom the principal johndoe/admin@MYDOMAIN.COM, with aseparate password, and separate permissions. In the case of ahost, the instance is the fully qualified hostname,i.e., osd1.MYDOMAIN.COM.

      • The realm is the Kerberos realm. Usually, the Kerberos realmis the domain name, in upper-case letters. For example, themachine osd1.MYDOMAIN.COM would be in the realmMYDOMAIN.COM.

  • Keytab

    • A keytab file stores the actual encryption key that can be used inlieu of a password challenge for a given principal. Creating keytabfiles are useful for noninteractive principals, such as ServicePrincipal Names, which are often associated with long-runningprocesses like Ceph daemons. A keytab file does not have to be a“1:1 mapping” to a single principal. Multiple different principalkeys can be stored in a single keytab file:

      • The keytab file allows a user/service to authenticate withoutknowledge of the password. Due to this, keytabs should beprotected with appropriate controls to prevent unauthorizedusers from authenticating with it.

      • The default client keytab file is /etc/krb5.keytab

The ‘Ceph side’ of the things

In order to configure connections (from Ceph nodes) to the KDC:

  • Login to the Kerberos client (Ceph server nodes) and confirm it is properlyconfigured, by checking and editing /etc/krb5.conf file properly:
  1. /etc/krb5.conf
  2. [libdefaults]
  3.     dns_canonicalize_hostname = false
  4.     rdns = false
  5.     forwardable = true
  6.     dns_lookup_realm = true
  7.     dns_lookup_kdc = true
  8.     allow_weak_crypto = false
  9.     default_realm = MYDOMAIN.COM
  10.     default_ccache_name = KEYRING:persistent:%{uid}
  11. [realms]
  12.     MYDOMAIN.COM = {
  13.         kdc = kerberos.mydomain.com
  14.         admin_server = kerberos.mydomain.com
  15.         ...
  16.     }
  17. ...
  • Login to the KDC Server and confirm it is properly configured toauthenticate to the Kerberos realm in question:

  1. Kerberos related DNS RRs:

    1. /var/lib/named/master/mydomain.comkerberos                IN A        192.168.10.21kerberos-slave          IN A        192.168.10.22_kerberos               IN TXT      "MYDOMAIN.COM"_kerberos._udp          IN SRV      1 0 88 kerberos_kerberos._tcp          IN SRV      1 0 88 kerberos_kerberos._udp          IN SRV      20 0 88 kerberos-slave_kerberos-master._udp   IN SRV      0 0 88 kerberos_kerberos-adm._tcp      IN SRV      0 0 749 kerberos_kpasswd._udp           IN SRV      0 0 464 kerberos…

  2. KDC configuration file:

    1. /var/lib/kerberos/krb5kdc/kdc.conf[kdcdefaults]        kdc_ports = 750,88[realms]        MYDOMAIN.COM = {            acl_file = /var/lib/kerberos/krb5kdc/kadm5.acl            admin_keytab = FILE:/var/lib/kerberos/krb5kdc/kadm5.keytab            default_principal_flags = +postdateable +forwardable +renewable +proxiable                                                    +dup-skey -preauth -hwauth +service                                                    +tgt-based +allow-tickets -pwchange                                                    -pwservice            dict_file = /var/lib/kerberos/krb5kdc/kadm5.dict            key_stash_file = /var/lib/kerberos/krb5kdc/.k5.MYDOMAIN.COM            kdc_ports = 750,88            max_life = 0d 10h 0m 0s            max_renewable_life = 7d 0h 0m 0s        }…
  • Still on the KDC Server, run the Kerberos administration utility;kadmin.local so we can list all the principals already created.
  1. kadmin.local:  listprincs
  2. K/M@MYDOMAIN.COM
  3. krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
  4. kadmin/admin@MYDOMAIN.COM
  5. kadmin/changepw@MYDOMAIN.COM
  6. kadmin/history@MYDOMAIN.COM
  7. kadmin/kerberos.mydomain.com@MYDOMAIN.COM
  8. root/admin@MYDOMAIN.COM
  9. ...
  • Add a principal for each Ceph cluster node we want to be authenticated byKerberos:

  1. Adding principals:

    1. kadmin.local:  addprinc -randkey ceph/ceph-mon1Principal "ceph/ceph-mon1@MYDOMAIN.COM" created.kadmin.local:  addprinc -randkey ceph/ceph-osd1Principal "ceph/ceph-osd1@MYDOMAIN.COM" created.kadmin.local:  addprinc -randkey ceph/ceph-osd2Principal "ceph/ceph-osd2@MYDOMAIN.COM" created.kadmin.local:  addprinc -randkey ceph/ceph-osd3Principal "ceph/ceph-osd3@MYDOMAIN.COM" created.kadmin.local:  addprinc -randkey ceph/ceph-osd4Principal "ceph/ceph-osd4@MYDOMAIN.COM" created.kadmin.local:  listprincsK/M@MYDOMAIN.COMkrbtgt/MYDOMAIN.COM@MYDOMAIN.COMkadmin/admin@MYDOMAIN.COMkadmin/changepw@MYDOMAIN.COMkadmin/history@MYDOMAIN.COMkadmin/kerberos.mydomain.com@MYDOMAIN.COMroot/admin@MYDOMAIN.COMceph/ceph-mon1@MYDOMAIN.COMceph/ceph-osd1@MYDOMAIN.COMceph/ceph-osd2@MYDOMAIN.COMceph/ceph-osd3@MYDOMAIN.COMceph/ceph-osd4@MYDOMAIN.COM…

  2. This follows the same idea if we are creating a user principal

    1. kadmin.local:  addprinc johndoeWARNING: no policy specified for johndoe@MYDOMAIN.COM; defaulting to no policyEnter password for principal "johndoe@MYDOMAIN.COM":Re-enter password for principal "johndoe@MYDOMAIN.COM":Principal "johndoe@MYDOMAIN.COM" created.…
  • Create a keytab file for each Ceph cluster node:

As the default client keytab file is /etc/krb5.keytab, we will want touse a different file name, so we especify which keytab file to create andwhich principal to export keys from:

  1. kadmin.local:  ktadd -k /etc/gssclient_mon1.ktab ceph/ceph-mon1Entry for principal ceph/ceph-mon1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_mon1.ktab.Entry for principal ceph/ceph-mon1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_mon1.ktab.Entry for principal ceph/ceph-mon1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/gss_client_mon1.ktab.Entry for principal ceph/ceph-mon1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/gss_client_mon1.ktab.kadmin.local:  ktadd -k /etc/gss_client_osd1.ktab ceph/ceph-osd1Entry for principal ceph/ceph-osd1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd1.ktab.Entry for principal ceph/ceph-osd1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd1.ktab.Entry for principal ceph/ceph-osd1 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/gss_client_osd1.ktab.Entry for principal ceph/ceph-osd1 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/gss_client_osd1.ktab.kadmin.local:  ktadd -k /etc/gss_client_osd2.ktab ceph/ceph-osd2Entry for principal ceph/ceph-osd2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd2.ktab.Entry for principal ceph/ceph-osd2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd2.ktab.Entry for principal ceph/ceph-osd2 with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/gss_client_osd2.ktab.Entry for principal ceph/ceph-osd2 with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/gss_client_osd2.ktab.kadmin.local:  ktadd -k /etc/gss_client_osd3.ktab ceph/ceph-osd3Entry for principal ceph/ceph-osd3 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd3.ktab.Entry for principal ceph/ceph-osd3 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd3.ktab.Entry for principal ceph/ceph-osd3 with kvno 3, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/gss_client_osd3.ktab.Entry for principal ceph/ceph-osd3 with kvno 3, encryption type arcfour-hmac added to keytab WRFILE:/etc/gss_client_osd3.ktab.kadmin.local:  ktadd -k /etc/gss_client_osd4.ktab ceph/ceph-osd4Entry for principal ceph/ceph-osd4 with kvno 4, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd4.ktab.Entry for principal ceph/ceph-osd4 with kvno 4, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/gss_client_osd4.ktab.Entry for principal ceph/ceph-osd4 with kvno 4, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/gss_client_osd4.ktab.Entry for principal ceph/ceph-osd4 with kvno 4, encryption type arcfour-hmac added to keytab WRFILE:/etc/gss_client_osd4.ktab.# ls -1 /etc/gss_client*/etc/gss_client_mon1.ktab/etc/gss_client_osd1.ktab/etc/gss_client_osd2.ktab/etc/gss_client_osd3.ktab/etc/gss_client_osd4.ktab

We can also check these newly created keytab client files by:

  1. # klist -kte /etc/gss_client_mon1.ktabKeytab name: FILE:/etc/gss_client_mon1.ktabKVNO Timestamp           Principal—— —————————- ———————————————————————————   2 10/8/2018 14:35:30 ceph/ceph-mon1@MYDOMAIN.COM (aes256-cts-hmac-sha1-96)   2 10/8/2018 14:35:31 ceph/ceph-mon1@MYDOMAIN.COM (aes128-cts-hmac-sha1-96)   2 10/8/2018 14:35:31 ceph/ceph-mon1@MYDOMAIN.COM (des3-cbc-sha1)   2 10/8/2018 14:35:31 ceph/ceph-mon1@MYDOMAIN.COM (arcfour-hmac)…
  • A new set parameter was added in Ceph, gss ktab client file whichpoints to the keytab file related to the Ceph node (or principal) inquestion.

By default it points to /var/lib/ceph/$name/gssclient$name.ktab. So,in the case of a Ceph server osd1.mydomain.com, the location and nameof the keytab file should be: /var/lib/ceph/osd1/gssclient_osd1.ktab

Therefore, we need to scp each of these newly created keytab files fromthe KDC to their respective Ceph cluster nodes (i.e):# for node in mon1 osd1 osd2 osd3 osd4; do scp /etc/gss_client$node*.ktab root@ceph-$node:/var/lib/ceph/$node/; done

Or whatever other way one feels comfortable with, as long as each keytabclient file gets copied over to the proper location.

At this point, even without using any keytab client file we should bealready able to authenticate a user principal:

  1. # kdestroy -A && kinit -f johndoe && klist -fPassword for johndoe@MYDOMAIN.COM:Ticket cache: KEYRING:persistent:0:0Default principal: johndoe@MYDOMAIN.COMValid starting       Expires              Service principal10/10/2018 15:32:01  10/11/2018 07:32:01  krbtgt/MYDOMAIN.COM@MYDOMAIN.COM    renew until 10/11/2018 15:32:01, Flags: FRI…

Given that the keytab client file is/should already be copied and available at theKerberos client (Ceph cluster node), we should be able to athenticate using it beforegoing forward:

  1. # kdestroy -A && kinit -k -t /etc/gss_client_mon1.ktab -f 'ceph/ceph-mon1@MYDOMAIN.COM' && klist -fTicket cache: KEYRING:persistent:0:0Default principal: ceph/ceph-mon1@MYDOMAIN.COMValid starting       Expires              Service principal10/10/2018 15:54:25  10/11/2018 07:54:25  krbtgt/MYDOMAIN.COM@MYDOMAIN.COM    renew until 10/11/2018 15:54:25, Flags: FRI…
  • The default client keytab is used, if it is present and readable, toautomatically obtain initial credentials for GSSAPI client applications. Theprincipal name of the first entry in the client keytab is used by defaultwhen obtaining initial credentials:

  1. The KRB5_CLIENT_KTNAME environment variable.

  2. The default_client_keytab_name profile variable in [libdefaults].

  3. The hardcoded default, DEFCKTNAME.

So, what we do is to internally, set the environment variableKRB5_CLIENT_KTNAME to the same location as gss_ktab_client_file,so /var/lib/ceph/osd1/gss_client_osd1.ktab, and change the ceph.conffile to add the new authentication method.

  1. /etc/ceph/ceph.conf[global]    …    auth cluster required = gss    auth service required = gss    auth client required = gss    gss ktab client file = /{$my_new_location}/{$my_new_ktab_client_file.keytab}    …
  • With that the GSSAPIs will then be able to read the keytab file and usingthe process of name and service resolution (provided by the DNS), able torequest a TGT as follows:

  1. User/Client sends principal identity and credentials to the KDC Server(TGT request).

  2. KDC checks its internal database for the principal in question.

  3. a TGT is created and wrapped by the KDC, using the principal’s key(TGT + Key).

  4. The newly created TGT, is decrypted and stored in the credentialscache.

  5. At this point, Kerberos/GSSAPI aware applications (and/or services) areable to check the list of active TGT in the keytab file.

For Ceph Developers Only

We certainly could have used straight native KRB5 APIs (instead ofGSSAPIs), but we wanted a more portable option as regards network security,which is the hallmark of the GSS (Generic Security Standard) -API.It does not actually provide security services itself.

Rather, it is a framework that provides security services to callers in ageneric way.

  1. +---------------------------------+
  2. |        Application              |
  3. +---------------------------------+
  4. | Protocol (RPC, Etc. [Optional]) |
  5. +---------------------------------+
  6. |         GSS-API                 |
  7. +---------------------------------+
  8. |   Security Mechs (Krb v5, Etc)  |
  9. +---------------------------------+

The GSS-API does two main things:

  1. It creates a security context in which data can be passed betweenapplications. A context can be thought of as a sort of “state of trust”between two applications.

    Applications that share a context know who each other are and thus canpermit data transfers between them as long as the context lasts.

  2. It applies one or more types of protection, known as “security services”,to the data to be transmitted.

GSS-API provides several types of portability for applications:

  1. Mechanism independence. GSS-API provides a generic interface to themechanisms for which it has been implemented. By specifying a defaultsecurity mechanism, an application does not need to know which mechanismit is using (for example, Kerberos v5), or even what type of mechanismit uses. As an example, when an application forwards a user’s credentialto a server, it does not need to know if that credential has a Kerberosformat or the format used by some other mechanism, nor how thecredentials are stored by the mechanism and accessed by the application.(If necessary, an application can specify a particular mechanism to use)

  2. Protocol independence. The GSS-API is independent of anycommunications protocol or protocol suite. It can be used withapplications that use, for example, sockets, RCP, or TCP/IP.RPCSEC_GSS “RPCSEC_GSS Layer” is an additional layer that smoothlyintegrates GSS-API with RPC.

  3. Platform independence. The GSS-API is completely oblivious to thetype of operating system on which an application is running.

  4. Quality of Protection independence. Quality of Protection (QOP) isthe name given to the type of algorithm used in encrypting data orgenerating cryptographic tags; the GSS-API allows a programmer to ignoreQOP, using a default provided by the GSS-API.(On the other hand, an application can specify the QOP if necessary.)

The basic security offered by the GSS-API is authentication. Authenticationis the verification of an identity: if you are authenticated, it meansthat you are recognized to be who you say you are.

The GSS-API provides for two additional security services, if supported by theunderlying mechanisms:

  1. Integrity: It’s not always sufficient to know that an applicationsending you data is who it claims to be. The data itself could havebecome corrupted or compromised.

    The GSS-API provides for data to be accompanied by a cryptographic tag,known as an Message Integrity Code (MIC), to prove that the datathat arrives at your doorstep is the same as the data that the sendertransmitted. This verification of the data’s validity is known as“integrity”.

  2. Confidentiality: Both authentication and integrity, however, leavethe data itself alone, so if it’s somehow intercepted, others can readit.

    The GSS-API therefore allows data to be encrypted, if underlyingmechanisms support it. This encryption of data is known as “confidentiality”.

Mechanisms Available With GSS-API:

The current implementation of the GSS-API works only with the Kerberos v5 securitymechanism.

  1. Mechanism Name          Object Identifier       Shared Library  Kernel Module———————————  ———————————  ———————  ———————diffie_hellman_640_0    1.3.6.4.1.42.2.26.2.4   dh640-0.so.1diffie_hellman_1024_0   1.3.6.4.1.42.2.26.2.5   dh1024-0.so.1SPNEGO                  1.3.6.1.5.5.2iakerb                  1.3.6.1.5.2.5SCRAM-SHA-1             1.3.6.1.5.5.14SCRAM-SHA-256           1.3.6.1.5.5.18GSS-EAP (arc)           1.3.6.1.5.5.15.1.1.*kerberos_v5             1.2.840.113554.1.2.2    gl/mech_krb5.so gl_kmech_krb5Therefore:    Kerberos Version 5 GSS-API Mechanism    OID {1.2.840.113554.1.2.2}    Kerberos Version 5 GSS-API Mechanism    Simple and Protected GSS-API Negotiation Mechanism    OID {1.3.6.1.5.5.2}

There are two different formats:

  1. The first, { 1 2 3 4 }, is officially mandated by the GSS-APIspecs. gss_str_to_oid() expects this first format.

  2. The second, 1.2.3.4, is more widely used but is not an officialstandard format.

Although the GSS-API makes protecting data simple, it does not do certainthings, in order to maximize its generic nature. These include:

  1. Provide security credentials for a user or application. These mustbe provided by the underlying security mechanism(s). The GSS-APIdoes allow applications to acquire credentials, either automaticallyor explicitly.

  2. Transfer data between applications. It is the application’sresponsibility to handle the transfer of all data between peers,whether it is security-related or “plain” data.

  3. Distinguish between different types of transmitted data (forexample, to know or determine that a data packet is plain data andnot GSS-API related).

  4. Indicate status due to remote (asynchronous) errors.

  5. Automatically protect information sent between processes of amultiprocess program.

  6. Allocate string buffers (“Strings and Similar Data”) to be passed toGSS-API functions.

  7. Deallocate GSS-API data spaces. These must be explicitly deallocatedwith functions such as gss_release_buffer() andgss_delete_name().

These are the basic steps in using the GSS-API:

  1. Each application, sender and recipient, acquires credentials explicitly,if credentials have not been acquired automatically.

  2. The sender initiates a security context and the recipient accepts it.

  3. The sender applies security protection to the message (data) it wants totransmit. This means that it either encrypts the message or stamps itwith an identification tag. The sender transmits the protected message.(The sender can choose not to apply either security protection, in whichcase the message has only the default GSS-API security serviceassociated with it. That is authentication, in which the recipient knowsthat the sender is who it claims to be.)

  4. The recipient decrypts the message (if needed) and verifies it(if appropriate).

  5. (Optional) The recipient returns an identification tag to the sender forconfirmation.

  6. Both applications destroy the shared security context. If necessary,they can also deallocate any “leftover” GSS-API data.

Applications that use the GSS-API should include the file gssapi.h.

Good References:

Kerberos Server Setup

First and foremost, this is not a recommendation for a productionenvironment. We are not covering Master/Slave replication cluster oranything production environment related (ntp/chrony, dns, pam/nss, sssd, etc).

Also, on the server side there might be different dependencies and/orconfiguration steps needed, depending on which backend database will be used.LDAP as a backend database is a good example of that.

On the client side there are different steps depending on which client backendconfiguration will be used. For example PAM/NSS or SSSD (along withLDAP for identity service, [and Kerberos for authentication service]) which isthe best suited option for joining MS Active Directory domains, and doingUser Logon Management.

By no means we intend to cover every possible scenario/combination here. Thesesteps are for a simple get a (MIT) Kerberos Server up and running.

Please, note that rpm packages might have slightly different names, as wellas the locations for the binaries and/or configuration files, depending onwhich Linux distro we are referring to.

Finally, keep in mind that some Linux distros will have their own wizards,which can perform the basic needed configuration:

  1. SUSE:
  2.     Kerberos server:
  3.         yast2 auth-server
  4.  
  5.     Kerberos client:
  6.         pam/nss: yast2 ldapkrb
  7.         sssd: yast2 auth-client

However, we are going through the manual configuration.

In order to get a new MIT KDC Server running:

  • Install the KDC server by:

  1. Install the needed packages:

    1. SUSE: zypper install krb5 krb5-server krb5-client    Additionally:        for development: krb5-devel        if using 'sssd': sssd-krb5 sssd-krb5-commonREDHAT: yum install krb5-server krb5-libs krb5-workstation    Additionally: 'Needs to be checked'

  2. Edit the KDC Server configuration file:

    1. /var/lib/kerberos/krb5kdc/kdc.conf[kdcdefaults]        kdc_ports = 750,88[realms]        MYDOMAIN.COM = {            acl_file = /var/lib/kerberos/krb5kdc/kadm5.acl            admin_keytab = FILE:/var/lib/kerberos/krb5kdc/kadm5.keytab            default_principal_flags = +postdateable +forwardable +renewable +proxiable                                                    +dup-skey -preauth -hwauth +service                                                    +tgt-based +allow-tickets -pwchange                                                    -pwservice            dict_file = /var/lib/kerberos/krb5kdc/kadm5.dict            key_stash_file = /var/lib/kerberos/krb5kdc/.k5.MYDOMAIN.COM            kdc_ports = 750,88            max_life = 0d 10h 0m 0s            max_renewable_life = 7d 0h 0m 0s        }…

  3. Edit the Kerberos Client configuration file:

    1. /etc/krb5.conf[libdefaults]    dns_canonicalize_hostname = false    rdns = false    forwardable = true    dns_lookup_realm = true     //—> if using DNS/DNSMasq    dns_lookup_kdc = true       //—> if using DNS/DNSMasq    allow_weak_crypto = false    default_realm = MYDOMAIN.COM    default_ccache_name = KEYRING:persistent:%{uid}[realms]    MYDOMAIN.COM = {        kdc = kerberos.mydomain.com        admin_server = kerberos.mydomain.com        …    }…
  • Create the Kerberos database:
  1. SUSE: kdb5_util create -s
  2.  
  3. REDHAT: kdb5_util create -s
  • Enable and Start both ‘KDC and KDC admin’ servers:
  1. SUSE: systemctl enable/start krb5kdc
  2.       systemctl enable/start kadmind
  3.  
  4. REDHAT: systemctl enable/start krb5kdc
  5.         systemctl enable/start kadmin
    • Create a Kerberos Administrator

    • Kerberos principals can be created either locally on the KDC server itselfor through the network, using an ‘admin principal’. On the KDC server,using kadmin.local:

      • List the existing principals:
  1. kadmin.local:  listprincs
  2. K/M@MYDOMAIN.COM
  3. krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
  4. kadmin/admin@MYDOMAIN.COM
  5. kadmin/changepw@MYDOMAIN.COM
  6. kadmin/history@MYDOMAIN.COM
  7. kadmin/kerberos.mydomain.com@MYDOMAIN.COM
  8. root/admin@MYDOMAIN.COM
  9. ...

b. In case we don’t have a built-in ‘admin principal’, we then create one(whatever principal name, we are using root, once by defaultkinit tries to authenticate using the same system login user name,unless a principal is passed as an argument kinit principal):

  1. # kadmin.local -q "addprinc root/admin"
  2. Authenticating as principal root/admin@MYDOMAIN.COM with password.
  3. WARNING: no policy specified for root/admin@MYDOMAIN.COM; defaulting to no policy
  4. Enter password for principal "root/admin@MYDOMAIN.COM":
  1. -

Confirm the newly created ‘admin principal’ has the needed permissionsin the KDC ACL (if ACLs are changed, kadmind needs to be restarted):

  1. SUSE: /var/lib/kerberos/krb5kdc/kadm5.acl
  2. REDHAT: /var/kerberos/krb5kdc/kadm5.acl
  3.  
  4. ###############################################################################
  5. #Kerberos_principal      permissions     [target_principal]      [restrictions]
  6. ###############################################################################
  7. #
  8. */admin@MYDOMAIN.COM     *
  1. -

Create a simple ‘user principal’ (same steps as by The ‘Ceph side’ ofthe things; 4a):

  1. kadmin.local:  addprinc johndoe
  2. WARNING: no policy specified for johndoe@MYDOMAIN.COM; defaulting to no policy
  3. Enter password for principal "johndoe@MYDOMAIN.COM":
  4. Re-enter password for principal "johndoe@MYDOMAIN.COM":
  5. Principal "johndoe@MYDOMAIN.COM" created.
  1. -

Confirm the newly created ‘user principal’ is able to authenticate (samesteps as by The ‘Ceph side’ of the things; 6):

  1. # kdestroy -A && kinit -f johndoe && klist -f
  2. Password for johndoe@MYDOMAIN.COM:
  3. Ticket cache: KEYRING:persistent:0:0
  4. Default principal: johndoe@MYDOMAIN.COM
  5.  
  6. Valid starting       Expires              Service principal
  7. 11/16/2018 13:11:16  11/16/2018 23:11:16  krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
  8.         renew until 11/17/2018 13:11:16, Flags: FRI
  9. ...
  • At this point, we should have a simple (MIT) Kerberos Server up and running:

  1. Considering we will want to work with keytab files, for both ‘user andservice’ principals, refer to The ‘Ceph side’ of the things startingat step 4.

  2. Make sure you are comfortable with following and their manpages:

    1. krb5.conf       -> Krb client config filekdc.conf        -> KDC server config filekrb5kdc         -> KDC server daemonkadmind         -> KDC administration daemonkadmin          -> Krb administration toolkdb5_util       -> Krb low-level database administration toolkinit           -> Obtain and cache Kerberos ticket-granting ticket toolklist           -> List cached Kerberos tickets toolkdestroy        -> Destroy Kerberos tickets tool
    • Name Resolution

    • As mentioned earlier, Kerberos relies heavly on name resolution. Most ofthe Kerberos issues are usually related to name resolution, since Kerberosis very picky on both systems names and host lookups.

      • As described in The ‘Ceph side’ of the things; step 2a, DNS RRsgreatly improves service location and host/domain resolution, by using(srv resources) and (txt record) respectively (as perBefore We Start; DNS resolution).
  1. /var/lib/named/master/mydomain.com
  2. kerberos                IN A        192.168.10.21
  3. kerberos-slave          IN A        192.168.10.22
  4. _kerberos               IN TXT      "MYDOMAIN.COM"
  5. _kerberos._udp          IN SRV      1 0 88 kerberos
  6. _kerberos._tcp          IN SRV      1 0 88 kerberos
  7. _kerberos._udp          IN SRV      20 0 88 kerberos-slave
  8. _kerberos-master._udp   IN SRV      0 0 88 kerberos
  9. _kerberos-adm._tcp      IN SRV      0 0 749 kerberos
  10. _kpasswd._udp           IN SRV      0 0 464 kerberos
  11. ...
  1. -

For a small network or development environment, where a DNS server isnot available, we have the option to use DNSMasq, anease-to-configure lightweight DNS server (along with some othercapabilities).

These records can be added to /etc/dnsmasq.conf (in addition to theneeded ‘host records’):

  1. /etc/dnsmasq.conf
  2. ...
  3. txt-record=_kerberos.mydomain.com,"MYDOMAIN.COM"
  4. srv-host=_kerberos._udp.mydomain.com,kerberos.mydomain.com,88,1
  5. srv-host=_kerberos._udp.mydomain.com,kerberos-2.mydomain.com,88,20
  6. srv-host=_kerberos-master._udp.mydomain.com,kerberos.mydomain.com,88,0
  7. srv-host=_kerberos-adm._tcp.mydomain.com,kerberos.mydomain.com,749,0
  8. srv-host=_kpasswd._udp.mydomain.com,kerberos.mydomain.com,464,0
  9. srv-host=_kerberos._tcp.mydomain.com,kerberos.mydomain.com,88,1
  10. ...
  1. -

After ‘b)’ is all set, and dnsmasq service up and running, we cantest it using:

  1. # nslookup kerberos
  2. Server:     192.168.10.1
  3. Address:    192.168.10.1#53
  4.  
  5. Name:   kerberos.mydomain.com
  6. Address: 192.168.10.21
  7.  
  8. # host -t SRV _kerberos._tcp.mydomain.com
  9. _kerberos._tcp.mydomain.com has SRV record 1 0 88 kerberos.mydomain.com.
  10.  
  11. # host -t SRV {each srv-host record}
  12. # host -t TXT _kerberos.mydomain.com
  13. _kerberos.mydomain.com descriptive text "MYDOMAIN.COM"
  14. ...
  1. -

As long as name resolution is working properly, either dnsmasqor named, Kerberos should be able to find the needed servicerecords.