Enabling the API Audit Log to Record System Events

You can enable the API audit log to record the sequence of system events initiated by individual users. You can know what happened, when it happened, who initiated it, and what cluster it affected. When you enable this feature, all requests to the Rancher API and all responses from it are written to a log.

You can enable API Auditing during Rancher installation or upgrade.

Enabling API Audit Log

The Audit Log is enabled and configured by passing environment variables to the Rancher server container. See the following to enable on your installation.

API Audit Log Options

The usage below defines rules about what the audit log should record and what data it should include:

ParameterDescription
AUDIT_LEVEL0 - Disable audit log (default setting).
1 - Log event metadata.
2 - Log event metadata and request body.
3 - Log event metadata, request body, and response body. Each log transaction for a request/response pair uses the same auditID value.

See Audit Level Logging for a table that displays what each setting logs.
AUDIT_LOG_PATHLog path for Rancher Server API. Default path is /var/log/auditlog/rancher-api-audit.log. You can mount the log directory to host.

Usage Example: AUDIT_LOG_PATH=/my/custom/path/
AUDIT_LOG_MAXAGEDefined the maximum number of days to retain old audit log files. Default is 10 days.
AUDIT_LOG_MAXBACKUPDefines the maximum number of audit log files to retain. Default is 10.
AUDIT_LOG_MAXSIZEDefines the maximum size in megabytes of the audit log file before it gets rotated. Default size is 100M.

Audit Log Levels

The following table displays what parts of API transactions are logged for each AUDIT_LEVEL setting.

AUDIT_LEVEL SettingRequest MetadataRequest BodyResponse MetadataResponse Body
0
1✓
2✓✓
3✓✓✓✓

Viewing API Audit Logs

Docker Install

Share the AUDIT_LOG_PATH directory (Default: /var/log/auditlog) with the host system. The log can be parsed by standard CLI tools or forwarded on to a log collection tool like Fluentd, Filebeat, Logstash, etc.

Kubernetes Install

Enabling the API Audit Log with the Helm chart install will create a rancher-audit-log sidecar container in the Rancher pod. This container will stream the log to standard output (stdout). You can view the log as you would any container log.

The rancher-audit-log container is part of the rancher pod in the cattle-system namespace.

CLI

  1. kubectl -n cattle-system logs -f rancher-84d886bdbb-s4s69 rancher-audit-log

Shipping the Audit Log

You can enable Rancher’s built in log collection and shipping for the cluster to ship the audit and other services logs to a supported collection endpoint. See Rancher Tools - Logging for details.

Audit Log Samples

After you enable auditing, each API request or response is logged by Rancher in the form of JSON. Each of the following code samples provide examples of how to identify each API transaction.

Metadata Level

If you set your AUDIT_LEVEL to 1, Rancher logs the metadata header for every API request, but not the body. The header provides basic information about the API transaction, such as the transaction’s ID, who initiated the transaction, the time it occurred, etc.

  1. {
  2.   "auditID": "30022177-9e2e-43d1-b0d0-06ef9d3db183",
  3.   "requestURI": "/v3/schemas",
  4.   "sourceIPs": ["::1"],
  5.   "user": {
  6.     "name": "user-f4tt2",
  7.     "group": ["system:authenticated"]
  8.   },
  9.   "verb": "GET",
  10.   "stage": "RequestReceived",
  11.   "stageTimestamp": "2018-07-20 10:22:43 +0800"
  12. }

Metadata and Request Body Level

If you set your AUDIT_LEVEL to 2, Rancher logs the metadata header and body for every API request.

The code sample below depicts an API request, with both its metadata header and body.

  1. {
  2.   "auditID": "ef1d249e-bfac-4fd0-a61f-cbdcad53b9bb",
  3.   "requestURI": "/v3/project/c-bcz5t:p-fdr4s/workloads/deployment:default:nginx",
  4.   "sourceIPs": ["::1"],
  5.   "user": {
  6.     "name": "user-f4tt2",
  7.     "group": ["system:authenticated"]
  8.   },
  9.   "verb": "PUT",
  10.   "stage": "RequestReceived",
  11.   "stageTimestamp": "2018-07-20 10:28:08 +0800",
  12.   "requestBody": {
  13.     "hostIPC": false,
  14.     "hostNetwork": false,
  15.     "hostPID": false,
  16.     "paused": false,
  17.     "annotations": {},
  18.     "baseType": "workload",
  19.     "containers": [
  20.       {
  21.         "allowPrivilegeEscalation": false,
  22.         "image": "nginx",
  23.         "imagePullPolicy": "Always",
  24.         "initContainer": false,
  25.         "name": "nginx",
  26.         "ports": [
  27.           {
  28.             "containerPort": 80,
  29.             "dnsName": "nginx-nodeport",
  30.             "kind": "NodePort",
  31.             "name": "80tcp01",
  32.             "protocol": "TCP",
  33.             "sourcePort": 0,
  34.             "type": "/v3/project/schemas/containerPort"
  35.           }
  36.         ],
  37.         "privileged": false,
  38.         "readOnly": false,
  39.         "resources": {
  40.           "type": "/v3/project/schemas/resourceRequirements",
  41.           "requests": {},
  42.           "limits": {}
  43.         },
  44.         "restartCount": 0,
  45.         "runAsNonRoot": false,
  46.         "stdin": true,
  47.         "stdinOnce": false,
  48.         "terminationMessagePath": "/dev/termination-log",
  49.         "terminationMessagePolicy": "File",
  50.         "tty": true,
  51.         "type": "/v3/project/schemas/container",
  52.         "environmentFrom": [],
  53.         "capAdd": [],
  54.         "capDrop": [],
  55.         "livenessProbe": null,
  56.         "volumeMounts": []
  57.       }
  58.     ],
  59.     "created": "2018-07-18T07:34:16Z",
  60.     "createdTS": 1531899256000,
  61.     "creatorId": null,
  62.     "deploymentConfig": {
  63.       "maxSurge": 1,
  64.       "maxUnavailable": 0,
  65.       "minReadySeconds": 0,
  66.       "progressDeadlineSeconds": 600,
  67.       "revisionHistoryLimit": 10,
  68.       "strategy": "RollingUpdate"
  69.     },
  70.     "deploymentStatus": {
  71.       "availableReplicas": 1,
  72.       "conditions": [
  73.         {
  74.           "lastTransitionTime": "2018-07-18T07:34:38Z",
  75.           "lastTransitionTimeTS": 1531899278000,
  76.           "lastUpdateTime": "2018-07-18T07:34:38Z",
  77.           "lastUpdateTimeTS": 1531899278000,
  78.           "message": "Deployment has minimum availability.",
  79.           "reason": "MinimumReplicasAvailable",
  80.           "status": "True",
  81.           "type": "Available"
  82.         },
  83.         {
  84.           "lastTransitionTime": "2018-07-18T07:34:16Z",
  85.           "lastTransitionTimeTS": 1531899256000,
  86.           "lastUpdateTime": "2018-07-18T07:34:38Z",
  87.           "lastUpdateTimeTS": 1531899278000,
  88.           "message": "ReplicaSet \"nginx-64d85666f9\" has successfully progressed.",
  89.           "reason": "NewReplicaSetAvailable",
  90.           "status": "True",
  91.           "type": "Progressing"
  92.         }
  93.       ],
  94.       "observedGeneration": 2,
  95.       "readyReplicas": 1,
  96.       "replicas": 1,
  97.       "type": "/v3/project/schemas/deploymentStatus",
  98.       "unavailableReplicas": 0,
  99.       "updatedReplicas": 1
  100.     },
  101.     "dnsPolicy": "ClusterFirst",
  102.     "id": "deployment:default:nginx",
  103.     "labels": {
  104.       "workload.user.cattle.io/workloadselector": "deployment-default-nginx"
  105.     },
  106.     "name": "nginx",
  107.     "namespaceId": "default",
  108.     "projectId": "c-bcz5t:p-fdr4s",
  109.     "publicEndpoints": [
  110.       {
  111.         "addresses": ["10.64.3.58"],
  112.         "allNodes": true,
  113.         "ingressId": null,
  114.         "nodeId": null,
  115.         "podId": null,
  116.         "port": 30917,
  117.         "protocol": "TCP",
  118.         "serviceId": "default:nginx-nodeport",
  119.         "type": "publicEndpoint"
  120.       }
  121.     ],
  122.     "restartPolicy": "Always",
  123.     "scale": 1,
  124.     "schedulerName": "default-scheduler",
  125.     "selector": {
  126.       "matchLabels": {
  127.         "workload.user.cattle.io/workloadselector": "deployment-default-nginx"
  128.       },
  129.       "type": "/v3/project/schemas/labelSelector"
  130.     },
  131.     "state": "active",
  132.     "terminationGracePeriodSeconds": 30,
  133.     "transitioning": "no",
  134.     "transitioningMessage": "",
  135.     "type": "deployment",
  136.     "uuid": "f998037d-8a5c-11e8-a4cf-0245a7ebb0fd",
  137.     "workloadAnnotations": {
  138.       "deployment.kubernetes.io/revision": "1",
  139.       "field.cattle.io/creatorId": "user-f4tt2"
  140.     },
  141.     "workloadLabels": {
  142.       "workload.user.cattle.io/workloadselector": "deployment-default-nginx"
  143.     },
  144.     "scheduling": {
  145.       "node": {}
  146.     },
  147.     "description": "my description",
  148.     "volumes": []
  149.   }
  150. }

Metadata, Request Body, and Response Body Level

If you set your AUDIT_LEVEL to 3, Rancher logs:

  • The metadata header and body for every API request.
  • The metadata header and body for every API response.

Request

The code sample below depicts an API request, with both its metadata header and body.

  1. {
  2.   "auditID": "a886fd9f-5d6b-4ae3-9a10-5bff8f3d68af",
  3.   "requestURI": "/v3/project/c-bcz5t:p-fdr4s/workloads/deployment:default:nginx",
  4.   "sourceIPs": ["::1"],
  5.   "user": {
  6.     "name": "user-f4tt2",
  7.     "group": ["system:authenticated"]
  8.   },
  9.   "verb": "PUT",
  10.   "stage": "RequestReceived",
  11.   "stageTimestamp": "2018-07-20 10:33:06 +0800",
  12.   "requestBody": {
  13.     "hostIPC": false,
  14.     "hostNetwork": false,
  15.     "hostPID": false,
  16.     "paused": false,
  17.     "annotations": {},
  18.     "baseType": "workload",
  19.     "containers": [
  20.       {
  21.         "allowPrivilegeEscalation": false,
  22.         "image": "nginx",
  23.         "imagePullPolicy": "Always",
  24.         "initContainer": false,
  25.         "name": "nginx",
  26.         "ports": [
  27.           {
  28.             "containerPort": 80,
  29.             "dnsName": "nginx-nodeport",
  30.             "kind": "NodePort",
  31.             "name": "80tcp01",
  32.             "protocol": "TCP",
  33.             "sourcePort": 0,
  34.             "type": "/v3/project/schemas/containerPort"
  35.           }
  36.         ],
  37.         "privileged": false,
  38.         "readOnly": false,
  39.         "resources": {
  40.           "type": "/v3/project/schemas/resourceRequirements",
  41.           "requests": {},
  42.           "limits": {}
  43.         },
  44.         "restartCount": 0,
  45.         "runAsNonRoot": false,
  46.         "stdin": true,
  47.         "stdinOnce": false,
  48.         "terminationMessagePath": "/dev/termination-log",
  49.         "terminationMessagePolicy": "File",
  50.         "tty": true,
  51.         "type": "/v3/project/schemas/container",
  52.         "environmentFrom": [],
  53.         "capAdd": [],
  54.         "capDrop": [],
  55.         "livenessProbe": null,
  56.         "volumeMounts": []
  57.       }
  58.     ],
  59.     "created": "2018-07-18T07:34:16Z",
  60.     "createdTS": 1531899256000,
  61.     "creatorId": null,
  62.     "deploymentConfig": {
  63.       "maxSurge": 1,
  64.       "maxUnavailable": 0,
  65.       "minReadySeconds": 0,
  66.       "progressDeadlineSeconds": 600,
  67.       "revisionHistoryLimit": 10,
  68.       "strategy": "RollingUpdate"
  69.     },
  70.     "deploymentStatus": {
  71.       "availableReplicas": 1,
  72.       "conditions": [
  73.         {
  74.           "lastTransitionTime": "2018-07-18T07:34:38Z",
  75.           "lastTransitionTimeTS": 1531899278000,
  76.           "lastUpdateTime": "2018-07-18T07:34:38Z",
  77.           "lastUpdateTimeTS": 1531899278000,
  78.           "message": "Deployment has minimum availability.",
  79.           "reason": "MinimumReplicasAvailable",
  80.           "status": "True",
  81.           "type": "Available"
  82.         },
  83.         {
  84.           "lastTransitionTime": "2018-07-18T07:34:16Z",
  85.           "lastTransitionTimeTS": 1531899256000,
  86.           "lastUpdateTime": "2018-07-18T07:34:38Z",
  87.           "lastUpdateTimeTS": 1531899278000,
  88.           "message": "ReplicaSet \"nginx-64d85666f9\" has successfully progressed.",
  89.           "reason": "NewReplicaSetAvailable",
  90.           "status": "True",
  91.           "type": "Progressing"
  92.         }
  93.       ],
  94.       "observedGeneration": 2,
  95.       "readyReplicas": 1,
  96.       "replicas": 1,
  97.       "type": "/v3/project/schemas/deploymentStatus",
  98.       "unavailableReplicas": 0,
  99.       "updatedReplicas": 1
  100.     },
  101.     "dnsPolicy": "ClusterFirst",
  102.     "id": "deployment:default:nginx",
  103.     "labels": {
  104.       "workload.user.cattle.io/workloadselector": "deployment-default-nginx"
  105.     },
  106.     "name": "nginx",
  107.     "namespaceId": "default",
  108.     "projectId": "c-bcz5t:p-fdr4s",
  109.     "publicEndpoints": [
  110.       {
  111.         "addresses": ["10.64.3.58"],
  112.         "allNodes": true,
  113.         "ingressId": null,
  114.         "nodeId": null,
  115.         "podId": null,
  116.         "port": 30917,
  117.         "protocol": "TCP",
  118.         "serviceId": "default:nginx-nodeport",
  119.         "type": "publicEndpoint"
  120.       }
  121.     ],
  122.     "restartPolicy": "Always",
  123.     "scale": 1,
  124.     "schedulerName": "default-scheduler",
  125.     "selector": {
  126.       "matchLabels": {
  127.         "workload.user.cattle.io/workloadselector": "deployment-default-nginx"
  128.       },
  129.       "type": "/v3/project/schemas/labelSelector"
  130.     },
  131.     "state": "active",
  132.     "terminationGracePeriodSeconds": 30,
  133.     "transitioning": "no",
  134.     "transitioningMessage": "",
  135.     "type": "deployment",
  136.     "uuid": "f998037d-8a5c-11e8-a4cf-0245a7ebb0fd",
  137.     "workloadAnnotations": {
  138.       "deployment.kubernetes.io/revision": "1",
  139.       "field.cattle.io/creatorId": "user-f4tt2"
  140.     },
  141.     "workloadLabels": {
  142.       "workload.user.cattle.io/workloadselector": "deployment-default-nginx"
  143.     },
  144.     "scheduling": {
  145.       "node": {}
  146.     },
  147.     "description": "my decript",
  148.     "volumes": []
  149.   }
  150. }

Response

The code sample below depicts an API response, with both its metadata header and body.

  1. {
  2.   "auditID": "a886fd9f-5d6b-4ae3-9a10-5bff8f3d68af",
  3.   "responseStatus": "200",
  4.   "stage": "ResponseComplete",
  5.   "stageTimestamp": "2018-07-20 10:33:06 +0800",
  6.   "responseBody": {
  7.     "actionLinks": {
  8.       "pause": "https://localhost:8443/v3/project/c-bcz5t:p-fdr4s/workloads/deployment:default:nginx?action=pause",
  9.       "resume": "https://localhost:8443/v3/project/c-bcz5t:p-fdr4s/workloads/deployment:default:nginx?action=resume",
  10.       "rollback": "https://localhost:8443/v3/project/c-bcz5t:p-fdr4s/workloads/deployment:default:nginx?action=rollback"
  11.     },
  12.     "annotations": {},
  13.     "baseType": "workload",
  14.     "containers": [
  15.       {
  16.         "allowPrivilegeEscalation": false,
  17.         "image": "nginx",
  18.         "imagePullPolicy": "Always",
  19.         "initContainer": false,
  20.         "name": "nginx",
  21.         "ports": [
  22.           {
  23.             "containerPort": 80,
  24.             "dnsName": "nginx-nodeport",
  25.             "kind": "NodePort",
  26.             "name": "80tcp01",
  27.             "protocol": "TCP",
  28.             "sourcePort": 0,
  29.             "type": "/v3/project/schemas/containerPort"
  30.           }
  31.         ],
  32.         "privileged": false,
  33.         "readOnly": false,
  34.         "resources": {
  35.           "type": "/v3/project/schemas/resourceRequirements"
  36.         },
  37.         "restartCount": 0,
  38.         "runAsNonRoot": false,
  39.         "stdin": true,
  40.         "stdinOnce": false,
  41.         "terminationMessagePath": "/dev/termination-log",
  42.         "terminationMessagePolicy": "File",
  43.         "tty": true,
  44.         "type": "/v3/project/schemas/container"
  45.       }
  46.     ],
  47.     "created": "2018-07-18T07:34:16Z",
  48.     "createdTS": 1531899256000,
  49.     "creatorId": null,
  50.     "deploymentConfig": {
  51.       "maxSurge": 1,
  52.       "maxUnavailable": 0,
  53.       "minReadySeconds": 0,
  54.       "progressDeadlineSeconds": 600,
  55.       "revisionHistoryLimit": 10,
  56.       "strategy": "RollingUpdate"
  57.     },
  58.     "deploymentStatus": {
  59.       "availableReplicas": 1,
  60.       "conditions": [
  61.         {
  62.           "lastTransitionTime": "2018-07-18T07:34:38Z",
  63.           "lastTransitionTimeTS": 1531899278000,
  64.           "lastUpdateTime": "2018-07-18T07:34:38Z",
  65.           "lastUpdateTimeTS": 1531899278000,
  66.           "message": "Deployment has minimum availability.",
  67.           "reason": "MinimumReplicasAvailable",
  68.           "status": "True",
  69.           "type": "Available"
  70.         },
  71.         {
  72.           "lastTransitionTime": "2018-07-18T07:34:16Z",
  73.           "lastTransitionTimeTS": 1531899256000,
  74.           "lastUpdateTime": "2018-07-18T07:34:38Z",
  75.           "lastUpdateTimeTS": 1531899278000,
  76.           "message": "ReplicaSet \"nginx-64d85666f9\" has successfully progressed.",
  77.           "reason": "NewReplicaSetAvailable",
  78.           "status": "True",
  79.           "type": "Progressing"
  80.         }
  81.       ],
  82.       "observedGeneration": 2,
  83.       "readyReplicas": 1,
  84.       "replicas": 1,
  85.       "type": "/v3/project/schemas/deploymentStatus",
  86.       "unavailableReplicas": 0,
  87.       "updatedReplicas": 1
  88.     },
  89.     "dnsPolicy": "ClusterFirst",
  90.     "hostIPC": false,
  91.     "hostNetwork": false,
  92.     "hostPID": false,
  93.     "id": "deployment:default:nginx",
  94.     "labels": {
  95.       "workload.user.cattle.io/workloadselector": "deployment-default-nginx"
  96.     },
  97.     "links": {
  98.       "remove": "https://localhost:8443/v3/project/c-bcz5t:p-fdr4s/workloads/deployment:default:nginx",
  99.       "revisions": "https://localhost:8443/v3/project/c-bcz5t:p-fdr4s/workloads/deployment:default:nginx/revisions",
  100.       "self": "https://localhost:8443/v3/project/c-bcz5t:p-fdr4s/workloads/deployment:default:nginx",
  101.       "update": "https://localhost:8443/v3/project/c-bcz5t:p-fdr4s/workloads/deployment:default:nginx",
  102.       "yaml": "https://localhost:8443/v3/project/c-bcz5t:p-fdr4s/workloads/deployment:default:nginx/yaml"
  103.     },
  104.     "name": "nginx",
  105.     "namespaceId": "default",
  106.     "paused": false,
  107.     "projectId": "c-bcz5t:p-fdr4s",
  108.     "publicEndpoints": [
  109.       {
  110.         "addresses": ["10.64.3.58"],
  111.         "allNodes": true,
  112.         "ingressId": null,
  113.         "nodeId": null,
  114.         "podId": null,
  115.         "port": 30917,
  116.         "protocol": "TCP",
  117.         "serviceId": "default:nginx-nodeport"
  118.       }
  119.     ],
  120.     "restartPolicy": "Always",
  121.     "scale": 1,
  122.     "schedulerName": "default-scheduler",
  123.     "selector": {
  124.       "matchLabels": {
  125.         "workload.user.cattle.io/workloadselector": "deployment-default-nginx"
  126.       },
  127.       "type": "/v3/project/schemas/labelSelector"
  128.     },
  129.     "state": "active",
  130.     "terminationGracePeriodSeconds": 30,
  131.     "transitioning": "no",
  132.     "transitioningMessage": "",
  133.     "type": "deployment",
  134.     "uuid": "f998037d-8a5c-11e8-a4cf-0245a7ebb0fd",
  135.     "workloadAnnotations": {
  136.       "deployment.kubernetes.io/revision": "1",
  137.       "field.cattle.io/creatorId": "user-f4tt2"
  138.     },
  139.     "workloadLabels": {
  140.       "workload.user.cattle.io/workloadselector": "deployment-default-nginx"
  141.     }
  142.   }
  143. }