Formatting Log Files

This section covers the creation of logging formats. All but a few logging output related settings in Traffic Server are performed in logging.yaml and consulting the documentation for that file is recommended in addition to this section. Any configurations or settings performed outside that file will be clearly noted.

Defining Formats

Logging formats in Traffic Server are defined by editing logging.yaml and adding new format entries for each format you wish to define. The syntax is fairly simple: every format must contain a Format attribute, which is the string defining the contents of each line in the log, and may also contain an optional Interval attribute defining the log aggregation interval for any logs which use the format (see Summary Logs for more information).

The return value from the format function is the log format object which may then be supplied to the appropriate log.* functions that define your logging destinations.

A very simple example, which contains only the timestamp of when the event began and the canonical URL of the request, would look like:

formats:
- name: myformat
  format: '%<cqtq> %<cauc>'

You may include as many custom field codes as you wish. The full list of codes available can be found in Log Fields. You may also include any literal characters in your format. For example, if we wished to separate the timestamp and canonical URL in our customer format above with a slash instead of a space, or even a slash surrounded by spaces, we could do so by just adding the desired characters to the format string:

formats:
- name: myformat
  format: '%<cqtq> / %<cauc>'

You may define as many custom formats as you wish. To apply changes to custom formats, you will need to run the command traffic_ctl config reload after saving your changes to logging.yaml.

Log Fields

The following sections detail all the available Traffic Server logging fields, broken down into the following broad categories for (hopefully) easier reference:

• Authentication
• Cache Details
• Connections and Transactions
• Content Types
• Hierarchical Proxies
• HTTP Headers
• HTTP Methods
• Identifiers
• Lengths and Sizes
• Log Collation
• Network Addresses, Ports, and Interfaces
• Plugin Details
• Protocols and Versions
• Request Details
• SSL / Encryption
• Status Codes
• TCP Details
• Timestamps and Durations
• URLs, Schemes, and Paths

Individual log fields are used within a log format string by enclosing them in angle brackets and prefixing with a percent sign. For example, to use the log field cqhl (the length in bytes of the client request headers), you would do the following:

Format = '%<cqhl>'

Literal characters may be used, but they must be outside the log fields’ placeholders, as so:

Format = 'Client Header Length (bytes): %<cqhl>'

You may combine many fields into a single format string (logs wouldn’t be very useful if you couldn’t). Some fields do require a little extra treatment, which is noted clearly in their descriptions below. This affects, primarily, those fields which provide access to HTTP header values as you need to specify which header’s value you wish to appear in the log data. For these, the header name is noted inside the angle brackets, before the log field name, and are enclosed within a curly braces pair. For example, to include the value of the Age header from an origin server response you would do:

Format = '%<{Age}ssh>'

Authentication

These log fields provide access to various details of a client or proxy’s means of request authentication to their destination (whether it be the client request to a proxy server, or the proxy server’s request to an origin).

Cache Details

These log fields reveal details of the Traffic Server proxy interaction with its own cache while attempting to service incoming client requests.

Connections and Transactions

The following log fields are used to list various details of connections and transactions between Traffic Server proxies and origin servers.

Content Types

Log fields used to reveal the HTTP content types in effect for transactions.

Error Code

The log fields of error code which is triggered session close or transaction close. The first byte of this field indicates that the error code is session level (S) or transaction level (T). When no error code is received or transmitted, these fields are -. For HTTP/2, error code are described in RFC 7540 section 7.

Hierarchical Proxies

The log fields detail aspects of transactions involving hierarchical caches.

HTTP Headers

The following log tables provide access to the values of specified HTTP headers from each phase of the transaction lifecycle. Unlike many of the other log fields, these require a little extra notation in the log format string, so that Traffic Server knows the individual HTTP header from which you aim to extract a value for the log entry.

This is done by specifying the name of the HTTP header in curly braces, just prior to the log field’s name, as so:

Format = '%<{User-agent}cqh>'

The above would insert the User Agent string from the client request headers into your log entry (or a blank string if no such header was present, or it did not contain a value).

Each of these also includes a URI-encoded variant, which replaces various characters in the string with entity encodings - rendering them safe for use in URL path components or query parameters. The variants’ names follow the pattern of the origin field named prefixed with e, as shown here:

HTTP Methods

These fields are used to log information about the HTTP methods/verbs used by requests.

Identifiers

Logging fields used to obtain various unique identifiers for transactions or objects.

Lengths and Sizes

These log fields are used to obtain various lengths and sizes of transaction components (headers, content bodies, etc.) between clients, proxies, and origins. Unless otherwise noted, all lengths are in bytes.

Log Collation

Logging fields related to Log Collation.

Note

Log collation is a deprecated feature as of ATS v8.0.0, and will be removed in ATS v9.0.0.

Network Addresses, Ports, and Interfaces

The following log fields are used to log details of the network (IP) addresses, incoming/outgoing ports, and network interfaces used during transactions.

Note

This can be misleading for cached documents. For example: if the first request was a cache miss and came from IP1 for server S and the second request for server S resolved to IP2 but came from the cache, then the log entry for the second request will show IP2.

Plugin Details

Logging fields which may be used to obtain details of plugins involved in the transaction.

Protocols and Versions

These logging fields may be used to determine which protocols and/or versions were in effect for a given event.

Request Details

The following logging fields are used to obtain the actual HTTP request details.

SSL / Encryption

Fields which expose the use, or lack thereof, of specific SSL and encryption features.

Status Codes

These log fields provide a variety of status codes, some numeric and some as strings, relating to client, proxy, and origin transactions.

TCP Details

The following logging fields reveal information about the TCP layer of client, proxy, and origin server connections.

Timestamps and Durations

The logging fields expose a variety of timing related information about client, proxy, and origin transactions. Variants of some of the fields provide timing resolution of the same underlying detail in milliseconds and seconds (both fractional and rounded-down integers). These variants are particularly useful in accommodating the emulation of other HTTP proxy softwares’ logging formats.

Other fields in this category provide variously formatted timestamps of particular events within the current transaction (e.g. the time at which a client request was received by Traffic Server).

Note

Logging fields for transaction milestones require specifying which of the milestones to use. Similar to how header logging fields are used, these log fields take the milestone name(s) in between curly braces, immediately before the logging field name, as so:

%<{Milestone field name}ms>
%<{Milestone field name1-Milestone field name2}msdms>

For more information on transaction milestones in Traffic Server, refer to the documentation on TSHttpTxnMilestoneGet().

URLs, Schemes, and Paths

These log fields allow capture of URLs, or components (such as schemes and paths), from transactions processed by Traffic Server.

Log Field Slicing

It is sometimes desirable to slice a log field to limit the length of a given log field’s output.

Log Field slicing can be specified as below:

%<field[start:end]>
%<{field}container[start:end]>

Omitting the slice notation defaults to the entire log field.

Slice notation only applies to a log field that is of type string and can not be applied to IPs or timestamp which are converted to string from integer.

The below slice specifiers are allowed.

[start:end]

Log field value from start through end-1

[start:]

Log field value from start through the rest of the string

[:end]

Log field value from the beginning through end-1

[:]

Default - entire Log field

Some examples below

'%<cqup>'       // the whole characters of <cqup>.
'%<cqup>[:]'    // the whole characters of <cqup>.
'%<cqup[0:30]>' // the first 30 characters of <cqup>.
'%<cqup[-10:]>' // the last 10 characters of <cqup>.
'%<cqup[:-5]>'  // everything except the last 5 characters of <cqup>.