Logging

Logging

A quick logging primer

Django uses Python’s builtin logging module to perform system logging.The usage of this module is discussed in detail in Python’s own documentation.However, if you’ve never used Python’s logging framework (or even if you have),here’s a quick primer.

The cast of players

A Python logging configuration consists of four parts:

Loggers

A logger is the entry point into the logging system. Each logger isa named bucket to which messages can be written for processing.

A logger is configured to have a log level. This log level describesthe severity of the messages that the logger will handle. Pythondefines the following log levels:

DEBUG: Low level system information for debugging purposes
INFO: General system information
WARNING: Information describing a minor problem that hasoccurred.
ERROR: Information describing a major problem that hasoccurred.
CRITICAL: Information describing a critical problem that hasoccurred.Each message that is written to the logger is a Log Record. Each logrecord also has a log level indicating the severity of that specificmessage. A log record can also contain useful metadata that describesthe event that is being logged. This can include details such as astack trace or an error code.

When a message is given to the logger, the log level of the message iscompared to the log level of the logger. If the log level of themessage meets or exceeds the log level of the logger itself, themessage will undergo further processing. If it doesn’t, the messagewill be ignored.

Once a logger has determined that a message needs to be processed,it is passed to a Handler.

Handlers

The handler is the engine that determines what happens to each messagein a logger. It describes a particular logging behavior, such aswriting a message to the screen, to a file, or to a network socket.

Like loggers, handlers also have a log level. If the log level of alog record doesn’t meet or exceed the level of the handler, thehandler will ignore the message.

A logger can have multiple handlers, and each handler can have adifferent log level. In this way, it is possible to provide differentforms of notification depending on the importance of a message. Forexample, you could install one handler that forwards ERROR andCRITICAL messages to a paging service, while a second handlerlogs all messages (including ERROR and CRITICAL messages) to afile for later analysis.

Filters

A filter is used to provide additional control over which log recordsare passed from logger to handler.

By default, any log message that meets log level requirements will behandled. However, by installing a filter, you can place additionalcriteria on the logging process. For example, you could install afilter that only allows ERROR messages from a particular source tobe emitted.

Filters can also be used to modify the logging record prior to beingemitted. For example, you could write a filter that downgradesERROR log records to WARNING records if a particular set ofcriteria are met.

Filters can be installed on loggers or on handlers; multiple filterscan be used in a chain to perform multiple filtering actions.

Formatters

Ultimately, a log record needs to be rendered as text. Formattersdescribe the exact format of that text. A formatter usually consistsof a Python formatting string containingLogRecord attributes; however,you can also write custom formatters to implement specific formatting behavior.

Using logging

Once you have configured your loggers, handlers, filters andformatters, you need to place logging calls into your code. Using thelogging framework works like this:

# import the logging library
import logging
 
# Get an instance of a logger
logger = logging.getLogger(__name__)
 
def my_view(request, arg1, arg):
    ...
    if bad_mojo:
        # Log an error message
        logger.error('Something went wrong!')

And that’s it! Every time the bad_mojo condition is activated, anerror log record will be written.

Naming loggers

The call to logging.getLogger() obtains (creating, ifnecessary) an instance of a logger. The logger instance is identifiedby a name. This name is used to identify the logger for configurationpurposes.

By convention, the logger name is usually name, the name ofthe Python module that contains the logger. This allows you to filterand handle logging calls on a per-module basis. However, if you havesome other way of organizing your logging messages, you can provideany dot-separated name to identify your logger:

# Get an instance of a specific named logger
logger = logging.getLogger('project.interesting.stuff')

The dotted paths of logger names define a hierarchy. Theproject.interesting logger is considered to be a parent of theproject.interesting.stuff logger; the project loggeris a parent of the project.interesting logger.

Why is the hierarchy important? Well, because loggers can be set topropagate their logging calls to their parents. In this way, you candefine a single set of handlers at the root of a logger tree, andcapture all logging calls in the subtree of loggers. A logging handlerdefined in the project namespace will catch all logging messagesissued on the project.interesting andproject.interesting.stuff loggers.

This propagation can be controlled on a per-logger basis. Ifyou don’t want a particular logger to propagate to its parents, youcan turn off this behavior.

Making logging calls

The logger instance contains an entry method for each of the defaultlog levels:

logger.debug()
logger.info()
logger.warning()
logger.error()
logger.critical()There are two other logging calls available:
logger.log(): Manually emits a logging message with aspecific log level.
logger.exception(): Creates an ERROR level loggingmessage wrapping the current exception stack frame.

Configuring logging

Of course, it isn’t enough to just put logging calls into your code.You also need to configure the loggers, handlers, filters andformatters to ensure that logging output is output in a useful way.

Python’s logging library provides several techniques to configurelogging, ranging from a programmatic interface to configuration files.By default, Django uses the dictConfig format.

In order to configure logging, you use LOGGING to define adictionary of logging settings. These settings describes the loggers,handlers, filters and formatters that you want in your logging setup,and the log levels and other properties that you want those componentsto have.

By default, the LOGGING setting is merged with Django’sdefault logging configuration using thefollowing scheme.

If the disable_existing_loggers key in the LOGGING dictConfig isset to True (which is the dictConfig default if the key is missing)then all loggers from the default configuration will be disabled. Disabledloggers are not the same as removed; the logger will still exist, but willsilently discard anything logged to it, not even propagating entries to aparent logger. Thus you should be very careful using'disable_existing_loggers': True; it’s probably not what you want. Instead,you can set disable_existing_loggers to False and redefine some or allof the default loggers; or you can set LOGGING_CONFIG to Noneand handle logging config yourself.

Logging is configured as part of the general Django setup() function.Therefore, you can be certain that loggers are always ready for use in yourproject code.

Examples

The full documentation for dictConfig formatis the best source of information about logging configuration dictionaries.However, to give you a taste of what is possible, here are several examples.

First, here’s a configuration which writes all logging from thedjango logger to a local file:

.. code-block:: python
  :caption: settings.py
 
  LOGGING = {
      'version': 1,
      'disable_existing_loggers': False,
      'handlers': {
          'file': {
              'level': 'DEBUG',
              'class': 'logging.FileHandler',
              'filename': '/path/to/django/debug.log',
          },
      },
      'loggers': {
          'django': {
              'handlers': ['file'],
              'level': 'DEBUG',
              'propagate': True,
          },
      },
  }

If you use this example, be sure to change the 'filename' path to alocation that’s writable by the user that’s running the Django application.

Second, here’s an example of how to make the logging system print Django’slogging to the console. It may be useful during local development.

By default, this config only sends messages of level INFO or higher to theconsole (same as Django’s default logging config, except that the default onlydisplays log records when DEBUG=True). Django does not log many suchmessages. With this config, however, you can also set the environment variableDJANGO_LOG_LEVEL=DEBUG to see all of Django’s debug logging which is veryverbose as it includes all database queries:

.. code-block:: python
  :caption: settings.py
 
  import os
 
  LOGGING = {
      'version': 1,
      'disable_existing_loggers': False,
      'handlers': {
          'console': {
              'class': 'logging.StreamHandler',
          },
      },
      'loggers': {
          'django': {
              'handlers': ['console'],
              'level': os.getenv('DJANGO_LOG_LEVEL', 'INFO'),
          },
      },
  }

Finally, here’s an example of a fairly complex logging setup:

.. code-block:: python
  :caption: settings.py
 
  LOGGING = {
      'version': 1,
      'disable_existing_loggers': False,
      'formatters': {
          'verbose': {
              'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
              'style': '{',
          },
          'simple': {
              'format': '{levelname} {message}',
              'style': '{',
          },
      },
      'filters': {
          'special': {
              '()': 'project.logging.SpecialFilter',
              'foo': 'bar',
          },
          'require_debug_true': {
              '()': 'django.utils.log.RequireDebugTrue',
          },
      },
      'handlers': {
          'console': {
              'level': 'INFO',
              'filters': ['require_debug_true'],
              'class': 'logging.StreamHandler',
              'formatter': 'simple'
          },
          'mail_admins': {
              'level': 'ERROR',
              'class': 'django.utils.log.AdminEmailHandler',
              'filters': ['special']
          }
      },
      'loggers': {
          'django': {
              'handlers': ['console'],
              'propagate': True,
          },
          'django.request': {
              'handlers': ['mail_admins'],
              'level': 'ERROR',
              'propagate': False,
          },
          'myproject.custom': {
              'handlers': ['console', 'mail_admins'],
              'level': 'INFO',
              'filters': ['special']
          }
      }
  }

This logging configuration does the following things:

Identifies the configuration as being in ‘dictConfig version 1’format. At present, this is the only dictConfig format version.
Defines two formatters:
- simple, that outputs the log level name (e.g., DEBUG) and the logmessage.

The format string is a normal Python formatting stringdescribing the details that are to be output on each loggingline. The full list of detail that can be output can befound in Formatter Objects.

verbose, that outputs the log level name, the logmessage, plus the time, process, thread and module thatgenerate the log message.

Defines two filters:
- project.logging.SpecialFilter, using the alias special. If thisfilter required additional arguments, they can be provided as additionalkeys in the filter configuration dictionary. In this case, the argumentfoo will be given a value of bar when instantiatingSpecialFilter.
- django.utils.log.RequireDebugTrue, which passes on records whenDEBUG is True.
Defines two handlers:
- console, a StreamHandler, which prints any INFO(or higher) message to sys.stderr. This handler uses the simpleoutput format.
- mail_admins, an AdminEmailHandler, which emails any ERROR(or higher) message to the site ADMINS. This handler uses thespecial filter.
Configures three loggers:
- django, which passes all messages to the console handler.
- django.request, which passes all ERROR messages tothe mailadmins handler. In addition, this logger ismarked to _not propagate messages. This means that logmessages written to django.request will not be handledby the django logger.
- myproject.custom, which passes all messages at INFOor higher that also pass the special filter to twohandlers – the console, and mail_admins. Thismeans that all INFO level messages (or higher) will beprinted to the console; ERROR and CRITICALmessages will also be output via email.

Custom logging configuration

If you don’t want to use Python’s dictConfig format to configure yourlogger, you can specify your own configuration scheme.

The LOGGING_CONFIG setting defines the callable that willbe used to configure Django’s loggers. By default, it points atPython’s logging.config.dictConfig() function. However, if you want touse a different configuration process, you can use any other callablethat takes a single argument. The contents of LOGGING willbe provided as the value of that argument when logging is configured.

Disabling logging configuration

If you don’t want to configure logging at all (or you want to manuallyconfigure logging using your own approach), you can setLOGGING_CONFIG to None. This will disable theconfiguration process for Django’s default logging. Here’s an example that disables Django’slogging configuration and then manually configures logging:

settings.py

LOGGING_CONFIG = None
 
import logging.config
logging.config.dictConfig(...)

Setting LOGGING_CONFIG to None only means that the automaticconfiguration process is disabled, not logging itself. If you disable theconfiguration process, Django will still make logging calls, falling back towhatever default logging behavior is defined.

Django’s logging extensions

Django provides a number of utilities to handle the uniquerequirements of logging in Web server environment.

Loggers

Django provides several built-in loggers.

django

The catch-all logger for messages in the django hierarchy. No messages areposted using this name but instead using one of the loggers below.

django.request

Log messages related to the handling of requests. 5XX responses areraised as ERROR messages; 4XX responses are raised as WARNINGmessages. Requests that are logged to the django.security logger aren’tlogged to django.request.

Messages to this logger have the following extra context:

status_code: The HTTP response code associated with therequest.
request: The request object that generated the loggingmessage.

django.server

Log messages related to the handling of requests received by the server invokedby the runserver command. HTTP 5XX responses are logged as ERRORmessages, 4XX responses are logged as WARNING messages, and everything elseis logged as INFO.

Messages to this logger have the following extra context:

status_code: The HTTP response code associated with the request.
request: The request object that generated the logging message.

django.template

Log messages related to the rendering of templates.

Missing context variables are logged as DEBUG messages.

django.db.backends

Messages relating to the interaction of code with the database. For example,every application-level SQL statement executed by a request is logged at theDEBUG level to this logger.

Messages to this logger have the following extra context:

duration: The time taken to execute the SQL statement.
sql: The SQL statement that was executed.
params: The parameters that were used in the SQL call.For performance reasons, SQL logging is only enabled whensettings.DEBUG is set to True, regardless of the logginglevel or handlers that are installed.

This logging does not include framework-level initialization (e.g.SET TIMEZONE) or transaction management queries (e.g. BEGIN,COMMIT, and ROLLBACK). Turn on query logging in your database if youwish to view all database queries.

django.security.*

The security loggers will receive messages on any occurrence ofSuspiciousOperation and other security-relatederrors. There is a sub-logger for each subtype of security error, including allSuspiciousOperations. The level of the log event depends on where theexception is handled. Most occurrences are logged as a warning, whileany SuspiciousOperation that reaches the WSGI handler will be logged as anerror. For example, when an HTTP Host header is included in a request froma client that does not match ALLOWED_HOSTS, Django will return a 400response, and an error message will be logged to thedjango.security.DisallowedHost logger.

These log events will reach the django logger by default, which mails errorevents to admins when DEBUG=False. Requests resulting in a 400 response dueto a SuspiciousOperation will not be logged to the django.requestlogger, but only to the django.security logger.

To silence a particular type of SuspiciousOperation, you can override thatspecific logger following this example:

'handlers': {
    'null': {
        'class': 'logging.NullHandler',
    },
},
'loggers': {
    'django.security.DisallowedHost': {
        'handlers': ['null'],
        'propagate': False,
    },
},

Other django.security loggers not based on SuspiciousOperation are:

django.security.csrf: For CSRF failures.

django.db.backends.schema

Logs the SQL queries that are executed during schema changes to the database bythe migrations framework. Note that it won’t log thequeries executed by RunPython.Messages to this logger have params and sql in their extra context (butunlike django.db.backends, not duration). The values have the same meaningas explained in django.db.backends.

Handlers

Django provides one log handler in addition to those provided by thePython logging module.

class AdminEmailHandler(include_html=False, email_backend=None, reporter_class=None)
This handler sends an email to the site ADMINS for each logmessage it receives.

If the log record contains a request attribute, the full detailsof the request will be included in the email. The email subject willinclude the phrase “internal IP” if the client’s IP address is in theINTERNAL_IPS setting; if not, it will include “EXTERNAL IP”.

If the log record contains stack trace information, that stacktrace will be included in the email.

The include_html argument of AdminEmailHandler is used tocontrol whether the traceback email includes an HTML attachmentcontaining the full content of the debug Web page that would have beenproduced if DEBUG were True. To set this value in yourconfiguration, include it in the handler definition fordjango.utils.log.AdminEmailHandler, like this:

'handlers': {
    'mail_admins': {
        'level': 'ERROR',
        'class': 'django.utils.log.AdminEmailHandler',
        'include_html': True,
    }
},

Note that this HTML version of the email contains a full traceback,with names and values of local variables at each level of the stack, plusthe values of your Django settings. This information is potentially verysensitive, and you may not want to send it over email. Consider usingsomething such as Sentry to get the best of both worlds – therich information of full tracebacks plus the security of not sending theinformation over email. You may also explicitly designate certainsensitive information to be filtered out of error reports – learn more onFiltering error reports.

By setting the email_backend argument of AdminEmailHandler, theemail backend that is being used by thehandler can be overridden, like this:

'handlers': {
    'mail_admins': {
        'level': 'ERROR',
        'class': 'django.utils.log.AdminEmailHandler',
        'email_backend': 'django.core.mail.backends.filebased.EmailBackend',
    }
},

By default, an instance of the email backend specified inEMAIL_BACKEND will be used.

The reporter_class argument of AdminEmailHandler allows providingan django.views.debug.ExceptionReporter subclass to customize thetraceback text sent in the email body. You provide a string import path tothe class you wish to use, like this:

'handlers': {
    'mail_admins': {
        'level': 'ERROR',
        'class': 'django.utils.log.AdminEmailHandler',
        'include_html': True,
        'reporter_class': 'somepackage.error_reporter.CustomErrorReporter'
    }
},

New in Django 3.0:The reporter_class argument was added.

sendmail(_subject, message, *args, **kwargs)
Sends emails to admin users. To customize this behavior, you cansubclass the AdminEmailHandler class andoverride this method.

Filters

Django provides some log filters in addition to those provided by the Pythonlogging module.

class CallbackFilter(callback)
This filter accepts a callback function (which should accept a singleargument, the record to be logged), and calls it for each record thatpasses through the filter. Handling of that record will not proceed if thecallback returns False.

For instance, to filter out UnreadablePostError(raised when a user cancels an upload) from the admin emails, you wouldcreate a filter function:

from django.http import UnreadablePostError
 
def skip_unreadable_post(record):
    if record.exc_info:
        exc_type, exc_value = record.exc_info[:2]
        if isinstance(exc_value, UnreadablePostError):
            return False
    return True

and then add it to your logging config:

'filters': {
    'skip_unreadable_posts': {
        '()': 'django.utils.log.CallbackFilter',
        'callback': skip_unreadable_post,
    }
},
'handlers': {
    'mail_admins': {
        'level': 'ERROR',
        'filters': ['skip_unreadable_posts'],
        'class': 'django.utils.log.AdminEmailHandler'
    }
},

class RequireDebugFalse
This filter will only pass on records when settings.DEBUG is False.

This filter is used as follows in the default LOGGINGconfiguration to ensure that the AdminEmailHandler only sendserror emails to admins when DEBUG is False:

'filters': {
    'require_debug_false': {
        '()': 'django.utils.log.RequireDebugFalse',
    }
},
'handlers': {
    'mail_admins': {
        'level': 'ERROR',
        'filters': ['require_debug_false'],
        'class': 'django.utils.log.AdminEmailHandler'
    }
},

class RequireDebugTrue
This filter is similar to RequireDebugFalse, except that records arepassed only when DEBUG is True.

Django’s default logging configuration

By default, Django configures the following logging:

When DEBUG is True:

The django logger sends messages in the django hierarchy (exceptdjango.server) at the INFO level or higher to the console.When DEBUG is False:
The django logger sends messages in the django hierarchy (exceptdjango.server) with ERROR or CRITICAL level toAdminEmailHandler.Independent of the value of DEBUG:
The django.server logger sends messages at the INFO levelor higher to the console.All loggers except django.server propagate logging to theirparents, up to the root django logger. The console and mail_adminshandlers are attached to the root logger to provide the behavior describedabove.

See also Configuring logging to learn how you cancomplement or replace this default logging configuration defined indjango/utils/log.py.