overview_python_logging

Overview of the Python logging Module

Logging is the process of keeping records of activities and data of a software program. It is an important aspect of developing, debugging, and running software as it helps developers track their program, better understand the flow and discover unexpected scenarios and problems. Logging is essential when creating any complex software as it helps the developers to locate an exact region of the application that needs attention.

The Python logging module (part of the standard Python library) allows code developers and users to record information about their Python applications and keep track of events generated by the applications as they take place. It consists of a class (logging class) and a set of functions that implement a flexible event logging and tracking system for Python applications.

Main Components

The main components of the Python logging module are:

• Loggers: Expose an interface that an application can use to log messages at run time. They also determine which log messages to act upon based upon severity (the default filtering facility). Loggers have a hierarchy. On top of the hierarchy is the root logger. When a new logger is created, its parent will be set to the root logger. A logger has three main components:

  • Propagate: Decides whether a log should be propagated to the logger’s parent.
  • A level: Like the log handler level, the logger level is used to filter out “less important” logs. Except, unlike the log handler, the level is only checked at the “child” logger; once the log is propagated to its parents, the level will not be checked.
  • Handlers: The list of handlers that a log will be sent to when it arrives to a logger. A log will be broadcast to all handlers once it passes the logger level check.

• Handlers: Send the logs created by loggers to their destination. Each log handler has two important fields:

  • A formatter which adds context information to a log.
  • A log level that filters out logs whose levels are lower.

  Popular handlers include:

  • FileHandler: For sending log messages to a file.
  • RotatingFileHandler: For rotating log statements into a new file every time the current log file reaches a certain size.
  • StreamHandler: For sending log messages to an output stream like stdout.

• Filters: Provide a mechanism to determine which logs are recorded. They are used to perform more advanced filtering than level filtering.

• Formatters: Determine the output formatting of log messages. They are used by the Python logging handlers to enhance the information available through LogRecord attributes. They are useful to know when the log is sent, where (file, line number, subroutine, etc.), and additional context such as the process.

To use Python logging in an existing workflow:

1. First, create a logger object using the basicConfig() method from the logging module.
2. Then, set up handlers for each type of log message to be captured (e.g., debug messages or errors).
3. Finally, add filters and formatters as needed to customize how logs are displayed or stored.

The logging flow can be summarized in the following diagram:

Logging Levels

The logging module has six predefined logging levels that are of incremental severity:

Level	Numeric Value	Description
NOTSET	0	Default level when a new logger is created. Setting the root logger to NOTSET logs all messages. For other loggers, NOTSET messages are delegated to parent loggers until a level other than NOTSET is found.
DEBUG	10	Messages useful for debugging how an application is running.
INFO	20	Messages that log standard/expected events.
WARNING	30	Potentially negative events that may cause abnormal operation or otherwise require attention (such as the use of a deprecated API).
ERROR	40	High severity messages generally related to an operation failing that does not cause the program to crash or exit.
CRITICAL	50	Highest severity messages; may cause a program to crash or exit.

Severity levels are used by the framework to decide whether a given message can be ignored. E.g., typically DEBUG messages will be ignored unless a configuration lowers the threshold for the associated logger. When you set a logging level in Python, you’re telling the library you want to handle all events from that level up. Setting the log level to INFO will include INFO, WARNING, ERROR, and CRITICAL messages. Therefore, NOTSET and DEBUG messages will not be included.

Usage

To use the logging module, a user first needs to include the line:

import logging

to their Python code and then add statements to create log messages.

To emit a log message, a caller first requests a named logger. The application can use the name to configure different rules for different loggers. This logger can then be used to emit simply-formatted messages at different log levels (DEBUG, INFO, ERROR, etc.), which can be used by the application to handle messages of higher priority other than those of a lower priority.

Assume that the user doesn't set a specific logging level, the following code snippet shows how you can use all the five logging levels with the syntax:

logging.debug("A DEBUG Message")
logging.info("An INFO")
logging.warning("A WARNING")
logging.error("An ERROR")
logging.critical("A message of CRITICAL severity")

The above statements will produce messages corresponding only to warning, error, and critical:

WARNING:root:A WARNING
ERROR:root:An ERROR
CRITICAL:root:A message of CRITICAL severity

This is because the default logging level is warning and the code will only print messages that have levels equal to warning or above. Note also that the default message format is:

  <SEVERITY>:<NAME>:<MESSAGE>

where <NAME> is the name of our logger. We can customized the messages by creating our own logger (instead of using root) using a function call (logger = logging.getLogger()) at the top of the file we want to monitor.