Skip to content

Structured logging with structlog

The logger of a project is configured in one module, core/logger.py, and every other module gets its logger from get_logger.

What to read

  • _configure picks the renderers from the ENV variable. With prod or production it writes one JSON object per event, and dict_tracebacks turns an exception into data. Otherwise it writes a console line, and the console renderer formats the exception itself. It reads the level once from LOG_LEVEL and adds the name of the logger to each event. functools.cache runs it once, on the first call to get_logger, so importing the module configures nothing.
  • get_logger returns a FilteringBoundLogger. A module calls get_logger(__name__), binds context with .bind(request_id=rid) and logs events as keyword pairs: log.info("user_created", user_id=user.id).

What to adapt

  • The path of the module is <your_package>/core/logger.py.
  • Keep the JSON and console split and the LOG_LEVEL variable. The standards of logging assume them.
  • logging.getLevelNamesMapping needs Python 3.11 or newer.

References

code/app/core/logger.py:28-49

the configuration: JSON in production, a console in development, applied once

@functools.cache
def _configure() -> None:
    shared: list[Processor] = [
        structlog.contextvars.merge_contextvars,
        structlog.stdlib.add_logger_name,
        structlog.stdlib.add_log_level,
        structlog.processors.TimeStamper(fmt="iso", utc=True),
        structlog.processors.StackInfoRenderer(),
    ]
    renderers: list[Processor] = (
        [structlog.processors.dict_tracebacks, structlog.processors.JSONRenderer()]
        if _is_prod()
        else [structlog.dev.ConsoleRenderer()]
    )
    level = _level()
    logging.basicConfig(format="%(message)s", stream=sys.stdout, level=level)
    structlog.configure(
        processors=[*shared, *renderers],
        wrapper_class=structlog.make_filtering_bound_logger(level),
        logger_factory=structlog.stdlib.LoggerFactory(),
        cache_logger_on_first_use=True,
    )

code/app/core/logger.py:52-55

the function that every module imports

def get_logger(name: str | None = None) -> FilteringBoundLogger:
    """Return the logger of `name`, usually `__name__`, configuring structlog on the first call."""
    _configure()
    return structlog.get_logger(name)

https://www.structlog.org/en/stable/getting-started.html#building-a-context

binding context: a logger is immutable, and bind() returns a new one that carries the context

Each logger with its context is immutable. You manipulate the context by creating new loggers using bind() and unbind().

https://www.structlog.org/en/stable/getting-started.html#rendering

the renderer is the last processor, and JSONRenderer is the one for production

So assuming you want to follow best practices and render your event dictionary to JSON that is picked up by a log aggregation system like ELK or Graylog, structlog comes with batteries included - you just have to tell it to use its JSONRenderer.

https://www.structlog.org/en/stable/api.html#structlog.dev.ConsoleRenderer

the console renderer formats exceptions itself, so the processor that turns them into data goes only before JSONRenderer

Changed in version 21.2.0: ConsoleRenderer now handles the exc_info event dict key itself. Do not use the structlog.processors.format_exc_info processor together with ConsoleRenderer anymore!

Cited by