Architecture of `coola.summary`¶

This page describes the design of the coola.summary package for contributors: the invariants, the contracts between the parts, and the trade-offs of the main design choices. The rules that users rely on are in the user guide; the API is in the reference.

Overview¶

summarize uses one engine (coola.summary.engine), one extension point and one option object (SummaryConfig). A summarizer only describes one type: summarize(data, config, *, compact) returns a str for a leaf, or a Node (a header, the lazy (label, child) pairs and their number) for a container. The engine owns the invariants: the summarizer lookup (by MRO), the display depth, max_items and max_characters, the cycle check, the layout, and the errors (a summarizer that raises is replaced by a placeholder). It walks the nodes with an explicit stack, so there is no RecursionError. A container at the display depth is shown on one line by CompactRepr, whose work is bounded by max_items, so the cost depends on the size of the summary and not on the size of the data. This is the same split as the equality and the recursive engines.

Constraints¶

Simple and easy to maintain. Summarizers are stateless: no constructor, no __repr__, no equal. The options live in one config, the traversal in one engine, and a summarizer only describes one type.
Bounded output and work. summarize is called to build a log line. The cost depends on the size of the summary, not on the size of the data: a huge input must not cost more than a small one. Small inputs (tens of microseconds) pay a little for the fixed cost of the engine.
Consistent with the siblings. Same engine shape as equality (the engine owns the invariants, a plugin describes one type), same registry, same option rules as recursive and equality (keyword-only, validated, one frozen config).
Optional libraries stay lazy. The package imports no optional library with import coola.summary; the summarizers of NumPy, torch, pandas, ... are loaded the first time the registry needs a type of the library (register_lazy). The plugin group coola.summary.summarizers is the extension point.

Architecture in detail¶

summarize(data, *, max_depth=1, max_items=5, max_characters=1000, show_data=False, registry=None)
        |   one SummaryConfig (frozen, validated), one registry (default: the singleton)
        v
render(data, registry, config)                coola/summary/engine.py
        |   explicit stack of frames; ids of the nodes being expanded; lines are appended once
        |
        |-- summarizer = registry.find_summarizer(type(obj))            MRO, fallbacks
        |-- result     = summarizer.summarize(obj, config, compact=...)  pure, no recursion
        |        str                        -> a leaf: the text, cut at max_characters
        |        Node(header, children, n)  -> a container:
        |              compact (depth >= max_depth)  one bounded line (CompactRepr)
        |              already open (cycle)          "<cycle>"
        |              otherwise                     header, then at most max_items children,
        |                                            then "... (+N more)"
        |-- an exception of the summarizer or of the children iterator -> a placeholder line
        v
the text (lines joined once)

The engine owns the invariants (bounded output, bounded work, depth, cycles, totality, layout). A summarizer only describes one type: a string for a leaf, or a header and lazy children for a container. That is the same split as equality (the engine owns identity, type, depth and cycles; a tester returns a verdict or the child pairs) and as recursive (a transformer returns the children). registry, depth and max_depth are not arguments of the summarizer, so a summarizer cannot forget a protection and cannot recurse.

The summarizer contract¶

@dataclass(frozen=True, slots=True)
class Node:
    header: str  # the first line: "<class 'list'> (length=3)"
    children: Iterable[tuple[Any, Any]]  # lazy (label, child) pairs, in display order
    total: int  # the number of children, to announce "... (+N more)"


class Summarizer(Protocol[T_contra]):
    def summarize(self, data: T_contra, config: SummaryConfig, *, compact: bool) -> str | Node:
        """A string (a leaf) or a ``Node`` (a container)."""


class BaseSummarizer(ABC, Generic[T]):  # convenience base class
    @abstractmethod
    def summarize(self, data: T, config: SummaryConfig, *, compact: bool) -> str | Node: ...

    def equal(self, other: object) -> bool:  # concrete: same type (all the built-ins are stateless)
        return type(other) is type(self)

Rules of the contract (documented in BaseSummarizer, checked by one parametrised unit test over the default registry, like NATIVE_SAMPLES in equality):

summarize is pure and cheap: it does not call the registry, does not recurse, does not modify data, and does not iterate the whole data. For a container it reads len(data) and returns the children lazily (enumerate(data), data.items(), a generator): the engine takes at most max_items of them. A set is the exception: it returns the smallest max_items items.
compact is True when the node is at the display depth. A leaf may use it (the default summarizer shows the type only when compact is False); a container ignores it (the engine uses CompactRepr).
A leaf returns a str, which may have several lines. The engine cuts it to at most max_characters characters (the ... is part of the limit, as in truncate_str), so a leaf does not have to.
Labels are anything: the engine writes safe_str(label) cut at max_characters.
An exception (of summarize, of the children iterator, of a label's str) is the engine's problem: it prints a placeholder, the summarizer does not guard itself.

Node is a dataclass and not a tuple so that the contract is readable and a later field (for example a closing line) can be added without breaking the plugins. Summarizer (protocol) and BaseSummarizer (ABC) follow the family convention: a summarizer from another library needs no import of coola.

The config¶

@dataclass(frozen=True, slots=True)
class SummaryConfig:  # shareable, hashable, thread-safe; validated in __post_init__
    max_depth: int = 1  # non-negative int, not a bool
    max_items: int | None = 5  # non-negative int, or None for no limit
    max_characters: int | None = 1000  # non-negative int, or None for no limit
    show_data: bool = False  # metadata summarizers show a bounded repr of the values

All the options of the package are here, in one place, with one validation. None means "no limit" (there is no negative sentinel). The consequences of one config instead of options in the constructors:

A person who wants three items writes summarize(data, max_items=3); they do not build a registry.
Summarizers are stateless, so they need no __init__, __repr__, equal or display mixin.
There is no per-type option (max_items=3 for lists and 10 for dicts); a user who needs it registers their own summarizer (a subclass that returns a Node with a smaller total).
One limit for items, fields and columns: max_items (default 5) counts the entries shown, so a frame lists 5 columns by default and summarize(df, max_items=20) lists more.
The indentation is fixed (2 spaces per level): a regular layout needs no num_spaces. Adding it would be one field of the config.

The engine¶

INDENT = "  "


def render(data, registry, config):
    lines: list[str] = []
    ancestors: set[int] = set()  # ids of the nodes being expanded
    stack: list[Frame] = []  # Frame: children iterator, depth, total, shown, id

    def cut(text):  # the character limit, for a leaf and for a label
        n = config.max_characters
        return text if n is None or len(text) <= n else text[:n] + "..."

    def visit(label, obj, depth):
        compact = depth >= config.max_depth
        prefix = INDENT * depth
        head = prefix if label is None else f"{prefix}({cut(safe_str(label))}): "
        try:
            result = registry.find_summarizer(type(obj)).summarize(obj, config, compact=compact)
            if not isinstance(result, Node):
                text = cut(result)
            elif compact:
                text = CompactRepr(config).repr(obj)  # one bounded line
            elif id(obj) in ancestors:
                text = "<cycle>"
            else:
                text = result.header
                ancestors.add(id(obj))
                children = islice(result.children, config.max_items)
                stack.append(Frame(children, depth + 1, result.total, 0, id(obj)))
        except Exception as exc:  # the data is broken, not the summary
            text = f"{type(obj)} <summary failed: {type(exc).__name__}>"
        lines.append(head + text.replace("\n", "\n" + prefix + INDENT))

    visit(None, data, 0)
    while stack:
        frame = stack[-1]
        try:
            label, child = next(frame.children)
        except StopIteration:
            stack.pop()
            ancestors.discard(frame.id)
            if frame.total > frame.shown:
                lines.append(f"{INDENT * frame.depth}... (+{frame.total - frame.shown:,} more)")
        except Exception as exc:  # a property that raises, a broken __iter__
            frame.children = iter(())
            lines.append(f"{INDENT * frame.depth}<summary failed: {type(exc).__name__}>")
        else:
            frame.shown += 1
            visit(label, child, frame.depth)
    return "\n".join(lines)

Design points:

No RecursionError, no thread-local, no lock. The state (stack, ancestors, lines) is local to the call.
summarize of a container runs at the boundary too. It is cheap by contract (rule 1), and it keeps the engine free of a "is this a container?" question that only the summarizer can answer.
The cycle check is on the open nodes only (ancestors), so a DAG is not reported as a cycle, and a leaf (a string, an array) never enters the set.
The layout is one rule: a node at depth d is written with 2 * d spaces, and a text with several lines is aligned under its first character. Nothing is re-indented.
A failure is local. A node that fails prints its type and the exception name (never the message, which may carry data) and the rest of the summary is intact. A summary is for logs and error messages, so raising there would hide the original problem. The cost is that a bug in a custom summarizer shows as <summary failed: ValueError> instead of a traceback; the exception type is in the text, and the tests of the plugin show it.
max_depth is a display depth here (the nodes deeper than it are shown on one line), whereas it is a limit that raises in objects_are_equal; the docs say so in both places. expand_depth would remove the ambiguity, but max_depth is the name every user of summarize knows.

The compact form¶

The compact form is what the engine prints for a container at the display depth: one line, with bounded work. It is a reprlib.Repr configured from the config and with two overrides:

class CompactRepr(reprlib.Repr):
    def __init__(self, config):
        super().__init__()
        self.maxlevel = 3
        self.maxlist = self.maxtuple = self.maxdict = self.maxset = ... = config.max_items
        self.maxstring = self.maxother = config.max_characters

    def repr_dict(self, x, level):  # keeps the insertion order: reprlib sorts the keys
        ...

    def repr_set(self, x, level):  # the smallest items, as SetSummarizer (reprlib sorts everything)
        ...

list, tuple, deque, dict, set, frozenset and array are cut at max_items ([0, 1, 2, 3, 4, ...]) in O(max_items), nested containers are cut at 3 levels ([...]), and a string is shortened in the middle ('abc...xyz'). It is the only code that shortens a container.
Any other type (a subclass such as Counter, a dataclass, an array) is shown with its repr, cut at max_characters. It is computed in full, which is the documented exception of "Bounded work".
It is built only when a container is at the boundary (lazy), so a flat summary pays nothing.
Fast path (required by the benchmark gate): a container with at most max_items scalar entries is shown with repr() directly, because reprlib is slower than repr on small containers.

A "header only" form (<class 'list'> (length=100) ...) would be bounded and need no reprlib, but it would hide the content of a small nested configuration, which is the most common use of the default max_depth=1. Asking the summarizer for an inline form (Node.inline) would be exact for every type, but would double the contract. reprlib is slower than repr on small containers, so the fast path keeps the common case cheap; the gate is "up to 2x and at most 10 µs more" for inputs under 50 µs.

The layout of the text¶

Rules (the examples are the output of the function):

>>> from coola.summary import summarize
>>> print(summarize({"a": [1, 2, 3, 4, 5, 6, 7], "b": {"c": {1, 2}, "d": "x\ny"}}, max_depth=3))
<class 'dict'> (length=2)
  (a): <class 'list'> (length=7)
    (0): <class 'int'> 1
    (1): <class 'int'> 2
    (2): <class 'int'> 3
    (3): <class 'int'> 4
    (4): <class 'int'> 5
    ... (+2 more)
  (b): <class 'dict'> (length=2)
    (c): <class 'set'> (length=2)
      (0): 1
      (1): 2
    (d): <class 'str'> x
      y
>>> print(summarize([1, "a", [1, 2]]))           # max_depth=1: the items are on one line
<class 'list'> (length=3)
  (0): 1
  (1): a
  (2): [1, 2]
>>> a = [1]; a.append(a); print(summarize(a, max_depth=3))
<class 'list'> (length=2)
  (0): <class 'int'> 1
  (1): <cycle>

A node is header, then its children one level deeper, then ... (+N more) if some are not shown. An empty container is its header alone: <class 'list'> (length=0).
A child is (label): text. The label of a sequence or a set is the position, of a mapping the str of the key, of a dataclass or a pydantic model the field name, of a named tuple the field name.
The type is shown for the nodes that are expanded and for their leaves; the boundary shows values. A leaf shows <class 'int'> 1 when compact is False and 1 when it is True: summarize(42) is <class 'int'> 42, and the items of summarize([1, "a"]) are bare because they are at the display depth. It needs one flag (compact). Showing the type of every leaf would be a simpler rule, but twice as long on the default output.
A metadata summarizer always shows its type: <class 'numpy.ndarray'> | shape=(3,) | dtype=float64.
Indentation is a fixed 2 spaces per level.

An empty container is (length=0), a list at the boundary is [0, 1, 2, 3, 4, ...], a dict at the boundary keeps its order but a Counter or other subclass is shown with its repr, and the hidden entries are announced by ... (+N more).

Built-in summarizers¶

Summarizer	`summarize(data, config, compact=...)`	Default registration
`DefaultSummarizer`	`safe_str(data)`, preceded by `<class ...>` unless `compact`; a leaf	`object`, `UserString`
`SequenceSummarizer`	`Node("<type> (length=n)", enumerate(data), n)` (the field names for a named tuple)	`list`, `tuple`, `deque`, `Sequence`
`SetSummarizer`	`Node("<type> (length=n)", enumerate(smallest(data, config.max_items)), n)`	`set`, `frozenset`
`MappingSummarizer`	`Node("<type> (length=n)", data.items(), n)`	`dict`, `Mapping`, `MappingProxyType`
`DataclassSummarizer`	`Node("<type> (fields=n)", ((f.name, getattr(data, f.name)) for f in fields(data) if f.repr), n)`	fallback: `dataclasses.is_dataclass`
`PydanticModelSummarizer`	the same with `model_fields` without the secrets, plus `model_extra`	`pydantic.BaseModel` (lazy)
`NDArraySummarizer`, `TensorSummarizer`, `JaxArraySummarizer`, `Pandas`, `Polars`, `Pyarrow`, `Xarray`	`" \\| ".join([type, *self._describe(data, config)])`, or `safe_repr(data)` if `config.show_data`	by library (lazy)

Container summarizers are three one-line classes on a common helper; the dataclass and pydantic summarizers share BaseObjectFieldsSummarizer (one abstract method _get_fields). All of them are stateless: no constructor and no __repr__.

A dataclass hides the field(repr=False) fields: it is how a dataclass author declares "do not print this" (tokens, large buffers), and the pydantic summarizer already hides secrets. The summary is not a comparison, so unlike equality and hashing it does not need every field. A named tuple uses its field names as labels (data._fields), consistent with dataclasses.

BaseMetadataSummarizer (_describe(data, config) returns the fields, _check_dependencies raises MissingPackageError at construction if the library is missing) is the base class of the array and table summarizers. The summarizers of frames and tables call format_fields(items, config.max_items) (a: int64, b: float64, ... (+3 more)) themselves.

str, int, float, complex and bool are not registered: they resolve through object. UserString and MappingProxyType are registered. The plugin group is coola.summary.summarizers.

The extension example:

class PointSummarizer(BaseSummarizer[Point]):
    def summarize(self, data, config, *, compact):
        return f"Point(x={data.x}, y={data.y})"  # a leaf


class NodeSummarizer(BaseSummarizer[Tree]):
    def summarize(self, data, config, *, compact):  # a container
        return Node(
            f"{type(data)} (children={len(data.children)})",
            enumerate(data.children),
            len(data.children),
        )


register_summarizers({Point: PointSummarizer(), Tree: NodeSummarizer()})

Public API¶

from coola.summary import (
    summarize,  # (data, *, max_depth=1, max_items=5, max_characters=1000, show_data=False, registry=None)
    SummaryConfig,
    SummarizerRegistry,  # find_summarizer, has_summarizer, summarize(data, config=None)
    get_default_registry,
    get_default_summarizer_registry,  # alias
    register_summarizers,  # (mapping, *, exist_ok=False)
    Summarizer,  # protocol: summarize(data, config, *, compact) -> str | Node
    BaseSummarizer,
    Node,
    DefaultSummarizer,
    SequenceSummarizer,
    SetSummarizer,
    MappingSummarizer,
    DataclassSummarizer,
    BaseMetadataSummarizer,
    format_fields,  # (fields, max_fields) -> str
    # lazy: NDArraySummarizer, TensorSummarizer, JaxArraySummarizer, PandasDataFrameSummarizer,
    # PandasSeriesSummarizer, PolarsDataFrameSummarizer, PolarsLazyFrameSummarizer,
    # PolarsSeriesSummarizer, PyarrowArraySummarizer, PyarrowTableSummarizer,
    # XarrayDataArraySummarizer, XarrayDatasetSummarizer, XarrayVariableSummarizer,
    # PydanticModelSummarizer
)

summarize(...) builds a SummaryConfig (so the options are validated in one place) and calls registry.summarize(data, config), which calls the engine; registry.summarize is the registry-level entry point, as registry.objects_are_equal is for equality.

Dependents¶

Nothing in src imports coola.summary (the mentions in equality and utils.singleton are docstrings). The package is a leaf.

Dependent	Use
`tests/properties/test_summary_random.py`, `test_cross_module`, `test_library_types`	Properties of `summarize(data, max_depth=...)`.
`tests/integration/test_lazy_imports.py`, `test_support_matrix.py`, `test_default_registries.py`	Lazy loading, the support matrix and the registered types.
`pyproject.toml` import-linter contracts	`coola.summary` imports only `coola.registry`, `coola.utils`, `coola.validation`.
Plugins of `coola.summary.summarizers`	Implement `summarize(data, config, *, compact)`; a plugin that returns a string is a leaf.

Design choices and trade-offs¶

The choices that shape the package, with the alternative each was preferred to:

A config instead of constructor options (see The config): simpler and stateless, at the cost of per-type options.
A reprlib-based compact form: it keeps the content of a small nested structure and it is bounded.
A total engine: an exception of a summarizer is a placeholder, not a raised error.
Fixed indentation, one limit for items, fields and columns: fewer options to validate and document. A second max_fields option or a larger default for max_items (a list of 20 would show 10 lines) are the alternatives.
The rule "type shown inside the expansion, values at the boundary": needs one flag.

Not covered¶

Plain objects with __dict__, attrs classes, __slots__: leaves. equality has ObjectAttributesEqualityTester; a summarizer of the attributes is a separate item.
A machine-readable form (JSON, a tree of Node): the package produces text for humans. A summary_tree(...) that returns the nodes is possible later, since the engine already walks them.
Width-aware or colored output, a str.format-style template, and options per type.
Summaries of the elements of an array (min, max, mean, number of NaN): a metadata summarizer can compute them for the types that want it; the default stays shape and dtype (cheap, no copy, no device transfer).

Architecture of coola.summary¶