Architecture of `coola.rounding`¶

This page describes the design of the coola.rounding package for contributors: the invariants, the contracts between the parts, and the trade-offs of the main design choices. The rules that users rely on are in the user guide; the API is in the reference.

Overview¶

round_values has one rule, two kernels, one adapter per table library and one extension point. A float is read as the decimal that repr shows, and "floor", "ceil" and "round" (halfway cases to the even digit) act on that decimal. The NumPy kernel (coola.rounding.numpy) finds the grid point with float arithmetic and two checks that compare the grid points in the dtype of the data, so the result equals Decimal(repr(x)).quantize(...) without Decimal. The torch kernel and the native scalar are the same formula with the operations of their library, and the tests compare them bit for bit with the NumPy kernel and with a Decimal oracle that is independent of the implementation. The adapters of pandas, polars and pyarrow find the floating-point columns, round each with the NumPy kernel, restore the nulls and put the column back: "coola owns the semantics, the libraries own the structure". A rounder is a plain function (data, mode, decimals) -> result registered in a TypeRegistry (lazy per library, MRO lookup, the plugin group coola.rounding.rounders). round_values owns the validation, the dispatch, the recursive walk and the error contract (UnsupportedTypeError, on_missing_package). See the user guide for the properties of the package.

Constraints¶

Simple and easy to maintain. Two kernels (NumPy and torch, like the numeric kernel of equality), a scalar version of the same formula for Python floats, and the three table libraries as thin adapters over the NumPy kernel. Copies of the formula per backend, each with its own fix-ups, are how backends disagree.
Correct first, then fast. A rule that is exact and costs a few nanoseconds per element more is preferred to a fast rule that is wrong on 1 % of the inputs. The benchmark gate bounds the cost (floor/ceil at the speed of the plain kernel, round at most 5 times slower on a large array, a scalar and the recursive path fast).
Consistent with the siblings. One extension point (a registry, register_*, a plugin group), keyword-only options, errors raised before any work with a message that names the remedy, dataclasses entered with recursive=True, stateless and thread-safe.
Optional libraries stay lazy. import coola.rounding imports no optional library (verified: numpy, torch, pandas, polars and pyarrow are all absent from sys.modules after it) and coola still has zero hard dependencies. Rounding a polars or a pyarrow object needs NumPy; without it on_missing_package decides: raise (default), warn or ignore.

Architecture in detail¶

round_values(data, *, mode="round", decimals=0, recursive=False, on_unsupported="ignore", on_missing_package="raise", registry=None)
        |   validate mode, decimals, on_unsupported and on_missing_package once; one registry (default: the singleton)
        |   recursive ? recursive_apply(data, leaf) : leaf(data)
        v
leaf(x): rounder = registry.resolve(type(x))                  MRO, lazy libraries
        |   rounder(x, mode, decimals)  ->  result | raises UnsupportedTypeError
        v
rounders (coola/rounding/*.py)
        |-- native.py    float, int (and bool)          the scalar version of the formula
        |-- numpy.py     ndarray, generic               THE kernel (about 20 lines)
        |-- torch.py     Tensor                         the same formula, torch operations
        |-- pandas.py    Series, DataFrame  \
        |-- polars.py    Series, DataFrame   >  float columns -> round_numpy, nulls restored
        '-- pyarrow.py   Array, ChunkedArray, Table, RecordBatch  /

The interface owns the invariants (validation, dispatch, the recursive walk, the error contract). A rounder only describes one type: it receives a validated mode and decimals and returns a new object or raises UnsupportedTypeError. This is the split of equality ("coola owns the semantics, the libraries own the structure"): every number goes through the kernel; a library says which columns are floating point and gives the values back, never what 2.675 rounds to.

The rule¶

For a finite x and step = 10**-decimals, with repr(x) the shortest string that round-trips in the dtype of x:

"floor": the largest grid point that is <= Decimal(repr(x)); "ceil": the smallest that is >=; "round": the nearest, a halfway case goes to the grid point whose last digit is even.
The result is that grid point converted to the dtype (the nearest float).
Not changed: integers, booleans, NaN, infinities, missing values, and abs(x) * 10**decimals >= 2**53 (the value is already on the grid, and 10**decimals that overflows a float has no effect).
A zero result takes the sign of x.

The grid point is found with float arithmetic and two checks, which is what makes the rule exact without Decimal. With s = x * scale and k = floor(s) the index is wrong by at most one in either direction, because x * scale is rounded; comparing the grid points in the dtype of x repairs it in both directions. For "round" the midpoint (2k + 1) / (2 * scale) is computed as a float of the dtype: x == mid is a halfway case if and only if repr(x) is the midpoint, as long as the grid is coarser than the spacing of the floats (the bound of the "Exact" row).

def round_numpy(data, mode, decimals):  # the float case of the kernel
    scale = get_scale(decimals)  # 10.0**decimals, None if it overflows
    x = data.astype(np.float64, copy=False)
    low = data.dtype != np.float64

    def grid(k):  # a grid point as the dtype holds it, compared in float64
        g = k / scale
        return g.astype(data.dtype).astype(np.float64) if low else g

    s = x * scale
    k = np.floor(s)
    k += grid(k + 1) <= x  # s was rounded down: the next grid point is not above x
    k -= grid(k) > x  # s was rounded up: this grid point is above x
    if mode == "ceil":
        k += grid(k) < x  # x is strictly between two grid points
    elif mode == "round":
        mid = (2 * k + 1) / (2 * scale)
        if low:
            mid = mid.astype(data.dtype).astype(np.float64)
        k = np.where(x > mid, k + 1, np.where(x == mid, np.rint(k + 0.5), k))  # rint: even
    result = np.where(np.abs(s) < MAX_EXACT, k / scale, x)  # NaN, inf, huge: unchanged
    return np.copysign(result, x).astype(data.dtype, copy=False)

The code above omits the errstate and the integer/boolean branch; the real function wraps it (copy for integers and booleans, UnsupportedTypeError for the other dtypes, np.errstate(over="ignore", invalid="ignore")).

The scalar, the tensor, the tables¶

round_float (native). The same formula on one float, with math.floor, integer k and math.copysign: about 15 lines, 0.30 µs per call, bit-identical to the NumPy kernel (checked on 100,010 values, 3 modes, 5 values of decimals, signs of zero included). int (and bool) are returned as they are. The Decimal implementation is the specification and lives in the tests only.
round_torch. The same formula with torch.floor, torch.where, torch.round (which is half-to-even), the dtype of the tensor for the grid, float32 instead of float64 on mps, detach().clone() for the integers. The result is bit-identical to the NumPy kernel for float64, float32 and float16. A non-strided layout (sparse, nested) raises UnsupportedTypeError instead of the raw NotImplementedError of torch.
Type preservation. NumPy: out = data.copy() then np.copyto(out, rounded). It keeps a matrix, a subclass and a MaskedArray with its mask and fill value, so the masked branch goes away. np.generic goes through a 0-d array and [()].
Tables. The three adapters share one shape: find the floating-point columns, round each with round_numpy on its NumPy values, restore the nulls, put the column back with its name and dtype.
pandas: isetitem keeps duplicate column names; the nullable and Arrow dtypes go through to_numpy(na_value=nan); a Series of integers is returned as a copy.
polars: pl.Series(name, round_numpy(s.to_numpy(), ...), dtype=s.dtype) then zip_with( s.is_not_null(), s) when there are nulls. No polars expression and no Series / scalar are needed. The result equals the kernel on 1,000,000 values, in three modes, with nulls and NaN, and Float32 is kept.
pyarrow: pa.array(round_numpy(a.to_numpy(zero_copy_only=False), ...), type=a.type, mask=...) per column and type(data).from_arrays(columns, schema=data.schema) for Table and RecordBatch. The result equals the kernel, float32 and nulls are kept.
A column that is not floating point is returned as it is (an integer column is not copied for polars and pyarrow: their buffers are immutable).
NumPy is imported when the adapter runs, not when it is registered. If it is missing the adapter raises MissingPackageError before any work, and round_values applies on_missing_package.

The rounder contract and the registry¶

Rounder = Callable[[Any, RoundMode, int], Any]


class UnsupportedTypeError(TypeError):
    """The object, or its dtype, cannot be rounded."""

A rounder is a plain function: no base class, no protocol, no state. It receives a mode and a decimals that round_values validated, returns a new object (never data itself unless data is immutable and unchanged), and raises UnsupportedTypeError for a type or a dtype it does not support. The default registry is a TypeRegistry[Rounder] built by make_default_registry_singleton, like the registries of the siblings:

Registered for	Rounder	Loaded
`float`, `int` (covers `bool` and `IntEnum`)	`round_float`, `round_int` (identity)	at import
`np.ndarray`, `np.generic` (before `float` in the MRO of `np.float64`)	`round_numpy`	`register_lazy(("numpy",), ...)`
`torch.Tensor`	`round_torch`	lazily
`pd.Series`, `pd.DataFrame`	`round_pandas`	lazily
`pl.Series`, `pl.DataFrame`	`round_polars`	lazily
`pa.Array`, `pa.ChunkedArray`, `pa.Table`, `pa.RecordBatch`	`round_pyarrow`	lazily

Each library module has a get_default_rounders() returning its {type: rounder} mapping (and {} when the library is missing), and interface.py has a _LAZY_ROUNDERS table, as hashing does. The rounders of polars and pyarrow are registered when their library is installed and import NumPy when they run. The plugin group is coola.rounding.rounders (a callable that receives the registry). A type with no rounder makes resolve raise TypeNotRegisteredError; round_values turns it into UnsupportedTypeError("Cannot round an object of type X; register a rounder with register_rounders"), with the registry error as the cause.

The example of the guide, a complete extension in six lines:

from decimal import Decimal
from coola.rounding import register_rounders

register_rounders(
    {Decimal: lambda x, mode, decimals: x.quantize(Decimal(1).scaleb(-decimals), ...)}
)

The interface¶

def round_values(
    data,
    *,
    mode="round",
    decimals=0,
    recursive=False,
    on_unsupported="ignore",
    on_missing_package="raise",
    registry=None,
):
    validate_arguments(mode, decimals, on_missing_package)
    validate_in(on_unsupported, ("raise", "ignore"), name="on_unsupported")
    registry = get_default_registry() if registry is None else registry
    if recursive:
        return recursive_apply(
            data,
            lambda x: _round_leaf(
                x, registry, mode, decimals, on_missing_package, on_unsupported=on_unsupported
            ),
        )
    return _round_object(data, registry, mode, decimals, on_missing_package)

_round_object does the lookup and the call; it is the only place that converts the registry error and that applies on_missing_package. When a rounder raises MissingPackageError: "raise" raises UnsupportedTypeError naming the package, with the MissingPackageError as the cause; "warn" emits a UserWarning naming the type and the package and returns data unchanged; "ignore" returns data unchanged. The policy covers any rounder (a user rounder that needs a missing package gets it too). A type with no rounder is not concerned: it raises UnsupportedTypeError (kept by recursive=True), whatever the option.
With recursive=True the policy applies to each leaf, so "raise" stops the walk at the first polars or pyarrow leaf and "warn" warns once per leaf (the default warning filter shows one per location).
_round_leaf calls it and catches only UnsupportedTypeError. A string, None or a date inside a nested structure is kept; a bug in a rounder is not hidden. With on_unsupported="raise" a leaf that has no rounder and is not a non-numeric value (None, str, bytes, bool, dates, times, timedelta, enums) raises too, so that a Decimal, a complex number or a JAX array is not left unrounded without a signal. The default is "ignore".
validate_arguments also checks on_missing_package ("raise", "warn" or "ignore") before any work. It accepts any integer (operator.index), so np.int64(2) works, and still rejects bool and float. The messages do not change.

Public API¶

from coola.rounding import (
    round_values,  # (data, *, mode="round", decimals=0, recursive=False, on_unsupported="ignore", on_missing_package="raise", registry=None)
    RoundMode,  # Literal["round", "floor", "ceil"]
    Rounder,  # Callable[[Any, RoundMode, int], Any]
    UnsupportedTypeError,  # TypeError
    register_rounders,  # (mapping, *, exist_ok=False)
    get_default_registry,  # TypeRegistry[Rounder]
)

The modules coola.rounding.{native,numpy,torch,pandas,polars,pyarrow} are public with their functions (round_numpy(data, mode, decimals), ...) so that a user can compose or wrap a rounder; they are not re-exported from coola.rounding. get_scale, MAX_EXACT and validate_arguments are in coola.rounding.common.

Dependents¶

Package	Use
`coola.recursive`	`round_values(recursive=True)` calls `recursive_apply(data, func)` with the default registry of `recursive`; dataclasses are entered.
`coola.utils.singleton`	`make_default_registry_singleton` and the plugin loader; `temporary_default_registries` knows the rounding singleton (checked by `tests/integration/test_default_registries.py`).
`tests/integration/test_lazy_imports.py`	Checks that importing `coola.rounding` imports no optional library.
`import-linter`	`coola.rounding` is forbidden to the core packages; it imports `coola.registry`, `coola.recursive` and `coola.utils` only.

Design choices and trade-offs¶

The rule for "round" is the decimal value, halfway to even. floor and ceil read 0.29 as 0.29; round reads 2.675 as 2.675, a tie, and gives 2.68. This is the only rule under which the three modes mean the same thing, under which the backends can agree, and that a user can predict by looking at the number. The alternatives are the exact binary value like Python (round(2.675, 2) is 2.67, but then floor(0.29, 2) must be 0.28, which the package refuses on purpose, or the modes disagree) and scale-and-rint (inexact). The consequence is that round_values does not agree with round() on the halfway cases of a decimal literal, which the guide says in one sentence.

The native scalar uses the float formula, not Decimal. Decimal(repr(x)).quantize(...) is simple and exact, but costs about 1.2 µs and needs a context for huge exponents (it raises InvalidOperation for decimals=400). The float formula costs 0.30 µs and is bit-identical to the arrays by construction. The Decimal version lives in the tests as the independent oracle: a kernel checked against a copy of itself proves nothing.

Tables go through NumPy. polars cannot do the exact division, and a second and a third copy of the formula is how backends disagree. The cost is a conversion per column (to_numpy, an array back; 0.8x to 2.2x of a native kernel), and NumPy is required to round a polars or pyarrow object (pandas already needs it). The error is the usual MissingPackageError, and on_missing_package decides what happens (below). Keeping a polars and a pyarrow kernel would make the conformance test the only guard, and polars has no exact operation to build one from.

A rounder is a function, not a class. A rounder has no state, no children and one job; Hasher and BaseHasher exist because a hasher has a protocol with several members. A plain TypeRegistry[Rounder] with register_rounders and get_default_registry is about 40 lines; a BaseRounder hierarchy with a RounderRegistry would triple it. Reconsider if a rounder ever needs more than one method.

recursive=False is the default. Strict by default (a string or a Decimal raises) is what a user rounding one object wants; recursive=True is the explicit opt-in to the lenient walk where unsupported leaves are kept. Always-recursive would lose the error for "abc", and a separate function would be two names for one concept (nested already has the walk).

Negative decimals are not supported. Python, NumPy and pandas accept them (round to tens, hundreds), but x * 10**-2 is not exact, so the kernel needs a second path (divide by 10**n, multiply back) in three implementations. The error message names the rule. The rule extends to it (step = 10**n) if it is requested.

Nested columns of a table (list<float>, struct) are left unchanged, and documented. Entering them needs a second walk (Arrow lists, polars List) in two libraries and is a different feature (rounding the values inside the cells). The property "only floating-point columns" states it.

Where the guarantee stops. The guide documents 1e13 and does not widen it: the first mismatch measured is at 9.9e13, and the error is one step, but a proof is out of reach with floats. Past the bound the answer is still a grid point within one step; a user who needs exactness at 15 digits uses Decimal with a registered rounder.

on_missing_package. An option with three values; the default raises.

Value	Behaviour
`"raise"` (default)	`UnsupportedTypeError` naming the missing package, `MissingPackageError` as the cause.
`"warn"`	`UserWarning` naming the type and the package; the object is returned unchanged.
`"ignore"`	The object is returned unchanged, silently.

The option is about a missing package, not about every unsupported type. It applies to any rounder that raises MissingPackageError (NumPy for polars and pyarrow, and a user rounder that needs another package). A type with no rounder (a str, a Decimal, a sparse tensor) is not affected: UnsupportedTypeError, kept by recursive=True. This keeps the lenient walk of recursive=True.
The rounders are registered when the library is installed and check NumPy when they run, so the registry, the lazy loading and import coola.rounding do not depend on NumPy.
"warn" and "ignore" can hide an unrounded column; they are opt-in. "warn" is the compromise for a pipeline that must not fail on an optional step.
The name is on_missing_package (not on_missing_numpy) because the mechanism is general; the keyword is keyword-only and last before registry.

Not covered¶

pandas.Index, polars.LazyFrame, pyarrow scalars: UnsupportedTypeError. A LazyFrame could be rounded with a map_batches.
Decimal, Fraction, complex numbers, dates, times and durations: not numerical in the sense of the package. Decimal is the example of the extension guide.
JAX, xarray, CuPy: a rounder is about 20 lines on the pattern of round_torch; registered by a plugin or by the user.
Sparse, nested and quantized tensors: UnsupportedTypeError.
Rounding to significant digits, "trunc" and "half_up" modes: new RoundMode values and the same kernel with a different last step.
In-place rounding (out=): the package is pure by design.

Architecture of coola.rounding¶