Migration Guide: v1.1 to v2.0¶
This guide explains how to migrate your code from coola version 1.1.x to version 2.0.x.
Version 2.0 is a large release. Most code that only calls objects_are_equal, hash_object or
summarize keeps working, but the results of some calls change (hashes, summaries, a few
comparisons). Code that extends coola (custom testers, hashers, summarizers, transformers, child
finders, loaders or registries) must be ported.
How to use this guide¶
- Read Before you start and check the requirements.
- Go through the checklist and keep the items that match what you use.
- Open the section of each item: every section starts with who is affected, then lists the changes in a table (Before / After) and shows a code example when a port is needed.
At a glance¶
| Area | Main change | Section |
|---|---|---|
| Python and extras | Python 3.11 or later; new scipy and testing extras |
1 |
| Exceptions | AlreadyRegisteredError, MissingPackageError and MaxDepthExceededError replace RuntimeError / RecursionError |
2 |
coola.registry |
One storage class (Registry) and one dispatch class (TypeRegistry), fallbacks, pickling |
3 |
coola.equality |
One engine; a tester returns a verdict or child pairs; compare replaces show_difference |
4 |
coola.hashing |
One engine and encode; every hash changes |
5 |
coola.summary |
One engine, one option object (SummaryConfig); new layout |
6 |
coola.recursive |
Transformers implement expand; the engine recurses |
7 |
coola.iterator |
One child finder (find_children) for dfs_iterate and bfs_iterate |
8 |
coola.factory |
Strict contracts: one way to import, call and check a target | 9 |
coola.random |
One seed domain, a strict set_rng_state, one registry |
10 |
coola.reducer |
BaseReducer owns the checks; strict input |
11 |
coola.nested |
Built on the engines; keyword-only options; stricter conversions | 12 |
coola.utils |
Leaf layer: formatting and registry helpers moved; import helpers replaced | 13 |
coola.display, coola.testing |
Optional rich; marks and fixtures modules |
14, 15 |
coola.io, coola.identifier, coola.rounding, coola.validation |
New packages (nothing to migrate from the release 1.1.10) |
16 to 19 |
Before you start¶
Requirements. coola 2.0 requires Python 3.11 or later (section 1).
Baseline. The baseline is the release 1.1.10. Some tables also compare with states of the
development branch between 1.1.10 and 2.0, so a "Before" cell can name something the release never
had. If you only used the release 1.1.10, skip these parts:
- The packages
coola.io(section 16),coola.identifier(17),coola.rounding(18),coola.validation(19),coola.formattingandcoola.display.richdid not exist. They are new (see section 20). - These names were not in
1.1.10:DataclassMarker,BaseTypeDispatchRegistry,resolve_or,register_fallback,register_lazy,enter_dataclasses,SupportsHash,HashableHasher,_allowed_prefixes_,is_object_config,sample_torch_seed,LazySingleton,add_prefix_suffix_to_keys,coola.testing.registries,coola.testing.marks.requiresandplural=,smallest_itemsandint_repr.
The rows that name a class, a function or an argument of 1.1.10 are the ones to apply.
Migration checklist¶
Everyone¶
- [ ] Use Python 3.11 or later (section 1).
- [ ] Regenerate every stored hash and stable identifier: the hash format changed (section 5).
- [ ] Pass the options of
objects_are_equal,hash_object,recursive_apply,random_seed,dfs_iterate,bfs_iterateand theregister_*functions by keyword (sections 3 to 10). - [ ] Replace
show_difference=Truebycompare(...)orassert_objects_equal(...)(section 4). - [ ] Catch
AlreadyRegisteredError,MaxDepthExceededErrorandMissingPackageErrorinstead ofRuntimeError/RecursionError, and do not rely onmax_depth=1000(section 2). - [ ] Read the new summary layout if you compare or parse summary text (section 6).
If you extend coola¶
- [ ] Equality: rewrite testers and handlers as testers that return a verdict or child pairs (section 4).
- [ ] Hashing: port hashers to
encode, replaceSupportsHash.hashby__coola_hash__, move the secret policy to the registry (section 5). - [ ] Summary: port summarizers to
summarize(data, config, *, compact)(section 6). - [ ] Recursive: port transformers to
expand(section 7). - [ ] Iterator: port iterators and child finders to
find_children(section 8). - [ ] Registry: replace
DataclassMarker,find,resolve_orandBaseTypeDispatchRegistry(section 3). - [ ] Reducer: subclass
BaseReducerinstead ofBaseBasicReducer(section 11). - [ ] I/O: rename
loadto_loadin custom loaders and drop theirequal(section 16).
If you use a specific package¶
- [ ] Factory: replace
_init_: __new__and the_init_names that start with_, and resolve nested configurations before passing them under an allow-list (section 9). - [ ] Random: replace the removed names and give
set_rng_statea state with the registered keys (section 10). - [ ] Reducer: pass
descendingby keyword and createTorchReducer()explicitly for tensors (section 11). - [ ] Nested: pass the options by keyword, import
get_first_valuefromcoola.utils.mapping, and read the stricter conversions (section 12). - [ ] Utils: update the import paths, replace the import decorators and the
raise_<pkg>_missing_errorfunctions, and read the smaller behaviour changes (section 13). - [ ] Display: pass
forceofconfigure_colorlog_loggingby keyword and read the new pydantic output (section 14). - [ ] Testing: import the marks from
coola.testing.marksand load the fixture withpytest_plugins(section 15); install thetestingextra (section 1). - [ ] SciPy sparse matrices: install the
scipyextra to compare them (section 1).
Detailed changes¶
Sections 1 to 3 apply to the whole library. Sections 4 to 11 follow the packages in order of use,
section 12 covers coola.nested, sections 13 to 15 the helpers, and sections 16 to 19 the new packages.
Part 1: Whole library (sections 1 to 3)¶
1. Packaging: Python 3.11, New Extras¶
Affects: Everyone who installs coola.
| Before | After |
|---|---|
requires-python = ">=3.10" |
>=3.11 (Python 3.15 is supported) |
| No extra for SciPy | pip install coola[scipy] (scipy>=1.13): the equality tester of the sparse matrices |
coola.testing imports pytest with no declared dependency |
pip install coola[testing] (pytest>=8.0) |
polars>=1.0,<2.0 |
polars>=1.0,<3.0 |
torch extra on every Python version |
Not installed on Python 3.15 (python_full_version < '3.15' marker) |
dependencies = [] |
Unchanged: coola still has no required dependency |
The jax, numpy, packaging, pandas, pyarrow, pydantic, xarray, colorlog and rich
extras are unchanged. The development tools changed (ty replaces pyright; black and mutmut are
removed), which matters to contributors only.
2. Exceptions That Changed Type¶
Affects: Code that catches RuntimeError, KeyError or RecursionError raised by coola.
| Before | After |
|---|---|
registry.register(...), register_many(...) and the register_* functions raise RuntimeError for a duplicate |
AlreadyRegisteredError, a ValueError: except RuntimeError no longer catches it (exist_ok=True is unchanged) |
registry[key], unregister(key) raise a KeyError |
KeyNotRegisteredError, or TypeNotRegisteredError for a type: both are KeyError (the message is no longer escaped by repr) |
raise_package_missing_error, check_package and raise_<pkg>_missing_error raise RuntimeError |
MissingPackageError, a subclass of ModuleNotFoundError and RuntimeError: both except clauses still work |
objects_are_equal and objects_are_allclose raise RecursionError beyond max_depth=1000 |
MaxDepthExceededError, an Exception, beyond max_depth=100: raise max_depth for a deeper structure |
EmptySequenceError is an Exception |
A ValueError |
timeblock("no placeholder") raises RuntimeError |
ValueError (the message must contain {time}) |
| A rename that gives two equal keys keeps one of them | DuplicateKeyError, a KeyError and a ValueError (section 12.3) |
RecursionError for a deep structure in recursive_apply, dfs_iterate, summarize and the coola.nested functions |
No limit: the engines use an explicit stack (a cycle raises ValueError) |
The messages of the validations changed too (section 19): a test that matches a message must be updated.
3. coola.registry: Two Classes, Fallbacks, Pickling¶
Affects: Code that builds, subclasses or registers into a registry, or catches its errors.
| Before | After |
|---|---|
registry.register(DataclassMarker, handler) |
registry.register_fallback(dataclasses.is_dataclass, handler) |
from coola.registry import DataclassMarker |
Removed; the dataclass handler is in registry.fallbacks(), not in items() / len() |
BaseTypeDispatchRegistry, registry.find(t), registry.resolve_or(t) |
Removed; the registries of the packages are TypeRegistry subclasses: use registry.resolve(t) |
registry.register(key, value, True), register_many(mapping, True) |
exist_ok is keyword-only: register(key, value, exist_ok=True) |
registry.equal(other, equal_nan=True) |
registry.equal(other): the entries are compared with ==, and a registry equals its copy |
A duplicate registration raises RuntimeError; a missing key raises KeyError |
AlreadyRegisteredError (a ValueError, not a RuntimeError); KeyNotRegisteredError (a KeyError) |
A hasher for object makes the hashing fallbacks unreachable |
object is the last resort everywhere: type > fallback > object |
HasherRegistry.register_fallback(predicate, hasher) accepts a duplicate predicate |
A duplicate predicate raises AlreadyRegisteredError (exist_ok=True replaces it, keeping its place) |
HasherRegistry.unregister_fallback(predicate) returns None |
It returns the hasher and raises KeyNotRegisteredError for an unknown predicate |
repr(registry) printed (state): TypeRegistry(...); a hashing fallbacks list |
One display: one (key): value line per entry, then (fallback) lines and the pending lazy groups |
register_lazy(["builtins", "decimal"], f) then register_lazy("builtins", g) failed |
Both groups stay and load; register_lazy is a method of TypeRegistry only |
pickle.dumps(registry) and copy.deepcopy(registry) raised TypeError |
They work; a spawn worker can receive a registry |
Before:
from coola.registry import DataclassMarker
registry.register(DataclassMarker, MyDataclassHandler())
After:
from dataclasses import is_dataclass
registry.register_fallback(is_dataclass, MyDataclassHandler())
A handler registered for the type of a dataclass still has priority over the fallback. A fallback
also supports the virtual subclasses of an abstract base class (the MRO lookup does not call
issubclass):
registry.register_fallback(lambda t: issubclass(t, Sequence), handler).
Part 2: Core packages (sections 4 to 11)¶
4. coola.equality: One Engine, Testers That Return a Verdict or Child Pairs¶
Affects: Custom equality testers and handlers, code that uses show_difference=True, and tests that depend on comparison edge cases.
The handler chain is replaced by testers that use an engine. A tester never recurses and
never checks the identity, the type, the depth or the cycles: the engine does. The tester receives
two objects of the same type and the engine (ctx), and returns either a verdict or the child pairs
(actual, expected, path part) that must all be equal.
Example (v2.0):
>>> from coola.equality import AttributeName, BaseEqualityTester, compare, objects_are_equal
>>> class Point:
... def __init__(self, x, y):
... self.x = x
... self.y = y
>>> class PointEqualityTester(BaseEqualityTester[Point]):
... def objects_are_equal(self, actual, expected, ctx):
... return (
... (actual.x, expected.x, AttributeName("x")),
... (actual.y, expected.y, AttributeName("y")),
... )
>>> from coola.equality import get_default_registry
>>> registry = get_default_registry().copy()
>>> registry.register(Point, PointEqualityTester()) # or register_equality_testers({...})
>>> objects_are_equal(Point(1, 2.0), Point(1, 2.0), registry=registry)
True
>>> result = compare(Point(1, 2.0), Point(1, 3.0), registry=registry)
>>> result.path
('y',)
A leaf tester returns a verdict and gives the reason with ctx.fail, which returns False. The
reason can be a function: it is called only when the difference is explained.
class VersionEqualityTester(BaseEqualityTester[Version]):
def objects_are_equal(self, actual, expected, ctx):
return actual.number == expected.number or ctx.fail(
lambda: f"versions differ: {actual.number} vs {expected.number}"
)
Migration of the v1.1 API (handlers and chain testers):
| Before | After |
|---|---|
BaseChainEqualityTester, HandlerEqualityTester, create_chain |
A BaseEqualityTester with one method objects_are_equal(a, e, ctx) |
BaseEqualityHandler, SameObjectHandler, SameTypeHandler, ... |
Removed: the engine checks the identity and the type |
config.registry.objects_are_equal(a, b, config) (recursion) |
Return the child pair (a, b, path part) |
report_difference(config, message, path=...) |
return ctx.fail(reason, *path) |
check_recursion_depth, track_comparison, HandlerCycleError |
Removed: the engine owns the depth and the cycles |
tester.equal(other) |
tester == other (same type and same attributes) |
EqualityConfig(atol=..., rtol=...), config.reports |
EqualityConfig(tolerance=Tolerance(...)); the config is frozen |
Engine(registry, config) |
Engine(config, explaining=False) |
PandasDataFrameEqualityTester, PolarsSeriesEqualityTester, ... |
TableEqualityTester(adapter), registered by the default registry |
from coola.equality.handler import ... |
Private (coola.equality._helpers); use coola.equality |
RecursionError when max_depth is exceeded (default 1000) |
MaxDepthExceededError, a plain Exception (default max_depth=100) |
objects_are_equal(..., show_difference=True) |
Removed (TypeError): see "Explaining a difference" below |
BaseEqualityHandler and the handlers: SameObjectHandler, SameTypeHandler, SameLengthHandler, SameAttributeHandler, SameShapeHandler, SameDTypeHandler, SameDataHandler, TrueHandler, FalseHandler, EqualHandler, EqualNanHandler, ScalarEqualHandler, NanEqualHandler, ObjectEqualHandler, TolerantEqualHandler, AllCloseNanHandler, MappingSameKeysHandler, MappingSameValuesHandler, SequenceSameValuesHandler and the handlers of NumPy, JAX, torch, pandas, polars and PyArrow |
Removed: a tester returns a verdict or child pairs |
The protocols SupportsEqual, SupportsEqualNan, SupportsAllCloseNan, SupportsTolerantEqual, SupportsShape, SupportsDType, SupportsData |
Removed: the objects keep the methods (equal(other), equal(other, equal_nan), allclose(other, rtol, atol, equal_nan)) and EqualEqualityTester, EqualNanEqualityTester and TolerantEqualEqualityTester call them (a protocol is not needed to register a type) |
PandasDataFrameEqualityTester, PandasSeriesEqualityTester, PolarsDataFrameEqualityTester, PolarsSeriesEqualityTester, PolarsLazyFrameEqualityTester, PyarrowEqualityTester |
TableEqualityTester(adapter) (a polars LazyFrame is compared like any other table) |
frame_equal, series_equal, handlers_are_equal, get_array_impl_class |
Removed |
Changes of the results:
The rules are the same for every container and every library. A comparison can change in these cases:
booland the members of anenum.Enumare labels: the tolerance never applies to them (alsoIntEnum,numpy.bool_and pandas booleans).- An integer difference is computed in integers:
2**62and2**62 + 1are different withatol=0.5in an array, as they are in a list. float16,bfloat16andfloat32are compared in double precision (no wrongTrueat the edge of the range).MappingProxyTypeandarray.arrayare compared like a mapping and a sequence.- pandas, polars and PyArrow numbers follow the tolerance; the nested and
objectcells are compared by the library (equals), without tolerance. The tolerance does not apply to an index. - A table (data frame, series, index, Arrow table) is one leaf of the explanation: the path stops at the table and the reason names the first column that differs.
OrderedDictkeeps its order, adictdoes not. A cycle is equal. A nesting deeper thanmax_depthraisesMaxDepthExceededError.
Explaining a difference: show_difference removed.
show_difference=True (a log message) is replaced by two functions that explain the first
difference only, on demand: compare returns a ComparisonResult and assert_objects_equal /
assert_objects_allclose raise an AssertionError. objects_are_equal and objects_are_allclose
format nothing.
Before:
import logging
logging.basicConfig(level=logging.INFO)
objects_are_equal(actual, expected, show_difference=True)
After:
>>> from coola.equality import compare, assert_objects_equal
>>> result = compare({"a": [1, 2, 3]}, {"a": [1, 2, 4]})
>>> result.equal
False
>>> result.path
('a', 2)
>>> print(result.format_message())
objects are not equal at actual['a'][2]
numbers are different:
actual : 3
expected : 4
>>> assert_objects_equal({"a": 1}, {"a": 2})
Traceback (most recent call last):
...
AssertionError: objects are not equal at actual['a']
numbers are different:
actual : 1
expected : 2
compare(actual, expected, rtol=0.0, atol=0.0, ...) takes the same keywords as
objects_are_allclose (exact by default). ComparisonResult has equal, path (keys, positions,
AttributeName, Element) and reason, and no actual / expected fields.
5. coola.hashing: One Engine, encode, a New Format¶
Affects: Everyone who stores a hash or a stable identifier, custom hashers, and code that handles secrets.
Every hash value changes (new encoding, format version coola-hash-2, short leaves inlined in the preimage of their parent), so every stable
identifier changes too: generate_stable_content_id and generate_stable_uuid5 return other values.
Regenerate any stored hash or identifier.
| Before | After |
|---|---|
Hasher.hash(data, registry, *, length, ignore_unhashable) |
Hasher.encode(data): returns tagged(tag, payload) (a leaf) or Node(tag, children) |
A hasher calls registry.hash(child, length=..., ignore_unhashable=...) |
The children are in the Node; the engine hashes them |
hash_string, hash_bytes, hash_tagged, hash_enum, hash_path |
Removed: use hash_object; tagged and digest are the building blocks of a hasher |
SupportsHash.hash(registry, length, ignore_unhashable), HashableHasher |
__coola_hash__(self) -> object (the content, not a hash), SupportsCoolaHash, ContentHasher |
PydanticModelHasher(on_secret=...), hash_pydantic_model, unwrap_secrets |
SecretHasher(on_secret) and get_secret_hashers(on_secret), registered in a registry |
"exclude" removed the key or the item of a secret |
"exclude" hashes a constant: the position of a secret is kept |
A secret in a dict or a dataclass raised, or was TypeNotRegisteredError |
The policy of the registry applies wherever the secret is |
hash_object(data, registry, 32), register_hashers(mapping, True) |
Keyword-only: hash_object(data, registry=registry, length=32), register_hashers(mapping, exist_ok=True) |
HasherRegistry.hash(data, 16) |
HasherRegistry.hash(data, length=16) |
ValueError("... nested too deeply") at about 500 levels |
No limit: the traversal uses an explicit stack |
| A DAG costs one visit per path | A shared container is hashed once |
length applied to every node: hash(x, length=16) was not a prefix of length=128 |
One digest cut at the end: a hash is the start of the longer ones |
A dataclass field with compare=False was hashed |
Only the compare=True fields are hashed |
memoryview hashed like bytes; UserString and MappingProxyType failed |
Own tag (memoryview), UserString and MappingProxyType are hashed |
A pandas object named NaN and one named None had different hashes |
Same hash |
Before:
class PointHasher(BaseHasher[Point]):
def hash(self, data, registry, *, length=64, ignore_unhashable=False):
return registry.hash((data.x, data.y), length=length, ignore_unhashable=ignore_unhashable)
After:
class PointHasher(BaseHasher[Point]):
def encode(self, data):
return Node("Point", [data.x, data.y])
To reveal or exclude secrets, build a registry:
registry = get_default_registry().copy()
registry.register_many(get_secret_hashers("reveal"), exist_ok=True)
hash_object(model, registry=registry)
6. coola.summary: One Engine, One Option Object¶
Affects: Custom summarizers, code that passes options to a summarizer, and code that parses summary text.
| Before | After |
|---|---|
Summarizer.summarize(data, registry, depth, max_depth) returns a str and recurses |
Summarizer.summarize(data, config, *, compact) returns a str (a leaf) or a Node (a container) |
registry.summarize(data, depth, max_depth) |
registry.summarize(data, config=None) with a SummaryConfig |
SequenceSummarizer(max_items=3), DefaultSummarizer(max_characters=50), num_spaces=4 |
summarize(data, max_items=3, max_characters=50); the indentation is 2 spaces and is not an option |
NDArraySummarizer(show_data=True), PandasDataFrameSummarizer(max_fields=2) |
summarize(data, show_data=True), summarize(data, max_items=2) |
max_items=-1 and max_characters=-1 for "no limit" |
max_items=None and max_characters=None |
summarize(data, 2) and register_summarizers(mapping, True) |
TypeError: summarize(data, max_depth=2) and register_summarizers(mapping, exist_ok=True) |
max_depth=True and max_depth=1.5 accepted |
TypeError |
BaseCollectionSummarizer, BaseFieldsSummarizer, MAX_FIELDS |
Removed. format_fields(fields, max_fields) has no default for max_fields |
BaseMetadataSummarizer(show_data) with _describe(data) |
BaseMetadataSummarizer() with _describe(data) returning fields; a table returns ("columns", pairs) |
repr and equal to write in every summarizer |
BaseSummarizer gives Name() and "same type" |
RecursionError for a nesting of a few hundred levels |
No RecursionError: the engine uses an explicit stack |
A self-referencing container is repeated up to max_depth |
<cycle> where it comes back |
A __len__, a property or a __iter__ that raises makes summarize raise |
<class ...> <summary failed: ExcType> and the rest of the summary is intact |
field(repr=False) fields of a dataclass are printed |
They are not shown |
UserString is a list of characters, MappingProxyType is printed whole |
A leaf, and a mapping |
A named tuple has the labels (0), (1) |
The field names |
A container at the display depth is [0, 1, 2, 3, 4] ..., its str is computed in full |
[0, 1, 2, 3, 4, ...], bounded work (three levels, max_items items) |
Hidden entries: ...; empty container: <class 'list'> []; nested levels indented by 2 then 4 |
... (+N more); <class 'list'> (length=0); 2 spaces per level |
| A frame, a table or a dataset lists 10 columns | It lists max_items (5) columns |
A custom summarizer that returned a string only needs the new signature:
from coola.summary import BaseSummarizer, Node, SummaryConfig
class PointSummarizer(BaseSummarizer[Point]):
def summarize(self, data: Point, config: SummaryConfig, *, compact: bool) -> str:
return f"Point(x={data.x}, y={data.y})" # a leaf
class TreeSummarizer(BaseSummarizer[Tree]):
def summarize(self, data: Tree, config: SummaryConfig, *, compact: bool) -> Node:
# a container: the engine recurses, summarizes the children with the summarizer of their
# type and applies max_items, max_depth and the cycle check
return Node(
f"Tree (children={len(data.children)})", enumerate(data.children), len(data.children)
)
A summarizer must be pure and cheap: it does not call the registry, does not recurse and returns
the children lazily. A plugin of the entry-point group coola.summary.summarizers that implemented
summarize(data, registry, depth, max_depth) must be ported. The options that were per type
(max_items=3 for lists and 10 for dicts) do not exist any more: register a summarizer that
returns a Node with a smaller total if you need it. The text of a summary is meant to be read,
not parsed, and its layout changes in this release.
7. coola.recursive: the expand Protocol¶
Affects: Custom transformers, and code that relied on dataclasses being leaves.
| Before | After |
|---|---|
Transformer.transform(data, func, registry) |
Transformer.expand(data): returns None (a leaf) or (children, rebuild) |
A transformer recurses through registry.transform |
The engine recurses: a transformer cannot forget the cycle check or the depth limit |
A dataclass is a leaf; enter_dataclasses() enters it |
A dataclass is entered by default; enter_dataclasses is removed |
dfs_iterate and bfs_iterate: a dataclass is a leaf; enter_dataclasses() enters it |
A dataclass is entered by default; enter_dataclasses is removed from coola.iterator |
dataclasses.replace rebuilds a dataclass |
A shallow copy with the fields set: init=False fields are transformed, no __post_init__ |
ConditionalTransformer |
Removed: guard func, or register IdentityTransformer for the type |
recursive_apply(data, func, registry) |
recursive_apply(data, func, registry=registry) |
KeyFilterTransformer() with func as the key predicate |
Removed (see section 12.3): use coola.nested.remove_keys_if |
register_transformers(mapping, True) |
register_transformers(mapping, exist_ok=True) |
RecursionError for a deep structure |
No limit: the traversal uses an explicit stack |
Before:
class PointTransformer(BaseTransformer[Point]):
def transform(self, data, func, registry):
return Point(registry.transform(data.x, func), registry.transform(data.y, func))
After:
class PointTransformer(BaseTransformer[Point]):
def expand(self, data):
return [data.x, data.y], lambda xs: Point(*xs)
A plugin of the group coola.recursive.transformers that implements transform must be ported to
expand. To keep a dataclass as a leaf, register DefaultTransformer() for its type (DefaultChildFinder() for
dfs_iterate and bfs_iterate).
8. coola.iterator: One Child Finder for Both Orders¶
Affects: Custom iterators and child finders, and plugins of the groups coola.iterator.dfs and coola.iterator.bfs.
| Before | After |
|---|---|
coola.iterator.dfs and coola.iterator.bfs |
Removed: import from coola.iterator |
BaseIterator.iterate(data, registry) (DFS) and BaseChildFinder.find_children (BFS) |
BaseChildFinder.find_children(data): returns the children, or None for a leaf |
IteratorRegistry, PathRegistry |
Removed: ChildFinderRegistry serves both orders |
register_iterators and register_child_finders |
register_child_finders(mapping, *, exist_ok=False) |
get_default_iterator_registry() |
get_default_registry() (alias get_default_child_finder_registry()) |
DefaultIterator, IterableIterator, MappingIterator, DataclassIterator |
DefaultChildFinder, IterableChildFinder, MappingChildFinder, DataclassChildFinder |
A leaf overrides is_leaf (BFS) |
A leaf returns None from find_children; an empty container returns an empty iterable |
registry.iterate(data) and registry.find_children(data) |
Removed: use dfs_iterate(data, registry=registry) or bfs_iterate(data, registry=registry) |
dfs_iterate(data, registry) and bfs_iterate(data, registry) (deprecated) |
registry is keyword-only |
Plugin groups coola.iterator.dfs and coola.iterator.bfs |
One group: coola.iterator.child_finders |
RecursionError for a dfs_iterate structure nested about a thousand levels |
No limit: both orders use an explicit stack or queue |
range, a user Collection and a user collections.abc.Set are entered |
They are leaves, like in recursive_apply (register a child finder to enter them) |
bfs_iterate could raise a false Cycle detected for objects built by a child finder |
Fixed: the ancestors are compared by identity |
Before:
class ReversedListIterator(BaseIterator[list]):
def iterate(self, data, registry):
for item in reversed(data):
yield from registry.iterate(item)
After:
class ReversedListChildFinder(BaseChildFinder[list]):
def find_children(self, data):
return reversed(data)
A child finder returns the children in the order they are visited, and the same order is used by
dfs_iterate and bfs_iterate: a custom order for one traversal only is no longer possible. A
plugin of the group coola.iterator.dfs or coola.iterator.bfs must register a ChildFinder in
coola.iterator.child_finders instead.
9. coola.factory: Strict Contracts¶
Affects: Configurations that use _init_ or _allowed_prefixes_, and code that nests configurations.
| Before | After |
|---|---|
A target cannot receive an argument named obj or cls |
It receives every key that is not reserved, whatever its name |
_init_="__new__" or a name that starts with _ |
TypeError: use a public class method or static method |
_allowed_prefixes_ in the configuration of resolve_object (used when the caller has none) |
Ignored: the allow-list always comes from the caller |
factory(..., _allowed_prefixes_=[...]) with an argument that is {"_target_": ...} |
ValueError: resolve the nested configuration first |
resolve_object with a nested configuration and allowed_prefixes |
ValueError (same reason) |
A dict with _target_ and a dict subclass cls |
Built (the key _target_ is reserved) |
A dict without _target_ |
Returned if it is an instance of cls, else TypeError |
Counter({"_target_": 2}) passed to resolve_object |
Resolved as a configuration |
| Missing module or attribute of a target | ImportError: The target object does not exist: ... everywhere |
A target that returns A \| B with no subclass of cls |
Refused before the call |
is_object_config({"_target_": 5}, int) |
False (was TypeError) |
Explicit _init_="__init__" with resolve_object |
The target is checked, like the default |
Before:
resolve_object({"_target_": "mypkg.Model", "_init_": "__new__"})
After:
resolve_object({"_target_": "mypkg.Model", "_init_": "from_config", "config": {...}})
An allow-list is a filter on a path, not a sandbox: never allow a prefix that contains a function
that resolves its arguments ("coola", "coola.factory", pickle, eval, ...).
10. coola.random: One Seed Domain, One Registry¶
Affects: Code that seeds NumPy or PyTorch through coola.random, and saved RNG states.
| Before | After |
|---|---|
RandomRandomManager (coola.random.random) |
PythonRandomManager (coola.random.python) |
numpy_seed(seed) and torch_seed(seed) |
NumpyRandomManager().seed(seed) and TorchRandomManager().seed(seed) |
random_seed(seed, manager) |
random_seed(seed, *, registry=None) |
register_managers(mapping, True) |
register_managers(mapping, exist_ok=True) |
manager.manual_seed(-1), 1.5, True, 2**63 accepted or folded |
TypeError / ValueError: every entry validates the seed |
NumPy: manual_seed(2**32 + 1) gives the numbers of manual_seed(1) |
A seed >= 2**32 is split in two 32-bit words: its numbers change |
registry.has_manager(key) and registry.find_manager(key) |
key in registry (or registry.has(key)) and registry[key] |
set_rng_state with different keys logs a warning and restores a part of the state |
ValueError, nothing restored; all the managers are restored before an error is raised |
get_random_seed(seed) in [-2**63, 2**63 - 1) |
In [0, 2**63 - 1), valid for manual_seed: the value for a given seed changes |
get_torch_generator(random_seed=1, device) (not exported) |
make_torch_generator(seed, *, device="cpu") from coola.random; seed is required |
repr(RandomManagerRegistry) has a (state): Registry(...) line |
The managers are listed directly |
A state saved with torch and restored on a machine without torch now raises a ValueError. Restore
it into a registry with the same keys.
11. coola.reducer: One Base Class That Owns the Checks¶
Affects: Custom reducers, and code that calls auto_reducer() for tensors.
The changes of coola.reducer:
auto_reducer()returnsNumpyReducerif NumPy is installed, elseNativeReducer. It no longer returnsTorchReducerwhen PyTorch is installed (it is slower on a list and slow to import): createTorchReducer()explicitly for tensors that live on a device.BaseBasicReduceris removed. SubclassBaseReducer, which now owns the checks (empty input, iterator, options, shape) and calls the private computations_max,_mean,_median,_min,_quantile,_stdand_sort._is_emptyand_check_valuesare removed; implement_prepareto convert or validate the input if needed.sort(values, *, descending=False):descendingis keyword-only.EmptySequenceErroris aValueError.- The input is a sequence of real numbers (
list,tuple,range) or a one-dimensional array / tensor for the backend. An iterator, a set, astr, an n-d or 0-d array, a complex number, an integer beyond 64 bits (NumPy and PyTorch) or aDecimalraiseTypeErrororValueErrorwith the same text in every backend (they used to return a value or raiseAttributeError,RuntimeError, ...).NativeReducerrejects arrays and tensors: useNumpyReducerorTorchReducer. quantileraisesValueErrorfor a quantile outside[0, 1](orNaN) in the three backends, before the values are read.NativeReducer.meanusesstatistics.fmean: it is much faster and the last bit of a float mean can differ.
Part 3: Nested data (section 12)¶
12. coola.nested: Built on the Engines¶
Affects: Users of coola.nested (flattening, key filtering, mapping helpers) and of the JSON conversion.
12.1 to_flat_dict on dfs_items¶
to_flat_dict joins the paths given by coola.iterator.dfs_items instead of walking the data with
its own recursion.
| Before | After |
|---|---|
| A dataclass was a leaf of the flat dict | It is entered: one entry per field ("p.x", "p.y") |
A range was entered ("r.0", "r.1", ...) |
A leaf, like in the other packages |
RecursionError for a deep structure |
No limit on the depth |
| Its own cycle check | The cycle check of the iterator: ValueError |
to_flat_dict(data, prefix, separator, to_str) (positional) |
Keyword-only options |
12.2 Keyword-only options, smaller public surface¶
The options of the coola.nested functions are keyword-only, get_first_value moves next to
sort_by_keys, and the helpers that only served one function are private.
| Before | After |
|---|---|
to_flat_dict(data, "prefix", "/", str) |
to_flat_dict(data, prefix="prefix", separator="/", to_str=str) |
flatten_mapping(mapping, "prefix", True, "/") |
flatten_mapping(mapping, on_duplicate="prefix", always_prefix=True, separator="/") |
merge_mappings(mappings, "last") |
merge_mappings(mappings, on_duplicate="last") |
add_prefix_suffix_to_keys(mapping, "x_", "_y", True) |
add_prefix_suffix_to_keys(mapping, "x_", "_y", recursive=True) |
from_flat_dict(data, ".") |
from_flat_dict(data, separator="."); a key that is not a str raises TypeError |
merge_mappings(..., on_duplicate="x"), flatten_mapping(..., on_duplicate="x") with a string |
Same call; the annotation is a Literal, and a new keyword equal_nan decides whether two NaN are equal duplicates |
from coola.nested import get_first_value |
from coola.utils.mapping import get_first_value |
coola.nested.try_sorted_list, coola.nested.conversion.SetToListTransformer |
Private: sort with sorted, or use coola.jsonable.to_jsonable for a set |
coola.nested.polars.{is_nested_struct, unnest_with_separator, unnest_one_level, expand_list_columns} |
Private: use flatten_frame |
12.3 Keys on the engine¶
remove_keys_* and add_prefix_suffix_to_keys(recursive=True) use the engine of coola.recursive
and its default registry, instead of a traversal of their own.
| Before | After |
|---|---|
coola.recursive.KeyFilterTransformer(key_predicate) |
Removed: coola.nested.remove_keys_if(mapping, predicate) |
A dataclass was a leaf of remove_keys_* and add_prefix_suffix_to_keys |
It is entered, and rebuilt with the filtered or renamed fields |
The nested OrderedDict, defaultdict were rebuilt as a dict (rename) |
The type of every container is kept, at every depth |
remove_keys_if([1], ...) returned the list; the annotation said dict |
TypeError for a non-mapping; the result has the type of the input |
RecursionError for a cycle or a deep structure |
ValueError for a cycle; no limit on the depth |
| A rename that gives two equal keys silently kept the last entry | DuplicateKeyError |
12.4 from_flat_dict without recursion¶
| Before | After |
|---|---|
from_flat_dict({"a": {"0": 1}}, restore_lists=True) gave {"a": [1]} |
{"a": {"0": 1}}: a dict given as a value is kept as it is |
RecursionError for about 3,000 levels with restore_lists=True |
No limit on the depth |
The annotation of data was dict[str, Any] |
Mapping[str, Any] (any mapping worked) |
12.5 convert_to_jsonable moved to coola.jsonable.to_jsonable¶
coola.nested.convert_to_jsonable and coola.nested.conversion.to_jsonable are removed, with no
alias. The conversion is coola.jsonable.to_jsonable (see Converting Data to JSON),
and coola/nested/conversion.py is now coola/nested/records.py.
| Before | After |
|---|---|
convert_to_jsonable(x) and to_jsonable(x) |
coola.jsonable.to_jsonable(x); it enters containers and dataclasses, so to_jsonable([Path("a")]) converts the path |
A value of an unknown type was returned unchanged (json.dumps failed later) |
TypeError that names the type. Register a ConvertTransformer with register_transformers, or pass on_unsupported="keep" for the old behaviour ("warn" shows which values are kept) |
A tuple, a deque, a named tuple or a range kept its type (a range became a list) |
A list. OrderedDict, defaultdict and Counter become a dict |
The keys were not converted: {(1, 2): 1} failed in json.dumps, {1: "a", "1": "b"} wrote "1" twice |
A key is a str, or written as json.dumps writes it ("true", "1", "null"); a Path, a UUID or an Enum key is converted; a tuple key raises TypeError; two keys that give one string raise ValueError |
The output of NumPy and PyTorch was not checked: [(1+2j)], the objects of an object array, a datetime.date for datetime64, timedelta, bytes |
Complex numbers and objects raise (by default); a datetime64 coarser than the nanosecond gives an ISO string, a timedelta64 gives seconds, np.bytes_ gives Base64. A nanosecond datetime64 still gives an int |
A pydantic model was dumped with model_dump(mode="json"): a NumPy field raised, a set field depended on PYTHONHASHSEED |
model_dump() (Python mode), converted by the registry: the sets are sorted and the arrays are converted. A serializer declared with when_used="json" is no longer applied (register ConvertTransformer(lambda m: m.model_dump(mode="json")) for the class) |
pd.NaT gave "NaT"; pd.NA, pd.Series and pd.Index were returned unchanged |
None, None, and a list. A DataFrame raises TypeError: register the shape you want (df.to_dict(orient="records")) |
to_jsonable(Point(1, 2)) converted the dataclass |
Same, and a set in a dataclass is sorted like every set, in every process |
| A tuple in a dataclass stayed a tuple | A list |
A set of mixed types ({1, "a"}), or with NaN, kept the iteration order |
Sorted by (type name, repr) |
RecursionError for 2,000 nested dataclasses |
No limit; a cycle through a dataclass raises ValueError |
coola.nested.try_sorted_list, coola.nested.conversion.SetToListTransformer |
Private |
NaN and the infinities are still kept (as json.dumps writes them by default): pass allow_nan=False
to json.dumps or save_json to refuse them.
12.6 Mappings and records¶
| Before | After |
|---|---|
merge_mappings([{"a": 1}, {"a": 2}, {"a_1": 3}], on_duplicate="suffix") raised |
{'a': 1, 'a_2': 2, 'a_1': 3}: a generated key skips the keys of the inputs |
flatten_mapping(on_duplicate="prefix") moved the renamed first occurrence to the end |
The first occurrence is renamed in place: the output follows the input order |
convert_to_list_of_dicts({"a": {1, 2}}) paired the elements in an arbitrary order |
TypeError that names the type; the values must be Sequence |
A generator as a value: TypeError: object of type 'generator' has no len() |
TypeError that names the type |
convert_to_dict_of_lists(generator): 'generator' object is not subscriptable |
Any iterable of mappings is accepted |
convert_to_dict_of_lists([{"a": 1, "b": 2}, {"a": 3}]) raised KeyError; an extra key of a later mapping was ignored |
ValueError that names the index and the keys: every mapping must have the keys of the first one |
convert_to_list_of_dicts({"a": "xyz"}) split the string per character |
TypeError for a str, bytes or bytearray value |
12.7 flatten_frame¶
| Before | After |
|---|---|
from coola.nested.polars import flatten_frame |
from coola.nested import flatten_frame (imports polars on first use) |
A LazyFrame was accepted (with a PerformanceWarning) |
TypeError: call collect() first |
The version differences of polars were found with except TypeError |
compare_version: unnest(separator=) from 1.35.0, explode(empty_as_null=) from 1.36.0; an unrelated TypeError propagates |
Part 4: Helpers (sections 13 to 15)¶
13. coola.utils: Leaf Layer, Formatting¶
Affects: Code that imports from coola.utils, uses its import helpers, or relies on the behaviour of its functions.
coola.utils imports the standard library and coola.validation only. Some modules moved, with
no alias:
| Before | After |
|---|---|
coola.utils.format |
coola.formatting.structure (str_*, repr_*) and coola.formatting.quantity (make_bar, byte sizes, durations) |
coola.utils.string, coola.utils.secret, coola.utils.text_diff |
coola.formatting.text, coola.formatting.secret, coola.formatting.diff |
coola.utils.conversion.to_jsonable |
coola.jsonable.to_jsonable |
coola.utils.singleton.make_default_registry_singleton, reset_default_registries, temporary_default_registries |
coola.registry.default.<name> (plugin_group is keyword-only) |
coola.utils.singleton.load_registry_plugins, get_loaded_plugins, LoadedPlugin, STRICT_PLUGINS_ENV_VAR, DISABLE_PLUGINS_ENV_VAR |
coola.registry.plugins.<name> |
LazySingleton._replace_with_copy, _restore |
LazySingleton.set(value) |
coola.utils.deprecation.deprecate_positional_options |
Removed: no function of coola is deprecated any more |
coola.utils.path.working_directory |
contextlib.chdir |
coola.utils.introspection.get_fully_qualified_name(obj) |
get_qualified_class_name(type(obj)) |
numpy_available, torch_available, ... (decorators that returned None), torch_numpy_available, decorator_package_available |
Removed. Use is_<pkg>_available() |
raise_numpy_missing_error(), ... and raise_package_missing_error(name, install_cmd) |
raise_package_missing_error(name, *, install=None) |
check_package(package, command) |
check_package(package, *, install=None): the message has the pip install and uv pip install commands, and the package is imported |
LazyModule, lazy_import, coola.utils.imports.import_object("pkg.mod:name") |
Removed. Use pkgutil.resolve_name("pkg.mod:name") |
patch("coola.utils.imports.numpy.is_numpy_available", ...) |
patch("coola.utils.imports.optional.is_numpy_available", ...) |
str_indent(x, 4), str_mapping(m, True), smallest_items(data, 3) |
str_indent(x, num_spaces=4), str_mapping(m, sorted_keys=True), smallest_items(data, limit=3) |
str_indent(x, num_spaces=-1): Incorrect num_spaces. Expected a non-negative integer...; quantile(v, [2.0]): quantiles must be in [0, 1] |
num_spaces must be greater than or equal to 0, got -1; quantiles must be in the range [0.0, 1.0], got 2.0 |
int_repr(-(1 << 20000)) returned 0x-1000... |
-0x1000...: the hash of a negative integer with more than 4300 digits changes |
smallest_items(iter([3, "a", 1])) returned []; LazySingleton(lambda: s.reset()) deadlocked |
smallest_items takes a collection; reset and set raise RuntimeError from the factory |
Rebuild error of recursive: Cannot rebuild an object of type builtins.frozenset ... |
... of type frozenset ... |
Smaller behaviour changes¶
The functions below keep their name and most of their behaviour.
| Before | After |
|---|---|
check_env_vars(["X"]) reports a variable set to "" or " " as defined |
It is reported as not defined, like get_required_env_var; new missing_level= (default logging.WARNING) and the log messages have no emoji |
to_date(datetime(2020, 1, 2, 3)) returns the datetime |
Returns date(2020, 1, 2) |
sort_by_keys({"a": 1, 1: 2}) raises TypeError |
Sorts by type name first, then by key (by repr if the keys of one type cannot be compared) |
sanitize_path("file://server/share/x") returns /share/x |
ValueError: a file:// URI with a host other than localhost is not supported |
sanitize_path(path) always resolves the path |
sanitize_path(path, resolve=False) keeps a relative path and the symbolic links |
find_root_package_parent("/dir/missing") starts from the path itself |
A path that is not a directory is treated as a file: the search starts from its parent |
quantile(values, [2.0]) returns the largest value, quantile([], [0.5]) raises IndexError |
ValueError for a quantile outside [0, 1] and for empty values |
batchify(items, size=0): size must be >= 1 |
size must be greater than or equal to 1, got 0 (ValueError, from coola.validation) |
get_package_version("pkg") raises InvalidVersion for a version that is not PEP 440 |
Returns None, like a package that is not installed; the cache is reset by clear_availability_caches() |
BloomFilter hashes with SHA-512; no add, no in |
BLAKE2b (the positions of the bits change: do not reuse a stored filter), add(item) and item in bloom, at most 32 hash functions |
to_array(torch.ones(2, dtype=torch.bfloat16)) raises TypeError |
Converted to float32 |
to_tensor(array) with a negative stride raises ValueError |
The array is copied; MissingPackageError if PyTorch is not installed |
hmac.compare_digest(password, confirmation) on str: TypeError for a non-ASCII password |
Compares the encoded bytes |
make_fake_class(raise_error), make_fake_function(raise_error) (coola.utils.fallback.factory) |
make_fake_class(package), make_fake_function(package): the fake raises MissingPackageError |
coola.__version__ is read when coola is imported |
Read the first time it is used (PEP 562): the value is the same |
14. coola.display: Optional rich, Pydantic Formatting¶
Affects: Users of coola.display, the logging helpers and the pydantic formatting.
import coola.display.rich works without rich and imports no rich module: the first call of a
function raises MissingPackageError.
| Before | After |
|---|---|
import coola.display.rich without rich: ModuleNotFoundError |
The import works; the first call raises MissingPackageError (pip install rich) |
import coola.display.rich imports rich |
It imports no rich module until a name is used |
configure_colorlog_logging(logging.INFO, True) |
TypeError: force is keyword-only |
configure_colorlog_logging always attached the coloured handler (raw ANSI escape codes in a file or in CI) |
The coloured handler is used only if colorlog is installed and the stream is a terminal; otherwise logging.basicConfig |
configure_colorlog_logging(level, force=...) had no other option |
**kwargs go to logging.basicConfig (stream, datefmt, ...); handlers, filename, filemode, format or style bypass the coloured handler |
str_pydantic_model(P(c=C(...))) shows c={'name': 'a'} |
Shows c=C(name=a): a nested model has its name, with the options of the root |
A field with Field(repr=False) is shown; a field_serializer runs |
The field is hidden like in the repr of pydantic; no user code runs |
A computed_field is shown (it is in model_dump()) |
It is not shown (it is not a field) |
print_markdown(msg, max_length=-1) cut from the end silently; an open code fence in the cut text swallows the notice |
ValueError for max_length < 1; the cut prefers paragraph and line breaks and closes a fence |
configure_rich_logging, make_progressbar, make_spinner always write to get_console() |
console= chooses the console; the default is unchanged |
15. coola.testing: marks and fixtures¶
Affects: Test suites that import marks or fixtures from coola.testing.
The two modules of coola.testing are renamed after what they contain, and the fixture is loaded
as a pytest plugin.
| Before | After |
|---|---|
from coola.testing.fixtures import numpy_available, requires |
from coola.testing.marks import numpy_available, requires |
from coola.testing.registries import isolated_default_registries |
pytest_plugins = ["coola.testing.fixtures"] in the top-level conftest.py |
requires(True, "PyTorch and CUDA", plural=True) |
requires(True, "PyTorch with CUDA"): plural= is removed |
| Reasons "Requires PyTorch and CUDA" / "Skip if PyTorch and CUDA are available" | "Requires PyTorch with CUDA" / "Skip if PyTorch with CUDA is available" (also for MPS and NumPy) |
The names of the marks (numpy_available, numpy_not_available, ...) are unchanged. The re-export
of the fixture in a conftest.py (with an __all__) is no longer needed.
Part 5: Packages that are new since 1.1.10 (sections 16 to 19)¶
16. coola.io: Template Methods, No Lock¶
Affects: Only code written against the development branch: the package is new since 1.1.10.
| Before | After |
|---|---|
BaseLoader.load(path) is abstract and each loader calls sanitize_path |
BaseLoader._load(path: Path) is abstract; load is concrete and converts the path |
Every loader and saver implements equal(other, equal_nan) |
equal is concrete in BaseLoader and BaseSaver: same type and same _display_fields() |
InlineDisplayMixin is a base of each format |
It is a base of BaseLoader and BaseSaver |
JsonLoader[T], PickleSaver[T], TorchLoader[T] |
BaseLoader[Any] / BaseFileSaver[Any]: the classes are not subscripted |
save_text(123, path) writes 123; save_text(None, path) writes None |
TypeError for anything but a str: call str() first |
TorchLoader(**kwargs), weights_only read from kwargs |
TorchLoader(*, weights_only=True, **kwargs); shown in the repr, compared by equal() |
from coola.io import add_uuid_suffix; coola.io.utils |
from coola.utils.path import add_uuid_suffix |
save_*(..., lock=True), BaseFileSaver.save(..., lock=True) |
lock is removed (TypeError); no .lock file, no heartbeat thread |
| Saves of one path are serialized | Saves are atomic and never mix; which one wins is unspecified |
exist_ok=False without hard links is protected by the lock |
Two processes creating the same file at the same instant can both succeed (last wins) |
_fsync_file opens the file read-only |
It opens it with O_RDWR |
Before:
class UpperTextLoader(BaseLoader[str]):
def equal(self, other, equal_nan=False):
return type(other) is type(self)
def load(self, path):
return sanitize_path(path).read_text().upper()
After:
class UpperTextLoader(BaseLoader[str]):
def _load(self, path):
return path.read_text().upper()
A caller that needs mutual exclusion around a read-modify-write of one file takes a lock of its own
around the read and the write: save never provided it. A temporary file .<name>-<uuid><ext> left
by a process killed during a save is hidden and safe to delete.
17. coola.identifier: Four Generators Removed, Strict Inputs, One Clock Rule¶
Affects: Only code written against the development branch: the package is new since 1.1.10.
generate_uuid4, generate_nano_id, generate_prefixed_id, generate_obfuscated_id and
decode_obfuscated_id are removed: each is a line of the standard library or, for the obfuscated
identifiers, an affine map that two known pairs break. The stable identifiers keep their values.
| Before | After |
|---|---|
generate_uuid4() |
str(uuid.uuid4()) |
generate_nano_id(n, a) |
secrets.token_urlsafe(16), or "".join(secrets.choice(a) for _ in range(n)) |
generate_prefixed_id |
f"cus_{generate_ulid()}" |
generate_obfuscated_id |
a library made for it (sqids), or a keyed permutation; never an affine map |
The inputs are strict, and the extractors are the strict inverse of the generators:
| Before | After |
|---|---|
extract_object_id_timestamp(object_id) |
extract_object_id_timestamp_s(object_id) (the unit is in the name) |
generate_ulid(True), generate_snowflake_id(worker_id=True) |
TypeError: a bool is not an int |
generate_ulid(1.5) raised AttributeError or TypeError from the code |
TypeError: timestamp_ms must be an int |
extract_object_id_timestamp accepted whitespace, upper case and bytes |
Only 24 lowercase hexadecimal characters; TypeError for anything but a str |
extract_uuid7_timestamp_ms accepted {...}, urn:uuid:..., no hyphens |
Only the canonical 36-character form (the case is ignored); pass str(uuid) |
extract_uuid7_timestamp_ms(uuid.UUID(...)) raised AttributeError |
TypeError |
extract_ulid_timestamp_ms rejected O, I and L |
They are read as 0, 1, 1 (Crockford), like verify_checksummed_id |
extract_snowflake_timestamp_ms documented 64 bits and checked 63 |
ValueError outside [0, 2**63); TypeError for a bool or a non-int |
ValueError: ... must fit in 48 bits |
ValueError: ... must be in the range [0, ...] (from coola.validation) |
| Before | After |
|---|---|
generate_ulid(ts), generate_uuid7(ts), generate_object_id(ts) |
Keyword-only: generate_ulid(timestamp_ms=ts), generate_object_id(timestamp_s=ts) |
ObjectIdGenerator, objectid.get_default_generator() |
Removed: use generate_object_id(); pass timestamp_s for a deterministic test |
| ObjectId: 5-byte per-process value and 3-byte counter, no order in a second | 4 bytes of seconds and a 64-bit incremented random counter: strictly increasing |
identifier.monotonic.MonotonicRandom |
identifier.sequence.TickSequence(bits=..., tick_ns=..., random_start=...) (internal) |
An ObjectId has the same 24 hexadecimal characters, the same timestamp field and the same sort order; only a tool that decodes the former process field and counter notices.
| Before | After |
|---|---|
generate_snowflake_id(worker_id, timestamp_ms), get_default_generator() |
Removed: SnowflakeIdGenerator(worker_id=...) created once per process |
SnowflakeIdGenerator().generate(worker_id, timestamp_ms) |
SnowflakeIdGenerator(worker_id=...).generate(timestamp_ms=...): no default worker_id |
SnowflakeIdGenerator(last_timestamp_ms, sequence) |
Removed |
RuntimeError("clock moved backward"); time.sleep when 4,096 IDs in a ms |
Never raises or sleeps: the timestamp is kept (or advanced) and the sequence goes on |
A forked child continued the sequence of its parent (same IDs with worker_id=0) |
RuntimeError in the child: create a generator with its own worker_id |
| Before | After |
|---|---|
generate_checksummed_id(12, 4, "-"), verify_checksummed_id(id, "-") |
Keyword-only: generate_checksummed_id(length=12, group_size=4, sep="-"), verify_checksummed_id(id, sep="-") |
5 of 37 identifiers ended with *, ~, $, = or U |
Never: the payload is drawn again; verify_checksummed_id still accepts the 5 symbols |
sep could be a lowercase letter or O, I, L |
ValueError: sep must not be a character of the alphabet (case-insensitive) |
verify_checksummed_id(id, sep=1) raised AttributeError |
TypeError (and ValueError for an invalid sep) |
18. coola.rounding: One Rule, One Registry¶
Affects: Only code written against the development branch: the package is new since 1.1.10.
round_values follows one rule: a float is read as the decimal that repr shows, and the three
modes act on that decimal (see the user guide).
| Before | After |
|---|---|
round_values(2.675, decimals=2) is 2.67 for a float, 2.68 for a NumPy array |
2.68 everywhere: the decimal is a halfway case, which goes to the even digit |
round_values(8.995, decimals=2) is 8.99 |
9.0 |
ceil(5.5200000000000005, 2) is 5.52 (below the input) |
5.53: floor(x) <= x <= ceil(x) always |
polars: round_values(pl.Series([-4.822]), decimals=1) is -4.800000000000001 |
-4.8: polars goes through the NumPy kernel, the results are the same as NumPy |
| Rounding a polars or pyarrow object does not need NumPy | It needs NumPy: without it UnsupportedTypeError (cause MissingPackageError), or on_missing_package="warn" / "ignore" |
round_values(-0.4, mode="ceil") is 0.0 for a float |
-0.0, like numpy.ceil: a zero result has the sign of the input |
np.matrix and ndarray subclasses become ndarray |
They keep their type (and a masked array its mask and fill value) |
decimals=np.int64(1): TypeError |
Accepted (any integer; bool and float are still rejected) |
pa.RecordBatch: TypeError |
Supported |
TypeError: Unsupported type: ... for an unsupported object |
UnsupportedTypeError (a TypeError): Cannot round an object of type X; register a rounder with register_rounders |
A sparse tensor: NotImplementedError from torch |
UnsupportedTypeError |
recursive=True keeps a leaf that raises any TypeError |
It keeps only a leaf that raises UnsupportedTypeError; any other exception propagates |
| The dispatch is a hard-coded table | register_rounders({Type: rounder}), get_default_registry(), round_values(..., registry=...), plugin group coola.rounding.rounders |
coola.rounding.native.round_native(data, mode, decimals) |
Removed: round_float and round_int, the rounders registered for float and int |
19. coola.validation: Eleven Functions, One Message Format¶
Affects: Only code written against the development branch: the package is new since 1.1.10.
coola.validation keeps the validators that state a constraint that isinstance or one comparison
cannot state, and removes the synonyms, with no alias:
| Before | After |
|---|---|
validate_bool(x), validate_float(x), validate_str(x), validate_list(x), validate_tuple(x), validate_dict(x), validate_set(x), validate_frozenset(x) |
validate_isinstance(x, bool), validate_isinstance(x, float), ... |
validate_mapping(x) |
validate_isinstance(x, Mapping) |
validate_positive(x), validate_non_negative(x) |
validate_gt(x, 0), validate_ge(x, 0) |
validate_negative(x), validate_non_positive(x) |
validate_lt(x, 0), validate_le(x, 0) |
validate_closed_range(x, low, high) |
validate_range(x, low, high) |
validate_open_range(x, low, high) |
validate_range(x, low, high, low_inclusive=False, high_inclusive=False) |
validate_half_open_range(x, low, high) |
validate_range(x, low, high, high_inclusive=False) |
validate_range with low > high (validate_closed_range(0.5, 1, 0) blamed the value) |
ValueError: ratio: the range [1, 0] is empty |
value must be an instance of <class 'int'>, got <class 'str'> |
value must be an instance of int, got str |
value must be a number that can be compared with 0, got None |
value cannot be compared with 0, got None |
value must be positive, got 0, ... in the closed range [0, 1], got 2 |
value must be greater than 0, got 0, ... in the range [0, 1], got 2 |
validate_finite("a"), validate_not_empty(5): a raw TypeError with no name |
TypeError: value must be a real number, got str, value must have a length, got int |
validate_finite(True), validate_finite(Decimal("1")) were accepted |
TypeError: validate_finite checks validate_number first |
validate_in([], {1}): TypeError: cannot use 'list' as a set element |
ValueError: value must be one of {1}, got [] |
validate_in(-1, list(range(100000))): a message of 688,919 characters |
The listing is limited to 20 items |
hash_object(x, length="64"), length=True: ValueError |
TypeError: length must be an int, got str |
round_values(x, mode="trunc"): Incorrect mode 'trunc'. The valid modes are: ... |
ValueError: mode must be one of ('round', 'floor', 'ceil'), got 'trunc' |
merge_mappings(..., on_duplicate="x"): Incorrect on_duplicate value: 'x'. Valid values are {...} |
ValueError: on_duplicate must be one of ('raise', 'first', 'last', 'suffix'), got 'x' |
Additions¶
20. New in 2.0¶
Affects: Nobody: these additions need no migration.
These additions need no migration. Each package has a page in the user guide.
| Addition | Where |
|---|---|
Registry and TypeRegistry (fallback predicates, lazy groups, pickling, plugin loading), LazySingleton |
coola.registry, coola.utils.singleton |
compare, assert_objects_equal, assert_objects_allclose, ComparisonResult, Tolerance; testers for dataclasses, sets, ordered mappings, objects with attributes, pydantic models, SciPy sparse matrices, NumPy time scalars, tables and DifferenceCloseEqualityTester |
coola.equality |
Hashers for dataclasses, enums, sets, JAX, NumPy, pandas, polars, PyArrow, PyTorch and xarray objects, SecretHasher, get_stable_registry, tagged, digest |
coola.hashing |
Summarizers for dataclasses, pydantic models, pandas, polars, PyArrow, xarray and JAX objects; SummaryConfig, Node |
coola.summary |
dfs_items |
coola.iterator |
flatten_frame, add_prefix_suffix_to_keys, DuplicateKeyError, escape= of to_flat_dict and from_flat_dict, restore_lists= |
coola.nested |
PythonRandomManager, sample_torch_seed, make_torch_generator, validate_seed |
coola.random |
coola.io (load_*, save_*, atomic saves), coola.identifier (ULID, UUID7, ObjectId, Snowflake, checksummed and stable identifiers), coola.rounding, coola.validation, coola.formatting, coola.display.rich |
New packages (sections 13, 14, 16, 17, 18 and 19) |
get_file_size, ForkAware, track_fork, get_current_branch, get_current_commit_hash, is_git_repo, run_git_command, int_repr, CaptureHandler, LRUCache, safe_repr, safe_str, smallest_items, raise_recursion_limit_error |
coola.utils |
MissingPackageError, is_scipy_available, check_scipy, is_module_imported, lazy_exports, availability_cache, clear_availability_caches |
coola.utils.imports |
requires, scipy_available, scipy_not_available, torch_cuda_not_available, torch_mps_not_available, torch_numpy_not_available, isolated_default_registries |
coola.testing.marks, coola.testing.fixtures |
BaseDisplayMixin (__repr__ and __str__ from _display_fields) |
coola.display |
configure_colorlog_logging(..., **kwargs), sanitize_path(resolve=), check_env_vars(missing_level=) |
coola.display, coola.utils |