Nested

coola.nested

Helpers to reshape, filter and convert nested data structures.

nested is not one algorithm: it is a set of helpers built on the traversal packages, in four groups. flat: to_flat_dict and from_flat_dict. keys: remove_keys_if, remove_keys_containing, remove_keys_starting_with and add_prefix_suffix_to_keys. mappings: merge_mappings, flatten_mapping and DuplicateKeyError. records: convert_to_dict_of_lists and convert_to_list_of_dicts. The conversion to JSON data is in coola.jsonable.

Properties:

• Pure: the input is never modified. A function that rebuilds containers returns new ones; the leaves are shared, never copied.
• One notion of container: a function that looks inside the values enters the containers of the other packages (sequences, mappings, sets, dataclasses) and treats text, bytes, iterators and unknown types as leaves. to_flat_dict does not enter sets.
• Structure-preserving: a function that rebuilds a structure keeps the type of every container and the order of the entries.
• Any depth: no RecursionError, the depth is bounded by memory only.
• Cycle-safe: a container that contains itself raises a ValueError.
• Keys are labels: only the flat and keys functions read or change keys.
• No silent loss: two entries are never merged into one without a decision of the caller, a key collision raises. A wrong input type raises a TypeError that names the type.
• Documented round trips: from_flat_dict(to_flat_dict(x)) == x and convert_to_list_of_dicts(convert_to_dict_of_lists(r)) == r under the conditions of the guide.
• Explicit options: options are keyword-only, an invalid option raises a ValueError before any work is done.
• Lazy: import coola.nested imports no optional library.
• Thread-safe: the state of a call is local to the call.
• Fast: the cost is linear in the number of nodes.

Not guaranteed: the cost of a container shared by several paths (it is visited once per path), and the round trip of dataclasses and of non-str keys.

Several of these properties are reached step by step; the guide says which ones.

Example

>>> from coola.nested import to_flat_dict
>>> to_flat_dict({"a": {"b": 1}, "c": [2, 3]})
{'a.b': 1, 'c.0': 2, 'c.1': 3}

coola.nested.DuplicateKeyError

Bases: KeyError, ValueError

Raised when a key cannot be added because it is a duplicate.

It derives from KeyError (the historical exception) and from ValueError (the exception used for the other invalid inputs of this module), so both can be used to catch it.

coola.nested.add_prefix_suffix_to_keys

add_prefix_suffix_to_keys(
    mapping: M,
    prefix: str = "",
    suffix: str = "",
    *,
    recursive: bool = False,
) -> M

Add a prefix and/or a suffix to the keys of a mapping.

Only keys of type str are renamed; every other key is kept as-is because a prefix/suffix cannot be concatenated to it. The type of every mapping is kept (an OrderedDict gives an OrderedDict).

Parameters:

Name	Type	Description	Default
mapping	M	The input mapping.	required
prefix	str	The prefix to prepend to the keys.	''
suffix	str	The suffix to append to the keys.	''
recursive	bool	If True, the prefix and suffix are also added to the keys of every nested mapping, including the mappings in the lists, the tuples and the dataclasses. If False, only the keys of the top-level mapping are renamed.	False

Returns:

Type	Description
M	A new mapping of the same type with the renamed keys.

Raises:

Type	Description
TypeError	If mapping is not a mapping.
ValueError	If a container contains itself.

Example

>>> from coola.nested import add_prefix_suffix_to_keys
>>> add_prefix_suffix_to_keys({"key1": 1, "key2": 2}, prefix="prefix_")
{'prefix_key1': 1, 'prefix_key2': 2}
>>> add_prefix_suffix_to_keys(
...     {"key1": 1, "key2": {"key3": 3}}, suffix="_suffix", recursive=True
... )
{'key1_suffix': 1, 'key2_suffix': {'key3_suffix': 3}}

coola.nested.convert_to_dict_of_lists

convert_to_dict_of_lists(
    seq_of_mappings: Iterable[Mapping[Any, Any]],
) -> dict[Any, list[Any]]

Convert an iterable of mappings to a dictionary of lists.

All the mappings must have the same keys as the first mapping, which is used to find the keys. The order of the keys is the one of the first mapping. An iterable that is not a sequence (a generator) is consumed once.

Parameters:

Name	Type	Description	Default
seq_of_mappings	Iterable[Mapping[Any, Any]]	The iterable of mappings to convert.	required

Returns:

Type	Description
dict[Any, list[Any]]	A dictionary of lists.

Raises:

Type	Description
ValueError	If a mapping's keys differ from the first mapping's keys.

Example

>>> from coola.nested import convert_to_dict_of_lists
>>> convert_to_dict_of_lists(
...     [{"key1": 1, "key2": 10}, {"key1": 2, "key2": 20}, {"key1": 3, "key2": 30}]
... )
{'key1': [1, 2, 3], 'key2': [10, 20, 30]}

coola.nested.convert_to_list_of_dicts

convert_to_list_of_dicts(
    mapping_of_seqs: Mapping[Any, Sequence[Any]],
) -> list[dict[Any, Any]]

Convert a mapping of sequences to a list of dictionaries.

All the sequences must have the same length. A string (or bytes) is not accepted as a sequence, because it would be split per character, and a set is not accepted either, because its elements have no position (the rows of several sets would be paired in an arbitrary order). A value must be a Sequence (a generator is not one). An empty mapping gives an empty list, so a list of empty dictionaries ([{}, {}]) does not round trip with convert_to_dict_of_lists.

Parameters:

Name	Type	Description	Default
mapping_of_seqs	Mapping[Any, Sequence[Any]]	The mapping of sequences to convert.	required

Returns:

Type	Description
list[dict[Any, Any]]	A list of dictionaries.

Raises:

Type	Description
TypeError	If a value is a str, bytes or bytearray, a set, or is not a Sequence.
ValueError	If the sequences do not all have the same length.

Example

>>> from coola.nested import convert_to_list_of_dicts
>>> convert_to_list_of_dicts({"key1": [1, 2, 3], "key2": [10, 20, 30]})
[{'key1': 1, 'key2': 10}, {'key1': 2, 'key2': 20}, {'key1': 3, 'key2': 30}]

coola.nested.flatten_frame

flatten_frame(
    frame: DataFrame,
    separator: str = ".",
    depth: int | None = None,
    expand_lists: bool = False,
) -> DataFrame

Recursively unnest struct columns and optionally expand list/array columns.

Non-nested columns are passed through unchanged. Struct columns are expanded level by level up to depth levels. Columns that are lists or arrays of structs are automatically exploded to expose their inner structs before unnesting. Several such columns are exploded together if their lists have the same length in every row, otherwise one after the other, which gives one row for each combination of their elements. If expand_lists is enabled, remaining List and Array columns are expanded into one column per index position following the <col><separator><index> pattern, filled with None where the list is shorter than the maximum length.

Parameters:

Name	Type	Description	Default
frame	DataFrame	A DataFrame that may contain struct, list, or array columns at any nesting level.	required
separator	str	Separator used to build output column names following the <parent><separator><field> pattern. Defaults to ".".	'.'
depth	int | None	Maximum number of struct nesting levels to expand. None (default) expands all levels until no struct columns remain. 0 expands no struct column.	None
expand_lists	bool	If True, List and Array columns are expanded into one column per index position. If False (default), list and array columns are left as-is.	False

Returns:

Type	Description
DataFrame	A DataFrame where struct columns are recursively unnested and, if
DataFrame	expand_lists is True, list and array columns are expanded into
DataFrame	positional columns.

Raises:

Type	Description
TypeError	If frame is not a polars.DataFrame, for example a LazyFrame (call collect() first: a lazy frame has no cheap schema, and the number of rows of an exploded list is only known after the evaluation).
ValueError	If depth is negative.

Example

>>> import polars as pl
>>> from coola.nested import flatten_frame
>>> frame = pl.DataFrame(
...     {
...         "id": [1, 2],
...         "coords": [
...             {"x": 10, "meta": {"label": "a"}},
...             {"x": 30, "meta": {"label": "b"}},
...         ],
...         "tags": [["foo", "bar"], ["baz"]],
...     }
... )
>>> frame
shape: (2, 3)
┌─────┬────────────┬────────────────┐
│ id  ┆ coords     ┆ tags           │
│ --- ┆ ---        ┆ ---            │
│ i64 ┆ struct[2]  ┆ list[str]      │
╞═════╪════════════╪════════════════╡
│ 1   ┆ {10,{"a"}} ┆ ["foo", "bar"] │
│ 2   ┆ {30,{"b"}} ┆ ["baz"]        │
└─────┴────────────┴────────────────┘
>>> flatten_frame(frame)
shape: (2, 4)
┌─────┬──────────┬───────────────────┬────────────────┐
│ id  ┆ coords.x ┆ coords.meta.label ┆ tags           │
│ --- ┆ ---      ┆ ---               ┆ ---            │
│ i64 ┆ i64      ┆ str               ┆ list[str]      │
╞═════╪══════════╪═══════════════════╪════════════════╡
│ 1   ┆ 10       ┆ a                 ┆ ["foo", "bar"] │
│ 2   ┆ 30       ┆ b                 ┆ ["baz"]        │
└─────┴──────────┴───────────────────┴────────────────┘
>>> flatten_frame(frame, expand_lists=True)
shape: (2, 5)
┌─────┬──────────┬───────────────────┬────────┬────────┐
│ id  ┆ coords.x ┆ coords.meta.label ┆ tags.0 ┆ tags.1 │
│ --- ┆ ---      ┆ ---               ┆ ---    ┆ ---    │
│ i64 ┆ i64      ┆ str               ┆ str    ┆ str    │
╞═════╪══════════╪═══════════════════╪════════╪════════╡
│ 1   ┆ 10       ┆ a                 ┆ foo    ┆ bar    │
│ 2   ┆ 30       ┆ b                 ┆ baz    ┆ null   │
└─────┴──────────┴───────────────────┴────────┴────────┘

coola.nested.flatten_mapping

flatten_mapping(
    mapping: Mapping[Any, Mapping[Any, Any]],
    *,
    on_duplicate: Literal[
        "raise", "first", "last", "prefix"
    ] = "raise",
    always_prefix: bool = False,
    separator: str = ".",
    equal_nan: bool = False,
) -> dict[Any, Any]

Flatten a mapping of mappings into a single dict.

Each outer key is used as a prefix for its inner keys. If an inner key appears in more than one outer mapping with the same value in every occurrence, the plain inner key is kept and on_duplicate is not triggered. on_duplicate only applies when an inner key appears more than once with different values.

Parameters:

Name	Type	Description	Default
mapping	Mapping[Any, Mapping[Any, Any]]	The mapping of mappings to flatten.	required
on_duplicate	Literal['raise', 'first', 'last', 'prefix']	The strategy used to manage duplicate inner keys that have different values across outer mappings. The valid values are: - 'raise': raise an exception if an inner key appears more than once with different values. - 'first': keep the value from the first occurrence, under the plain inner key. - 'last': keep the value from the last occurrence, under the plain inner key. - 'prefix': keep all the values, renaming every occurrence of a conflicting key as '{outer_key}{separator}{inner_key}', including the first occurrence. The result depends on the order of the mappings: a key is renamed only when a conflict is found, in place (the output follows the order of the input). This differs from merge_mappings's on_duplicate='suffix' strategy, which leaves the first occurrence under the plain key.	'raise'
always_prefix	bool	If True, every key in the output is named '{outer_key}{separator}{inner_key}', regardless of whether it is a duplicate. If False, only keys involved in a genuine conflict are prefixed (only relevant when on_duplicate='prefix').	False
separator	str	The separator used to join the outer and inner keys when prefixing.	'.'
equal_nan	bool	If True, NaN values are considered equal, so they are not duplicates of each other. Defaults to False.	False

Returns:

Type	Description
dict[Any, Any]	The flattened dict.

Raises:

Type	Description
ValueError	if on_duplicate is not a valid value.
TypeError	if a value of mapping is not a mapping.
DuplicateKeyError	if on_duplicate='raise' and an inner key appears more than once with different values, or if two different keys are flattened to the same key (for example {"a.b": {"c": 1}, "a": {"b.c": 2}} with always_prefix=True).

Example

>>> from coola.nested import flatten_mapping
>>> flatten_mapping(
...     {"module1": {"a": 1, "b": 2}, "module2": {"b": 3, "c": 4}},
...     on_duplicate="prefix",
... )
{'a': 1, 'module1.b': 2, 'module2.b': 3, 'c': 4}

coola.nested.from_flat_dict

from_flat_dict(
    data: Mapping[str, Any],
    *,
    separator: str = ".",
    restore_lists: bool = False,
    escape: bool = False,
) -> dict[str, Any]

Return a nested dict from a flat dict produced by to_flat_dict.

Each key in data is split on separator to reconstruct the nesting depth. By default the keys whose segments are decimal integers are not converted to lists; the output only contains dict objects, so the round-trip is lossless for the dict-of-dicts case and the integer keys are preserved as strings for the list-originated case. Use restore_lists=True to rebuild the lists.

Parameters:

Name	Type	Description	Default
data	Mapping[str, Any]	A flat dictionary whose keys use separator to encode nesting depth, as produced by to_flat_dict.	required
separator	str	The separator that was used when the flat dict was created. Defaults to ".".	'.'
restore_lists	bool	If True, every dict whose keys are exactly the strings "0", "1", ..., "n-1" (n >= 1) is converted to a list. A dict that was originally built with such keys is converted too, so the original container type is not always recovered, and tuple objects come back as list objects. Only the nodes built from the keys are converted: a dict given as a value is kept as it is. Defaults to False.	False
escape	bool	If True, a backslash escapes the separator (and the backslash itself) in the keys, as written by to_flat_dict(..., escape=True), so the keys of the original dict that contain the separator are restored. If False (default), every separator splits the key.	False

Returns:

Type	Description
dict[str, Any]	A nested dict. The root is always a dict, even with
dict[str, Any]	restore_lists=True and the keys "0", "1", ...

Raises:

Type	Description
ValueError	If separator is empty, which would make key splitting ambiguous.
ValueError	If two keys in data imply conflicting types at the same path (e.g. one key treats a node as a dict and another treats it as a scalar leaf).

Example

>>> from coola.nested import from_flat_dict
>>> from_flat_dict({"str": "def", "module.component.float": 3.5, "module.component.int": 2})
{'str': 'def', 'module': {'component': {'float': 3.5, 'int': 2}}}
>>> # Integer-string keys are preserved as strings
>>> from_flat_dict({"module.0.0": 1, "module.0.1": 2, "module.1.bool": True, "str": "abc"})
{'module': {'0': {'0': 1, '1': 2}, '1': {'bool': True}}, 'str': 'abc'}
>>> # The lists can be restored
>>> from_flat_dict(
...     {"module.0.0": 1, "module.0.1": 2, "module.1.bool": True, "str": "abc"},
...     restore_lists=True,
... )
{'module': [[1, 2], {'bool': True}], 'str': 'abc'}

coola.nested.merge_mappings

merge_mappings(
    mappings: Iterable[Mapping[Any, Any]],
    *,
    on_duplicate: Literal[
        "raise", "first", "last", "suffix"
    ] = "raise",
    equal_nan: bool = False,
) -> dict[Any, Any]

Merge an iterable of mappings into a single dict.

If a key appears more than once with the same value in every occurrence, the value is kept as-is and on_duplicate is not triggered. on_duplicate only applies when a key appears more than once with different values.

Parameters:

Name	Type	Description	Default
mappings	Iterable[Mapping[Any, Any]]	The mappings to merge.	required
on_duplicate	Literal['raise', 'first', 'last', 'suffix']	The strategy used to manage duplicate keys that have different values across mappings. The valid values are: - 'raise': raise an exception if a key appears more than once with different values. - 'first': keep the value from the first occurrence. - 'last': keep the value from the last occurrence. - 'suffix': keep all the distinct values by adding a '_n' suffix to the key name, where n is incremented for each new distinct value. A value that is equal to one already kept for the key is dropped.	'raise'
equal_nan	bool	If True, NaN values are considered equal, so they are not duplicates of each other. Defaults to False.	False

Note

The keys of the output follow the order of the input. With on_duplicate='suffix', a generated key never takes the name of a key of an input mapping (whatever its position): the suffix is incremented until the key is free, so the result does not depend on the order of the mappings. Only the later occurrences of a conflicting key are renamed ('{key}_1', '{key}_2', ...); the first occurrence keeps the plain key. This differs from flatten_mapping's on_duplicate='prefix' strategy, which renames every occurrence, including the first, once a genuine conflict is detected for a key.

Returns:

Type	Description
dict[Any, Any]	The merged dict.

Raises:

Type	Description
ValueError	if on_duplicate is not a valid value.
DuplicateKeyError	if on_duplicate='raise' and a key appears more than once with different values.

Example

>>> from coola.nested import merge_mappings
>>> merge_mappings([{"key1": 1, "key2": 2}, {"key2": 3, "key3": 4}], on_duplicate="last")
{'key1': 1, 'key2': 3, 'key3': 4}

coola.nested.remove_keys_containing

remove_keys_containing(mapping: M, substring: str) -> M

Recursively remove the keys that contain a given substring.

Only the str keys are checked; the other keys are kept.

Parameters:

Name	Type	Description	Default
mapping	M	The original mapping.	required
substring	str	The substring used to filter the keys.	required

Returns:

Type	Description
M	A new mapping of the same type without the removed keys.

Raises:

Type	Description
TypeError	If mapping is not a mapping.
ValueError	If a container contains itself.

Example

>>> from coola.nested import remove_keys_containing
>>> remove_keys_containing(
...     {"key": 1, "key.abc": 2, "abc": 3, "abc.key": 4, 1: 5, (2, 3): 6},
...     "key",
... )
{'abc': 3, 1: 5, (2, 3): 6}
>>> remove_keys_containing(
...     {"abc": {"key": 1, "abc": 2}, "list": [{"key": 1, "abc": 2}]},
...     "key",
... )
{'abc': {'abc': 2}, 'list': [{'abc': 2}]}

coola.nested.remove_keys_if

remove_keys_if(
    mapping: M, predicate: Callable[[Any], bool]
) -> M

Recursively remove the keys that satisfy a given predicate.

The mapping is traversed with the engine of coola.recursive and the containers of the other packages: mappings (dict, OrderedDict, ...), sequences (list, tuple, named tuples, ...), sets and dataclasses are entered, and the type of each container is kept. Every other value is kept as-is.

Parameters:

Name	Type	Description	Default
mapping	M	The original mapping.	required
predicate	Callable[[Any], bool]	A function that takes a key and returns True if the key (and its value) should be removed.	required

Returns:

Type	Description
M	A new mapping of the same type without the removed keys.

Raises:

Type	Description
TypeError	If mapping is not a mapping.
ValueError	If a container contains itself.

Example

>>> from coola.nested import remove_keys_if
>>> remove_keys_if(
...     {"key": 1, "key.abc": 2, "abc": 3, "abc.key": 4, 1: 5, (2, 3): 6},
...     lambda key: isinstance(key, str) and "key" in key,
... )
{'abc': 3, 1: 5, (2, 3): 6}
>>> remove_keys_if(
...     {"abc": {"key": 1, "abc": 2}, "list": [{"key": 1, "abc": 2}]},
...     lambda key: isinstance(key, str) and "key" in key,
... )
{'abc': {'abc': 2}, 'list': [{'abc': 2}]}

coola.nested.remove_keys_starting_with

remove_keys_starting_with(mapping: M, prefix: str) -> M

Recursively remove the keys that start with a given prefix.

Only the str keys are checked; the other keys are kept.

Parameters:

Name	Type	Description	Default
mapping	M	The original mapping.	required
prefix	str	The prefix used to filter the keys.	required

Returns:

Type	Description
M	A new mapping of the same type without the removed keys.

Raises:

Type	Description
TypeError	If mapping is not a mapping.
ValueError	If a container contains itself.

Example

>>> from coola.nested import remove_keys_starting_with
>>> remove_keys_starting_with(
...     {"key": 1, "key.abc": 2, "abc": 3, "abc.key": 4, 1: 5, (2, 3): 6},
...     "key",
... )
{'abc': 3, 'abc.key': 4, 1: 5, (2, 3): 6}

coola.nested.to_flat_dict

to_flat_dict(
    data: object,
    *,
    prefix: str | None = None,
    separator: str = ".",
    to_str: type | tuple[type, ...] | None = None,
    escape: bool = False,
) -> dict[str, Any]

Return a flat representation of a nested structure as a dict using dotted keys.

The keys of the nested dicts are joined with separator, so by default a key that contains the separator is ambiguous: {"a.b": 1} and {"a": {"b": 1}} give the same flat dict, and from_flat_dict rebuilds the second one. Use escape=True to keep them apart.

Parameters:

Name	Type	Description	Default
data	object	The nested structure to flatten. The containers of coola.iterator are recursed into (mappings, sequences, dataclasses, ...); str, bytes, iterators and every unknown type are leaves. The sets are leaves too, whatever the registry: they are not ordered, so they have no index.	required
prefix	str | None	The prefix prepended to each key. None means no prefix; only valid when data is a mapping or sequence — a ValueError is raised if a bare scalar is passed without a prefix.	None
separator	str	The separator used to join nested keys.	'.'
to_str	type | tuple[type, ...] | None	A type or tuple of types that should be converted to their string representation instead of being recursed into. None (default) recurses into all supported containers.	None
escape	bool	If True, a backslash is added before every separator and every backslash in the keys of the mappings (the prefix is used as is), so that from_flat_dict(..., escape=True) restores the keys that contain the separator. If False (default), the keys are joined as they are.	False

Returns:

Type	Description
dict[str, Any]	A flat dictionary whose keys are the separator-joined
dict[str, Any]	paths to every leaf value.

Raises:

Type	Description
ValueError	If data is a scalar (non-mapping, non-sequence) and prefix is None, since this would produce a None key, or if a container contains itself.

Example

>>> from coola.nested import to_flat_dict
>>> data = {
...     "str": "def",
...     "module": {
...         "component": {
...             "float": 3.5,
...             "int": 2,
...         },
...     },
... }
>>> to_flat_dict(data)
{'str': 'def', 'module.component.float': 3.5, 'module.component.int': 2}

>>> # Lists and tuples are also supported
>>> data = {
...     "module": [[1, 2, 3], {"bool": True}],
...     "str": "abc",
... }
>>> to_flat_dict(data)
{'module.0.0': 1, 'module.0.1': 2, 'module.0.2': 3, 'module.1.bool': True, 'str': 'abc'}

>>> # Use to_str to prevent recursion into specific types
>>> to_flat_dict(data, to_str=list)
{'module': "[[1, 2, 3], {'bool': True}]", 'str': 'abc'}