arkas.evaluator2

arkas.evaluator2

Contain data evaluators.

arkas.evaluator2.AccuracyEvaluator

Bases: BaseStateCachedEvaluator[AccuracyState]

Implement the accuracy evaluator.

Parameters:

Name	Type	Description	Default
state	T	The state containing the ground truth and predicted labels.	required

Example usage:

>>> import numpy as np
>>> from arkas.evaluator2 import AccuracyEvaluator
>>> from arkas.state import AccuracyState
>>> evaluator = AccuracyEvaluator(
...     AccuracyState(
...         y_true=np.array([1, 0, 0, 1, 1]),
...         y_pred=np.array([1, 0, 0, 1, 1]),
...         y_true_name="target",
...         y_pred_name="pred",
...     )
... )
>>> evaluator
AccuracyEvaluator(
  (state): AccuracyState(y_true=(5,), y_pred=(5,), y_true_name='target', y_pred_name='pred', nan_policy='propagate')
)
>>> evaluator.evaluate()
{'accuracy': 1.0, 'count_correct': 5, 'count_incorrect': 0, 'count': 5, 'error': 0.0}

arkas.evaluator2.BalancedAccuracyEvaluator

Bases: BaseStateCachedEvaluator[AccuracyState]

Implement the balanced accuracy evaluator.

Parameters:

Name	Type	Description	Default
state	T	The state containing the ground truth and predicted labels.	required
nan_policy		The policy on how to handle NaN values in the input arrays. The following options are available: 'omit', 'propagate', and 'raise'.	required

Example usage:

>>> import numpy as np
>>> from arkas.evaluator2 import BalancedAccuracyEvaluator
>>> from arkas.state import AccuracyState
>>> evaluator = BalancedAccuracyEvaluator(
...     AccuracyState(
...         y_true=np.array([1, 0, 0, 1, 1]),
...         y_pred=np.array([1, 0, 0, 1, 1]),
...         y_true_name="target",
...         y_pred_name="pred",
...     )
... )
>>> evaluator
BalancedAccuracyEvaluator(
  (state): AccuracyState(y_true=(5,), y_pred=(5,), y_true_name='target', y_pred_name='pred', nan_policy='propagate')
)
>>> evaluator.evaluate()
{'balanced_accuracy': 1.0, 'count': 5}

arkas.evaluator2.BaseCachedEvaluator

Bases: BaseEvaluator

Define the base class to implement an evaluator that caches the metrics after the first evaluation.

arkas.evaluator2.BaseEvaluator

Bases: ABC

Define the base class to implement an evaluator.

Example usage:

>>> import numpy as np
>>> from arkas.evaluator2 import AccuracyEvaluator
>>> from arkas.state import AccuracyState
>>> evaluator = AccuracyEvaluator(
...     AccuracyState(
...         y_true=np.array([1, 0, 0, 1, 1]),
...         y_pred=np.array([1, 0, 0, 1, 1]),
...         y_true_name="target",
...         y_pred_name="pred",
...     )
... )
>>> evaluator
AccuracyEvaluator(
  (state): AccuracyState(y_true=(5,), y_pred=(5,), y_true_name='target', y_pred_name='pred', nan_policy='propagate')
)

arkas.evaluator2.BaseEvaluator.compute abstractmethod

compute() -> BaseEvaluator

Compute the metrics and return a new evaluator.

Returns:

Type	Description
BaseEvaluator	A new evaluator with the computed metrics.

Example usage:

>>> import numpy as np
>>> from arkas.evaluator2 import AccuracyEvaluator
>>> from arkas.state import AccuracyState
>>> evaluator = AccuracyEvaluator(
...     AccuracyState(
...         y_true=np.array([1, 0, 0, 1, 1]),
...         y_pred=np.array([1, 0, 0, 1, 1]),
...         y_true_name="target",
...         y_pred_name="pred",
...     )
... )
>>> evaluator2 = evaluator.compute()
>>> evaluator2
Evaluator(count=5)

arkas.evaluator2.BaseEvaluator.equal abstractmethod

equal(other: Any, equal_nan: bool = False) -> bool

Indicate if two evaluators are equal or not.

Parameters:

Name	Type	Description	Default
other	Any	The other evaluator to compare.	required
equal_nan	bool	Whether to compare NaN's as equal. If True, NaN's in both objects will be considered equal.	False

Returns:

Type	Description
bool	True if the two evaluators are equal, otherwise False.

Example usage:

>>> import numpy as np
>>> from arkas.evaluator2 import AccuracyEvaluator
>>> from arkas.state import AccuracyState
>>> evaluator1 = AccuracyEvaluator(
...     AccuracyState(
...         y_true=np.array([1, 0, 0, 1, 1]),
...         y_pred=np.array([1, 0, 0, 1, 1]),
...         y_true_name="target",
...         y_pred_name="pred",
...     )
... )
>>> evaluator2 = AccuracyEvaluator(
...     AccuracyState(
...         y_true=np.array([1, 0, 0, 1, 1]),
...         y_pred=np.array([1, 0, 0, 1, 1]),
...         y_true_name="target",
...         y_pred_name="pred",
...     )
... )
>>> evaluator3 = AccuracyEvaluator(
...     AccuracyState(
...         y_true=np.array([1, 0, 0, 0, 0]),
...         y_pred=np.array([1, 0, 0, 1, 1]),
...         y_true_name="target",
...         y_pred_name="pred",
...     )
... )
>>> evaluator1.equal(evaluator2)
True
>>> evaluator1.equal(evaluator3)
False

arkas.evaluator2.BaseEvaluator.evaluate abstractmethod

evaluate(prefix: str = '', suffix: str = '') -> dict

Evaluate the metrics.

Parameters:

Name	Type	Description	Default
prefix	str	The key prefix in the returned dictionary.	''
suffix	str	The key suffix in the returned dictionary.	''

Returns:

Type	Description
dict	The metrics.

Example usage:

>>> import numpy as np
>>> from arkas.evaluator2 import AccuracyEvaluator
>>> from arkas.state import AccuracyState
>>> evaluator = AccuracyEvaluator(
...     AccuracyState(
...         y_true=np.array([1, 0, 0, 1, 1]),
...         y_pred=np.array([1, 0, 0, 1, 1]),
...         y_true_name="target",
...         y_pred_name="pred",
...     )
... )
>>> evaluator.evaluate()
{'accuracy': 1.0, 'count_correct': 5, 'count_incorrect': 0, 'count': 5, 'error': 0.0}

arkas.evaluator2.BaseStateCachedEvaluator

Bases: BaseCachedEvaluator, Generic[T]

Define the base class to implement an evaluator that caches the metrics after the first evaluation, and computes the metrics from a state object.

Parameters:

Name	Type	Description	Default
state	T	The state with the data.	required

arkas.evaluator2.ColumnCooccurrenceEvaluator

Bases: BaseStateCachedEvaluator[ColumnCooccurrenceState]

Implement the pairwise column co-occurrence evaluator.

Parameters:

Name	Type	Description	Default
state	T	The state with the co-occurrence matrix.	required

Example usage:

>>> import numpy as np
>>> from arkas.evaluator2 import ColumnCooccurrenceEvaluator
>>> from arkas.state import ColumnCooccurrenceState
>>> evaluator = ColumnCooccurrenceEvaluator(
...     ColumnCooccurrenceState(matrix=np.ones((3, 3)), columns=["a", "b", "c"])
... )
>>> evaluator
ColumnCooccurrenceEvaluator(
  (state): ColumnCooccurrenceState(matrix=(3, 3), figure_config=MatplotlibFigureConfig())
)
>>> evaluator.evaluate()
{'column_cooccurrence': array([[1., 1., 1.],
       [1., 1., 1.],
       [1., 1., 1.]])}

arkas.evaluator2.ColumnCorrelationEvaluator

Bases: BaseStateCachedEvaluator[TargetDataFrameState]

Implement the column correlation evaluator.

Parameters:

Name	Type	Description	Default
state	T	The state with the DataFrame to analyze.	required

Example usage:

>>> import polars as pl
>>> from arkas.evaluator2 import ColumnCorrelationEvaluator
>>> from arkas.state import TargetDataFrameState
>>> frame = pl.DataFrame(
...     {
...         "col1": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0],
...         "col2": [7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0],
...         "col3": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0],
...     },
... )
>>> evaluator = ColumnCorrelationEvaluator(
...     TargetDataFrameState(frame, target_column="col3")
... )
>>> evaluator
ColumnCorrelationEvaluator(
  (state): TargetDataFrameState(dataframe=(7, 3), target_column='col3', nan_policy='propagate', figure_config=MatplotlibFigureConfig())
)
>>> evaluator.evaluate()
{'col1': {'count': 7, 'pearson_coeff': 1.0, 'pearson_pvalue': 0.0, 'spearman_coeff': 1.0, 'spearman_pvalue': 0.0},
 'col2': {'count': 7, 'pearson_coeff': -1.0, 'pearson_pvalue': 0.0, 'spearman_coeff': -1.0, 'spearman_pvalue': 0.0}}

arkas.evaluator2.CorrelationEvaluator

Bases: BaseStateCachedEvaluator[TwoColumnDataFrameState]

Implement the pairwise column correlation evaluator.

Parameters:

Name	Type	Description	Default
state	TwoColumnDataFrameState	The state with the DataFrame to analyze.	required

Example usage:

>>> import polars as pl
>>> from arkas.evaluator2 import CorrelationEvaluator
>>> from arkas.state import DataFrameState
>>> frame = pl.DataFrame(
...     {
...         "col1": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0],
...         "col2": [2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0],
...     },
... )
>>> evaluator = CorrelationEvaluator(
...     TwoColumnDataFrameState(frame, column1="col1", column2="col2")
... )
>>> evaluator
CorrelationEvaluator(
  (state): TwoColumnDataFrameState(dataframe=(7, 2), column1='col1', column2='col2', nan_policy='propagate', figure_config=MatplotlibFigureConfig())
)
>>> evaluator.evaluate()
{'count': 7, 'pearson_coeff': 1.0, 'pearson_pvalue': 0.0, 'spearman_coeff': 1.0, 'spearman_pvalue': 0.0}

arkas.evaluator2.Evaluator

Bases: BaseEvaluator

Implement a simple evaluator.

Parameters:

Name	Type	Description	Default
metrics	dict | None	The dictionary of metrics.	None

Example usage:

>>> import numpy as np
>>> from arkas.evaluator2 import Evaluator
>>> evaluator = Evaluator({"accuracy": 1.0, "total": 42})
>>> evaluator
Evaluator(count=2)
>>> evaluator.evaluate()
{'accuracy': 1.0, 'total': 42}

arkas.evaluator2.EvaluatorDict

Bases: BaseCachedEvaluator

Implement an evaluator that sequentially evaluates a mapping of evaluators.

Parameters:

Name	Type	Description	Default
evaluators	Mapping[Hashable, BaseEvaluator]	The mapping of evaluators to evaluate.	required

Example usage:

>>> import numpy as np
>>> from arkas.state import AccuracyState
>>> from arkas.evaluator2 import EvaluatorDict, AccuracyEvaluator, Evaluator
>>> evaluator = EvaluatorDict(
...     {
...         "one": AccuracyEvaluator(
...             AccuracyState(
...                 y_true=np.array([1, 0, 0, 1, 1]),
...                 y_pred=np.array([1, 0, 0, 1, 1]),
...                 y_true_name="target",
...                 y_pred_name="pred",
...             )
...         ),
...         "two": Evaluator({"accuracy": 1.0, "total": 42}),
...     }
... )
>>> evaluator
EvaluatorDict(
  (one): AccuracyEvaluator(
      (state): AccuracyState(y_true=(5,), y_pred=(5,), y_true_name='target', y_pred_name='pred', nan_policy='propagate')
    )
  (two): Evaluator(count=2)
)
>>> metrics = evaluator.evaluate()
>>> metrics
{'one': {'accuracy': 1.0, 'count_correct': 5, 'count_incorrect': 0, 'count': 5, 'error': 0.0},
 'two': {'accuracy': 1.0, 'total': 42}}

arkas.evaluator2.NumericStatisticsEvaluator

Bases: BaseStateCachedEvaluator[DataFrameState]

Implement an evaluator to compute statistics of numerical columns.

Parameters:

Name	Type	Description	Default
state	T	The state containing the DataFrame to analyze.	required

Example usage:

>>> import polars as pl
>>> from arkas.evaluator2 import NumericStatisticsEvaluator
>>> from arkas.state import DataFrameState
>>> dataframe = pl.DataFrame(
...     {
...         "col1": [0, 1, 1, 0, 0, 1, 0],
...         "col2": [0, 1, 0, 1, 0, 1, 0],
...         "col3": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0],
...     }
... )
>>> evaluator = NumericStatisticsEvaluator(DataFrameState(dataframe))
>>> evaluator
NumericStatisticsEvaluator(
  (state): DataFrameState(dataframe=(7, 3), nan_policy='propagate', figure_config=MatplotlibFigureConfig())
)
>>> evaluator.evaluate()
{'col1': {'count': 7, ...}, 'col2': {'count': 7, ...}}

arkas.evaluator2.PrecisionEvaluator

Bases: BaseStateCachedEvaluator[PrecisionRecallState]

Implement the precision evaluator.

This evaluator can be used in 3 different settings:

• binary: y_true must be an array of shape (n_samples,) with 0 and 1 values, and y_pred must be an array of shape (n_samples,).
• multiclass: y_true must be an array of shape (n_samples,) with values in {0, ..., n_classes-1}, and y_pred must be an array of shape (n_samples,).
• multilabel: y_true must be an array of shape (n_samples, n_classes) with 0 and 1 values, and y_pred must be an array of shape (n_samples, n_classes).

Parameters:

Name	Type	Description	Default
state	T	The state containing the ground truth and predicted labels.	required

Example usage:

>>> import numpy as np
>>> from arkas.evaluator2 import PrecisionEvaluator
>>> from arkas.state import PrecisionRecallState
>>> # binary
>>> evaluator = PrecisionEvaluator(
...     PrecisionRecallState(
...         y_true=np.array([1, 0, 0, 1, 1]),
...         y_pred=np.array([1, 0, 0, 1, 1]),
...         y_true_name="target",
...         y_pred_name="pred",
...         label_type="binary",
...     ),
... )
>>> evaluator
PrecisionEvaluator(
  (state): PrecisionRecallState(y_true=(5,), y_pred=(5,), y_true_name='target', y_pred_name='pred', label_type='binary', nan_policy='propagate')
)
>>> evaluator.evaluate()
{'count': 5, 'precision': 1.0}
>>> # multilabel
>>> evaluator = PrecisionEvaluator(
...     PrecisionRecallState(
...         y_true=np.array([[1, 0, 1], [0, 1, 0], [0, 1, 0], [1, 0, 1], [1, 0, 1]]),
...         y_pred=np.array([[1, 0, 0], [0, 1, 1], [0, 1, 1], [1, 0, 0], [1, 0, 0]]),
...         y_true_name="target",
...         y_pred_name="pred",
...         label_type="multilabel",
...     )
... )
>>> evaluator
PrecisionEvaluator(
  (state): PrecisionRecallState(y_true=(5, 3), y_pred=(5, 3), y_true_name='target', y_pred_name='pred', label_type='multilabel', nan_policy='propagate')
)
>>> evaluator.evaluate()
{'count': 5,
 'macro_precision': 0.666...,
 'micro_precision': 0.714...,
 'precision': array([1., 1., 0.]),
 'weighted_precision': 0.625}
>>> # multiclass
>>> evaluator = PrecisionEvaluator(
...     PrecisionRecallState(
...         y_true=np.array([0, 0, 1, 1, 2, 2]),
...         y_pred=np.array([0, 0, 1, 1, 2, 2]),
...         y_true_name="target",
...         y_pred_name="pred",
...         label_type="multiclass",
...     ),
... )
>>> evaluator
PrecisionEvaluator(
  (state): PrecisionRecallState(y_true=(6,), y_pred=(6,), y_true_name='target', y_pred_name='pred', label_type='multiclass', nan_policy='propagate')
)
>>> evaluator.evaluate()
{'count': 6,
 'macro_precision': 1.0,
 'micro_precision': 1.0,
 'precision': array([1., 1., 1.]),
 'weighted_precision': 1.0}
>>> # auto
>>> evaluator = PrecisionEvaluator(
...     PrecisionRecallState(
...         y_true=np.array([1, 0, 0, 1, 1]),
...         y_pred=np.array([1, 0, 0, 1, 1]),
...         y_true_name="target",
...         y_pred_name="pred",
...     )
... )
>>> evaluator
PrecisionEvaluator(
  (state): PrecisionRecallState(y_true=(5,), y_pred=(5,), y_true_name='target', y_pred_name='pred', label_type='binary', nan_policy='propagate')
)
>>> evaluator.evaluate()
{'count': 5, 'precision': 1.0}