Validation

docculus.validation

Contain validation functions for docculus.

docculus.validation.DocumentConsistencyError

Bases: ValueError

Raised when two documents share an id but have different page_content or metadata.

docculus.validation.validate_document_consistency

validate_document_consistency(
    docs: list[Document], *, raise_error: bool = False
) -> bool

Check that documents sharing the same id have the same page_content and metadata.

Documents with id=None are ignored, since None does not identify a single logical document. metadata is compared via a canonical JSON serialization (:func:json.dumps with sort_keys=True), so metadata key order does not affect equality. This means metadata values must be JSON-serializable.

Parameters:

Name	Type	Description	Default
docs	list[Document]	The list of :class:~langchain_core.documents.Document instances to check.	required
raise_error	bool	If True, raises :class:DocumentConsistencyError on the first inconsistency found. If False, logs a warning for each inconsistent document and continues checking the rest.	False

Returns:

Type	Description
bool	True if all documents are consistent, False if at least
bool	one inconsistency was found. Always returns True when
bool	raise_error=True, because an inconsistency raises instead of
bool	returning False.

Raises:

Type	Description
DocumentConsistencyError	if raise_error=True and two documents with the same id have different page_content or metadata.

Example

>>> from langchain_core.documents import Document
>>> from docculus.validation import validate_document_consistency
>>> docs = [
...     Document(id="1", page_content="A", metadata={"source": "a.txt"}),
...     Document(id="1", page_content="A", metadata={"source": "a.txt"}),
... ]
>>> validate_document_consistency(docs)
True