Key-Value Stores¶
This page describes the persista.store package, which provides a uniform key-value
store interface backed by several storage engines. This page explains the BaseStore interface
(which supports both synchronous and asynchronous usage) and how to use the concrete store
implementations: InMemoryStore, SQLiteStore, DuckDBStore, LmdbStore, RedisStore,
PostgresStore, and their "typed"/"pickle" variants.
Prerequisites: You'll need to know a bit of Python. For a refresher, see the Python tutorial.
Overview¶
The persista.store package provides a single, consistent interface for storing dict values
under string keys, regardless of the backend used to persist them:
BaseStore: a single abstract interface that supports both synchronous methods (get,set, ...) and their asynchronous,a-prefixed counterparts (aget,aset, ...) on the same instance
Both modes expose the same set of operations:
get/aget,get_many/aget_many: read one or several values by keyset/aset,set_many/aset_many,set_batches/aset_batches: write one, several, or a stream of valuesfilter/afilter: retrieve values matching field conditionsdelete/adelete,delete_many/adelete_many: remove valuesclear/aclear: remove all valuescontains_many/acontains_many: check which keys existkeys/akeys,values/avalues,iter_batches/aiter_batches: iterate over the store's contentcount/acount: number of entriesclose/aclose: release underlying resources
Because every store implements the same interface, application code written against BaseStore
can be moved between backends — for example using InMemoryStore in unit tests and
PostgresStore in production — without changes, and can freely mix sync and async calls on the
same store instance.
Getting Started¶
In-Memory Store¶
InMemoryStore keeps data in a plain Python dict. It requires no setup and is a good default
for tests and prototyping:
>>> from persista.store import InMemoryStore
>>> with InMemoryStore() as store:
... store.set("1", {"title": "Intro to Python", "author": "Alice"})
... print(store.count())
... print(store.get("1"))
...
1
{'title': 'Intro to Python', 'author': 'Alice'}
Constructing a store does not connect to the underlying backend -- every method (other than
open/aopen) raises RuntimeError until the store has been opened, either by calling
open()/aopen() explicitly or by using it as a context manager, as above. Every store supports
the context manager protocol, which calls open() on entry and close() automatically on exit --
prefer it over calling open()/close() manually so the underlying resources are always released.
Setting Multiple Values¶
set_many writes several values in a single call. filter retrieves the values whose fields
match the given keyword arguments:
>>> from persista.store import InMemoryStore
>>> with InMemoryStore() as store:
... store.set_many(
... {
... "1": {
... "title": "Intro to Python",
... "author": "Alice",
... "category": "Programming",
... },
... "2": {
... "title": "Advanced Python",
... "author": "Alice",
... "category": "Programming",
... },
... "3": {"title": "History of Rome", "author": "Bob", "category": "History"},
... }
... )
... print(len(store.filter(author="Alice")))
... print(len(store.filter(author="Alice", category="Programming")))
... print(len(store.filter(category="History")))
...
2
2
1
Handling Conflicts¶
Every write method accepts an on_conflict argument controlling what happens when a key already
exists:
"overwrite"(default): replace the existing value"raise": raise aKeyError, leaving the existing value unchanged"skip": leave the existing value unchanged"merge": shallow-merge the new value into the existing one; new fields win
>>> from persista.store import InMemoryStore
>>> with InMemoryStore() as store:
... store.set("1", {"title": "Intro to Python", "views": 10})
... store.set("1", {"views": 11}, on_conflict="merge")
... print(store.get("1"))
... store.set("1", {"title": "New title"}, on_conflict="skip")
... print(store.get("1"))
...
{'title': 'Intro to Python', 'views': 11}
{'title': 'Intro to Python', 'views': 11}
Deleting and Clearing¶
>>> from persista.store import InMemoryStore
>>> with InMemoryStore() as store:
... store.set_many({"1": {"a": 1}, "2": {"a": 2}})
... store.delete("1")
... print(store.count())
... store.clear()
... print(store.count())
...
1
0
SQL-Backed Stores¶
SQLite¶
SQLiteStore persists values in a SQLite database, storing each value as a single JSON column.
It works both with a file path and with ":memory:":
>>> from persista.store import SQLiteStore
>>> with SQLiteStore(":memory:") as store:
... store.set_many(
... {
... "1": {
... "title": "Intro to Python",
... "author": "Alice",
... "category": "Programming",
... },
... "2": {
... "title": "Advanced Python",
... "author": "Alice",
... "category": "Programming",
... },
... "3": {"title": "History of Rome", "author": "Bob", "category": "History"},
... }
... )
... print(len(store.filter(author="Alice")))
...
2
To persist to disk, pass a file path instead of ":memory:":
from pathlib import Path
from persista.store import SQLiteStore
with SQLiteStore(Path("tmp/data.sqlite")) as store:
...
DuckDB¶
DuckDBStore works the same way as SQLiteStore but is backed by DuckDB
(requires the duckdb extra):
>>> from persista.store import DuckDBStore
>>> with DuckDBStore(":memory:") as store:
... store.set_many(
... {
... "1": {
... "title": "Intro to Python",
... "author": "Alice",
... "category": "Programming",
... },
... "2": {
... "title": "Advanced Python",
... "author": "Alice",
... "category": "Programming",
... },
... "3": {"title": "History of Rome", "author": "Bob", "category": "History"},
... }
... )
... print(len(store.filter(author="Alice")))
...
2
Typed Stores¶
TypedSQLiteStore, TypedDuckDBStore, and TypedPostgresStore map selected fields onto native
SQL columns instead of storing the whole value as JSON, using a value_schema that maps field
names to SQL types. Fields that are not listed in the schema are still stored (in an extra JSON
overflow column), so filtering by those fields still works, but only the fields declared in the
schema can be used efficiently in filter/indexes:
>>> from persista.store import TypedSQLiteStore
>>> schema = {"author": "TEXT", "year": "INTEGER", "category": "TEXT"}
>>> with TypedSQLiteStore(":memory:", value_schema=schema) as store:
... store.set_many(
... {
... "1": {
... "title": "Intro to Python",
... "author": "Alice",
... "category": "Programming",
... },
... "2": {
... "title": "Advanced Python",
... "author": "Alice",
... "category": "Programming",
... },
... "3": {"title": "History of Rome", "author": "Bob", "category": "History"},
... }
... )
... print(len(store.filter(author="Alice")))
...
2
PostgreSQL¶
PostgresStore (and TypedPostgresStore) connect to a PostgreSQL database using a connection
string, and store values in a configurable table (requires the psycopg extra):
from persista.store import PostgresStore
with PostgresStore(
"postgresql://user:pass@localhost/dbname", table="documents"
) as store:
store.set_many(
{
"1": {"title": "Intro to Python", "author": "Alice"},
"2": {"title": "Advanced Python", "author": "Alice"},
}
)
len(store.filter(author="Alice")) # 2
!!! warning
Unlike the SQLite, DuckDB, LMDB, and Redis stores, BasePostgresStore does not automatically
reopen a closed connection when re-entering a with block. Create a new store instance
instead of reusing one after close().
Embedded and Server-Backed Stores¶
LMDB¶
LmdbStore persists values to a memory-mapped LMDB database on
disk, with no separate server process required (requires the lmdb extra):
from persista.store import LmdbStore
with LmdbStore("/tmp/lmdb_store") as store:
store.set_many(
{
"1": {
"title": "Intro to Python",
"author": "Alice",
"category": "Programming",
},
"2": {
"title": "Advanced Python",
"author": "Alice",
"category": "Programming",
},
"3": {"title": "History of Rome", "author": "Bob", "category": "History"},
}
)
len(store.filter(author="Alice")) # 2
Use PickleLmdbStore to store arbitrary Python objects (not just JSON-serializable dict
values) using pickle instead of JSON:
from persista.store import PickleLmdbStore
with PickleLmdbStore("/tmp/lmdb_store") as store:
store.set("1", {"title": "Intro to Python", "tags": {"python", "intro"}})
store.get("1") # {'title': 'Intro to Python', 'tags': {'python', 'intro'}}
!!! warning
pickle.loads can execute arbitrary code. Only use PickleLmdbStore (and
PickleRedisStore) with data from trusted sources.
Redis¶
RedisStore stores values in Redis, encoded as JSON (requires the redis
extra):
from persista.store import RedisStore
with RedisStore("redis://localhost:6379/0") as store:
store.set_many(
{
"1": {
"title": "Intro to Python",
"author": "Alice",
"category": "Programming",
},
"2": {
"title": "Advanced Python",
"author": "Alice",
"category": "Programming",
},
"3": {"title": "History of Rome", "author": "Bob", "category": "History"},
}
)
len(store.filter(author="Alice")) # 2
Use PickleRedisStore to store arbitrary Python objects using pickle instead of JSON.
Async Usage¶
Every store also exposes a-prefixed asynchronous methods (aget, aset, acount, ...) that
are coroutines (or async iterators), so they must be awaited and used from an async function.
The same store instance can be used from both sync and async code -- there is no separate async
class. Async methods are available on every store, including InMemoryStore, SQLiteStore
(async methods use the aiosqlite extra when installed, falling back to a thread otherwise),
RedisStore, and PostgresStore (requires the psycopg extra); DuckDBStore and LmdbStore
also expose async methods, backed by a thread pool.
>>> import asyncio
>>> from persista.store import InMemoryStore
>>> async def main():
... async with InMemoryStore() as store:
... await store.aset("1", {"text": "hello"})
... print(await store.acount())
... print(await store.aget("1"))
...
>>> asyncio.run(main())
1
{'text': 'hello'}
SQLiteStore and TypedSQLiteStore's async methods behave like their sync counterparts:
>>> import asyncio
>>> from persista.store import SQLiteStore
>>> async def main():
... async with SQLiteStore(":memory:") as store:
... await store.aset_many(
... {
... "1": {"title": "Intro to Python", "author": "Alice"},
... "2": {"title": "Advanced Python", "author": "Alice"},
... "3": {"title": "History of Rome", "author": "Bob"},
... }
... )
... result = await store.afilter(author="Alice")
... print(len(result))
...
>>> asyncio.run(main())
2
RedisStore/PickleRedisStore and PostgresStore/TypedPostgresStore follow the same
pattern, connecting to a running Redis or PostgreSQL server:
import asyncio
from persista.store import PostgresStore
async def main():
async with PostgresStore("postgresql://user:pass@localhost/dbname") as store:
await store.aset_many(
{
"1": {"title": "Intro to Python", "author": "Alice"},
"2": {"title": "Advanced Python", "author": "Alice"},
}
)
result = await store.afilter(author="Alice")
print(len(result))
asyncio.run(main())
Using async with (as above) calls aopen() on entry and aclose() automatically on exit.
Checking Which Keys Exist¶
contains_many checks a batch of keys at once, returning a list of booleans in the same order
as the input, without fetching the values themselves:
>>> from persista.store import InMemoryStore
>>> with InMemoryStore() as store:
... store.set_many({"1": {"a": 1}, "2": {"a": 2}})
... print(store.contains_many(["1", "2", "3"]))
...
[True, True, False]
split_present_missing turns those flags into two key lists, which is often more convenient than
zipping keys and flags yourself:
>>> from persista.store import InMemoryStore, split_present_missing
>>> with InMemoryStore() as store:
... store.set_many({"1": {"a": 1}, "2": {"a": 2}})
... keys = ["1", "2", "3"]
... present, missing = split_present_missing(keys, store.contains_many(keys))
... print(present)
... print(missing)
...
['1', '2']
['3']
Iterating Over a Store¶
keys, values, and iter_batches iterate over a store's content without loading everything
into memory at once:
>>> from persista.store import InMemoryStore
>>> with InMemoryStore() as store:
... store.set_many({"1": {"a": 1}, "2": {"a": 2}, "3": {"a": 3}})
... print(sorted(store.keys()))
... print(sorted(v["a"] for v in store.values()))
...
['1', '2', '3']
[1, 2, 3]
set_batches mirrors set_many but consumes an iterable of (key, value) pairs and writes them
in mini-batches, which is useful when the source data does not fit comfortably in memory:
>>> from persista.store import InMemoryStore
>>> with InMemoryStore() as store:
... store.set_batches((str(i), {"value": i}) for i in range(5))
... print(store.count())
...
5
Store URIs¶
Every store implements to_uri(), which returns a URI identifying where its data lives, and the
matching from_uri(uri, *, read_only=False) classmethod, which reconstructs a store of the same
class from that URI:
>>> from persista.store import SQLiteStore
>>> with SQLiteStore("tmp/data.sqlite") as store:
... uri = store.to_uri()
...
>>> with SQLiteStore.from_uri(uri) as reloaded:
... pass
...
read_only is honored by the SQLite, DuckDB, and LMDB stores (and their Typed/Pickle
variants); it's accepted but ignored everywhere else. to_uri/from_uri do not preserve
constructor options like value_schema (typed stores) or table (Postgres stores) -- from_uri
always reconstructs with the defaults. InMemoryStore and NullStore always round-trip to a
fresh, empty store since they carry no reconnection information.
If you don't know the concrete store class ahead of time, store_from_uri dispatches on the
URI's scheme to the right class automatically:
>>> from persista.store import JsonFileStore, store_from_uri
>>> with JsonFileStore("data") as store:
... store.set("1", {"title": "Intro to Python"})
... uri = store.to_uri()
...
>>> with store_from_uri(uri) as reloaded:
... print(isinstance(reloaded, JsonFileStore))
... print(reloaded.get("1"))
...
True
{'title': 'Intro to Python'}
Store classes that share a scheme with another class (TypedPostgresStore and PostgresStore
both use postgresql://, PickleRedisStore and RedisStore both use redis://) aren't
reachable through the dispatcher -- call TheClass.from_uri(uri) directly for those.
Use register_scheme to register a custom store class (or override a built-in one) under a
given scheme:
from persista.store import register_scheme, store_from_uri
from my_project.stores import MyCustomStore
register_scheme("mycustom", MyCustomStore)
with store_from_uri("mycustom://...") as store:
...
Choosing a Store¶
| Store | Backend | Persisted | Typed columns | Pickle values | Async |
|---|---|---|---|---|---|
InMemoryStore |
Python dict |
No | No | N/A | Yes |
SQLiteStore |
SQLite | Yes | Yes (Typed…) |
No | Yes |
DuckDBStore |
DuckDB | Yes | Yes (Typed…) |
No | Yes (thread pool) |
LmdbStore |
LMDB | Yes | No | Yes (Pickle…) |
Yes (thread pool) |
RedisStore |
Redis | Yes | No | Yes (Pickle…) |
Yes |
PostgresStore |
PostgreSQL | Yes | Yes (Typed…) |
No | Yes |
NullStore |
None (discards everything) | No | No | N/A | Yes |
Use InMemoryStore for tests and prototyping, SQLiteStore/DuckDBStore for local
single-process persistence without a server, and RedisStore/PostgresStore when data needs to
be shared across processes or machines. NullStore never actually stores anything -- every
get/aget is a miss -- which is useful for plugging into Cache to disable caching entirely
without changing any calling code.
API Reference¶
See the reference documentation for the full API.