Memory¶
arcana.memory holds the concrete storage backends behind the
MemoryAdapter protocol. The
protocol is the seam: agents read and write through it and never learn which
backend is underneath, so swapping stores is a matter of supplying a different
adapter.
SQLiteAdapter is the first backend — a single-file, async store built on
aiosqlite. One adapter instance owns one
.db file; an agent's private memory lives at
~/.arcana/agents/{agent_id}/memory.db.
from uuid import uuid4
from arcana.memory import SQLiteAdapter
from arcana.types import MemoryEntry, MemoryQuery, MemoryType
agent_id = uuid4()
memory = SQLiteAdapter.for_agent(agent_id)
await memory.connect() # opens the file, runs migrations
await memory.write(
MemoryEntry(
agent_id=agent_id,
type=MemoryType.SEMANTIC,
content="The user prefers metric units.",
importance=0.8,
)
)
results = await memory.search(MemoryQuery(agent_id=agent_id, limit=5))
await memory.aclose()
Keyword search¶
When a query carries text, search() ranks results by full-text relevance
(BM25) using a SQLite FTS5 index over each entry's content and tags. The
usual metadata filters — scope, type, confidence, time range, conflict
exclusion — still apply on top of the text match. A query with no text falls
back to filter-and-order: pinned first, then importance, then recency.
from arcana.types import MemoryQuery, RetrievalMode
results = await memory.search(
MemoryQuery(
agent_id=agent_id,
text="metric units preference",
retrieval_mode=RetrievalMode.keyword,
limit=5,
)
)
Arbitrary user text is safe to pass directly: it is sanitized into a valid FTS5
MATCH expression (operator characters stripped, tokens quoted and OR-joined),
so it can never raise a syntax error. The index is kept in lockstep with the
entries table by database triggers, so writes, upserts, and deletes need no
extra bookkeeping.
FTS5 must be compiled into SQLite — connect() raises MemoryStorageError if
the build lacks it.
Importance-based promotion¶
When an adapter is constructed with a global_store, writing a PRIVATE entry
whose importance >= 0.9 also copies it into the global store as a GLOBAL
entry — the mechanism behind "all agents read; The World writes". Without a
global_store, promotion is a no-op.
global_store = SQLiteAdapter(global_db_path)
agent_memory = SQLiteAdapter.for_agent(agent_id, global_store=global_store)
Schema migrations¶
Schema is versioned with SQLite's built-in PRAGMA user_version — deliberately
no Alembic or SQLAlchemy, to keep arcana-core dependency-light.
connect() brings the database to the latest version automatically.
The arcana.memory.migrations package separates the runner (runner.py, the
forward-only engine that applies migrations) from the definitions
(versions/, one module per schema version). Definitions are append-only:
never edit a shipped migration — add a new versions/vNNN_*.py module and
register it. Each migration's DDL and its user_version bump share a
transaction, so a partial failure rolls back atomically.
Embedding gateway and model pinning¶
Vector search needs an embedder, and which one a database may use is pinned.
The first model to write an embedding is recorded in the database's
embedding_meta row (added by migration v3), and the database stays locked to
it. EmbeddingGateway.resolve() turns that pin into the
adapter to embed with:
- New database (no pin yet) → the first healthy adapter, in priority order; the database is then pinned to it.
- Pinned database → the adapter for its exact model when healthy; otherwise a
healthy adapter in the same
model_family(interchangeable vectors); otherwiseNone.
A None result is the signal to fall back to keyword (FTS5) search. The gateway
never substitutes a model from a different family — that would compare vectors
across incompatible spaces and corrupt similarity scores with no error raised.
from arcana.memory import EmbeddingGateway
from arcana.models.adapters.ollama_embedding import OllamaEmbeddingAdapter
from arcana.models.adapters.fastembed_embedding import FastEmbedEmbeddingAdapter
# Priority order: Ollama first, fastembed second.
gateway = EmbeddingGateway([OllamaEmbeddingAdapter(), FastEmbedEmbeddingAdapter()])
adapter = await gateway.resolve(db_meta) # db_meta: EmbeddingMeta | None
if adapter is not None:
vector = await adapter.embed("some text")
else:
... # fall back to FTS5 keyword search
The gateway is pure resolution logic: it takes the database's
EmbeddingMeta (or None) and
returns an adapter. Reading and writing the embedding_meta row belongs to the
vector backend that consumes the gateway — that backend is VectorAdapter,
below.
Vector search (semantic)¶
VectorAdapter adds semantic search on top of a SQLiteAdapter. It composes
(does not subclass) the SQLite store and an EmbeddingGateway, storing
embeddings in a sqlite-vec vec0 index
that lives in the same memory.db — so row and vector writes share one
connection and stay consistent. Vector storage is an optional extra:
from arcana.memory import EmbeddingGateway, SQLiteAdapter, VectorAdapter
from arcana.models.adapters.ollama_embedding import OllamaEmbeddingAdapter
from arcana.types import MemoryQuery, RetrievalMode
memory = VectorAdapter(
SQLiteAdapter.for_agent(agent_id),
EmbeddingGateway([OllamaEmbeddingAdapter()]),
)
await memory.connect()
await memory.write(entry) # embeds content, indexes the vector
results = await memory.search(
MemoryQuery(
agent_id=agent_id,
text="units the user likes",
retrieval_mode=RetrievalMode.semantic,
limit=5,
)
)
On write(), the adapter resolves an embedder through the gateway, embeds the
entry's content (when no vector is supplied), and stores it in the index. The
first embedded write both creates the dimension-sized index and pins the
database to the model (writing its embedding_meta row); vectors are
L2-normalized and the index ranks by cosine distance. A vector whose width
disagrees with the pin is refused with MemoryStorageError rather than
silently corrupting the index.
search() embeds the query, ranks by nearest-neighbour cosine distance, then
applies the same metadata filters as keyword search. It falls back to FTS5
keyword search (with a one-time warning) whenever no compatible embedder is
healthy — or when sqlite-vec is not installed — so memory stays usable without
the extra, just without semantic ranking. A query with no text uses the
filter-and-order path.
Hybrid retrieval¶
RetrievalMode.hybrid fuses the vector and keyword legs into one ranking.
Because cosine distance and BM25 sit on different, incompatible scales, each leg
is converted to a higher-is-better relevance and min-max normalized to
[0, 1] within the query's candidate pool, then combined:
Per-query normalization keeps either leg from dominating purely because of scale. The weights are set on the adapter and normalized to sum 1, defaulting to 0.7 vector / 0.3 keyword:
memory = VectorAdapter(sqlite, gateway, vector_weight=0.7, bm25_weight=0.3)
results = await memory.search(
MemoryQuery(
agent_id=agent_id,
text="metric units",
retrieval_mode=RetrievalMode.hybrid,
limit=5,
)
)
An entry surfaced by only one leg contributes 0 for the other. With no healthy embedder, hybrid degrades to keyword-only (the BM25 leg alone).
Folder connector (Markdown)¶
MarkdownFolderAdapter exposes a directory of Markdown notes as retrievable
memory — the zero-setup way to make an Obsidian vault (or any notes folder)
searchable by an agent. It implements the same MemoryAdapter protocol, so it
registers as a federation tier or attaches to an agent's memory slot with
nothing but a folder path: no plugin, no server, no sync job.
One .md/.markdown file becomes one MemoryEntry. The id is a stable uuid5
of the file's root-relative path, so re-reads — and any future ingest into
SQLite — upsert instead of duplicating, and the federation dedups a folder note
against an ingested copy by that shared id. YAML frontmatter maps to fields
(type, importance, pinned, tags), Obsidian #hashtags fold into tags,
and malformed frontmatter is tolerated — one bad note never fails the scan.
The folder has no FTS5 or vector index, so every retrieval mode collapses to
keyword: a semantic or hybrid query is served by the same in-process
substring/token scan with a one-line degraded notice, never an error. Results
rank pinned → lexical relevance → importance → recency. A process-local index
cache keyed by path → (mtime, size) keeps a live read fresh — each search()
restats the tree and re-parses only changed or new files — so the connector
works standalone with zero sync infrastructure.
It is read-only: reads never mutate files, and write() raises
MemoryWriteError (the folder is an external source of truth, not a sink).
Dotfiles and dot-dirs (.obsidian/, .trash/), symlinks, oversized files
(max_file_bytes, default 1 MiB), and any configurable ignore-glob are skipped;
health_check() is a cheap stat + readability probe on the root and never
raises.
from pathlib import Path
from uuid import uuid4
from arcana.memory import MarkdownFolderAdapter
from arcana.types import MemoryQuery, MemoryScope
vault = MarkdownFolderAdapter(
Path("~/Documents/MyVault").expanduser(),
agent_id=uuid4(),
scope=MemoryScope.SHARED, # e.g. registered as a shared read tier
pool_name="vault",
)
results = await vault.search(MemoryQuery(text="metric units", limit=5))
Pruning¶
prune() removes low-value entries per a
PrunePolicy. The min_importance
floor and the max_entries cap compose into one victim set (an entry is removed
if it falls below the floor or sits outside the top-N by importance), and
pinned entries are never touched. PruneMode.ARCHIVE soft-deletes (sets
archived, hiding the entry from search but keeping it recoverable);
PruneMode.PURGE hard-deletes the row — and its vector, if a vec0 index
exists. A PruneReport records what
was scanned, archived, and purged.
from arcana.types import PrunePolicy, PruneMode
report = await memory.prune(PrunePolicy(min_importance=0.2, max_entries=10_000))
Knowledge graph (edges)¶
Beyond similarity, memory carries an explicit graph: typed, directed edges
between nodes, stored in a memory_edges table (migration v4) that lives in
the same SQLite database as the rows — a property graph, no separate engine.
EdgeStore is its read/write layer. An edge is a
MemoryEdge: src_id → dst_id under
a relation, tagged by the source that produced it and carrying a
confidence. Endpoints are stable node ids, so an edge can relate nodes that
live in any tier — a folder connector's uuid5 note id is a valid endpoint even
though it is not a stored row.
The (src_id, dst_id, relation) primary key makes writes idempotent. A producer
owns its source and rewrites exactly that set with replace_source(), so a
re-index stays convergent as links are added, removed, or re-pointed.
neighbors() returns the ids one hop away in either direction — the seed→expand
primitive for graph-aware retrieval.
Wikilinks → edges¶
WikilinkEdgeExtractor populates the graph from a Markdown folder: it parses
Obsidian [[wikilinks]] (and ![[embeds]]) out of the notes a
MarkdownFolderAdapter already reads and writes them as references edges. It
is fully deterministic — regex over text, resolution by filename or path, no
model in the loop — so it carries none of the hallucinated-edge risk that
inferred edges would. Targets resolve within the folder (a bare [[Note]] by
basename, [[folder/Note]] by path); ambiguous and dangling links are skipped
and counted, never guessed. Each reindex() rewrites the whole wikilink edge
set and returns an EdgeIndexReport tally.
from arcana.memory import (
EdgeStore,
MarkdownFolderAdapter,
SQLiteAdapter,
WikilinkEdgeExtractor,
)
reader = MarkdownFolderAdapter(vault_path, agent_id)
edges = EdgeStore(SQLiteAdapter.for_agent(agent_id))
await edges.connect()
report = await WikilinkEdgeExtractor(reader, edges).reindex()
neighbours = await edges.neighbors(note_id) # ids one hop away
Populates, doesn't traverse
The extractor writes edges; consuming them at read time — seed by vector/keyword, expand along edges, re-rank — is a federation concern and is not wired into the read path yet.
Federation across tiers¶
A MemoryFederation presents many tier backends as a single MemoryAdapter, so
an Agent holds it exactly where it would hold one store and the topology stays
invisible. Three scopes make up the topology:
- private — the agent's own store (the durability anchor);
- shared — named pools an agent group reads and writes;
- global — the shared-by-all tier, the mechanism behind "all agents read; The World writes".
A MemoryRouter owns the pure routing policy; the federation performs the I/O.
- Fan-out writes —
route_write()decides the target tiers, and the federation writes each concurrently. A high-importance (>= 0.9)PRIVATEentry is also written toGLOBALas a scope-rewritten copy (promotion, sameid). Writes are not transactional across tiers. - Merged reads —
route_read()fans a query across the routed tiers concurrently; results are deduplicated byid(the most-local tier wins) and re-ranked by the agent'sMemoryWeightsbefore truncation toquery.limit.stream_search()yields the same ranked sequence one entry at a time.
Reads and writes treat failure differently, on purpose: a failed read tier is
dropped so the agent still sees the other tiers' results (partial memory beats
none), while a failed private write raises MemoryWriteError — a lost write to
the durability anchor is data loss the caller must know about. SHARED/GLOBAL
write failures degrade instead (a degraded GLOBAL simply pauses promotion).
from arcana.memory import MemoryFederation, MemoryRouter, SQLiteAdapter
router = MemoryRouter(
private=SQLiteAdapter.for_agent(agent_id),
global_=SQLiteAdapter(global_db_path),
pools={"team-research": SQLiteAdapter(pool_db_path)},
weights=agent_config.memory_weights,
)
memory = MemoryFederation(router)
await memory.write(entry) # fans out to every routed tier
results = await memory.search(MemoryQuery(agent_id=agent_id)) # merged + ranked
Assembling a federation for an agent¶
Wiring the router and tiers by hand (above) is the low-level API. In practice one
call does the whole assembly: build_federation() turns an agent id plus the
~/.arcana home into a ready MemoryFederation.
from pathlib import Path
from uuid import uuid4
from arcana.memory import EmbeddingGateway, PoolConfig, build_federation
from arcana.models.adapters.fastembed_embedding import FastEmbedEmbeddingAdapter
federation = await build_federation(
uuid4(),
home=Path.home() / ".arcana",
embedding=EmbeddingGateway([FastEmbedEmbeddingAdapter()]), # optional
pools=[PoolConfig("team-research", pool_adapter)], # optional
)
# ... agent.run(...) ...
await federation.aclose() # release the private handle and any vector store
It builds the tiers the way the runtime expects:
- private — per-agent SQLite at
~/.arcana/agents/{id}/memory.db, opened eagerly so migrations run and a corrupt store is quarantined before first use. This tier is always present, the durability anchor. - global — a shared vector store at
~/.arcana/vector/global.db, wired only when anEmbeddingGatewayis supplied. It is semantic when the embedder is healthy and keyword (FTS5) when not; with no embedder the tier is dropped and the agent runs private-only, so a zero-config install still works. - shared — each
PoolConfigis registered on the router by name.
Degradations route to the observability audit log by
default (pass on_degraded to override), so a thinned SHARED/GLOBAL tier is
visible without failing the run.
Injection and configuration¶
Callers rarely invoke build_federation directly.
AgentRegistry.build_runtime_with_memory() assembles and injects a
federation for a stored agent, returning (agent, federation) so the caller owns
teardown. The CLI arcana run path uses it, so agents remember across sessions
by default. Opt out of a single run with --no-memory, or globally via the
memory block in ~/.arcana/config.json:
{
"memory": {
"enabled": true,
"private": "sqlite",
"global": "vector",
"pools": [],
"extraction": {
"strategy": "heuristic",
"agent_confidence_cap": 0.7,
"summarise_on_close": true,
"min_confidence_to_store": 0.3
}
}
}
The extraction block selects the memory extractor and its thresholds: strategy
is "heuristic" (default) or "llm", agent_confidence_cap bounds agent-asserted
confidence below 1.0, summarise_on_close toggles the consolidated
session-summary memory, and min_confidence_to_store drops weak entries before
they are written. An absent block yields these defaults, and "llm" with no model
provider falls back to the heuristic.
The global vector tier activates when an embedding provider is available: the CLI
uses in-process fastembed when the
arcana-os[embed] extra is installed, and stays private-SQLite-only otherwise.
Extraction and summarisation¶
What an agent writes to memory is decided by a MemoryExtractor: it turns one
completed turn — (prompt, response, session) — into a small list of typed,
confidence-scored MemoryEntrys,
rather than one blanket slice of the response. Correct typing matters because
MemoryType selects the decay profile — episodic decays fast, semantic slow,
procedural very slow — so a durable fact and a throwaway exchange are retained on
different clocks.
Two strategies sit behind one interface:
HeuristicExtractor— the default: deterministic and model-free. It records oneEPISODICentry per turn, promotes a stated user preference toSEMANTIC, and turns a how-to answer into aPROCEDURALentry. A promotedSEMANTICentry stores the distilled clause — the fact itself with its conversational framing ("by the way, remember that …") stripped — rather than the whole prompt, so the stored memory (and every later prompt injection of it) is the fact, not the wrapper. Free, testable, and it adds no round-trip.LLMExtractor— opt-in: a single low-temperature gateway call returning a small JSON list of candidate memories. Any model error, malformed JSON, or empty result falls back to the heuristic for that turn, so extraction can never crash a run.
Honest confidence is the anti-poisoning guarantee: agent-generated text is
capped below 1.0 and sourced as AGENT (so better evidence can override it
later), while an explicit user statement ("remember that I …") is trusted higher
and sourced as USER_CONFIRMED. Entries below min_confidence_to_store are
dropped before the write. Importance is derived from signal — imperative or
"remember" language, a pin — not a constant.
Extraction is best-effort throughout: a failure is logged to the audit log and
swallowed, never surfaced into the user-facing run.
from arcana.memory import HeuristicExtractor, build_extractor, ExtractionConfig
# The default, wired automatically by the runtime.
extractor = HeuristicExtractor()
entries = await extractor.extract(prompt, response, session) # list[MemoryEntry]
# Or select by config; "llm" with no model configured falls back to heuristic.
extractor = build_extractor(ExtractionConfig(strategy="llm"), gateway=gw, model="ollama/hermes-3")
Session summaries. SessionManager distils a whole session into
Session.summary and writes one consolidated memory on close. summarise() sets
the summary (using the extractor, or a deterministic heuristic fallback) and
persists it; close_and_summarise() then writes a single consolidated entry —
SEMANTIC when the session stated a durable fact, else EPISODIC — carrying that
summary at a higher baseline importance. close() stays synchronous and does
neither, so a sync caller never triggers a model call by accident.
summary = await session_manager.summarise(session) # sets session.summary
await session_manager.close_and_summarise(session, memory=federation)
Selection and thresholds live in the extraction sub-block of the memory
config (see below); with no model provider, extraction is forced to the
deterministic heuristic. strategy is the
ExtractionStrategy enum
(heuristic / llm).
Env-overridable tunables. The scoring knobs — confidence caps, importance
baselines, and the per-entry content cap (MAX_ENTRY_CONTENT, the ceiling
trim_content enforces) — ship as defaults but read ARCANA_EXTRACTION_*
environment variables at import time, so code built on Arcana can tune extraction
without a release. Precedence, high to low: config.json → ARCANA_EXTRACTION_*
env → built-in default. For example, ARCANA_EXTRACTION_MAX_CONTENT=1000 widens
stored entries and ARCANA_EXTRACTION_AGENT_CONFIDENCE_CAP=0.6 lowers the
agent-confidence ceiling. Bounds are validated: agent_confidence_cap must stay
strictly below 1.0 — a value of 1.0 would let agent-asserted text become
un-overridable, defeating anti-poisoning, so it is rejected at load rather than
silently accepted.
Language signals. The heuristic's surface cues (imperative/"remember"
language, stated preferences, how-to questions, step lists, and the leading
framing stripped when distilling a durable clause) are language-specific and live
in arcana.memory.extraction.signals, decoupled from the extractor.
English ships as the default; a new language is a SignalPatterns registered via
register_language() (or passed straight to HeuristicExtractor(signals=...)) —
the extractor itself never changes.
Resilience¶
The router wraps every tier in a ResilientTier before handing it to the
federation, so a slow, locked, or corrupt store can never stall or sink a whole
session. Each wrapper bounds calls with a timeout, contains failures behind a
per-tier circuit breaker, and surfaces the thinning as a
MemoryDegradedEvent rather than swallowing it
silently.
- Reads are total — a timeout, open breaker, corruption, or backend error
yields
[]after emitting a degraded event. - Writes are partial — the same conditions raise
TierWriteFailedcarrying the tier's scope, so the federation decides the blast radius (PRIVATEfatal,SHARED/GLOBALdegrade).
A CircuitBreaker trips after fail_threshold consecutive failures during real
traffic, fails fast while OPEN, then allows one HALF_OPEN probe once
reset_after_seconds elapses. Corruption is special: a MemoryCorruptError is a
session-long condition, so it forces the breaker open (quarantine) rather than
counting as one transient failure.
Timeout budgets and breaker thresholds are per tier — a keyword read hits local
SQLite, while a semantic read may call a remote embedder — and configurable via
~/.arcana/connections/memory-adapters.json (a missing file yields safe
defaults, so existing programmatic wiring keeps working):
{
"private": { "read_timeout_ms": 250, "semantic_timeout_ms": 1500, "write_timeout_ms": 500 },
"global": { "read_timeout_ms": 400, "write_timeout_ms": 600 },
"shared": { "team-research": { "read_timeout_ms": 400 } },
"default_shared": { "read_timeout_ms": 400, "write_timeout_ms": 600 }
}
Background jobs¶
Post-session extraction and periodic consolidation are meant to run off the
request path. BackgroundJobQueue models that as a bounded queue with
load-shedding: submit() never blocks, always accepting critical jobs (e.g.
a user-confirmed preference) while shedding non-critical jobs once the
backlog's drain estimate (depth × EWMA(service time)) exceeds its headroom.
Design-ahead, not yet wired
Extraction runs inline on every agent turn
(Agent.run and Agent.stream) via the memory extractor; nothing drains this
queue in the live path yet. Only the
queue interface and its depth/drain metrics ship today — the consumer loop
lands when extraction and consolidation actually move off the request path.
Adapters¶
Async SQLite memory backend. Implements the MemoryAdapter protocol.
Source code in packages/arcana-core/arcana/memory/adapters/sqlite.py
property
¶
The live connection, for sibling adapters sharing this database.
VectorAdapter layers a vec0 index onto the same memory.db and
needs this exact connection so its vector writes sit in the same database
as the rows. Raises if the adapter was never connected.
classmethod
¶
Build an adapter at ~/.arcana/agents/{agent_id}/memory.db.
Mirrors SessionManager's path convention so an agent's memory lives
beside its sessions.
Source code in packages/arcana-core/arcana/memory/adapters/sqlite.py
async
¶
Open the connection, set pragmas, and migrate to the latest schema.
Idempotent — safe to call repeatedly. search/write lazily call
this, so explicit connect() is optional.
Source code in packages/arcana-core/arcana/memory/adapters/sqlite.py
async
¶
async
¶
Probe the backend with a trivial query. Never raises.
Ensures the connection (which validates the store), then runs
SELECT 1. Any failure — unconnectable, corrupt, or FTS5-less —
reports unhealthy so the resilience layer can gate on it as its
half-open recovery probe without exception handling.
Source code in packages/arcana-core/arcana/memory/adapters/sqlite.py
async
¶
Upsert one entry (keyed on id), then promote to GLOBAL if eligible.
Source code in packages/arcana-core/arcana/memory/adapters/sqlite.py
async
¶
Return entries matching query.
When the query carries usable text, results are ranked by FTS5 BM25
relevance (keyword search). Otherwise it's filter-and-order:
pinned → importance → recency. Any text query — whatever its
retrieval_mode — is currently served by the keyword path.
Source code in packages/arcana-core/arcana/memory/adapters/sqlite.py
async
¶
Post-read bookkeeping: refresh access tracking and emit the read event.
Public so a sibling adapter (VectorAdapter) whose semantic path
bypasses search shares the same access-refresh + audit behaviour.
started is a time.perf_counter() stamp taken before the query ran.
Source code in packages/arcana-core/arcana/memory/adapters/sqlite.py
async
¶
Remove low-value entries per policy; pinned entries are never touched.
ARCHIVE soft-deletes (sets archived, hiding entries from search but
keeping them recoverable); PURGE hard-deletes the row — and its vector,
if a vec index exists. The importance floor and max-entries cap compose
into a single victim set, removed in one committed transaction.
Source code in packages/arcana-core/arcana/memory/adapters/sqlite.py
Semantic memory backend: sqlite-vec KNN over a SQLiteAdapter's database.
Implements the MemoryAdapter protocol (search/write). Composes —
does not subclass — a SQLiteAdapter, reusing its row storage, FTS5 index,
importance-based promotion, and access tracking unchanged, and adds the vector
layer on top.
Source code in packages/arcana-core/arcana/memory/adapters/vector.py
async
¶
Connect the underlying store and load the sqlite-vec extension once.
Idempotent. If sqlite-vec is missing or the SQLite build cannot load
extensions, the adapter degrades to keyword-only (one visible warning)
rather than failing — arcana stays usable without the [vector] extra.
Source code in packages/arcana-core/arcana/memory/adapters/vector.py
async
¶
async
¶
Report health of the underlying store. Never raises.
Storage health is the underlying SQLite store: if rows are readable, the
tier is usable. A missing vector layer is degraded but healthy —
keyword retrieval still answers — so it is noted in message rather
than reported as unhealthy.
Source code in packages/arcana-core/arcana/memory/adapters/vector.py
async
¶
Prune the underlying store. Shares the connection, so a PURGE here also clears the vec0 index rows for the removed entries.
Source code in packages/arcana-core/arcana/memory/adapters/vector.py
async
¶
Store entry (row + FTS5 + promotion) and index its vector.
Resolves the embedder via the gateway. With no healthy embedder the entry
is still stored and keyword-searchable, just without a vector. The first
embedded write pins the database to the model (embedding_meta) and
creates the dimension-sized vec0 index.
Source code in packages/arcana-core/arcana/memory/adapters/vector.py
async
¶
Return entries matching query, semantically when possible.
Semantic KNN runs for semantic/hybrid queries that carry usable
text against a pinned database with a healthy compatible embedder.
Otherwise — keyword mode, no text, an unpinned database, or no healthy
embedder — results come from the keyword/filter path on the underlying
store.
Source code in packages/arcana-core/arcana/memory/adapters/vector.py
MarkdownFolderAdapter(
root,
agent_id,
*,
scope=PRIVATE,
pool_name=None,
default_type=SEMANTIC,
ignore_globs=_DEFAULT_IGNORE_GLOBS,
max_file_bytes=1048576,
)
Read-only MemoryAdapter backed by a folder of Markdown notes.
One instance owns exactly one folder root and is scoped to one agent_id;
composing multiple folders is the federation layer's job. Implements search,
write (raises — the source is read-only), and health_check.
Source code in packages/arcana-core/arcana/memory/adapters/markdown.py
async
¶
Scan the folder and return entries matching query, keyword-ranked.
The folder has no semantic or full-text index, so semantic/hybrid
requests are served by the same keyword path with a one-line degraded
notice — the mode is never an error. Ranking is pinned → lexical relevance
→ importance → recency, ties broken by path for determinism, truncated to
query.limit.
Source code in packages/arcana-core/arcana/memory/adapters/markdown.py
async
¶
Reject writes — the folder is an external source of truth, not a sink.
Silently dropping a write would hide data loss, and the federation's writer propagates tier failures deliberately, so a read-only tier must say so.
Source code in packages/arcana-core/arcana/memory/adapters/markdown.py
async
¶
Report whether the root is a readable directory. Never raises.
Kept cheap — a single stat + os.access on the root, no tree walk —
so the resilience layer can use it as a half-open recovery probe.
Source code in packages/arcana-core/arcana/memory/adapters/markdown.py
async
¶
Return every note as a public (entry, rel_path) pair.
Exposes the folder's parsed view — the same cached scan search uses —
for consumers that need the on-disk path alongside the entry, such as
resolving wikilink targets to note ids.
Source code in packages/arcana-core/arcana/memory/adapters/markdown.py
async
¶
Return the note with stable id entry_id, or None if absent.
Lets a graph-expansion read rehydrate a folder-resident neighbour by id.
The id is a one-way uuid5 of the path, so this resolves by matching over
the (cached) tree rather than reversing the hash.
Source code in packages/arcana-core/arcana/memory/adapters/markdown.py
dataclass
¶
A note as seen by a consumer: its entry plus the on-disk rel_path.
The public projection of the folder's parsed view. Exposes the root-relative POSIX path alongside the entry so a consumer (e.g. wikilink resolution) can map link targets to notes without re-walking the tree.
Knowledge graph¶
Async CRUD over memory_edges. Composes a :class:SQLiteAdapter.
Source code in packages/arcana-core/arcana/memory/edges.py
async
¶
async
¶
Insert or replace edges (keyed on src_id, dst_id, relation).
Returns the number of edge rows written.
Source code in packages/arcana-core/arcana/memory/edges.py
async
¶
Atomically replace every edge tagged source with edges.
Delete-then-insert in one transaction: a producer owns its source and
rewrites that whole set each pass, so removed or re-pointed edges disappear
without any per-edge diffing. Returns the number of edges written.
Source code in packages/arcana-core/arcana/memory/edges.py
async
¶
async
¶
async
¶
Distinct node ids adjacent to node_id in either direction.
The seed→expand step of graph-aware retrieval: given a node, the ids of everything one hop away, deduped and order-stable (outgoing then incoming).
Source code in packages/arcana-core/arcana/memory/edges.py
Extract [[wikilink]] edges from a folder into an EdgeStore.
Source code in packages/arcana-core/arcana/memory/wikilinks.py
async
¶
Scan the folder, resolve every wikilink, and rewrite the edge set.
Rewrites the whole wikilink source in one shot, so the graph reflects
exactly the links present on disk right now. Returns a tally of what was
found and written.
Source code in packages/arcana-core/arcana/memory/wikilinks.py
dataclass
¶
EdgeIndexReport(
notes_scanned=0,
links_found=0,
resolved=0,
dangling=0,
ambiguous=0,
edges_written=0,
)
Outcome of a re-index pass. All counts are per-occurrence except the last.
Federation and routing¶
async
¶
build_federation(
agent_id,
*,
home,
embedding=None,
pools=None,
decay_profiles=None,
on_degraded=None,
)
Assemble the tier stack for one agent and return its federation.
- PRIVATE — per-agent SQLite at
<home>/agents/{id}/memory.db, the durability anchor. Opening it runs pending migrations and, on a corrupt file, quarantines the store before first use. - GLOBAL — a shared vector store at
<home>/vector/global.db, present only whenembeddingis given. It is semantic when the embedder is healthy and keyword (FTS5) when not; with no embedder the tier is dropped entirely and the agent runs private-only. Shared across agents, so its open-time integrity check is off (it detects corruption on read instead). - SHARED — each pool in
poolsis registered on the router by name.
decay_profiles sets the per-type half-lives the router ranks retrieval
with; when omitted the system defaults apply. Callers derive these from a
card via :func:~arcana.memory.decay.resolve_decay_profiles.
on_degraded defaults, inside the federation, to the observability audit
log, so a degraded SHARED/GLOBAL tier is recorded without failing the run.
Source code in packages/arcana-core/arcana/memory/assembly.py
dataclass
¶
A shared pool to register on the federation: a name and its backend.
Bases: BaseModel
The memory block of ~/.arcana/config.json.
An absent block yields these defaults, so an install that predates the block
keeps working: memory on, private SQLite, global vector, no shared pools.
global is a Python reserved word, so it maps to global_ via alias.
Fan-out writes and merged reads across a router's memory tiers.
Implements the MemoryAdapter protocol. Composes a MemoryRouter for
every routing and ranking decision; this layer only performs the I/O.
Source code in packages/arcana-core/arcana/memory/federation.py
async
¶
Write an entry to every tier the router selects, by failure policy.
When a target tier's scope differs from the entry's, a scope-rewritten
copy is sent there instead — this is the promotion case, where a
high-importance PRIVATE entry is also stored GLOBAL. The copy keeps the
same id (idempotent) and drops pool_name.
Writes fan out concurrently and are not transactional across tiers, and a
failing tier is handled by its scope: PRIVATE is the durability anchor,
so its failure raises MemoryWriteError; a SHARED or GLOBAL failure
degrades (the tier wrapper has already surfaced a MemoryDegradedEvent)
so the session proceeds. A degraded GLOBAL therefore pauses promotion
without ever failing the private write.
Source code in packages/arcana-core/arcana/memory/federation.py
async
¶
Fan a query across all routed tiers, then merge, dedup, and rank.
Tiers are queried concurrently. A tier that fails is logged and dropped
rather than sinking the whole read. Results are deduplicated by id
(keeping the copy from the most-local tier in routing order) and ranked
by the agent's memory weights, then truncated to query.limit.
Source code in packages/arcana-core/arcana/memory/federation.py
async
¶
Yield the merged, ranked entries best-first, one at a time.
Produces the same sequence as :meth:search but as an async generator,
so a caller can stop iterating the moment it has what it needs (e.g. once
a context budget is full) instead of materialising the whole list.
Best-first order needs the full candidate pool, so the fan-out and rank still happen up front; the streaming is on the consumption side. An empty routing set yields nothing.
Source code in packages/arcana-core/arcana/memory/federation.py
async
¶
Aggregate health across all registered tiers. Never raises.
The federation is usable as long as at least one tier is reachable —
consistent with degraded reads returning partial context. Unhealthy
tiers are named in message so a caller can see what dropped out.
Source code in packages/arcana-core/arcana/memory/federation.py
async
¶
Prune every tier whose backend supports it; aggregate the reports.
Read-only backends (no prune method) are skipped. The resilient tier
wrapper does not forward pruning, so we prune the backend it wraps
directly: pruning is a destructive maintenance op that deliberately
bypasses the read/write resilience path — a tier failure propagates,
because losing track of a destructive operation should surface rather
than be silently swallowed.
Source code in packages/arcana-core/arcana/memory/federation.py
async
¶
Close every tier's backend connection. Safe to call more than once.
Reaches past the router's resilience wrappers to the real backend and
closes it when it supports aclose (read-only tiers may not). This is
the teardown seam a caller uses to release the private SQLite handle and
any vector store when a run ends, leaving no open connections behind.
Source code in packages/arcana-core/arcana/memory/federation.py
MemoryRouter(
*,
private,
global_=None,
pools=None,
weights=None,
decay_profiles=None,
clock=now_utc,
resilience=None,
)
Routes memory writes and reads across private, shared, and global tiers.
A neutral set of weights (all 0.5) is used when none is supplied, so an agent with no card preferences still gets a stable importance-driven order.
Source code in packages/arcana-core/arcana/memory/router.py
Register (or replace) the backend serving a named shared pool.
Return the tier backend(s) an entry must be written to.
PRIVATE→ the private tier, plus the global tier when the entry is eligible for promotion (importance >= 0.9) and a global backend is registered. The caller writes ascope=GLOBALcopy to that target.SHARED→ the backend forentry.pool_name.GLOBAL→ the global tier.
Raises MemoryRoutingError when a required tier is missing: a SHARED
write with no or unknown pool_name, or a GLOBAL write with no global
backend. A PRIVATE entry eligible for promotion but lacking a global
backend is not an error — it simply stays private.
Source code in packages/arcana-core/arcana/memory/router.py
Return the tier backend(s) a query should fan across.
scope is None(federated read) → every registered tier.PRIVATE/GLOBAL→ that single tier.SHARED→ the named pool ifpool_nameis set, else every pool.
Reads degrade rather than raise: a tier that isn't registered (e.g. no
global backend) is silently skipped. An explicitly named pool that does
not exist is a caller error and raises MemoryRoutingError.
Source code in packages/arcana-core/arcana/memory/router.py
Re-rank merged candidates by decayed effective importance and weights.
Layers card preference on top of each adapter's own ordering, using the age-discounted importance (decay is a score, not a delete) rather than the raw stored value:
score = effective_importance(entry, profile, now) × weights.for_type(type)
Entries that have aged out — decayed below their consolidation threshold
(:func:~arcana.memory.decay.should_consolidate) — are dropped from the
result, so a fast-decaying type stops surfacing once stale. Pinned entries
are exempt from decay and always sort first; ties break on score then
recency. Assumes the input is already deduplicated by id (the
federation merges per-tier results before ranking). Truncates to
query.limit.
Source code in packages/arcana-core/arcana/memory/router.py
Every registered tier — private, each shared pool, then global.
The same private → shared → global order a federated read fans across, and the set a store-wide operation (e.g. pruning) iterates.
Source code in packages/arcana-core/arcana/memory/router.py
dataclass
¶
A registered memory backend and the scope it serves.
pool_name is set only for SHARED backends; it names the pool this
adapter stands in for.
Extraction and summarisation¶
Bases: Protocol
Turns an exchange (and a whole session) into memories.
property
¶
The language cue set this extractor detects with — drives consolidation typing.
Deterministic, model-free extraction.
Per turn it always records one EPISODIC entry (what happened), promotes a
stated user preference/fact to SEMANTIC, and — when the user asked a
how-to and the answer contains steps — records a PROCEDURAL entry. Turn
entries are agent-asserted, so confidence is capped; an explicit user
statement is USER_CONFIRMED at higher confidence.
signals selects the language cue set (English by default); pass another
:class:SignalPatterns to extract in a different language.
Source code in packages/arcana-core/arcana/memory/extraction/extractors.py
LLMExtractor(
gateway,
model,
*,
agent_confidence_cap=DEFAULT_AGENT_CONFIDENCE_CAP,
temperature=0.0,
fallback=None,
)
Low-temperature gateway extraction with a heuristic safety net.
A single small completion returns a JSON list of candidate memories, each
validated into a :class:MemoryEntry at agent-capped confidence (a model
never asserts a fact at 1.0). Any gateway error, malformed JSON, or empty
result falls back to :class:HeuristicExtractor for that turn, so extraction
never fails a run.
Source code in packages/arcana-core/arcana/memory/extraction/extractors.py
Select an extractor from config, falling back to heuristic without a model.
ExtractionStrategy.LLM needs both a gateway and a non-empty model; absent
either (the no-provider case), extraction is forced to the deterministic
heuristic.
Source code in packages/arcana-core/arcana/memory/extraction/extractors.py
Collapse whitespace and cap length, appending an ellipsis when truncated.
The default limit is the env-tunable per-entry content cap; pass an
explicit limit to frame model input. The result is never longer than
limit characters.
Source code in packages/arcana-core/arcana/memory/extraction/scoring.py
Reduce a durable user statement to the clause worth storing as SEMANTIC.
A stated fact usually arrives wrapped in conversational framing — "by the way, please remember that my project is called Arcana". Persisting the whole prompt carries that framing into memory and back into every future prompt injection; this keeps the fact itself: "my project is called Arcana".
It picks the sentence bearing the durable preference cue (the same signal
that classified the turn as SEMANTIC) and strips a leading framing wrapper,
then trims. Falls back to the full trimmed text whenever stripping would leave
nothing — so a bare "remember that." is never reduced to an empty memory.
Pure and deterministic; signals selects the language, so a non-English cue
set distils in its own language (or, absent framing cues, simply trims).
Source code in packages/arcana-core/arcana/memory/extraction/scoring.py
Bases: BaseModel
The extraction sub-block of the memory config.
strategy selects the extractor; llm needs a model/gateway or it is
forced back to heuristic by build_extractor. Numeric fields default
to the env-overridable tunables above and carry the same bounds — notably
agent_confidence_cap must stay strictly below 1.0 (the anti-poisoning
invariant), so a config setting it to 1.0 is rejected at load.
dataclass
¶
Compiled surface-cue patterns for a single language.
imperative— emphasis/"remember" language that raises importance.preference— a durable, user-stated fact/preference (→ SEMANTIC).howto— the user is asking for a procedure (paired withsteps).steps— a step/ordered list in a response (→ PROCEDURAL).framing— a leading conversational/imperative wrapper ("remember that", "by the way") stripped when distilling a durable statement down to the clause worth storing. Anchored at the start; defaults to a never-match pattern for languages that define no framing cues.
Register a language's signal patterns under an ISO code (e.g. "pt").
Resilience¶
Wraps one memory backend with a timeout, a circuit breaker, and degraded
reporting. Drops into TierBackend.adapter in place of the raw adapter.
Reads are total — a timeout, open breaker, corruption, or backend error
yields [] (partial context beats none) after emitting a
MemoryDegradedEvent. Writes are partial — the same conditions raise
TierWriteFailed carrying this tier's scope, so the federation can decide
the blast radius (PRIVATE fatal; SHARED/GLOBAL degrade).
Corruption is special: it is a session-long condition, so a
MemoryCorruptError forces the breaker open (quarantine) rather than
counting as one transient failure.
Source code in packages/arcana-core/arcana/memory/resilience.py
property
¶
The wrapped backend. Lets callers reach capabilities the wrapper does not forward (e.g. pruning, a destructive op that deliberately bypasses the read/write resilience path).
async
¶
Fan a query to the backend within budget; never raise.
Source code in packages/arcana-core/arcana/memory/resilience.py
async
¶
Write to the backend within budget; raise TierWriteFailed on failure.
A failed write surfaces by raising (carrying scope + reason) rather than
emitting a degraded event: only the federation knows whether this leg is a
promotion and whether the failure is fatal (PRIVATE) or a degradation
(SHARED / GLOBAL), so it owns the MemoryDegradedEvent.
Source code in packages/arcana-core/arcana/memory/resilience.py
async
¶
Trips after consecutive failures; recovers via a single probe call.
The breaker is tripped by observed failures during real traffic — a passing
health check does not prove queries succeed, and we will not poll on every
call. Once open it fails fast (allow returns False) until
reset_after_seconds passes, after which one HALF_OPEN probe decides
whether to close again or re-open.
The clock is injected so callers (and tests) control time without sleeping; it must be a monotonic source of seconds.
Source code in packages/arcana-core/arcana/memory/resilience.py
A call succeeded: reset failures and close the breaker.
A call failed: count it and open the breaker at the threshold.
A failure while HALF_OPEN re-opens immediately regardless of the count — the probe told us the backend is still unhealthy.
Source code in packages/arcana-core/arcana/memory/resilience.py
Open the breaker immediately, ignoring the failure count.
Used for conditions known to be session-long rather than transient — chiefly store corruption, where retrying within the session is pointless.
Source code in packages/arcana-core/arcana/memory/resilience.py
Bases: StrEnum
Lifecycle of a per-adapter circuit breaker.
Configuration¶
Bases: BaseModel
Resilience config for every tier the federation may register.
shared is keyed by pool name; a pool with no explicit entry falls back to
default_shared. global_ uses the global JSON key (Python reserved
word) via its field alias.
Config for a named shared pool, falling back to default_shared.
classmethod
¶
Load config from disk. A missing file yields all defaults (never raises).
Source code in packages/arcana-core/arcana/memory/config.py
Bases: BaseModel
Timeout + breaker budget for a single memory tier.
The keyword/semantic split is per operation: the wrapper applies
semantic_timeout_ms when a query would touch the embedding path
(retrieval_mode != keyword) and read_timeout_ms otherwise, so a slow
embedder degrades to no vector tier rather than a stalled session.
Background jobs¶
A bounded FIFO queue for background memory work with load-shedding.
submit never blocks: critical jobs are always enqueued, non-critical jobs
are dropped (submit returns False) once the estimated drain time
reaches max_drain_seconds (so a headroom of 0 sheds all non-critical
work). The drain estimate is depth × EWMA(service
time); the EWMA is seeded and refined by a consumer via
record_service_time once one exists.
Source code in packages/arcana-core/arcana/memory/jobs.py
Enqueue a job. Returns False if a non-critical job was load-shed.
Critical work (e.g. a user-confirmed preference) is always accepted; non-critical work is shed when the backlog would take longer to drain than the configured headroom allows.
Source code in packages/arcana-core/arcana/memory/jobs.py
Remove and return the oldest job, or None if the queue is empty.
The single primitive a future consumer loop needs; updates the metrics.
Source code in packages/arcana-core/arcana/memory/jobs.py
Fold an observed job service time into the EWMA used for drain estimates.
dataclass
¶
A unit of deferred memory work.
run is the coroutine factory a future consumer awaits; keeping the work
behind a thunk lets the queue stay agnostic about what the job does.
Bases: StrEnum
What a deferred memory job does — and, by nature, how sheddable it is.
Embedding gateway¶
Picks the embedding adapter for a database, honouring its model pin.
Source code in packages/arcana-core/arcana/memory/embedding_gateway.py
async
¶
Return the adapter to embed with, or None to fall back to FTS5.
Source code in packages/arcana-core/arcana/memory/embedding_gateway.py
Migrations¶
async
¶
Apply every migration newer than the DB's current user_version, in order.
Each migration runs in its own transaction together with the user_version
bump, so a partial failure leaves the database exactly at its prior version.
Idempotent: a database already at head applies nothing. Returns the version
the database is at after running.
migrations is injectable purely so tests can exercise the runner (e.g.
rollback on a deliberately failing statement) without touching real schema.
Source code in packages/arcana-core/arcana/memory/migrations/runner.py
Highest version defined. Zero when no migrations exist.
Errors¶
Bases: Exception
Base class for all memory-backend errors.
Bases: MemoryError
A read/write against the backend failed (I/O, corruption, constraint).
Bases: MemoryStorageError
The backend reported a malformed or unreadable database image.
A subclass of MemoryStorageError so existing except MemoryStorageError
sites still catch it, but distinct so the resilience layer can treat it
specially: corruption is a session-long condition, not a transient failure,
so a tier that raises this is quarantined (breaker forced open) rather than
merely counted as one failure.
Bases: MemoryError
The adapter was used before a connection/schema was established.
Bases: MemoryError
A write targets a tier that is missing or under-specified.
Raised when an entry must reach a backend that was never registered — a
GLOBAL write with no global backend, or a SHARED write whose pool_name
is absent or names an unknown pool.
Bases: MemoryError
A write to an agent's PRIVATE store failed and could not be recovered.
PRIVATE memory is the durability anchor: an agent that cannot persist its
own memory must learn of it rather than silently lose data it believes was
written. The federation raises this when the private write leg fails; SHARED
and GLOBAL write failures degrade instead (surfaced via MemoryDegradedEvent).
Bases: MemoryError
A single tier's write failed inside the resilience wrapper.
Carries the scope of the failing tier so the federation can decide the
blast radius — PRIVATE is fatal (re-raised as MemoryWriteError), SHARED
and GLOBAL degrade. cause is the original backend exception (or a
breaker-open sentinel); reason is the degradation category
(timeout / breaker_open / backend_error / corruption) the
federation puts on the MemoryDegradedEvent it emits for a degraded tier.
The wrapper raises rather than emits, so the federation can label the event
write vs promote and suppress it entirely for the fatal PRIVATE case.