![\"Four-Layer](\"/diagrams/four-layer-architecture.svg\")
\nThe MCP-native agent stack: four layers with the LLM as a probabilistic reasoning engine, bounded by deterministic architectural layers.The diagram on the front page of this site is not an illustration. Every box in it is a\npublished Rust crate you can cargo add today, read the documentation for, and use\nwithout taking the rest of the system. This article explains what sits behind each box,\nand why the decomposition is the interesting part.
\n
Most agent frameworks ship as a single dependency. You take the orchestrator, the memory\nlayer, the retrieval stack and the tool protocol together, or you take none of it. That\nis convenient at first and expensive later: when one layer misbehaves you cannot isolate\nit, and when you want to replace one you replace all of them.
\nThe pattern this site argues for is the opposite. Draw the boundaries first, make each\none independently useful, and let the seams stay visible. Terraphim is built that way —\n52 crates across six layers, each with a single responsibility — which makes it a\nreasonable worked example rather than a product pitch.
\nThe test of a boundary is whether you can use one side without the other. Here you can:\nterraphim_automata is a text-matching engine that happens to be used by a knowledge\ngraph; terraphim_persistence is a storage abstraction that does not know what it is\nstoring.
Terraphim organises its crates into a dependency graph with six levels. Lower layers do\nnot know the higher ones exist.
\n| Layer | Concern | Representative crates |
|---|---|---|
| 6 | User interfaces | terraphim_agent, terraphim-cli, terraphim_server |
| 5 | Orchestration | terraphim_orchestrator, terraphim_kg_orchestration, terraphim_symphony |
| 4 | Agent system | terraphim_spawner, terraphim_router, terraphim_agent_supervisor |
| 3 | Service layer | terraphim_service, terraphim_middleware, haystack_* |
| 2 | Core engine | terraphim_automata, terraphim_rolegraph, terraphim_persistence |
| 1 | Types and configuration | terraphim_types, terraphim_config, terraphim_settings |
That ordering matters more than the crate count. An agent system that cannot say which\nlayer a failure came from is a system you cannot debug in production.
\nterraphim_types is the shared vocabulary — the structures every other crate agrees on.\nterraphim_config holds role definitions, haystacks and LLM routing;\nterraphim_settings handles runtime preferences.
Putting roles in configuration rather than code is what makes the context boundary\nstructural. A role has access to certain haystacks; the knowledge graph is built from\nthose haystacks; therefore the agent cannot reach information outside its role. That is\nnot a permission check at runtime that could be bypassed — it is an absence. The data\nwas never loaded.
\nThis is the layer the front-page diagram mostly depicts.
\nterraphim_automata is the matching engine:\nAho-Corasick finite state automata that match thousands of patterns simultaneously in\nO(n) time, where n is the length of the text. Not \"usually fast\" — bounded, and the same\nbound every time. It compiles to WebAssembly, which is why the same matcher runs in a\nbrowser as on a server.
terraphim_rolegraph is the knowledge graph.\nIt is deliberately not a general-purpose graph database. It maps search roles to\ndomain-specific graph views, and it exists to do one thing quickly: turn text into\nconcepts, deterministically. Terraphim reports knowledge-graph inference in the 5-10\nnanosecond range and a 15 MB memory footprint, with no GPU — figures worth checking\nagainst your own workload, but the shape of the claim follows from the design rather\nthan from optimisation tricks.
terraphim_persistence provides the\nPersistable trait and DeviceStorage backends across memory, SQLite and redb. Storage is\nan interface here, not an assumption, which is what lets the same engine run on a laptop\nand at the edge.
terraphim_middleware searches haystacks —\npluggable data-source backends. This is the boundary the front-page diagram labels\nDocument Input. A haystack might be a local folder, a repository, or a connector to\nsomething else; the layers above do not change when you add one.
terraphim_service handles requests and responses\nfor the core, which keeps request handling out of the engine itself.
Above the service layer sit the agent system (terraphim_spawner, terraphim_router,\nterraphim_agent_supervisor) and orchestration (terraphim_orchestrator,\nterraphim_kg_orchestration, terraphim_symphony).
The distinction is worth holding onto. The agent system is concerned with individual\nagents — spawning them, routing to them, supervising them. Orchestration is concerned\nwith what happens between them. Collapsing those two into one \"agent framework\" is the\nusual mistake, and it is why so many systems cannot tell a stuck agent from a stuck\nworkflow.
\nThe MCP-Native Agent Architecture goes\ninto the protocol surface these layers expose, and why treating MCP as an architectural\nprimitive rather than a transport changes what the system can guarantee.
\nterraphim_agent is the interactive REPL with session\nsearch and learning capture. terraphim-cli is the\nautomation-shaped counterpart, with JSON output for scripting.\nterraphim_server provides the REST API.
Three interfaces over one engine, none of them privileged. That is the payoff of the\nlayering: the CLI is not a thin wrapper around the server, and the server is not a\nspecial case. They are peers over the same core.
\nYou do not need to adopt Terraphim to use the pattern. The transferable parts are:
\nEvery crate named above is published and documented:
\nterraphim_automata for matching,\nterraphim_rolegraph for the knowledge graph,\nor terraphim_persistence for storage.Installation is a single command, or cargo install terraphim_agent if you would prefer\nto start from the REPL.
A reference architecture for production AI systems, drawn from five years building Terraphim — a privacy-first AI assistant with ~50 Rust crates, MCP-native interfaces, and role-based multi-agent orchestration.
\nIn the spring of 2024, I watched a well-funded AI startup burn through $180,000 in cloud credits in six weeks. Their crime? They let an LLM-powered agent manage their Kubernetes cluster. The agent had root access, a vague prompt (\"optimize resource utilization\"), and no guardrails. It deleted a production namespace, scaled a stateful set to zero, and triggered a cascading failure that took down their primary revenue pipeline for four hours.
\nThe post-mortem was illuminating. The LLM wasn't \"malicious.\" It was doing exactly what LLMs do: generating plausible-sounding text based on pattern matching. The prompt \"optimize resource utilization\" is semantically close to \"remove unused resources.\" The agent found a namespace with low CPU utilization and removed it. Logical, if you squint. Catastrophic, if you're the on-call engineer.
\nThis is the production agent crisis in miniature: we've given probabilistic systems deterministic powers without deterministic boundaries.
\nThe current generation of agent frameworks — LangChain, CrewAI, AutoGPT, and their descendants — share a common architectural flaw: they treat the LLM as both the decision-maker AND the execution engine. The LLM decides what to do, how to do it, and when to stop. This is fine for demos. It's a liability for production.
\nConsider what production systems actually require:
\nNone of these are properties of LLMs. LLMs are probabilistic text generators. They hallucinate. They drift. They respond to temperature. They have no concept of \"side effects\" or \"resource limits\" or \"this action is irreversible.\"
\nThe solution is architectural, not algorithmic. We need a boundary layer between the probabilistic decision-maker (the LLM) and the deterministic execution environment (the system). That boundary layer is the Model Context Protocol (MCP), and treating it as an architectural primitive rather than a communication protocol changes everything.
\nMCP was introduced by Anthropic in late 2024 as a standardized protocol for connecting AI assistants to external data sources and tools. At its simplest, it defines how an AI client discovers capabilities, requests context, and invokes tools from a server.
\nBut MCP is more than a wire format. It is a contract layer that enforces capability boundaries. And when you treat it as an architectural primitive — a fundamental building block of your system, not just a communication detail — you get something powerful: the ability to build deterministic agent systems on top of probabilistic reasoning engines.
\nAn MCP server exposes three things:
\nThe key insight: the server defines WHAT is possible. The LLM decides WHICH possibility to pursue. The protocol enforces the boundary between them.
\nTerraphim's MCP server (terraphim_mcp_server) implements this boundary in ~200 lines of Rust. It exposes three tools: search (read-only, queries the knowledge graph), build_autocomplete_index (idempotent, rebuilds the Aho-Corasick automata), and update_config (destructive, modifies runtime configuration). Each tool has a schema. Each tool has a known effect. The LLM cannot invoke a tool it hasn't discovered, and it cannot pass arguments that don't match the schema.
This is the difference between \"I hope the LLM doesn't delete something\" and \"the system is physically incapable of deleting something without authorization.\"
\n\nThe MCP-native agent stack: four layers with the LLM as a probabilistic reasoning engine, bounded by deterministic architectural layers.
Based on production experience building Terraphim — a privacy-first AI assistant with ~50 Rust crates, MCP server integration, and multi-agent orchestration — I've distilled the following reference architecture. It separates concerns that current frameworks conflate and introduces verification at every boundary.
\nWhat the agent IS.
\nIn Terraphim, everything starts with a Role. A Role is not a prompt. It is a structured configuration that defines an agent's identity, knowledge sources, capabilities, and constraints. Think of it as a compile-time contract for an agent's behavior.
// From terraphim_config — the actual Role struct\n#[derive(Debug, Serialize, Deserialize, Clone)]\npub struct Role {\n pub name: RoleName,\n pub relevance_function: RelevanceFunction, // BM25, TitleScorer, etc.\n pub haystacks: Vec<Haystack>, // Data sources\n pub kg: Option<KnowledgeGraph>, // Linked knowledge graph\n pub llm_enabled: bool,\n pub llm_model: Option<String>, // e.g., "gemma3:270m"\n pub llm_router_enabled: bool, // 6-phase intelligent routing\n pub extra: AHashMap<String, Value>, // Extensible config\n}Key properties:
\nextra hashmap allows domain-specific configuration without schema changesThe Role is the \"type system\" of your agent. Just as Rust's borrow checker prevents data races at compile time, the Role configuration prevents capability leakage at configuration time.
\nWhat the agent KNOWS.
\nTerraphim's knowledge graph (terraphim_rolegraph) is not a general-purpose graph database. It is a specialized structure optimized for one thing: fast, deterministic text-to-concept matching using Aho-Corasick automata.
// From terraphim_rolegraph — the actual RoleGraph\npub struct RoleGraph {\n pub role: RoleName,\n nodes: AHashMap<u64, Node>,\n edges: AHashMap<u64, Edge>,\n documents: AHashMap<String, IndexedDocument>,\n pub thesaurus: Thesaurus,\n pub ac: AhoCorasick, // Compiled automata\n pub ac_reverse_nterm: AHashMap<u64, NormalizedTermValue>,\n}\n\nimpl RoleGraph {\n pub fn find_matching_node_ids(&self, text: &str) -> Vec<u64> {\n self.ac.find_iter(text)\n .map(|mat| self.aho_corasick_values[mat.pattern()])\n .collect()\n }\n}Key properties:
\nterraphim_automata compiles to WebAssembly for browser-based autocomplete.The Context Boundary emerges naturally from the Role configuration: if a Role only has access to certain haystacks, and the RoleGraph is built from those haystacks, the agent cannot access information outside its Role. This is not a runtime permission check. It is a structural impossibility.
\nWhat the agent DOES with others.
\nTerraphim's multi-agent system (terraphim_multi_agent) treats each Role as an autonomous agent. Agents discover each other through capability registration, communicate through structured messages, and coordinate through workflow patterns.
// From terraphim_multi_agent — the actual AgentRegistry\npub struct AgentRegistry {\n agents: Arc<RwLock<HashMap<AgentId, Arc<TerraphimAgent>>>>,\n capabilities: Arc<RwLock<HashMap<String, Vec<AgentId>>>>,\n role_agents: Arc<RwLock<HashMap<String, AgentId>>>,\n agent_load: Arc<RwLock<HashMap<AgentId, LoadMetrics>>>,\n}\n\nimpl AgentRegistry {\n pub async fn find_agents_by_capability(&self, capability: &str) -> Vec<AgentId> {\n let capabilities = self.capabilities.read().await;\n capabilities.get(capability).cloned().unwrap_or_default()\n }\n}Key properties:
\nThe Execution Harness is not a Wasm sandbox (though we have WASM support in terraphim_automata for browser contexts). It is the agent's own TerraphimAgent struct, which enforces resource limits through Rust's type system: token budgets are usize, timeouts are Duration, and context windows are bounded by configuration.
How the agent COMMUNICATES.
\nThe terraphim_mcp_server crate exposes Terraphim's capabilities through the Model Context Protocol. It acts as both server (exposing Terraphim tools to external clients) and client (discovering external tools for Terraphim agents).
// From terraphim_mcp_server — the actual McpService\n#[derive(Clone)]\npub struct McpService {\n config_state: Arc<ConfigState>,\n resource_mapper: Arc<TerraphimResourceMapper>,\n autocomplete_index: Arc<tokio::sync::RwLock<Option<AutocompleteIndex>>>,\n}\n\nimpl McpService {\n pub async fn search(\n &self,\n query: String,\n role: Option<String>,\n limit: Option<i32>,\n ) -> Result<CallToolResult, ErrorData> {\n // 1. Resolve role (tenant boundary)\n let role_name = if let Some(role_str) = role {\n RoleName::from(role_str)\n } else {\n self.config_state.get_selected_role().await\n };\n\n // 2. Query knowledge graph within role boundary\n let search_query = SearchQuery {\n search_term: NormalizedTermValue::from(query),\n role: Some(role_name),\n limit: limit.map(|l| l as usize),\n ..Default::default()\n };\n\n // 3. Return structured results\n match service.search(&search_query).await {\n Ok(documents) => { /* ... */ }\n Err(e) => { /* ... */ }\n }\n }\n}Key properties:
\nrole parameter in every tool call enforces tenant isolation at the protocol level.TerraphimMcpError maps to MCP ErrorData with structured error codes.tokio for concurrent request handling.The reference architecture is implemented in Rust. This is not aesthetic preference. It is an engineering requirement.
\nZero-cost abstractions: The Role configuration, knowledge graph, and agent registry introduce no runtime overhead. The type system enforces safety at compile time.
\nDeterministic resource usage: No garbage collection pauses. Memory is explicitly managed. The terraphim_automata Aho-Corasick matcher runs in bounded time and space.
Fearless concurrency: The AgentRegistry is accessed by multiple agents simultaneously. Rust's Arc<RwLock<_>> prevents data races without runtime overhead.
WASM portability: terraphim_types and terraphim_automata compile to WebAssembly, enabling browser-based autocomplete with TypeScript bindings.
![\"Traditional](\"/diagrams/traditional-vs-mcp.svg\")
\nTraditional agents conflate decision and execution. MCP-native agents enforce a hard boundary: the LLM decides, the architecture executes.
In Terraphim, multiple users share the same agent infrastructure but have completely isolated contexts:
\n// Each tenant gets their own Role configuration\nlet tenant_role = Role::new("tenant_acme")\n .with_haystacks(vec![acme_documents, acme_wiki])\n .with_llm_model("gemma3:270m")\n .with_relevance_function(RelevanceFunction::BM25);\n\n// The RoleGraph is built ONLY from tenant's haystacks\nlet role_graph = RoleGraph::new("tenant_acme".into(), tenant_thesaurus).await?;\n\n// MCP search automatically scopes to tenant's role\nlet results = mcp_service.search(query, Some("tenant_acme".to_string()), None).await?;Key insight: The boundary is per-tenant, but the automata engine and MCP server are shared. This gives you multi-tenant isolation without multi-tenant cost.
\nWhen a bug occurs in production, you need to reproduce it exactly. With probabilistic agents, this is nearly impossible. With MCP-native architecture, it's trivial:
\n![\"Deterministic](\"/diagrams/deterministic-replay.svg\")
\nRecording LLM responses (not regenerating them) makes replay deterministic. The same inputs produce the same execution trace.
// The RoleGraph + automata are deterministic\n// Same query + same thesaurus → same matches, every time\nlet matches = role_graph.find_matching_node_ids("rust programming");\n// Always returns the same node IDs for the same input\n\n// For LLM variability: record responses, don't regenerate\nlet recorded_response = load_recording(task_id)?;\nlet plan = agent.generate_plan_with_response(context, recorded_response)?;Key insight: The knowledge graph is deterministic. The LLM is not. Separate the two, and replay becomes deterministic where it matters.
\n![\"Approval](\"/diagrams/approval-flow.svg\")
\nDestructive operations require explicit approval. The system logs every request but does not execute without authorization.
// terraphim_mcp_server's update_config tool is inherently bounded\npub async fn update_config_tool(&self, config_str: String) -> Result<CallToolResult, ErrorData> {\n match serde_json::from_str::<Config>(&config_str) {\n Ok(new_config) => {\n // Config validation happens BEFORE mutation\n self.validate_config(&new_config)?;\n self.update_config(new_config).await?;\n Ok(CallToolResult::success(vec![Content::text(\n "Configuration updated".to_string()\n )]))\n }\n Err(e) => {\n Ok(CallToolResult::error(vec![Content::text(\n format!("Invalid configuration: {}", e)\n )]))\n }\n }\n}Key insight: The MCP schema IS the approval gate. The LLM cannot construct a valid Config JSON without knowing the schema, and invalid configs are rejected before mutation.
![\"Terraphim](\"/diagrams/terraphim-multi-agent.svg\")
\nTerraphim instances share an MCP protocol layer while maintaining isolated Role configurations and Knowledge Graphs. Each agent discovers tools through MCP without data leakage.
Terraphim implements this architecture across ~50 Rust crates. Here's how the layers map to real code:
\n| Layer | Crate | Responsibility |
|---|---|---|
| Role & Config | terraphim_config | Role definitions, haystacks, LLM routing |
| Knowledge Graph | terraphim_rolegraph | Aho-Corasick automata, concept matching |
| Fast Matching | terraphim_automata | FST autocomplete, link generation, WASM |
| Multi-Agent | terraphim_multi_agent | Agent registry, capability discovery, workflows |
| MCP Server | terraphim_mcp_server | Protocol surface, schema enforcement |
| Capabilities | terraphim_agent_registry | Capability matching, score-based discovery |
| Persistence | terraphim_persistence | DeviceStorage, memory/file backends |
The MCP integration is bidirectional. Terraphim acts as an MCP server (exposing search and config tools to external clients) and can discover external MCP servers for additional capabilities. This enables multi-agent orchestration where each agent has its own Role and Knowledge Graph, but they can discover and invoke each other's tools through MCP.
\nLet's be precise about what this architecture guarantees and what it doesn't.
\nWhat IS guaranteed:
\nWhat is NOT guaranteed:
\nThe key insight: We don't need the LLM to be deterministic. We need the KNOWLEDGE GRAPH and the PROTOCOL BOUNDARY to be deterministic. The LLM provides variable intelligence within fixed architectural bounds. The bounds are what matter for production.
\nNot every AI system needs four layers of deterministic boundaries. Here's when you DON'T need MCP-native architecture:
\nAdd MCP-native boundaries when:
\nBuilding MCP-native agents?
\nThe current generation of agent frameworks treats LLMs as the center of the universe. The LLM decides everything, and the framework provides convenient wrappers around API calls.
\nThis is backwards.
\nThe protocol is the architecture. The LLM is just one implementation detail.
\nA production agent system needs:
\nMCP provides the protocol surface. The reference architecture provides the safety properties. Rust provides the implementation guarantees.
\nThe LLM? It's just the reasoning engine. Powerful, probabilistic, and properly contained.
\nThe architecture described in this article is implemented in Terraphim, an open-source privacy-first AI assistant:
\nterraphim_mcp_server crateContributions welcome. Issues tracked in the Terraphim Gitea.
\nAlexander Mikhalev is CTO & Head of AI at Zestic AI, where he architects AI-native platforms with deterministic safety guarantees. He is the creator of Terraphim, an open-source privacy-first AI assistant built in Rust. His project \"The Pattern\" won the $10,000 Platinum Prize at the Redis \"Build on Redis\" Hackathon 2021 for ML-powered knowledge discovery. He previously led AI/ML architecture at Nationwide Building Society, where he co-authored the organization's first technology patent — a blockchain-inspired distributed system for resilient consensus. He speaks on AI architecture, Rust, and knowledge graphs.
\n" },{"title": "Harness Engineering for Self-Improving Agents", "url": "https://reference-architecture.ai/posts/harness-engineering-agents/", "body": "Many AI agent systems plateau after deployment. They execute tasks, but the surrounding system may not measure whether they are getting better, repeating old work, or normalizing stalled execution. This article proposes a reference pattern for agent harnesses: supervised feedback loops that make agent behavior observable, measurable, and easier to improve.
\nMost agent architectures follow a simple pattern:
\nUser Request -> Agent Reasoning -> Tool Execution -> ResponseThis works for single-turn tasks. But over hundreds or thousands of runs, the agent can make the same mistakes, retry the same failing strategies, and generate the same redundant outputs. It may have no mechanism to:
\nThese are plausible failure modes in long-running agent systems: repeated status updates, hand-waved health scores, accumulated \"lessons learned\" that never change system behavior, and stale priorities that remain nominally critical while no one acts on them.
\nThis is not necessarily a bug in the agent. It is often a missing architectural layer.
\nA harness is a feedback loop that monitors an agent's output, detects patterns, and triggers bounded corrective action. The proposed pattern uses four harness types, each addressing a specific failure mode:
\n| Harness | Failure Mode | Trigger | Action |
|---|---|---|---|
| Novelty Filter | Redundant documentation accumulation | Semantic similarity > threshold | Suppress redundant output |
| Calculated Health | Narrated metrics drift from reality | Daily evaluation window | Compute objective health score |
| Break-Glass | System paralysis without action | Threshold breach | Notify, escalate, or reduce authorized scope |
| Lesson Window | Lessons identified but never applied | Age > window or count > limit | Archive stale active lessons |
+-------------------------------------------------------------+\n| AGENT CORE |\n| +-----------+ +-----------+ +----------------+ |\n| | Reasoning | ---> | Execution | ---> | Output | |\n| +-----------+ +-----------+ +----------------+ |\n+-------------------------------------------------------------+\n |\n v\n+-------------------------------------------------------------+\n| HARNESS LAYER |\n| +----------------+ +----------------+ +-------------+ |\n| | Novelty Filter| | Calc. Health | | Break-Glass | |\n| | (dedup) | | (metrics) | | escalation | |\n| +----------------+ +----------------+ +-------------+ |\n| +-------------------------------------------------------+ |\n| | Lesson Window (time-bounded) | |\n| +-------------------------------------------------------+ |\n+-------------------------------------------------------------+\n |\n v\n+-------------------------------------------------------------+\n| FEEDBACK LOOP |\n| Metrics -> Review -> Approved Changes -> Policies |\n+-------------------------------------------------------------+Problem: The agent may produce repeated documentation or status text that restates known issues without adding decision value.
\nIllustrative implementation: This example is self-contained enough to show the state-management rule. In production, persist history outside the process and choose the embedding model, threshold, and retention period through evaluation.
from dataclasses import dataclass\nfrom datetime import datetime, timedelta, timezone\n\nfrom sentence_transformers import SentenceTransformer\nimport numpy as np\n\n@dataclass(frozen=True)\nclass HistoryEntry:\n content: str\n embedding: np.ndarray\n created_at: datetime\n\nclass NoveltyFilter:\n def __init__(self, threshold=0.82, window_days=7):\n self.model = SentenceTransformer("all-MiniLM-L6-v2")\n self.threshold = threshold\n self.window_days = window_days\n self.history: list[HistoryEntry] = []\n\n def is_novel(self, content: str) -> bool:\n """Return True only if content is sufficiently novel."""\n now = datetime.now(timezone.utc)\n self._evict_expired(now)\n new_embedding = self.model.encode(content)\n\n for entry in self.history:\n similarity = np.dot(new_embedding, entry.embedding) / (\n np.linalg.norm(new_embedding) * np.linalg.norm(entry.embedding)\n )\n if similarity > self.threshold:\n return False\n\n self.history.append(HistoryEntry(content, new_embedding, now))\n return True\n\n def _evict_expired(self, now: datetime) -> None:\n cutoff = now - timedelta(days=self.window_days)\n self.history = [\n entry for entry in self.history\n if entry.created_at >= cutoff\n ]Hypothesis: A correctly tuned novelty filter should reduce redundant persisted text while preserving new decisions, new evidence, and changed conclusions. A proposed target is a 30-50% reduction in persisted daily status text with manual review showing no loss of material decisions.
\nProblem: Health metrics are often narrated rather than calculated. A stable phrase such as \"roughly healthy\" can mask variation in the underlying jobs.
\nIllustrative implementation: The health calculator should receive concrete check functions. This avoids hidden dependencies on undefined methods and makes the metric auditable.
\nfrom dataclasses import dataclass\nfrom datetime import datetime, timedelta, timezone\nfrom pathlib import Path\nfrom typing import Callable\n\n@dataclass\nclass JobMetrics:\n name: str\n weight: float\n check_func: Callable[[int], float]\n\nclass HealthCalculator:\n def __init__(self, jobs: list[JobMetrics]):\n total_weight = sum(job.weight for job in jobs)\n if round(total_weight, 6) != 1.0:\n raise ValueError("job weights must sum to 1.0")\n self.jobs = jobs\n\n def calculate(self, days: int = 7) -> float:\n """Calculate health as weighted average of job success rates."""\n total = 0.0\n for job in self.jobs:\n success_rate = job.check_func(days)\n total += success_rate * job.weight\n return round(total * 100, 1)\n\ndef recent_output_check(directory: Path, min_bytes: int = 1024) -> Callable[[int], float]:\n """Return a check where success means one meaningful output per day."""\n def check(days: int) -> float:\n cutoff = datetime.now(timezone.utc) - timedelta(days=days)\n outputs = [\n path for path in directory.glob("*.md")\n if datetime.fromtimestamp(path.stat().st_mtime, timezone.utc) >= cutoff\n and path.stat().st_size >= min_bytes\n ]\n return min(len(outputs) / max(days, 1), 1.0)\n return check\n\ncalculator = HealthCalculator([\n JobMetrics("extraction", 0.40, recent_output_check(Path("runs/extraction"))),\n JobMetrics("briefing", 0.30, recent_output_check(Path("runs/briefing"))),\n JobMetrics("weekly_review", 0.30, recent_output_check(Path("runs/review"))),\n])Measurement definition: Health is the weighted average of job-specific success rates over a fixed evaluation window. Each job must define its own observable success condition before the run starts.
\nProblem: Some systems need a fail-safe escalation path when important work stalls or health drops below an agreed threshold. Break-glass rules should not silently grant broad autonomy to the agent.
\nCanonical configuration:
\n[break_glass]\nenabled = true\nmode = "supervised"\n\n[[break_glass.rules]]\nname = "commit_gap"\ncondition = "days_since_commit > 14"\nseverity = "critical"\naction = "notify_and_require_ack"\nmessage = "Critical commit gap: human acknowledgement required."\n\n[[break_glass.rules]]\nname = "health_critical"\ncondition = "calculated_health < 15"\nseverity = "critical"\naction = "reduce_authorized_scope"\nallowed_jobs = ["extraction", "health_check"]\nmessage = "Health critical: non-essential jobs are paused within pre-authorized limits."\n\n[[break_glass.rules]]\nname = "lesson_accumulation"\ncondition = "unapplied_lessons > 50 AND oldest_unapplied_days > 14"\nseverity = "warning"\naction = "notify_owner"Rule semantics:
\nnotify_and_require_ack:\n Send an escalation and block risky automatic changes.\n\nreduce_authorized_scope:\n Automatically disable only jobs already marked non-essential.\n\nnotify_owner:\n Create a visible warning with no automatic operational change.Abbreviated trigger evaluation: This sketch assumes Rule.evaluate is a safe expression evaluator over a constrained state object, not arbitrary code execution.
class BreakGlass:\n def evaluate(self, system_state: dict) -> list[Trigger]:\n triggers = []\n for rule in self.rules:\n if rule.evaluate(system_state):\n triggers.append(Trigger(\n rule=rule.name,\n severity=rule.severity,\n action=rule.action,\n message=rule.message.format(**system_state),\n ))\n return triggersFail-safe principle: Notification and escalation remain available through approved channels and rate limits. Automatic action is limited to pre-authorized scope reduction, such as pausing non-essential jobs. Anything outside that predefined reduction policy—including changing data, deleting history, shipping code, or broadening production behavior—requires human authorization.
\nProblem: \"Lessons learned\" can become a growing active queue that never changes system behavior.
\nIllustrative implementation: The window archives stale active lessons. It does not prove that lessons have been applied, and it does not force application.
\nfrom datetime import datetime, timedelta, timezone\n\nclass LessonWindow:\n def __init__(self, window_days: int = 7, max_unapplied: int = 30):\n self.window = timedelta(days=window_days)\n self.max_unapplied = max_unapplied\n\n def process(self, lessons: list[Lesson]) -> tuple[list[Lesson], list[Lesson]]:\n """Return (active_lessons, archived_lessons)."""\n active = []\n archived = []\n now = datetime.now(timezone.utc)\n candidates = [\n lesson for lesson in lessons\n if lesson.status not in {"applied", "archived"}\n ]\n overflow = max(len(candidates) - self.max_unapplied, 0)\n\n for lesson in lessons:\n if lesson.status == "archived":\n archived.append(lesson)\n continue\n if lesson.status == "applied":\n active.append(lesson)\n continue\n\n age = now - lesson.created_at\n\n if age > self.window:\n lesson.archive_once(reason=f"expired_after_{self.window.days}_days")\n archived.append(lesson)\n elif overflow > 0:\n lesson.archive_once(reason="overflow")\n archived.append(lesson)\n overflow -= 1\n else:\n active.append(lesson)\n\n return active, archivedRule: A lesson is only \"learned\" if system state changes because of it. If not applied within the active window, it can be archived, not forgotten, so that it no longer clutters the active queue.
\nThe harnesses integrate at the output boundary of the agent core, before content is persisted to memory:
\nAgent Output\n |\n v\n+-----------------+\n| Novelty Filter | --> Suppress? Yes: discard and log filter event\n+-----------------+ No: continue\n |\n v\n+-----------------+\n| Calculate Health| --> Update metrics store\n+-----------------+\n |\n v\n+-----------------+\n| Break-Glass | --> Notify or apply authorized reduction\n+-----------------+\n |\n v\n+-----------------+\n| Lesson Window | --> Archive expired, queue active\n+-----------------+\n |\n v\nPersist to MemoryThis placement is important: harnesses operate on output, not input. Filtering input could prevent useful ideas from reaching the agent. Filtering output ensures only valuable, non-redundant content is preserved.
\n# harness.toml\n[novelty_filter]\nenabled = true\nthreshold = 0.82\nwindow_days = 7\nmodel = "all-MiniLM-L6-v2"\n\n[health]\nenabled = true\ncalculation_window_days = 7\njobs = [\n { name = "extraction", weight = 0.40, check = "recent_output:runs/extraction" },\n { name = "briefing", weight = 0.30, check = "recent_output:runs/briefing" },\n { name = "weekly_review", weight = 0.30, check = "recent_output:runs/review" },\n]\n\n[break_glass]\nenabled = true\nmode = "supervised"\n\n[[break_glass.rules]]\nname = "commit_gap"\ncondition = "days_since_commit > 14"\nseverity = "critical"\naction = "notify_and_require_ack"\n\n[[break_glass.rules]]\nname = "health_critical"\ncondition = "calculated_health < 15"\nseverity = "critical"\naction = "reduce_authorized_scope"\nallowed_jobs = ["extraction", "health_check"]\n\n[[break_glass.rules]]\nname = "lesson_accumulation"\ncondition = "unapplied_lessons > 50 AND oldest_unapplied_days > 14"\nseverity = "warning"\naction = "notify_owner"\n\n[lesson_window]\nenabled = true\nwindow_days = 7\nmax_unapplied = 30The following is an illustrative contract, not a schema from a real product. The goal is to show the integration boundary a scheduler, orchestrator, or agent runtime would need to provide.
\n{\n "event": "agent.output.proposed",\n "timestamp": "2026-09-06T23:00:00Z",\n "run_id": "run_123",\n "agent_id": "research_agent",\n "content": "Proposed output content",\n "metrics": {\n "days_since_commit": 15,\n "calculated_health": 42.5,\n "unapplied_lessons": 12\n }\n}As of 2026-09-07, a 14-day evaluation starting on 2026-09-06 is not complete. A coherent 14-day inclusive window would run from 2026-09-06 through 2026-09-19. The table below is a prospective evaluation plan, not reported results.
\n| Metric | Baseline Definition | Treatment Definition | Target / Hypothesis |
|---|---|---|---|
| Daily persisted text | Bytes of agent-generated status or lesson text per day | Same measure after novelty filtering | 30-50% reduction without losing material decisions |
| Health score stability | Seven-day weighted health score from fixed checks | Same measure after calculated health is introduced | Changes should track job outcomes rather than remain narratively fixed |
| Break-glass behavior | Number of threshold breaches that produce no visible escalation | Number of threshold breaches with notification, acknowledgement, or authorized scope reduction | Critical breaches should create reviewable events |
| Lesson queue age | Age of oldest active unapplied lesson | Same measure after lesson window archival | Active queue should remain bounded |
Sample size and limitations: A single 14-day run is enough to detect obvious integration failures and generate hypotheses, but it is not enough to claim general performance. Workload mix, agent prompts, human review behavior, and calendar effects can dominate the result. Treat the outcome as local evidence for this system, not proof that the pattern works everywhere.
\nRelated-work caveat: This pattern overlaps with established ideas in observability, control loops, MLOps monitoring, guardrails, and human-in-the-loop operations. The novelty here, if any, is the packaging of those ideas around agent output boundaries. This article does not claim a new algorithm or publish comparative research.
\nIf the novelty threshold is too aggressive, genuinely novel but structurally similar content may be suppressed. Mitigation: Start with a conservative threshold, then adjust based on measured filter hit rate and manual false-positive review. The target should be meaningful reduction, not maximum suppression.
\nIf thresholds are too tight, break-glass triggers too frequently and is ignored. Mitigation: Reserve critical alerts for conditions that require review. Warning-level thresholds should log or notify without interrupting work.
\nArchiving lessons after a fixed window may hide valuable but long-term insights. Mitigation: Archived lessons remain searchable. The archive is cold storage: retrievable, but not cluttering the active queue.
\nThe harness architecture addresses a practical gap in agent design: agents need feedback loops about their own behavior, not just about the tasks they execute. Without these loops, systems can accumulate redundant memory, drifted metrics, and normalized paralysis.
\nThe four harnesses (novelty filter, calculated health, break-glass, lesson window) form a proposed minimal layer for supervised agent improvement. They require a small, explicit contract from the agent core—output events, job metrics, and authorization state—plus careful placement at the output boundary and human authorization for risky actions.
\nKey Takeaway: An agent that cannot detect its own stagnation will stagnate. Harnesses are the architectural layer that makes stagnation visible, reviewable, and fixable.
\nThis is a proposed reference pattern. The methodology above is intended to make future evaluation reproducible without relying on private supporting material.
\nLicense: CC BY-SA 4.0
\n" },{"title": "Turning Open Source project into Product with Redis Enterprise", "url": "https://reference-architecture.ai/posts/github-oauth2/", "body": "Last year, my reference project, \"The Pattern\", was the hackathon winner 2021 and got a bit of publicity and, in total, seven forks. But as with many open source projects, it is now stale. Time to revive \"The Pattern\" with new features and GitHub sponsors or Patreon patrons to help and inspire developers and creatives. In return, it's common to provide sponsor-only features and articles. Nevertheless, how can we do it with a large Redis-based machine learning pipeline?
\nThis article will introduce a simple first step:\nfor GitHub sponsors, we start with offering persistent storage of preferences: I have a simple flask POST API which adds nodes into the user's preference storage - a simple Redis set per user. And it will be a foundation to build other sponsor-only features.\nFor now, let's cover the basics:
\n\nflowchart LR\n id1(User) --> flask_login(Flask Login API)--> github(GitHub OAuth2)\n github-->flask_callback(Flask API callback)-->GitHubGraphQL(GitHub GraphQL)\n
There are a number of API's that GitHub offers to help developers, but the GitHub Authentication API is one of the most popular. This API allows you to log in to GitHub using your username and password, or OAuth token.
\nA login button with a standard OIDC/OAuth2 dance is one of the most common ways for a user to authenticate to an API.\nBelow is code taken from this gist and is very common for OAuth2 flows:
\nimport os \nclient_id = os.getenv('GITHUB_CLIENT_ID')\nclient_secret = os.getenv('GITHUB_SECRET')\n@app.route('/', methods=['GET', 'POST'])\ndef index():\n url = 'https://github.com/login/oauth/authorize'\n params = {\n 'client_id': client_id,\n\n 'scope': 'read:user,read:email',\n 'state': str(uuid4().hex),\n 'allow_signup': 'true'\n }\n url = furl(url).set(params)\n return redirect(str(url), 302)where GITHUB_CLIENT_ID and GITHUB_SECRET are client/secret GitHub Oauth2 apps. Register for following process on GitHub
\norg_name="applied-knowledge-systems"\n@app.route('/oauth2/callback')\ndef oauth2_callback():\n\n code = request.args.get('code')\n access_token_url = 'https://github.com/login/oauth/access_token'\n payload = {\n 'client_id': client_id,\n 'client_secret': client_secret,\n 'code': code,\n # 'redirect_uri':\n 'state': str(uuid4().hex)\n }\n r = requests.post(access_token_url, json=payload, headers={'Accept': 'application/json'})\n access_token = json.loads(r.text).get('access_token')\n print(access_token)\n \n access_user_url = 'https://api.github.com/user'\n response = requests.get(access_user_url, headers={'Authorization': 'token ' + access_token})\n data=response.json()\n user_email=data["email"]\n user_login=data["login"]\n user_id=data["id"]\n # response=redirect(url_for('login',next=redirect_url()))\n # response.set_cookie('user_id', str(user_id))\n # response.set_cookie('user_login', str(user_login))\n # return response\n query = """\n {\n viewer {\n sponsorshipsAsSponsor(first: 100) {\n nodes {\n sponsorable {\n ... on User {\n id\n email\n url\n }\n ... on Organization {\n id\n email\n name\n url\n }\n }\n tier {\n id\n name\n monthlyPriceInDollars\n monthlyPriceInCents\n }\n }\n }\n }\n }\n """\n response_graphql = requests.post('https://api.github.com/graphql', json={'query': query}, headers={'Authorization': 'token ' + access_token})\n response_graphql_data=response_graphql.json()["data"]\n if isinstance (response_graphql_data["viewer"]["sponsorshipsAsSponsor"]["nodes"], list):\n if response_graphql_data["viewer"]["sponsorshipsAsSponsor"]["nodes"][0]["sponsorable"]["name"]==org_name:\n # if user is a sponsor of Applied Knowledge System add them to set of sponsors\n redis_client.sadd(f'sponsors:{org_name}',user_id)\n # if RedisJSON enabled:\n # redis_client.json().set(f"user_details:{user_id}", '$', {\n # 'email': user_email,\n # 'id': user_id,\n # 'user_login': user_login,\n # 'graphql': response_graphql_data,\n # }) \n #if not\n redis_client.hset(f"user_details:{user_id}", mapping={\n 'email': user_email,\n 'id': user_id,\n 'user_login': user_login\n })\n return jsonify({\n 'status': 'success',\n 'email': user_email,\n 'id': user_id,\n 'user_login': user_login\n })The API we are using for our sponsor-only feature is straightforward:
\n@app.route('/exclude', methods=['POST','GET'])\ndef mark_node():\n if request.method == 'POST':\n if 'id' in request.json:\n node_id=request.json['id']\n else:\n if 'id' in request.args:\n node_id=request.args.get('id')\n user_id = session.get('user_id')\n log(f"Got user {user_id} from session")\n if not user_id:\n user_id = request.cookies.get('user_id')\n log(f"Got user {user_id} from cookie")\n redis_client.sadd("user:%s:mnodes" % user_id,node_id)\n response = jsonify(message=f"Finished {node_id} and {user_id}")\n return responseAnd the only purpose of this API is to mark nodes as unimportant for the given user by adding nodes to RedisSet, and those nodes will be excluded from search API output. So far, everything was pretty standard: basic flask API and GitHub Social login flow. Now let's add Redis Enterprise and synchronise sponsors preferences.
\nWhy not use Redis Enterprise directly for everything?\nThe project is memory-heavy, with a lot of data and machine learning inside Redis. This allows to achieve state-of-the-art performance, but it also takes over 120 GB RAM (or as much RAM as you can give it), and 128 GB Redis Enterprise instance will exceed my budget for open-source project. Obviously if there will be enough sponsors we can move more functionality into Redis Enterprise, but for that we need to finish building basic blocks. Register on Redis.com cloud and create a database with the subscription.
\n![\"Redis](\"https://reference-architecture.ai/posts/github-oauth2/redis_enterprise_screen.png\")
\nTake a note host, port and password for Redis Enterprise and create docker enviroment file:
cat .env.gears \nREDISENT_PWD="123"\nREDISENT_PORT="13444"\nREDISENT_HOST="hostname.cloud.redislabs.com"and create a docker compose with section passing .env.gears. Mine looks like this:
\n redisgraph:\n image: redislabs/redismod\n container_name: redisgears\n hostname: redisgears\n env_file:\n - ./.env.gears\n ports:\n - 127.0.0.1:9001:6379\nFirst flow:\nWe will be using RedisGears to synchronize all preferences with Redis Enterprise
\n\nflowchart LR\n id1(User Preferences Redis OSS) --> redis_gears1(Redis Gears)--> redise(Redis Enterprise)\n
If you are new to RedisGears, there is a pattern rgsync that covers exacly this use case, but I already have RedisGears, so I am going to build it step by step:
\n# gears_sync_preferences.py\nrconn=None\n\ndef connecttoRedisEnterise():\n import redis \n import os \n log(str(os.environ))\n # Get environment variables\n\n HOST = os.getenv('REDISENT_HOST')\n PASSWORD = os.getenv('REDISENT_PWD')\n PORT = os.getenv('REDISENT_PORT')\n\n redis_client=redis.Redis(host=HOST,port=PORT,charset="utf-8", password=PASSWORD, decode_responses=True)\n return redis_client\n\ndef sync_users(record):\n global rconn\n if not rconn:\n rconn=connecttoRedisEnterise()\n # Uncomment logs to check \n # log(str(record['key']))\n # log(str(record['value']))\n\n rconn.hset(record['key'],mapping=record['value'])\n\ngb = GB()\ngb.foreach(sync_users)\ngb.count()\ngb.run('user_details:*')this is a \"batch\" mode for RedisGears, which is easier to debug than streams. Install gears-cli](https://github.com/RedisGears/gears-cli) with pip install gears-cli run above script:
gears-cli run --host 127.0.0.1 --port 9001 gears_sync_preferences.py --requirements req_sync.txtwhere req_sync.txt
\nredis==3.5.3This RedisGears will copy all user's profiles into RedisEnterprise. Now let us add sponsors:
\n# gears_sync_sponsors.py\nrconn=None\n\ndef remove_prefix(text, prefix):\n return text[text.startswith(prefix) and len(prefix):]\n\ndef connecttoRedisEnterise():\n import redis \n import os \n log(str(os.environ))\n # Get environment variables\n\n HOST = os.getenv('REDISENT_HOST')\n PASSWORD = os.getenv('REDISENT_PWD')\n PORT = os.getenv('REDISENT_PORT')\n log(HOST)\n log(PORT)\n log(PASSWORD)\n redis_client=redis.Redis(host=HOST,port=PORT,charset="utf-8", password=PASSWORD)\n return redis_client\n\n\n\ndef sync_sponsors(record):\n global rconn\n if not rconn:\n rconn=connecttoRedisEnterise()\n\n log(str(record['key']))\n values=execute('SMEMBERS',record['key'])\n log(str(values))\n for each_value in values: \n rconn.sadd(record['key'],each_value)\n\ngb = GB('KeysReader')\ngb.foreach(sync_sponsors)\ngb.count()\ngb.run('user:*')But this one will sync all user's preferences, but we only need sponsors - let us add another feature of RedisGears - filter:
\nrconn=None\n\ndef remove_prefix(text, prefix):\n return text[text.startswith(prefix) and len(prefix):]\n\ndef connecttoRedisEnterise():\n import redis \n import os \n log(str(os.environ))\n # Get environment variables\n\n HOST = os.getenv('REDISENT_HOST')\n PASSWORD = os.getenv('REDISENT_PWD')\n PORT = os.getenv('REDISENT_PORT')\n\n redis_client=redis.Redis(host=HOST,port=PORT,charset="utf-8", password=PASSWORD, decode_responses=True)\n return redis_client\n\ndef filter_sponsors(record):\n org_name="applied-knowledge-systems"\n user_id = remove_prefix(record['key'],'user:')\n sponsor=execute('SISMEMBER',f'sponsors:{org_name}',user_id)\n return bool(sponsor==True)\n\ndef sync_sponsors(record):\n global rconn\n if not rconn:\n rconn=connecttoRedisEnterise()\n\n log(str(record['key']))\n values=execute('SMEMBERS',record['key'])\n log(str(values))\n for each_value in values: \n rconn.sadd(record['key'],each_value)\n\ngb = GB()\ngb.filter(filter_sponsors)\ngb.foreach(sync_sponsors)\ngb.count()\ngb.run('user:*')Then we are going to use Key miss events from Redis Gears to fetch data for all users:
\n\nflowchart LR\n redise(Redis Enterprise)\n redis_gears2(Redis Gears)--key miss--->redise\n redis_gears2-->redisOSS[Redis OSS]\n
and it's very easy, right from key miss example:
\n\ndef fetch_data(r):\n key = r['key']\n global rconn\n if not rconn:\n rconn=connecttoRedisEnterise()\n values=rconn.smembers(record['key'])\n log(str(values))\n for each_value in values: \n execute('SADD',record['key'],each_value)\n\nGB().foreach(fetch_data).register(prefix='user:*', commands=['smember'],eventTypes=['keymiss'], mode="async_local")There is one more option - to turn fetch_data into the async call, by wrapping it into async/await, but Redis Enterprise is fairly fast, and I don't think it's worth adding an async call in this case. For curiosity, see the example code in The Pattern repository.
\nIn this article, we walked through steps on how to create sponsor-specific \"nanoservices\" using RedisOSS, RedisGears and Redis Enterprise. This allows us to leverage the best of all worlds open source Redis, high availability and persistence with Redis Enterprise and RedisGears as the glue which holds everything together.
\nThis post is in collaboration with Redis.
\nCheck sponsors.go - query whether a GitHub user is your sponsor at a given tier (dollar amount)
\nThere are tools for advanced analytics, including free ones from Google and Kaggle.
\nThere are well-known and validated deployment architectures for applications and the cloud.
\nYet the number of practical applications is still tiny, and they retained niche implementations.\nWhile the benefits of AI are clear, there are still many gaps in AI architecture that need to be filled. For example, there is a gap between analytical tools and verified architectures for real-time deployments. This gap often stems from a lack of specific reference architectures and patterns, demonstrating the trade-offs between technologies, libraries, and tools.
\nLet's bridge the gap in knowledge and drive a connection between science and engineering to make fast, efficient, and practical AI deployments.\nThree things need to be in place to build an AI product:
\nI will use The Pattern, my [“Build on Redis” Hackathon prize-winning open source](https://github.com/applied-knowledge-systems/the-pattern) project, to illustrate how the capabilities below can be implemented and invite you to contribute or donate.
\nWe launch in two full-featured articles - NLP ML pipeline for turning unstructured JSON text into a knowledge graph and fresh off the press Benchmarks for BERT Large Question Answering inference for RedisAI and RedisGears with Grafana Dashboards by Mikhail Volkov
\n" },{"title": "Support the project", "url": "https://reference-architecture.ai/docs/donate/", "body": "The patterns described on this site are not theory. They are extracted from\nTerraphim, a privacy-first AI assistant built across 52 Rust\ncrates, and every one of those crates is open source.
\nThe most valuable support is use and scrutiny.
\nterraphim_automata for Aho-Corasick matching,\nterraphim_rolegraph for role-based knowledge\ngraphs, or terraphim_persistence for the\nstorage layer.Issues, benchmarks and architecture critique are worth more than money here. If a\ntrade-off on this site looks wrong, say so — with evidence — at\ngithub.com/terraphim. Deployment guides and reproducible\nbenchmarks are especially welcome.
\nIf your organisation depends on this work, Terraphim Pro funds\nthe open-source engine directly: connectors, an LLM proxy and agentic skills for\nenterprise deployment.
\n" },{"title": "Capability Map (2020-2022)", "url": "https://reference-architecture.ai/docs/legacy-capability-map/", "body": "There are tools for advanced analytics, including free ones from Google and Kaggle.
\nThere are well-known and validated deployment architectures for applications and the cloud.
\nYet the number of practical applications is still tiny, and they retained niche implementations.\nWhile the benefits of AI are clear, there are still many gaps in AI architecture that need to be filled. For example, there is a gap between analytical tools and verified architectures for real-time deployments. This gap often stems from a lack of specific reference architectures and patterns, demonstrating the trade-offs between technologies, libraries, and tools.
\nLet's bridge the gap in knowledge and drive a connection between science and engineering to make fast, efficient, and practical AI deployments.
\nThree things need to be in place to build an AI product:
\nI will use The Pattern, my Build on Redis Hackathon prize-winning open source project to illustrate how the capabilities below can be implemented and invite you to contribute or donate.\nThe diagrams below are clickable.
\n![](\"/images/ai_product.drawio.svg\")
\nflowchart LR; \nsubgraph \" \"\n data_intake(Data Acquisition);\n data_preprocessing(Data Preprocessing);\n data_validation(Data Validation);\n data_streaming(Data Streaming);\n ai_ml_pipe(AI ML Pipeline
Model Training/Test/Validation);\n kg(Knowledge Graph);\n ML_inference(ML Inference);\n interaction(Interaction Layer
Voice/VR/AR/Meta);\n click data_intake \"/docs/intake/\" \"Data Acquisition\"\n click ML_inference \"/docs/bert-qa-benchmarking/\" \"BERT Large Question Answering\"\n click kg \"/docs/nlp/#what-is-a-knowledge-graph\" \"Knowledge Graph\"\n click data_preprocessing \"/docs/nlp/#redisgears-for-nlp-pre-processing\" \"Data pre-processing\"\n click data_streaming \"/docs/nlp/#goal\" \"Data Streaming\"\n click ai_ml_pipe \"/docs/nlp/#overall-architecture-overview-components-diagram\" \"Overall ML pipeline\"\nend\n
\nflowchart LR;\nsubgraph 2 [Enabling Capabilities]\n subgraph \" \"\nD1[Data Governance]\nD2[Data Quality Management]\nD3[Metadata Management]\nclick D3 \"/docs/metadata/\" \"Metadata Management\"\nend\nsubgraph \" \"\n id2[ML performance and bias monitoring];\n id3[Application Performance Monitoring];\n id4[Hardware Performance Monitoring];\n click id3 \"/docs/bert-qa-benchmarking/#running-the-benchmark\" \"Running Benchmarks\"\n click id4 \"/docs/bert-qa-benchmarking/#using-grafana-to-monitor-redisgears-throughput-cpu-and-memory-usage\" \"CPU and Memory Benchmarks\"\nend\nsubgraph \" \"\n id5[DevOps: Continuous Integration/Continuous Deployment];\n c1[Configuration management]\n c2[Collaboration and knowledge management tooling]\n c3[Change Management]\nend \n subgraph 3 [Self Serving Infrastructure]\n s1[Computing Infrastructure]\n s2[Serving infrastructure]\n end\n end\n style 2 fill:#485fc754,stroke:#333,stroke-width:4px;\n
This article will explore the challenges and opportunities of deploying a large BERT Question Answering Transformer model(bert-large-uncased-whole-word-masking-finetuned-squad) from inside Huggingface, where RedisGears and RedisAI perform heavy lifting while leveraging in-memory datastore Redis.
\nSome numbers for inspiration and why to read this article:
\npython3 transformers_plain_bert_qa.py \nairborne transmission of respiratory infections is the lack of established methods for the detection of airborne respiratory microorganisms\n10.351818372 secondstime curl -i -H "Content-Type: application/json" -X POST -d '{"search":"Who performs viral transmission among adults"}' http://localhost:8080/qasearch\n\nreal\t0m0.747s\nuser\t0m0.004s\nsys\t0m0.000s\nBERT Question Answering inference works where the ML model selects an answer from the given text. In other words, BERT QA \"thinks\" through the following: \"What is the answer from the text, assuming the answer to the question exists within the paragraph selected.\"
\nSo it's important to select text potentially containing an answer. A typical pattern is to use Wikipedia data to build Open Domain Question Answering.
\nOur QA system is a medical domain-specific question/answering pipeline, hence we need a first pipeline that turns data into a knowledge graph. This NLP pipeline is available at Redis LaunchPad, is fully open source, and is described in a previous article. Here is a 5 minute video describing it, and below you will find an architectural overview:
\n![\"featured\"](\"https://reference-architecture.ai/docs/bert-qa-benchmarking/featured.png\")
In the BERT QA pipeline (or in any other modern NLP inference task), there are two steps:
\nWith Redis, we have the opportunity to pre-compute everything and store it in memory, but how do we do it? Unlike with the summarization ML learning task, the question is not known in advance, so we can't pre-compute all possible answers. However, we can pre-tokenize all potential answers (i.e. all paragraphs in the dataset) using RedisGears:
\ndef parse_sentence(record):\n import redisAI\n import numpy as np\n global tokenizer\n if not tokenizer:\n tokenizer=loadTokeniser()\n hash_tag="{%s}" % hashtag()\n\n for idx, value in sorted(record['value'].items(), key=lambda item: int(item[0])):\n tokens = tokenizer.encode(value, add_special_tokens=False, max_length=511, truncation=True, return_tensors="np")\n tokens = np.append(tokens,tokenizer.sep_token_id).astype(np.int64)\n tensor=redisAI.createTensorFromBlob('INT64', tokens.shape, tokens.tobytes())\n\n key_prefix='sentence:'\n sentence_key=remove_prefix(record['key'],key_prefix)\n token_key = f"tokenized:bert:qa:{sentence_key}:{idx}"\n redisAI.setTensorInKey(token_key, tensor)\n execute('SADD',f'processed_docs_stage3_tokenized{hash_tag}', token_key)\nSee the full code on GitHub.
\nThen for each Redis Cluster shard, we pre-load the BERT QA model by downloading, exporting it into torchscript, then loading it into each shard:
\ndef load_bert():\n model_file = 'traced_bert_qa.pt'\n\n with open(model_file, 'rb') as f:\n model = f.read()\n startup_nodes = [{"host": "127.0.0.1", "port": "30001"}, {"host": "127.0.0.1", "port":"30002"}, {"host":"127.0.0.1", "port":"30003"}]\n cc = ClusterClient(startup_nodes = startup_nodes)\n hash_tags = cc.execute_command("RG.PYEXECUTE", "gb = GB('ShardsIDReader').map(lambda x:hashtag()).run()")[0]\n print(hash_tags)\n for hash_tag in hash_tags:\n print("Loading model bert-qa{%s}" %hash_tag.decode('utf-8'))\n cc.modelset('bert-qa{%s}' %hash_tag.decode('utf-8'), 'TORCH', 'CPU', model)\n print(cc.infoget('bert-qa{%s}' %hash_tag.decode('utf-8')))The full code is available on GitHub.
\nAnd when a question comes from the user, we tokenize and append the question to the list of potential answers before running the RedisAI model:
\n token_key = f"tokenized:bert:qa:{sentence_key}"\n # encode question\n input_ids_question = tokenizer.encode(question, add_special_tokens=True, truncation=True, return_tensors="np")\n t=redisAI.getTensorFromKey(token_key)\n input_ids_context=to_np(t,np.int64)\n # merge (append) with potential answer, context - is pre-tokenized paragraph\n input_ids = np.append(input_ids_question,input_ids_context)\n attention_mask = np.array([[1]*len(input_ids)])\n input_idss=np.array([input_ids])\n num_seg_a=input_ids_question.shape[1]\n num_seg_b=input_ids_context.shape[0]\n token_type_ids = np.array([0]*num_seg_a + [1]*num_seg_b)\n # create actual model runner for RedisAI\n modelRunner = redisAI.createModelRunner(f'bert-qa{hash_tag}')\n # make sure all types are correct\n input_idss_ts=redisAI.createTensorFromBlob('INT64', input_idss.shape, input_idss.tobytes())\n attention_mask_ts=redisAI.createTensorFromBlob('INT64', attention_mask.shape, attention_mask.tobytes())\n token_type_ids_ts=redisAI.createTensorFromBlob('INT64', token_type_ids.shape, token_type_ids.tobytes())\n redisAI.modelRunnerAddInput(modelRunner, 'input_ids', input_idss_ts)\n redisAI.modelRunnerAddInput(modelRunner, 'attention_mask', attention_mask_ts)\n redisAI.modelRunnerAddInput(modelRunner, 'token_type_ids', token_type_ids_ts)\n redisAI.modelRunnerAddOutput(modelRunner, 'answer_start_scores')\n redisAI.modelRunnerAddOutput(modelRunner, 'answer_end_scores')\n # run RedisAI model runner\n res = await redisAI.modelRunnerRunAsync(modelRunner)\n answer_start_scores=to_np(res[0],np.float32)\n answer_end_scores = to_np(res[1],np.float32)\n answer_start = np.argmax(answer_start_scores)\n answer_end = np.argmax(answer_end_scores) + 1\n answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end],skip_special_tokens = True))\n log("Answer "+str(answer))\n return answer\nCheckout the full code, available on GitHub.
\nThe process for making a BERT QA API call looks like this:
\n![\"Architecture](\"https://reference-architecture.ai/docs/bert-qa-benchmarking/diagrams_02.png\")
Here I use two cool features of RedisGears: capturing events on key miss and using async/await to run RedisAI on each shard without locking the primary thread - so that Redis Cluster can continue to serve other customers. For benchmarks, caching responses from RedisAI is disabled. If you are getting response times in nanoseconds on the second call rather then milliseconds, check to make sure the line linked above is commented out.
\nPre-requisites for running the benchmark:
\nAssuming you are running Debian or Ubuntu and have Docker and docker-compose installed (or can create a virtual environment via conda), run the following commands:
\ngit clone --recurse-submodules https://github.com/applied-knowledge-systems/the-pattern.git\ncd the-pattern\n./bootstrap_benchmark.shThe above commands should end with a curl call to the qasearch API, since Redis caching is disabled for the benchmark.
\nNext, invoke curl like this:
\ntime curl -i -H "Content-Type: application/json" -X POST -d '{"search":"Who performs viral transmission among adults"}' http://localhost:8080/qasearchExpect the following output, or something similar based on your runtime environment:
\nHTTP/1.1 200 OK\nServer: nginx/1.18.0 (Ubuntu)\nDate: Sun, 29 May 2022 12:05:39 GMT\nContent-Type: application/json\nContent-Length: 2120\nConnection: keep-alive\n\n{"links":[{"created_at":"2002","rank":13,"source":"C0001486","target":"C0152083"}],"results":[{"answer":"adenovirus","sentence":"The medium of 40 T150 flasks of adenovirus transducer dec CAR CHO cells yielded 0 5 1 my of purified msCEACAM1a 1 4 protein","sentencekey":"sentence:PMC125375.xml:{mG}:202","title":"Crystal structure of murine sCEACAM1a[1,4]: a coronavirus receptor in the CEA family"}] OUTPUT_REDUCTED}I modified the output of API for the benchmark to return results from all shards - even if the answer is empty, in the run above five shards return answers, overall API call response under second with all additional hops to search in RedisGraph.
\nI modified the output of the API for the benchmark to return results from all shards - even if the answer is empty. In the run above five shards return answers. The overall API call response takes less than one second with all additional hops to search in RedisGraph!
\n![\"Architecture](\"https://reference-architecture.ai/docs/bert-qa-benchmarking/diagrams_01.png\")
Let's dig deeper into what's happening under the hood:
\nYou should have a sentence key with shard id, which you get by looking at the \"Cache key\" from docker logs -f rgcluster. In my setup the cache key is, \"bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults\". If you think it looks like a function call it's because it is a function call. It is triggered if the key isn't present in the Redis Cluster, which for the benchmark will be every time since if you remember we disabled caching the output.
One more thing to figure out from the logs is the port of the shard corresponding to the hashtag, also known as the shard id. It is the text found in betweeen the curly brackets – looks like {6fd} above. The same will be in the output for the export_load script. In my case the cache key was found in \"30012.log\", so my port is 30012.
Next I run the following command:
\nredis-cli -c -p 300012 -h 127.0.0.1 get "bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults"and then run the benchmark:
\nredis-benchmark -p 30012 -h 127.0.0.1 -n 10 get "bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults"\n====== get bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults ======\n 10 requests completed in 0.04 seconds\n 50 parallel clients\n 3 bytes payload\n keep alive: 1\n\n10.00% <= 41 milliseconds\n100.00% <= 41 milliseconds\n238.10 requests per secondIf you are wondering, -n = number of times. In this case we run the benchmark 10 times. You can also add:
– csv if you want to output in CSV format
– precision 3 if you want more decimals in the ms
More information about the benchmarking tool can be found on the redis.io Benchmarks page.
\nif you don't have redis-utils installed locally, you can use Docker as follows:
\ndocker exec -it rgcluster /bin/bash\nredis-benchmark -p 30012 -h 127.0.0.1 -n 10 get "bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults"\n====== get bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults ======\n 10 requests completed in 1.75 seconds\n 50 parallel clients\n 99 bytes payload\n keep alive: 1\n host configuration "save":\n host configuration "appendonly": no\n multi-thread: no\n\nLatency by percentile distribution:\n0.000% <= 243.711 milliseconds (cumulative count 1)\n50.000% <= 987.135 milliseconds (cumulative count 5)\n75.000% <= 1577.983 milliseconds (cumulative count 8)\n87.500% <= 1662.975 milliseconds (cumulative count 9)\n93.750% <= 1744.895 milliseconds (cumulative count 10)\n100.000% <= 1744.895 milliseconds (cumulative count 10)\n\nCumulative distribution of latencies:\n0.000% <= 0.103 milliseconds (cumulative count 0)\n10.000% <= 244.223 milliseconds (cumulative count 1)\n20.000% <= 409.343 milliseconds (cumulative count 2)\n30.000% <= 575.487 milliseconds (cumulative count 3)\n40.000% <= 821.247 milliseconds (cumulative count 4)\n50.000% <= 987.135 milliseconds (cumulative count 5)\n60.000% <= 1157.119 milliseconds (cumulative count 6)\n70.000% <= 1497.087 milliseconds (cumulative count 7)\n80.000% <= 1577.983 milliseconds (cumulative count 8)\n90.000% <= 1662.975 milliseconds (cumulative count 9)\n100.000% <= 1744.895 milliseconds (cumulative count 10)\n\nSummary:\n throughput summary: 5.73 requests per second\n latency summary (msec):\n avg min p50 p95 p99 max\n 1067.296 243.584 987.135 1744.895 1744.895 1744.895The platform only has 20 articles and 8 Redis nodes (4 masters + 4 slaves), so relevance would be wrong and it doesn't need a lot of memory.
\nNow let's check how long our RedisAI model runs on the {6fd} shard:
127.0.0.1:30012> AI.INFO bert-qa{6fd}\n 1) "key"\n 2) "bert-qa{6fd}"\n 3) "type"\n 4) "MODEL"\n 5) "backend"\n 6) "TORCH"\n 7) "device"\n 8) "CPU"\n 9) "tag"\n10) ""\n11) "duration"\n12) (integer) 8928136\n13) "samples"\n14) (integer) 58\n15) "calls"\n16) (integer) 58\n17) "errors"\n18) (integer) 0\nbert-qa{6fd} is the key of the actual (very large) model saved. The AI.INFO command gives us a cumulative duration of 8928136 microseconds and 58 calls, which is approximately 153 milliseconds per call.
Let's double-check to make sure that's right by resetting the stats and then re-runnning the benchmark.
\nFirst, reset the stats:
\n127.0.0.1:30012> AI.INFO bert-qa{6fd} RESETSTAT\nOK\n127.0.0.1:30012> AI.INFO bert-qa{6fd}\n 1) "key"\n 2) "bert-qa{6fd}"\n 3) "type"\n 4) "MODEL"\n 5) "backend"\n 6) "TORCH"\n 7) "device"\n 8) "CPU"\n 9) "tag"\n10) ""\n11) "duration"\n12) (integer) 0\n13) "samples"\n14) (integer) 0\n15) "calls"\n16) (integer) 0\n17) "errors"\n18) (integer) 0Then, re-run the benchmark:
\nredis-benchmark -p 30012 -h 127.0.0.1 -n 10 get "bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults"\n====== get bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults ======\n 10 requests completed in 1.78 seconds\n 50 parallel clients\n 99 bytes payload\n keep alive: 1\n host configuration "save":\n host configuration "appendonly": no\n multi-thread: no\n\nLatency by percentile distribution:\n0.000% <= 188.927 milliseconds (cumulative count 1)\n50.000% <= 995.839 milliseconds (cumulative count 5)\n75.000% <= 1606.655 milliseconds (cumulative count 8)\n87.500% <= 1692.671 milliseconds (cumulative count 9)\n93.750% <= 1779.711 milliseconds (cumulative count 10)\n100.000% <= 1779.711 milliseconds (cumulative count 10)\n\nCumulative distribution of latencies:\n0.000% <= 0.103 milliseconds (cumulative count 0)\n10.000% <= 189.183 milliseconds (cumulative count 1)\n20.000% <= 392.191 milliseconds (cumulative count 2)\n30.000% <= 540.159 milliseconds (cumulative count 3)\n40.000% <= 896.511 milliseconds (cumulative count 4)\n50.000% <= 996.351 milliseconds (cumulative count 5)\n60.000% <= 1260.543 milliseconds (cumulative count 6)\n70.000% <= 1456.127 milliseconds (cumulative count 7)\n80.000% <= 1606.655 milliseconds (cumulative count 8)\n90.000% <= 1692.671 milliseconds (cumulative count 9)\n100.000% <= 1779.711 milliseconds (cumulative count 10)\n\nSummary:\n throughput summary: 5.62 requests per second\n latency summary (msec):\n avg min p50 p95 p99 max\n 1080.454 188.800 995.839 1779.711 1779.711 1779.711Now check the stats again:
\nAI.INFO bert-qa{6fd}\n 1) "key"\n 2) "bert-qa{6fd}"\n 3) "type"\n 4) "MODEL"\n 5) "backend"\n 6) "TORCH"\n 7) "device"\n 8) "CPU"\n 9) "tag"\n10) ""\n11) "duration"\n12) (integer) 1767749\n13) "samples"\n14) (integer) 20\n15) "calls"\n16) (integer) 20\n17) "errors"\n18) (integer) 0Now we get 88387.45 microseconds per call ~0.088387 seconds, which is pretty fast! Also, considering we started with 10 seconds per call, I think the benefits of using RedisAI in combination with RedisGears are pretty obvious. However, the trade-off is high memory usage.
\nThere are many ways to optimize this deployment. For example, you can add a FP16 quantization and ONNX runtime. If you would like to try that, this script will be a good starting point.
\nThanks to the contribution of Mikhail Volkov, we can now observe RedisGears and RedisGraph throughput and memory consumption using Grafana. When you cloned repository it started Graphana Docker, which has pre-build templates to monitor RedisCluster, including RedisGears and RedisAI, and Graph - which is Redis with RedisGraph. \"The Pattern\" dashboard provides an overview, with all the key benchmark metrics you care about:
\n![\"Grafana](\"https://reference-architecture.ai/docs/bert-qa-benchmarking/graphana_redis_graph.png\")
![\"Grafana](\"https://reference-architecture.ai/docs/bert-qa-benchmarking/graphana_cluster_overview.png\")
This post is in collaboration with Redis.
\n" },{"title": "Building a Pipeline for Natural Language Processing using RedisGears", "url": "https://reference-architecture.ai/docs/nlp/", "body": "Disclaimer originally published in collaboration with Ajeet Raina on Developer.Redis.Com
\nIn this tutorial, you will learn how to build a pipeline for Natural Language Processing(NLP) using RedisGears. For this demonstration, we will be leveraging the Kaggle CORD19 datasets. The implementation is designed to avoid running out of memory, leveraging Redis Cluster and RedisGears, where the use of RedisGears allows for processing data on storage without the need to move data in and out of the Redis Cluster—using Redis Cluster as data fabric. Redis Cluster allows for horizontal scalability up to 1000 nodes, and together with RedisGears, provides a distributed system where data science/ML engineers can focus on processing steps, without the worry of writing tons of scaffoldings for distributed calculations.
\n![\"nlp\"](\"https://reference-architecture.ai/docs/nlp/nlparch.png\")
This project was built with the aim to make it easier for other people to contribute and build better information and knowledge management products.
\nRedisGears have enormous potential, particularly for text processing—you can process your data “on data” without needing to move them in and out of memory. Summary of the important points:
\nToday, we live in the world of new systems that operate not just files, folders, or web pages, but entities with their properties and relationships between them, organized into hierarchies of classes and categories. These systems are used everywhere from the military-industrial complex to our everyday lives. Palantir, Primer, and other data companies enable massive intelligence and counterintelligence projects in military and security forces, Quid and RecordedFuture enable competitive analytics, Bottlenose and similar enterprises enable online reputation analytics. Microsoft Graph enables new kinds of productivity apps for the enterprises, Google Knowledge Graph and Microsoft’s Satori enable everyday search queries, and together with Amazon Information Graph they power corresponding AI assistants by enabling them to answer questions about the world facts
\nAll these (and many other more specialized) systems are used in different domains, but all of them use Knowledge Graphs as their foundation.
\nKnowledge graphs are one of the best ways to connect and make sense out of information from different data sources, following the motto of one of the vendors— “It’s about things not strings”.
\nKnowledge Graph consists of thesaurus, taxonomy and ontology. In this pipeline I assume knowledge is captured in medical metathesaurus UMLS and concepts in text are related if they are part of the same sentence, therefore concept become node, their relationship becomes edge:
\n![\"concepts1\"](\"https://reference-architecture.ai/docs/nlp/concepts1.png\")
\n![\"concepts2\"](\"https://reference-architecture.ai/docs/nlp/concepts2.png\")
Concepts have CUI (Concept Unique Identifiers) and those will be primary keys in nodes, linked to UMLS thesaurus. For example, if you search, “How does temperature and humidity affect the transmission of 2019-nCoV?” on the demo website http://thepattern.digital/ and move slider to 1996, there is an edge-connecting transmission (C5190195) and birth (C5195639) and the part of sentence matched, “the rate of transmission to an infant born to,” from the report titled, “Afebrile Pneumonia in infants.”
\n![\"concepts3\"](\"https://reference-architecture.ai/docs/nlp/concepts3.png\")
![\"component_diagram\"](\"https://reference-architecture.ai/docs/nlp/component_diagram.png\")
Intake step - is very simple put all JSON records into RedisCluster, then NLP pipeline starts processing all records, code is here.
\nFor each record — detect language (discard non English), it’s filter
\nMap paragraphs into a sentence — flatmap
\nSentences spellchecker — it’s map
\nSave sentences into hash — processor
\nEnsure that you install virtualenv in your system
\n git clone --recurse-submodules https://github.com/applied-knowledge-systems/the-pattern.git\n cd the-pattern docker-compose -f docker-compose.dev.yml up --build -dYou can deploy PyTorch and spacy to run on RedisGears.
\n bash post_start_dev.shFor Data science-focused deployment, RedisCluster should be in HA mode with at least one slave for each master.\nOne need to change a few default parameters for rgcluster to accommodate the size of PyTorch and spacy libraries (each over 1GB zipped), gist with settings.
\n cd ./the-pattern-platform/You can create it via
\n conda create -n pattern_env python=3.8or
\nAlternatively, you can activate by using the below CLI:
\n source ~/venv_cord19/bin/activate #or create new venv\n pip install -r requirements.txt bash cluster_pipeline.shWait for a bit and then check:
\n redis-cli -p 9001 -h 127.0.0.1 GRAPH.QUERY cord19medical "MATCH (n:entity) RETURN count(n) as entity_count" \n redis-cli -p 9001 -h 127.0.0.1 GRAPH.QUERY cord19medical "MATCH (e:entity)-[r]->(t:entity) RETURN count(r) as edge_count" curl -i -H "Content-Type: application/json" -X POST -d '{"search":"How does temperature and humidity affect the transmission of 2019-nCoV"}' \n http://localhost:8080/gsearchWhile RedisGears allows to deploy and run Machine Learning libraries like spacy and BERT transformers, the solution above uses simpler approach:
\n gb = GB('KeysReader')\n gb.filter(filter_language)\n gb.flatmap(parse_paragraphs)\n gb.map(spellcheck_sentences)\n gb.foreach(save_sentences)\n gb.count()\n gb.register('paragraphs:*',keyTypes=['string','hash'], mode="async_local")This is the overall pipeline: those 7 lines allow you to run logic in a distributed cluster or on a single machine using all available CPUs - no changes required until you need to scale over more 1000 nodes. I use KeysReader registered for namespace paragraphs for all strings or hashes. My pipeline would need to run in async mode. For data scientists, I would recommend using gb.run to make sure gears function work and it will run in batch mode and then change it to register - to capture new data. By default, functions will return output, hence the need for count() - to prevent fetching the whole dataset back to the command issuing machine (90 GB for Cord19).
\nOverall pre-processing is a straightforward - full code is here.
\nThings to keep in mind:
\n execute('SADD','processed_docs_{%s}' % hashtag(),article_id) """\n load symspell and relevant dictionaries\n """\n sym_spell=None \n\n def load_symspell():\n import pkg_resources\n from symspellpy import SymSpell, Verbosity\n sym_spell = SymSpell(max_dictionary_edit_distance=1, prefix_length=7)\n dictionary_path = pkg_resources.resource_filename(\n "symspellpy", "frequency_dictionary_en_82_765.txt")\n bigram_path = pkg_resources.resource_filename(\n "symspellpy", "frequency_bigramdictionary_en_243_342.txt")\n # term_index is the column of the term and count_index is the\n # column of the term frequency\n sym_spell.load_dictionary(dictionary_path, term_index=0, count_index=1)\n sym_spell.load_bigram_dictionary(bigram_path, term_index=0, count_index=2)\n return sym_spell bg = GearsBuilder('KeysReader')\n bg.foreach(process_item)\n bg.count()\n bg.register('sentence:*', mode="async_local",onRegistered=OnRegisteredAutomata)\nOnRegisteredAutomata will perform similarly to symspell example above except it will download pre-build Aho-Corasick automata (30Mb).\nAho-Corasick is a very fast matcher and allows to perform >900 Mb text per second even on commodity laptop, RedisGears cluster makes a very smooth distribution of data and ML model and matching using available CPU and Memory. Full matcher code.
\nOutput of the matcher: nodes and edges are candidates to use another RedisGears pattern rgsync where you can write fast into Redis and RedisGears are going to replicate data into slower storage using RedisStreams.\nBut I decided to use streams and handcraft the population of the RedisGraph database, which will be focus of the next blog post.
\nOutput of the matcher: nodes and edges are candidates to use another RedisGears pattern rgsync where you can write fast into Redis and RedisGears are going to replicate data into slower storage using RedisStreams, while this demo uses streams and populates RedisGraph database with nodes and edges calculating rank of each.
\nWe took OCR scans in JSON format and turned them into Knowledge Graph, demonstrating how you can traditional Semantic Network/OWL/Methathesaurus technique based on Unified Medical Language System. Redis Ecosystem offers a lot to the data science community, and can take place at the core of Kaggle notebooks, ML frameworks and make deployment and distribution of data more enjoyable. The success of our industry depends on how our tools work together — regardless of whether they are engineering, data science, machine learning and organisational or architectural.
\nWith the collaboration of RedisLabs and community, the full pipeline code is available via https://github.com/applied-knowledge-systems/the-pattern-platform. In case, you want to try it locally, then you can find a Docker Launch script in the root of the repository along with short quickstart guide. PR and suggestions are welcome. The overall goal of the project is to allow other to build their more interesting pipeline on top of it.
\nIn CORD 19 dataset mentioned in Data Acquisition Metadata stored in the separate csv file from the source data. Here simple script to parse date/times and attach it to JSON/XML files
\n" },{"title": "Contribution Guidelines", "url": "https://reference-architecture.ai/docs/contribution/", "body": "General guidelines for contributing to the project.
\nThere should be a a path to be implemented in a real world - good prototype or production deployment.
\nWelcome pull requests on \n\n\n\n. Check out supported shortcuts Extended Shortcuts
\nWhen contributing you agreeing to share your contribution under ![](\"https://img.shields.io/badge/License-CC%20BY--SA%204.0-lightgrey.svg\"){kind=icon}
For the Reference Architecture for AI, we used Kaggle Cord19 dataset, \"COVID-19 Open Research Dataset (CORD-19). CORD-19 is a resource of over 1,000,000 scholarly articles, including over 400,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. This freely available dataset is provided to the global research community to apply recent advances in natural language processing and other AI techniques to generate new insights in support of the ongoing fight against this infectious disease.\"
\nExample script parses documents taking out body_text and saves under paragraphs in Redis cluster.
\n" },{"title": "The Pattern: Machine Learning Natural Language Processing meets VR/AR", "url": "https://reference-architecture.ai/docs/ai-product/", "body": "To fight ever-increasing complexity, \"The Pattern\" projects help find relevant knowledge using Artificial Intelligence and novel UX elements, all powered by Redis - a new generation real-time data fabric turned into knowledge fabric
\nOverall repository for CORD19 medical NLP pipeline, API and UI, design and architecture.
\nDemo Video:
\n\nDemo Server (no persistance): https://thepattern.digital/
\nThe medical profession put a lot of effort into collaboration, starting from Latin as a common language to industry-wide thesauruses like UMLS. However, if full of scandals where publications in a prestigious journal would be retracted, and the World Health Organisation would change its policy advice based on the article. I think \"paper claiming that eating a bat-like Pokémon sparked the spread of COVID-19\" takes a prize. One would say that editors in those journals don't do their job, and while it may seem true, I would say they had no chance: with a number of publications about COVID (SARS-V) passing 300+ per day, we need better tools to navigate via such flow of information.\nWhen exploring science or engineering topics, I look at the diversity of the opinion, not the variety of the same cluster of words or the same thought. I want to avoid confirmation bias. I want to find articles relevant to the same concept, not necessarily the ones which have similar words. My focus is to build a natural language processing pipeline capable of handling a large number of documents and concepts, incorporating System 1 AI (fast, intuitive reasoning) and System 2 (high-level reasoning) and then present knowledge in a modern VR/AR visualisation. Search or rather information exploration should be spatial, preferably in VR (memory palace, see Theatre of Giulio Camillo). A force-directed graph is a path towards it, where visuals are assisted by text — relevant text pops up on the connection and where people explore the concepts and then dig deeper into the text. The purpose of the pipeline is that knowledge should be reusable and shareable.
\nJoin our community on Discord or post on GitHub Discussions](https://github.com/applied-knowledge-systems/the-pattern/discussions)
\n" }]