![\"Proposed](\"/diagrams/roles-contract-architecture.png\")
An organisational AI agent needs defined responsibilities, relevant knowledge and bounded authority. This paper proposes roles as contracts for all three. A Project Manager plans delivery; a Product Manager decomposes product outcomes into features. Each uses a governed role-based knowledge graph. Agent services realise role capabilities through tools whose actions require contextual permission. Verification and validation require compatible concepts, relationships and operating conditions across the input and evaluation criteria, with explicit mappings where views differ. A cross-check against ISO/IEC/IEEE 42010:2022 and 29148:2018 informs these distinctions without claiming conformity. Terraphim provides graph retrieval and checklists; Terraph connects concept matching to proof-gated verbs; ADF configurations define responsibilities and artefact templates. The proposed integration would allow a Project Manager agent to create a checked plan from an approved template while reserving production deployment for a separately authorised role.
\n\n
This paper continues the argument developed by David Hunt and Alexander Mikhalev in AI Risk is evolving, every day., published on 31 October 2026. Their starting concern is that an agent can pursue a legitimate objective through unauthorised methods. They connect oversight of business activity with cybersecurity and knowledge integrity, and argue that changes to agents and their surroundings can invalidate earlier assurance.
\nSupervision needs an inspectable mandate: the organisational role, the capability being exercised, the service providing it, and the tool action requested against a particular resource. Role-based knowledge graphs provide the relevant concepts and relationships; an independently enforced permission decision determines whether the action may proceed. A Project Manager can prepare a delivery plan using approved templates while production execution remains assigned to a separately authorised role. The sections below distinguish existing components from the integration still required.
\nA successful check needs criteria that apply to the input's meaning, obligations and operating context. Matching words cannot establish that a plan preserves a prohibition, uses the correct environment, or retains an acceptance threshold. The semantic compatibility contract developed here connects requirements, criteria and evidence through a versioned baseline and explicit mappings. Changes to that baseline trigger reassessment of affected results. The standards cross-check helps define the responsibilities and evaluation claims; it does not establish that the supervisory system has been implemented.
\nThe common concepts connect the earlier article's concerns to the mechanisms examined in this paper:
\n| Shared concept | Engineering continuation in this paper |
|---|---|
| Mandate and justified authority | Trace the assigned role through capability, service, tool and action to a contextual permission decision. |
| Knowledge integrity and context | Use governed concept identities, relationships and requirement baselines to preserve meaning across inputs, outputs and criteria. |
| Independent supervision | Separate the agent's proposal from the gateway's permission decision and the evidence used to assess its result. |
| Continued validity of assurance | Bind findings to the evaluated artefact, criteria and context; reassess affected claims when relevant versions change. |
| Learning and authority to change | Govern changes to role graphs, templates and policies separately from permission to perform the task. |
| Prediction and operational evidence | Keep illustrative scenarios, expected outcomes and recorded executions distinguishable; validate predictions before relying on them. |
| Consequences across organisational scales | Make resource and project scope explicit, and identify where wider consequences require escalation beyond the acting role. |
This mapping is our architectural synthesis. The examples below illustrate individual mechanisms; connecting them across systems remains engineering work.
\n“Planning agent” describes a computational activity. It does not identify who owns the outcome, which planning conventions apply, or what authority accompanies the work. The same ambiguity affects “research agent”, “review agent” and “execution agent”. These labels can be useful implementation names, but they leave organisational meaning unresolved.
\nA Project Manager plans delivery against agreed scope, dependencies, resources and milestones. A Product Manager decomposes desired outcomes into features, evaluates value and defines acceptance criteria. Both plan, but they plan different objects, draw on different evidence, and answer to different accountabilities. An organisation should therefore delegate to an agent acting in a defined role, with planning as one of that role’s capabilities.
\nThe role contract can remain stable when an agent’s model, prompt, framework or supplier changes. A replacement implementation must demonstrate the required service behaviour and operate within the same authority boundary. Governed role definitions and capability contracts give the organisation a basis for evaluating that replacement.
\nThe central proposal is a traceable relationship:
\n\n\nOrganisation → role → capability → service → tool → action → contextual permission.
\n
This sequence is an explanation path, not an assertion that authority flows automatically through every edge. The organisation assigns responsibilities to roles; an agent principal is separately authorised to enact a role. Its implementation supplies services. Tools make actions technically available. Policy determines whether an exact requested action is permitted in its current context.
\nThe organisational semantic layer describes concepts, relationships, templates, standards and responsibilities using explicit identifiers and versioned definitions. Here, conceptual space means this governed system of organisational meaning. The proposal does not require a fully implemented geometric model of human conceptual spaces.
\n| Element | Organisational meaning | Planning example |
|---|---|---|
| Role | A position with responsibilities and bounded authority | Project Manager |
| Capability | An ability the organisation needs to achieve an outcome | Delivery planning |
| Service | An accessible behaviour provided by an implementation | Produce a delivery plan |
| Agent implementation | Software that provides the service while enacting an assigned role | Project planning agent instance |
| Tool | A concrete interface through which work is performed | Template reader or document writer |
| Action | A specific operation on an identified resource | Create a draft plan in project workspace |
| Permission | A policy decision over principal, role, action, resource and context | Draft creation allowed for the assigned project |
| Evidence | Records supporting a decision and its observed result | Template version, checks and execution receipt |
This distinction between capabilities and services is compatible with enterprise architecture’s separation of organisational behaviour from its realisation and exposed services. It is used here as an architectural vocabulary, without claiming formal compliance with an ArchiMate model. Open Group ArchiMate 101
\nA capability describes what the organisation can accomplish. It is not a permission token. Likewise, an implementation’s demonstrated competence does not grant authority. A model may understand deployment instructions while having no right to invoke a deployment endpoint.
\nRole capabilities contribute to broader organisational capabilities. For example, an organisation’s Product Delivery capability can depend on Product Discovery performed by the Product Manager, Delivery Planning performed by the Project Manager, and Release Assurance performed by the Release Manager. The graph records these contribution relationships explicitly. Separate agent services realise the role capabilities; their combined performance supports the organisational outcome without granting each service the authority of every contributing role.
\nRoles should also be sufficiently specific to support ownership. An organisation can assign a person or agent multiple roles, but each action should use an explicit active role and bounded assignment. Combining role knowledge for a discussion must not silently combine execution privileges.
\n| Role | Dedicated conceptual view | Principal service | Example boundary |
|---|---|---|---|
| Product Manager | Outcomes, user needs, features, value hypotheses and acceptance criteria | Produce an evidence-linked feature breakdown | Cannot approve production deployment |
| Project Manager | Scope, dependencies, resources, risks and delivery milestones | Produce a delivery plan from an approved template | Can write a project draft; cannot deploy it |
| Release Manager | Release readiness, change records, environments and rollback procedures | Coordinate an authorised release | Production action requires a separate assignment and satisfied release policy |
These are example contracts, not universal job descriptions. The organisation owns the actual definitions and decides where responsibilities overlap.
\nEach role receives a governed projection of organisational knowledge. This can select which concepts and relationships are available to the role, as well as how retrieval ranks them. Roles can share an underlying graph while retaining different conceptual views.
\nShared concepts retain shared identifiers. For example, a feature’s acceptance criteria connect product intent with delivery planning and release verification. Role-specific relationships express what matters for each responsibility: a Product Manager connects a criterion to user value; a Project Manager connects it to delivery evidence and dependencies; a Release Manager connects it to a release decision. The shared identity supports handoffs without making the roles interchangeable.
\nRole-owned overlays can define local vocabulary, specialised relationships, approved templates and checklist requirements. Their provenance should identify the owner, source, revision and approval state. Changes to the shared definition of “accepted” require different governance from adding a convenient synonym to a role’s retrieval vocabulary.
\nGraph projections must also respect access controls. Knowing that another team’s confidential plan exists does not authorise retrieving its contents. A role graph is a semantic view; a separately authenticated assignment and resource policy determine what information the agent may actually read.
\nThe graph is the inspectable, or white-box, component of this architecture. A reviewer can examine its concepts and relationships, the embeddings derived from them, and the definitions that shaped retrieval. This visibility neither makes the language model transparent nor establishes that the graph's contents are correct.
\nISO/IEC/IEEE 42010:2022 concerns architecture descriptions, including viewpoints and model kinds. It distinguishes an architecture from its description and does not prescribe an agent implementation or execution policy. In that standard, ADF means architecture description framework; the ADF agent orchestration system discussed here is a different use of the acronym. ISO/IEC/IEEE 42010:2022
\nFor this architecture, stakeholder concerns, viewpoint conventions and resulting views must remain distinguishable from organisational roles. A viewpoint establishes conventions for constructing and interpreting views that address concerns. ISO 24641:2023 preview
\nA role can serve several stakeholder concerns and use several viewpoints. A role graph becomes useful architectural material when its concepts, relationships, conventions and intended questions are explicit. It is not automatically a viewpoint; a project plan is not automatically an architecture view. Terraph's language constructs likewise do not establish that it is a conforming architecture description language. The standards do not assign the paper's Project Manager, Product Manager or Release Manager jobs their permissions.
\nISO/IEC/IEEE 29148:2018 addresses requirements engineering and its information across the life cycle. Its terminology distinguishes checks on requirements from checks on the realised product or system. The following four questions must not be collapsed into a single “validation passed” label. ISO/IEC/IEEE 29148:2018 official preview
\n| Activity | Question and object |
|---|---|
| Requirements verification | Are the requirement statements and set well formed? |
| Requirements validation | Do those requirements express the intended stakeholder needs? |
| Product or system verification | Does the realised item fulfil its specified requirements? |
| Product or system validation | Does it support its intended application and use? |
Each decision needs the applicable activity and evidence; an individual tool check need not perform all four. A reviewed input baseline is particularly important: an output checklist cannot retrospectively establish what the stakeholder meant.
\nThe following crosswalk is this paper's proposed design, not an ISO clause-compliance matrix:
\n| Standards-related distinction | Application to the proposed agent contract |
|---|---|
| Stakeholder concern and viewpoint | Name the decision owner, concern and representation conventions; bind the agent's organisational assignment separately. |
| View and modelling conventions | Specify graph concept and relationship types, interpretation rules and correspondences between role views. |
| Requirement and traceability | Connect each obligation to its need, responsible role, input baseline, criteria, output and evidence. |
| Verification and validation | Name the checked object, the predicate or intended-use scenario, its applicable context and the decision supported. |
This is a bounded cross-check using official public material. It is neither certification nor a full-text normative clause audit. The richer semantic contract below is the paper's engineering proposal, not a claim that either standard mandates knowledge graphs or this permission scheme.
\nA check is meaningful only if its criteria apply to the relevant meaning and context of its input. “Release” can mean software deployment or approval of a payment; the same token need not denote the same concept. A plan for staging cannot be validated against production assumptions merely because both mention “release”. A criterion about elapsed hours cannot silently become one about business days. Evaluation requires compatible conceptual spaces, not identical wording or complete identity between two role graphs.
\nThe proposed trace is:
\n\n\nRequirement ID and revision → stakeholder need → responsible role and relevant viewpoints → input concepts, propositions and context baseline → verification criterion and validation scenario → evidence → decision.
\n
The baseline identifies the entity and project, source and requirement revisions, environment, relevant time and calendar, units and thresholds, assumptions, exceptions and prohibited actions. A proposition preserves who must do what to which object, under which conditions, with its polarity and quantifiers. “Every accepted feature” and “some features” can share concepts while imposing different obligations.
\nVerification and validation may use different abstractions. A product view can describe a customer outcome while a delivery view describes work packages. Their mapping must explain how the relevant obligation survives decomposition and which evidence supports it. Approved unit conversions and concept mappings may establish compatibility; fuzzy similarity alone cannot. Additional derived criteria can be legitimate when their rationale and relationship to the stakeholder need are recorded. An obligation cannot disappear merely because a role projection omits its vocabulary.
\nThis paper proposes the following contract. These SHALL statements are authored requirements for the proposed integration, not quotations from ISO:
\nContext compatibility is necessary but insufficient. A perfectly aligned test can still have a weak predicate, unreliable evidence or inadequate coverage. Criteria must themselves be verified and validated for their purpose; their evaluation must establish the promised claim. Checks should report only what their evidence supports.
\nThe architecture needs four cooperating parts: identity and assignment, semantic knowledge, mediated execution, and audit evidence.
\n
Figure 1. Connecting meaning with authority. Open the diagram source in Mermaid Live.
\nRole-based access control provides a foundation: subjects are assigned authorised roles, and operations are mediated through the permissions of those roles. A session can activate a subset of the roles available to a subject. NIST RBAC
\nThe proposed architecture adds organisational meaning to this assignment and applies contextual conditions to execution. Project identity, resource scope, environment, approval state and expiry can all matter. This is consistent with attribute-based access control, which evaluates attributes of the subject, object, operation and environment against policy. NIST SP 800-162
\nLet a request be q = (principal, role, service, tool, action, resource, arguments, context). A simplified decision rule is:
permit(q) =\n authenticated(principal)\n AND active_assignment(principal, role, scope, expiry)\n AND capability_service_binding(role, service)\n AND registered_service_tool_action(service, tool, action)\n AND resource_and_argument_scope_match(q)\n AND required_evidence_valid_for_exact_request(q)\n AND contextual_policy_allows(q)\n AND NOT explicit_deny(q)An absent or indeterminate condition produces no permit. This is a proposed contract, not a description of a policy engine already supplied by Terraphim.
\nThe gateway must mediate actual effects. Hiding a deployment tool in a user interface is insufficient if the same principal can deploy through a shell, network client or overprivileged service credential. The action boundary includes alternative routes, delegated agents and mutable resource references. A delegate receives only an authorised subset of the parent’s scope; it cannot turn a planning request into a production privilege.
\nTerraphim’s graph approach supplies a concrete semantic mechanism for the role view. Concepts and surface forms are represented explicitly, and graph relationships support retrieval and explanation. Its documented graph retrieval path uses normalised concepts, synonym matching and document co-occurrence structure, rather than requiring a neural vector embedding model. Terraphim graph embeddings
\nThe inspected general RoleGraph implementation uses Aho–Corasick matching to resolve surface forms to concept identifiers. Its document indexer creates edges between consecutive recognised concepts in the indexed text and retains supporting document references. Retrieval then accumulates node, edge and document rank. Matched edges and tags make the retrieval basis inspectable. A high rank indicates retrieval support under that graph, not factual certainty or authority.
For organisational planning, this allows terms such as “delivery roadmap” and “project plan” to resolve according to the organisation’s chosen vocabulary. The interpretation is role-dependent: a delivery roadmap may emphasise dependencies and milestones for a Project Manager, while a product roadmap emphasises outcomes and prioritisation.
\nThe organisation should keep equivalence and relationship separate. A synonym maps a surface form to a concept. A relationship links distinct concepts. “Risk log” may be an agreed synonym for “risk register”, but a register and an individual risk remain distinct concepts. “Risk” and “deployment permission” must not become equivalent merely because they occur together in operational documents.
\nGraph embeddings make the retrieval context reproducible. A result can identify the recognised concept and supporting relationship evidence, explaining why a planning template or dependency policy was selected. Template currency, the truth of retrieved claims and action authority require separate checks against authoritative metadata, evidence and policy.
\nThe existing concept vocabulary and the associations found in text play different parts in constructing the role's conceptual space. The role's thesaurus supplies the concepts that can be recognised and the synonyms that resolve to them. Indexing identifies occurrences of those concepts in notes and documents, then constructs connections from the matching text. Retrieval uses that structure to select relevant material. In this path, a connection is discovered between recognised concepts; it does not establish a new concept definition or a new organisational rule.
\nSentence-level co-occurrence provides an intuitive starting assumption: concepts mentioned together may be related. The inspected implementation has a broader boundary. The document indexer takes the matching concept sequence from the indexed document and connects adjacent matches without invoking the available sentence-splitting helper. Associations can therefore cross sentence boundaries. A sentence-bounded association rule would require an explicit indexing change; it is not a current guarantee.
\nEven within one sentence, proximity leaves the relationship unspecified. A note can mention a requirement and a test because the test satisfies it, fails it, or has not yet been performed. The association graph cannot decide which statement holds. The proposed semantic layer must establish the connection's meaning, direction and conditions before admitting it as a typed organisational claim.
\n| Layer | Representation | What it establishes |
|---|---|---|
| Role vocabulary | Canonical concepts and approved synonyms | Which concept a recognised expression denotes within the role's domain. |
| Text association | Connections between matched concepts, with document evidence | Where concepts occur together or in the indexed matching sequence. |
| Organisational interpretation | Typed, contextual propositions governed by role templates | Whether a connection expresses a dependency, responsibility, requirement, evidence claim or other admitted meaning. |
| Evaluation and execution | Applicable criteria, evidence checks and permission decisions | Whether the specific claim is supported and whether the requested action is authorised. |
The organisational semantic layer supplies the relationship vocabulary. A role selects an applicable profile of that vocabulary, together with the templates for its work products. A Project Manager's delivery-plan template may admit dependencies between milestones and assignments of responsibility. A Product Manager's feature template may admit derivation from stakeholder needs and links to acceptance criteria. Shared concepts retain their identities across these views; the role determines which governed relationships are relevant, rather than inventing their meanings during each interaction.
\nThe proposed governance rule is that every admitted organisational relationship type SHALL be defined in a versioned, approved template or semantic profile, with traceability to the applicable adopted standards or an explicitly approved organisational extension. The profile records the standard edition and relevant requirement or definition where applicable. Where a standard leaves a modelling choice open, the organisation records its extension and rationale. This preserves standards-based discipline without asserting that industry standards already name every graph predicate used by an organisation. ISO/IEC/IEEE 42010:2022 and ISO/IEC/IEEE 29148:2018 inform the distinctions discussed earlier; the illustrative predicate names here are not standard-prescribed vocabulary.
\nA relation definition needs more than a label. It identifies the permitted source and target concept types, direction, interpretation, applicable role and work product, and required qualifiers such as environment, time, polarity or evidence state. For example, verifies might connect a verification activity to a requirement, while producedEvidence connects its execution to a result. A planned test and a completed test cannot be substituted merely because both connect to the same requirement.
Consider the sentence: “The Release Manager must approve production deployment after verification.” An association graph can connect the recognised concepts Release Manager, production deployment and verification. In a governed release template, those associations could support the candidate propositions deployment requiresApprovalFrom ReleaseManager and deployment requiresEvidence verificationResult. These describe obligations; they do not record that approval was given or that verification passed. An observed approval requires separate evidence identifying its issuer, subject, decision and applicable version. The runtime must still establish that the issuer and acting agent have the necessary authority.
Typing a connection is therefore a controlled interpretation step. A candidate keeps its source text and concept matches; the role template supplies permissible predicates and structural constraints; a check establishes whether the text and independent evidence support the proposed proposition. An extraction model can propose a type, but a plausible label is not sufficient to admit the claim as organisational knowledge. Ambiguous, unsupported or conflicting propositions remain unresolved. The association evidence is retained so that a reviewer can reconstruct how the candidate arose.
\nVerification and validation must compare typed propositions and their qualifiers across the input, output and evaluation criteria, using approved mappings where views differ. A test that finds every concept but converts requiresApprovalFrom into approvedBy, or removes a prohibition, has changed the claim. The audit must expose that change. Graph coverage can find omitted concepts; checking typed relationships addresses whether the connections preserve the intended meaning.
The following command is an illustrative invocation using the installed terraphim-grep interface. The paths represent a proposed organisational knowledge layout; they are not supplied sample data or a claim that this particular corpus was executed.
terraphim-grep "delivery plan" \\\n --role "Project Manager" \\\n --haystack all \\\n --paths ./organisation/templates/planning \\\n --thesaurus ./organisation/roles/project-manager/thesaurus.json \\\n --search-only -n 8 -C 2 --jsonThe dedicated thesaurus makes the role’s vocabulary concrete. --search-only keeps this invocation on its search path without LLM synthesis. The implementation can fall back to thesaurus matching when populated graph evidence is absent; a result alone does not demonstrate graph traversal. Search produces candidates; the service then checks each candidate against the authoritative template registry. The chosen template must be approved, applicable to the project type, available to the active role and identified by an immutable revision or content digest. Restricting search paths narrows retrieval; it does not establish an execution sandbox.
This distinction matters when a semantically excellent match is obsolete. The organisation’s semantic layer may help locate it, but the planning service must select an eligible template. If none exists, the service should request a governed template decision rather than manufacture an “approved” template itself.
\nA checklist makes an expected output contract explicit. Terraphim’s published checklist approach places enumerated concepts outside the model and uses the same graph machinery for recognition and retrieval. Graph embeddings for checklists
\nThe current source also supports a paired input/output graph-coverage mode. An illustrative planning checklist, saved as .terraphim/checklists/project_plan.md, is:
# Project plan\n\nmode:: graph-coverage-v1\nchecklist:: scope, dependencies, risk register, acceptance criteria\n\n### risk register\n\nsynonyms:: risk logIts concepts and aliases must resolve uniquely in the configured role graph; declaring an alias in a checklist does not create the missing graph concept. The checklist is not a substitute for building and governing that graph. A source-level invocation for the inspected implementation is:
\nterraphim-agent validate \\\n --role "Project Manager" \\\n --checklist project_plan \\\n --input-file brief.md \\\n --json < plan.mdThis --input-file form is supported by the inspected client source, not by the locally installed 1.21.22 command interface. The example is therefore a source-supported integration example, not an installed-binary demonstration.
The paired check requires recognised input concepts and checklist concepts to appear in the output, with supporting stored graph paths. Its report distinguishes pass, fail and indeterminate. A service using it as a gate must inspect that verdict; successful CLI completion alone does not establish a passing checklist.
Concept coverage is one limited checking predicate. A plan can mention “dependencies” while assigning them incorrectly, or include “acceptance criteria” that do not test the promised outcome. The complete planning contract consequently layers structural checks, graph concept coverage and evidence review. The graph checker supplies an inspectable signal within that contract. It does not establish the full semantic compatibility contract, verify every asserted relationship or confer permission to publish or execute the plan. Its lexical negation handling does not constitute general verification of proposition meaning.
\nConsider a project to deliver a customer-facing feature. The Product Manager agent receives an authorised assignment to prepare a feature breakdown. Its graph connects the product outcome to needs, features, value assumptions and acceptance criteria. It produces a versioned breakdown and hands that artefact to the Project Manager service.
\nThe Project Manager agent authenticates under its own scoped assignment. It retrieves candidate planning templates through its role thesaurus, selects an approved revision, and creates a draft connecting the agreed features to milestones, dependencies, owners and risks. It preserves the incoming acceptance criteria and identifies unresolved assumptions.
\nThe service checks the draft’s structure and invokes the graph checklist. A missing recognised requirement produces a failed or indeterminate result that must be resolved before the relevant transition. A passing result is recorded alongside its limitations. Further review establishes whether the proposed dependencies and acceptance evidence are adequate.
\nWhen the service requests create_draft on the assigned project’s document store, the gateway evaluates the exact operation, destination and content digest. Assuming the role assignment, template, checks and policy conditions are valid, it permits that write and records its outcome.
Now suppose the agent requests deploy against production. The request fails the role-to-service/action binding and the production policy. Mentioning deployment in the plan, retrieving a release runbook, or passing the planning checklist cannot repair that missing authority. The gateway records a denial before an effect occurs.
The agent may instead hand the approved plan to the release management process. That handoff supplies evidence, not inherited production credentials. A separately authorised release principal must satisfy its own contract. The organisation can therefore automate preparation without collapsing preparation, approval and execution into a single unrestricted agent.
\nConsider an illustrative requirement R-17 revision 3: the plan must specify a staging rollback rehearsal whose planned start is no earlier than the simulated-failure trigger and no later than eight elapsed hours afterwards. The baseline defines that trigger as the rehearsal protocol's injected-failure timestamp. Production deployment is outside the Project Manager's scope. The stakeholder need is to coordinate a usable rehearsal without delegating production execution.
\nA verification criterion checks the plan's environment, planned start offset and preserved prohibition against that baseline. A validation scenario asks the delivery and release stakeholders whether the proposed handoff enables the intended rehearsal with the available people and resources. The plan's check does not prove that a rehearsal occurred. Later testing of the delivered system requires its own evidence; a good plan does not establish successful system behaviour.
\n| Candidate interpretation or evidence | Required treatment under the proposed contract |
|---|---|
| “One business day” replaces eight elapsed hours | No silent equivalence. Resolve the calendar and conversion; fail if the interval changes, otherwise retain the approved mapping. |
| The plan schedules the rehearsal in production | Fail the known environment mismatch; separately deny unauthorised production actions. |
| The evaluator uses revision 2's 24-hour threshold | Reject the stale criterion binding and reassess against revision 3. |
| “Must not deploy” becomes “may deploy” | Fail proposition preservation despite matching deployment concepts. |
| Relevant calendar or requirement revision is unknown | Return indeterminate until the baseline is resolved. |
The same protection applies when views use different language. “Pre-production” can map to the named staging environment if the organisation explicitly defines that equivalence. Matching the spelling “staging” in an unrelated project's evidence cannot establish that the requirement was met.
\nThis Terraph example connects a role graph, a checklist and three verbs: audit, check and deploy. Here, engineer is the example role; it does not change the Project Manager’s authority. The graph and checklist are Markdown inputs compiled into constant matching automata. embed and satisfies are language intrinsics, so retrieval and checking use the same concept definitions.
import std;\n\nrole engineer from "kg";\nchecklist preflight from "checklists/preflight.md";\n\n// Audit concept coverage and report the matching source spans.\nfn audit(path: str) -> i64 {\n let src = read_file(path);\n if src.len == 0 {\n print("cannot read ", path);\n return 1;\n }\n let h = embed(src);\n let cov = coverage(h);\n print("file: ", path);\n print("kg_hits: ", cov.kg_hits, " concepts: ", cov.concepts,\n " confidence: ", cov.confidence, " diversity: ", cov.diversity);\n for m in h {\n print(" ", m.concept, " @", m.start, "..", m.end);\n }\n if cov.sufficient() {\n print("sufficient: yes");\n return 0;\n }\n print("sufficient: no");\n return 1;\n}\n\n// Diagnose missing checklist items. This verb does not issue a proof.\nfn check(text: str) -> i64 {\n let miss = text satisfies preflight;\n if miss.any() {\n print("checklist preflight: incomplete");\n miss.report();\n return 4;\n }\n print("checklist preflight: complete");\n return 0;\n}\n\n// The generated wrapper supplies a sealed proof for the checked snapshot.\nfn deploy(plan: str, ev: Proof<preflight>) -> i64 {\n print("deploy: preflight evidence present, proceeding");\n return 0;\n}audit exposes which concepts matched and where. check reports missing checklist items. At the gated entry point, the generated verb wrapper obtains a sealed Proof<preflight> through .require(), which succeeds only when no items are missing. The function signature makes that evidence a prerequisite for entering deploy.
The proof carries the checked snapshot as ev.subject. Any operation justified by the gate must consume that snapshot; reopening the path in plan could read changed, unchecked bytes. The sample body only prints a message. Actual deployment still requires an authorised role assignment and resource policy, and concept coverage alone does not establish that the plan is correct. The example shows the language boundary; it is not a recorded deployment.
The ADF agent orchestration configurations assign organisational responsibilities. The checked-in Product Owner template names an implementation and declares these fields. Product Owner is the configured role; it is distinct from the illustrative Product Manager role used earlier:
\n[[agents]]\nname = "product-owner"\npersona = "Themis"\ncapabilities = [\n "planning",\n "roadmap",\n "issue-creation",\n "prioritisation",\n]\nproject = "terraphim"Its skill chain includes research, design, specification, architecture, business scenario design, requirements traceability and acceptance testing. The task defines the work product: each proposed issue must identify its organisational goal, describe the problem and proposed solution, and supply Given–When–Then acceptance criteria. The template also requires a priority estimate, a communication hint and the suggested delivery agent. The workflow narrows the backlog, scores the surviving items and proposes up to two new issues.
\nThese selected literal fields show the existing issue template. Its brackets indicate fields the Product Owner must fill; WIG means the organisation’s current Wildly Important Goal.
\nTitle: [concise, imperative]\nWIG: [which WIG this serves, or skip if none]\nProblem: [2-3 sentences describing the problem]\nProposed solution: [bullet points, no hand-waving]\nAcceptance criteria (mini-UAT):\n Given [precondition]\n When [action]\n Then [observable outcome]The configuration connects the Product Owner’s planning capability to a defined work product and template. It records operational intent; this inspection does not establish that the agent is deployed or that every configured step succeeds.
\nThe repo-local ADF configuration separately declares a Developer, Reviewer and Merge Coordinator, with distinct capability lists and named Terraphim roles:
\n| Configured agent | Declared Terraphim role | Responsibility expressed in the configuration |
|---|---|---|
developer | Terraphim AI Developer | Implementation, Rust development and test-driven development |
reviewer | Terraphim AI Reviewer | Code review, quality gates and review coordination |
merge-coordinator | Terraphim AI Merge Coordinator | Merge coordination, CI/CD and conflict resolution |
The orchestrator’s merge path checks four required status contexts: native-ci / build (push), adf/pr-reviewer, adf/validation and adf/verification. It also requires canonical gate-result records for the current pull request head. Blocking findings and the project's treatment of concerns affect the decision. Evidence for an older commit cannot substitute for the current head’s review, binding the decision to that part of its context baseline. The labels adf/validation and adf/verification do not establish that their producers perform the corresponding ISO requirements or system activities.
Configuration intent and runtime enforcement must remain distinguishable. The inspected repo-local loader sets terraphim_role to None; the declaration alone does not prove that an agent receives the named role graph. Instructions such as “do not merge” are task text, and the Product Owner’s broad shell access is not a production-deployment barrier. The merge gate controls the orchestrator’s merge path; it does not prove that every alternative credential or tool path is mediated. Its structured records are not independently authenticated merely by having the expected fields.
ADF declares responsibility, capabilities, workflow and expected artefacts. Terraph implements bounded tool access, checks and receipts; Terraphim provides the role vocabulary and graph context. Connecting them still requires authenticated role assignments, capability-to-service identifiers, approved artefact versions and policy decisions tied to exact requests.
\nW3C PROV provides terms for recording entities, activities and agents, including derivation, association and delegated responsibility. These relationships can describe an artefact’s origin and an agent’s participation; a provenance statement alone is not an authorisation decision. W3C PROV-O
\nThe proposal can be evaluated without assuming that more graph coverage means safer behaviour. A test corpus should pair legitimate planning tasks with boundary cases and preserve the same role assignments and policy across agent implementations.
\n| Test | Expected observable result |
|---|---|
| Valid plan from an approved template | Draft created in the assigned project with complete evidence links |
| Semantically relevant but withdrawn template | Candidate rejected by template eligibility policy |
| Required concept omitted | Graph report identifies a fail or indeterminate result; gated transition does not proceed |
| All terms present but a dependency is wrong | Concept coverage may pass; substantive review detects the error |
| Project Manager requests production deployment | Denied before any deployment effect |
| Deployment attempted through an alternative tool | Same authority boundary enforced |
| Artefact changes after checking | Previous evidence rejected for the changed request or bytes |
| Role assignment expires before execution | Gateway denies or requires a fresh valid assignment |
| Agent implementation is replaced | Service contract and authority boundaries remain unchanged |
| Concepts match but polarity, environment or units differ | Relevant semantic predicate fails, or unresolved meaning is indeterminate |
| A role projection drops a critical obligation | Trace check rejects the transformation or requires an approved change |
| Different role views have an approved meaning-preserving mapping | Compatibility can be established; substantive criteria still require evidence |
Measurements should include accepted legitimate tasks, incorrect denials, unauthorised effects, trace completeness and the cost of reconstructing a decision. Report semantic-check accuracy separately from policy enforcement. A test that finds no bypass in a bounded suite is evidence about that suite, not proof that all possible tool paths are mediated.
\nTerraphim already supplies relevant semantic components, and Terraph supplies concrete prototype mechanisms for compiled checks and evidence. ADF provides configured organisational responsibilities and an implemented orchestrator merge gate. The end-to-end organisational contract, contextual requirement-to-criterion alignment, authenticated role binding, complete execution mediation and linked audit schema remain the integration proposed in this paper. No production deployment result or benchmark is claimed here.
\nRole owners must maintain concepts, templates and policies, and reviewers must understand what each checklist establishes. Graph updates need controlled promotion because a vocabulary change can alter which obligations appear satisfied. Audit records should retain enough versioned evidence to reconstruct decisions while limiting sensitive content.
\nAn agent’s organisational role provides a durable basis for knowledge, responsibility and delegated authority. Capabilities explain the outcomes the role must support; services describe how an agent implementation supplies them; tools expose actions; contextual policy determines which actions may occur. A role-based knowledge graph makes the concepts and relationships behind the work inspectable. Meaningful verification and validation also require those concepts, their propositions and the operating context to align with the applicable criteria through explicit, traceable mappings. A shared vocabulary alone cannot preserve an obligation.
\nTerraphim supplies explicit concept matching, co-occurrence associations, graph retrieval and graph-based checklists. The proposed next layer types those associations using governed role templates and standards-traceable relationship definitions, distinguishing a mention from an obligation and an obligation from an evidenced event. Terraph illustrates how compiled checks can produce evidence tied to the subject being checked. Connecting these mechanisms to authenticated role assignments and mediated execution yields a traceable organisational contract.
\nFor a Project Manager agent, the contract requires an approved planning template, preserved requirements, a checked plan and operation within the assigned project scope. Production deployment belongs to a separately authorised role and process. The evidence must allow a reviewer to reconstruct both the planning result and the authority decision.
\nPublic sources are linked beside the claims they support. The preceding article by David Hunt and Alexander Mikhalev was published on 31 October 2026 and read on 4 November. The continuation and concept mapping are this paper's synthesis.
\nTerraphim and ADF implementation details were checked against local source on 3–4 November 2026. The Terraph example above was supplied by Alexander Mikhalev; its comments have been shortened and its function bodies retained. Source revisions, implementation paths, recorded fixture results and inspection limits remain in the internal research notes.
\nThe standards cross-check used official catalogue records and accessible previews. The complete normative text of ISO/IEC/IEEE 42010:2022 was not inspected, so no clause-by-clause conformity claim is made. The role contracts, typed relationship profiles, contextual criteria and cross-system bindings are proposed integration work. The examples do not demonstrate a live production deployment.
\n", "https://reference-architecture.ai/properties/date": "2026-11-07", "https://atomicdata.dev/properties/tags": {"categories":["Architecture"],"tags":["ai-agents","governance","knowledge-graphs","terraphim","verification"]}, "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"] },{"localId": "https://reference-architecture.ai/posts/from-ai-risk-to-accountable-agents/", "https://atomicdata.dev/properties/name": "From AI Risk to Accountable Agents", "https://reference-architecture.ai/properties/url": "https://reference-architecture.ai/posts/from-ai-risk-to-accountable-agents/", "https://atomicdata.dev/properties/description": "In AI Risk is evolving, every day., David Hunt and I examined how an AI agent can pursue a legitimate objective through methods outside its mandate. Oversight needs to connect organisational intent, knowledge, authority, actions and consequences.
\n\n
This article summarises the full paper. To supervise that agent, we need to specify its responsibilities, the concepts it uses, the evidence its work requires and the actions it may take.
\n“Planning agent” tells us what an agent does, but leaves its responsibility unclear.
\nA Product Manager breaks outcomes into features and acceptance criteria. A Project Manager turns agreed scope into a delivery plan with dependencies, owners and milestones. Both perform planning, but their work products and authority differ.
\nAn agent's planning capability should belong to a defined organisational role. Its implementation provides services that use tools to perform specific actions.
\nThe chain becomes:
\nOrganisation → role → capability → service → tool → action → contextual permission.
\nA Project Manager agent might select an approved planning template and create a draft in its assigned project. That assignment does not authorise production deployment. The permission decision must govern the actual tool operation, including alternative routes through shells or delegated agents.
\nEach role needs a governed view of organisational knowledge. A Product Manager works with user needs, value hypotheses and acceptance criteria. A Project Manager needs dependencies, resources, risks and delivery commitments. Shared concepts connect their work without making the roles interchangeable.
\n![\"Role-based](\"/diagrams/accountable-agents-01-role-and-authority.png\")
Figure 1. Proposed role contract: the knowledge needed to plan and the authority to act are explicit.
\nTerraphim's role-based knowledge graphs make this vocabulary explicit through concepts and synonyms. The inspected indexer connects consecutive recognised concepts in document text, retaining supporting document references. Those associations help retrieve relevant notes and explain why they were selected.
\nThe next layer is to type the relationships. A connection might mean “depends on”, “addresses a requirement” or “requires approval”. The role selects the applicable relationship profile from approved templates, traced to adopted standards or documented organisational extensions.
\nA sentence mentioning a release, a manager and approval does not establish that approval occurred. “Requires approval” and “was approved” are different claims.
\nA checklist can recognise every expected concept while missing a change in meaning.
\nSuppose a delivery plan must place a rollback rehearsal in staging, with its planned start within eight elapsed hours of a defined failure trigger. An output can contain “rollback”, “rehearsal” and “hours” while moving the activity to production or replacing elapsed hours with business days.
\nSuccessful checking therefore requires compatible concepts, relationships and context across the input, output and criteria. Different terminology can be acceptable through an explicit mapping. Changed scope, units, conditions or prohibitions cannot pass silently.
\nOur paper uses ISO/IEC/IEEE 42010:2022 to frame architecture descriptions and ISO/IEC/IEEE 29148:2018 to distinguish requirements quality and stakeholder intent from evidence about the realised system. This is a standards-informed design, not a conformity claim.
\nThe practical requirement is traceability: identify the requirement version, relevant context, applicable criterion and evidence behind each result. When those conditions change, reassess the affected findings. Unknown context should produce an indeterminate result, not an assumed pass.
\n![\"Contextual](\"/diagrams/accountable-agents-02-contextual-verification.png\")
Figure 2. Proposed contextual check: passing a planning criterion neither proves the rehearsal occurred nor grants deployment authority.
\nTerraph, the Terraphim language, connects a role graph, a checklist and a gated action in one program. The graph and checklist compile from Markdown into constant matching automata. embed identifies concepts and source spans; text satisfies preflight reports which checklist items are missing.
The deploy verb declares the evidence it needs. Here, engineer is the example role; it does not change the Project Manager’s authority:
import std;\n\nrole engineer from "kg";\nchecklist preflight from "checklists/preflight.md";\n\nfn deploy(plan: str, ev: Proof<preflight>) -> i64 {\n print("deploy: preflight evidence present, proceeding");\n return 0;\n}The generated verb wrapper obtains a sealed Proof<preflight> through .require() only when no checklist items are missing. The proof binds to the checked snapshot, available as ev.subject.
![\"Terraph](\"/diagrams/accountable-agents-03-terraph-audit.png\")
Figure 3. The Terraph proof gate. The example body prints a message; it does not deploy a system.
\nAny actual operation must use that checked snapshot and pass a separate authority check. Reopening the plan could read changed bytes. Passing the checklist cannot give a Project Manager production privileges, nor can concept coverage establish that the plan is correct.
\nOur earlier argument calls for supervision that follows an agent's changing context. An inspectable role contract gives that supervision a defined mandate to check. A Project Manager agent can prepare a plan, preserve its requirements and record the evidence, while release authority stays with the role responsible for exercising it.
\n", "https://reference-architecture.ai/properties/date": "2026-11-07", "https://atomicdata.dev/properties/tags": {"categories":["Architecture"],"tags":["ai-agents","governance","knowledge-graphs","terraphim","verification"]}, "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"] },{"localId": "https://reference-architecture.ai/posts/terraphim-engine-architecture/", "https://atomicdata.dev/properties/name": "The Terraphim Engine: Six Layers, Fifty-Two Crates", "https://reference-architecture.ai/properties/url": "https://reference-architecture.ai/posts/terraphim-engine-architecture/", "https://atomicdata.dev/properties/description": "The diagram on the front page of this site is not an illustration. Every box in it is a\npublished Rust crate you can cargo add today, read the documentation for, and use\nwithout taking the rest of the system. This article explains what sits behind each box,\nand why the decomposition is the interesting part.
\n
Most agent frameworks ship as a single dependency. You take the orchestrator, the memory\nlayer, the retrieval stack and the tool protocol together, or you take none of it. That\nis convenient at first and expensive later: when one layer misbehaves you cannot isolate\nit, and when you want to replace one you replace all of them.
\nThe pattern this site argues for is the opposite. Draw the boundaries first, make each\none independently useful, and let the seams stay visible. Terraphim is built that way —\n52 crates across six layers, each with a single responsibility — which makes it a\nreasonable worked example rather than a product pitch.
\nThe test of a boundary is whether you can use one side without the other. Here you can:\nterraphim_automata is a text-matching engine that happens to be used by a knowledge\ngraph; terraphim_persistence is a storage abstraction that does not know what it is\nstoring.
Terraphim organises its crates into a dependency graph with six levels. Lower layers do\nnot know the higher ones exist.
\n| Layer | Concern | Representative crates |
|---|---|---|
| 6 | User interfaces | terraphim_agent, terraphim-cli, terraphim_server |
| 5 | Orchestration | terraphim_orchestrator, terraphim_kg_orchestration, terraphim_symphony |
| 4 | Agent system | terraphim_spawner, terraphim_router, terraphim_agent_supervisor |
| 3 | Service layer | terraphim_service, terraphim_middleware, haystack_* |
| 2 | Core engine | terraphim_automata, terraphim_rolegraph, terraphim_persistence |
| 1 | Types and configuration | terraphim_types, terraphim_config, terraphim_settings |
That ordering matters more than the crate count. An agent system that cannot say which\nlayer a failure came from is a system you cannot debug in production.
\nterraphim_types is the shared vocabulary — the structures every other crate agrees on.\nterraphim_config holds role definitions, haystacks and LLM routing;\nterraphim_settings handles runtime preferences.
Putting roles in configuration rather than code is what makes the context boundary\nstructural. A role has access to certain haystacks; the knowledge graph is built from\nthose haystacks; therefore the agent cannot reach information outside its role. That is\nnot a permission check at runtime that could be bypassed — it is an absence. The data\nwas never loaded.
\nThis is the layer the front-page diagram mostly depicts.
\nterraphim_automata is the matching engine:\nAho-Corasick finite state automata that match thousands of patterns simultaneously in\nO(n) time, where n is the length of the text. Not \"usually fast\" — bounded, and the same\nbound every time. It compiles to WebAssembly, which is why the same matcher runs in a\nbrowser as on a server.
terraphim_rolegraph is the knowledge graph.\nIt is deliberately not a general-purpose graph database. It maps search roles to\ndomain-specific graph views, and it exists to do one thing quickly: turn text into\nconcepts, deterministically. Terraphim reports knowledge-graph inference in the 5-10\nnanosecond range and a 15 MB memory footprint, with no GPU — figures worth checking\nagainst your own workload, but the shape of the claim follows from the design rather\nthan from optimisation tricks.
terraphim_persistence provides the\nPersistable trait and DeviceStorage backends across memory, SQLite and redb. Storage is\nan interface here, not an assumption, which is what lets the same engine run on a laptop\nand at the edge.
terraphim_middleware searches haystacks —\npluggable data-source backends. This is the boundary the front-page diagram labels\nDocument Input. A haystack might be a local folder, a repository, or a connector to\nsomething else; the layers above do not change when you add one.
terraphim_service handles requests and responses\nfor the core, which keeps request handling out of the engine itself.
Above the service layer sit the agent system (terraphim_spawner, terraphim_router,\nterraphim_agent_supervisor) and orchestration (terraphim_orchestrator,\nterraphim_kg_orchestration, terraphim_symphony).
The distinction is worth holding onto. The agent system is concerned with individual\nagents — spawning them, routing to them, supervising them. Orchestration is concerned\nwith what happens between them. Collapsing those two into one \"agent framework\" is the\nusual mistake, and it is why so many systems cannot tell a stuck agent from a stuck\nworkflow.
\nThe MCP-Native Agent Architecture goes\ninto the protocol surface these layers expose, and why treating MCP as an architectural\nprimitive rather than a transport changes what the system can guarantee.
\nterraphim_agent is the interactive REPL with session\nsearch and learning capture. terraphim-cli is the\nautomation-shaped counterpart, with JSON output for scripting.\nterraphim_server provides the REST API.
Three interfaces over one engine, none of them privileged. That is the payoff of the\nlayering: the CLI is not a thin wrapper around the server, and the server is not a\nspecial case. They are peers over the same core.
\nYou do not need to adopt Terraphim to use the pattern. The transferable parts are:
\nEvery crate named above is published and documented:
\nterraphim_automata for matching,\nterraphim_rolegraph for the knowledge graph,\nor terraphim_persistence for storage.Installation is a single command, or cargo install terraphim_agent if you would prefer\nto start from the REPL.
A reference architecture for production AI systems, drawn from five years building Terraphim — a privacy-first AI assistant with ~50 Rust crates, MCP-native interfaces, and role-based multi-agent orchestration.
\nIn the spring of 2024, I watched a well-funded AI startup burn through $180,000 in cloud credits in six weeks. Their crime? They let an LLM-powered agent manage their Kubernetes cluster. The agent had root access, a vague prompt (\"optimize resource utilization\"), and no guardrails. It deleted a production namespace, scaled a stateful set to zero, and triggered a cascading failure that took down their primary revenue pipeline for four hours.
\nThe post-mortem was illuminating. The LLM wasn't \"malicious.\" It was doing exactly what LLMs do: generating plausible-sounding text based on pattern matching. The prompt \"optimize resource utilization\" is semantically close to \"remove unused resources.\" The agent found a namespace with low CPU utilization and removed it. Logical, if you squint. Catastrophic, if you're the on-call engineer.
\nThis is the production agent crisis in miniature: we've given probabilistic systems deterministic powers without deterministic boundaries.
\nThe current generation of agent frameworks — LangChain, CrewAI, AutoGPT, and their descendants — share a common architectural flaw: they treat the LLM as both the decision-maker AND the execution engine. The LLM decides what to do, how to do it, and when to stop. This is fine for demos. It's a liability for production.
\nConsider what production systems actually require:
\nNone of these are properties of LLMs. LLMs are probabilistic text generators. They hallucinate. They drift. They respond to temperature. They have no concept of \"side effects\" or \"resource limits\" or \"this action is irreversible.\"
\nThe solution is architectural, not algorithmic. We need a boundary layer between the probabilistic decision-maker (the LLM) and the deterministic execution environment (the system). That boundary layer is the Model Context Protocol (MCP), and treating it as an architectural primitive rather than a communication protocol changes everything.
\nMCP was introduced by Anthropic in late 2024 as a standardized protocol for connecting AI assistants to external data sources and tools. At its simplest, it defines how an AI client discovers capabilities, requests context, and invokes tools from a server.
\nBut MCP is more than a wire format. It is a contract layer that enforces capability boundaries. And when you treat it as an architectural primitive — a fundamental building block of your system, not just a communication detail — you get something powerful: the ability to build deterministic agent systems on top of probabilistic reasoning engines.
\nAn MCP server exposes three things:
\nThe key insight: the server defines WHAT is possible. The LLM decides WHICH possibility to pursue. The protocol enforces the boundary between them.
\nTerraphim's MCP server (terraphim_mcp_server) implements this boundary in ~200 lines of Rust. It exposes three tools: search (read-only, queries the knowledge graph), build_autocomplete_index (idempotent, rebuilds the Aho-Corasick automata), and update_config (destructive, modifies runtime configuration). Each tool has a schema. Each tool has a known effect. The LLM cannot invoke a tool it hasn't discovered, and it cannot pass arguments that don't match the schema.
This is the difference between \"I hope the LLM doesn't delete something\" and \"the system is physically incapable of deleting something without authorization.\"
\n![\"Four-Layer](\"/diagrams/four-layer-architecture.svg\")
\nThe MCP-native agent stack: four layers with the LLM as a probabilistic reasoning engine, bounded by deterministic architectural layers.
Based on production experience building Terraphim — a privacy-first AI assistant with ~50 Rust crates, MCP server integration, and multi-agent orchestration — I've distilled the following reference architecture. It separates concerns that current frameworks conflate and introduces verification at every boundary.
\nWhat the agent IS.
\nIn Terraphim, everything starts with a Role. A Role is not a prompt. It is a structured configuration that defines an agent's identity, knowledge sources, capabilities, and constraints. Think of it as a compile-time contract for an agent's behavior.
// From terraphim_config — the actual Role struct\n#[derive(Debug, Serialize, Deserialize, Clone)]\npub struct Role {\n pub name: RoleName,\n pub relevance_function: RelevanceFunction, // BM25, TitleScorer, etc.\n pub haystacks: Vec<Haystack>, // Data sources\n pub kg: Option<KnowledgeGraph>, // Linked knowledge graph\n pub llm_enabled: bool,\n pub llm_model: Option<String>, // e.g., "gemma3:270m"\n pub llm_router_enabled: bool, // 6-phase intelligent routing\n pub extra: AHashMap<String, Value>, // Extensible config\n}Key properties:
\nextra hashmap allows domain-specific configuration without schema changesThe Role is the \"type system\" of your agent. Just as Rust's borrow checker prevents data races at compile time, the Role configuration prevents capability leakage at configuration time.
\nWhat the agent KNOWS.
\nTerraphim's knowledge graph (terraphim_rolegraph) is not a general-purpose graph database. It is a specialized structure optimized for one thing: fast, deterministic text-to-concept matching using Aho-Corasick automata.
// From terraphim_rolegraph — the actual RoleGraph\npub struct RoleGraph {\n pub role: RoleName,\n nodes: AHashMap<u64, Node>,\n edges: AHashMap<u64, Edge>,\n documents: AHashMap<String, IndexedDocument>,\n pub thesaurus: Thesaurus,\n pub ac: AhoCorasick, // Compiled automata\n pub ac_reverse_nterm: AHashMap<u64, NormalizedTermValue>,\n}\n\nimpl RoleGraph {\n pub fn find_matching_node_ids(&self, text: &str) -> Vec<u64> {\n self.ac.find_iter(text)\n .map(|mat| self.aho_corasick_values[mat.pattern()])\n .collect()\n }\n}Key properties:
\nterraphim_automata compiles to WebAssembly for browser-based autocomplete.The Context Boundary emerges naturally from the Role configuration: if a Role only has access to certain haystacks, and the RoleGraph is built from those haystacks, the agent cannot access information outside its Role. This is not a runtime permission check. It is a structural impossibility.
\nWhat the agent DOES with others.
\nTerraphim's multi-agent system (terraphim_multi_agent) treats each Role as an autonomous agent. Agents discover each other through capability registration, communicate through structured messages, and coordinate through workflow patterns.
// From terraphim_multi_agent — the actual AgentRegistry\npub struct AgentRegistry {\n agents: Arc<RwLock<HashMap<AgentId, Arc<TerraphimAgent>>>>,\n capabilities: Arc<RwLock<HashMap<String, Vec<AgentId>>>>,\n role_agents: Arc<RwLock<HashMap<String, AgentId>>>,\n agent_load: Arc<RwLock<HashMap<AgentId, LoadMetrics>>>,\n}\n\nimpl AgentRegistry {\n pub async fn find_agents_by_capability(&self, capability: &str) -> Vec<AgentId> {\n let capabilities = self.capabilities.read().await;\n capabilities.get(capability).cloned().unwrap_or_default()\n }\n}Key properties:
\nThe Execution Harness is not a Wasm sandbox (though we have WASM support in terraphim_automata for browser contexts). It is the agent's own TerraphimAgent struct, which enforces resource limits through Rust's type system: token budgets are usize, timeouts are Duration, and context windows are bounded by configuration.
How the agent COMMUNICATES.
\nThe terraphim_mcp_server crate exposes Terraphim's capabilities through the Model Context Protocol. It acts as both server (exposing Terraphim tools to external clients) and client (discovering external tools for Terraphim agents).
// From terraphim_mcp_server — the actual McpService\n#[derive(Clone)]\npub struct McpService {\n config_state: Arc<ConfigState>,\n resource_mapper: Arc<TerraphimResourceMapper>,\n autocomplete_index: Arc<tokio::sync::RwLock<Option<AutocompleteIndex>>>,\n}\n\nimpl McpService {\n pub async fn search(\n &self,\n query: String,\n role: Option<String>,\n limit: Option<i32>,\n ) -> Result<CallToolResult, ErrorData> {\n // 1. Resolve role (tenant boundary)\n let role_name = if let Some(role_str) = role {\n RoleName::from(role_str)\n } else {\n self.config_state.get_selected_role().await\n };\n\n // 2. Query knowledge graph within role boundary\n let search_query = SearchQuery {\n search_term: NormalizedTermValue::from(query),\n role: Some(role_name),\n limit: limit.map(|l| l as usize),\n ..Default::default()\n };\n\n // 3. Return structured results\n match service.search(&search_query).await {\n Ok(documents) => { /* ... */ }\n Err(e) => { /* ... */ }\n }\n }\n}Key properties:
\nrole parameter in every tool call enforces tenant isolation at the protocol level.TerraphimMcpError maps to MCP ErrorData with structured error codes.tokio for concurrent request handling.The reference architecture is implemented in Rust. This is not aesthetic preference. It is an engineering requirement.
\nZero-cost abstractions: The Role configuration, knowledge graph, and agent registry introduce no runtime overhead. The type system enforces safety at compile time.
\nDeterministic resource usage: No garbage collection pauses. Memory is explicitly managed. The terraphim_automata Aho-Corasick matcher runs in bounded time and space.
Fearless concurrency: The AgentRegistry is accessed by multiple agents simultaneously. Rust's Arc<RwLock<_>> prevents data races without runtime overhead.
WASM portability: terraphim_types and terraphim_automata compile to WebAssembly, enabling browser-based autocomplete with TypeScript bindings.
![\"Traditional](\"/diagrams/traditional-vs-mcp.svg\")
\nTraditional agents conflate decision and execution. MCP-native agents enforce a hard boundary: the LLM decides, the architecture executes.
In Terraphim, multiple users share the same agent infrastructure but have completely isolated contexts:
\n// Each tenant gets their own Role configuration\nlet tenant_role = Role::new("tenant_acme")\n .with_haystacks(vec![acme_documents, acme_wiki])\n .with_llm_model("gemma3:270m")\n .with_relevance_function(RelevanceFunction::BM25);\n\n// The RoleGraph is built ONLY from tenant's haystacks\nlet role_graph = RoleGraph::new("tenant_acme".into(), tenant_thesaurus).await?;\n\n// MCP search automatically scopes to tenant's role\nlet results = mcp_service.search(query, Some("tenant_acme".to_string()), None).await?;Key insight: The boundary is per-tenant, but the automata engine and MCP server are shared. This gives you multi-tenant isolation without multi-tenant cost.
\nWhen a bug occurs in production, you need to reproduce it exactly. With probabilistic agents, this is nearly impossible. With MCP-native architecture, it's trivial:
\n![\"Deterministic](\"/diagrams/deterministic-replay.svg\")
\nRecording LLM responses (not regenerating them) makes replay deterministic. The same inputs produce the same execution trace.
// The RoleGraph + automata are deterministic\n// Same query + same thesaurus → same matches, every time\nlet matches = role_graph.find_matching_node_ids("rust programming");\n// Always returns the same node IDs for the same input\n\n// For LLM variability: record responses, don't regenerate\nlet recorded_response = load_recording(task_id)?;\nlet plan = agent.generate_plan_with_response(context, recorded_response)?;Key insight: The knowledge graph is deterministic. The LLM is not. Separate the two, and replay becomes deterministic where it matters.
\n![\"Approval](\"/diagrams/approval-flow.svg\")
\nDestructive operations require explicit approval. The system logs every request but does not execute without authorization.
// terraphim_mcp_server's update_config tool is inherently bounded\npub async fn update_config_tool(&self, config_str: String) -> Result<CallToolResult, ErrorData> {\n match serde_json::from_str::<Config>(&config_str) {\n Ok(new_config) => {\n // Config validation happens BEFORE mutation\n self.validate_config(&new_config)?;\n self.update_config(new_config).await?;\n Ok(CallToolResult::success(vec![Content::text(\n "Configuration updated".to_string()\n )]))\n }\n Err(e) => {\n Ok(CallToolResult::error(vec![Content::text(\n format!("Invalid configuration: {}", e)\n )]))\n }\n }\n}Key insight: The MCP schema IS the approval gate. The LLM cannot construct a valid Config JSON without knowing the schema, and invalid configs are rejected before mutation.
![\"Terraphim](\"/diagrams/terraphim-multi-agent.svg\")
\nTerraphim instances share an MCP protocol layer while maintaining isolated Role configurations and Knowledge Graphs. Each agent discovers tools through MCP without data leakage.
Terraphim implements this architecture across ~50 Rust crates. Here's how the layers map to real code:
\n| Layer | Crate | Responsibility |
|---|---|---|
| Role & Config | terraphim_config | Role definitions, haystacks, LLM routing |
| Knowledge Graph | terraphim_rolegraph | Aho-Corasick automata, concept matching |
| Fast Matching | terraphim_automata | FST autocomplete, link generation, WASM |
| Multi-Agent | terraphim_multi_agent | Agent registry, capability discovery, workflows |
| MCP Server | terraphim_mcp_server | Protocol surface, schema enforcement |
| Capabilities | terraphim_agent_registry | Capability matching, score-based discovery |
| Persistence | terraphim_persistence | DeviceStorage, memory/file backends |
The MCP integration is bidirectional. Terraphim acts as an MCP server (exposing search and config tools to external clients) and can discover external MCP servers for additional capabilities. This enables multi-agent orchestration where each agent has its own Role and Knowledge Graph, but they can discover and invoke each other's tools through MCP.
\nLet's be precise about what this architecture guarantees and what it doesn't.
\nWhat IS guaranteed:
\nWhat is NOT guaranteed:
\nThe key insight: We don't need the LLM to be deterministic. We need the KNOWLEDGE GRAPH and the PROTOCOL BOUNDARY to be deterministic. The LLM provides variable intelligence within fixed architectural bounds. The bounds are what matter for production.
\nNot every AI system needs four layers of deterministic boundaries. Here's when you DON'T need MCP-native architecture:
\nAdd MCP-native boundaries when:
\nBuilding MCP-native agents?
\nThe current generation of agent frameworks treats LLMs as the center of the universe. The LLM decides everything, and the framework provides convenient wrappers around API calls.
\nThis is backwards.
\nThe protocol is the architecture. The LLM is just one implementation detail.
\nA production agent system needs:
\nMCP provides the protocol surface. The reference architecture provides the safety properties. Rust provides the implementation guarantees.
\nThe LLM? It's just the reasoning engine. Powerful, probabilistic, and properly contained.
\nThe architecture described in this article is implemented in Terraphim, an open-source privacy-first AI assistant:
\nterraphim_mcp_server crateContributions welcome. Issues tracked in the Terraphim Gitea.
\nAlexander Mikhalev is CTO & Head of AI at Zestic AI, where he architects AI-native platforms with deterministic safety guarantees. He is the creator of Terraphim, an open-source privacy-first AI assistant built in Rust. His project \"The Pattern\" won the $10,000 Platinum Prize at the Redis \"Build on Redis\" Hackathon 2021 for ML-powered knowledge discovery. He previously led AI/ML architecture at Nationwide Building Society, where he co-authored the organization's first technology patent — a blockchain-inspired distributed system for resilient consensus. He speaks on AI architecture, Rust, and knowledge graphs.
\n", "https://reference-architecture.ai/properties/date": "2026-09-07", "https://atomicdata.dev/properties/tags": {"categories":["Architecture"],"tags":["mcp","agents","rust","knowledge-graphs","production","terraphim"]}, "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"] },{"localId": "https://reference-architecture.ai/posts/harness-engineering-agents/", "https://atomicdata.dev/properties/name": "Harness Engineering for Self-Improving Agents", "https://reference-architecture.ai/properties/url": "https://reference-architecture.ai/posts/harness-engineering-agents/", "https://atomicdata.dev/properties/description": "Many AI agent systems plateau after deployment. They execute tasks, but the surrounding system may not measure whether they are getting better, repeating old work, or normalizing stalled execution. This article proposes a reference pattern for agent harnesses: supervised feedback loops that make agent behavior observable, measurable, and easier to improve.
\nMost agent architectures follow a simple pattern:
\nUser Request -> Agent Reasoning -> Tool Execution -> ResponseThis works for single-turn tasks. But over hundreds or thousands of runs, the agent can make the same mistakes, retry the same failing strategies, and generate the same redundant outputs. It may have no mechanism to:
\nThese are plausible failure modes in long-running agent systems: repeated status updates, hand-waved health scores, accumulated \"lessons learned\" that never change system behavior, and stale priorities that remain nominally critical while no one acts on them.
\nThis is not necessarily a bug in the agent. It is often a missing architectural layer.
\nA harness is a feedback loop that monitors an agent's output, detects patterns, and triggers bounded corrective action. The proposed pattern uses four harness types, each addressing a specific failure mode:
\n| Harness | Failure Mode | Trigger | Action |
|---|---|---|---|
| Novelty Filter | Redundant documentation accumulation | Semantic similarity > threshold | Suppress redundant output |
| Calculated Health | Narrated metrics drift from reality | Daily evaluation window | Compute objective health score |
| Break-Glass | System paralysis without action | Threshold breach | Notify, escalate, or reduce authorized scope |
| Lesson Window | Lessons identified but never applied | Age > window or count > limit | Archive stale active lessons |
+-------------------------------------------------------------+\n| AGENT CORE |\n| +-----------+ +-----------+ +----------------+ |\n| | Reasoning | ---> | Execution | ---> | Output | |\n| +-----------+ +-----------+ +----------------+ |\n+-------------------------------------------------------------+\n |\n v\n+-------------------------------------------------------------+\n| HARNESS LAYER |\n| +----------------+ +----------------+ +-------------+ |\n| | Novelty Filter| | Calc. Health | | Break-Glass | |\n| | (dedup) | | (metrics) | | escalation | |\n| +----------------+ +----------------+ +-------------+ |\n| +-------------------------------------------------------+ |\n| | Lesson Window (time-bounded) | |\n| +-------------------------------------------------------+ |\n+-------------------------------------------------------------+\n |\n v\n+-------------------------------------------------------------+\n| FEEDBACK LOOP |\n| Metrics -> Review -> Approved Changes -> Policies |\n+-------------------------------------------------------------+Problem: The agent may produce repeated documentation or status text that restates known issues without adding decision value.
\nIllustrative implementation: This example is self-contained enough to show the state-management rule. In production, persist history outside the process and choose the embedding model, threshold, and retention period through evaluation.
from dataclasses import dataclass\nfrom datetime import datetime, timedelta, timezone\n\nfrom sentence_transformers import SentenceTransformer\nimport numpy as np\n\n@dataclass(frozen=True)\nclass HistoryEntry:\n content: str\n embedding: np.ndarray\n created_at: datetime\n\nclass NoveltyFilter:\n def __init__(self, threshold=0.82, window_days=7):\n self.model = SentenceTransformer("all-MiniLM-L6-v2")\n self.threshold = threshold\n self.window_days = window_days\n self.history: list[HistoryEntry] = []\n\n def is_novel(self, content: str) -> bool:\n """Return True only if content is sufficiently novel."""\n now = datetime.now(timezone.utc)\n self._evict_expired(now)\n new_embedding = self.model.encode(content)\n\n for entry in self.history:\n similarity = np.dot(new_embedding, entry.embedding) / (\n np.linalg.norm(new_embedding) * np.linalg.norm(entry.embedding)\n )\n if similarity > self.threshold:\n return False\n\n self.history.append(HistoryEntry(content, new_embedding, now))\n return True\n\n def _evict_expired(self, now: datetime) -> None:\n cutoff = now - timedelta(days=self.window_days)\n self.history = [\n entry for entry in self.history\n if entry.created_at >= cutoff\n ]Hypothesis: A correctly tuned novelty filter should reduce redundant persisted text while preserving new decisions, new evidence, and changed conclusions. A proposed target is a 30-50% reduction in persisted daily status text with manual review showing no loss of material decisions.
\nProblem: Health metrics are often narrated rather than calculated. A stable phrase such as \"roughly healthy\" can mask variation in the underlying jobs.
\nIllustrative implementation: The health calculator should receive concrete check functions. This avoids hidden dependencies on undefined methods and makes the metric auditable.
\nfrom dataclasses import dataclass\nfrom datetime import datetime, timedelta, timezone\nfrom pathlib import Path\nfrom typing import Callable\n\n@dataclass\nclass JobMetrics:\n name: str\n weight: float\n check_func: Callable[[int], float]\n\nclass HealthCalculator:\n def __init__(self, jobs: list[JobMetrics]):\n total_weight = sum(job.weight for job in jobs)\n if round(total_weight, 6) != 1.0:\n raise ValueError("job weights must sum to 1.0")\n self.jobs = jobs\n\n def calculate(self, days: int = 7) -> float:\n """Calculate health as weighted average of job success rates."""\n total = 0.0\n for job in self.jobs:\n success_rate = job.check_func(days)\n total += success_rate * job.weight\n return round(total * 100, 1)\n\ndef recent_output_check(directory: Path, min_bytes: int = 1024) -> Callable[[int], float]:\n """Return a check where success means one meaningful output per day."""\n def check(days: int) -> float:\n cutoff = datetime.now(timezone.utc) - timedelta(days=days)\n outputs = [\n path for path in directory.glob("*.md")\n if datetime.fromtimestamp(path.stat().st_mtime, timezone.utc) >= cutoff\n and path.stat().st_size >= min_bytes\n ]\n return min(len(outputs) / max(days, 1), 1.0)\n return check\n\ncalculator = HealthCalculator([\n JobMetrics("extraction", 0.40, recent_output_check(Path("runs/extraction"))),\n JobMetrics("briefing", 0.30, recent_output_check(Path("runs/briefing"))),\n JobMetrics("weekly_review", 0.30, recent_output_check(Path("runs/review"))),\n])Measurement definition: Health is the weighted average of job-specific success rates over a fixed evaluation window. Each job must define its own observable success condition before the run starts.
\nProblem: Some systems need a fail-safe escalation path when important work stalls or health drops below an agreed threshold. Break-glass rules should not silently grant broad autonomy to the agent.
\nCanonical configuration:
\n[break_glass]\nenabled = true\nmode = "supervised"\n\n[[break_glass.rules]]\nname = "commit_gap"\ncondition = "days_since_commit > 14"\nseverity = "critical"\naction = "notify_and_require_ack"\nmessage = "Critical commit gap: human acknowledgement required."\n\n[[break_glass.rules]]\nname = "health_critical"\ncondition = "calculated_health < 15"\nseverity = "critical"\naction = "reduce_authorized_scope"\nallowed_jobs = ["extraction", "health_check"]\nmessage = "Health critical: non-essential jobs are paused within pre-authorized limits."\n\n[[break_glass.rules]]\nname = "lesson_accumulation"\ncondition = "unapplied_lessons > 50 AND oldest_unapplied_days > 14"\nseverity = "warning"\naction = "notify_owner"Rule semantics:
\nnotify_and_require_ack:\n Send an escalation and block risky automatic changes.\n\nreduce_authorized_scope:\n Automatically disable only jobs already marked non-essential.\n\nnotify_owner:\n Create a visible warning with no automatic operational change.Abbreviated trigger evaluation: This sketch assumes Rule.evaluate is a safe expression evaluator over a constrained state object, not arbitrary code execution.
class BreakGlass:\n def evaluate(self, system_state: dict) -> list[Trigger]:\n triggers = []\n for rule in self.rules:\n if rule.evaluate(system_state):\n triggers.append(Trigger(\n rule=rule.name,\n severity=rule.severity,\n action=rule.action,\n message=rule.message.format(**system_state),\n ))\n return triggersFail-safe principle: Notification and escalation remain available through approved channels and rate limits. Automatic action is limited to pre-authorized scope reduction, such as pausing non-essential jobs. Anything outside that predefined reduction policy—including changing data, deleting history, shipping code, or broadening production behavior—requires human authorization.
\nProblem: \"Lessons learned\" can become a growing active queue that never changes system behavior.
\nIllustrative implementation: The window archives stale active lessons. It does not prove that lessons have been applied, and it does not force application.
\nfrom datetime import datetime, timedelta, timezone\n\nclass LessonWindow:\n def __init__(self, window_days: int = 7, max_unapplied: int = 30):\n self.window = timedelta(days=window_days)\n self.max_unapplied = max_unapplied\n\n def process(self, lessons: list[Lesson]) -> tuple[list[Lesson], list[Lesson]]:\n """Return (active_lessons, archived_lessons)."""\n active = []\n archived = []\n now = datetime.now(timezone.utc)\n candidates = [\n lesson for lesson in lessons\n if lesson.status not in {"applied", "archived"}\n ]\n overflow = max(len(candidates) - self.max_unapplied, 0)\n\n for lesson in lessons:\n if lesson.status == "archived":\n archived.append(lesson)\n continue\n if lesson.status == "applied":\n active.append(lesson)\n continue\n\n age = now - lesson.created_at\n\n if age > self.window:\n lesson.archive_once(reason=f"expired_after_{self.window.days}_days")\n archived.append(lesson)\n elif overflow > 0:\n lesson.archive_once(reason="overflow")\n archived.append(lesson)\n overflow -= 1\n else:\n active.append(lesson)\n\n return active, archivedRule: A lesson is only \"learned\" if system state changes because of it. If not applied within the active window, it can be archived, not forgotten, so that it no longer clutters the active queue.
\nThe harnesses integrate at the output boundary of the agent core, before content is persisted to memory:
\nAgent Output\n |\n v\n+-----------------+\n| Novelty Filter | --> Suppress? Yes: discard and log filter event\n+-----------------+ No: continue\n |\n v\n+-----------------+\n| Calculate Health| --> Update metrics store\n+-----------------+\n |\n v\n+-----------------+\n| Break-Glass | --> Notify or apply authorized reduction\n+-----------------+\n |\n v\n+-----------------+\n| Lesson Window | --> Archive expired, queue active\n+-----------------+\n |\n v\nPersist to MemoryThis placement is important: harnesses operate on output, not input. Filtering input could prevent useful ideas from reaching the agent. Filtering output ensures only valuable, non-redundant content is preserved.
\n# harness.toml\n[novelty_filter]\nenabled = true\nthreshold = 0.82\nwindow_days = 7\nmodel = "all-MiniLM-L6-v2"\n\n[health]\nenabled = true\ncalculation_window_days = 7\njobs = [\n { name = "extraction", weight = 0.40, check = "recent_output:runs/extraction" },\n { name = "briefing", weight = 0.30, check = "recent_output:runs/briefing" },\n { name = "weekly_review", weight = 0.30, check = "recent_output:runs/review" },\n]\n\n[break_glass]\nenabled = true\nmode = "supervised"\n\n[[break_glass.rules]]\nname = "commit_gap"\ncondition = "days_since_commit > 14"\nseverity = "critical"\naction = "notify_and_require_ack"\n\n[[break_glass.rules]]\nname = "health_critical"\ncondition = "calculated_health < 15"\nseverity = "critical"\naction = "reduce_authorized_scope"\nallowed_jobs = ["extraction", "health_check"]\n\n[[break_glass.rules]]\nname = "lesson_accumulation"\ncondition = "unapplied_lessons > 50 AND oldest_unapplied_days > 14"\nseverity = "warning"\naction = "notify_owner"\n\n[lesson_window]\nenabled = true\nwindow_days = 7\nmax_unapplied = 30The following is an illustrative contract, not a schema from a real product. The goal is to show the integration boundary a scheduler, orchestrator, or agent runtime would need to provide.
\n{\n "event": "agent.output.proposed",\n "timestamp": "2026-09-06T23:00:00Z",\n "run_id": "run_123",\n "agent_id": "research_agent",\n "content": "Proposed output content",\n "metrics": {\n "days_since_commit": 15,\n "calculated_health": 42.5,\n "unapplied_lessons": 12\n }\n}As of 2026-09-07, a 14-day evaluation starting on 2026-09-06 is not complete. A coherent 14-day inclusive window would run from 2026-09-06 through 2026-09-19. The table below is a prospective evaluation plan, not reported results.
\n| Metric | Baseline Definition | Treatment Definition | Target / Hypothesis |
|---|---|---|---|
| Daily persisted text | Bytes of agent-generated status or lesson text per day | Same measure after novelty filtering | 30-50% reduction without losing material decisions |
| Health score stability | Seven-day weighted health score from fixed checks | Same measure after calculated health is introduced | Changes should track job outcomes rather than remain narratively fixed |
| Break-glass behavior | Number of threshold breaches that produce no visible escalation | Number of threshold breaches with notification, acknowledgement, or authorized scope reduction | Critical breaches should create reviewable events |
| Lesson queue age | Age of oldest active unapplied lesson | Same measure after lesson window archival | Active queue should remain bounded |
Sample size and limitations: A single 14-day run is enough to detect obvious integration failures and generate hypotheses, but it is not enough to claim general performance. Workload mix, agent prompts, human review behavior, and calendar effects can dominate the result. Treat the outcome as local evidence for this system, not proof that the pattern works everywhere.
\nRelated-work caveat: This pattern overlaps with established ideas in observability, control loops, MLOps monitoring, guardrails, and human-in-the-loop operations. The novelty here, if any, is the packaging of those ideas around agent output boundaries. This article does not claim a new algorithm or publish comparative research.
\nIf the novelty threshold is too aggressive, genuinely novel but structurally similar content may be suppressed. Mitigation: Start with a conservative threshold, then adjust based on measured filter hit rate and manual false-positive review. The target should be meaningful reduction, not maximum suppression.
\nIf thresholds are too tight, break-glass triggers too frequently and is ignored. Mitigation: Reserve critical alerts for conditions that require review. Warning-level thresholds should log or notify without interrupting work.
\nArchiving lessons after a fixed window may hide valuable but long-term insights. Mitigation: Archived lessons remain searchable. The archive is cold storage: retrievable, but not cluttering the active queue.
\nThe harness architecture addresses a practical gap in agent design: agents need feedback loops about their own behavior, not just about the tasks they execute. Without these loops, systems can accumulate redundant memory, drifted metrics, and normalized paralysis.
\nThe four harnesses (novelty filter, calculated health, break-glass, lesson window) form a proposed minimal layer for supervised agent improvement. They require a small, explicit contract from the agent core—output events, job metrics, and authorization state—plus careful placement at the output boundary and human authorization for risky actions.
\nKey Takeaway: An agent that cannot detect its own stagnation will stagnate. Harnesses are the architectural layer that makes stagnation visible, reviewable, and fixable.
\nThis is a proposed reference pattern. The methodology above is intended to make future evaluation reproducible without relying on private supporting material.
\nLicense: CC BY-SA 4.0
\n", "https://reference-architecture.ai/properties/date": "2026-09-06", "https://atomicdata.dev/properties/tags": {"categories":["Architecture"],"tags":["agents","self-improvement","harnesses","feedback-loops"]}, "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"] },{"localId": "https://reference-architecture.ai/posts/github-oauth2/", "https://atomicdata.dev/properties/name": "Turning Open Source project into Product with Redis Enterprise", "https://reference-architecture.ai/properties/url": "https://reference-architecture.ai/posts/github-oauth2/", "https://atomicdata.dev/properties/description": "Last year, my reference project, \"The Pattern\", was the hackathon winner 2021 and got a bit of publicity and, in total, seven forks. But as with many open source projects, it is now stale. Time to revive \"The Pattern\" with new features and GitHub sponsors or Patreon patrons to help and inspire developers and creatives. In return, it's common to provide sponsor-only features and articles. Nevertheless, how can we do it with a large Redis-based machine learning pipeline?
\nThis article will introduce a simple first step:\nfor GitHub sponsors, we start with offering persistent storage of preferences: I have a simple flask POST API which adds nodes into the user's preference storage - a simple Redis set per user. And it will be a foundation to build other sponsor-only features.\nFor now, let's cover the basics:
\n\nflowchart LR\n id1(User) --> flask_login(Flask Login API)--> github(GitHub OAuth2)\n github-->flask_callback(Flask API callback)-->GitHubGraphQL(GitHub GraphQL)\n
There are a number of API's that GitHub offers to help developers, but the GitHub Authentication API is one of the most popular. This API allows you to log in to GitHub using your username and password, or OAuth token.
\nA login button with a standard OIDC/OAuth2 dance is one of the most common ways for a user to authenticate to an API.\nBelow is code taken from this gist and is very common for OAuth2 flows:
\nimport os \nclient_id = os.getenv('GITHUB_CLIENT_ID')\nclient_secret = os.getenv('GITHUB_SECRET')\n@app.route('/', methods=['GET', 'POST'])\ndef index():\n url = 'https://github.com/login/oauth/authorize'\n params = {\n 'client_id': client_id,\n\n 'scope': 'read:user,read:email',\n 'state': str(uuid4().hex),\n 'allow_signup': 'true'\n }\n url = furl(url).set(params)\n return redirect(str(url), 302)where GITHUB_CLIENT_ID and GITHUB_SECRET are client/secret GitHub Oauth2 apps. Register for following process on GitHub
\norg_name="applied-knowledge-systems"\n@app.route('/oauth2/callback')\ndef oauth2_callback():\n\n code = request.args.get('code')\n access_token_url = 'https://github.com/login/oauth/access_token'\n payload = {\n 'client_id': client_id,\n 'client_secret': client_secret,\n 'code': code,\n # 'redirect_uri':\n 'state': str(uuid4().hex)\n }\n r = requests.post(access_token_url, json=payload, headers={'Accept': 'application/json'})\n access_token = json.loads(r.text).get('access_token')\n print(access_token)\n \n access_user_url = 'https://api.github.com/user'\n response = requests.get(access_user_url, headers={'Authorization': 'token ' + access_token})\n data=response.json()\n user_email=data["email"]\n user_login=data["login"]\n user_id=data["id"]\n # response=redirect(url_for('login',next=redirect_url()))\n # response.set_cookie('user_id', str(user_id))\n # response.set_cookie('user_login', str(user_login))\n # return response\n query = """\n {\n viewer {\n sponsorshipsAsSponsor(first: 100) {\n nodes {\n sponsorable {\n ... on User {\n id\n email\n url\n }\n ... on Organization {\n id\n email\n name\n url\n }\n }\n tier {\n id\n name\n monthlyPriceInDollars\n monthlyPriceInCents\n }\n }\n }\n }\n }\n """\n response_graphql = requests.post('https://api.github.com/graphql', json={'query': query}, headers={'Authorization': 'token ' + access_token})\n response_graphql_data=response_graphql.json()["data"]\n if isinstance (response_graphql_data["viewer"]["sponsorshipsAsSponsor"]["nodes"], list):\n if response_graphql_data["viewer"]["sponsorshipsAsSponsor"]["nodes"][0]["sponsorable"]["name"]==org_name:\n # if user is a sponsor of Applied Knowledge System add them to set of sponsors\n redis_client.sadd(f'sponsors:{org_name}',user_id)\n # if RedisJSON enabled:\n # redis_client.json().set(f"user_details:{user_id}", '$', {\n # 'email': user_email,\n # 'id': user_id,\n # 'user_login': user_login,\n # 'graphql': response_graphql_data,\n # }) \n #if not\n redis_client.hset(f"user_details:{user_id}", mapping={\n 'email': user_email,\n 'id': user_id,\n 'user_login': user_login\n })\n return jsonify({\n 'status': 'success',\n 'email': user_email,\n 'id': user_id,\n 'user_login': user_login\n })The API we are using for our sponsor-only feature is straightforward:
\n@app.route('/exclude', methods=['POST','GET'])\ndef mark_node():\n if request.method == 'POST':\n if 'id' in request.json:\n node_id=request.json['id']\n else:\n if 'id' in request.args:\n node_id=request.args.get('id')\n user_id = session.get('user_id')\n log(f"Got user {user_id} from session")\n if not user_id:\n user_id = request.cookies.get('user_id')\n log(f"Got user {user_id} from cookie")\n redis_client.sadd("user:%s:mnodes" % user_id,node_id)\n response = jsonify(message=f"Finished {node_id} and {user_id}")\n return responseAnd the only purpose of this API is to mark nodes as unimportant for the given user by adding nodes to RedisSet, and those nodes will be excluded from search API output. So far, everything was pretty standard: basic flask API and GitHub Social login flow. Now let's add Redis Enterprise and synchronise sponsors preferences.
\nWhy not use Redis Enterprise directly for everything?\nThe project is memory-heavy, with a lot of data and machine learning inside Redis. This allows to achieve state-of-the-art performance, but it also takes over 120 GB RAM (or as much RAM as you can give it), and 128 GB Redis Enterprise instance will exceed my budget for open-source project. Obviously if there will be enough sponsors we can move more functionality into Redis Enterprise, but for that we need to finish building basic blocks. Register on Redis.com cloud and create a database with the subscription.
\n![\"Redis](\"https://reference-architecture.ai/posts/github-oauth2/redis_enterprise_screen.png\")
\nTake a note host, port and password for Redis Enterprise and create docker enviroment file:
cat .env.gears \nREDISENT_PWD="123"\nREDISENT_PORT="13444"\nREDISENT_HOST="hostname.cloud.redislabs.com"and create a docker compose with section passing .env.gears. Mine looks like this:
\n redisgraph:\n image: redislabs/redismod\n container_name: redisgears\n hostname: redisgears\n env_file:\n - ./.env.gears\n ports:\n - 127.0.0.1:9001:6379\nFirst flow:\nWe will be using RedisGears to synchronize all preferences with Redis Enterprise
\n\nflowchart LR\n id1(User Preferences Redis OSS) --> redis_gears1(Redis Gears)--> redise(Redis Enterprise)\n
If you are new to RedisGears, there is a pattern rgsync that covers exacly this use case, but I already have RedisGears, so I am going to build it step by step:
\n# gears_sync_preferences.py\nrconn=None\n\ndef connecttoRedisEnterise():\n import redis \n import os \n log(str(os.environ))\n # Get environment variables\n\n HOST = os.getenv('REDISENT_HOST')\n PASSWORD = os.getenv('REDISENT_PWD')\n PORT = os.getenv('REDISENT_PORT')\n\n redis_client=redis.Redis(host=HOST,port=PORT,charset="utf-8", password=PASSWORD, decode_responses=True)\n return redis_client\n\ndef sync_users(record):\n global rconn\n if not rconn:\n rconn=connecttoRedisEnterise()\n # Uncomment logs to check \n # log(str(record['key']))\n # log(str(record['value']))\n\n rconn.hset(record['key'],mapping=record['value'])\n\ngb = GB()\ngb.foreach(sync_users)\ngb.count()\ngb.run('user_details:*')this is a \"batch\" mode for RedisGears, which is easier to debug than streams. Install gears-cli](https://github.com/RedisGears/gears-cli) with pip install gears-cli run above script:
gears-cli run --host 127.0.0.1 --port 9001 gears_sync_preferences.py --requirements req_sync.txtwhere req_sync.txt
\nredis==3.5.3This RedisGears will copy all user's profiles into RedisEnterprise. Now let us add sponsors:
\n# gears_sync_sponsors.py\nrconn=None\n\ndef remove_prefix(text, prefix):\n return text[text.startswith(prefix) and len(prefix):]\n\ndef connecttoRedisEnterise():\n import redis \n import os \n log(str(os.environ))\n # Get environment variables\n\n HOST = os.getenv('REDISENT_HOST')\n PASSWORD = os.getenv('REDISENT_PWD')\n PORT = os.getenv('REDISENT_PORT')\n log(HOST)\n log(PORT)\n log(PASSWORD)\n redis_client=redis.Redis(host=HOST,port=PORT,charset="utf-8", password=PASSWORD)\n return redis_client\n\n\n\ndef sync_sponsors(record):\n global rconn\n if not rconn:\n rconn=connecttoRedisEnterise()\n\n log(str(record['key']))\n values=execute('SMEMBERS',record['key'])\n log(str(values))\n for each_value in values: \n rconn.sadd(record['key'],each_value)\n\ngb = GB('KeysReader')\ngb.foreach(sync_sponsors)\ngb.count()\ngb.run('user:*')But this one will sync all user's preferences, but we only need sponsors - let us add another feature of RedisGears - filter:
\nrconn=None\n\ndef remove_prefix(text, prefix):\n return text[text.startswith(prefix) and len(prefix):]\n\ndef connecttoRedisEnterise():\n import redis \n import os \n log(str(os.environ))\n # Get environment variables\n\n HOST = os.getenv('REDISENT_HOST')\n PASSWORD = os.getenv('REDISENT_PWD')\n PORT = os.getenv('REDISENT_PORT')\n\n redis_client=redis.Redis(host=HOST,port=PORT,charset="utf-8", password=PASSWORD, decode_responses=True)\n return redis_client\n\ndef filter_sponsors(record):\n org_name="applied-knowledge-systems"\n user_id = remove_prefix(record['key'],'user:')\n sponsor=execute('SISMEMBER',f'sponsors:{org_name}',user_id)\n return bool(sponsor==True)\n\ndef sync_sponsors(record):\n global rconn\n if not rconn:\n rconn=connecttoRedisEnterise()\n\n log(str(record['key']))\n values=execute('SMEMBERS',record['key'])\n log(str(values))\n for each_value in values: \n rconn.sadd(record['key'],each_value)\n\ngb = GB()\ngb.filter(filter_sponsors)\ngb.foreach(sync_sponsors)\ngb.count()\ngb.run('user:*')Then we are going to use Key miss events from Redis Gears to fetch data for all users:
\n\nflowchart LR\n redise(Redis Enterprise)\n redis_gears2(Redis Gears)--key miss--->redise\n redis_gears2-->redisOSS[Redis OSS]\n
and it's very easy, right from key miss example:
\n\ndef fetch_data(r):\n key = r['key']\n global rconn\n if not rconn:\n rconn=connecttoRedisEnterise()\n values=rconn.smembers(record['key'])\n log(str(values))\n for each_value in values: \n execute('SADD',record['key'],each_value)\n\nGB().foreach(fetch_data).register(prefix='user:*', commands=['smember'],eventTypes=['keymiss'], mode="async_local")There is one more option - to turn fetch_data into the async call, by wrapping it into async/await, but Redis Enterprise is fairly fast, and I don't think it's worth adding an async call in this case. For curiosity, see the example code in The Pattern repository.
\nIn this article, we walked through steps on how to create sponsor-specific \"nanoservices\" using RedisOSS, RedisGears and Redis Enterprise. This allows us to leverage the best of all worlds open source Redis, high availability and persistence with Redis Enterprise and RedisGears as the glue which holds everything together.
\nThis post is in collaboration with Redis.
\nCheck sponsors.go - query whether a GitHub user is your sponsor at a given tier (dollar amount)
\nThere are tools for advanced analytics, including free ones from Google and Kaggle.
\nThere are well-known and validated deployment architectures for applications and the cloud.
\nYet the number of practical applications is still tiny, and they retained niche implementations.\nWhile the benefits of AI are clear, there are still many gaps in AI architecture that need to be filled. For example, there is a gap between analytical tools and verified architectures for real-time deployments. This gap often stems from a lack of specific reference architectures and patterns, demonstrating the trade-offs between technologies, libraries, and tools.
\nLet's bridge the gap in knowledge and drive a connection between science and engineering to make fast, efficient, and practical AI deployments.\nThree things need to be in place to build an AI product:
\nI will use The Pattern, my [“Build on Redis” Hackathon prize-winning open source](https://github.com/applied-knowledge-systems/the-pattern) project, to illustrate how the capabilities below can be implemented and invite you to contribute or donate.
\nWe launch in two full-featured articles - NLP ML pipeline for turning unstructured JSON text into a knowledge graph and fresh off the press Benchmarks for BERT Large Question Answering inference for RedisAI and RedisGears with Grafana Dashboards by Mikhail Volkov
\n", "https://reference-architecture.ai/properties/date": "2022-06-16", "https://atomicdata.dev/properties/tags": {"categories":["Architecture"],"tags":["reference architecture","ai","announcement"]}, "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"] },{"localId": "https://reference-architecture.ai/docs/patterns/", "https://atomicdata.dev/properties/name": "The Twelve Core Patterns", "https://reference-architecture.ai/properties/url": "https://reference-architecture.ai/docs/patterns/", "https://atomicdata.dev/properties/description": "The harness, not the model, determines agent success. A harness has three jobs — deciding\nwhat the model sees, enforcing what it can do, and ensuring it learns from its own history\n— and each job is served by four patterns.
\nEvery pattern below is documented elsewhere on this site. This page is the index, not the\nexplanation: each entry links to where the argument is actually made.
\nWhat the model sees at each step. Context is a scarce resource: a 1M token window with\n800K of noise performs worse than 200K with 150K of curated signal.
\nPut the boundary in configuration, not in a runtime permission check. A role has access\nto certain sources; the knowledge graph is built from those sources; the agent therefore\ncannot reach anything outside its role. Not a check that could be bypassed — an absence.\nThe data was never loaded.
\n→ The Terraphim Engine, Layer 1
\nResolve text to concepts with finite state automata rather than vector similarity.\nAho-Corasick finds every match in O(n) time, with the same result every run. Probabilistic\nretrieval gives you a different context window each time you ask, which makes every\ndownstream failure irreproducible.
\n→ The MCP-Native Agent Architecture, Layer 2
\nCompact before the window fills, not when it overflows. By 100K tokens a working context\nis largely noise: superseded search results, abandoned reasoning paths, old file reads.\nAdding capacity without compaction makes performance worse, not better.
\n→ Three ways 1M token windows fail
\nInstructions compete, and by default the loudest wins rather than the most important. The\nsystem prompt says one thing, the project file another, the README a third. Weight them\nexplicitly and re-inject the high-priority ones, rather than trusting position in a\ntranscript.
\n→ Three ways 1M token windows fail
\nWhat the model can and cannot do. Risk-tiered, machine-readable, and enforced by the\nsystem rather than requested in a prompt.
\nAgents sharing a protocol layer must not share state. Each agent keeps its own role\nconfiguration and its own knowledge graph, discovering the others' tools without\ndiscovering their data.
\n→ Pattern 1: Multi-Tenant Agent Isolation
\nAn agent run you cannot replay is an agent run you cannot debug. Record the inputs and the\ndecisions such that the same run can be reconstructed exactly — which is only possible if\nthe layers underneath are themselves deterministic.
\n→ Pattern 2: Deterministic Replay
\nClassify tools by effect, not by name. Read-only operations run freely; destructive ones\nstop for approval. The protocol enforces the distinction, so an agent cannot talk its way\npast it.
\n→ Pattern 3: Human-in-the-Loop for Destructive Operations
\nEvery guardrail needs a deliberate, logged way through it. A guardrail with no override\ngets disabled wholesale the first time it blocks something urgent; an override that is\nnot recorded is indistinguishable from the guardrail not existing.
\n\nEnsuring the model learns from its own history. Without it, agents repeat solved\nproblems indefinitely and no amount of context helps.
\nDistinguish output that is new from output that merely looks new. An agent that cannot\ntell repetition from progress will re-report the same finding until someone stops reading.
\n\nDerive a health signal from observable metrics rather than asking the agent how it is\ndoing. Self-assessment is the least reliable input available; commit recency, unapplied\nlessons and stall time are not.
\n→ Harness 2: Calculated Health
\nCapture corrections at the moment of failure and apply them within a bounded window.\nLessons with no expiry become noise; lessons never applied were never lessons.
\n\nLong-running agents optimise for local coherence and lose the original objective. Ask\nperiodically, and mechanically: do we actually have the thing we set out to build? The\nquestion has to be asked by the harness, because an agent that has drifted cannot detect\nits own drift.
\n→ Three ways 1M token windows fail ·\nThe nine-step orchestration arc
\nThese twelve are the patterns this site currently makes an argument for, grouped by\nthe three jobs of an agent harness. The grouping is deliberate: a pattern\nthat does not serve one of those three jobs probably belongs in a different architecture.
\nThe list is not a claim to completeness. It is what is documented, and the honest test of\nany addition is whether there is an article behind it.
\n", "https://reference-architecture.ai/properties/date": "2026-09-08", "https://atomicdata.dev/properties/tags": {"categories":["Architecture"],"tags":["reference-architecture","patterns","agents","harnesses"]}, "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"] },{"localId": "https://reference-architecture.ai/docs/donate/", "https://atomicdata.dev/properties/name": "Support the project", "https://reference-architecture.ai/properties/url": "https://reference-architecture.ai/docs/donate/", "https://atomicdata.dev/properties/description": "The patterns described on this site are not theory. They are extracted from\nTerraphim, a privacy-first AI assistant built across 52 Rust\ncrates, and every one of those crates is open source.
\nThe most valuable support is use and scrutiny.
\nterraphim_automata for Aho-Corasick matching,\nterraphim_rolegraph for role-based knowledge\ngraphs, or terraphim_persistence for the\nstorage layer.Issues, benchmarks and architecture critique are worth more than money here. If a\ntrade-off on this site looks wrong, say so — with evidence — at\ngithub.com/terraphim. Deployment guides and reproducible\nbenchmarks are especially welcome.
\nIf your organisation depends on this work, Terraphim Pro funds\nthe open-source engine directly: connectors, an LLM proxy and agentic skills for\nenterprise deployment.
\n", "https://reference-architecture.ai/properties/date": "2026-09-08", "https://atomicdata.dev/properties/tags": {"categories":["donations","open-source"],"tags":["donate","support","terraphim"]}, "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"] },{"localId": "https://reference-architecture.ai/docs/legacy-capability-map/", "https://atomicdata.dev/properties/name": "Capability Map (2020-2022)", "https://reference-architecture.ai/properties/url": "https://reference-architecture.ai/docs/legacy-capability-map/", "https://atomicdata.dev/properties/description": "There are tools for advanced analytics, including free ones from Google and Kaggle.
\nThere are well-known and validated deployment architectures for applications and the cloud.
\nYet the number of practical applications is still tiny, and they retained niche implementations.\nWhile the benefits of AI are clear, there are still many gaps in AI architecture that need to be filled. For example, there is a gap between analytical tools and verified architectures for real-time deployments. This gap often stems from a lack of specific reference architectures and patterns, demonstrating the trade-offs between technologies, libraries, and tools.
\nLet's bridge the gap in knowledge and drive a connection between science and engineering to make fast, efficient, and practical AI deployments.
\nThree things need to be in place to build an AI product:
\nI will use The Pattern, my Build on Redis Hackathon prize-winning open source project to illustrate how the capabilities below can be implemented and invite you to contribute or donate.\nThe diagrams below are clickable.
\n![](\"/images/ai_product.drawio.svg\")
\nflowchart LR; \nsubgraph \" \"\n data_intake(Data Acquisition);\n data_preprocessing(Data Preprocessing);\n data_validation(Data Validation);\n data_streaming(Data Streaming);\n ai_ml_pipe(AI ML Pipeline
Model Training/Test/Validation);\n kg(Knowledge Graph);\n ML_inference(ML Inference);\n interaction(Interaction Layer
Voice/VR/AR/Meta);\n click data_intake \"/docs/intake/\" \"Data Acquisition\"\n click ML_inference \"/docs/bert-qa-benchmarking/\" \"BERT Large Question Answering\"\n click kg \"/docs/nlp/#what-is-a-knowledge-graph\" \"Knowledge Graph\"\n click data_preprocessing \"/docs/nlp/#redisgears-for-nlp-pre-processing\" \"Data pre-processing\"\n click data_streaming \"/docs/nlp/#goal\" \"Data Streaming\"\n click ai_ml_pipe \"/docs/nlp/#overall-architecture-overview-components-diagram\" \"Overall ML pipeline\"\nend\n
\nflowchart LR;\nsubgraph 2 [Enabling Capabilities]\n subgraph \" \"\nD1[Data Governance]\nD2[Data Quality Management]\nD3[Metadata Management]\nclick D3 \"/docs/metadata/\" \"Metadata Management\"\nend\nsubgraph \" \"\n id2[ML performance and bias monitoring];\n id3[Application Performance Monitoring];\n id4[Hardware Performance Monitoring];\n click id3 \"/docs/bert-qa-benchmarking/#running-the-benchmark\" \"Running Benchmarks\"\n click id4 \"/docs/bert-qa-benchmarking/#using-grafana-to-monitor-redisgears-throughput-cpu-and-memory-usage\" \"CPU and Memory Benchmarks\"\nend\nsubgraph \" \"\n id5[DevOps: Continuous Integration/Continuous Deployment];\n c1[Configuration management]\n c2[Collaboration and knowledge management tooling]\n c3[Change Management]\nend \n subgraph 3 [Self Serving Infrastructure]\n s1[Computing Infrastructure]\n s2[Serving infrastructure]\n end\n end\n style 2 fill:#485fc754,stroke:#333,stroke-width:4px;\n
This article will explore the challenges and opportunities of deploying a large BERT Question Answering Transformer model(bert-large-uncased-whole-word-masking-finetuned-squad) from inside Huggingface, where RedisGears and RedisAI perform heavy lifting while leveraging in-memory datastore Redis.
\nSome numbers for inspiration and why to read this article:
\npython3 transformers_plain_bert_qa.py \nairborne transmission of respiratory infections is the lack of established methods for the detection of airborne respiratory microorganisms\n10.351818372 secondstime curl -i -H "Content-Type: application/json" -X POST -d '{"search":"Who performs viral transmission among adults"}' http://localhost:8080/qasearch\n\nreal\t0m0.747s\nuser\t0m0.004s\nsys\t0m0.000s\nBERT Question Answering inference works where the ML model selects an answer from the given text. In other words, BERT QA \"thinks\" through the following: \"What is the answer from the text, assuming the answer to the question exists within the paragraph selected.\"
\nSo it's important to select text potentially containing an answer. A typical pattern is to use Wikipedia data to build Open Domain Question Answering.
\nOur QA system is a medical domain-specific question/answering pipeline, hence we need a first pipeline that turns data into a knowledge graph. This NLP pipeline is available at Redis LaunchPad, is fully open source, and is described in a previous article. Here is a 5 minute video describing it, and below you will find an architectural overview:
\n![\"featured\"](\"https://reference-architecture.ai/docs/bert-qa-benchmarking/featured.png\")
In the BERT QA pipeline (or in any other modern NLP inference task), there are two steps:
\nWith Redis, we have the opportunity to pre-compute everything and store it in memory, but how do we do it? Unlike with the summarization ML learning task, the question is not known in advance, so we can't pre-compute all possible answers. However, we can pre-tokenize all potential answers (i.e. all paragraphs in the dataset) using RedisGears:
\ndef parse_sentence(record):\n import redisAI\n import numpy as np\n global tokenizer\n if not tokenizer:\n tokenizer=loadTokeniser()\n hash_tag="{%s}" % hashtag()\n\n for idx, value in sorted(record['value'].items(), key=lambda item: int(item[0])):\n tokens = tokenizer.encode(value, add_special_tokens=False, max_length=511, truncation=True, return_tensors="np")\n tokens = np.append(tokens,tokenizer.sep_token_id).astype(np.int64)\n tensor=redisAI.createTensorFromBlob('INT64', tokens.shape, tokens.tobytes())\n\n key_prefix='sentence:'\n sentence_key=remove_prefix(record['key'],key_prefix)\n token_key = f"tokenized:bert:qa:{sentence_key}:{idx}"\n redisAI.setTensorInKey(token_key, tensor)\n execute('SADD',f'processed_docs_stage3_tokenized{hash_tag}', token_key)\nSee the full code on GitHub.
\nThen for each Redis Cluster shard, we pre-load the BERT QA model by downloading, exporting it into torchscript, then loading it into each shard:
\ndef load_bert():\n model_file = 'traced_bert_qa.pt'\n\n with open(model_file, 'rb') as f:\n model = f.read()\n startup_nodes = [{"host": "127.0.0.1", "port": "30001"}, {"host": "127.0.0.1", "port":"30002"}, {"host":"127.0.0.1", "port":"30003"}]\n cc = ClusterClient(startup_nodes = startup_nodes)\n hash_tags = cc.execute_command("RG.PYEXECUTE", "gb = GB('ShardsIDReader').map(lambda x:hashtag()).run()")[0]\n print(hash_tags)\n for hash_tag in hash_tags:\n print("Loading model bert-qa{%s}" %hash_tag.decode('utf-8'))\n cc.modelset('bert-qa{%s}' %hash_tag.decode('utf-8'), 'TORCH', 'CPU', model)\n print(cc.infoget('bert-qa{%s}' %hash_tag.decode('utf-8')))The full code is available on GitHub.
\nAnd when a question comes from the user, we tokenize and append the question to the list of potential answers before running the RedisAI model:
\n token_key = f"tokenized:bert:qa:{sentence_key}"\n # encode question\n input_ids_question = tokenizer.encode(question, add_special_tokens=True, truncation=True, return_tensors="np")\n t=redisAI.getTensorFromKey(token_key)\n input_ids_context=to_np(t,np.int64)\n # merge (append) with potential answer, context - is pre-tokenized paragraph\n input_ids = np.append(input_ids_question,input_ids_context)\n attention_mask = np.array([[1]*len(input_ids)])\n input_idss=np.array([input_ids])\n num_seg_a=input_ids_question.shape[1]\n num_seg_b=input_ids_context.shape[0]\n token_type_ids = np.array([0]*num_seg_a + [1]*num_seg_b)\n # create actual model runner for RedisAI\n modelRunner = redisAI.createModelRunner(f'bert-qa{hash_tag}')\n # make sure all types are correct\n input_idss_ts=redisAI.createTensorFromBlob('INT64', input_idss.shape, input_idss.tobytes())\n attention_mask_ts=redisAI.createTensorFromBlob('INT64', attention_mask.shape, attention_mask.tobytes())\n token_type_ids_ts=redisAI.createTensorFromBlob('INT64', token_type_ids.shape, token_type_ids.tobytes())\n redisAI.modelRunnerAddInput(modelRunner, 'input_ids', input_idss_ts)\n redisAI.modelRunnerAddInput(modelRunner, 'attention_mask', attention_mask_ts)\n redisAI.modelRunnerAddInput(modelRunner, 'token_type_ids', token_type_ids_ts)\n redisAI.modelRunnerAddOutput(modelRunner, 'answer_start_scores')\n redisAI.modelRunnerAddOutput(modelRunner, 'answer_end_scores')\n # run RedisAI model runner\n res = await redisAI.modelRunnerRunAsync(modelRunner)\n answer_start_scores=to_np(res[0],np.float32)\n answer_end_scores = to_np(res[1],np.float32)\n answer_start = np.argmax(answer_start_scores)\n answer_end = np.argmax(answer_end_scores) + 1\n answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end],skip_special_tokens = True))\n log("Answer "+str(answer))\n return answer\nCheckout the full code, available on GitHub.
\nThe process for making a BERT QA API call looks like this:
\n![\"Architecture](\"https://reference-architecture.ai/docs/bert-qa-benchmarking/diagrams_02.png\")
Here I use two cool features of RedisGears: capturing events on key miss and using async/await to run RedisAI on each shard without locking the primary thread - so that Redis Cluster can continue to serve other customers. For benchmarks, caching responses from RedisAI is disabled. If you are getting response times in nanoseconds on the second call rather then milliseconds, check to make sure the line linked above is commented out.
\nPre-requisites for running the benchmark:
\nAssuming you are running Debian or Ubuntu and have Docker and docker-compose installed (or can create a virtual environment via conda), run the following commands:
\ngit clone --recurse-submodules https://github.com/applied-knowledge-systems/the-pattern.git\ncd the-pattern\n./bootstrap_benchmark.shThe above commands should end with a curl call to the qasearch API, since Redis caching is disabled for the benchmark.
\nNext, invoke curl like this:
\ntime curl -i -H "Content-Type: application/json" -X POST -d '{"search":"Who performs viral transmission among adults"}' http://localhost:8080/qasearchExpect the following output, or something similar based on your runtime environment:
\nHTTP/1.1 200 OK\nServer: nginx/1.18.0 (Ubuntu)\nDate: Sun, 29 May 2022 12:05:39 GMT\nContent-Type: application/json\nContent-Length: 2120\nConnection: keep-alive\n\n{"links":[{"created_at":"2002","rank":13,"source":"C0001486","target":"C0152083"}],"results":[{"answer":"adenovirus","sentence":"The medium of 40 T150 flasks of adenovirus transducer dec CAR CHO cells yielded 0 5 1 my of purified msCEACAM1a 1 4 protein","sentencekey":"sentence:PMC125375.xml:{mG}:202","title":"Crystal structure of murine sCEACAM1a[1,4]: a coronavirus receptor in the CEA family"}] OUTPUT_REDUCTED}I modified the output of API for the benchmark to return results from all shards - even if the answer is empty, in the run above five shards return answers, overall API call response under second with all additional hops to search in RedisGraph.
\nI modified the output of the API for the benchmark to return results from all shards - even if the answer is empty. In the run above five shards return answers. The overall API call response takes less than one second with all additional hops to search in RedisGraph!
\n![\"Architecture](\"https://reference-architecture.ai/docs/bert-qa-benchmarking/diagrams_01.png\")
Let's dig deeper into what's happening under the hood:
\nYou should have a sentence key with shard id, which you get by looking at the \"Cache key\" from docker logs -f rgcluster. In my setup the cache key is, \"bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults\". If you think it looks like a function call it's because it is a function call. It is triggered if the key isn't present in the Redis Cluster, which for the benchmark will be every time since if you remember we disabled caching the output.
One more thing to figure out from the logs is the port of the shard corresponding to the hashtag, also known as the shard id. It is the text found in betweeen the curly brackets – looks like {6fd} above. The same will be in the output for the export_load script. In my case the cache key was found in \"30012.log\", so my port is 30012.
Next I run the following command:
\nredis-cli -c -p 300012 -h 127.0.0.1 get "bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults"and then run the benchmark:
\nredis-benchmark -p 30012 -h 127.0.0.1 -n 10 get "bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults"\n====== get bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults ======\n 10 requests completed in 0.04 seconds\n 50 parallel clients\n 3 bytes payload\n keep alive: 1\n\n10.00% <= 41 milliseconds\n100.00% <= 41 milliseconds\n238.10 requests per secondIf you are wondering, -n = number of times. In this case we run the benchmark 10 times. You can also add:
– csv if you want to output in CSV format
– precision 3 if you want more decimals in the ms
More information about the benchmarking tool can be found on the redis.io Benchmarks page.
\nif you don't have redis-utils installed locally, you can use Docker as follows:
\ndocker exec -it rgcluster /bin/bash\nredis-benchmark -p 30012 -h 127.0.0.1 -n 10 get "bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults"\n====== get bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults ======\n 10 requests completed in 1.75 seconds\n 50 parallel clients\n 99 bytes payload\n keep alive: 1\n host configuration "save":\n host configuration "appendonly": no\n multi-thread: no\n\nLatency by percentile distribution:\n0.000% <= 243.711 milliseconds (cumulative count 1)\n50.000% <= 987.135 milliseconds (cumulative count 5)\n75.000% <= 1577.983 milliseconds (cumulative count 8)\n87.500% <= 1662.975 milliseconds (cumulative count 9)\n93.750% <= 1744.895 milliseconds (cumulative count 10)\n100.000% <= 1744.895 milliseconds (cumulative count 10)\n\nCumulative distribution of latencies:\n0.000% <= 0.103 milliseconds (cumulative count 0)\n10.000% <= 244.223 milliseconds (cumulative count 1)\n20.000% <= 409.343 milliseconds (cumulative count 2)\n30.000% <= 575.487 milliseconds (cumulative count 3)\n40.000% <= 821.247 milliseconds (cumulative count 4)\n50.000% <= 987.135 milliseconds (cumulative count 5)\n60.000% <= 1157.119 milliseconds (cumulative count 6)\n70.000% <= 1497.087 milliseconds (cumulative count 7)\n80.000% <= 1577.983 milliseconds (cumulative count 8)\n90.000% <= 1662.975 milliseconds (cumulative count 9)\n100.000% <= 1744.895 milliseconds (cumulative count 10)\n\nSummary:\n throughput summary: 5.73 requests per second\n latency summary (msec):\n avg min p50 p95 p99 max\n 1067.296 243.584 987.135 1744.895 1744.895 1744.895The platform only has 20 articles and 8 Redis nodes (4 masters + 4 slaves), so relevance would be wrong and it doesn't need a lot of memory.
\nNow let's check how long our RedisAI model runs on the {6fd} shard:
127.0.0.1:30012> AI.INFO bert-qa{6fd}\n 1) "key"\n 2) "bert-qa{6fd}"\n 3) "type"\n 4) "MODEL"\n 5) "backend"\n 6) "TORCH"\n 7) "device"\n 8) "CPU"\n 9) "tag"\n10) ""\n11) "duration"\n12) (integer) 8928136\n13) "samples"\n14) (integer) 58\n15) "calls"\n16) (integer) 58\n17) "errors"\n18) (integer) 0\nbert-qa{6fd} is the key of the actual (very large) model saved. The AI.INFO command gives us a cumulative duration of 8928136 microseconds and 58 calls, which is approximately 153 milliseconds per call.
Let's double-check to make sure that's right by resetting the stats and then re-runnning the benchmark.
\nFirst, reset the stats:
\n127.0.0.1:30012> AI.INFO bert-qa{6fd} RESETSTAT\nOK\n127.0.0.1:30012> AI.INFO bert-qa{6fd}\n 1) "key"\n 2) "bert-qa{6fd}"\n 3) "type"\n 4) "MODEL"\n 5) "backend"\n 6) "TORCH"\n 7) "device"\n 8) "CPU"\n 9) "tag"\n10) ""\n11) "duration"\n12) (integer) 0\n13) "samples"\n14) (integer) 0\n15) "calls"\n16) (integer) 0\n17) "errors"\n18) (integer) 0Then, re-run the benchmark:
\nredis-benchmark -p 30012 -h 127.0.0.1 -n 10 get "bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults"\n====== get bertqa{6fd}_PMC169038.xml:{6fd}:33_Who performs viral transmission among adults ======\n 10 requests completed in 1.78 seconds\n 50 parallel clients\n 99 bytes payload\n keep alive: 1\n host configuration "save":\n host configuration "appendonly": no\n multi-thread: no\n\nLatency by percentile distribution:\n0.000% <= 188.927 milliseconds (cumulative count 1)\n50.000% <= 995.839 milliseconds (cumulative count 5)\n75.000% <= 1606.655 milliseconds (cumulative count 8)\n87.500% <= 1692.671 milliseconds (cumulative count 9)\n93.750% <= 1779.711 milliseconds (cumulative count 10)\n100.000% <= 1779.711 milliseconds (cumulative count 10)\n\nCumulative distribution of latencies:\n0.000% <= 0.103 milliseconds (cumulative count 0)\n10.000% <= 189.183 milliseconds (cumulative count 1)\n20.000% <= 392.191 milliseconds (cumulative count 2)\n30.000% <= 540.159 milliseconds (cumulative count 3)\n40.000% <= 896.511 milliseconds (cumulative count 4)\n50.000% <= 996.351 milliseconds (cumulative count 5)\n60.000% <= 1260.543 milliseconds (cumulative count 6)\n70.000% <= 1456.127 milliseconds (cumulative count 7)\n80.000% <= 1606.655 milliseconds (cumulative count 8)\n90.000% <= 1692.671 milliseconds (cumulative count 9)\n100.000% <= 1779.711 milliseconds (cumulative count 10)\n\nSummary:\n throughput summary: 5.62 requests per second\n latency summary (msec):\n avg min p50 p95 p99 max\n 1080.454 188.800 995.839 1779.711 1779.711 1779.711Now check the stats again:
\nAI.INFO bert-qa{6fd}\n 1) "key"\n 2) "bert-qa{6fd}"\n 3) "type"\n 4) "MODEL"\n 5) "backend"\n 6) "TORCH"\n 7) "device"\n 8) "CPU"\n 9) "tag"\n10) ""\n11) "duration"\n12) (integer) 1767749\n13) "samples"\n14) (integer) 20\n15) "calls"\n16) (integer) 20\n17) "errors"\n18) (integer) 0Now we get 88387.45 microseconds per call ~0.088387 seconds, which is pretty fast! Also, considering we started with 10 seconds per call, I think the benefits of using RedisAI in combination with RedisGears are pretty obvious. However, the trade-off is high memory usage.
\nThere are many ways to optimize this deployment. For example, you can add a FP16 quantization and ONNX runtime. If you would like to try that, this script will be a good starting point.
\nThanks to the contribution of Mikhail Volkov, we can now observe RedisGears and RedisGraph throughput and memory consumption using Grafana. When you cloned repository it started Graphana Docker, which has pre-build templates to monitor RedisCluster, including RedisGears and RedisAI, and Graph - which is Redis with RedisGraph. \"The Pattern\" dashboard provides an overview, with all the key benchmark metrics you care about:
\n![\"Grafana](\"https://reference-architecture.ai/docs/bert-qa-benchmarking/graphana_redis_graph.png\")
![\"Grafana](\"https://reference-architecture.ai/docs/bert-qa-benchmarking/graphana_cluster_overview.png\")
This post is in collaboration with Redis.
\n", "https://reference-architecture.ai/properties/date": "2022-06-16", "https://atomicdata.dev/properties/tags": {"categories":["inference","performance","monitoring"],"tags":["redis","redis-cluster","redisai","redisgears","benchmark","performance","benchmarks"]}, "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"] },{"localId": "https://reference-architecture.ai/docs/nlp/", "https://atomicdata.dev/properties/name": "Building a Pipeline for Natural Language Processing using RedisGears", "https://reference-architecture.ai/properties/url": "https://reference-architecture.ai/docs/nlp/", "https://atomicdata.dev/properties/description": "Disclaimer originally published in collaboration with Ajeet Raina on Developer.Redis.Com
\nIn this tutorial, you will learn how to build a pipeline for Natural Language Processing(NLP) using RedisGears. For this demonstration, we will be leveraging the Kaggle CORD19 datasets. The implementation is designed to avoid running out of memory, leveraging Redis Cluster and RedisGears, where the use of RedisGears allows for processing data on storage without the need to move data in and out of the Redis Cluster—using Redis Cluster as data fabric. Redis Cluster allows for horizontal scalability up to 1000 nodes, and together with RedisGears, provides a distributed system where data science/ML engineers can focus on processing steps, without the worry of writing tons of scaffoldings for distributed calculations.
\n![\"nlp\"](\"https://reference-architecture.ai/docs/nlp/nlparch.png\")
This project was built with the aim to make it easier for other people to contribute and build better information and knowledge management products.
\nRedisGears have enormous potential, particularly for text processing—you can process your data “on data” without needing to move them in and out of memory. Summary of the important points:
\nToday, we live in the world of new systems that operate not just files, folders, or web pages, but entities with their properties and relationships between them, organized into hierarchies of classes and categories. These systems are used everywhere from the military-industrial complex to our everyday lives. Palantir, Primer, and other data companies enable massive intelligence and counterintelligence projects in military and security forces, Quid and RecordedFuture enable competitive analytics, Bottlenose and similar enterprises enable online reputation analytics. Microsoft Graph enables new kinds of productivity apps for the enterprises, Google Knowledge Graph and Microsoft’s Satori enable everyday search queries, and together with Amazon Information Graph they power corresponding AI assistants by enabling them to answer questions about the world facts
\nAll these (and many other more specialized) systems are used in different domains, but all of them use Knowledge Graphs as their foundation.
\nKnowledge graphs are one of the best ways to connect and make sense out of information from different data sources, following the motto of one of the vendors— “It’s about things not strings”.
\nKnowledge Graph consists of thesaurus, taxonomy and ontology. In this pipeline I assume knowledge is captured in medical metathesaurus UMLS and concepts in text are related if they are part of the same sentence, therefore concept become node, their relationship becomes edge:
\n![\"concepts1\"](\"https://reference-architecture.ai/docs/nlp/concepts1.png\")
\n![\"concepts2\"](\"https://reference-architecture.ai/docs/nlp/concepts2.png\")
Concepts have CUI (Concept Unique Identifiers) and those will be primary keys in nodes, linked to UMLS thesaurus. For example, if you search, “How does temperature and humidity affect the transmission of 2019-nCoV?” on the demo website http://thepattern.digital/ and move slider to 1996, there is an edge-connecting transmission (C5190195) and birth (C5195639) and the part of sentence matched, “the rate of transmission to an infant born to,” from the report titled, “Afebrile Pneumonia in infants.”
\n![\"concepts3\"](\"https://reference-architecture.ai/docs/nlp/concepts3.png\")
![\"component_diagram\"](\"https://reference-architecture.ai/docs/nlp/component_diagram.png\")
Intake step - is very simple put all JSON records into RedisCluster, then NLP pipeline starts processing all records, code is here.
\nFor each record — detect language (discard non English), it’s filter
\nMap paragraphs into a sentence — flatmap
\nSentences spellchecker — it’s map
\nSave sentences into hash — processor
\nEnsure that you install virtualenv in your system
\n git clone --recurse-submodules https://github.com/applied-knowledge-systems/the-pattern.git\n cd the-pattern docker-compose -f docker-compose.dev.yml up --build -dYou can deploy PyTorch and spacy to run on RedisGears.
\n bash post_start_dev.shFor Data science-focused deployment, RedisCluster should be in HA mode with at least one slave for each master.\nOne need to change a few default parameters for rgcluster to accommodate the size of PyTorch and spacy libraries (each over 1GB zipped), gist with settings.
\n cd ./the-pattern-platform/You can create it via
\n conda create -n pattern_env python=3.8or
\nAlternatively, you can activate by using the below CLI:
\n source ~/venv_cord19/bin/activate #or create new venv\n pip install -r requirements.txt bash cluster_pipeline.shWait for a bit and then check:
\n redis-cli -p 9001 -h 127.0.0.1 GRAPH.QUERY cord19medical "MATCH (n:entity) RETURN count(n) as entity_count" \n redis-cli -p 9001 -h 127.0.0.1 GRAPH.QUERY cord19medical "MATCH (e:entity)-[r]->(t:entity) RETURN count(r) as edge_count" curl -i -H "Content-Type: application/json" -X POST -d '{"search":"How does temperature and humidity affect the transmission of 2019-nCoV"}' \n http://localhost:8080/gsearchWhile RedisGears allows to deploy and run Machine Learning libraries like spacy and BERT transformers, the solution above uses simpler approach:
\n gb = GB('KeysReader')\n gb.filter(filter_language)\n gb.flatmap(parse_paragraphs)\n gb.map(spellcheck_sentences)\n gb.foreach(save_sentences)\n gb.count()\n gb.register('paragraphs:*',keyTypes=['string','hash'], mode="async_local")This is the overall pipeline: those 7 lines allow you to run logic in a distributed cluster or on a single machine using all available CPUs - no changes required until you need to scale over more 1000 nodes. I use KeysReader registered for namespace paragraphs for all strings or hashes. My pipeline would need to run in async mode. For data scientists, I would recommend using gb.run to make sure gears function work and it will run in batch mode and then change it to register - to capture new data. By default, functions will return output, hence the need for count() - to prevent fetching the whole dataset back to the command issuing machine (90 GB for Cord19).
\nOverall pre-processing is a straightforward - full code is here.
\nThings to keep in mind:
\n execute('SADD','processed_docs_{%s}' % hashtag(),article_id) """\n load symspell and relevant dictionaries\n """\n sym_spell=None \n\n def load_symspell():\n import pkg_resources\n from symspellpy import SymSpell, Verbosity\n sym_spell = SymSpell(max_dictionary_edit_distance=1, prefix_length=7)\n dictionary_path = pkg_resources.resource_filename(\n "symspellpy", "frequency_dictionary_en_82_765.txt")\n bigram_path = pkg_resources.resource_filename(\n "symspellpy", "frequency_bigramdictionary_en_243_342.txt")\n # term_index is the column of the term and count_index is the\n # column of the term frequency\n sym_spell.load_dictionary(dictionary_path, term_index=0, count_index=1)\n sym_spell.load_bigram_dictionary(bigram_path, term_index=0, count_index=2)\n return sym_spell bg = GearsBuilder('KeysReader')\n bg.foreach(process_item)\n bg.count()\n bg.register('sentence:*', mode="async_local",onRegistered=OnRegisteredAutomata)\nOnRegisteredAutomata will perform similarly to symspell example above except it will download pre-build Aho-Corasick automata (30Mb).\nAho-Corasick is a very fast matcher and allows to perform >900 Mb text per second even on commodity laptop, RedisGears cluster makes a very smooth distribution of data and ML model and matching using available CPU and Memory. Full matcher code.
\nOutput of the matcher: nodes and edges are candidates to use another RedisGears pattern rgsync where you can write fast into Redis and RedisGears are going to replicate data into slower storage using RedisStreams.\nBut I decided to use streams and handcraft the population of the RedisGraph database, which will be focus of the next blog post.
\nOutput of the matcher: nodes and edges are candidates to use another RedisGears pattern rgsync where you can write fast into Redis and RedisGears are going to replicate data into slower storage using RedisStreams, while this demo uses streams and populates RedisGraph database with nodes and edges calculating rank of each.
\nWe took OCR scans in JSON format and turned them into Knowledge Graph, demonstrating how you can traditional Semantic Network/OWL/Methathesaurus technique based on Unified Medical Language System. Redis Ecosystem offers a lot to the data science community, and can take place at the core of Kaggle notebooks, ML frameworks and make deployment and distribution of data more enjoyable. The success of our industry depends on how our tools work together — regardless of whether they are engineering, data science, machine learning and organisational or architectural.
\nWith the collaboration of RedisLabs and community, the full pipeline code is available via https://github.com/applied-knowledge-systems/the-pattern-platform. In case, you want to try it locally, then you can find a Docker Launch script in the root of the repository along with short quickstart guide. PR and suggestions are welcome. The overall goal of the project is to allow other to build their more interesting pipeline on top of it.
\nIn CORD 19 dataset mentioned in Data Acquisition Metadata stored in the separate csv file from the source data. Here simple script to parse date/times and attach it to JSON/XML files
\n", "https://reference-architecture.ai/properties/date": "2022-05-14", "https://atomicdata.dev/properties/tags": {"categories":["metadata"],"tags":["data","metadata"]}, "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"] },{"localId": "https://reference-architecture.ai/docs/contribution/", "https://atomicdata.dev/properties/name": "Contribution Guidelines", "https://reference-architecture.ai/properties/url": "https://reference-architecture.ai/docs/contribution/", "https://atomicdata.dev/properties/description": "General guidelines for contributing to the project.
\nThere should be a a path to be implemented in a real world - good prototype or production deployment.
\nWelcome pull requests on \n\n\n\n. Check out supported shortcuts Extended Shortcuts
\nWhen contributing you agreeing to share your contribution under ![](\"https://img.shields.io/badge/License-CC%20BY--SA%204.0-lightgrey.svg\"){kind=icon}
For the Reference Architecture for AI, we used Kaggle Cord19 dataset, \"COVID-19 Open Research Dataset (CORD-19). CORD-19 is a resource of over 1,000,000 scholarly articles, including over 400,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. This freely available dataset is provided to the global research community to apply recent advances in natural language processing and other AI techniques to generate new insights in support of the ongoing fight against this infectious disease.\"
\nExample script parses documents taking out body_text and saves under paragraphs in Redis cluster.
\n", "https://reference-architecture.ai/properties/date": "2020-08-31", "https://atomicdata.dev/properties/tags": {"categories":["intake","data acquisition"],"tags":["data","acquisition","intake"]}, "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"] },{"localId": "https://reference-architecture.ai/docs/ai-product/", "https://atomicdata.dev/properties/name": "The Pattern: Machine Learning Natural Language Processing meets VR/AR", "https://reference-architecture.ai/properties/url": "https://reference-architecture.ai/docs/ai-product/", "https://atomicdata.dev/properties/description": "To fight ever-increasing complexity, \"The Pattern\" projects help find relevant knowledge using Artificial Intelligence and novel UX elements, all powered by Redis - a new generation real-time data fabric turned into knowledge fabric
\nOverall repository for CORD19 medical NLP pipeline, API and UI, design and architecture.
\nDemo Video:
\n\nDemo Server (no persistance): https://thepattern.digital/
\nThe medical profession put a lot of effort into collaboration, starting from Latin as a common language to industry-wide thesauruses like UMLS. However, if full of scandals where publications in a prestigious journal would be retracted, and the World Health Organisation would change its policy advice based on the article. I think \"paper claiming that eating a bat-like Pokémon sparked the spread of COVID-19\" takes a prize. One would say that editors in those journals don't do their job, and while it may seem true, I would say they had no chance: with a number of publications about COVID (SARS-V) passing 300+ per day, we need better tools to navigate via such flow of information.\nWhen exploring science or engineering topics, I look at the diversity of the opinion, not the variety of the same cluster of words or the same thought. I want to avoid confirmation bias. I want to find articles relevant to the same concept, not necessarily the ones which have similar words. My focus is to build a natural language processing pipeline capable of handling a large number of documents and concepts, incorporating System 1 AI (fast, intuitive reasoning) and System 2 (high-level reasoning) and then present knowledge in a modern VR/AR visualisation. Search or rather information exploration should be spatial, preferably in VR (memory palace, see Theatre of Giulio Camillo). A force-directed graph is a path towards it, where visuals are assisted by text — relevant text pops up on the connection and where people explore the concepts and then dig deeper into the text. The purpose of the pipeline is that knowledge should be reusable and shareable.
\nJoin our community on Discord or post on GitHub Discussions](https://github.com/applied-knowledge-systems/the-pattern/discussions)
\n", "https://reference-architecture.ai/properties/date": "2020-08-31", "https://atomicdata.dev/properties/tags": {"categories":["ai product","nlp","medical"],"tags":["ai product","ai","medical"]}, "https://atomicdata.dev/properties/isA": ["https://atomicdata.dev/classes/Article"] }]