Brain Ingest Gate

Brain Ingest Gate — Resolve and Dedup Before Anything Enters the Brain

Convention: see conventions/brain-first.md — the lookup chain (gbrain entity → search → query → get) is the same chain this gate runs before every write.

Convention: see _brain-filing-rules.md — when the gate's verdict is "write", the primary subject picks the directory.

Convention: skills/conventions/quality.md owns the cross-cutting page rules (citations, Iron Law back-linking, notability) — every page the gate lets through follows them. Gate-specific delta: the gate only decides write/link/skip; the admitting skill applies the quality rules on write.

The Rule

No content enters the brain without passing this gate. A raw cp or mv into the brain repo is a bug.

One insight, one place. If it already exists, link to it — don't clone it. Before any new page is written (file migration, bulk import, manual gbrain put, subagent output), two checks run in order:

1. Named-Entity Resolution Gate — is this about a named thing that already has a page under its chosen name?
2. Dedup Gate — does the brain already state this insight somewhere?

Scope honesty: this gate is a routing convention — the harness resolves it into context when an ingest-shaped intent matches, and a well-behaved agent follows it. Nothing in the gbrain runtime mechanically blocks an unenriched or duplicate write if the skill never loads.

Why gbrain needs this gate

The native pipeline does NOT do semantic dedup for you:

• gbrain import / gbrain sync skip only matching frontmatter IDs. Identical content under a different slug or ID indexes twice — every duplicate becomes a second search hit competing with the canonical page.
• gbrain capture's dedup is a 24-hour exact content-hash — it catches re-captures of identical bytes, not the same insight reworded.
• The remember verb dedupes facts, not pages.

Semantic dedup and named-entity resolution are this skill's job, in full.

When This Gate Fires

1. File migration — moving files already in the workspace into the brain repo ("move this to brain").
2. Bulk imports — batch moves of any kind into brain directories, BEFORE gbrain sync or gbrain import indexes them. For batches, also read conventions/test-before-bulk.md: gate 3-5 items and inspect the decisions before running the rest.
3. Manual writes — gbrain put or gbrain capture of rich content, or direct file writes into the brain repo.
4. Subagent output — background agents writing notes or pages into the brain.

What This Gate Owns vs Delegates

This skill is a gate, not a pipeline. It owns the pre-write checks below. Everything downstream of a "write" verdict is delegated to shipped skills — do not restate their steps here or inline:

Named-Entity Resolution Gate (runs FIRST)

Fires whenever the content is about a NAMED project, place, company, person, or anything someone "wants to build / found / make."

Vector similarity alone cannot be trusted to catch named-entity dupes: a page stored under its chosen NAME will not embed close to the generic English phrase someone happens to describe it with. The classic failure: a search for a descriptive phrase scores the canonical named page below the prose floor, so a duplicate stub gets written on top of a years-old page. Stored by named meaning; retrieval attempted by literal generic phrase.

The rules

1. Resolve registry-first, not by the generic phrase. gbrain's native registry is the entity surface:

   bashCopy code
   gbrain entity "<name>"        # zero-LLM card: page, aka list, near-miss suggestions
   

   A card hit means the page exists — STOP, link, don't clone. On a miss (or for concept-shaped nouns), fall through to gbrain query "<name>" --limit 3. If the brain also keeps an explicit index of named initiatives (e.g. a page under concepts/), read it before concluding anything is new.

2. Expand through aliases before searching. Named pages should carry an aliases: frontmatter list (generic label + chosen name + any nickname + signature phrase). Search EACH alias and the generic label, not just the phrase the user happened to say.

3. A vector score is a floor for prose, NEVER a gate for named things. If there is ANY plausible named match, open and read the candidate page (gbrain get <slug>) before concluding it doesn't exist. A named page can be the right answer at a score that would be a clear miss for prose.

4. When a NEW named thing appears, bake its aliases in the same write. Create the page with the full aliases: list so every future synonym resolves through gbrain entity. One frontmatter list covers all future phrasings — O(1), not a per-instance reminder.

Why a gate and not a memory note

A memory reminder ("query the real name, not the generic phrase") is a per-instance sticky note: it only works if it happens to be in hot context that turn, doesn't generalize to the next named entity, and rots. This skill loads when an ingest-shaped task routes here. Process rules belong in the triggered gate, not in hot memory.

Dedup Gate (runs SECOND)

Before writing ANY new page (for named things, the resolution gate above runs first and takes precedence):

1. Extract the core claim — 1-2 sentences capturing what's novel about the new content.

2. Search for it:

   bashCopy code
   gbrain search "<core claim>" --limit 5
   

3. OPEN AND READ the top hit (gbrain get <slug>). Never band on the score alone. Donor systems publish cosine cutoffs for this step — do NOT port them: gbrain search returns fused hybrid rank scores, not cosine similarity, and no numeric threshold maps across. The band comes from reading, not from the number.

4. Assign a band:

   Band	Meaning	Action
   clear-dup	The top hit already states the same insight about the same subject	STOP. Link to the existing page (gbrain link / gbrain timeline-add) instead of writing.
   plausible-dup	Same territory; possibly a new angle	Read both fully. Same insight → link, don't write. Genuinely new angle → write WITH a cross-link to the existing page.
   clear	Nothing in the top results covers the claim	Write normally through the delegated enrichment skills.

Decision tree

Copy code
New content to write
  ├─ Named thing? → Named-Entity Resolution Gate first
  │    (entity card → alias-expanded search → READ the candidate)
  ├─ Extract core claim (1-2 sentences)
  ├─ gbrain search "<core claim>" --limit 5
  └─ OPEN AND READ the top hit (gbrain get <slug>)
      ├─ clear-dup     → STOP. Link to existing. Report "duplicate".
      ├─ plausible-dup → Read both. Same insight?
      │    ├─ yes → STOP. Link to existing. Report "duplicate".
      │    └─ no  → Write with cross-link. Report "new angle".
      └─ clear         → Write via enrichment skills. Report "unique".

When to skip dedup

• Operational/state files — time-series records, not knowledge.
• Meeting transcripts — each meeting is unique by definition (entities INSIDE it still go through the named-entity gate via the delegated skills).
• Timeline entries on existing pages — back-links are additive, not duplicative.
• Media files — dedup by filename/hash, not semantic similarity.

Verification

After the batch, verify the gate's output holds:

bashCopy code
gbrain check-backlinks check            # mentioned entities link back (fix with: check-backlinks fix)
gbrain backlinks <new-slug>             # each new page has inbound links
gbrain search "<core claim>" --limit 3  # the insight has exactly ONE home

If check-backlinks check reports gaps on pages the gate just admitted, the enrichment delegation was skipped — route back through enrich before declaring the ingest done.

Contract

This skill guarantees:

• No new page enters the brain through this skill's flows without the named-entity resolution check and the dedup check running first.
• Every "duplicate" verdict names the matched slug and produces a link or timeline entry instead of a clone.
• New named-entity pages carry an aliases: frontmatter list in the same write that creates them.
• Dedup bands are assigned by READING the top hit, never by score alone; no numeric similarity thresholds are used against gbrain's fused scores.
• Enrichment is delegated to shipped skills (ingest, enrich, signal-detector, concept-synthesis) — never restated or reimplemented inline.
• Batches end with a gbrain check-backlinks check verification pass.
• Routing matches the canonical triggers in the frontmatter.
• Output written under the directories listed in writes_to:.
• Privacy contract preserved: no real names, no fork-specific filesystem path literals, no upstream-fork references.

The full behavior contract is documented in the body sections above; this section exists for the conformance test.

Output Format

One decision line per item checked, then the verification result:

Copy code
Ingest gate — 3 item(s) checked

| item | entity resolution | band | action |
|---|---|---|---|
| notes-on-widget-co.md | resolved: companies/widget-co | clear-dup | linked (timeline entry on companies/widget-co) |
| pricing-thesis.md | n/a (prose) | plausible-dup | new angle — written to concepts/ with cross-link to concepts/pricing-power |
| charlie-example-intro.md | miss (near-miss: people/charlie-example) | — | read near-miss; same person → linked, no new page |

Verification: check-backlinks check → 0 gaps on admitted pages

Every "linked" or "duplicate" row MUST name the matched slug. If any row says "written", the enrichment delegation (which skill handled it) should be recoverable from the conversation.

Anti-Patterns

• ❌ cp file.md <brain-repo>/concepts/ — raw copy, no gate, no enrichment.
• ❌ Bulk mv of a folder into the brain repo, then gbrain sync — sync happily indexes every duplicate; matching-ID skip will not save you.
• ❌ Trusting a low vector score as proof a named thing has no page — named pages don't embed near generic descriptions of them.
• ❌ Banding on the search score without opening the top hit.
• ❌ Porting numeric dedup thresholds from other systems onto gbrain's fused scores.
• ❌ Writing a new named page without its aliases: list — the next synonym creates the next duplicate.
• ❌ Reimplementing entity detection, backlinking, or concept linking inline instead of delegating to the shipped skills.
• ❌ Skipping the gate because the write is "just one page" via gbrain put — single manual writes are where duplicate stubs come from.

Dedup (sharp boundaries)

• capture — the quick-save front door; its dedup is a 24h exact content-hash on identical bytes. This gate is the SEMANTIC + named-entity layer for content entering the brain as real pages (migrations, bulk imports, inbox graduation). "capture this thought" → capture; "migrate these files into the brain" → this gate.
• ingest — the router for NEW external content (meetings, articles, media) and its enrichment pipeline. ingest decides what to DO with content; this gate decides whether a page should EXIST at all. The gate fires before the write; ingest and its specialized skills handle everything after a "write" verdict.
• enrich — page creation/update mechanics (tiers, citations, timelines, backlinks) AFTER this gate says "write" or "link".
• concept-synthesis — retroactive, at-scale dedup of concept stubs that already slipped in. This gate is prevention at write time; concept-synthesis is the cleanup pass. "dedupe my existing concepts" → concept-synthesis.
• frontmatter-guard (host-side) — the same standalone-gate pattern on an orthogonal axis: structural validity of what's written vs (here) semantic novelty of whether to write.
• bulk-ingestion — the bulk sibling. Its pipeline dedup key (source + source_id) only makes RE-RUNS idempotent; it does not catch cross-source duplicates or resolve named entities. This gate is the semantic + named-entity layer bulk-ingestion runs on its Phase 3 trial items and bakes into the codified pipeline (its Phase 1d/6). "Build a large-corpus pipeline" → bulk-ingestion; "does this page already exist before I write it" → this gate.
• data-loss-gate — the inverse gate: it stops data LEAVING the brain without confirmation; this gate stops data ENTERING without resolution + dedup.