Rag Pipeline

Bright Data — RAG / Search-Engine Pipeline

Use Discover as the retrieval layer for an LLM app or a custom search engine. Discover already returns intent-ranked, relevance-scored results with parsed page content, so it does the "search + fetch + clean" stage of RAG for you. This is a code/architecture skill built on the discover-api skill — read that for API mechanics (trigger/poll, modes, params, limits).

Pick the right neighbor: a written brief → live-research; markdown of specific URLs you already have → scrape; structured platform records → data-feeds.

Two architectures — choose first

Copy code
Does the corpus change every query, or is it a stable knowledge base?

  ├── Per-query, always-fresh ("ground each answer in live web data")
  │     → LIVE RETRIEVAL: Discover(include_content) at query time → top-k → LLM
  │       Pros: always current, no storage. Cons: per-query latency + cost.
  │
  └── Reused across many queries ("build a knowledge base / search engine")
        → INGESTION: Discover(include_content) → chunk → embed → vector store
          then at query time: embed query → vector search → (rerank) → LLM
          Pros: fast queries, cacheable. Cons: can go stale (re-ingest on a schedule).

Many systems do both: an ingested base for breadth + a live Discover call for freshness, merged before the LLM.

Live retrieval (web-grounded answers)

Pattern: on each user question, run Discover with a sharp intent, take the top-k by relevance_score, and pass their content as context to the LLM. The LLM cites the links.

javascriptCopy code
import { bdclient } from '@brightdata/sdk';
const client = new bdclient(); // BRIGHTDATA_API_TOKEN

async function retrieve(question, k = 6) {
  const res = await client.discover(question, {
    intent: `authoritative sources that directly answer: ${question}`,
    includeContent: true,
    numResults: Math.min(k * 2, 20),  // over-fetch, then trim
  });
  // NOTE: the JS SDK returns a WRAPPER object, not a bare array:
  //   { success, data: [ {link,title,description,relevance_score,content?} ], totalResults, cost, taskId, ... }
  // The result rows are in `.data` (CLI/REST use `.results` instead — see discover-api).
  if (!res.success) throw new Error(`discover failed: ${res.error ?? 'unknown'}`);
  return (res.data ?? [])
    .filter(r => r.content && !/just a moment|captcha|access denied|not found/i.test(r.content) && r.content.length > 200)
    .sort((a, b) => b.relevance_score - a.relevance_score)
    .slice(0, k);
}
// → build a prompt from sources[].content, ask the LLM to answer WITH [n] citations to sources[].link

Full prompt-assembly + citation pattern: references/code.md.

Ingestion (build a vector knowledge base / search engine)

Pattern: discover broadly (high volume — zeroRanking via REST is ideal here), chunk each page's content, embed the chunks, upsert into a vector store with the source URL as metadata. At query time: embed the query, vector-search, optionally rerank, then feed to the LLM.

Stages: discover → dedup → chunk → embed → upsert (ingest), then embed query → search → rerank → generate (serve). Provider-agnostic code for both stages, including chunking and metadata, is in references/code.md.

For bulk corpus building, prefer the raw REST "mode":"zeroRanking" flow (max raw results, no ranking) from the discover-api skill — but note it ignores num_results and does not support include_content, so you fetch content separately (Discover standard/deep with content, or the scrape skill).

Design rules

• Store provenance. Every chunk keeps its source link (and ideally title + relevance_score). RAG without citations is unverifiable.
• Chunk for the model, not the page. ~500–1500 tokens with overlap; split on headings/paragraphs, not mid-sentence.
• Validate content before embedding. Skip block pages and empty bodies (oversized PDFs return null content). Embedding garbage poisons retrieval.
• Over-fetch then trim by relevance_score. Discover's score is a strong prior for top-k selection before (or instead of) a reranker.
• Re-ingest on a schedule if freshness matters — web content drifts. The ingested base goes stale; live retrieval doesn't.
• Cap and dedup. num_results ≤ 20 per call; dedup by normalized URL across calls so one article via three aggregators isn't triple-weighted.
• Keep the embedder/vector store pluggable. Discover is the retrieval source; the embedding model and vector DB are your choice — don't hardwire one.

Verification gate

1. Retrieval returns non-empty, on-topic chunks for a known test query (eyeball top-k links).
2. No block-page / empty content made it into the index — spot-check stored chunks.
3. Citations resolve — every [n] the LLM emits maps to a real source link in the retrieved set.
4. Freshness is honored — if the app promises current data, confirm live retrieval (or a recent re-ingest), not a stale index.
5. Grounding check — answers are supported by retrieved content, not the model's prior; test with a question whose answer only exists in a retrieved page.

Red flags

• Building an ingestion pipeline when the user needs fresh answers (use live retrieval), or hammering Discover live when a cached index would do.
• Embedding content without filtering block pages / nulls.
• Dropping source URLs — you can't cite or refresh what you didn't store.
• Treating num_results as unlimited (cap 20) or expecting include_content under zeroRanking.
• Letting the LLM answer from training data — enforce "answer only from provided sources; if absent, say so."
• One giant chunk per page (kills retrieval precision) or mid-sentence splits.

References

• references/code.md — runnable JS + Python for both architectures: live retrieval with prompt+citation assembly, and the full ingestion pipeline (discover → dedup → chunk → embed → upsert → query), with a provider-agnostic embedder/vector-store interface.

Related skills

• discover-api — the retrieval API (trigger/poll, modes, include_content, limits). Read first.
• live-research — one-off synthesized report instead of a standing system.
• scrape — fetch markdown for specific URLs you already have.
• js-sdk-best-practices / python-sdk-best-practices — client.discover() option details and batch patterns.