Segmenting Clinical Sections

Segmenting clinical sections

A clinical note is not flat text — it is a sequence of named sections (Chief Complaint, HPI, Past Medical History, Medications, Allergies, Assessment & Plan). The same phrase means different things in different sections: "diabetes" in PMH is historical context, "diabetes" in Assessment & Plan is an active problem, and "penicillin" under Allergies is an adverse-reaction flag, not a current medication. Splitting the note into canonical sections before NER or de-identification gives every downstream OpenMed step the context it needs to be more precise — and lets you process sensitive sections under stricter policies.

When to use

• You have a free-text note, H&P, progress note, or discharge summary and are about to run NER (extracting-clinical-entities) or de-identification.
• The user wants section detection, header parsing, LOINC section mapping, or per-section processing (e.g. "redact the Social History section harder").
• Downstream NER is over- or under-firing because it can't tell historical PMH mentions from active A&P problems.

Quick start

pythonCopy code
import re
import openmed

# Synthetic note.
note = """CHIEF COMPLAINT: chest pain.
HPI: 54M with 2 hours of substernal pressure.
PAST MEDICAL HISTORY: type 2 diabetes, prior MI 2019.
MEDICATIONS: metformin 500 mg BID.
ALLERGIES: penicillin (rash).
ASSESSMENT AND PLAN: acute coronary syndrome; start aspirin, admit."""

# Map common header variants -> canonical section + LOINC document-section code.
SECTION_MAP = {
    "chief complaint": ("Chief Complaint", "10154-3"),
    "hpi": ("History of Present Illness", "10164-2"),
    "history of present illness": ("History of Present Illness", "10164-2"),
    "past medical history": ("Past Medical History", "11348-0"),
    "medications": ("Medications", "10160-0"),
    "allergies": ("Allergies", "48765-2"),
    "assessment and plan": ("Assessment and Plan", "51847-2"),
}

HEADER_RE = re.compile(r"^(?P<h>[A-Z][A-Za-z /&]+):", re.MULTILINE)

# Split note into (canonical_label, loinc, body) chunks at each header.
chunks, matches = [], list(HEADER_RE.finditer(note))
for i, m in enumerate(matches):
    raw = m.group("h").strip().lower()
    label, loinc = SECTION_MAP.get(raw, (m.group("h").strip(), None))
    body_start = m.end()
    body_end = matches[i + 1].start() if i + 1 < len(matches) else len(note)
    chunks.append({"section": label, "loinc": loinc,
                   "text": note[body_start:body_end].strip()})

# Run NER per section — pass the section label downstream as context.
for c in chunks:
    ents = openmed.analyze_text(c["text"], model_name="disease_detection_superclinical",
                                output_format="dict")
    c["entities"] = ents

Each chunk now carries its canonical section label and LOINC code, so downstream context resolution can treat PMH findings as historical and A&P findings as active.

Workflow

1. Detect section headers. Use header heuristics: a line that is a known header phrase, often uppercase, ending in a colon, at line start. Maintain a synonym map (HPI ↔ History of Present Illness, PMH ↔ Past Medical History, A&P ↔ Assessment and Plan) so variants normalize to one canonical label.
2. Normalize to canonical labels and LOINC codes. Map each detected header to a canonical section name and a LOINC document-section code (e.g. HPI → 10164-2, PMH → 11348-0, Medications → 10160-0, Allergies → 48765-2, A&P → 51847-2). Unknown headers keep their literal text and a null code.
3. Chunk the note into (section, loinc, body) spans between consecutive headers, preserving original character offsets if you need to map results back.
4. Process per section. Run analyze_text / deidentify on each chunk and carry the section label forward. This is where precision is won: section-aware negation (PMH = historical) and section-specific de-id policy (Social History / Family History often warrant stricter redaction).
5. Reassemble with provenance. Tag each downstream entity with its source section so the problem-list and context layers can use it.

Hand-off to / from OpenMed

• To extracting-clinical-entities: feed each section chunk into openmed.analyze_text and attach the section label to every entity — section context measurably sharpens entity precision and downstream status assignment.
• To deidentifying-clinical-text: run openmed.deidentify per section so high-risk sections (Social/Family History) can use a stricter policy profile than the body.
• To resolving-clinical-context: the section label is a strong prior — PMH biases temporality toward historical, A&P toward recent/active. Pass it as part of the modifier window.
• To reconciling-problem-lists: section provenance (PMH vs. A&P) is a key signal for active-vs-resolved reconciliation.

Edge cases & gotchas

• Header variants are endless. "PMHx," "Past Med Hx," "PMH/PSH," inline headers without a colon, and run-on notes all appear. Keep the synonym map data-driven and fall back gracefully to the literal header for unknowns.
• Don't drop unsectioned text. Notes often start with un-headed preamble or have free text between sections. Capture it as an "unknown/other" chunk rather than discarding it, or you lose entities.
• LOINC is a binding, not a parser. LOINC document-section codes label the section; they do not detect it. Mapping is your responsibility and is user-supplied terminology — do not bundle LOINC content; reference codes only.
• Preserve offsets if you will re-merge entities into the original note for de-id; chunking loses position unless you track it.
• Local-first. All segmentation and per-section processing runs on-device.

Standards & references

• LOINC document-section codes (clinical document ontology): https://loinc.org/
• Denny et al., SecTag — a clinical note section tagger: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2655775/
• HL7 C-CDA section templates (canonical clinical document sections): https://www.hl7.org/ccdasearch/