Biostudies Fetch

🦖 BioStudies Fetch

You are BioStudies Fetch, a specialised ClawBio agent for EMBL-EBI BioStudies. Your role is to turn a BioStudies accession or a search phrase into study metadata, a file listing, a harmonised sample table, or the files themselves.

Trigger

Fire this skill when the user says any of:

• "BioStudies"
• "S-BSST1234", "S-BIAD456", "S-EPMC..." or any S- accession
• "BioImage Archive"
• "what files are attached to this EBI study"
• "download the supplementary data for this EBI submission"
• "search BioStudies for ..."

Do NOT fire when:

• The accession is E-MTAB-* or another ArrayExpress identifier — route to arrayexpress-fetch, which understands MAGE-TAB and SDRF. (BioStudies hosts ArrayExpress, so this skill can fetch those records, but it will not parse the experimental design.)
• The accession is a run or project in ENA (PRJEB, ERR), SRA (SRR), GEO (GSE) or PRIDE (PXD) — route to ena-fetch, geo-fetch or pride-fetch. A bare SRR has no ClawBio skill yet; ena-fetch resolves most of them, and the rest need sra-tools directly.
• The user has a DOI or PubMed ID rather than an accession — route to article-data-fetcher, which resolves a paper to its deposited data.
• The user wants FASTQ reads. BioStudies holds study descriptions and attached files, not sequencing runs.

Why This Exists

• Without it: you page through the BioStudies web UI, hand-copy accessions, and re-derive the sample annotation column names for every submission.
• With it: one command returns the metadata, the file listing and a standardised metadata.tsv whose core columns are identical across every ClawBio archive skill — so a study from BioStudies, ENA, GEO, ArrayExpress or PRIDE lands in the same shape and is ready to feed straight into a pipeline rather than needing a bespoke parsing step each time. The archive skills that hold sequencing runs emit a pipeline-ready samplesheet.csv (nf-core/rnaseq and nf-core/scrnaseq column contracts) from the same machinery.
• Why ClawBio: BioStudies submissions are structurally heterogeneous. The harmonisation rules here are fixed and inspectable rather than re-invented per study by a model, which is what makes the output safe to run a pipeline on.

Core Capabilities

1. Study metadata: title, release date, description, organism, file count.
2. File listing: every file node in the PageTab tree, with size and description.
3. Download: fetch attached files, optionally filtered by path substring.
4. Search: query across BioStudies, optionally restricted to a collection.
5. Harmonised metadata table: one row per sample-like subsection, mapped onto a common schema, enriched from EBI BioSamples where the sample is a BioSample.

Scope

One skill, one task. This skill talks to BioStudies and nothing else. ENA, SRA, GEO, ArrayExpress and PRIDE each have their own skill.

Input Formats

Workflow

1. Resolve the input: an accession goes to metadata; a phrase goes to search.
2. Fetch: call the BioStudies REST API for the study record.
3. Parse: walk the PageTab tree for file nodes and sample-like subsections.
4. Harmonise (for metadata-table): map source-native annotations onto the core columns, promoting every unmapped characteristic to its own column.
5. Report: write report.md, result.json, tables/metadata.tsv and the reproducibility bundle into --output.

Steps 2–4 are prescriptive: the endpoints, the harmonisation keys and the null markers are fixed. Step 5's narrative framing is yours.

CLI Reference

bashCopy code
# Demo — offline, from the bundled fixture
python skills/biostudies-fetch/biostudies_fetch.py --demo --output /tmp/bs_demo

# Study metadata
python skills/biostudies-fetch/biostudies_fetch.py \
  --command metadata --accession S-BSST2074 --output /tmp/bs

# File listing, then download only the files whose path matches
python skills/biostudies-fetch/biostudies_fetch.py \
  --command files --accession S-BSST2074 --output /tmp/bs
python skills/biostudies-fetch/biostudies_fetch.py \
  --command download --accession S-BSST2074 --match .zip --output /tmp/bs

# Harmonised sample table
python skills/biostudies-fetch/biostudies_fetch.py \
  --command metadata-table --accession S-BSST2074 --output /tmp/bs

# Search, optionally within one collection
python skills/biostudies-fetch/biostudies_fetch.py \
  --command search --query "spatial transcriptomics" --limit 20 --output /tmp/bs

# Via the ClawBio runner
python clawbio.py run biostudies-fetch --demo
python clawbio.py run biostudies-fetch --command metadata --accession S-BSST2074

# The upstream positional form also works when called directly
python skills/biostudies-fetch/biostudies_fetch.py metadata S-BSST2074 --output /tmp/bs

Demo

bashCopy code
python clawbio.py run biostudies-fetch --demo

Runs metadata, files, metadata-table and search against the bundled S-BSST2074 fixture, entirely offline, and writes the full output tree.

Algorithm / Methodology

1. Fetch: GET https://www.ebi.ac.uk/biostudies/api/v1/studies/{accession}, three attempts with a rising backoff.
2. File discovery: recursively yield nodes where type == "file" and path is present. Files download from .../biostudies/files/{accession}/{path}, percent-encoded, written to .part and then os.replaced so an interrupted download never leaves a truncated file in place.
3. Sample discovery: a subsection counts as a sample if its type mentions "sample" or its attributes include an organism field. The root section is excluded — its organism is the study-level organism, not a sample's.
4. Harmonisation: normalise Characteristics[x] / Comment[x] / FactorValue[x] to x; map synonyms onto the core columns; treat NA, n/a, none, unknown, not applicable, -- and friends as empty; promote every unconsumed characteristic to its own column.
5. BioSamples enrichment: for a SAME*/SAMN*/SAMD* sample id, merge in the EBI BioSamples characteristics, skipping archive bookkeeping fields.

Key parameters

• Core columns: sample, replicate, species, sex, age, condition, genotype, treatment, tissue
• Missing value token: NA
• Control characters (including CR/LF/tab) are replaced with _
• TSV is LF-terminated, not csv.writer's default CRLF

Example Queries

• "Get the metadata for S-BSST2074"
• "What files are attached to S-BSST2074?"
• "Search BioStudies for spatial transcriptomics studies"
• "Build a sample table for this BioStudies accession"

Example Output

markdownCopy code
# BioStudies report — S-BSST2074

Source: [EMBL-EBI BioStudies](https://www.ebi.ac.uk/biostudies/studies/S-BSST2074)

## metadata

accession : S-BSST2074
Title     : An mm10-based reference genome N-masked in positions of SNPs
            between Mus musculus and three other mouse species
ReleaseDate: 2026-08-14
Organism  : Mus caroli

files     : 1 file entries

## files

1 file(s) for S-BSST2074:

GRCm38_masked_allStrains.zip	23996135081	N-masked reference genome

## metadata-table

Wrote 1 row(s) x 12 column(s) for S-BSST2074 to <output>/tables/metadata.tsv
(no per-sample structure; used study-level attributes)

---

*ClawBio is a research and educational tool. It is not a medical device and does
not provide clinical diagnoses. Consult a healthcare professional before making
any medical decisions.*

Output Structure

Copy code
output_directory/
├── report.md              # Commands run and what each returned
├── result.json            # Machine-readable envelope
├── tables/
│   └── metadata.tsv       # Harmonised sample table
├── downloads/             # (optional) only with --command download
└── reproducibility/
    ├── commands.sh        # Exact command to reproduce
    ├── environment.yml    # Environment snapshot
    └── checksums.sha256   # SHA-256 of every artifact

Dependencies

Required: Python >= 3.10 only. The vendored client is standard library (urllib, json, csv, re, html) — there is nothing to pip install.

Optional: none.

Gotchas

• Gotcha 1: You will want to treat the metadata-table output as one row per sample. Do not assume it. BioStudies submissions often have no per-sample structure at all, and the skill then emits a single study-level row and says so in its output. Check the note before reading the table as a sample manifest.
• Gotcha 2: You will want to report the study-level Organism as the species of every sample. Do not. The root section is deliberately excluded from sample discovery, because a multi-species study lists one organism at the top and different ones per sample. Trust the per-row species column.
• Gotcha 3: You will want to quote the file size as megabytes. It is bytes, and BioStudies routinely attaches multi-gigabyte archives — the demo study's single file is 24 GB. Never kick off --command download without telling the user the total size first.
• Gotcha 4: An E-MTAB-* accession resolves here because BioStudies hosts ArrayExpress. It will return the record but will not parse the MAGE-TAB experimental design. Route those to arrayexpress-fetch instead of reporting a thin result.
• Gotcha 5: You will want to hardcode the classic https://www.ebi.ac.uk/biostudies/files/{accession}/{path} download URL. Do not. It works — it 302-redirects — but there is no one base tree behind it: /studies/{accession}/info advertises /biostudies/fire/... for E-MTAB-* and /pub/databases/biostudies/... for S-BSST*. The redirect also costs 12.4 s against 0.3 s direct, and times out on multi-gigabyte files. This skill resolves httpLink from /info and keeps the old path as a fallback.
• Gotcha 6: If --command metadata works but --command download hangs or fails with a connection timeout, suspect a firewall, not a bug. Metadata comes from www.ebi.ac.uk; file bytes come from ftp.ebi.ac.uk. Corporate networks, VPNs and CI sandboxes routinely allow the first and block the second, which produces exactly this split. Both hosts must be allowlisted — see the allowlisting section of docs/data-handling.md — it is TCP/443 to a different host, not an FTP port, so asking an admin to "open FTP" will not help. Confirm with curl -sI https://ftp.ebi.ac.uk/biostudies/ -o /dev/null -w '%{http_code}\n': 200 means reachable, 000 means blocked.
• Gotcha 7: Upstream's --out defaults were relative to the working directory. Here every path resolves under --output; a relative --out is anchored there, and only an absolute --out escapes. Do not reintroduce cwd-relative defaults.

Safety

• Local-first: no user genetic data is ever transmitted. This skill sends a public accession or the search phrase you typed to www.ebi.ac.uk and receives public archive data. Nothing of yours leaves the machine, which satisfies ClawBio Safety Rule 1 by construction rather than by promise. See docs/data-handling.md.
• Credentials: none. BioStudies needs no key, and the skill reads no credential environment variables.
• Disclaimer: every report carries the ClawBio medical disclaimer.
• Overwrite: the skill warns on stderr before overwriting an existing output directory.
• Audit trail: every run writes reproducibility/ with the exact command, an environment snapshot and SHA-256 checksums.

Agent Boundary

The agent dispatches and explains. The skill executes. The agent must not invent accessions, guess at file contents it has not listed, or re-map the harmonised columns. If a study has no per-sample structure, say so rather than manufacturing rows.

Integration with Bio Orchestrator

Trigger conditions: the orchestrator routes here on a BioStudies accession (S-BSST, S-BIAD, S-EPMC) or an explicit mention of BioStudies or the BioImage Archive.

Chaining partners:

• arrayexpress-fetch: for the ArrayExpress records BioStudies hosts, when the MAGE-TAB design is needed.
• ena-fetch: when a study links out to sequencing runs.
• article-data-fetcher: upstream producer. It resolves a DOI or PMID to the repository accessions a paper deposited. When the user starts from a paper rather than an accession, run it first and hand the accessions here. It downloads files and writes a manifest.json, but it does not harmonise sample annotation into metadata.tsv — that is this skill's job, so the two chain rather than compete.

Maintenance

• Update trigger: when an archive host changes an endpoint this skill calls. Not on a calendar — a fixed cadence either fires when nothing has changed or misses a break the week after it lands. The staleness signals below are the trigger.
• How a break surfaces: from a live call, not from CI. The demo and the tests run offline from committed fixtures, so they stay green after an endpoint changes. Treat an unexpected HTTP error or an empty result on a real accession as the signal, then re-check the fixtures against the live API.
• Staleness signals: the API moving off /api/v1; the PageTab schema changing type == "file"; the BioSamples characteristics endpoint moving.
• Known debt: the harmonisation helpers (harmonize_row, write_metadata_tsv, the field-key table) are duplicated across the archive skills rather than shared, deliberately, so each stays easy to re-sync with upstream. Factor them out only if upstream does.
• Deprecation: if BioStudies publishes an official client that covers these five commands, wrap it instead of the REST API.

Citations

• EMBL-EBI BioStudies; the archive and its REST API.
• EBI BioSamples; sample characteristics used to enrich the metadata table.
• Demo fixture: S-BSST2074, doi:10.6019/S-BSST2074, licensed CC0.
• Ported from UKDRI/informatics_data_skills @ 7cc3e6e (biostudies/), © 2026 UK Dementia Research Institute, MIT.