PortalJS — Migrate
Overview
Harvest datasets from an external open-data platform into a PortalJS portal, or
push them into a CKAN instance. Every source reads into one canonical shape —
{ slug, namespace, name, description, resources[] } — then writes to the target
from that shape, so adding a source covers every target automatically.
Supported sources: CKAN, a DCAT-US /data.json catalog (DKAN, ArcGIS Hub,
data.gov), a DCAT / DCAT-AP RDF feed, Socrata, OpenDataSoft, or an ArcGIS
FeatureServer/MapServer. Supported targets: a static PortalJS catalog
(datasets.json, resources linked by URL or downloaded into Cloudflare R2) or a
CKAN instance over its write API.
This is the copy-into-the-portal path, the inverse of /portaljs-connect-ckan
(which reads the source live at build time): a one-time, re-runnable snapshot so
the portal stands alone and needs no backend.
Prerequisites
- An existing PortalJS portal for the static target:
datasets.json,package.json, andpages/[owner]/[slug].tsxmust exist (theportaljs-catalogtemplate). - Node 18+ and npm, with portal dependencies installed (
npm install). - Network access to the source platform's API or feed URL.
- For a CKAN target: a write API key in
CKAN_API_KEY— never hardcoded. - For
downloadcopy mode:git-lfsand a PortalJS Arc account (or a self-hosted Giftless endpoint) to push resource files to R2. - For
dcat-rdfsources:lib/metadata/dcat-harvest.tsanddcat-profiles.ts(copy fromexamples/portaljs-catalog/lib/metadata/if missing), run vianpx tsx.
Instructions
Full step-by-step workflow:
.claude/commands/portaljs-migrate.md.
- Gather input — source type, source URL, target (
static/ckan), portal directory, copy mode (link/download), filters. Interview instead of erroring. - Validate the target — confirm the static catalog files exist, or that the CKAN URL and API key authenticate.
- Detect the source type from the URL (or
--source) and verify reachability. - Read the source into canonical dataset entries, per the source's field table in
the reference doc, applying the metadata hygiene rules (license sentinel — never
fabricate CC-BY; description sanitizer + fallback chain; source dates preserved
with
migratedAtseparate;cleanTitledisplay-name cleanup with slugs from the raw title;categoryfrom a meaningful theme). - For a static target, resolve resource paths by copy mode:
linkkeeps source URLs;downloadcopies files into the repo under Git LFS, pushed to R2. - Print a dry-run preview; stop if
--dry-runwas passed. - Write
datasets.json(upsert or--replace), or push packages/resources into CKAN over its action API. - Verify —
npm run build(static) or re-querypackage_search(CKAN). - Report datasets migrated, namespaces touched, and next steps.
Output
Static target: an updated datasets.json (upserted on (namespace, slug)),
optionally with resource files in Git LFS served from Cloudflare R2, plus a
passing npm run build. CKAN target: created/updated packages and resources;
per-dataset failures are logged and skipped, not fatal.
Error Handling
| Symptom | Cause | Fix |
|---|---|---|
"not a portaljs-catalog template" | datasets.json missing | Confirm the portal directory, or run /portaljs-new-portal |
| CKAN target rejects every write (403) | Key missing, expired, or lacks org permission | Set a valid CKAN_API_KEY; stop rather than partially migrating |
| Source URL matches no known format | Auto-detection failed | Pass --source explicitly (ckan, dcat, dcat-rdf, socrata, ods, arcgis) |
| DCAT-RDF fetch returns HTML, not a feed | URL is a portal page, not the feed | Follow its <link rel="alternate"> href, or use the direct feed URL |
| Build fails after write | Malformed entry (missing slug/namespace/name) | Inspect /tmp/portaljs-migrate-build.log, fix, rebuild |
download mode push fails | git-lfs missing, or no Arc/Giftless token | Install git-lfs; mint a token via the Arc API or self-hosted Giftless |
| Duplicate datasets after re-running | --replace used unintentionally, or slug collisions | Omit --replace to upsert; keep slugs unique per namespace |
Examples
Example 1 — CKAN to static catalog, link mode
bash/portaljs-migrate --source ckan --source-url https://demo.dev.datopian.com \ --target static --portal-dir . --copy-mode link
Upserts all packages from the CKAN demo instance into datasets.json, referencing
source resource URLs directly.
Example 2 — data.gov DCAT-US catalog, downloaded into R2
bash/portaljs-migrate --source dcat --source-url https://catalog.data.gov/data.json \ --target static --copy-mode download --org-filter epa-gov
Harvests one publisher and copies every resource file into the repo under Git LFS, served from Cloudflare R2.
Example 3 — DCAT-AP RDF feed from a national portal
bash/portaljs-migrate --source dcat-rdf \ --source-url https://data.europa.eu/api/hub/search/catalog.jsonld \ --target static --dry-run
Fetches the JSON-LD feed through the RDF harvester and previews the canonical datasets without writing anything.
Example 4 — CKAN to CKAN (platform-to-platform)
bash/portaljs-migrate --source ckan --source-url https://old-portal.example.org \ --target ckan --target-url https://new-portal.example.org --owner-org research
Pushes every package from the old CKAN instance into the new one, filed under
research.
Resources
- Command source:
.claude/commands/portaljs-migrate.md - Field mappings and troubleshooting:
references/reference.md - Related skills:
/portaljs-connect-ckan,/portaljs-add-dcat,/portaljs-check-data-quality,/portaljs-define-schema - CKAN Action API reference: https://docs.ckan.org/en/latest/api/
