Portaljs Check Data Quality

PortalJS — Check Data Quality

Overview

Run a read-only quality audit of one CSV or TSV file, local or remote, and return a structured JSON report. The audit profiles every column — null/blank counts, inferred value types, numeric ranges, likely year/date fields — and flags duplicate rows, duplicate values in identifier-like columns, ambiguous overlapping year columns (e.g. calendar year vs fiscal year), and mixed-type columns. It never edits the source file, datasets.json, or any other project file; it only reads the target file (a remote URL is downloaded to a temp file that is deleted before the run ends) and prints a report. Use it before publishing a dataset with portaljs-add-dataset, or to diagnose why a showcase renders wrong.

Prerequisites

• python3 on PATH — the audit logic runs as an embedded Python script; nothing is installed.
• One CSV or TSV file, given as a local path or an http/https URL. Only one file per run.

Instructions

The canonical, full step-by-step workflow is .claude/commands/portaljs-check-data-quality.md — the single source of truth. Read and follow it when executing. Summary:

1. Gather input — the file path or URL to audit. If missing, ask for it; never dead-end.
2. Resolve the source: if it's an http/https URL, download it to a temp file first; otherwise use the local path as given.
3. Validate the extension is .csv or .tsv. If not, or the file is missing, or the header row is empty, stop and surface the error JSON as-is — do not guess a fix.
4. Profile every column: null/blank counts, distinct values, sample values, inferred per-value type (boolean/integer/float/date/string), numeric min/max, and year range for columns whose name looks year-like.
5. Derive findings from the profiles — duplicate rows, missing-value ratios, invalid year values, mixed types, suspect negative values, duplicate identifier values, and ambiguous overlapping year columns — each tagged critical, warning, or info.
6. Assemble the JSON report (status, file metadata, findings, recommendations, column_profiles), print it, and clean up the temp file if one was created.
7. Relay the report to the user as-is; do not modify the source file, datasets.json, or any other project file based on the findings — that's a separate, explicit step.

Output

A single JSON object printed to stdout:

• status — ok, warning, or critical.
• file, file_name, source_type (local or url), row_count, column_count.
• findings — structured issues, most severe first.
• recommendations — de-duplicated suggested next steps.
• column_profiles — per-column summary (nulls, blanks, distinct count, sample values, inferred types, numeric/year ranges).

No files are created or modified. A remote URL's temp download is removed on exit, success or failure alike.

Error Handling

Examples

Example 1 — Audit a local CSV before publishing

Copy code
/portaljs-check-data-quality ./public/data/trash.csv

Example 2 — Audit a remote CSV over HTTPS

Copy code
/portaljs-check-data-quality https://example.com/trash.csv

Example 3 — Audit a TSV and save the report for review

bashCopy code
bash scripts/check-data-quality.sh ./data/emissions.tsv > /tmp/emissions-quality.json

Example 4 — Read a critical status report

jsonCopy code
{
  "status": "critical",
  "findings": [
    { "severity": "critical", "check": "duplicate_rows", "message": "42 duplicate rows found." }
  ],
  "recommendations": ["Review and deduplicate repeated rows if they are not intentional."]
}

Fix the flagged rows/columns, then re-run the audit before publishing.

Resources

• Full workflow: .claude/commands/portaljs-check-data-quality.md
• Detailed check catalog and troubleshooting: references/reference.md
• Related skills: portaljs-add-dataset, portaljs-define-schema
• Python csv module (parsing behavior this audit relies on): https://docs.python.org/3/library/csv.html