External Model Validation

External Model Validation

Input Validation

This skill accepts: an existing prognostic gene signature (model coefficient file with Gene and Coef columns), a bulk expression matrix in CSV format (genes as rows, samples as columns), and a clinical file with OS and OS.time survival columns.

If the user's request does not involve validating a pre-existing prognostic model on an external cohort — for example, asking to train a new model, perform feature selection, build a nomogram, run calibration curves, analyze single-cell data, or process data without survival endpoints — do not proceed with the workflow. Instead respond:

"external-model-validation is designed to validate an existing prognostic risk signature on an external bulk expression cohort with survival outcomes. Your request appears to be outside this scope. Please provide a fixed model coefficient file plus expression and clinical data with OS/OS.time columns, or use a more appropriate tool for model training, nomogram construction, or single-cell analysis."

When to Read External Files

Usage

bashCopy code
Rscript scripts/main.R \
  --exp_file ./expression.csv \
  --cli_file ./clinical.csv \
  --model_file ./model.csv \
  --output_dir ./output/ \
  --time_unit month \
  --seed 42

Arguments

When to Use

• You already have a fixed prognostic gene signature and coefficients.
• You need to test that model on an independent cohort with bulk expression and survival data.
• You want standard outputs for external validation: risk table, Kaplan-Meier curve, risk score plot, survival status plot, expression heatmap, and time-dependent ROC.

When Not to Use

• Do not use this skill to train or re-fit a prognostic model.
• Do not use it for nomogram construction, calibration curves, DCA, or diagnostic classification.
• Do not use it for single-cell expression matrices or cohorts without survival endpoints.
• Do not use identifiable patient data without de-identification and local compliance approval.
• Do not use for cohorts with very few events (fewer than 5 events may produce unreliable Kaplan-Meier and ROC results).

Research Use Notice

• This skill is for research and validation workflows only.
• It does not provide diagnosis, treatment recommendations, or clinical decision support.
• Use de-identified data and follow IRB, ethics, and data-use requirements before running on human cohorts.

Input Format

Expression Matrix (exp_file)

CSV with genes as rows and samples as columns. The first column must contain gene identifiers.

csvCopy code
"","Sample_1","Sample_2","Sample_3"
"TSPAN6",3.87,4.54,8.12
"TNMD",9.98,5.86,5.38
"DPM1",7.95,6.11,5.41

Clinical File (cli_file)

CSV with sample IDs as row names and at least OS and OS.time columns.

csvCopy code
,Age,OS,OS.time
Sample_1,59,0,133.5
Sample_2,60,0,49.13
Sample_3,59,1,22.40

• OS must use 0/1 encoding.
• OS.time must be positive and interpretable under --time_unit.

Model Coefficient File (model_file)

CSV with two required columns: Gene and Coef.

csvCopy code
Gene,Coef
TSPAN6,-0.25
TNMD,0.15
DPM1,0.32

Output Files

Workflow

Step 1: Validate Inputs

• Check required files and CSV extensions.
• Validate color strings, timeout, seed, KM break setting, and time unit choice.
• Parse --roc_times and --roc_cols.

Step 2: Build Matched Validation Dataset

• Read expression, clinical, and model files.
• Match samples shared by expression columns and clinical row names.
• Check all model genes exist in the expression matrix.
• Remove incomplete cases before downstream analysis.

Step 3: Calculate Risk Scores and Groups

• Compute risk scores with the supplied linear predictor.
• Convert follow-up time into years.
• Split patients into low and high groups using the median risk score.

Step 4: Generate Validation Outputs

• Save the full risk table and RDS object.
• Produce Kaplan-Meier, risk score, survival status, heatmap, and time-dependent ROC plots.
• Save session metadata and exact run parameters.

Methods

Risk Score Formula

For sample i, the skill computes:

textCopy code
riskScore_i = sum(expression_ig * coefficient_g)

using all genes listed in model_file.

Risk Stratification

• Samples are ordered by riskScore.
• The median risk score is used as the cutoff.
• Samples with scores above the median are labeled high; the others are labeled low.

Survival Analysis

• Kaplan-Meier curves are fit with survival::survfit.
• Group difference is shown with the default log-rank p-value in survminer::ggsurvplot.

Time-Dependent ROC

• ROC analysis is performed with timeROC::timeROC using follow-up time in years.
• All --roc_times values must be smaller than the maximum observed follow-up time.
• --roc_times is always interpreted in years, regardless of --time_unit.

Examples

Basic Usage

bashCopy code
Rscript scripts/main.R \
  -e tests/data/BRCA_data.csv \
  -c tests/data/BRCA_clinic.csv \
  -m tests/data/BRCA_coef.csv \
  -o ./output/

Input Follow-up Recorded in Days

bashCopy code
Rscript scripts/main.R \
  -e expression.csv \
  -c clinical.csv \
  -m model.csv \
  -o ./output \
  -u day \
  --roc_times 1,2,3

Note: --roc_times 1,2,3 means 1, 2, and 3 years — even though --time_unit day was supplied. The skill converts OS.time from days to years internally before ROC computation.

Custom Plot Colors and ROC Settings

bashCopy code
Rscript scripts/main.R \
  -e expression.csv \
  -c clinical.csv \
  -m model.csv \
  -o ./output \
  --col_high '#B2182B' \
  --col_low '#2166AC' \
  --roc_cols '#B2182B,#4D9221,#2166AC' \
  --roc_pos topleft \
  --km_breaks 2

Error Handling

Common Errors

IF error persists, READ: references/troubleshooting.md

Testing

Test with Included Data

bashCopy code
# Check CLI
Rscript scripts/main.R --help

# Run with bundled test data in a fresh output directory
Rscript scripts/main.R \
  -e tests/data/BRCA_data.csv \
  -c tests/data/BRCA_clinic.csv \
  -m tests/data/BRCA_coef.csv \
  -o ./output/

Validation Commands

bashCopy code
# Run R tests
Rscript tests/testthat.R

# Refresh the retained example output bundle
Rscript tests/refresh_example_output.R

# Inspect the generated risk table
wc -l tests/output/table/out_varifyRisk.txt

# Review the retained example outputs
ls -la tests/output/

Real-data Baseline

The repository stores a documented real-data baseline summary in references/baseline-run.md.

IF you need exact benchmark outputs or runtime expectations, READ: references/baseline-run.md

→ Directory structure and implementation details: references/project-structure.md