Protein Design Qc

Protein Design Quality Control And Ranking

Plain-language role: Use this skill to decide which designs pass QC and which ones should move forward.

Critical Limitation

Individual metrics have weak predictive power for binding. Research shows:

• Individual metric ROC AUC: 0.64-0.66 (slightly better than random)
• Metrics are pre-screening filters, not affinity predictors
• Composite scoring is essential for meaningful ranking

These thresholds filter out poor designs but do NOT predict binding affinity.

QC Organization

QC is organized by purpose and level:

Each category has two levels:

• Metric-level: Calculated values with thresholds (pLDDT > 0.85)
• Design-level: Pattern/motif detection (odd cysteines, NG sites)

Quick Reference: All Thresholds

Design-Level Checks (Expression)

See: references/binding-qc.md, references/expression-qc.md, references/structural-qc.md

Sequential Filtering Pipeline

pythonCopy code
import pandas as pd

designs = pd.read_csv('designs.csv')

# Stage 1: Structural confidence
designs = designs[designs['pLDDT'] > 0.85]

# Stage 2: Self-consistency
designs = designs[designs['scRMSD'] < 2.0]

# Stage 3: Binding quality
designs = designs[(designs['ipTM'] > 0.5) & (designs['PAE_interaction'] < 10)]

# Stage 4: Sequence plausibility
designs = designs[designs['esm2_pll_normalized'] > 0.0]

# Stage 5: Expression checks (design-level)
designs = designs[designs['cysteine_count'] % 2 == 0]  # Even cysteines
designs = designs[designs['instability_index'] < 40]

Composite Scoring (Required for Ranking)

Individual metrics alone are too weak. Use composite scoring:

pythonCopy code
def composite_score(row):
    return (
        0.30 * row['pLDDT'] +
        0.20 * row['ipTM'] +
        0.20 * (1 - row['PAE_interaction'] / 20) +
        0.15 * row['shape_complementarity'] +
        0.15 * row['esm2_pll_normalized']
    )

designs['score'] = designs.apply(composite_score, axis=1)
top_designs = designs.nlargest(100, 'score')

For advanced composite scoring, see references/composite-scoring.md.

Tool-Specific Filtering

BindCraft Filter Levels

BoltzGen Filtering

bashCopy code
boltzgen run ... \
  --budget 60 \
  --alpha 0.01 \
  --filter_biased true \
  --refolding_rmsd_threshold 2.0 \
  --additional_filters 'ALA_fraction<0.3'

• alpha=0.0: Quality-only ranking
• alpha=0.01: Default (slight diversity)
• alpha=1.0: Diversity-only

Design-Level Severity Scoring

For pattern-based checks, use severity scoring:

Experimental Correlation

Key insight: Metrics work as filters (eliminating failures) not predictors (ranking successes).

Campaign Health Assessment

Quick assessment of your design campaign:

Failure Recovery Trees

Too Few Pass pLDDT Filter (< 5% with pLDDT > 0.85)

Copy code
Low pLDDT across campaign
├── Check scRMSD distribution
│   ├── High scRMSD (>2.5Å): Backbone issue
│   │   └── Fix: Regenerate backbones with lower noise_scale (0.5-0.8)
│   └── Low scRMSD but low pLDDT: Disordered regions
│       └── Fix: Check design length, simplify topology
├── Try more sequences per backbone
│   └── modal run modal_proteinmpnn.py --num-seq-per-target 32 --sampling-temp 0.1
├── Use SolubleMPNN instead of ProteinMPNN
│   └── Better for expression-optimized sequences
└── Consider different design tool
    └── BindCraft (integrated design) may work better

Too Few Pass ipTM Filter (< 5% with ipTM > 0.5)

Copy code
Low ipTM across campaign
├── Review hotspot selection
│   ├── Are hotspots surface-exposed? (SASA > 20Ų)
│   ├── Are hotspots conserved? (check MSA)
│   └── Try 3-6 different hotspot combinations
├── Increase binder length (more contact area)
│   └── Try 80-100 AA instead of 60-80 AA
├── Check interface geometry
│   ├── Is target flat? → Try helical binders
│   └── Is target concave? → Try smaller binders
└── Try all-atom design tool
    └── BoltzGen (all-atom, better packing)

High scRMSD (> 50% with scRMSD > 2.0Å)

Copy code
Sequences don't specify intended structure
├── ProteinMPNN issue
│   ├── Lower temperature: --sampling-temp 0.1
│   ├── Increase sequences: --num-seq-per-target 32
│   └── Check fixed_positions aren't over-constraining
├── Backbone geometry issue
│   ├── Backbones may be unusual/strained
│   ├── Regenerate with lower noise_scale (0.5-0.8)
│   └── Reduce diffuser.T to 30-40
└── Try different sequence design
    └── ColabDesign (AF2 gradient-based) may work better

Everything Passes But No Experimental Hits

Copy code
In silico metrics don't predict affinity
├── Generate MORE designs (10x current)
│   └── Computational metrics have high false positive rate
├── Increase diversity
│   ├── Higher ProteinMPNN temperature (0.2-0.3)
│   ├── Different backbone topologies
│   └── Different hotspot combinations
├── Try different design approach
│   ├── BindCraft (different algorithm)
│   ├── ColabDesign (AF2 hallucination)
│   └── BoltzGen (all-atom diffusion)
└── Check if target is druggable
    └── Some targets are inherently difficult

Too Many Designs Pass (> 50%)

Copy code
Suspiciously high pass rate
├── Check if thresholds are too lenient
│   └── Use stringent thresholds: pLDDT > 0.90, ipTM > 0.60
├── Verify prediction quality
│   ├── Are predictions actually running? Check output files
│   └── Are complexes being predicted, not just monomers?
├── Check for data issues
│   ├── Same sequence being predicted multiple times?
│   └── Wrong FASTA format (missing chain separator)?
└── Apply diversity filter
    └── Cluster at 70% identity, take top per cluster

Diagnostic Commands

Quick Campaign Assessment

pythonCopy code
import pandas as pd

df = pd.read_csv('designs.csv')

# Pass rates at each stage
print(f"Total designs: {len(df)}")
print(f"pLDDT > 0.85: {(df['pLDDT'] > 0.85).mean():.1%}")
print(f"ipTM > 0.50: {(df['ipTM'] > 0.50).mean():.1%}")
print(f"scRMSD < 2.0: {(df['scRMSD'] < 2.0).mean():.1%}")
print(f"All filters: {((df['pLDDT'] > 0.85) & (df['ipTM'] > 0.5) & (df['scRMSD'] < 2.0)).mean():.1%}")

# Identify top issue
if (df['pLDDT'] > 0.85).mean() < 0.1:
    print("ISSUE: Low pLDDT - check backbone or sequence quality")
elif (df['ipTM'] > 0.50).mean() < 0.1:
    print("ISSUE: Low ipTM - check hotspots or interface geometry")
elif (df['scRMSD'] < 2.0).mean() < 0.5:
    print("ISSUE: High scRMSD - sequences don't specify backbone")

Templates and Demo

• Executable scorer: scripts/protein_qc_score.py
• Sample metrics table: templates/protein-design-qc/design_metrics.csv
• Demo input table: examples/minimal-binder-campaign/inputs/design_metrics.csv

Inputs

• A CSV table of design metrics such as pLDDT, ipTM, PAE, scRMSD, and optional ESM or expression fields.
• Optional threshold overrides for stricter or more permissive filtering.
• A design campaign context so summary pass rates can be interpreted correctly.

Outputs

• Per-design pass or fail annotations across structural, binding, and expression checks.
• A composite score and ranked output table suitable for experimental triage.
• A campaign summary highlighting which filter stage is removing most candidates.

Next Step

Advance the top-ranked designs into ipsae or experimental testing, and use the summary to decide whether the generation stage needs adjustment.