AI Model Privacy Audit
Overview
AI model privacy auditing is the systematic assessment of whether trained ML models leak information about their training data. Models can memorize individual training records, enabling adversaries to extract personal data, determine dataset membership, reconstruct input features, or infer sensitive attributes. This skill implements a comprehensive model privacy audit methodology using established attack techniques and tools (ML Privacy Meter, ART, Foolbox) to quantify privacy leakage before deployment and periodically during operation. The audit results feed directly into the AI DPIA risk assessment and inform mitigation measure selection.
Privacy Attack Taxonomy
1. Training Data Extraction
Objective: Extract verbatim or near-verbatim records from the model's training data.
| Attack Vector | Description | Target Models |
|---|---|---|
| Prompt-based extraction | Craft prompts that cause LLMs to regurgitate training data | Language models, generative models |
| Canary extraction | Insert known canary strings into training data and test if model reproduces them | Any model (testing methodology) |
| Gradient-based extraction | Use model gradients to reconstruct training inputs | Models with accessible gradients |
| Generative reconstruction | Use the model as an oracle to iteratively reconstruct training samples | GANs, VAEs, diffusion models |
Risk Factors Increasing Extraction Likelihood:
- Large model capacity relative to training data size (overfitting)
- Training data containing duplicated or near-duplicated records
- Longer training duration (more epochs)
- Lower regularisation
- Models with high output granularity (logits, probabilities)
Testing Methodology:
- Insert canary records with unique identifiers into training data
- Train the model
- Attempt extraction through various prompting strategies
- Measure extraction success rate (percentage of canaries recovered)
- Threshold: extraction rate should be below 0.1% for acceptable risk
2. Membership Inference
Objective: Determine whether a specific record was in the model's training set.
| Attack Type | Method | Computational Cost |
|---|---|---|
| Shadow model attack | Train shadow models on similar data, build a binary classifier on model outputs | High — requires training multiple shadow models |
| Metric-based attack | Use model confidence, loss, or entropy to distinguish members from non-members | Low — single model query per sample |
| Label-only attack | Use predicted labels (no confidence scores) to infer membership | Medium — requires multiple queries |
| Likelihood ratio attack (LiRA) | Compare per-sample loss to reference distributions | High — most accurate, requires multiple models |
ML Privacy Meter Implementation:
- Population metric-based attack: compares target model's loss on a sample against population loss distribution
- Reference metric-based attack: uses reference models to compute per-sample metrics
- Shadow model attack: trains shadow models and uses the attack model to classify members
Testing Methodology:
- Partition data: training set (members) and held-out set (non-members)
- Run membership inference attacks using ML Privacy Meter
- Measure attack success: true positive rate at low false positive rate (TPR@FPR=0.1%, 1%)
- Generate ROC curves per sample and aggregate
- Threshold: TPR@1%FPR should be below 5% for acceptable privacy
3. Model Inversion
Objective: Reconstruct input features from model outputs.
| Attack Type | Method | Target |
|---|---|---|
| Confidence-based inversion | Iteratively optimise input to maximise model confidence for a known label | Classification models |
| Gradient-based inversion | Use model gradients to reconstruct inputs from outputs | White-box models |
| GAN-based inversion | Train a GAN to invert model outputs to input space | Face recognition, image classifiers |
Testing Methodology:
- Select target classes or individuals
- Run inversion attacks with various initializations
- Measure reconstruction quality (SSIM, PSNR for images; cosine similarity for embeddings)
- Assess re-identification risk: can reconstructed data identify specific individuals?
- Threshold: reconstruction similarity should be below 0.3 (SSIM) for acceptable risk
4. Attribute Inference
Objective: Infer sensitive attributes not present in the model's output.
| Attack Type | Description |
|---|---|
| Correlation exploitation | Use correlated features to infer sensitive attributes from model behaviour |
| Partial knowledge attack | Attacker knows some attributes and uses model to infer remaining sensitive ones |
| Group inference | Determine statistical properties of training subgroups |
Testing Methodology:
- Identify sensitive attributes (Art. 9 categories) that may be correlated with model features
- Train attack models to predict sensitive attributes from model outputs
- Measure inference accuracy for each sensitive attribute
- Compare against random baseline
- Threshold: inference accuracy should not exceed random baseline + 10%
Audit Methodology
Phase 1: Audit Scoping (Days 1-3)
- Define audit scope: which models, what deployment context, what threat model
- Identify assets: training data, model artefacts, deployment infrastructure
- Define threat model: who are the adversaries, what access do they have?
- Black-box: API access only (queries and responses)
- Grey-box: API access plus model architecture knowledge
- White-box: Full access to model weights and architecture
- Select attacks based on threat model and model type
- Define success criteria (acceptable leakage thresholds)
- Obtain audit authorisation from model owner and legal
Phase 2: Environment Setup (Days 4-7)
- Set up isolated audit environment (no production data leakage)
- Install audit tools: ML Privacy Meter, ART (Adversarial Robustness Toolbox), custom scripts
- Obtain model access (API endpoint or model weights depending on threat model)
- Prepare member/non-member datasets for membership inference
- Prepare canary data for extraction testing
- Configure monitoring to log all audit queries
Phase 3: Attack Execution (Days 8-18)
For each selected attack:
- Configure attack parameters
- Execute attack against the target model
- Collect results (success rates, confidence intervals)
- Vary attack parameters to find worst-case leakage
- Document: attack configuration, results, computational cost
Phase 4: Analysis and Reporting (Days 19-25)
- Aggregate results across all attacks
- Calculate privacy risk scores per attack type
- Identify high-risk data subsets (records most vulnerable to extraction)
- Cross-reference with DPIA risk register
- Generate audit report with:
- Executive summary
- Attack results per category
- Risk assessment with GDPR alignment
- Recommended mitigations
- Residual risk after proposed mitigations
Phase 5: Remediation Validation (Days 26-30)
- If mitigations are applied (differential privacy, output perturbation, etc.)
- Re-run key attacks to validate mitigation effectiveness
- Document residual leakage post-mitigation
- Compare against acceptable thresholds
- Issue final audit certificate or remediation requirements
Privacy Leakage Thresholds
| Metric | Acceptable | Elevated | Unacceptable |
|---|---|---|---|
| Membership inference TPR@1%FPR | < 5% | 5-15% | > 15% |
| Training data extraction rate | < 0.1% | 0.1-1% | > 1% |
| Model inversion SSIM | < 0.3 | 0.3-0.6 | > 0.6 |
| Attribute inference accuracy above baseline | < 10% | 10-25% | > 25% |
Mitigation Measures
| Mitigation | Attacks Mitigated | Trade-off |
|---|---|---|
| Differential privacy (DP-SGD) | All — provides mathematical guarantee | Model accuracy reduction (calibrate epsilon) |
| Training data deduplication | Extraction, membership inference | One-time preprocessing cost |
| Regularisation (dropout, weight decay) | Membership inference, overfitting-related leakage | May affect model performance |
| Output perturbation | Model inversion, attribute inference | Reduces output precision |
| Confidence score rounding | Metric-based membership inference | Minor output precision loss |
| Model distillation | Extraction, membership inference | Requires additional training |
| Rate limiting | All query-based attacks | Affects legitimate use |
| Input/output PII filtering | Extraction of PII from generative models | May affect model utility |
Tools and Frameworks
| Tool | Purpose | Source |
|---|---|---|
| ML Privacy Meter | Membership inference auditing | github.com/privacytrustlab/ml_privacy_meter |
| IBM ART | Adversarial robustness and privacy testing | github.com/Trusted-AI/adversarial-robustness-toolbox |
| TensorFlow Privacy | Differential privacy training | github.com/tensorflow/privacy |
| Opacus | PyTorch differential privacy | github.com/pytorch/opacus |
| Google DP Library | Differential privacy algorithms | github.com/google/differential-privacy |
| Foolbox | Adversarial attack library | github.com/bethgelab/foolbox |
Enforcement Relevance
Model privacy auditing is not explicitly required by the GDPR or AI Act, but is effectively mandated through:
- Art. 35 DPIA: Risk assessment for AI systems must evaluate privacy leakage risks — auditing is the standard methodology
- Art. 32 Security: Appropriate technical measures to ensure security of processing — privacy auditing validates these measures
- AI Act Art. 9: Risk management for high-risk AI requires identification and mitigation of privacy risks
- AI Act Art. 15: Accuracy, robustness, and cybersecurity requirements — privacy attacks are a cybersecurity concern
- EDPB Guidelines 04/2025: Controllers must assess whether AI models have effectively anonymised training data — auditing tests this claim
Integration Points
- ai-dpia: Audit results feed into DPIA Phase 3 risk assessment
- ai-data-retention: Audit validates whether deletion from training data is effective
- ai-deployment-checklist: Pre-deployment privacy audit is a checklist requirement
- ai-federated-learning: Federated learning models require distributed privacy auditing
