Trl

TRL Library Reference

Package Structure

TRL is organized around a trainer hierarchy that extends Hugging Face transformers.Trainer.

Copy code
trl/
├── trainer/
│   ├── grpo_trainer.py       # GRPOTrainer
│   ├── grpo_config.py        # GRPOConfig
│   ├── sft_trainer.py        # SFTTrainer (supervised fine-tuning)
│   ├── dpo_trainer.py        # DPOTrainer (direct preference optimization)
│   ├── kto_trainer.py        # KTOTrainer (Kahneman-Tversky optimization)
│   ├── online_dpo_trainer.py # OnlineDPOTrainer
│   ├── utils.py              # Shared utilities (log probs, decoding, padding)
│   └── ...
├── models/
│   └── modeling_value_head.py  # Value head for PPO-style trainers
├── data_utils.py
├── commands/                   # CLI entry points
└── ...

Trainer Hierarchy

All TRL trainers extend transformers.Trainer:

Copy code
transformers.Trainer
├── SFTTrainer          # Supervised fine-tuning
├── DPOTrainer          # Direct preference optimization
├── GRPOTrainer         # Group relative policy optimization
├── KTOTrainer          # Kahneman-Tversky optimization
└── OnlineDPOTrainer    # Online DPO

Each trainer overrides compute_loss with its specific objective, and RL-based trainers (GRPO, OnlineDPO) additionally override training_step to add a generation phase before the optimization step.

Shared Utility Functions (trainer/utils.py)

These utilities are used across multiple trainers. Read the source before modifying; the contracts below are what callers rely on.

selective_log_softmax(logits, index)

Memory-efficient per-token log-probability. Equivalent in value to F.log_softmax(logits, dim=-1).gather(...) at the selected token positions, but avoids materializing the full vocab-sized tensor.

Contract:

• Input: logits [B, T, V], index [B, T]
• Output: log_probs [B, T], each entry a valid log-probability (i.e. non-positive)
• Must agree with F.log_softmax to within numerical tolerance on the same inputs

decode_and_strip_padding(input_ids, tokenizer)

Converts a batch of token ID tensors into the cleaned text strings that the reward function will score.

Contract:

• Input: input_ids [B, T], tokenizer
• Output: list[str] of length B
• Strips padding and decoder artefacts
• Handles any reasoning-block conventions the library supports; the exact policy for complete, incomplete, and absent reasoning markers is defined in the implementation

Other Utilities

• pad / pad_to_length — Pad tensors to equal or specific lengths
• Various tokenizer helpers for batch processing

Configuration System

All TRL configs extend transformers.TrainingArguments. Each trainer adds its own fields:

Available References