DPO 2305.18290
logistic loss on log-ratiosIPO 2310.12036
resists overfitting on deterministic prefs
Every team rents the same foundation models. The difference is the data you train on, the reward you optimize and the knowledge you retrieve.
Prompt, retrieve, tune or reinforce. Each rung up the ladder costs more data, more money and more platform risk.
“you use the right optimization tool for the right job”
“RAG only solves our in-context learning issues”
Missing, stale or private facts. Use prompting and RAG.
Ovadia 2023: “RAG consistently outperforms” unsupervised fine-tuning for knowledge injection.
Gekhman 2024: models learn new facts slowly, and then hallucinate more.
Format, tone, style and consistent reasoning. Use fine-tuning, preference tuning or RL.
Balaguer 2024 (agriculture): fine-tuning gave +6 p.p. RAG added +5 p.p. on top. The two levers combine.
Read 20 to 50 failures. Ask one question per failure: is a fact missing, or is the behavior wrong?
RAFT: fine-tune with distractor documents so the model ignores noise and cites the right chunk. Vertex: SFT first, then preference tuning.
| Rung | Data needed | Cost and latency | Wins when |
|---|---|---|---|
| 1 · Prompt / context | 0 to tens of examples + a 20+ case eval | Hours. Prompt caching offsets long prompts | Always first; the task fits the model's priors |
| 2 · Long context | KB under ~200k tokens (~500 pages) | Very low upfront; high per call unless cached | Small, stable corpora |
| 3 · RAG / agentic RAG | A corpus, no labels; eval Q/A pairs | Days. + a retrieval hop (tens to hundreds of ms) | Fresh, large, private, permissioned facts |
| 4 · SFT (LoRA / full) | 50 to 500 (OpenAI ≥50; Vertex 100 to 500) | Tokens × epochs + hosting; can shrink prompts | Format, tone, narrow tasks, distillation |
| 5 · Preference tuning (DPO) | Pairs (chosen, rejected) | ~1.1x to 4x the SFT token price (Together) | Judgments with no single gold answer |
| 6 · RL / RFT | Tens to thousands of prompts + a grader | Highest: rollouts + training | Verifiable tasks: code, math, tools |
Rung 1 is context engineering: keep “the optimal set of tokens” in the window. Fight context rot with just-in-time retrieval, compaction and sub-agents.
Doc Anthropic, Effective context engineering for AI agentsSet the bar in money: +$20 per success, −$40 per escalation, −$1,000 churn in 5% of failures. Break-even accuracy is 81.5%. Below it, climb a rung.
Doc OpenAI, Optimizing LLM accuracyLoRA is usually enough: it matches full fine-tuning on small and medium SFT sets, and in RL even at rank 1. Anthropic has no public fine-tuning API.
Blog Thinking Machines, LoRA Without Regret“Inference on fine-tuned models will be disabled only when the underlying base model is deprecated.” RFT ran only on o4-mini, so it ends today.
Doc OpenAI, Deprecations“Managed Training no longer accepts new RFT jobs”. The Training API gives “Async GRPO with a custom rollout function” and multi-turn tool harnesses.
Doc Fireworks, Reinforcement fine-tuningYou write the loop with forward_backward, optim_step, sample and save. LoRA on 28+ models, from 1B to 1T+ parameters, dense and MoE.
| Provider | What you can tune, Oct 2026 | Trend |
|---|---|---|
| Vertex AI | SFT: Gemini 3.5 Flash-Lite, 3.5 Flash, 3.1 Flash-Lite, 2.5 Pro, 2.5 Flash-Lite, 2.5 Flash. PREFERENCE_TUNING | growth |
| Amazon Bedrock | SFT, RFT (Nova 2 Lite), distillation | growth |
| Together AI | LoRA, full fine-tuning, DPO on open models | open |
| Fireworks AI | Training API: async GRPO, custom rollouts | DIY RL |
| Tinker | forward_backward / optim_step, LoRA | DIY RL |
| OpenAI | gpt-4.1 family, existing customers only | sunset |
| Anthropic | No first-party fine-tuning; context, caching, Skills | context |
# Portable recipe: same JSONL, new provider
from google import genai
from google.genai import types
from google.genai.types import HttpOptions
DATA = "gs://your-gcs-bucket/your-tuning-data.jsonl"
client = genai.Client(
http_options=HttpOptions(api_version="v1"))
job = client.tunings.tune(
base_model="gemini-3.5-flash",
training_dataset=types.TuningDataset(gcs_uri=DATA),
config=types.CreateTuningJobConfig(
tuned_model_display_name="support-tone-v1",
epoch_count=1),
)
# Keep the eval set and the grader in your own repo.
SFT minimizes the cross-entropy of the target tokens. The data and the mask decide what the model learns.
Fine-tuning is “the most reliable way to enforce a specific output format”.
The sum skips tokens with label −100 (the ignore index).
Data quality beats size: LIMA used 1,000 curated examples.
A Jinja template adds role markers and special tokens. SFTTrainer applies it to conversational data: messages, or prompt + completion; tool_calls and images also work.
completion_only_loss=None (default): prompt-completion data gets loss on the completion only.assistant_only_loss=True: loss on assistant turns only. The template needs {% generation %} … {% endgeneration %}.Qwen/Qwen2.5-1.5B, set eos_token="<|im_end|>". If not, the model never learns to stop.packing=True; max_length (default 1024) sets the row length.packing_strategy: "bfd" (default, best-fit decreasing), "bfd_split", "wrapped".padding_free=True (FlashAttention): no attention across examples.
from peft import LoraConfig, get_peft_model
config = LoraConfig(
r=16,
lora_alpha=32,
target_modules="all-linear", # include MLP layers
lora_dropout=0.05,
use_rslora=True, # alpha/sqrt(r) scaling
use_dora=False, # True -> DoRA (merge before inference)
init_lora_weights="pissa_niter_4", # PiSSA init via fast SVD
task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()
Other init options in PEFT v0.21.0: olora, eva, corda, loftq, orthogonal.
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer
args = SFTConfig(
output_dir="qwen3-sft",
max_length=2048,
packing=True, packing_strategy="bfd", # padding-free BFD packing
assistant_only_loss=True, # mask user/system tokens
learning_rate=1e-4, # adapters -> higher LR
)
trainer = SFTTrainer(
model="Qwen/Qwen3-0.6B",
args=args,
train_dataset=load_dataset("trl-lib/Capybara", split="train"),
peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear"),
quantization_config=BitsAndBytesConfig(load_in_4bit=True,
bnb_4bit_quant_type="nf4"), # QLoRA
)
trainer.train()
v1.13.0: chunked-CE lm_head, 1.2–1.7× throughput. v1.15.0: fused LM head, selective activation checkpointing. v1.16.0: pad_to_multiple_of="auto".
Loss only on assistant turns. New loss option: loss_type="dft" (Dynamic Fine-Tuning, arXiv 2508.05629).
The docs say: “Combine with peft_config for QLoRA training.” Only the LoRA matrices train.
SFTConfig defaults: learning_rate=2e-5, gradient_checkpointing=True, bf16=True, logging_steps=10. CLI: trl sft …
| Pre-1.0, on SFTTrainer | TRL 1.x |
|---|---|
max_seq_length= | SFTConfig(max_length=) |
dataset_text_field=, packing= | in SFTConfig |
tokenizer= | processing_class= |
$ pip install "trl==1.16.0" "peft==0.21.0" bitsandbytes
$ pip show trl peft | grep Version
Upgrade one library at a time. Run a short smoke test with max_steps=10 after each upgrade.
| QLoRA page | google/gemma-4-E2B (or E4B, 12B, 31B, 26B-A4B) · synthetic text-to-SQL data · r=16, α=16, dropout 0.05 · saves lm_head + embed_tokens · 3 epochs, LR 2e-4, constant schedule |
| Cloud Run Jobs codelab | gemma-3-1b-it on Text2Emoji · one L4 GPU · r=8, α=16 on q_proj, v_proj · serve with vLLM on Cloud Run |
| Own your AI | Gemma 3 270M · QLoRA on a free T4 “in minutes” · 4-bit model under 300 MB · LiteRT / MediaPipe, or Transformers.js with WebGPU |
| Unsloth | “2× faster with 70% less VRAM” · r 16 or 32, α = r or 2r, LR 2e-4, 1–3 epochs · train_on_responses_only with <start_of_turn>user / <start_of_turn>model |
Gemma docs now point to gemma.ai. The Keras LoRA tutorial is marked deprecated.
Models: Gemini 3.5 Flash-Lite, 3.5 Flash, 3.1 Flash-Lite, 2.5 Pro, 2.5 Flash-Lite, 2.5 Flash. Data: JSONL in Cloud Storage, “at least 100 to 500 examples”. Knobs: epoch_count, adapter_size, learning_rate_multiplier (default 1). Run the first job with the defaults. For tuned tasks, turn thinking off or use the minimum budget.
import time
from google import genai
from google.genai import types
client = genai.Client(http_options=types.HttpOptions(api_version="v1"))
job = client.tunings.tune(
base_model="gemini-3.5-flash",
training_dataset=types.TuningDataset(gcs_uri="gs://b/tune.jsonl"),
config=types.CreateTuningJobConfig(
tuned_model_display_name="support-tone-v1", epoch_count=1))
done = {"JOB_STATE_SUCCEEDED", "JOB_STATE_FAILED", "JOB_STATE_CANCELLED"}
while job.state not in done:
time.sleep(60); job = client.tunings.get(name=job.name)
resp = client.models.generate_content(
model=job.tuned_model.endpoint, contents="why is the sky blue?")
A linear merge of 0.9 × the APO soup + 0.1 × a mid-training checkpoint recovered the long-context RULER scores.
| Task arithmetic | τ = θft − θbase; add or negate task vectors (2212.04089) |
| TIES | trim redundant values, elect the sign, average the values that agree (2306.01708) |
| DARE | drop a fraction p of the deltas, rescale by 1/(1−p); often dare_ties (2311.03099) |
| SLERP | merge exactly two models along the arc, not the chord |
mergekit (LGPL-3.0, 18+ methods): mergekit-yaml config.yml ./out. PEFT: add_weighted_adapter(…, combination_type="dare_ties", density=0.2).
SFT on teacher outputs, as in the DeepSeek-R1-Distill Qwen and Llama students. The student never sees its own mistakes.
The student writes, and the teacher scores every token. This removes the train/inference mismatch.
DistillationTrainerStable since v1.10.0 (2026-08-13). beta: 0.0 = forward KL, 0.5 = JSD, 1.0 = reverse KL (default). Teacher and student must share a vocabulary. Tool-result tokens are masked.
eos_token (for example "<|im_end|>" on a Qwen2.5 base). Use the same template in training and serving. Check the mask: the end-of-turn token must be unmasked.Three eras: RLHF with PPO (2022-23), direct preference optimization (2023-24), verifiable rewards with GRPO-family algorithms (2024-26). The 2026 frontier is agentic, multi-turn, tool-using RL with async rollouts.
“outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3”
−log σ(r(x,yw) − r(x,yl)).The closed-form optimal policy of the KL-regularized RLHF objective turns RLHF into a logistic loss. DPO “eliminat[es] the need for sampling from the LM during fine-tuning”.
logps/chosen can fall. Watch rewards/margins.
from trl import DPOConfig, DPOTrainer
from peft import LoraConfig
from datasets import load_dataset
args = DPOConfig(
beta=0.1, learning_rate=1e-5, # higher LR for adapters
loss_type=["sigmoid", "bco_pair", "sft"], # MPO
loss_weights=[0.8, 0.2, 1.0])
ds = load_dataset("trl-lib/ultrafeedback_binarized",
split="train")
trainer = DPOTrainer("Qwen/Qwen3-0.6B", args=args,
train_dataset=ds,
peft_config=LoraConfig())
trainer.train()
beta 0.1, learning_rate 1e-6, max_length 1024.ref_model=None uses the initial policy.sync_ref_model=True gives TR-DPO. label_smoothing gives Robust DPO.beta 0.01 to 0.5. Do SFT on the preferred responses first, then preference-tune.num_generations=8.Ai = (ri − mean(r)) / std(r). The group mean is the baseline. Then apply a PPO-style clipped ratio, with an optional KL term.Cost of the RL stage: about US$294K on 648 H800 GPUs, base model not included (Nature supplementary). The Register argues the true cost is far higher.
Set εlow 0.2 and εhigh 0.28. Rare tokens can gain probability. It “avoids entropy collapse”.
Drop prompts with group accuracy 0 or 1. They give zero advantage and zero gradient.
Average the loss over all tokens in the batch, not per sample. Long answers keep their weight.
Apply a soft length penalty. Max length 16,384 tokens, with a 4,096-token cache before it.
GRPO “artificially increases response length”, mostly for wrong answers. Fix: divide by a constant, drop the std term.
GSPO (Qwen, 2507.18071) uses sequence-level ratios and clipping. It stabilizes MoE RL and trained the Qwen3 models.
import re
from trl import GRPOConfig, GRPOTrainer
def correct_answer(completions, ground_truth, **kwargs):
texts = [c[0]["content"] for c in completions] # conversational
preds = [re.search(r"\\boxed\{(.+?)\}", t) for t in texts]
return [1.0 if p and p.group(1).strip() == gt else 0.0
for p, gt in zip(preds, ground_truth)]
def format_reward(completions, **kwargs):
return [0.2 if "<think>" in c[0]["content"] else 0.0
for c in completions]
args = GRPOConfig(loss_type="dr_grpo", scale_rewards=False, # Dr. GRPO
num_generations=8, beta=0.0, use_vllm=True)
trainer = GRPOTrainer(model="Qwen/Qwen2.5-0.5B-Instruct",
reward_funcs=[correct_answer, format_reward],
args=args, train_dataset=dataset)
trainer.train()
| field | default | other values |
|---|---|---|
| loss_type | "dapo" | dr_grpo, sapo, cispo, vespo |
| beta | 0.0 | no KL, no reference model |
| num_generations | 8 | group size G |
| scale_rewards | "group" | "batch"; False = Dr. GRPO |
| vllm_mode | "colocate" | "server"; set use_vllm=True |
list[float]. Unsloth: expect the reward to rise only after about 300 steps. Use 500+ rows and a model of 1.5B or larger.A reward can come from people, from a model trained on people, from a judge, from a rubric or from a program. The source sets the cost, the scale and the attack surface.
Tulu 3 named it: “a novel method we call Reinforcement Learning with Verifiable Rewards”.
math-verify)A program is cheap per label and hard to fool. But most real tasks (writing, medicine, law) are not verifiable. Judges and rubrics are the bridge.
RLAIF. Constitutional AI: the model critiques and revises its outputs with a list of principles, then RL uses AI preference labels. Lee et al.: AI labels perform comparably to human labels on summarization and dialogue.
An ORM scores only the final answer. A PRM scores each reasoning step.
“Let's Verify Step by Step”: the process-supervised model solves 78% of a MATH test subset. PRM800K has 800,000 step labels.
In 2025-26, teams use PRMs mostly for test-time reranking. DeepSeek-R1 did not train with PRMs.
Rubrics as Rewards (Scale AI): a judge grades a checklist for each prompt. Up to 28% relative gain on HealthBench-1k vs Likert scores.
OpenAI RFT graders: string_check, text_similarity, score_model, python, multi. The platform is winding down, so keep the grader as a design pattern. Bedrock RFT: reward in AWS Lambda or model-as-judge.
Fireworks: test the grader on full, partial, zero, malformed and edge-case answers before you train.
$ ray job submit --address="http://127.0.0.1:8265" \
--runtime-env-json='{"working_dir": "/openrlhf"}' \
-- python3 -m openrlhf.cli.train_ppo_ray \
--actor.model_name_or_path OpenRLHF/Llama-3-8b-sft-mixture \
--reward.model_name_or_path OpenRLHF/Llama-3-8b-rm-700k \
--algo.advantage.estimator reinforce_baseline \
--vllm.num_engines 4 --vllm.tensor_parallel_size 2 \
--train.colocate_all --ds.zero_stage 3 --ds.param_dtype bf16
Cursor (2025-09-12): policy gradient on live accept and reject signals. “we learn a policy that targets a particular accept rate”
AgentInstanceBase reset()/step(), --train.agent_func_path, --train.async_enable.forward_backward, optim_step).OpenAI RFT: Ambience ICD-10 0.39 → 0.57; Harvey citation F1 0.563 → 0.6765; SafetyKit F1 86% → 90%. Fireworks: Genspark +10% quality vs a SOTA closed model, +33% tool calls, ~50% cost cut.
rewards/margins and an eval. Add an SFT term (MPO) to anchor the chosen responses.dapo, beta=0, dynamic sampling. Run the tests in a sandbox and decontaminate the eval. Start from an SFT checkpoint of 1.5B or larger. Expect the reward to rise only after about 300 steps.Retrieval changes the tokens in the window, not the weights. It wins when knowledge is large, changing, permissioned or needs citations.
A chunk such as “revenue grew 3%” loses its company and its quarter. An LLM reads the full document and writes a short header. The header goes into the embedding and the BM25 text.
Source: Anthropic, 2024-09-19.
Failure rate = 1 − recall@20.
import agentplatform
from agentplatform import types
from google import genai
from google.genai import types as genai_types
client = agentplatform.Client(project=PROJECT, location="us-central1")
corpus = client.rag.create_corpus(display_name="policies")
client.rag.import_files(corpus.name, paths=["gs://bucket/policies/"],
transformation_config=types.RagFileTransformationConfig(
chunking_config=types.RagFileChunkingConfig(
chunk_size=512, chunk_overlap=100)))
ctx = client.rag.retrieve_contexts(rag_resources=[corpus.name],
rag_retrieval_config=types.RagRetrievalConfig(
top_k=3, vector_distance_threshold=0.5),
text="What is the refund window?")
rag_tool = genai_types.Tool(retrieval=genai_types.Retrieval(
vertex_rag_store=genai_types.VertexRagStore(...))) # corpus ref
gem = genai.Client(vertexai=True, project=PROJECT, location="us-central1")
resp = gem.models.generate_content(model="gemini-3.5-flash",
contents="What is the refund window?",
config=genai_types.GenerateContentConfig(tools=[rag_tool]))
1 LLM call. Use it for single-hop lookups with a tight latency budget.
Plan: split the question into sub-questions.
Choose a tool: vector search, SQL, web or an API. Retrieve.
Reflect: “enough evidence?” No. Re-query with a new tool.
Yes: answer and cite. Use the loop for multi-hop or ambiguous questions.
The agent keeps only paths and queries in context and loads the data with tools. Sub-agents return short summaries.
Genspark’s RFT-trained Deep Research agent: +33% tool calls, +10% quality vs a SOTA closed model.
| Criterion | Choose RAG | Choose fine-tuning |
|---|---|---|
| Freshness | Facts change often | The task spec is fixed |
| Corpus size | Over ~200k tokens | Knowledge is not the gap |
| Citations / ACL | Citations, audit trail, per-user ACLs | Not needed |
| Output format | Not the problem | Consistent format and style |
| Latency budget | +1 retrieval hop; more if agentic | Small model, short prompts |
| Data available | A corpus + eval Q/A pairs | Gold examples, pairs or a grader |
“RAG consistently outperforms” unsupervised FT for knowledge.
Fine-tuning on new facts makes hallucination rise.
Behavior task: FT 87 BLEU; FT + RAG 83.
FT gave +6 p.p., and RAG added +5 p.p. on top.
The model learns what the data covers. Gaps in coverage become gaps in capability, and missing provenance becomes legal risk.
In this chapter: document the data for people, machines and CI. Make the contract executable. Measure coverage against a taxonomy and name the gaps as slices. Then compare live data with a reference to find drift.
README.md + YAML (license, task_categories, tags). New in 2026: rl-environment.RecordSet / Field, semantics.
import mlcroissant as mlc
ds = mlc.Dataset(
"https://huggingface.co/api/"
"datasets/fashion_mnist/croissant")
print(ds.metadata.to_json()["name"])
Schema, semantics, SLAs and a change policy as code. CI runs it on every batch and stops the build when it fails. Pandera, Great Expectations, Soda, dbt, ODCS (next slide).
Training, validation and testing data must be “relevant, sufficiently representative, and to the best extent possible, free of errors and complete”.
Article 10 also asks for documented design choices, data origin, preparation steps and bias examination. A Data Card maps onto these items.
Fields, types, nullability, keys.
Allowed values, ranges, formats, units, meaning: “amount is gross EUR incl. VAT”.
Freshness, completeness, volume, uniqueness, an owner.
Versioned in git. CI finds breaking changes.
import pandas as pd
import pandera.pandas as pa # not: import pandera
class SFTRow(pa.DataFrameModel): # contract for an SFT shard
prompt: str = pa.Field(str_length={"min_value": 1})
response: str = pa.Field(str_length={"min_value": 1})
lang: str = pa.Field(isin=["en", "de", "fr"])
quality: float = pa.Field(ge=0.0, le=1.0)
class Config:
strict = True # no unexpected columns
SFTRow.validate(df, lazy=True) # collect all failures
$ datacontract lint odcs.yaml
$ datacontract test odcs.yaml # schema + quality on the live source
$ datacontract breaking v1.odcs.yaml v2.odcs.yaml
# v2 removes a column -> breaking change -> the CI job fails
| Need | Best fit |
|---|---|
| Validate frames in Python training code | Pandera 0.34.1 |
Expectation classes with severity, checkpoints | Great Expectations 1.24.1 |
YAML contract on a table, soda contract verify | Soda Core 4 |
SQL models: enforced: true, breaking-change gate | dbt model contracts |
| Vendor-neutral spec, executable SLAs | ODCS v3.1.0 + datacontract-cli |
Taxonomy = domain × task × skill × difficulty × language × format × risk category. Tag every row with its leaf ID.
GLAN (2024): a taxonomy is the only input; to add a skill, insert a node. #InsTag (2023): 6.6K open-set tags. TagLM, tuned on 6K tag-selected samples, beats far larger SFT sets on MT-Bench.
Also measure embedding coverage (production clusters with training mass), nearest-neighbour distance from production queries, and diversity (tags, distinct n-grams, self-BLEU).
slice: shipping_customs
predicate: "domain == 'shipping' and task == 'customs'"
min_eval_n: 200
metric: [task_success, groundedness]
gate: "task_success >= overall - 0.03"
owner: support-ml@company
| Evidently 0.7.23 default | n ≤ 1000 | n > 1000 |
|---|---|---|
| Numerical | ks (p < 0.05) | wasserstein (≥ 0.1) |
| Categorical | chisquare (z if binary) | jensenshannon (≥ 0.1) |
| Text | domain classifier (95th pct) | domain classifier (ROC AUC 0.55) |
Dataset drift: the share of drifted columns ≥ drift_share (default 0.5). Embedding drift (legacy API): model AUC 0.55, ratio 0.2, distance 0.2, mmd.
| Statistic | Definition | Default |
|---|---|---|
| PSI | Σ (ai − ei) · ln(ai / ei), 10 bins. Convention: < 0.1 stable, 0.1–0.25 moderate, > 0.25 major | 0.1 |
| KS | D = sup |Fref(x) − Fcur(x)|, gives a p-value. Numeric only | p < 0.05 |
| Jensen-Shannon | Distance: symmetric and bounded | 0.1 |
| Wasserstein | Earth mover’s distance ÷ reference std | 0.1 |
from sklearn import datasets
from evidently import Report
from evidently.presets import DataDriftPreset
from evidently.metrics import ValueDrift
iris = datasets.load_iris(as_frame=True).frame
report = Report([
DataDriftPreset(method="psi", drift_share=0.7),
ValueDrift(column="sepal length (cm)", method="ks"),
], include_tests=True)
my_eval = report.run(iris.iloc[:60], iris.iloc[60:]) # cur, ref
my_eval.save_html("drift.html")
Test sets contain errors, annotators disagree, and leaked splits inflate scores. Measure each one before you trust a number.
2,916 label errors in the ImageNet validation set (Northcutt et al. 2021). Confident learning flagged them. Humans confirmed about half of the candidates.
po is the observed agreement. pe comes from the label shares of each rater. The quadratic-weighted κ on the same data is 0.800, because it gives partial credit on ordinal scales. Kappa paradox: with skewed classes pe is high, so κ is low while raw agreement is high. Report both.
Observed over expected disagreement. It accepts any number of raters, missing ratings (grey cells), and nominal to ratio levels. Use it for crowd and LLM + human panels. Fleiss' κ needs a fixed number of raters per item.
Loop: guidelines → pilot 50–200 items with 3+ raters → α per label → adjudicate → revise → repeat until α ≥ 0.8 → scale with 5–10% overlap items. Low agreement can be signal on subjective tasks: keep soft or per-annotator labels.
import numpy as np, krippendorff
from sklearn.metrics import cohen_kappa_score
from statsmodels.stats.inter_rater import aggregate_raters, fleiss_kappa
a = [0, 1, 1, 2, 0, 2, 1]
b = [0, 1, 2, 2, 0, 2, 0]
cohen_kappa_score(a, b) # nominal: 0.588
cohen_kappa_score(a, b, weights="quadratic") # ordinal: 0.800
ratings = np.array([[0, 0, 1], [1, 1, 1], [2, 2, 1], [0, 0, 0]])
table, _ = aggregate_raters(ratings) # items x raters -> counts
fleiss_kappa(table, method="fleiss")
# Krippendorff: raters x items, np.nan = missing rating
rel = np.array([[0, 1, 2, np.nan], [0, 1, 2, 0], [1, 1, np.nan, 0]])
krippendorff.alpha(reliability_data=rel, level_of_measurement="nominal")
Landis and Koch (1977) bands for κ. They are widely cited, but the cut points are arbitrary.
Argilla 2.8.0 and Label Studio 1.23.2 show LLM pre-annotations as suggestions. A human accepts or corrects each one.
On corrected ImageNet labels, ResNet-18 outperforms ResNet-50 if the share of originally mislabeled test examples rises by 6%. The model ranking flips.
cross_val_predict.
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_predict
from cleanlab.filter import find_label_issues
from cleanlab.classification import CleanLearning
from cleanlab import Datalab
model = LogisticRegression(max_iter=1000)
# 1) out-of-sample probabilities (critical!)
pred_probs = cross_val_predict(model, X, labels, cv=5,
method="predict_proba")
# 2) ranked indices of likely label errors
issues = find_label_issues(
labels=labels, pred_probs=pred_probs,
filter_by="both", return_indices_ranked_by="self_confidence")
# 3) finds the issues, then trains on the cleaned data
cl = CleanLearning(model).fit(X, labels) # cl.get_label_issues()
# 4) Datalab: label issues, outliers, near-duplicates, non-IID
lab = Datalab({"X": X, "y": labels}, label_name="y")
lab.find_issues(pred_probs=pred_probs, features=X)
lab.report()
pred_probs columns must be in the order of the class labels. A wrong column order makes every label look wrong.Record p(true label | x) at the end of each epoch. Confidence = the mean over epochs. Variability = the standard deviation.
| region | signature | use |
|---|---|---|
| easy-to-learn | high confidence, low variability | largest region; helps optimization |
| ambiguous | high variability | helps out-of-distribution generalization; train on about 33% |
| hard-to-learn | low confidence, low variability | often label errors: send to the relabel queue |
Loop: seed → train → score the pool → select a batch of B → a human or an LLM labels it → add → repeat until the budget ends.
Least confidence 1 − max p, margin p1 − p2, or entropy. It is cheap. But near points get the same score, so one batch holds redundant examples.
Core-set (Sener and Savarese, ICLR 2018): greedy k-center in embedding space. BADGE (Ash et al., ICLR 2020): embed each point as the last-layer loss gradient, with the predicted label as a hypothetical label. The norm gives uncertainty, the direction gives diversity. Then run k-means++ seeding. It “balances uncertainty and diversity” with no hand-tuned trade-off.
Uncertainty: LLM disagreement or self-consistency. Diversity: sentence embeddings. Route to an LLM annotator or a human. Problems: cold start; sampling bias (test on a separate random sample); small gains over random with strong embeddings.
Labeling functions (rules, regexes, LLM prompts) vote or abstain. The LabelModel learns the accuracy of each function from their agreements, without ground truth. An end model then trains on the probabilistic labels.
Several scans per patient. A random split puts one patient in both folds.
Train on the past, test on the future. Each training set is a superset of the one before. A gap stops boundary bleed.
A scaler, imputer, SMOTE or vocabulary fit on all rows sees the test set. Fit each step inside the fold.
| leakage type | example | fix |
|---|---|---|
| entity / group | same patient, user, speaker or repo in both splits | GroupKFold |
| temporal | train on the future; "next 30 days" features | TimeSeriesSplit |
| preprocessing | scaler, imputer, SMOTE fit before the split | Pipeline |
| near-duplicate | the same or a paraphrased item in both splits | MinHash, embeddings |
| target | a proxy for the label, e.g. “treatment given” | feature audit |
| contamination | the test set is in the pretraining corpus | fresh, dated sets |
from sklearn.model_selection import GroupKFold, StratifiedGroupKFold
from sklearn.model_selection import TimeSeriesSplit, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
pipe = make_pipeline(StandardScaler(), # fit per fold
LogisticRegression(max_iter=1000))
# entity: no patient_id in both train and test
cross_val_score(pipe, X, y, groups=patient_id,
cv=GroupKFold(n_splits=5))
cross_val_score(pipe, X, y, groups=patient_id, # + class balance
cv=StratifiedGroupKFold(n_splits=5))
# temporal: rows sorted by time; the gap stops boundary bleed
tscv = TimeSeriesSplit(n_splits=5, gap=7)
cross_val_score(pipe, X_sorted, y_sorted, cv=tscv)
scikit-learn: i.i.d. data “rarely holds in practice”; with group structure “it is safer to use group-wise cross-validation”. For LLM and RAG evals, hold out by time: test only on documents and questions created after the training cutoff.
Docs scikit-learn: Cross-validation: evaluating estimator performance
from datasketch import MinHash, MinHashLSH
def mh(text, num_perm=128, n=5):
toks = text.lower().split()
m = MinHash(num_perm=num_perm)
for sh in {" ".join(toks[i:i + n])
for i in range(max(1, len(toks) - n + 1))}:
m.update(sh.encode("utf8"))
return m
lsh = MinHashLSH(threshold=0.8, num_perm=128)
keep = []
for doc_id, text in corpus.items():
m = mh(text)
if lsh.query(m): # near-dup of a kept doc
continue
lsh.insert(doc_id, m); keep.append(doc_id)
GroupKFold on the patient id. Fit all preprocessing inside a Pipeline. Then search for near-duplicate images across the splits.From 2023 to 2026 most gains came from model-based filtering and verified synthetic data, not from more raw tokens.
PyPI, Oct 2026: datatrove 0.10.1 · nemo-curator 1.4.0 · ragas 0.4.3 · distilabel 1.5.3 (no PyPI release since Jan 2025)
Positives: OpenHermes 2.5 and top r/ExplainLikeImFive posts. Keep the top 10%.
Core score: fastText 30.2 · perplexity 29.0 · AskLLM 28.6 · RefinedWeb 27.5
DCLM-7B gets 64% MMLU after 2.6T tokens. Llama 3 8B gets 66% with 6.6x more compute. “model-based filtering is key”.
A large LLM labels about 0.5M documents on a rubric. A small classifier learns the labels. Then the classifier runs over trillions of tokens at a low cost.
6.3T tokens from a classifier ensemble and synthetic rephrasing. Its 8B model gets +5 MMLU over Llama 3.1 8B. The authors say that a 90% cut hurts long training horizons.
from transformers import AutoTokenizer
from transformers import AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained(
"HuggingFaceTB/fineweb-edu-classifier")
model = AutoModelForSequenceClassification.from_pretrained(
"HuggingFaceTB/fineweb-edu-classifier")
text = "This is a test sentence."
inputs = tokenizer(text, return_tensors="pt",
padding="longest", truncation=True)
outputs = model(**inputs)
logits = outputs.logits.squeeze(-1).float().detach().numpy()
score = logits.item()
result = {"text": text, "score": score,
"int_score": int(round(max(0, min(score, 5))))}
keep = result["int_score"] >= 3 # the card advice
exclusion_writer. Put each removal rate on the quality card.The LLM writes new tasks from seed tasks. Drop near-duplicates. +33% absolute on Super-NaturalInstructions over vanilla GPT-3.
Rewrite each instruction “step by step into more complex instructions”. WizardLM: >90% of ChatGPT capacity on 17 of 29 skills (GPT-4 judge).
Send the template only up to the user slot. The model writes the query. 4M made, 300K kept; Base + Magpie SFT is close to Llama-3-8B-Instruct on some tasks.
1B personas; 370M “elite personas” (2025-02-18). Tulu 3 SFT, 939,344 rows: persona MATH 149,960 · GSM 49,980 · Python 34,999 · IF 29,980.
from distilabel.models import TransformersLLM
from distilabel.steps.tasks import MagpieGenerator
generator = MagpieGenerator(
llm=TransformersLLM(
model="meta-llama/Meta-Llama-3-8B-Instruct",
magpie_pre_query_template="llama3",
generation_kwargs={"temperature": 1.0,
"max_new_tokens": 256},
device="mps",
),
only_instruction=True,
num_rows=5,
)
generator.load()
result = next(generator.process())
Judges are biased. GPT-4 judges agree with humans “over 80%”, but they show position, verbosity and self-enhancement bias. Swap the answer order. Use rubrics and reference answers. Use a judge from another model family.
RLVR (Tulu 3) moves the same check into the RL reward: the reward is 1 only if the answer is verifiably correct.
import subprocess, tempfile, textwrap
def passes_tests(solution: str, tests: str, timeout=5) -> bool:
prog = solution + "\n\n" + textwrap.dedent(tests)
with tempfile.NamedTemporaryFile("w", suffix=".py",
delete=False) as f:
f.write(prog)
try:
r = subprocess.run(["python", "-I", f.name],
capture_output=True, timeout=timeout)
return r.returncode == 0
except subprocess.TimeoutExpired:
return False
def rejection_sample(prompt, tests, generate, k=16):
cands = [generate(prompt, temperature=0.8) for _ in range(k)]
good = [c for c in cands if passes_tests(c, tests)]
# keep all passing (SFT) or pair passing/failing (DPO)
return good
python -I and a timeout are not a sandbox. Llama 3 ran its tests in containers.Replace: training on model output causes “irreversible defects in the resulting models, where tails of the original content distribution disappear”.
Accumulate: “accumulating the successive generations of synthetic data alongside the original real data avoids model collapse”.
SFT distillation from a strong teacher: “none gave significant performance improvements” (OpenThoughts).
RLVR rewards and code: verification is essential. Tulu 3 puts it in the reward; Llama 3 kept only code that passed all checks.
Anchored synthesis works: Phi-4 “substantially surpasses its teacher model on STEM-focused QA”.
source=synthetic.Collapse by curation: most labs now filter for “educational-looking” text. That risks one narrow style and lost tails.
RAGAS 0.4.3 loads each document as a node of a KnowledgeGraph.
default_transforms and apply_transforms add summaries, entities, embeddings and relationships. Docs example: 10 nodes become 48 nodes with 605 relationships. The graph gives scenarios. Each scenario gives a query.
Default mix: 50% single-hop specific, 25% multi-hop abstract, 25% multi-hop specific. Each row has four columns:
import openai
from langchain_community.document_loaders import DirectoryLoader
from langchain_openai import ChatOpenAI
from ragas.embeddings import OpenAIEmbeddings
from ragas.llms import LangchainLLMWrapper
from ragas.testset import TestsetGenerator
docs = DirectoryLoader("Sample_Docs_Markdown/", glob="**/*.md").load()
llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
embeddings = OpenAIEmbeddings(client=openai.OpenAI())
generator = TestsetGenerator(llm=llm, embedding_model=embeddings)
dataset = generator.generate_with_langchain_docs(docs, testset_size=10)
dataset.to_pandas()
# user_input, reference_contexts, reference, synthesizer_name
“Fundamentally, evaluations are experiments.”
The eval set decides what ships, so it needs more care than the training set.
“Error analysis is the most important activity in evals.”
One benevolent dictator owns the quality definition. Align the judge to that expert (TPR / TNR). Gate CI on each slice. Watch criteria drift: “grading outputs helps users define criteria” (Shankar et al., arXiv 2404.12272). Evals before code? The FAQ says “Generally no”: the loop is error-analysis-first.
| property | rule |
|---|---|
| Source mix | Production traces + targeted synthetic edge cases from a dimensions × values grid. A human validates each synthetic item. |
| Size | 100–500 to start. n = 400, p = 0.8 → 95% CI ≈ ±3.9 pts. |
| Slices | Intent, language, user tier, length, failure mode, safety. |
| Decontam. | n-gram + embedding dedup against SFT and training data. Keep a private holdout. |
| Versions | Immutable (golden_v3.2), changelog, one owner. |
| Splits | dev (iterate) / test (report) / judge-calibration (expert labels). |
| Refresh | Add new failure modes monthly. Retire saturated items. |
proportion_confint(method="wilson").At α = 0.05 and power 0.8, the minimum detectable effect is ≈ 2.8 × SE(Δ). Unpaired, p = 0.8, n = 500 per arm: SE(Δ) ≈ 0.025, so you detect only ≈ 7-point deltas. Pairing shrinks this.
# Paired bootstrap on the same golden set
import numpy as np
rng = np.random.default_rng(0)
a = np.load("scores_A.npy") # per-item scores, same order
b = np.load("scores_B.npy")
d = b - a
boots = np.array([
rng.choice(d, size=len(d), replace=True).mean()
for _ in range(10_000)])
lo, hi = np.percentile(boots, [2.5, 97.5])
print(f"delta={d.mean():+.3f} 95% CI [{lo:+.3f}, {hi:+.3f}] "
f"P(delta<=0)={np.mean(boots<=0):.3f}")
SciPy: scipy.stats.bootstrap((d,), np.mean, confidence_level=0.95, method="BCa"). For clustered items, resample the clusters (passages, conversations), not the items.
GPT-4 as a judge reaches “over 80% agreement, the same level of agreement between humans.”
Zheng et al. also find position, verbosity and self-enhancement bias, and limited reasoning.
Managed: Vertex Gen AI evaluation service. The default GENERAL_QUALITY is an adaptive rubric metric: rubrics per prompt, then judged. AutoSxS runs side-by-side checks for migrations.
Pairwise: Chatbot Arena (240K+ votes) fits Bradley-Terry, P(i beats j) = σ(βi − βj), with bootstrap CIs and style control.
In pairwise grading, the judge prefers the answer in one fixed position.
Swap A and B. Count a win only if it holds in both orders. Else score a tie.
The judge gives longer answers higher scores.
Use length-controlled scoring (Length-Controlled AlpacaEval, arXiv 2404.04475). Write “do not reward length” in the rubric.
The judge prefers outputs of its own model family (self-enhancement).
Use a judge from a different model family than the candidate. The bias is not fully solved.
A Likert score does not say what “good” means, so labels vary.
One pass/fail judge per failure mode. G-Eval (arXiv 2303.16634) adds a CoT rubric; DeepEval GEval uses it.
Rubric drift → freeze and version the judge prompt like code. Never validated → expert calibration split with TPR, TNR, κ. Reference leakage → grade against a reference only where one exists.
θ = (p_obs + TNR − 1) / (TPR + TNR − 1)
Bootstrap the CI over the calibration items and the eval items together.
from sklearn.metrics import cohen_kappa_score, confusion_matrix
# human, judge: 0/1 labels on the calibration split
k = cohen_kappa_score(human, judge)
tn, fp, fn, tp = confusion_matrix(
human, judge, labels=[0, 1]).ravel()
tpr, tnr = tp/(tp+fn), tn/(tn+fp)
print(f"kappa={k:.2f} TPR={tpr:.2f} TNR={tnr:.2f}")
# p_obs: the judge pass rate on the eval set
theta = (p_obs + tnr - 1) / (tpr + tnr - 1) # Rogan-Gladen
“Modern neural networks, unlike those from a decade ago, are poorly calibrated.”
ECE = Σb (|Bb| / n) · |acc(Bb) − conf(Bb)|
Use 10 or 15 equal-width bins. ECE depends on the binning, so also report the Brier score or log loss, and show a reliability diagram.
Temperature scaling, a one-parameter form of Platt scaling, “is surprisingly effective”. Or use CalibratedClassifierCV(method="sigmoid") or "isotonic".
LLM uses: guardrail and classifier heads; verbalized or token-probability confidence for abstention; judge confidence to send hard items to humans.
With calibrated p, flag when
p > t* = CFP / (CFP + CFN)
A miss costs 20× a false alarm → t* = 1/21 ≈ 0.048.
EC(t) = π·FNR(t)·CFN + (1 − π)·FPR(t)·CFP
π is the prevalence in production. On the ROC curve, the optimum touches an iso-cost line of slope CFP(1 − π) / (CFN·π).
import numpy as np
from sklearn.metrics import roc_curve
fpr, tpr, thr = roc_curve(y_true, scores)
pi, C_fp, C_fn = 0.02, 1.0, 20.0 # prod prevalence, costs
ec = pi*(1-tpr)*C_fn + (1-pi)*fpr*C_fp
best = np.argmin(ec)
print(f"threshold={thr[best]:.3f} TPR={tpr[best]:.2f} "
f"FPR={fpr[best]:.3f} EC={ec[best]:.4f}")
Design, generate, select, curate, label, denoise and document every release. Then measure what still breaks: reward hacking, contamination, collapse, licensing and platform risk.
“The model flow is the full lifecycle of an LM: every stage, checkpoint, dataset, and dependency.”
Write the taxonomy, slices and contract first: ODCS v3.1.0 with datacontract-cli, or Pandera 0.34.1 in Python.
Mix logs with synthetic rows: distilabel Magpie, personas, teacher distillation (TRL DistillationTrainer).
Score with a FineWeb-Edu-style classifier. Target with DSIR or LESS: the LESS 5% often beats the full set.
MinHash + semantic dedup, then decontaminate. datatrove 0.10.1; NeMo Curator 1.4.0 GPU fuzzy dedup: 10.7 h → 0.65 h.
Experts accept or fix LLM suggestions in Label Studio 1.23.2. Measure agreement before you scale.
Find label issues with cleanlab 2.9.0. Run verifiers and unit tests on every synthetic row.
TRL v1.16.0, Unsloth or Vertex tuning. Inspect AI 0.3.283, lm-eval 0.4.13, DeepEval 4.3.1, promptfoo, Vertex client.evals.
Data Card + Croissant-RAI + eval card. Evidently 0.7.23 tests drift on descriptors.
Datasheets ask why the data exists, what is in it and how it was collected.
Paper Datasheets for DatasetsModel Cards: intended use, factors, metrics, evaluation data, quantitative analyses, caveats.
Paper Model Cards for Model Reporting“documentation must be treated as a user-centric product in and of itself”
Paper Data Cards (FAccT 2022)OlmoTrace traces OLMo 3 outputs back to training documents in real time.
Blog Ai2, Olmo 3
# eval card: the eval set is a dataset product
eval_set: support-bot-golden v3.2 # owner: domain expert, 2026-10
provenance: {prod_traces: 62%, synthetic: 38%} # PII-scrubbed / human-validated
size: {n: 640, slices: 9, min_slice_n: 48, positive_rate: 4.1%}
decontamination: 13-gram + embedding (cos > 0.92) vs SFT set -> 11 removed
labels: {experts: 2, adjudication: true, cohen_kappa: 0.78, noise_audit: 3.1%}
judge: {model: "<model>@<version>", prompt: v7, TPR: 0.91, TNR: 0.88, n: 200}
headline: "pass 84.2% [81.3, 86.9]" # bootstrap 95% CI
worst_slice: "refunds-ES 71% [58, 81]"
calibration: "ECE 0.04 (15 bins) after temperature scaling"
operating_point: {t: 0.05, C_FN_over_C_FP: 20, prod_prevalence: 2%}
known_gaps: [multi-turn > 8 turns, voice channel]
# data card: fields from FineWeb and Llama 3 practice
sources: Common Crawl snapshots, each with its date
filters: every filter with its removal rate
dedup: {level: per-snapshot, minhash: 5-gram, 14 buckets x 8 hashes}
classifier: {annotator: Llama-3-70B-Instruct, labels: 460k, F1: 0.82, keep: ">= 3"}
synthetic: {share: "...", generator_or_teacher: "..."}
verifiers: {types: [exec, unit tests], note: "about 20% of code self-corrected"}
decontamination_train: {method: n-gram, benchmarks_checked: [...]}
diversity: [distinct-n, embedding spread]
license: ODC-BY
known_biases: grade-school bias # FineWeb-Edu card
Raw web, PDF and code → filter, dedup, classifiers → pretrain → mid-train → SFT on decontaminated synthetic + curated data → DPO / APO → RLVR → merge → eval on dev and unseen splits → card + traceability.
OpenThoughts3 7B: +15.3 / +17.2 / +20.5 pts over R1-Distill-Qwen-7B on AIME25 / LiveCodeBench / GPQA-D.
Gemma 3: all models are trained with distillation, “making Gemma3-4B-IT competitive with Gemma2-27B-IT”. Gemma 4: E2B, E4B, 12B, 26B A4B MoE, 31B dense; 128K / 256K context; Apache 2.0 license link.
Phi-4 (14B): a recipe “centrally focused on data quality”, with synthetic data throughout training.
“the base models achieve a higher pass@k score when k is large”
RL wins at pass@1. Report pass@k for large k before you claim new reasoning.
GRPO on Qwen2.5-Math-7B: +21.4 pts on MATH-500 with random rewards, +29.1 with ground truth. Test other model families and decontaminated evals.
GPT-4o monitored o3-mini CoT. Training against the monitor gave obfuscated reward hacking, so pay a “monitorability tax”. Anthropic: hacking coding tasks spread to ~12% sabotage and ~50% alignment-faking. Inoculation prompting fixed it: “Learning to cheat no longer generalized to other misaligned behaviors.”
Humanity's Last Exam: 2,500 questions, because “benchmarks are not keeping pace in difficulty”. Dynamic benchmarks have no standard criteria yet.
Recursive training makes the tails vanish (Nature 631, 2024). Classifier monoculture removes tails by curation.
Bartz v. Anthropic: lawful books were fair use (2025-06-23); pirated copies went to trial; US$1.5B settlement. Provenance decides the risk.
OpenAI self-serve fine-tuning winds down (last jobs 2027-01-06). Fireworks: “Managed Training no longer accepts new RFT jobs”.
Non-verifiable tasks need graders: rubric quality, position, verbosity and self-preference bias.
Trajectory scoring, nondeterminism, cost and side effects. pass@k is not pass^k reliability.
66% (Bedrock RFT), up to 73% (Salesforce), 40x (Vercel v0): vendor-reported, not replicated.
Cards are written once and go stale. Regenerate them in CI from contracts and profiles.
Which model, prompt, seed and filter made each row? Few formats have fields for it.