LEDGAR Contract Clause Classifier & Retrieval

Commercial contract review requires tagging clauses across 100 legal categories and retrieving precedent clauses matching specific constraints (e.g., 30-day termination notice). Deploying transformers in legal tech requires rigorous justification against cheaper linear baselines and calibrated confidence scores.

Does full fine-tuning of DistilBERT (67M parameters) or LoRA (815k parameters) significantly outperform a tuned TF-IDF + Logistic Regression baseline on 100 imbalanced clause types, and which embedding geometry best supports clause-to-clause and natural-language lawyer retrieval?

LexGLUE LEDGAR benchmark (coastalcph/lex_glue, SEC filings): 60,000 training, 10,000 validation, and 10,000 held-out test clauses across 100 highly imbalanced clause types (138x ratio between largest and smallest class), plus derived bias-audit slices (contract family, jurisdiction, party role, ALL-CAPS boilerplate).

Executed a pre-registered experimental protocol using macro-F1 on validation and reading the 10,000-clause test split only once per final model. Compared TF-IDF (unigrams+bigrams, L2 C=30), BiLSTM+GloVe, DistilBERT full fine-tuning (512 tokens, length-grouped batches), and LoRA (rank 8 on query/value) across 3 seeds using paired bootstrap (1,000 resamples) and McNemar's test. Applied post-hoc temperature scaling for Expected Calibration Error (ECE) and benchmarked 13 retrieval encodings (TF-IDF, MiniLM, E5, BGE, Legal-BERT, LoRA features, with mean-centering and hybrid type filtering) on perturbation self-retrieval and regex-verified lawyer queries.

10-epoch DistilBERT achieved 0.827 ± 0.004 test macro-F1 (88.0% accuracy), leading LoRA by +1.26 pts (95% CI [+0.54, +1.93] while LoRA trained only 1.2% of parameters at 4x lower memory), and statistically tied the 2.3-minute TF-IDF baseline (0.829, McNemar p=0.60). Temperature scaling (T=2.47) reduced 10-epoch test ECE by 73% (from 0.086 to 0.023, and by 84% to 0.011 on LoRA). In retrieval, centered BGE-small beat TF-IDF by +0.76 macro P@10 (+4.64 pts on low-lexical-overlap clauses), while E5 achieved 86.2% constraint precision (4.0x lift) on natural-language lawyer queries.

Rigorous ML engineering study illustrating pre-registered hypothesis testing, parameter-efficient fine-tuning (LoRA), probability calibration, slice-based bias auditing, and embedding anisotropy correction.