Memory-Efficient Training with Block Coordinate Descent
Ph.D. research on block coordinate descent methods, one with a selective block rule, for training under tight GPU memory: only the active block keeps gradients and optimizer state, so memory drops to 2M + 16M/D bytes (M parameters, D blocks) — up to 95% less optimizer memory than Adam/AdamW. Benchmarked in fine-tuning and post-training (SFT, DPO) of language and vision models such as RoBERTa-Large and ViT-H, and extended to multi-GPU training with PyTorch FSDP. Ongoing.
- PyTorch
- FSDP
- SFT
- DPO
- Optimization