知识蒸馏全面综述_47页_1mb

Knowledge Distillation: A Comprehensive Survey

Knowledge Distillation (KD) is a technique for transferring knowledge from a large teacher model to a smaller student model, improving the student's performance while reducing computational costs. The survey covers recent advances in KD, categorizing methods into sources (logits, features, similarities), schemes (offline, online, self), and algorithms (attention-based, adversarial, multi-teacher). It explores applications in various domains, including vision, NLP, and medical imaging, and addresses challenges like capacity gap and teacher-student mismatches.

Key Insights:

• KD is particularly critical for efficient deployment of Large Language Models (LLMs) and Vision-Language Models (VLMs).
• Recent trends include adaptive and contrastive distillation, which enhance transfer efficiency.
• Online and self-distillation are growing areas, especially for scenarios with limited pre-trained models or data constraints.
• Performance comparisons highlight that feature-based KD often yields better results than logit-based methods.
• Future research should focus on robust knowledge transfer, handling imbalanced data, and distillation for multi-modal and 3D data.