We use cookies to improve your experience with our site.

用于不平衡多任务学习的序列协同蒸馏

Sequential Cooperative Distillation for Imbalanced Multi-Task Learning

  • 摘要: 多任务学习(MTL)可通过多个相关任务之间的相互学习来提升单个任务的性能。然而,当这些任务具有不同的复杂度时,MTL 目标中涉及的相应损失不可避免地会相互竞争,最终使学习偏向于简单任务而非复杂任务。为了解决这种不平衡的学习问题,我们提出了一种新颖的MTL 方法,它可为现有的多个深度MTL 模型框架配备一个序列协同蒸馏(SCD)模块。具体地,我们首先引入一种有效的机制来衡量任务之间的相似性,并将相似的任务分组到同一个区块中,使它们能够相互协同学习。在此基础上,将分组后的任务块按队列排序,根据定义的性能指标估算出任务复杂度,并确定任务的学习顺序。最后,在单个任务特定模型和MTL 模型之间进行从复杂到简单逐块蒸馏,从而实现多个任务学习之间竞争与合作的平衡。广泛的实验证明,与最先进的方法相比,我们的方法更有竞争力,在多个数据集上的平均性能排名第一,与OMTL 和MTLKD 相比,分别提高了12.95% 和3.72%。

     

    Abstract: Multi-task learning (MTL) can boost the performance of individual tasks by mutual learning among multiple related tasks. However, when these tasks assume diverse complexities, their corresponding losses involved in the MTL objective inevitably compete with each other and ultimately make the learning biased towards simple tasks rather than complex ones. To address this imbalanced learning problem, we propose a novel MTL method that can equip multiple existing deep MTL model architectures with a sequential cooperative distillation (SCD) module. Specifically, we first introduce an efficient mechanism to measure the similarity between tasks, and group similar tasks into the same block to allow their cooperative learning from each other. Based on this, the grouped task blocks are sorted in a queue to determine the learning sequence of the tasks according to their complexities estimated with the defined performance indicator. Finally, a distillation between the individual task-specific models and the MTL model is performed block by block from complex to simple manner, achieving a balance between competition and cooperation among learning multiple tasks. Extensive experiments demonstrate that our method is significantly more competitive compared with state-of-the-art methods, ranking No.1 with average performances across multiple datasets by improving 12.95% and 3.72% compared with OMTL and MTLKD, respectively.

     

/

返回文章
返回