Multitask Training: Train the model on all tasks/datasets all at once;
Sequential Multitask Training: Train the model on some tasks/datasets first before continue training it with the target tasks/datasets; ^3f3b27
Multitask Training and Fine-tuning: Train the model on some tasks first and fine-tuning it with target tasks/datasets;
Findings from @lourie_2021 that
-
[[]]❌ almost always outperform other methods. - Multitask training helps the most when target data is scarce.
- Larger models see more gains from transfer learning.
References: