I noticed that there are many studies in recent years on how to train/update neural networks faster/quicker with equal or better performance. I find the following methods(except the chips arms race):
- using few-shot learning, for instance, pre-taining and etc.
- using the minimum viable dataset, for instance using (guided) progressive sampling.
- model compression, for instance, efficent transformers
- Data echoing, or simply put let the data pass multiple times in the graph(or GPU)
Is there a systematic structure on this topic and how can we update or train a model faster without loss of its capacity?
