How could Bayesian neural networks be used for transfer learning?

Question

In transfer learning, we use big data from similar tasks to learn the parameters of a neural network, and then fine-tune the neural network on our own task that has little data available for it. Here, we can think of the transfer learning step as learning a (proper) prior, and then fine-tuning as learning the posterior.

So, we can argue that Bayesian networks can also solve the problem of small data-set regimes. But, what are the directions that we can mix Bayesian neural networks with similar tasks to transfer learning, for example, few-shot learning?

They make sense when they both take a role as a solution to the low data regime problems, but I can't think of a mix of them to tackle this issue.

Is it possible, for example, to learn a BNN for which we have picked a good prior to learn the posterior with little data and use the weight distribution for learning our new task? Is there any benefit in this?

Answer 1

Well, I would say, that purpose of Bayesian inference is not transfer learning, but uncertainty estimation.

In case you have good feature extractor in the beginning, you can adjust small number of parameters, like few last layers to achieve good quality in few epochs.

However, this is about adjusting the means of distributions over each weight.

Concerning the variance, I think transfer learning is inapplicable since the source and target distributions can be very different. For example, ImageNet is a broad and diverse dataset with many classes, and the target problem can involve only a few classes. Most probably, uncertainty estimate and the standard deviations of model weights on the ImageNet would be larger, than for the model, trained solely on the target task.