2

Could it make any sense to choose a larger dimension for the latent space of the VAE with respect to the original input?

For example, we may want to learn how to reconstruct a relatively low-dimensional input (let's say $20$ dimensions), then could I define my encoder and decoder to have $64,256,512...$ hidden neurons before bringing back the reconstruction?

EDIT: Well I've thought about that and I think it would still be reasonable as in latent-variable models we are actually assuming that our original observations are generated from unseen 'hidden' variables. And (I think) the lower dimension of the latent space is only assumed for an original dimensionality-reduction purpose.

James Arten
  • 307
  • 1
  • 12

1 Answers1

1

I am not sure about the VAE in particular, but the convnext presented here uses the "inverse bottleneck" (i.e. internal representations being higher dimensional than inputs) as one of the core changes that leads to an increase in its performance when compared to prior convolutional networks. This and some other changes make the convnet be more competitive against the visual transformer (that is pretrained in a large dataset).

But for really low-dimensional data (as in your example) and for VAEs, I am not sure if the same idea would prove useful.

P.S. I wanted to add this as a comment but I do not have enough reputation to do so.

avio11
  • 11
  • 1