When I was reading about discriminative vs generative models, I came across their definitions:
Given a distribution of inputs $X$ and labels $Y:$
Discriminative models learn the conditional distribution $P(Y|X)$.
Generative models learn the joint distribution $P(X,Y)$. If you only have $X$, then you can still learn $P(X)$.
My questions are:
Learning the distribution: When we talk about learning a distribution, we are essentially trying to capture the underlying statistical properties of the data. In other words, we try to capture the distribution from which the data points in our dataset our sampled. This involves estimating parameters that define the distribution (such as mean, variance, etc.) or learning a model that can generate data points similar to those observed in the dataset.
What does the distribution contain?: The distribution contains information about the likelihood of different values or configurations of the variables in the dataset. For example, in a simple case where X represents the features of a dataset and Y represents the labels, the conditional distribution P(Y|X) would describe the probability of observing a particular label given the input features.
What does the distribution look like?: Depends on the nature of the data and the relationships between variables. It could take various forms, such as Gaussian (bell-shaped), uniform, exponential, etc., depending on the characteristics of the data.
