3

I'm reading about how Conditional Probability/ Bayes Theorem is used in Naive Bayes in Intro to Statistical Learning, but it seems like it isn't that "groundbreaking" as it is described?

If I'm not mistaken doesn't every single ML classifier use conditional probability/Bayes in its underlying assumptions, not just Naive Bayes? We are always trying to find the most likely class/label, given a set of features. And we can only deduce that using Bayes rule since we are (usually) solving for P(class|features) with P(features|class)?

nbro
  • 42,615
  • 12
  • 119
  • 217

2 Answers2

2

Conditional probability and Bayes rule are related but they are not the same thing, you can predict conditional probabilities without using Bayes rule.

So no, not all machine learning classifiers use Bayes rule, standard neural networks do not use Bayes rule at all, SVMs and linear classifiers neither.

A better counterexample is Bayesian Neural Networks, which have a probability distribution over the weights, and Bayes rule is used during learning and inference, these are not the same as standard neural networks.

As reference for this statement, I leave the following quote from Section 3.1 of the the paper Uncertainty Quantification for Deep Neural Networks: An Empirical Comparison and Usage Guidelines:

BNNs are neural networks with probabilistic weights, instead of scalar weights as in PPNN, and are represented as probability density functions. To train a BNN, first, a prior distribution p(θ) over weights θ has to be defined. Then, given some data D, the posterior distribution p(θ|D), i.e., the trained BNN is inferred using Bayes rule:

Dr. Snoopy
  • 1,363
  • 8
  • 12
1

Probability is one way to solve classification problems. Still, there are other ways like clustering and K nearest neighbor approach where we tend to analyze the position of the current data point and its neighboring points to classify it. Also, in the decision tree classifier, information gain is the core concept used to classify.

oseekero
  • 21
  • 5