Probability Distributions and Random Variables

• A discrete random variable has some finite number of outcomes drawn from a sample space:
  • E.g. the sample space for \(Y\) might be \(\{cold, flu, healthy\}\)
  • The sample space for \(X\) might be \(\{True, False\}\)
• A probability distribution or probability mass function maps each event in the sample space to a probability between 0 and 1, where the total of all probabilities must sum to 1:
  • Disease:
    • \(P(Y = cold) = .7\)
    • \(P(Y = flu) = .2\)
    • \(P(Y = healthy) = .1\)
    • Or, more concisely:
    • \(P(cold) = .7\)
    • \(P(flu) = .2\)
    • \(P(healthy) = .1\)
  • Fever:
    • \(P(X = True) = .1\)
    • \(P(X = False) = .9\)
    • Or, more concisely:
    • \(P(x) = .1\)
    • \(P(\lnot x) = .9\)

Joint Probability Distributions

The joint probability distribution represents the probability of two events, described by different random variables, happening together:

\(X\)	\(Y\)	\(P(X,Y)\)
T	cold	.04
T	flu	.14
T	healthy	.02
F	cold	.08
F	flu	.04
F	healthy	.68

So \(P(X=True, Y=cold) = .04\).

This could also be written: \(P(X=True \;\; \cap\;\; Y=cold) = .04\)

Marginalization

\(X\)	\(Y\)	\(P(X,Y)\)
T	cold	.04
T	flu	.14
T	healthy	.02
F	cold	.08
F	flu	.04
F	healthy	.68

• Given the joint probability distribution we can use marginalization to retrieve the probability distribution for any individual variable:
  • \[P(Y) = \sum_{x \in X} P(Y, x) \]
• For example:
  • The probability that someone has the flu, regardless of fever:
    • \(P(Y=flu) = P(X=True, Y=flu) + P(X=False, Y=flu)\)
    • \(P(Y=flu) = .14 + .04 = .18\)
  • The probability that someone has a fever regardless of health:
    • \(P(X =True)= P(X=True, Y=cold) + P(X=True, Y=flu) + P(X=True, Y=healthy)\)
    • \(P(X =True)= .04 + .14 + .02 = .2\)

Conditional Probability

• Definition:
  • \(P(Y \mid X) = \frac{P(X,Y)}{P(X)}\)
• For example:
  • \(P(Y=flu \mid X=True) = \frac{P(X=True,Y=flu)}{P(X=True)} = \frac{.14}{.2} = .7\)
• It follows that:
  • \(P(X, Y) = P(Y \mid X) P(X)\) (the chain rule)
  • \(P(X, Y) = P(X \mid Y) P(Y)\)

Bayes Theorem / Bayes Rule

\[P(Y \mid X) = \frac{P(X \mid Y)P(Y)}{P(X)}\]

Note that

\[P(X) = \sum_{y \in Y} P(X \mid y) P(y)\]

(by combining marginalization with the chain rule)

So Bayes rule can be expressed as:

\[P(Y \mid X) = \frac{P(X \mid Y)P(Y)}{ \sum_{y \in Y} P(X \mid y) P(y)}\]

Bayes Classifier

We can use Bayes rule to build a classifier:

\[P(Y \mid X_1, X_2, ..., X_d) = \frac{P(X_1, X_2, ..., X_d \mid Y)P(Y)}{P(X_1, X_2, ..., X_d)}\]

Where \(Y\) corresponds to the class label and each \(X_i\) is an attribute.

There is a serious problem with this! What is it?

Naive Bayes Classifier

• We assume that the attributes are conditionally independent given class labels, so:
  • \[P(X_1, X_2, ..., X_d \mid Y) = \prod_{i=1}^d P(X_i \mid Y)\]
• We can also recognize that \(P(X_1, X_2, ..., X_d)\) is the same regardless of class so dividing with it won’t change the class with the largest value.
• These leads to the naive Bayes classifier:
  • \[P(Y \mid X_1, X_2, ..., X_d) \propto \prod_{i=1}^d P(X_i \mid Y)P(Y)\]