JMU CS 445: Probability and Expectation for ML

Probability Theory - Basic Terminology

  • Sample Space (): The set of all possible outcomes

    • Example: When rolling a die,

  • Outcome: A single result

    • Example: Rolling

  • Event: A subset of the sample space (a collection of one or more outcomes)

    • Example: "Rolling an even-numbered face"
    • Example: "Rolling a face numbered greater than "
    • Example: "Rolling face 3"

Probability

Probability is a numerical measure of the likelihood that an event will occur.

A probability function (or probability measure) is a function that assigns a number between 0 and 1 to each event in the sample space, where:

  • indicates the event is impossible
  • indicates the event is certain
  • Values between and indicate varying degrees of likelihood

Interpretation:

  • means event has a 20% chance of occurring
  • means event has an 80% chance of occurring

Examples with a fair six-sided die:

  • (technically, we would need to add 7 to the sample space for this to make sense)
  • (16.7% chance)
  • (certain)

Laws of Probability

  1. Non-negativity: For any event ,

    • Example:

  2. Total Probability: The probability of the sample space is 1,

    • Example:

  3. Additivity: For mutually exclusive events and ,

    • Example:

Random Variables

Definition: A random variable is a function that assigns a value (numeric or categorical) to each outcome in the sample space .

Examples:

  • Die roll - Parity: If , then random variable could map:

  • Die roll - Numerical: Same sample space, but random variable maps to numbers:

    • This gives us a numerical random variable with range

Range and Events:

  • The range of a random variable is the set of possible values it can take:
    • Parity RV : range is (categorical)
    • Numerical RV : range is (numerical)
  • For any value , the event is the subset of where the RV takes value :
    • is the event outcomes where die shows odd number
    • is the event outcome where die shows the face with 4 dots

Probability Mass Function (PMF)

A Probability Mass Function (PMF) assigns probabilities to the values of a discrete random variable.

Definition: For a discrete random variable , the PMF is:

Requirements: A valid PMF must satisfy:

  1. Non-negativity: for all
  2. Normalization: (probabilities sum to 1)

Example: For our die parity random variable :

Verification: ✓ and both probabilities ✓

Expectation, Variance

Numerical random variables only (not the parity RV).

  • Expectation (discrete): one term per value in the range of , weighted by its probability

  • Expectation (continuous): same idea, with a density in place of the PMF

    The expected value is where the distribution balances.

  • Variance (and , the standard deviation):

Quiz

You pay $2 to play. You roll a four-sided die and win the number of dollars
showing on the face that comes up. The die is weighted:

Questions:

  • What is the expected value of this distribution?
  • What is the variance?
  • Should you play?

Sample Mean and Variance

Expectation and variance are properties of distributions.
We can also calculate the sample mean and the sample variance for a dataset:

  • Sample mean:

  • Sample variance:

Normal Distribution

  • "Normal" because of the central limit theorem

Consolidates the probability foundations needed downstream (trimmed from prob.md) with the expectation material (formerly normal.md), in one deck. Covers: random variables and PMFs, then conditional probability, P(y|x) notation and independence (for logistic regression and maximum likelihood), then expectation and variance (for the bias/variance decomposition, which the sprinkles lab picks up immediately after). Expectation sits last so it is freshest going into that lab. Dropped from prob.md, since Naive Bayes is no longer in the sequence: learning a joint distribution from counts, the table-explosion problem, marginalization as a named operation, conditional independence, Bayes rule, the Bayes classifier, and the naive Bayes classifier. Those are still in prob.md. Supersedes prob_brief.md and normal.md.