Independence and Bayes' Theorem

Master independence and Bayes' theorem: dependent events, probability updating, and conditional reasoning through these 5 detailed exercises.

Independence and Bayes' Theorem Concepts
\(P(A \cap B) = P(A) \cdot P(B)\)
Independence Definition
\(P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}\)
Bayes' Theorem
\(P(B) = P(B|A) \cdot P(A) + P(B|A') \cdot P(A')\)
Law of Total Probability
Key Definitions:

Independent Events: Events A and B are independent if the occurrence of one does not affect the probability of the other.

Dependent Events: Events where the occurrence of one affects the probability of the other.

Bayes' Theorem: Formula to update the probability of a hypothesis based on new evidence.

Law of Total Probability: Method to calculate the probability of an event by considering all possible scenarios.

Posterior Probability: Updated probability after observing new evidence.

Prior Probability: Initial probability before observing evidence.

Key Relationships:
  1. Independence Test: \(P(A \cap B) = P(A) \cdot P(B)\) or \(P(A|B) = P(A)\)
  2. Bayes' Theorem: \(P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}\)
  3. Law of Total Probability: \(P(B) = \sum P(B|A_i) \cdot P(A_i)\)
  4. Chain Rule: \(P(A \cap B) = P(A|B) \cdot P(B)\)
Independence Property: If A and B are independent, then A and B' are also independent.
Bayes' Insight: Allows "reversing" conditional probability relationships.
Tip 1: Independence means P(A|B) = P(A) - knowledge of B doesn't change A's probability.
Tip 2: Use Bayes' theorem when you know P(B|A) but need P(A|B).
Tip 3: Tree diagrams help visualize sequential conditional probabilities.
Solution: Exercises 1 to 3
1 Independence Test
Exercise 1
A fair die is rolled. Let A = "roll is even" and B = "roll is greater than 3".
Are A and B independent events?
Define Events:

Sample space: S = {1, 2, 3, 4, 5, 6}

A = {2, 4, 6} (even numbers)

B = {4, 5, 6} (numbers > 3)

A ∩ B = {4, 6} (even and > 3)

Step 1: Calculate individual probabilities

\(P(A) = \frac{3}{6} = \frac{1}{2}\)

\(P(B) = \frac{3}{6} = \frac{1}{2}\)

\(P(A \cap B) = \frac{2}{6} = \frac{1}{3}\)

Step 2: Test independence using P(A ∩ B) = P(A) · P(B)

\(P(A) \cdot P(B) = \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4}\)

Since \(P(A \cap B) = \frac{1}{3} \neq \frac{1}{4} = P(A) \cdot P(B)\), A and B are NOT independent.

Step 3: Verify using conditional probability

\(P(A|B) = \frac{P(A \cap B)}{P(B)} = \frac{\frac{1}{3}}{\frac{1}{2}} = \frac{2}{3}\)

Since \(P(A|B) = \frac{2}{3} \neq \frac{1}{2} = P(A)\), A and B are NOT independent.

Step 4: Interpretation

Knowing that the die roll is greater than 3 increases the probability that it's even (from 1/2 to 2/3).

Events A and B are NOT independent
Final Answer:

Events A and B are dependent because \(P(A \cap B) \neq P(A) \cdot P(B)\).

2 Independent Events
Exercise 2
Two fair coins are flipped. Let A = "first coin is heads" and B = "second coin is tails".
Are A and B independent events?
Define Events:

Sample space: S = {HH, HT, TH, TT}

A = {HH, HT} (first coin is heads)

B = {HT, TT} (second coin is tails)

A ∩ B = {HT} (first heads, second tails)

Step 1: Calculate individual probabilities

\(P(A) = \frac{2}{4} = \frac{1}{2}\)

\(P(B) = \frac{2}{4} = \frac{1}{2}\)

\(P(A \cap B) = \frac{1}{4}\)

Step 2: Test independence

\(P(A) \cdot P(B) = \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4}\)

Since \(P(A \cap B) = \frac{1}{4} = P(A) \cdot P(B)\), A and B ARE independent.

Step 3: Verify using conditional probability

\(P(A|B) = \frac{P(A \cap B)}{P(B)} = \frac{\frac{1}{4}}{\frac{1}{2}} = \frac{1}{2}\)

Since \(P(A|B) = \frac{1}{2} = P(A)\), A and B are independent.

Step 4: Interpretation

The outcome of the first coin flip does not affect the probability of the second coin flip.

Events A and B ARE independent
Final Answer:

Events A and B are independent because \(P(A \cap B) = P(A) \cdot P(B)\).

3 Drawing Cards Without Replacement
Exercise 3
Two cards are drawn without replacement from a standard deck. Let A = "first card is red" and B = "second card is red".
Are A and B independent events?
Setup:

Standard deck: 52 cards (26 red, 26 black)

Without replacement: composition changes after first draw

Step 1: Calculate individual probabilities

\(P(A) = \frac{26}{52} = \frac{1}{2}\)

\(P(B) = \frac{26}{52} = \frac{1}{2}\) (before any information about A)

Step 2: Calculate conditional probabilities

If A occurs (first card is red): 25 red, 26 black remain → 51 total

\(P(B|A) = \frac{25}{51}\)

If A' occurs (first card is black): 26 red, 25 black remain → 51 total

\(P(B|A') = \frac{26}{51}\)

Step 3: Test independence

Since \(P(B|A) = \frac{25}{51} \neq \frac{1}{2} = P(B)\), A and B are NOT independent.

The first card drawn affects the probability of drawing a red card second.

Step 4: Calculate P(A ∩ B)

\(P(A \cap B) = P(A) \cdot P(B|A) = \frac{1}{2} \cdot \frac{25}{51} = \frac{25}{102}\)

\(P(A) \cdot P(B) = \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4} = \frac{51}{204} = \frac{25.5}{102}\)

Since \(\frac{25}{102} \neq \frac{25.5}{102}\), A and B are dependent.

Events A and B are NOT independent
Final Answer:

Events A and B are dependent because drawing the first card changes the composition of the deck for the second draw.

Solution: Exercises 4 to 5
4 Medical Diagnosis (Bayes' Theorem)
Exercise 4
A disease affects 1% of the population. A test for the disease has:
• 95% accuracy rate when the person has the disease
• 90% accuracy rate when the person doesn't have the disease
If a person tests positive, what is the probability they actually have the disease?
Define Events:

Let D = "person has disease", T = "test is positive"

Given: P(D) = 0.01, P(T|D) = 0.95, P(T'|D') = 0.90

Therefore: P(D') = 0.99, P(T|D') = 0.10

Step 1: Identify all given probabilities

P(D) = 0.01 (prior probability of disease)

P(D') = 0.99 (probability of no disease)

P(T|D) = 0.95 (probability of positive test given disease)

P(T|D') = 0.10 (probability of positive test given no disease)

Step 2: Apply Law of Total Probability to find P(T)

P(T) = P(T|D)·P(D) + P(T|D')·P(D')

P(T) = (0.95)(0.01) + (0.10)(0.99) = 0.0095 + 0.099 = 0.1085

Step 3: Apply Bayes' Theorem

\(P(D|T) = \frac{P(T|D) \cdot P(D)}{P(T)} = \frac{(0.95)(0.01)}{0.1085} = \frac{0.0095}{0.1085} = \frac{95}{1085} = \frac{19}{217} \approx 0.0876\)

Step 4: Interpret the result

Even though the test is quite accurate, only about 8.76% of positive results are true positives.

This demonstrates the importance of considering base rates in medical testing.

\(P(\text{disease}|\text{positive test}) = \frac{19}{217} \approx 0.0876\)
Final Answer:

The probability that a person who tests positive actually has the disease is approximately 8.76%.

5 Quality Control (Bayes' Theorem)
Exercise 5
A factory has two machines producing widgets:
• Machine A produces 60% of the widgets (2% defective rate)
• Machine B produces 40% of the widgets (5% defective rate)
If a randomly selected widget is defective, what is the probability it came from Machine A?
Define Events:

Let A = "widget from Machine A", B = "widget from Machine B", D = "widget is defective"

Given: P(A) = 0.6, P(B) = 0.4, P(D|A) = 0.02, P(D|B) = 0.05

Step 1: Identify all given probabilities

P(A) = 0.6 (probability widget from Machine A)

P(B) = 0.4 (probability widget from Machine B)

P(D|A) = 0.02 (probability defective given Machine A)

P(D|B) = 0.05 (probability defective given Machine B)

Step 2: Apply Law of Total Probability to find P(D)

P(D) = P(D|A)·P(A) + P(D|B)·P(B)

P(D) = (0.02)(0.6) + (0.05)(0.4) = 0.012 + 0.020 = 0.032

Step 3: Apply Bayes' Theorem to find P(A|D)

\(P(A|D) = \frac{P(D|A) \cdot P(A)}{P(D)} = \frac{(0.02)(0.6)}{0.032} = \frac{0.012}{0.032} = \frac{12}{32} = \frac{3}{8} = 0.375\)

Step 4: Verify with P(B|D)

\(P(B|D) = \frac{P(D|B) \cdot P(B)}{P(D)} = \frac{(0.05)(0.4)}{0.032} = \frac{0.020}{0.032} = \frac{20}{32} = \frac{5}{8} = 0.625\)

Check: \(P(A|D) + P(B|D) = 0.375 + 0.625 = 1.0\) ✓

Step 5: Interpret the result

Even though Machine A produces more widgets, defective widgets are more likely to come from Machine B due to its higher defect rate.

\(P(\text{Machine A}|\text{defective}) = \frac{3}{8} = 0.375\)
Final Answer:

The probability that a defective widget came from Machine A is 37.5%.

Comprehensive Summary: Independence and Bayes' Theorem
Core Concepts:

Independence: Two events are independent if the occurrence of one does not affect the probability of the other.

Bayes' Theorem: A method to update probabilities based on new evidence.

Conditional Probability: The probability of an event given that another event has occurred.

Essential Formulas:
  1. Independence Test: \(P(A \cap B) = P(A) \cdot P(B)\)
  2. Alternative Independence: \(P(A|B) = P(A)\) or \(P(B|A) = P(B)\)
  3. Bayes' Theorem: \(P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}\)
  4. Law of Total Probability: \(P(B) = \sum P(B|A_i) \cdot P(A_i)\)
  5. Chain Rule: \(P(A \cap B) = P(A|B) \cdot P(B)\)
Key Principles:

Base Rate Neglect: People often ignore prior probabilities when evaluating conditional probabilities.

Information Update: Bayes' theorem provides a rational way to update beliefs.

Independence Preservation: If A and B are independent, then A and B' are also independent.

Memory Tip: Independent means "no influence" - P(A|B) = P(A)
Strategy Tip: Use Bayes' when you need to reverse a conditional probability
Visualization Tip: Tree diagrams clarify sequential probability updates
Verification Tip: Posterior probabilities should sum to 1
Bayes' Theorem Problem-Solving
1
Identify Hypotheses
2
Find Prior Probabilities
3
Determine Likelihoods
4
Calculate Posterior

Questions & Answers

Question: How do I know if events are independent or dependent in a real-world scenario?

Answer: Consider whether the occurrence of one event affects the probability of the other:

  • Independent: Coin flips, dice rolls, drawing with replacement, weather in distant cities
  • Dependent: Drawing without replacement, taking multiple courses from the same professor, consecutive days' weather

Ask yourself: "Does knowing that event A happened change my belief about the probability of event B?" If yes, they're dependent. If no, they're independent.

Question: Why is Bayes' theorem important in real life?

Answer: Bayes' theorem is crucial for decision-making under uncertainty:

  • Medical Diagnosis: Updating disease probability based on test results
  • Spam Filtering: Calculating probability an email is spam based on its content
  • Jury Decisions: Updating belief in guilt based on evidence
  • Machine Learning: Many algorithms use Bayesian reasoning

It provides a systematic way to incorporate new evidence into our existing beliefs.

Question: Can you have three events that are pairwise independent but not mutually independent?

Answer: Yes, this is possible! Here's an example:

Consider flipping two coins. Let:

  • A = "first coin is heads"
  • B = "second coin is heads"
  • C = "both coins show the same result"

Each pair is independent (A and B, A and C, B and C), but all three together are not independent because if A and B both occur, then C must occur with probability 1, not 1/2.

For mutual independence, we need: \(P(A \cap B \cap C) = P(A) \cdot P(B) \cdot P(C)\), which fails in this case.