Solved Exercises on Regression Analysis Basics

Master regression analysis basics: scatter plots, correlation, linear regression, least squares method, and coefficient of determination through these 5 detailed exercises.

Solution: Exercises 1 to 3
1 Scatter Plot and Correlation
Exercise 1
The table shows the number of hours studied and test scores for 6 students. Create a scatter plot and describe the correlation.
Definition:

Scatter Plot: A graph that displays the relationship between two variables using ordered pairs plotted as points.

Correlation: A measure of the strength and direction of a linear relationship between two variables.

Scatter Plot Method:
  1. Identify the independent variable (x-axis) and dependent variable (y-axis)
  2. Create a coordinate plane with appropriate scales
  3. Plot each ordered pair as a point
  4. Analyze the pattern of points
  5. Determine the type and strength of correlation
Data points
(2, 65), (3, 70), (4, 75), (5, 80), (6, 85), (7, 90)
Pattern
Positive linear correlation
Correlation strength
Strong (r ≈ 1)
Step 1: Identify variables

Independent variable (x): Hours studied

Dependent variable (y): Test score

Step 2: Set up the coordinate system

X-axis: Hours studied (0 to 8)

Y-axis: Test score (60 to 95)

Scale: 1 unit = 1 hour on x-axis, 1 unit = 5 points on y-axis

Step 3: Plot the data points

(2, 65): 2 hours → 65 points

(3, 70): 3 hours → 70 points

(4, 75): 4 hours → 75 points

(5, 80): 5 hours → 80 points

(6, 85): 6 hours → 85 points

(7, 90): 7 hours → 90 points

Step 4: Analyze the pattern

Points form an upward-sloping line

As x increases, y increases consistently

Pattern is linear with minimal scatter

Step 5: Describe the correlation

Direction: Positive (upward trend)

Strength: Strong (points closely follow a straight line)

Type: Linear (forms a straight line pattern)

Positive strong linear correlation
Final answer:

The scatter plot shows a positive strong linear correlation between hours studied and test scores. As study time increases, test scores tend to increase.

Applied rules:

• Positive Correlation: Both variables increase together

• Linear Pattern: Points form a straight line

• Correlation Strength: Determined by how closely points follow a line

2 Linear Regression Line
Exercise 2
Find the equation of the least squares regression line for the data: (1, 3), (2, 5), (3, 7), (4, 9), (5, 11).
Definition:

Least Squares Regression Line: The line that minimizes the sum of the squared vertical distances from each data point to the line.

Regression Equation: y = mx + b, where m is the slope and b is the y-intercept.

Data points
(1, 3), (2, 5), (3, 7), (4, 9), (5, 11)
Means
x̄ = 3, ȳ = 7
Slope
m = 2
Y-intercept
b = 1
Regression line
y = 2x + 1
Step 1: Calculate the means

Mean of x values: x̄ = (1 + 2 + 3 + 4 + 5)/5 = 15/5 = 3

Mean of y values: ȳ = (3 + 5 + 7 + 9 + 11)/5 = 35/5 = 7

Step 2: Calculate necessary sums

Σx = 15, Σy = 35, Σx² = 1² + 2² + 3² + 4² + 5² = 1 + 4 + 9 + 16 + 25 = 55

Σxy = (1×3) + (2×5) + (3×7) + (4×9) + (5×11) = 3 + 10 + 21 + 36 + 55 = 125

Step 3: Calculate the slope (m)

m = [n(Σxy) - (Σx)(Σy)] / [n(Σx²) - (Σx)²]

m = [5(125) - (15)(35)] / [5(55) - (15)²]

m = [625 - 525] / [275 - 225]

m = 100 / 50 = 2

Step 4: Calculate the y-intercept (b)

b = ȳ - mx̄

b = 7 - 2(3) = 7 - 6 = 1

Step 5: Write the regression equation

y = mx + b

y = 2x + 1

Step 6: Verify the regression line

Check with point (3, 7): y = 2(3) + 1 = 7 ✓

Check with point (1, 3): y = 2(1) + 1 = 3 ✓

All points lie exactly on the line (perfect correlation)

y = 2x + 1
Final answer:

The equation of the least squares regression line is y = 2x + 1.

Applied rules:

• Slope Formula: m = [n(Σxy) - (Σx)(Σy)] / [n(Σx²) - (Σx)²]

• Y-intercept: b = ȳ - mx̄

• Regression Line: Passes through point (x̄, ȳ)

3 Correlation Coefficient
Exercise 3
Calculate the correlation coefficient for the data: (2, 8), (4, 12), (6, 16), (8, 20), (10, 24).
Definition:

Correlation Coefficient (r): A measure of the strength and direction of the linear relationship between two variables. Ranges from -1 to 1.

Perfect Correlation: r = 1 (perfect positive) or r = -1 (perfect negative).

Data points
(2, 8), (4, 12), (6, 16), (8, 20), (10, 24)
Required sums
Σx=30, Σy=80, Σx²=220, Σy²=1440, Σxy=560
Correlation coefficient
r = 1 (perfect positive)
Step 1: Organize the data

Number of data points: n = 5

x y x² y² xy
2 8 4 64 16
4 12 16 144 48
6 16 36 256 96
8 20 64 400 160
10 24 100 576 240
Step 2: Calculate the sums

Σx = 2 + 4 + 6 + 8 + 10 = 30

Σy = 8 + 12 + 16 + 20 + 24 = 80

Σx² = 4 + 16 + 36 + 64 + 100 = 220

Σy² = 64 + 144 + 256 + 400 + 576 = 1440

Σxy = 16 + 48 + 96 + 160 + 240 = 560

Step 3: Apply the correlation coefficient formula

r = [n(Σxy) - (Σx)(Σy)] / √[n(Σx²) - (Σx)²][n(Σy²) - (Σy)²]

r = [5(560) - (30)(80)] / √[5(220) - (30)²][5(1440) - (80)²]

r = [2800 - 2400] / √[1100 - 900][7200 - 6400]

r = 400 / √[200][800]

r = 400 / √160000

r = 400 / 400 = 1

Step 4: Interpret the result

r = 1 indicates a perfect positive linear correlation

All data points lie exactly on a straight line with positive slope

Step 5: Verify the pattern

Notice: y = 2x + 4 (from previous exercise)

Each increase of 2 in x results in an increase of 4 in y

Perfect linear relationship: r = 1

r = 1
Final answer:

The correlation coefficient is r = 1, indicating a perfect positive linear relationship.

Applied rules:

• Correlation Range: -1 ≤ r ≤ 1

• Interpretation: r = 1 (perfect positive), r = 0 (no correlation), r = -1 (perfect negative)

• Formula: Uses means, standard deviations, and covariance

Regression Analysis: Rules and Methods
y = mx + b
Linear Regression Equation
Slope
m = (n∑xy - ∑x∑y)/(n∑x² - (∑x)²)
Rate of change
Y-intercept
b = ȳ - mx̄
Value when x = 0
Correlation
r = (n∑xy - ∑x∑y)/√[n∑x²-(∑x)²][n∑y²-(∑y)²]
Strength of relationship
Scatter Plot: Visual representation of bivariate data showing patterns and relationships.
Best Fit Line: Minimizes the sum of squared residuals (least squares method).
Correlation Strength: |r| close to 1 indicates strong correlation, |r| close to 0 indicates weak correlation.
Solution: Exercises 4 to 5
4 Prediction Using Regression
Exercise 4
Using the regression equation y = 1.5x + 10 from a study of study hours vs. test scores, predict the test score for a student who studies 7.5 hours.
Definition:

Prediction: Using a regression equation to estimate the value of the dependent variable for a given value of the independent variable.

Interpolation: Predicting within the range of observed data.

Extrapolation: Predicting outside the range of observed data (less reliable).

Regression equation
y = 1.5x + 10
Given value
x = 7.5 hours
Predicted value
y = 21.25 points
Step 1: Identify the regression equation

Given: y = 1.5x + 10

Where x = hours studied and y = test score

Step 2: Identify the input value

Given: x = 7.5 hours of study

Step 3: Substitute into the equation

y = 1.5(7.5) + 10

y = 11.25 + 10

y = 21.25

Step 4: Interpret the result

The model predicts a test score of 21.25 for 7.5 hours of study

This is interpolation since 7.5 is likely within the observed range

Step 5: Consider reliability

The prediction is reliable if the correlation is strong

The model assumes the relationship continues beyond observed data

Predicted score: 21.25
Final answer:

The predicted test score for a student who studies 7.5 hours is 21.25 points.

Applied rules:

• Substitution: Replace x with the given value in the equation

• Interpolation vs Extrapolation: Assess reliability based on data range

• Correlation Strength: Stronger correlations yield more reliable predictions

5 Residual Analysis
Exercise 5
For the data point (4, 18) and regression line y = 2x + 5, calculate and interpret the residual.
Definition:

Residual: The difference between the actual y-value and the predicted y-value: residual = actual - predicted.

Residual Analysis: Examining residuals to assess the fit of a regression model.

Actual point
(4, 18)
Regression line
y = 2x + 5
Predicted value
y = 2(4) + 5 = 13
Residual
18 - 13 = 5
Step 1: Identify the actual value

For x = 4, the actual y-value is 18

Step 2: Calculate the predicted value

Using the regression equation: y = 2(4) + 5 = 8 + 5 = 13

Step 3: Calculate the residual

Residual = Actual - Predicted = 18 - 13 = 5

Step 4: Interpret the residual

Positive residual (5) means the actual value is 5 units above the predicted value

The point (4, 18) is above the regression line

Step 5: Assess model fit

Large residuals indicate poor model fit for that point

Sum of residuals should be close to zero for good fit

Residual = 5
Final answer:

The residual for the point (4, 18) is 5, meaning the actual value is 5 units above the predicted value.

Applied rules:

• Residual Formula: residual = y_actual - y_predicted

• Positive Residual: Point above regression line

• Negative Residual: Point below regression line

Complete Guide to Regression Analysis
\hat{y} = a + bx
Regression Equation
Key definitions:

Regression Analysis: A statistical method used to model and analyze the relationship between a dependent variable and one or more independent variables.

Correlation: A measure of the strength and direction of the linear relationship between two variables.

Least Squares Method: A technique that finds the line that minimizes the sum of the squared vertical distances from each data point to the line.

Complete regression methodology:
  1. Collect data: Gather paired observations for variables of interest
  2. Create scatter plot: Visualize the relationship between variables
  3. Calculate correlation coefficient: Assess strength and direction of relationship
  4. Determine regression line: Calculate slope and y-intercept
  5. Assess model fit: Examine residuals and goodness-of-fit measures
  6. Make predictions: Use the model for forecasting within or beyond observed range
Tip 1: Always plot your data first to visualize patterns and outliers.
Tip 2: Correlation does not imply causation - be careful about cause-and-effect conclusions.
Tip 3: Extrapolation (predicting outside observed range) is generally unreliable.
Tip 4: A correlation coefficient near 0 doesn't mean there's no relationship - just not a linear one.
Common errors: Confusing correlation with causation, extrapolating too far beyond data range, ignoring outliers.
Exam preparation: Practice calculating correlation coefficients, interpreting regression equations, understanding residual analysis.
Essential Rules:

• Correlation Range: -1 ≤ r ≤ 1

• Perfect Correlation: r = ±1 indicates perfect linear relationship

• No Linear Correlation: r = 0 indicates no linear relationship

• Regression Line: Always passes through the point (x̄, ȳ)

Exercise with Visualization: Regression Models
Exercise 6: Comparing Regression Models
Visualize different regression models for the same dataset:
Linear: \(y = 1.2x + 3\)
Quadratic: \(y = 0.1x^2 + 0.8x + 2\)
Exponential: \(y = 2.5e^{0.1x}\)

Analysis: The chart shows how different models fit the same data differently.

  • Linear model: Best for data with constant rate of change
  • Quadratic model: Captures curvature in the data
  • Exponential model: Suitable for data with increasing rate of change
  • Choose model based on pattern and correlation coefficient

Questions & Answers

Question: What's the difference between correlation and regression? Are they the same thing?

Answer: While related, correlation and regression serve different purposes:

Correlation:

  • Measures the strength and direction of a linear relationship
  • Does not distinguish between independent and dependent variables
  • Results in a single number (correlation coefficient r)
  • Range: -1 ≤ r ≤ 1
  • Example: "Hours studied and test scores are strongly correlated"

Regression:

  • Creates a model to predict one variable from another
  • Distinguishes between independent (predictor) and dependent (response) variables
  • Results in an equation (regression line)
  • Used for prediction and forecasting
  • Example: "Test score = 5 + 2(hours studied)"

Think of correlation as describing the relationship, and regression as using that relationship for prediction.

Question: How do I interpret different values of the correlation coefficient? What does r = 0.8 mean compared to r = -0.5?

Answer: The correlation coefficient has two components: strength and direction.

Direction:

  • Positive r (> 0): As x increases, y tends to increase
  • Negative r (< 0): As x increases, y tends to decrease
  • Zero r (= 0): No linear relationship

Strength: Determined by |r| (absolute value)

  • |r| = 1: Perfect linear relationship
  • 0.8 ≤ |r| < 1: Very strong correlation
  • 0.6 ≤ |r| < 0.8: Strong correlation
  • 0.4 ≤ |r| < 0.6: Moderate correlation
  • 0.2 ≤ |r| < 0.4: Weak correlation
  • 0 ≤ |r| < 0.2: Very weak or no correlation

So r = 0.8 means very strong positive correlation, while r = -0.5 means moderate negative correlation.

Question: What impact do outliers have on regression analysis? Should I remove them?

Answer: Outliers can significantly affect regression analysis:

Impact of outliers:

  • Can dramatically change the slope and y-intercept of the regression line
  • Can reduce the correlation coefficient, making the relationship appear weaker
  • Can create leverage points that pull the regression line toward them
  • Can increase the sum of squared residuals

Decision about removal:

  1. Investigate: Determine if outlier is due to error in data collection
  2. Assess: See how much it affects the regression results
  3. Consider: Whether the outlier represents a valid but unusual observation
  4. Decide: Remove if clearly an error; keep if it's a valid data point

Always report whether outliers were removed and why. Sometimes it's better to perform analysis both with and without outliers to compare results.

Question: How accurate are predictions made using regression equations? What factors affect accuracy?

Answer: The accuracy of regression predictions depends on several factors:

Factors affecting accuracy:

  • Correlation strength: Higher |r| values generally mean more accurate predictions
  • Data quality: Clean, representative data yields better models
  • Sample size: Larger samples typically provide more reliable estimates
  • Range of data: Interpolation is more accurate than extrapolation
  • Model appropriateness: Linear model for linear relationships, etc.

Measuring accuracy:

  • Coefficient of determination (r²): Proportion of variance explained by the model
  • Residual analysis: Examining differences between actual and predicted values
  • Standard error of estimate: Average deviation of actual values from predicted values

Remember: All predictions have uncertainty, and regression models are only approximations of real-world relationships.

Question: What should I do if my data doesn't show a linear relationship? Can I still use regression?

Answer: Yes, there are regression methods for non-linear relationships:

Types of non-linear regression:

  • Quadratic regression: For parabolic relationships (y = ax² + bx + c)
  • Exponential regression: For growth/decay patterns (y = abˣ)
  • Power regression: For relationships like y = axᵇ
  • Logarithmic regression: For logarithmic patterns (y = a + b ln x)

Approaches:

  1. Visual inspection: Look at scatter plot to identify pattern
  2. Transformation: Apply logarithms or other functions to linearize data
  3. Technology: Use calculators or software to find best-fitting model
  4. Comparison: Compare correlation coefficients of different models

Always choose the model that best represents the underlying relationship and has the highest correlation coefficient or lowest residual sum of squares.