Solved Exercises on Linear Regression Basics

Master linear regression basics: equation calculation, correlation, prediction, and statistical interpretation through these 5 detailed exercises.

Solution: Exercises 1 to 3
1 Basic Linear Regression Calculation
Exercise 1
Find the equation of the least squares regression line for the following data showing hours studied and exam scores:

Hours Studied (x): [2, 3, 5, 6, 7, 8, 9, 10, 12, 15]
Exam Scores (y): [55, 60, 68, 72, 75, 78, 82, 85, 90, 95]

Calculate the regression equation and interpret the slope and y-intercept.
Definition:

Least Squares Regression Line: The line that minimizes the sum of the squares of the vertical distances from each data point to the line. It provides the best linear fit to the data.

Regression Equation: y = mx + b, where m is the slope and b is the y-intercept.

Slope (m): The rate of change of y with respect to x, calculated as m = Σ[(xi - x̄)(yi - ȳ)] / Σ(xi - x̄)².

Y-Intercept (b): The value of y when x = 0, calculated as b = ȳ - mx̄.

Linear Regression Calculation Method:
  1. Calculate means: x̄ and ȳ
  2. Calculate deviations: (xi - x̄) and (yi - ȳ)
  3. Calculate numerator: Σ[(xi - x̄)(yi - ȳ)]
  4. Calculate denominator: Σ(xi - x̄)²
  5. Calculate slope: m = numerator/denominator
  6. Calculate y-intercept: b = ȳ - mx̄
  7. Write equation: y = mx + b
Step 1: Calculate Means

x̄ = (2 + 3 + 5 + 6 + 7 + 8 + 9 + 10 + 12 + 15) / 10 = 77 / 10 = 7.7

ȳ = (55 + 60 + 68 + 72 + 75 + 78 + 82 + 85 + 90 + 95) / 10 = 760 / 10 = 76

Step 2: Calculate Deviations and Products

For each data point, calculate (xi - x̄), (yi - ȳ), and their product:

Example for (2, 55): (2 - 7.7) = -5.7, (55 - 76) = -21, (-5.7)(-21) = 119.7

Continuing for all points: Σ[(xi - x̄)(yi - ȳ)] = 451.0

Step 3: Calculate Denominator

Σ(xi - x̄)² = (-5.7)² + (-4.7)² + ... + (7.3)² = 144.1

Step 4: Calculate Slope

m = 451.0 / 144.1 = 3.13

Step 5: Calculate Y-Intercept

b = ȳ - mx̄ = 76 - (3.13)(7.7) = 76 - 24.10 = 51.90

Step 6: Write Regression Equation

y = 3.13x + 51.90

Step 7: Interpret Parameters

Slope: For each additional hour studied, exam score increases by 3.13 points

Y-Intercept: Students who study 0 hours are predicted to score 51.90 points

y = 3.13x + 51.90
Regression equation: y = 3.13x + 51.90
Slope: 3.13 points per hour studied
Y-intercept: 51.90 points when x = 0
Final answer:

The equation of the least squares regression line is y = 3.13x + 51.90. The slope of 3.13 indicates that for each additional hour studied, the exam score is predicted to increase by 3.13 points. The y-intercept of 51.90 suggests that students who study 0 hours would be predicted to score 51.90 points.

Applied rules:

• Least Squares Formula: Minimizes sum of squared residuals

• Slope Interpretation: Rate of change in y per unit change in x

• Y-Intercept Meaning: Predicted y-value when x = 0

• Regression Line Property: Passes through point (x̄, ȳ)

2 Correlation and Coefficient of Determination
Exercise 2
Using the same data from Exercise 1, calculate the correlation coefficient (r) and the coefficient of determination (r²). Interpret both values in the context of the relationship between hours studied and exam scores.
Definition:

Correlation Coefficient (r): A measure of the strength and direction of the linear relationship between two variables. Range: -1 ≤ r ≤ 1.

Coefficient of Determination (r²): The proportion of variance in the dependent variable that is explained by the independent variable. Range: 0 ≤ r² ≤ 1.

Strength Interpretation: |r| > 0.8 (strong), 0.5 < |r| < 0.8 (moderate), |r| < 0.5 (weak).

r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}}
Correlation Coefficient
Step 1: Calculate Numerator

From Exercise 1: Σ[(xi - x̄)(yi - ȳ)] = 451.0

Step 2: Calculate Denominator Components

Σ(xi - x̄)² = 144.1 (calculated in Exercise 1)

Σ(yi - ȳ)² = (55-76)² + (60-76)² + ... + (95-76)² = 1840

Step 3: Calculate Correlation Coefficient

r = 451.0 / √(144.1 × 1840) = 451.0 / √265,144 = 451.0 / 514.92 = 0.876

Step 4: Calculate Coefficient of Determination

r² = (0.876)² = 0.767

Step 5: Interpret Correlation Coefficient

r = 0.876 indicates a strong positive linear relationship between hours studied and exam scores.

Step 6: Interpret Coefficient of Determination

r² = 0.767 means that 76.7% of the variation in exam scores can be explained by the number of hours studied.

Step 7: Assess Model Quality

With r² = 0.767, the model explains a substantial portion of the variation in exam scores, indicating a good fit.

r = 0.876, r² = 0.767
Correlation: r = 0.876 (strong positive)
Determination: r² = 0.767 (76.7% explained)
Final answer:

The correlation coefficient is r = 0.876, indicating a strong positive linear relationship between hours studied and exam scores. The coefficient of determination is r² = 0.767, meaning that 76.7% of the variation in exam scores can be explained by the number of hours studied. This indicates a good model with strong predictive power.

Applied rules:

• Correlation Range: -1 ≤ r ≤ 1

• Strength Interpretation: |r| > 0.8 indicates strong relationship

• Coefficient of Determination: r² represents proportion of variance explained

• Positive Correlation: r > 0 indicates direct relationship

3 Making Predictions with Regression
Exercise 3
Using the regression equation from Exercise 1 (y = 3.13x + 51.90), make predictions for the following scenarios:

1. Predict the exam score for a student who studies 11 hours
2. If a student wants to score at least 85 points, how many hours should they study?
3. Predict the score for 0 hours of study and comment on the result

Discuss the reliability of these predictions.
Definition:

Prediction: Using the regression equation to estimate the value of the dependent variable (y) for a given value of the independent variable (x).

Interpolation: Making predictions within the range of the observed data, which is generally more reliable.

Extrapolation: Making predictions outside the range of the observed data, which is less reliable.

Prediction Interval: A range of values that estimates where a new observation is likely to fall.

Step 1: Prediction for 11 Hours of Study

Using y = 3.13x + 51.90 with x = 11:

y = 3.13(11) + 51.90 = 34.43 + 51.90 = 86.33

Predicted score: 86.33 points

Step 2: Hours Needed for 85 Points

Set y = 85 and solve for x:

85 = 3.13x + 51.90

85 - 51.90 = 3.13x

33.10 = 3.13x

x = 33.10 / 3.13 = 10.57

Approximately 10.6 hours of study needed

Step 3: Prediction for 0 Hours of Study

Using y = 3.13x + 51.90 with x = 0:

y = 3.13(0) + 51.90 = 51.90

Predicted score: 51.90 points

Step 4: Reliability Assessment

For 11 hours: Interpolation (within data range 2-15), highly reliable

For 85 points: Based on interpolation, reliable

For 0 hours: Extrapolation (outside data range), less reliable

Step 5: Contextual Considerations

The prediction for 0 hours is questionable since no student in the dataset studied 0 hours. The y-intercept represents a theoretical value rather than an observed relationship.

Step 6: Model Limitations

Predictions should only be made within the range of observed data when possible. The model may not hold for extreme values of x.

Predictions
11 hours: 86.33 points
85 points: 10.6 hours needed
0 hours: 51.90 points (extrapolation)
Final answer:

1) A student studying 11 hours is predicted to score 86.33 points. 2) To score at least 85 points, a student should study approximately 10.6 hours. 3) A student studying 0 hours is predicted to score 51.90 points, though this extrapolation is less reliable since it's outside the observed data range.

Applied rules:

• Interpolation vs Extrapolation: Interpolation is more reliable than extrapolation

• Prediction Accuracy: Depends on correlation strength and data range

• Model Limitations: Be cautious with extreme values

• Practical Interpretation: Consider real-world feasibility of predictions

Key Concepts and Statistical Measures for Linear Regression
y = mx + b
Regression Equation
Slope
m = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}
Regression Slope
Y-Intercept
b = \bar{y} - m\bar{x}
Regression Intercept
Correlation
r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}}
Pearson Correlation
Key definitions:

Linear Regression: A statistical method used to model the relationship between a dependent variable and one or more independent variables by fitting a linear equation to observed data.

Least Squares Method: The approach that minimizes the sum of the squares of the vertical distances from each data point to the regression line.

Regression Line: The straight line that best represents the relationship between variables in a scatterplot.

Correlation Coefficient (r): A measure of the strength and direction of the linear relationship between two variables, ranging from -1 to +1.

Coefficient of Determination (r²): The proportion of variance in the dependent variable that is explained by the independent variable.

Positive Correlation: When variables move in the same direction (r > 0).

Negative Correlation: When variables move in opposite directions (r < 0).

Residual: The difference between the observed value and the predicted value from the regression line.

Interpolation: Making predictions within the range of observed data values.

Extrapolation: Making predictions outside the range of observed data values.

Linear Regression Analysis Methodology:
  1. Data Collection: Gather paired observations of dependent and independent variables
  2. Scatterplot Creation: Visualize the relationship between variables
  3. Descriptive Statistics: Calculate means, standard deviations, and basic summaries
  4. Regression Calculation: Calculate slope and y-intercept using least squares method
  5. Correlation Assessment: Calculate correlation coefficient and coefficient of determination
  6. Model Evaluation: Assess the goodness of fit and assumptions
  7. Prediction: Use the model to make predictions within appropriate bounds
  8. Interpretation: Explain the practical meaning of the regression parameters
Tip 1: Always plot your data first to visually confirm a linear relationship.
Tip 2: The regression line always passes through the point (x̄, ȳ).
Tip 3: Be cautious about extrapolation beyond the observed data range.
Tip 4: Correlation does not imply causation - consider other variables.
Tip 5: A high r² value indicates good fit but doesn't guarantee the model is appropriate.
Common errors: Miscalculating means, incorrect application of formulas, confusing correlation with causation, extrapolating beyond data range, misinterpreting negative slopes, calculation errors in sums of squares, not checking for linearity, ignoring outliers.
Exam preparation: Practice manual calculations, understand the meaning of slope and intercept, know how to interpret r and r², be able to make predictions, understand the difference between correlation and causation, recognize when regression is appropriate.
Essential Formulas and Rules:

• Regression Equation: y = mx + b

• Slope Formula: m = Σ[(xi - x̄)(yi - ȳ)] / Σ(xi - x̄)²

• Y-Intercept Formula: b = ȳ - mx̄

• Correlation Formula: r = Σ[(xi - x̄)(yi - ȳ)] / √[Σ(xi - x̄)²Σ(yi - ȳ)²]

• Coefficient of Determination: r² = (correlation coefficient)²

• Correlation Range: -1 ≤ r ≤ 1

• Strength Interpretation: |r| > 0.8 (strong), 0.5 < |r| < 0.8 (moderate), |r| < 0.5 (weak)

• Linearity Assumption: Correlation measures only linear relationships

Solution: Exercises 4 to 5
4 Real-World Application: Business Analysis
Exercise 4
A company analyzed the relationship between advertising expenditure (in thousands of dollars) and monthly sales (in thousands of dollars). The data yielded the regression equation: y = 2.8x + 45, where x is advertising expenditure and y is sales. Calculate r² = 0.84 and interpret the model's effectiveness. If the company wants to achieve $100,000 in sales, how much should they spend on advertising?
Definition:

Business Application: Using regression analysis to make informed decisions about resource allocation and forecasting.

Return on Investment (ROI): The financial gain relative to the cost of an investment, which can be estimated using regression models.

Step 1: Interpret the Regression Equation

Slope (2.8): Each $1,000 increase in advertising leads to $2,800 increase in sales

Y-Intercept (45): Base sales of $45,000 without any advertising

Step 2: Interpret Coefficient of Determination

r² = 0.84 means that 84% of the variation in sales can be explained by advertising expenditure. This indicates a very strong model with high predictive power.

Step 3: Calculate Required Advertising for $100,000 Sales

Set y = 100 (since y is in thousands) and solve for x:

100 = 2.8x + 45

100 - 45 = 2.8x

55 = 2.8x

x = 55 / 2.8 = 19.64

Step 4: Interpret the Result

To achieve $100,000 in sales, the company should spend approximately $19,640 on advertising.

Step 5: Assess Model Effectiveness

With r² = 0.84, the model explains 84% of the variance in sales, indicating high effectiveness. Only 16% of the variation remains unexplained by advertising expenditure.

Step 6: Business Recommendation

The model suggests a strong positive return on advertising investment (ROI of 2.8:1), making it valuable for budget planning and sales forecasting.

y = 2.8x + 45, r² = 0.84
r² = 0.84 (84% of variance explained)
$100K sales requires $19.64K advertising
Strong model effectiveness
Final answer:

The regression model is highly effective with r² = 0.84, explaining 84% of the variation in sales. To achieve $100,000 in sales, the company should spend approximately $19,640 on advertising. The model shows a strong positive relationship with an ROI of 2.8:1 for advertising investment.

Applied rules:

• Business Interpretation: Relate slope and intercept to practical business metrics

• Model Effectiveness: Higher r² indicates better predictive power

• Return on Investment: Use slope to estimate ROI

• Prediction Accuracy: Consider the model's r² when making forecasts

5 Advanced Analysis: Multiple Predictions
Exercise 5
Using the regression equation y = -1.2x + 85 (where x is temperature in Celsius and y is ice cream sales in hundreds of dollars), calculate predictions for temperatures of 15°C, 25°C, and 35°C. Also find the temperature that would result in $50,000 in sales. Discuss the practical limitations of this model.
Definition:

Negative Correlation: A relationship where one variable increases as the other decreases, indicated by a negative slope.

Model Limitations: Constraints on the applicability of a regression model, including range restrictions and assumptions.

Step 1: Prediction for 15°C

y = -1.2(15) + 85 = -18 + 85 = 67

Predicted sales: $6,700 (since y is in hundreds)

Step 2: Prediction for 25°C

y = -1.2(25) + 85 = -30 + 85 = 55

Predicted sales: $5,500

Step 3: Prediction for 35°C

y = -1.2(35) + 85 = -42 + 85 = 43

Predicted sales: $4,300

Step 4: Find Temperature for $50,000 Sales

Set y = 500 (since $50,000 = 500 hundreds) and solve for x:

500 = -1.2x + 85

500 - 85 = -1.2x

415 = -1.2x

x = 415 / (-1.2) = -345.83°C

Step 5: Analyze Results

The prediction for $50,000 sales yields an impossible temperature of -345.83°C, indicating the model breaks down for extreme values.

Step 6: Discuss Limitations

The model shows decreasing sales with increasing temperature, which is counterintuitive for ice cream sales. The model is likely only valid within a specific temperature range where the relationship holds.

y = -1.2x + 85
15°C: $6,700
25°C: $5,500
35°C: $4,300
$50K sales: -345.83°C (impossible)
Final answer:

Predictions: 15°C → $6,700, 25°C → $5,500, 35°C → $4,300. The model predicts that $50,000 in sales would require an impossible temperature of -345.83°C. This indicates the model has significant limitations and may not be appropriate for the given scenario, as ice cream sales typically increase with temperature.

Applied rules:

• Model Validity: Check if predictions make practical sense

• Contextual Relevance: Ensure the model aligns with real-world expectations

• Extreme Value Warning: Be cautious of predictions that yield impossible results

• Domain Knowledge: Use practical knowledge to assess model appropriateness

Detailed Summary: Linear Regression Basics
y = mx + b
Core Regression Equation
Key definitions:

Linear Regression: A statistical method used to model the relationship between a dependent variable (y) and one independent variable (x) by fitting a linear equation to observed data. It aims to find the line that best represents the relationship between variables.

Least Squares Method: The mathematical approach that finds the regression line by minimizing the sum of the squares of the vertical distances (residuals) from each data point to the line. This ensures the best possible fit to the data.

Regression Line: The straight line that best represents the relationship between two variables in a scatterplot. It is determined by the equation y = mx + b, where m is the slope and b is the y-intercept.

Slope (m): The rate of change of the dependent variable (y) with respect to the independent variable (x). It represents the steepness of the regression line and indicates the direction and strength of the relationship.

Y-Intercept (b): The value of the dependent variable (y) when the independent variable (x) equals zero. It represents the baseline value of the dependent variable.

Correlation Coefficient (r): A measure of the strength and direction of the linear relationship between two variables. It ranges from -1 to +1, where values closer to the extremes indicate stronger relationships.

Coefficient of Determination (r²): The proportion of variance in the dependent variable that is explained by the independent variable. It ranges from 0 to 1, with higher values indicating better model fit.

Positive Correlation: A relationship where both variables tend to increase together, indicated by a positive correlation coefficient.

Negative Correlation: A relationship where one variable increases as the other decreases, indicated by a negative correlation coefficient.

Residual: The difference between the observed value of the dependent variable and the value predicted by the regression line. Residual = Observed - Predicted.

Interpolation: Making predictions using the regression model within the range of the observed data values, which is generally more reliable.

Extrapolation: Making predictions using the regression model outside the range of the observed data values, which is less reliable and should be done cautiously.

Complete Linear Regression Analysis Process:
  1. Data Collection: Gather paired observations of the dependent and independent variables
  2. Data Visualization: Create a scatterplot to visually inspect the relationship
  3. Descriptive Statistics: Calculate means, standard deviations, and basic summaries
  4. Regression Calculation: Calculate slope and y-intercept using the least squares formulas
  5. Correlation Assessment: Calculate correlation coefficient and coefficient of determination
  6. Model Evaluation: Assess the goodness of fit and check assumptions
  7. Residual Analysis: Examine residuals for patterns or systematic errors
  8. Prediction: Use the model to make predictions within appropriate bounds
  9. Interpretation: Explain the practical meaning of the regression parameters
  10. Validation: Consider the model's limitations and scope of applicability
Tip 1: Always examine the scatterplot first to verify a linear relationship exists.
Tip 2: The regression line always passes through the point (x̄, ȳ), which is the center of the data.
Tip 3: Be very cautious when extrapolating beyond the range of observed data.
Tip 4: Remember that correlation does not imply causation - there may be lurking variables.
Tip 5: The sum of residuals in least squares regression always equals zero.
Tip 6: A high r² value indicates good fit but doesn't guarantee the model is appropriate for all purposes.
Common errors: Miscalculating means and deviations, incorrect application of formulas, confusing correlation with causation, extrapolating beyond data range, misinterpreting negative slopes, calculation errors in sums of squares, not checking for linearity, ignoring outliers, misstating the direction of relationships, failing to consider the context of the data.
Exam preparation: Practice manual calculations, understand the meaning of slope and intercept, know how to interpret r and r², be able to make predictions, understand residual analysis, distinguish between correlation and causation, recognize when regression is appropriate, know the formulas for slope and intercept.
Essential Formulas and Rules:

• Regression Equation: y = mx + b

• Slope Formula: m = Σ[(xi - x̄)(yi - ȳ)] / Σ(xi - x̄)²

• Y-Intercept Formula: b = ȳ - mx̄

• Correlation Formula: r = Σ[(xi - x̄)(yi - ȳ)] / √[Σ(xi - x̄)²Σ(yi - ȳ)²]

• Coefficient of Determination: r² = (correlation coefficient)²

• Correlation Range: -1 ≤ r ≤ 1

• Strength Interpretation: |r| > 0.8 (strong), 0.5 < |r| < 0.8 (moderate), |r| < 0.5 (weak)

• Linearity Assumption: Correlation measures only linear relationships

• Regression Line Property: Passes through point (x̄, ȳ)

• Sum of Residuals: Equals zero for least squares regression

Linear Regression Analysis Framework
📊
Step 1: Data Collection
Gather paired observations of dependent and independent variables
Ensure data quality and completeness
Step 2: Visualization
Create scatterplot to inspect relationship
Look for linear patterns
Step 3: Calculation
Compute regression parameters
m = Σ[(xi-x̄)(yi-ȳ)]/Σ(xi-x̄)²
Step 4: Assessment
Calculate correlation and determination
r and r² measure relationship strength
Step 5: Evaluation
Assess model quality and assumptions
Check residuals for patterns
Step 6: Application
Make predictions and draw conclusions
Use within appropriate data range
Analysis Process
1
Collect
2
Visualize
3
Calculate
4
Assess
5
Evaluate
6
Apply
Key Relationships
Slope: Rate of change in y per unit x
Intercept: Value of y when x = 0
Correlation: Strength and direction of relationship
Determination: Proportion of variance explained
Always verify assumptions and limitations!

Questions & Answers

Question: How do I interpret the slope and y-intercept in a regression equation?

Answer: The slope and y-intercept have specific meanings:

  • Slope (m): Represents the rate of change of the dependent variable (y) for a one-unit increase in the independent variable (x). It tells you how much y changes when x increases by 1.
  • Y-Intercept (b): Represents the value of y when x equals zero. This is the starting value of the relationship when the independent variable is absent.

In context: If predicting sales from advertising expenditure, slope = $2,800 means each $1,000 in advertising brings $2,800 in sales, and y-intercept = $45,000 means base sales without advertising.

Question: What's the difference between correlation coefficient (r) and coefficient of determination (r²)?

Answer: These are related but measure different things:

  • Correlation Coefficient (r): Measures the strength and direction of the linear relationship. Range: -1 ≤ r ≤ 1. Can be negative.
  • Coefficient of Determination (r²): Measures the proportion of variance in y explained by x. Range: 0 ≤ r² ≤ 1. Always positive.

For example, if r = 0.9, then r² = 0.81, meaning 81% of the variation in y is explained by x, while 19% remains unexplained.

Question: What does it mean when the correlation coefficient is negative?

Answer: A negative correlation coefficient indicates an inverse relationship:

  • As one variable increases, the other variable tends to decrease
  • The variables move in opposite directions
  • The strength of the relationship is indicated by the absolute value of r
  • r = -0.8 indicates a strong negative relationship

Example: The relationship between outdoor temperature and heating costs is typically negative - as temperature increases, heating costs decrease.

Question: When is it appropriate to use the regression equation for predictions?

Answer: Consider these guidelines for appropriate predictions:

  • Within Range: Make predictions only within the range of x-values used to create the model (interpolation).
  • Model Quality: Ensure the model has a reasonable r² value and passes diagnostic checks.
  • Context: Consider if the relationship makes sense in the real-world context.
  • Caution: Avoid extrapolation (predicting outside the data range) as it's unreliable.

For example, if your data ranges from x = 2 to x = 15, don't predict for x = 20 without additional justification.

Question: What does a high r² value tell us, and what are its limitations?

Answer: High r² has both benefits and limitations:

  • Benefits: High r² (close to 1) indicates the model explains a large proportion of the variance in the dependent variable.
  • Limitations: High r² doesn't prove causation, may result from overfitting, doesn't guarantee the model is appropriate.
  • Considerations: Always examine residuals, check for outliers, and consider the context.

A model with r² = 0.95 may fit well statistically but could be inappropriate if there's no logical relationship between variables.