Skip to main content

Chapter 11: Linear Regression and Regression Project

MODULE 3 | WEEK 11

Chapter 11: Linear Regression and Regression Project​

So far we have described data and tested claims. Now the biggest step: prediction . Linear Regression is the very first and most widely used algorithm of machine learning. Once you understand it, many later ML algorithms will feel easy. First Covariance and Correlation (the relationship between two things), then regression, and finally a complete project.

11.1 Covariance and Correlation​

Do two things change together? For example, when study hours increase, do marks increase too? There are two ways to measure this relationship.

Covariance​

Covariance tells whether two things change together or not, and in which direction .

  • Positive covariance: if one rises, the other rises too.

  • Negative covariance: if one rises, the other falls.

  • Near zero: no straight-line relationship.

The problem is that the covariance number depends on units (measure height in cm or in metres and the number changes), so by reading it we cannot tell "how strong the relationship is". The solution to this problem is correlation .

Correlation (Pearson r)​

Correlation = covariance / (std of x x std of y) . It standardizes the covariance, so it always stays between -1 and +1 and the effect of units disappears.

Value of rMeaning
+0.7 to +1Strong positive relationship
+0.3 to +0.7Moderate positive relationship
-0.3 to +0.3Weak or no (linear) relationship
-0.7 to -0.3Moderate negative relationship
-1 to -0.7Strong negative relationship
import numpy as np
import pandas as pd
from scipy import stats
import matplotlib.pyplot as plt
np.random.seed(42)
hours = np.random.uniform(0, 10, 80)
marks = 40 + 5 * hours + np.random.normal(0, 8, 80)
print("Covariance:", np.cov(hours, marks)[0, 1].round(2))
print("Correlation:", np.corrcoef(hours, marks)[0, 1].round(3))
df = pd.DataFrame({"hours": hours, "marks": marks})
print(df.corr().round(3))
OUTPUT​
Covariance: 50.08
Correlation: 0.906
hours marks
hours 1.000 0.906
marks 0.906 1.000

Pearson vs Spearman​

  • Pearson: only a measures straight-line (linear) relationships.

  • Spearman: works on the ranking (order) of values, so it measures every steadily increasing (or decreasing) relationship (monotonic) (straight or not) and is less affected by outliers.

x = np.arange(1, 21)
y = x ** 3 # increases, but not in a straight line
print("Pearson :", round(stats.pearsonr(x, y)[0], 3))
print("Spearman:", round(stats.spearmanr(x, y)[0], 3)) # exactly 1.0
OUTPUT​
Pearson : 0.922
Spearman: 1.0

Pictures of different relationships​

np.random.seed(0)
n = 100
x = np.random.normal(0, 1, n)
cases = {
"r = +0.9": 0.9 * x + np.random.normal(0, 0.45, n),
"r = +0.5": 0.5 * x + np.random.normal(0, 0.87, n),
"r = 0": np.random.normal(0, 1, n),
"r = -0.5": -0.5 * x + np.random.normal(0, 0.87, n),
"r = -0.9": -0.9 * x + np.random.normal(0, 0.45, n),
"A relationship, but r ~ 0": x ** 2 + np.random.normal(0, 0.2, n),
}
fig, axes = plt.subplots(2, 3, figsize=(10, 6))
for ax, (title, y) in zip(axes.ravel(), cases.items()):
ax.scatter(x, y, s=12, color="#6B5B95", alpha=0.7)
ax.set_title(f"{title} (actual r = {np.corrcoef(x, y)[0, 1]:.2f})", fontsize=10)
plt.tight_layout()
plt.show()
OUTPUT (chart)​
Pictures of different relationships: r = +0.9 (asli r = 0.90) r = +0.5 (asli r = 0.57) r = 0 (asli r = -0.06) -1 -1 -1 -2 -2 r = -0.5 (asli r = -0.51) (asli r= -0.90) Rishta hai, magar r ~ 0 r = -0.9 (asli r =...
Pictures of different relationships: r = +0.9 (asli r = 0.90) r = +0.5 (asli r = 0.57) r = 0 (asli r = -0.06) -1 -1 -1 -2 -2 r = -0.5 (asli r = -0.51) (asli r= -0.90) Rishta hai, magar r ~ 0 r = -0.9 (asli r =...

The last chart is a very important lesson: x and y have a clear relationship (U-shape), but Pearson r is almost 0 , because it is not a straight-line relationship . So do not trust the number alone; always look at the scatter plot too.

Anscombe's Quartet: four datasets, identical numbers​

The statistician Francis Anscombe created four different datasets whose mean, variance, correlation and regression line are exactly the same , yet the pictures are completely different.

import seaborn as sns
anscombe = sns.load_dataset("anscombe")
summary = anscombe.groupby("dataset").agg(
x_mean=("x", "mean"), y_mean=("y", "mean"), y_std=("y", "std"))
summary["corr"] = anscombe.groupby("dataset").apply(lambda d: d["x"].corr(d["y"]),
include_groups=False)
print(summary.round(2))
OUTPUT​
x_mean y_mean y_std corr
dataset
I 9.0 7.5 2.03 0.82
II 9.0 7.5 2.03 0.82
III 9.0 7.5 2.03 0.82
IV 9.0 7.5 2.03 0.82
sns.lmplot(data=anscombe, x="x", y="y", col="dataset", col_wrap=4, height=2.6,
ci=None, scatter_kws={"color": "#6B5B95"}, line_kws={"color": "red"})
plt.show()
OUTPUT (chart)​
Anscombe's Quartet: four datasets, identical numbers: dataset =1 dataset = Il dataset = 1 dataset = IV 10 10 15 10 15 10 15 10 15
Anscombe's Quartet: four datasets, identical numbers: dataset =1 dataset = Il dataset = 1 dataset = IV 10 10 15 10 15 10 15 10 15
LESSON FROM ANSCOMBE​

The numbers of all four are the same, but the first is fine, the second is curved, in the third an outlier is pulling the line, and in the fourth everything rests on a single point. Lesson: always plot the data before modelling.

11.2 Simple Linear Regression​

Simple Linear Regression uses one input (x) to predict one output (y) , by drawing a straight line through the data:

y = b0 + b1 x​
  • y: the thing we want to predict (target / dependent variable), e.g. marks.

  • x: what we predict from (feature / independent variable), e.g. study hours.

  • b1 (slope): how much y changes when there is a 1-unit increase in x.

  • b0 (intercept): what y will be when x = 0.

Which is the best line? (Least Squares)​

Thousands of lines can pass through the data. Best is the one that makes the sum of squared errors (residuals) the smallest. Residual = actual y - predicted y (i.e. how far each point is from the line).

np.random.seed(5)
x = np.random.uniform(1, 10, 15)
y = 35 + 6 * x + np.random.normal(0, 7, 15)
b1, b0 = np.polyfit(x, y, 1)
pred = b0 + b1 * x
plt.figure(figsize=(7.5, 4.2))
plt.scatter(x, y, color="#6B5B95", s=45, zorder=3, label="Actual data")
xs = np.linspace(0, 11, 50)
plt.plot(xs, b0 + b1 * xs, color="red", linewidth=2, label="Best-fit line")
for xi, yi, pi in zip(x, y, pred):
plt.plot([xi, xi], [yi, pi], color="gray", linestyle=":", linewidth=1) # residual
plt.title("Short dotted lines = residuals (errors)")
plt.xlabel("Study hours (x)")
plt.ylabel("Marks (y)")
plt.legend()
plt.show()
OUTPUT (chart)​
Which is the best line? (Least Squares): Chhoti dottedlakeerein= residuals (galtiyan) Asli data 100 Best-fit line 90 80 Marks (y) 70 60 50 40 30 10 Parhaike ghantay (x)
Which is the best line? (Least Squares): Chhoti dottedlakeerein= residuals (galtiyan) Asli data 100 Best-fit line 90 80 Marks (y) 70 60 50 40 30 10 Parhaike ghantay (x)

Formula for slope and intercept​

b1 = covariance(x, y) / variance(x) and b0 = mean(y) - b1 x mean(x) . Let us compute it three ways:

# 1) By hand (formula)
b1_manual = np.cov(x, y, ddof=1)[0, 1] / np.var(x, ddof=1)
b0_manual = y.mean() - b1_manual * x.mean()
print("Manual :", round(b1_manual, 3), round(b0_manual, 3))
# 2) NumPy polyfit (degree 1 = straight line)
b1_np, b0_np = np.polyfit(x, y, 1)
print("polyfit :", round(b1_np, 3), round(b0_np, 3))
# 3) SciPy linregress (also gives r and p-value)
res = stats.linregress(x, y)
print("linregress:", round(res.slope, 3), round(res.intercept, 3),
"| r =", round(res.rvalue, 3), "| p =", f"{res.pvalue:.2e}")
OUTPUT​
Manual : 6.182 33.15
polyfit : 6.182 33.15
linregress: 6.182 33.15 | r = 0.883 | p = 1.28e-05

All three methods gave the same answer. This gives us the line hidden in the data, y = b0 + b1 x .

Interpretation (what the result means)​

print(f"Marks = {res.intercept:.1f} + {res.slope:.1f} x hours")
print("Slope:", round(res.slope, 2), "-> marks rise by about this much for each extra hour")
print("Intercept:", round(res.intercept, 1), "-> predicted marks for 0 hours of study")
new_hours = 8
print(f"Prediction for {new_hours} hours:", round(res.intercept + res.slope * new_hours, 1))
OUTPUT​
Marks = 33.1 + 6.2 x hours
Slope: 6.18 -> marks rise by about this much for each extra hour
Intercept: 33.1 -> predicted marks for 0 hours of study
Prediction for 8 hours: 82.6

COMMON MISTAKES

  • Extrapolation: predicting outside the range of the data (e.g. 1 to 10 hours). At 100 hours you would get "marks = 600", which is meaningless. The model is reliable only within the range it was built on.

  • Always treating the intercept as meaningful. When x = 0 is not even possible (such as height 0), the intercept has no real meaning.

  • Mistaking the slope for a cause . Regression shows a relationship; it does not prove causation.

R-squared (R<sup>2</sup> ): how good is the model?​

R<sup>2</sup> = 1 - (sum of squared residuals) / (total spread) . It means: the share of the spread in y that the model can explain .

  • R<sup>2</sup> = 0.8 means that of the variation in marks, 80% can be explained by study hours.

  • R<sup>2</sup> = 0 means the model is no better than the average. R<sup>2</sup> = 1 means perfectly correct.

  • In simple regression R<sup>2</sup> = r<sup>2</sup> (the square of the correlation).

pred = res.intercept + res.slope * x
ss_res = ((y - pred) ** 2).sum() # sum of squared residuals
ss_tot = ((y - y.mean()) ** 2).sum() # total spread
r2 = 1 - ss_res / ss_tot
print("R-squared:", round(r2, 3), "| r**2:", round(res.rvalue ** 2, 3))
OUTPUT​
R-squared: 0.78 | r**2: 0.78

11.3 statsmodels: the full regression report​

In Data Science, it is important to know not only the prediction but also how reliable each coefficient is . statsmodels library gives a complete statistical report. The Formula API ("y ~ x") is as easy as in the R language.

import statsmodels.formula.api as smf
np.random.seed(42)
hours = np.random.uniform(0, 10, 100)
marks = 40 + 5 * hours + np.random.normal(0, 8, 100)
df = pd.DataFrame({"hours": hours, "marks": marks})
model = smf.ols("marks ~ hours", data=df).fit()
print(model.summary())
OUTPUT​
OLS Regression Results
==============================================================================
Dep. Variable: marks R-squared: 0.785
Model: OLS Adj. R-squared: 0.782
Method: Least Squares F-statistic: 356.9
Date: Tue, 13 Oct 2026 Prob (F-statistic): 1.95e-34
Time: 11:52:00 Log-Likelihood: -339.09
No. Observations: 100 AIC: 682.2
Df Residuals: 98 BIC: 687.4
Df Model: 1
Covariance Type: nonrobust
==============================================================================
coef std err t P>|t| [0.025 0.975]
------------------------------------------------------------------------------
Intercept 41.7208 1.362 30.625 0.000 39.017 44.424
hours 4.6322 0.245 18.892 0.000 4.146 5.119
==============================================================================
Omnibus: 0.900 Durbin-Watson: 2.285
Prob(Omnibus): 0.638 Jarque-Bera (JB): 0.808
Skew: 0.217 Prob(JB): 0.668
Kurtosis: 2.929 Cond. No. 10.7
==============================================================================
Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly sp...

How to read the report​

PartMeaning
R-squaredHow much of the spread in marks the model explained
Adj. R-squaredThe form of R-squared that penalizes useless features (important in multiple regression)
coef(Intercept, hours)b0 and b1 (the effect of each x)
std errThe standard error of the coefficient: how uncertain the estimate is
**t and P>t
[0.025, 0.975]The 95% confidence interval of the coefficient
F-statistic, Prob (F)Whether the whole model is useful (does at least one x have an effect?)

See how everything from the previous chapters (hypothesis test, p-value, confidence interval) comes together in one place here. This is what ties it all together.

print("Coefficients:")
print(model.params.round(3))
print("\nP-values:")
print(model.pvalues.round(4))
print("\n95% CI:")
print(model.conf_int().round(3))
print("\nR-squared:", round(model.rsquared, 3))
OUTPUT​
Coefficients:
Intercept 41.721
hours 4.632
dtype: float64
P-values:
Intercept 0.0
hours 0.0
dtype: float64
95% CI:
0 1
Intercept 39.017 44.424
hours 4.146 5.119
R-squared: 0.785
# Prediction on new data
new = pd.DataFrame({"hours": [2, 5, 9]})
print(model.predict(new).round(1))
OUTPUT​
0 51.0
1 64.9
2 83.4
dtype: float64

11.4 Multiple Linear Regression​

In real life one thing (y) depends on many factors. Multiple Linear Regression has several inputs:

y = b0 + b1 x1 + b2 x2 + ... + bk xk

OUTPUT​
price size bedrooms age city
0 373.428765 728.924868 1 6.433031 Lahore
1 621.493903 2839.756377 3 29.361627 Lahore
2 531.945474 1815.227694 5 32.316518 Lahore
3 986.987204 2670.395533 3 27.956046 Islamabad
4 472.921537 3433.968536 5 38.593854 Karachi
m2 = smf.ols("price ~ size + bedrooms + age + C(city)", data=houses).fit()
print(m2.params.round(3))
print("R-squared:", round(m2.rsquared, 3), "| Adj R-squared:", round(m2.rsquared_adj, 3))
OUTPUT​
Intercept601.643
C(city)[T.Karachi]-594.459
C(city)[T.Lahore]-386.205
size0.129
bedrooms40.920
age-3.933
dtype: float64
`R-squared: 0.955`

The model found numbers close to the true coefficients : the effect of size ~0.12, bedrooms ~40, age ~-4. C(city) automatically turned the city into dummy variables (0/1 columns). The city that comes first alphabetically ( Islamabad ) became the base category , so the coefficients of Karachi and Lahore show the difference compared to Islamabad : Karachi about -600 and Lahore about -400, which match the differences in our true formula (Islamabad +600, Lahore +200, Karachi 0).

How to read a coefficient​

In multiple regression each coefficient means: holding everything else equal , the change in y for a 1-unit increase in this x. For example, size 0.12 means if the number of rooms, age and city are the same , then each extra square foot raises the price by about 0.12 lakh.

Why Adjusted R-squared?​

The more features you add, R<sup>2</sup> only goes up (even if the feature is useless). Adjusted R<sup>2</sup> penalizes useless features, so it is better for comparing models.

If two inputs are highly correlated with each other (such as size in sq ft and size in sq meter ) the model cannot decide which one has the effect, and the coefficients become unreliable. First look at the correlation matrix , and also check the VIF (Variance Inflation Factor; be suspicious if above 5).

from statsmodels.stats.outliers_influence import variance_inflation_factor
X = houses[["size", "bedrooms", "age"]].copy()
X.insert(0, "const", 1.0)
vif = pd.Series([variance_inflation_factor(X.values, i) for i in range(1, X.shape[1])],
index=["size", "bedrooms", "age"])
print(vif.round(2)) # fine if all are below 5
OUTPUT​
size 1.01
bedrooms 1.01
age 1.01
dtype: float64

11.5 Regression assumptions and Residual Plots​

Linear regression is reliable when the data meets some conditions. An easy way to remember them: LINE .

AssumptionHow to check
L**Linearity:**the relationship between x and y is like a straight
line
Scatter plot; no curve in the residuals vs fitted plot
I**Independence:**each observation is independent of the
others
How the data was collected (can be a problem in time
series)
N**Normality:**residuals are roughly NormalHistogram and Q-Q plot of residuals
E**Equal variance:**the spread of residuals is the same
everywhere (homoscedasticity)
Residuals vs fitted: an evenly wide band-like pattern
throughout
import statsmodels.api as sm
fitted = m2.fittedvalues
resid = m2.resid
fig, axes = plt.subplots(1, 3, figsize=(13, 3.8))
axes[0].scatter(fitted, resid, color="#6B5B95", alpha=0.6, s=18)
axes[0].axhline(0, color="red", linestyle="--")
axes[0].set_title("Residuals vs Fitted\n(no pattern = good)")
axes[0].set_xlabel("Fitted values")
axes[0].set_ylabel("Residuals")
axes[1].hist(resid, bins=20, color="#A4457A", edgecolor="white")
axes[1].set_title("Histogram of residuals\n(bell-shaped = good)")
sm.qqplot(resid, line="45", fit=True, ax=axes[2], markerfacecolor="#6B5B95", markersize=3)
axes[2].set_title("Q-Q plot\n(points on the line = Normal)")
plt.tight_layout()
plt.show()
OUTPUT (chart)​
11.5 Regression assumptions and Residual Plots: Residuals ka histogram Residuals vs Fitted Q-Q plot (koi pattern nahi = achha) (ghanti jaisa = achha) (nuqtay lakeer par = Normal) 25 150 20 100 Sample Quantiles 50 Residuals...
11.5 Regression assumptions and Residual Plots: Residuals ka histogram Residuals vs Fitted Q-Q plot (koi pattern nahi = achha) (ghanti jaisa = achha) (nuqtay lakeer par = Normal) 25 150 20 100 Sample Quantiles 50 Residuals...

All three plots show a healthy situation. Now see what happens when the relationship is not a straight line and we still fit a straight line:

np.random.seed(3)
xc = np.random.uniform(0, 10, 100)
yc = 3 + 0.5 * xc ** 2 + np.random.normal(0, 3, 100) # curved relationship
bad = smf.ols("y ~ x", data=pd.DataFrame({"x": xc, "y": yc})).fit()
fig, axes = plt.subplots(1, 2, figsize=(10, 3.8))
axes[0].scatter(xc, yc, color="#6B5B95", s=15)
axes[0].plot(np.sort(xc), bad.predict(pd.DataFrame({"x": np.sort(xc)})), color="red")
axes[0].set_title("The straight line is not capturing the curve")
axes[1].scatter(bad.fittedvalues, bad.resid, color="#A4457A", s=15)
axes[1].axhline(0, color="black", linestyle="--")
axes[1].set_title("U-shape pattern in residuals = Linearity fails")
plt.tight_layout()
plt.show()
OUTPUT (chart)​
11.5 Regression assumptions and Residual Plots: Seedhilakeer curve ko nahipakad rahi Residuals me U-shape pattern = Linearity fail 50 10.0 7.5 40 5.0 30 2.5 20 0.0 2.5 10 5.0 -7.5 10 10 20 30 40
11.5 Regression assumptions and Residual Plots: Seedhilakeer curve ko nahipakad rahi Residuals me U-shape pattern = Linearity fail 50 10.0 7.5 40 5.0 30 2.5 20 0.0 2.5 10 5.0 -7.5 10 10 20 30 40

The residuals show a clear U-shape pattern , which shows that a straight line is the wrong model for this data. Fix: add x squared ( x ** 2 ) (polynomial regression), or apply a log transform , or use another algorithm (Decision Tree, Random Forest), which come in later modules.

11.6 Evaluation Metrics and Train/Test Split​

Testing a model on the same data it learned from is cheating . It is as if you memorized the exact questions that would come in the paper before the exam and scored well, but what happens with new questions? That is why we split the data into two parts:

  • Train set (usually 80%): the model learns from this.

  • Test set (20%): the model has never seen this. Real performance is measured on this.

MetricFormula / IdeaMeaning
MAEMean Absolute Error: average ofactual - predicted
MetricFormula / IdeaMeaning
MSEMean Squared Error: (actual - predicted)<sup>2</sup>differences
of each value from the mean
Punishes large errors more
RMSEsqrt(MSE)Like MSE but in original units, the most common
R<sup>2</sup>1 - SSres / SStotHow much of the spread the model explained
(closer to 1 is better)
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
X = pd.get_dummies(houses[["size", "bedrooms", "age", "city"]], drop_first=True)
y = houses["price"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print("Train:", X_train.shape, "| Test:", X_test.shape)
model_sk = LinearRegression().fit(X_train, y_train)
pred = model_sk.predict(X_test)
print("MAE :", round(mean_absolute_error(y_test, pred), 2))
print("RMSE:", round(np.sqrt(mean_squared_error(y_test, pred)), 2))
print("R2 (test) :", round(r2_score(y_test, pred), 3))
print("R2 (train):", round(model_sk.score(X_train, y_train), 3))
OUTPUT​
Train: (160, 5) | Test: (40, 5)
MAE : 56.39
RMSE: 71.14
R2 (test) : 0.931
R2 (train): 0.961
DIFFERENCE BETWEEN THE TWO​

In this chapter we saw both statsmodels (statistics: p-values, interpretation) and scikit-learn (machine learning: prediction, train/test). The nature of statistics is to understand , the nature of ML is to predict . Both are needed.

11.7 Regression Model Project: Car Fuel Efficiency (mpg)​

Question: predict a car's fuel efficiency (mpg: miles per gallon) from its characteristics (weight, horsepower, model year, etc.), and understand which factor affects efficiency the most. Data: 398 cars (seaborn's mpg dataset).

Step 1: Load and inspect the data​

import seaborn as sns
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.formula.api as smf
cars = sns.load_dataset("mpg")
print(cars.shape)
print(cars.isnull().sum()[cars.isnull().sum() > 0])
print(cars.head(3))
OUTPUT​
(398, 9)
horsepower 6
dtype: int64
mpg cylinders displacement horsepower weight acceleration \
0 18.0 8 307.0 130.0 3504 12.0
1 15.0 8 350.0 165.0 3693 11.5
2 18.0 8 318.0 150.0 3436 11.0
model_year origin name
0 70 usa chevrolet chevelle malibu
1 70 usa buick skylark 320
2 70 usa plymouth satellite

Step 2: Cleaning​

horsepower has 6 missing values. Only 6 rows (1.5%), so dropping them is fine.

cars = cars.dropna().reset_index(drop=True)
print(cars.shape)
print(cars[["mpg", "weight", "horsepower", "acceleration",
"model_year"]].describe().round(1))
OUTPUT​
(392, 9)
mpg weight horsepower acceleration model_year
count 392.0 392.0 392.0 392.0 392.0
mean 23.4 2977.6 104.5 15.5 76.0
std 7.8 849.4 38.5 2.8 3.7
min 9.0 1613.0 46.0 8.0 70.0
25% 17.0 2225.2 75.0 13.8 73.0
50% 22.8 2803.5 93.5 15.5 76.0
75% 29.0 3614.8 126.0 17.0 79.0
max 46.6 5140.0 230.0 24.8 82.0

Step 3: EDA​

fig, axes = plt.subplots(1, 3, figsize=(13, 3.8))
sns.histplot(cars["mpg"], bins=25, kde=True, ax=axes[0], color="#6B5B95")
axes[0].set_title("Distribution of mpg")
sns.scatterplot(data=cars, x="weight", y="mpg", hue="origin", ax=axes[1])
axes[1].set_title("Weight and mpg")
sns.heatmap(cars[["mpg", "cylinders", "displacement", "horsepower", "weight",
"acceleration", "model_year"]].corr(),
annot=True, fmt=".2f", cmap="coolwarm", vmin=-1, vmax=1, ax=axes[2],
annot_kws={"size": 7})
axes[2].set_title("Correlation")
plt.tight_layout()
plt.show()
OUTPUT (chart)​
Step 3: EDA: Wazan aur mpg mpg ki taqseem Correlation mpg 1.0 1.00 -0.780.81-0.78 -0.83 042 origin 0.58 30 40 usa cylinders -0.78 1.00 0.95 0.84 0.90 -0.50 0.35 0.5 japan displacement -0.81...
Step 3: EDA: Wazan aur mpg mpg ki taqseem Correlation mpg 1.0 1.00 -0.780.81-0.78 -0.83 042 origin 0.58 30 40 usa cylinders -0.78 1.00 0.95 0.84 0.90 -0.50 0.35 0.5 japan displacement -0.81...

Observations: The relationship of mpg with weight is strongly negative (heavy car = low mileage). displacement, horsepower, cylinders and weight are also highly correlated with each other (a multicollinearity risk). The relationship is not a perfect straight line; it is slightly curved.

Step 4: Model 1, weight only​

m1 = smf.ols("mpg ~ weight", data=cars).fit()
print(m1.params.round(4))
print("R-squared:", round(m1.rsquared, 3))
print("Change in mpg for every 1000 lb increase in weight:", round(m1.params["weight"] *
1000, 2))
OUTPUT​
Intercept 46.2165
weight -0.0076
dtype: float64
R-squared: 0.693
Change in mpg for every 1000 lb increase in weight: -7.65

Step 5: Model 2, several features​

m2 = smf.ols("mpg ~ weight + horsepower + acceleration + model_year + C(origin)",
data=cars).fit()
print(m2.summary().tables[1])
print("R-squared:", round(m2.rsquared, 3), "| Adj R-squared:", round(m2.rsquared_adj, 3))
OUTPUT​

|===============================
coef
-------------------------------|=======================================================
std err t P&gt;|t| [0.025 0.975]
-------------------------------------------------------| |---|---| |Intercept -16.0654|4.562 -3.522 0.000 -25.035 -7.096| |C(origin)[T.japan] 0.3326|0.569 0.585 0.559 -0.785 1.451| |C(origin)[T.usa] -1.9363|0.521 -3.715 0.000 -2.961 -0.912| |weight -0.0057|0.001 -11.252 0.000 -0.007 -0.005| |horsepower -0.0048|0.013 -0.360 0.719 -0.031 0.021| |acceleration 0.0392|0.098 0.401 0.689 -0.153 0.231| |model_year 0.7554
===============================|0.052 14.617 0.000 0.654 0.857
=======================================================| |R-squared: 0.819 | Adj R-square|d: 0.817|

p-values : features whose p-value is above 0.05 (such as acceleration and horsepower ) their effect, together with the other factors, is not proven , because weight and model_year already explain most of the effect. weight and model_year have a strong effect (the positive coefficient of model_year shows that cars from newer years are more fuel-efficient).

Step 6: Evaluate on Train/Test​

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
features = pd.get_dummies(cars[["weight", "horsepower", "acceleration", "model_year",
"origin"]], drop_first=True)
target = cars["mpg"]
X_tr, X_te, y_tr, y_te = train_test_split(features, target, test_size=0.2, random_state=42)
lr = LinearRegression().fit(X_tr, y_tr)
p = lr.predict(X_te)
print("MAE :", round(mean_absolute_error(y_te, p), 2), "mpg")
print("RMSE:", round(np.sqrt(mean_squared_error(y_te, p)), 2), "mpg")
print("R2 train:", round(lr.score(X_tr, y_tr), 3), "| R2 test:", round(r2_score(y_te, p),
3))
OUTPUT​
MAE : 2.51 mpg
RMSE: 3.34 mpg
R2 train: 0.826 | R2 test: 0.781

Step 7: Residuals and actual vs predicted​

fig, axes = plt.subplots(1, 2, figsize=(11, 4))
axes[0].scatter(y_te, p, color="#6B5B95", alpha=0.7)
lims = [y_te.min(), y_te.max()]
axes[0].plot(lims, lims, color="red", linestyle="--")
axes[0].set_xlabel("Actual mpg")
axes[0].set_ylabel("Predicted mpg")
axes[0].set_title("Actual vs Predicted (near the line = good)")
axes[1].scatter(p, y_te - p, color="#A4457A", alpha=0.7)
axes[1].axhline(0, color="black", linestyle="--")
axes[1].set_xlabel("Predicted mpg")
axes[1].set_ylabel("Residual")
axes[1].set_title("Residuals vs Predicted")
plt.tight_layout()
plt.show()
OUTPUT (chart)​
Step 7: Residuals and actual vs predicted: Actual vs Predicted (lakeer ke qareeb = achha) Residuals vs Predicted 45 10.0 40 7.5 Andaza shuda mpg 35 5.0 2.5 Residual 25 0.0 20 2.5 5.0 15 7.5 10 10.0 10 15 20 25 30 35 40...
Step 7: Residuals and actual vs predicted: Actual vs Predicted (lakeer ke qareeb = achha) Residuals vs Predicted 45 10.0 40 7.5 Andaza shuda mpg 35 5.0 2.5 Residual 25 0.0 20 2.5 5.0 15 7.5 10 10.0 10 15 20 25 30 35 40...

Step 8: How to do better? (one more attempt)​

A slight pattern may be visible in the residual plot (the relationship is slightly curved). A common fix: take the log of the target. We compare both models using the same features and same test set , and measure RMSE on the original scale of mpg (converting the log model's predictions back to mpg with np.exp ).

train, test = train_test_split(cars, test_size=0.2, random_state=42)
feats = "weight + model_year + C(origin)"
lin = smf.ols("mpg ~ " + feats, data=train).fit()
log = smf.ols("np.log(mpg) ~ " + feats, data=train).fit()
def rmse(a, b):
return float(np.sqrt(np.mean((a - b) ** 2)))
print("Linear model RMSE:", round(rmse(test["mpg"], lin.predict(test)), 2), "mpg")
print("Log model RMSE :", round(rmse(test["mpg"], np.exp(log.predict(test))), 2), "mpg")
OUTPUT​
Linear model RMSE: 3.33 mpg
Log model RMSE : 2.75 mpg

Taking the log made the test RMSE lower , i.e. the log transform improved the prediction on this data. This shows that a model is not meant to be built once and left alone , but improved through experimentation.

Step 9: Report (writing up the result)​

Be sure to write these parts in your project's README or report:

  1. Goal: predicting mpg and the factors that affect it.

  2. Data: 398 cars, 6 rows removed because of missing values (392 remain).

  3. EDA findings: a strong negative relationship between weight and mpg; features are correlated with each other.

  4. Models: Model 1 (weight only), Model 2 (several features), Model 3 (log mpg). For each: R<sup>2</sup> , test RMSE.

  5. Most important effects: weight and model_year (with coefficients and p-values).

  6. Limitations: old data (1970 to 1982), the relationship is not perfectly straight, features are correlated with each other.

  7. Next step: comparison with Decision Tree/Random Forest in Module 4.

11.8 Hands-on​

# Dataset: seaborn's 'tips'
tips = sns.load_dataset("tips")
print(tips[["total_bill", "tip", "size"]].corr().round(2))
OUTPUT​
total_bill tip size
total_bill 1.00 0.68 0.60
tip 0.68 1.00 0.49
size 0.60 0.49 1.00
  1. Simple regression: tip ~ total_bill — fit this model. Report the slope and R<sup>2</sup> . Explain what the slope means in your own words.

  2. Prediction: predict the tip for a bill of 40 dollars.

  3. Multiple regression: tip ~ total_bill + size + C(smoker) + C(day) . Which features have a p-value below 0.05?

  4. Residual plot: make residuals vs fitted. Is there any pattern? (Hint: the spread increases at larger bills)

  5. Train/Test: do an 80/20 split and compute RMSE and R<sup>2</sup> .

  6. Question to think about: is a higher bill the cause of a higher tip? What could a confounding variable be?

Challenge: Take the hours/marks dataset of Chapter 9, add an outlier (hours = 9, marks = 10) to it and rerun the regression. How much did the slope and R<sup>2</sup> change? This will give you a feel for the effect of outliers.

KEY TAKEAWAYS

  • Covariance gives direction; correlation (-1 to +1) gives strength too. Pearson is for straight-line relationships, Spearman for monotonic ones.

  • Correlation does not mean causation. The lesson of Anscombe: plot the data before modelling.

  • Simple linear regression: y = b0 + b1 x. Slope = the change in y for a 1-unit increase in x. Least squares minimizes the sum of squared residuals.

  • R-squared: how much of the spread in y the model explained. Avoid extrapolation.

  • The statsmodels report contains coef, std err, p-value, confidence interval, R-squared and the F-test.

  • In multiple regression a coefficient means: the effect of that x holding everything else equal. Check Adjusted R-squared and VIF.

  • Assumptions (LINE): Linearity, Independence, Normality of residuals, Equal variance. Check them with residual plots.

  • Metrics: MAE, MSE, RMSE, R-squared. Always evaluate on the test set (train/test split).