Multivariable Linear Regression, Lecture notes of Machine Learning

Download Multivariable Linear Regression and more Lecture notes Machine Learning in PDF only on Docsity!

Introduction to Machine Learning

Rohit Kate

Linear Regression – Part 1

Reading

• Chapter 7, up to Section 7.4.

Example

• Can we learn a relation between the size and its

rental price?

• What will be the use of that?

– To understand how the size affects the rental price

– To predict rental price for other sizes in future

• Note: This is a regression task, i.e. predicting a

numeric target

Example

Looks like a line (linear model) relates the two, but how do we determine it?

Simple Linear Regression

• Every line has an m and b; different values of

m and b represent different lines

y x b ∆y ∆x m=∆y/∆x

Simple Linear Regression

• Every line has an m and b; different values of

m and b represent different lines

y x b ∆y ∆x m=∆y/∆x Negative intercept

Simple Linear Regression

• If size was x and rental price was y then the

linear relation will take the form: rental

price = m*size + b

– Why is this linear? Because there is no squared or

higher degree term of size

• How do we determine the best values for m and

b that fits the training data?

• How do we quantify the “fit”?

• When does a line not “fit” the training data?

Simple Linear Regression

Which line fits best and how do we know?

Sum of Squared Errors

Errors Sum of squared errors measures how well the line fits the training data points.

Best Linear Model

• The best line (i.e. linear model) will be the one with

the least sum of squared errors (L

2

• How do we determine the line, i.e. parameter

values (m,b), which has the least L

2

for the training

data D?

• This task is called least square optimization

• It is computationally infeasible to try every possible

value of (m,b) and measure L

2

to see which gives

the least value

Best Linear Model

• Solve and

– Partial derivatives are zero at the minimum value

• This has an analytical solution:

where and are averages of the y (target) and x

(feature) in the training data respectively

Multivariable Linear Regression

• But so far our example had only one feature! That is when

it is called simple linear regression.

• Having multiple features is common in machine learning

tasks

• (^) Note: Feature is a term commonly used in machine learning which goes by other names in other areas (e.g. statistics, medicine): variable, covariate, independent variable, predictor
• (^) Similarly, target is called by other names: dependent variable, outcome

• When there are multiple features (i.e. variables) then

building a linear model to predict a numeric target is called

multivariable linear regression or multiple linear

regression

Sum of Squared Errors

• Sum of squared error, L

, is then analogously

computed for the training data D of n training

examples as:

– t

i

is the correct target for i th training example

– ½ is added for mathematical convenience

Actual Predicted

Best Linear Model

• The best linear model will be again the one with the least

sum of squared errors (L

2

• How do we determine the weight vector w that gives the

least L

2

• Again calculus can be used

• Solve for i=0 to m

• This also has an analytic solution but computing it requires

expensive matrix operations

• In practice, a numerical method, such as gradient descent,

is used to approximately find the best weight vector