Polynomial regression

Question

What if there is no linear relationship between input features and the output label?

If your data is more complex than a simple straight line, how will you fit a linear regression model to it? Take a look at the data shown below.

Let us fit a linear regression model \(h_\mathbf{w}(x)=\color{red}w_0\color{black}+\color{red}w_1\color{black}x\) for this data.

Clearly, this is not a great fit and we need to search for a better model.

Polynomial regression

We can create polynomial features by combining the existing input features as new features, then train a linear model on this extended set of features.

Let \(\phi_k\) (features) be called polynomial transformation of \(k\)-th order.

Comparison between linear regression and polynomial regression model.

Let's represent polynomial transformation in form of a vector \(\phi\) with \(k\) components.

\[ \mathbf{\phi} = \begin{bmatrix} \color{purple}{\phi_0} \\ \color{purple}{\phi_1} \\ \color{purple}{\vdots} \\ \color{purple}{\phi_k} \end{bmatrix} \]

Each component denotes a specific transform to be applied to the input feature. The polynomial regression model becomes:

\[ \begin{aligned} y &= \sum_{i=0}^{k} w_i ({\phi}(\mathbf{x}))_i \\ \\ y &= \mathbf{w}^T \mathbf{\phi}(\mathbf{x}) \\ \end{aligned} \]

Polynomial Regression is capable of finding relationships between features (which is something a plain Linear Regression model cannot do).

Note:

1. The model is a non-linear function of \((x)\), but is a linear function of weight vector (\(\textbf w=[w_0, w_1, \ldots, w_k]\))

2. Polynomial Features (degree = \(d\)) transforms an array containing \(n\) features into an array containing \(\cfrac{(n+d)!}{d!n!}\) features, where \(n!\) is the factorial of \(n\), equal to \(1 \times 2 \times 3 \times \cdots \times n\).