What the Mathematical Intuition Behind the Linear Regression Algorithm Is All About

It is appropriate to use a linear regression model when attempting to model a relationship between two variables, where one variable is an independent variable and the other is a dependent variable. Linear regression is used to understand the relationship between these two variables and to predict the value of the dependent variable based on the independent variable. In order to determine the best fitting line, a computer will use a mathematical algorithm to measure the differences between the observed data points and the predicted data points. The amount of error is calculated by measuring the differences between the predictions and the actual values. To reduce the amount of error, it is necessary to refine the model by adjusting the coefficients of the equation. This post will explain the linear regression method and answer any queries that readers may have. Shall we proceed?

What is the definition of linear regression?

Statistics utilises a technique known as linear regression, which is suitable for situations where the relationship between variables is linear and can be depicted as a straight line. Linear regression is a powerful tool that can be used to analyse and determine the correlation between two variables. By using this statistical method, insights can be gained into how changes in one variable will impact the other and how a linear relationship between the two exists.

Anticipating the connection between two quantitative variables – predictor variables, also known as independent variables, and dependent variables, which are those that are being predicted – is a typical application of this paradigm. As an example, if one wishes to gauge the value of a home based on its square footage, garage space, lot size, proximity to public transportation, and other features, the dependent variable in this case would be “Price,” while the independent variables would be “Square Footage,” “Garage Space,” “Land Slope,” and “Utilities.

Situations in which regression may be seen

Exhibit A: The First Example
Linear regression is a commonly employed statistical tool by businesses to examine the relationship between advertising expenditure and sales outcomes. They may, for example, construct a linear regression model in which advertising costs serve as the independent variable and sales figures constitute the response variable. The equation for this model is expressed as follows:

Productiveness = Breakeven + Growth (ad. spending)

Exhibit 2
In the field of medicine, linear regression can be employed to examine the relationship between the amount of medication administered and the corresponding blood pressure of the patient. By administering different doses of a designated drug, researchers may study the impact of the drug on a patient’s blood pressure or cardiac strain. To do this, they may use blood pressure as the dependent variable in a model with the dose of medication as the independent variable. To summarise, the formula for this linear regression model is:

Calculating blood pressure: 0+1 (dosage)

Here’s an Illustration Number three:
Linear regression is a commonly employed statistical technique used by agricultural experts to investigate the relationship between weather and crop yields. For example, to assess the influence of rainfall on crop output, researchers might use a multiple linear regression to assess the effects of precipitation and fertiliser application. In this case, crop output would be the dependent variable, and precipitation and fertiliser application would be the independent variables. In mathematical terms, the regression model could be expressed as follows:

plant output = 0 + 1 (rainfall) + 2 (fertiliser)

Numerous monikers for the statistical technique known as linear regression

The earliest documented evidence of linear regression can be traced back to 1805. Subsequently, numerous variations of linear regression have emerged, including but not limited to multiple linear regression, polynomial regression, and other similar approaches. Ultimately, all of these techniques aim to enable the prediction of a target value based on the existing characteristics of the input data.

It is possible to distinguish between the two scenarios based on the terminology associated with the analysis. Linear regression is employed when there is a single predictor variable, whereas multiple linear regression is employed when there are multiple predictor variables.

Linear regression is only applicable when there is a linear relationship between the variables in question. A pair plot or scatter plot is a useful tool for quickly and easily examining this relationship.

The results of linear regression indicate what sort of connection?

  • Constructive Partnership

    There is a correlation between two variables when the regression line between them slopes upwards, indicating that an increase in the value of the independent variable (x) will result in an increase in the value of the dependent variable. This correlation is said to be positive, as opposed to a negative correlation, which would suggest that an increase in the value of x would result in a decrease in the value of the dependent variable.
  • Unfavourable association

    A negative correlation is a relationship between two variables in which an increase in the value of one of the variables is associated with a decrease in the value of the other. This can be represented graphically by a regression line that has a negative slope, meaning that as the value of the independent variable (x) increases, the value of the dependent variable (y) decreases.
  • There is zero correlation.

    It is generally accepted that there is no relationship between the two variables if the best fit line of the regression analysis is a horizontal line. This indicates that any alterations in the value of the independent variable (x) will not influence the value of the dependent variable (y).

Using measures of correlation and covariance, we may learn about the nature of the connection between these variables.

Covariance is a measure of the linear relationship between two variables, X and Y. It provides an indication of whether an increase in the independent variable (X) is associated with a rise or a fall in the dependent variable (Y). A positive covariance value suggests that an increase in X is associated with an increase in Y, while a negative covariance value indicates that a rise in X is associated with a decrease in Y, and a decrease in X is associated with an increase in Y. It is important to bear in mind that covariance does not measure the strength of the relationship between the two variables, only whether the relationship is positive or negative.

The strength and direction of a relationship between two variables can be determined using the concept of correlation. Correlation is a measure that can range from -1 to +1, where a perfect correlation is extremely rare and is evidenced by every data point being exactly on the line of best fit.

The approach using the least squares

The linear regression model aims to identify the most optimal line that can fit the data points. This is done by employing the least squares approach, which minimises the sum of squared residuals. This sum is a measure of the difference between the actual values of the data points and the projected values from the fitted line, which is often referred to as the “error”.

It is common knowledge that a straight line is represented by the equation y = a + bx.

It is necessary to ask what mistake is being referred to here. Can we gain an understanding of what is being described? Is there an available method to identify it? The linear model is capable of performing all the necessary calculations, meaning that there is no need for concern.

Let’s suppose Yi is a free variable. The Sum of Squared Errors (SSE) can be used to measure the divergence between the actual value of Yi and the value that was predicted. The most precise result can be achieved by minimising this unexplained variability.

The sum of squared residuals (SSR) is a measure of the difference between the predicted value (ŷ) and the mean of the dependent variable. Maximising the explained variance of the model can be accomplished by minimising the SSR.
This mistake has the advantage of being able to be differentiated at all points, making it desirable for use as a loss function. However, it is susceptible to extreme values, which can cause issues.

The sum of squared total (SST) represents the overall variance in the model (SSR + SSE = SST).

Explain how linear regression is performed.

In order to gain an understanding of how accurately the bill can predict the amount of the tip, it is necessary to analyse the relationship between the independent variable (the bill) and the dependent variable (the tip). The bill can be considered as the predictor variable, while the tip is the response variable.

The best-fit line is found by minimising the sum of squared errors, which is the difference between the predicted value and the actual value.

The first thing to do is see whether the variables are related in a linear fashion.

It is essential to be aware that a straight line can be represented by the equation y = mx + c, or y = x * 1 + 0. To analyse the correlation between two factors, a scatter plot should be created. It is noteworthy that the centroid, i.e. the intersection of the mean values of x and y, will always be located on the best fit line.

It is evident that there is a positive correlation between leaving a larger gratuity and increasing the total amount of the bill. Linear regression could be used to predict the response variable based on this relationship.

The second step is to verify the data’s consistency.

The strength of a direction can be calculated by determining the correlation coefficient after plotting a scatter graph to illustrate the relationship between two variables. In this instance, the correlation coefficient is 0.866, indicating a very strong correlation between the two variables.

S Step 3: Do the Math.

The computations may begin now that you know the association is quite significant and positive.

Best-fit line equation = x * 1 + 0
By determining the slope or regression coefficient to be 1, we can make accurate predictions regarding the effect of an increase of one unit in the independent variable on the dependent variable. This understanding provides us with valuable insight that can be used to inform decisions and strategy.

, and if the constant term is computed as 0=-x*1,

To do this, plug in x = 74, = 10, (x-x)(y-x) = 615, and solve for x (x-x)2 = 4206.

If the total bill is rounded to the nearest one unit, the resulting figure is 10.1466. Consequently, increasing the total bill by one unit will result in an increase of 0.1462 units in the suggested gratuity.

To the same effect, 0 Equals -0.8203. Any practical significance of the intercept cannot be guaranteed.

Therefore, the best-fit line has the equation y = 0.1462x – 0.8203.

The extensive number of computations need the usage of Python packages to simplify the process.

In the next meeting, we will discuss how to make sense of the data from a linear regression analysis of healthcare costs.

Using linear regression to interpret

The following code will provide a summary of your linear regression model:

X = sm.add constant import statsmodels.api as sm (x)

equivalent = sm.ols (y,x).fit()


Statistical significance coefficient (R-squared)

R-squared (R2) is a measure of how well a statistical model fits the observed data. It quantifies the amount of variance in the data that is explained by the model; in other words, the extent to which the model accurately reflects the observed data. R2 can range from 0 (no fit) to 1 (perfect fit).

The following is an illustration of this:

It has been observed that in this particular scenario, “CGPA” serves as the independent variable and “Package” as the dependent variable. After conducting thorough research on the financial aid offered by XYZ University, you are now ready to apply. As you have not provided any additional information that would be of assistance for them to provide you with an appropriate response, the university will offer you the average package of the previous students. However, if you provide your CGPA when asked the same question, they will be able to estimate your compensation. Through the use of linear regression to analyse past data, they will be able to come to an appropriate conclusion.

As opposed to using the average of the findings of other students, linear regression produces much better results, as shown by the R2 score.

R squared may move between 0 and 1, but can it ever be negative? At what point will this number become 1, and what does it signify exactly?

  • Is R2 allowed to equal 0?

    If the errors on both the regression line and the mean line are equivalent, then it is unnecessary to reveal your cumulative grade point average to the university. This is because the end result would remain the same regardless, since the two lines would be sharing the same mistake when the regression line crosses the mean line.
  • Is an R-squared equal to one possible?

    It is necessary for the ratio of the error of the regression line to the error of the mean line to be equal to zero in order for the regression line to provide a perfect fit to all data points. This would only be possible if the error of the regression line is zero, implying that the regression line is an exact match for all of the data points.
  • Could the R-squared value be -1 or any other negative number?

    In cases when the mean line error is larger than the regression line error, the answer is affirmative. This is only possible if the Sum of Squares of Residuals (SSR) is greater than the Sum of Squares of Total (SST). It is only when the regression line does not intersect any of the data points that the error in the linear estimation exceeds the worst-case scenario.

If the model is able to explain 75.1% of the variance in the data set, or if the fitted values are a good representation of the original values, as demonstrated above, then the coefficient of determination is 0.751. This data set consists of tabular data, so the values in the target column should be continuous and varied. This figure indicates that 75.1% of the variance was explained by the characteristics given to the model. It is prohibited to discuss the remaining 25%.

Keep in mind that a high R-squared value indicates a robust model. Sure, but how high is too high?
If R2 is more than 0.8, the model fits the data well.
An acceptable fit is indicated by an R-squared value between 60% and 80%.
Having a R squared score of less than 60% indicates the model might need some tweaking.

If your R2 is low, you may want to double-check your independent variables for anomalies.

Quantitative Analysis Using the Modified R-Squared

It is important to note that the R-squared statistic increases as more input variables are included in the model, which can lead to inaccurate results if used to assess the inclusion of a new independent variable. To address this, we can use a modified version known as the adjusted R-squared. This modified version accounts for the nuances of the situation, by increasing in importance as extraneous factors are included in the model, while also decreasing when factors with no effect on the dependent variable are included. It is important to note that the adjusted R-squared is always less than the R-squared.

In most scenarios, the adjusted R-squared value will be very similar to the initial R-squared figure. Nevertheless, if there is a noticeable discrepancy between the two, it is recommended to re-evaluate the independent variables to determine if there is any association between the independent and dependent variables.

F-statistic: In this instance, the F-statistic value suggests that some of the model’s coefficients are not consistently equal to zero. If the overall F-test returns a significant value, this indicates that the correlation between the model and the dependent variable is statistically reliable due to the fact that the R-squared value is not equal to zero.

The null hypothesis in this case is that the model without any independent variables is equivalent in terms of its ability to fit the data when compared to the model that has independent variables.

In contrast to the intercept-only model, the model provides a superior fit to the data, as proposed by the null hypothesis (H1).

A p-value can be used to evaluate whether the evidence is strong enough to support the alternative hypothesis. In order to be 95% confident in the results of a test, the following conditions must be met:

The null hypothesis (H0) may be rejected since the p-value is less than the threshold of significance (5%).
If the p-value is higher than the significance threshold, then hypothesis (H0) cannot be rejected.


Let’s attempt to gain an understanding of this concept by interpreting it. It is important to note that it is essential to have a basic comprehension of statistics if one wishes to comprehend Machine Learning.

Coefficient: It is important to take into account that linear regression relies on the assumption that the independent variables in the model are not collinear, meaning that they are not highly correlated with one another. Furthermore, the coefficient of a variable reflects the increase of the dependent variable when the independent variable is increased by one unit, given that all other variables in the model remain constant.

To provide an example, if the age coefficient is 257.305, then for every additional year of age, there would be a corresponding increase of 257.305 points in the dependent variable. Conversely, if the coefficient was negative, such as with the “region” coefficient, a one-unit increase in the region’s value would lead to a decrease in the dependent variable by 353.4491 points.

A variable’s relevance to a model can be highlighted using the magnitude of its coefficient. If the coefficient value is close to zero, this indicates that there is no relationship between the two variables.

Dispersion, Relative to the norm: Gaining an understanding of standard deviation can assist in understanding standard error. If you are looking to measure the spread of your data or the distance of the numbers from the mean, then standard deviation can be employed. The majority of the data (approximately 95%) is concentrated within two standard deviations from the average.

To begin, we will take a sample of 10 from a larger population to illustrate the concept of standard error. Standard error is defined as the amount by which sample means deviate from the mean of the entire population when plotted on a normal distribution graph, providing an indication of how accurate the sample mean is in representation of the true population mean. This will allow us to gain a better understanding of the difference between the sample mean and the true population mean.

As such, the standard error in the regression model provides an approximation of the spread of coefficients.

Confusing? Let’s break it down, shall we?

For every one-year increase in age, the dependent variable is expected to increase by 257.4050. Re-running the model could potentially lead to a shift in this coefficient, with a variation of up to 11.878 standard errors, as suggested by the standard error.

t-stat – The t-statistic, also called a t-value, is arrived at by dividing the coefficient by the standard error.

The formula for the t-statistic is: Coefficient / Standard Error.
If the coefficient value is unusually high or low, the t-statistic can be used to help explain why this may be the case. If the t-statistic value is higher than what is expected, then it can be concluded that the hypothesis being tested is supported.

Because this number is inside the rejection interval, you will choose to reject the null hypothesis.
H0 states that all population-level coefficients are zero.
Coefficients are not zero at the population level (H1), the alternative hypothesis.
A larger t-statistic value indicates that the variable is more significant.

P > |t| – The p-value is a measure of the likelihood that the alternative hypothesis is incorrect and can be expressed as a percentage. In the case of age, the p-value is extremely low, suggesting that the null hypothesis is highly unlikely to be correct. Furthermore, the p-value indicates that the coefficients of the independent and dependent variables are non-zero at the population level, implying that there is a correlation between the two variables.

When analysing the correlation between two variables, a p-value of 5% or less is often accepted as an indicator of statistical significance. Therefore, if the calculated p-value is greater than 0.05, it is incorrect to conclude that there is no correlation between the two variables. In this case, the null hypothesis should be rejected, and it can be declared that the coefficients are not equal to 0 if the p-value is less than 0.05 (p-value < 0.05).

If the t-statistic is large, then the p-value is little, therefore it is more likely that the observed coefficient values are not random.

You have been highly praised for your independent research and subsequent expertise in understanding linear regression models. Acquiring a deep comprehension of the inner workings of such models is always a beneficial skill. In order to take your understanding of this model to the next level and generate precise predictions, it is essential that you first comprehend the definitions of the associated terms.

Join the Top 1% of Remote Developers and Designers

Works connects the top 1% of remote developers and designers with the leading brands and startups around the world. We focus on sophisticated, challenging tier-one projects which require highly skilled talent and problem solvers.
seasoned project manager reviewing remote software engineer's progress on software development project, hired from Works blog.join_marketplace.your_wayexperienced remote UI / UX designer working remotely at home while working on UI / UX & product design projects on Works blog.join_marketplace.freelance_jobs