Linear and Polynomial Regression in Python

[@GeriReshef]

Almost 4 years have passed since you made it, but it still contributes to people like me. I find it very clear and helpful. Thanks.

[@apm]

Thanks Geri, I'm glad it helped.

[@nackyding]

If you follow convention (import numpy as np) it makes it a lot easier for newbies like myself to follow along. Thank you for the awesome tutorial. Thank you.

[@apm]

I agree on the import call. It is much better to use: import numpy as np

[@nackyding]

I wasn't trying to be rude or sarcastic either. I actually enjoyed and learned from the tutorial and I'm a subscriber to your channel. Just saying it's a lot easier to follow along and make sense of things when convention is followed. You can delete my comment if you want. It's fine.

[@apm]

Magneto - it is a really good comment and very insightful. Importing numpy in the way that I showed in the video clutters the namespace, especially for more complex scripts. I really appreciate your input.

[@akashgillella]

Awesome video, Stuck me with its simplicity, Thanks a lot

[@maicolmoretti9504]

you are great!! In 42 seconds you explained everything I wanted to know!

[@apm]

I'm glad that you got the answers you needed. Check out additional course content at apmonitor.com/che263

[@daltondaytongkraatz5687]

Your video really helped me understand polyfit and why my polyfit function on my homework was not working! Please keep up the amazing video tutorials!

[@apm]

+Dalton Day Tong Kraatz Thanks for the positive feedback!

[@nickt423]

For those getting 𝙽𝚊𝚖𝚎𝙴𝚛𝚛𝚘𝚛: 𝚗𝚊𝚖𝚎 '𝚙𝚘𝚕𝚢𝚏𝚒𝚝' 𝚒𝚜 𝚗𝚘𝚝 𝚍𝚎𝚏𝚒𝚗𝚎𝚍 Its because you probably used 𝚒𝚖𝚙𝚘𝚛𝚝 𝚗𝚞𝚖𝚙𝚢 𝚊𝚜 𝚗𝚙 . Anytime you use 𝚙𝚘𝚕𝚢𝚏𝚒𝚝 or 𝚙𝚘𝚕𝚢𝚟𝚊𝚕, just add an np. before it. 𝚙𝚘𝚕𝚢𝚏𝚒𝚝 becomes 𝚗𝚙.𝚙𝚘𝚕𝚢𝚏𝚒𝚝 𝚙𝚘𝚕𝚢𝚟𝚊𝚕 becomes 𝚗𝚙.𝚙𝚘𝚕𝚢𝚟𝚊𝚕

[@fokjohnpainkiller]

You're the best

[@tristanyoung4997]

Thank you sir!

[@arijruwaii]

This helped me so much in my homework, i was looking for someone good at explaining the function. Thank you, keep up the good videos

[@Aziqfajar]

Thank you for the tutorial. For those having problem with y fit, perhaps you can use polyval() function from numpy on fitted polynomial and the x values.

[@apm]

That is the easy way!

[@sangam_319]

Is there any way to plot without using polyfit. i want to do as per prml book but couldn't write the code to find the value of the coefficient w and show the same exact thing without using polyfit or poly1D

[@apm]

Yes, you can plot any data with matplotlib. See the plotting module in this course: apmonitor.github.io/begin_matlab

[@yaboiiprettyeyez]

Hey was wondering if you could please show how one would go about getting all critical points from a given data set. specifically asking about time series data. thanks!

[@apm]

I don't know about critical points from data but you could find them from a function: www.khanacademy.org/math/ap-calculus-ab/ab-diff-analytical-applications-new/ab-5-2/v/minima-maxima-and-critical-points You would set the derivative of the function equal to zero and find all of the roots of that function (where it equals zero).

[@yaboiiprettyeyez]

@@apm ok so I found how to get all the critical points i.e. maxima and minima given a time series using scipy.signal.argrelextrema. Here is the example code I found to get the points. Now I would like to get the distance between the lowest price and highest price and plot the trend line. Here is some example code if your willing to help me expand on this thought... import pandas as pd import numpy as np import matplotlib.pyplot as plt from datetime import datetime import pandas_datareader as data from scipy.signal import argrelextrema START = datetime(2004,1,1) END = datetime(2020,12,16) df = data.DataReader('AAPL', 'yahoo', START, END) df.index = pd.to_datetime(df.index) dfc = df.copy() highp = dfc['High'].to_numpy() lowp = dfc['Low'].to_numpy() max_idx = argrelextrema(highp, np.greater) min_idx = argrelextrema(lowp, np.less) maximaPrices = highp[max_idx] minimaPPrices = lowp[min_idx] As I said, I would like to get the distance from each low to each high and plot them. Sorry if my last question was ambiguous. I do appreciate your content very much though! Thanks.

[@ajbaker1655]

Is it common practice to use an asterisk when importing the libraries? I find it confusing, I like to know which library is responsible for which function.

[@apm]

No, that is just me being lazy and it is generally bad practice because of name conflicts with other libraries for more complex code. I recommend "import numpy as np".

[@rvilleg95]

I get a problem there where you write "slope,intercept,r_value,std_err=linregress(x,y)" I get an error saying : "Traceback (most recent call last): File "", line 1, in slope,intercept,r_value,std_err=linregress(x,y) ValueError: too many values to unpack" why's that

[@apm]

At 13:52 make sure you add in the p_value as well. It looks like you are missing it. See docs.scipy.org/doc/scipy/reference/generated/scipy.stats.linregress.html for additional help on this function.

[@Clbnjoroge]

Loving your video, i do have a small complaint about your imports tho, i think its generally a good idea to only import the libs that you require instead of using * on imports, that way you know from which imports the functions are imported from when reading thro the code.

[@apm]

+Caleb Njoroge, I agree - It is a shortcut that can cause problems with more complex code.

[@eliotharreau7627]

Hello, somebody know how to put back the values of the linear regression in a dataframe ? Thnx any way.

[@apm]

See 7.3 on DataFrames: apmonitor.com/dde/index.php/Main/PythonPandas

[@mionome501]

How do I select a range of point on x and y on where to perform the fit ? (Do I have to create a shorter array x' and y' or there is a faster way ?) Thank you

[@apm]

Yes, you'll just need to slice the input arrays such as x[2:15], y[2:15].

[@2010aurnob]

Thanks for another great video Dr. Hedengren. Could we use this for multivariate non-linear regression ? If I have 3 columns of domain, could I define "x = dataset.iloc[:,:-1]", eliminating the dependent variable at the last column and use the polyfit function?

[@apm]

polyfit only supports 1 variable. However, you could use a matrix solution as shown here: stackoverflow.com/questions/34746724/fit-data-to-curve-using-polyfit-with-multiple-variables-in-python-using-numpy-po or use a tool such as gekko that includes general nonlinear (or linear) and multi-variate regression. Here is a relevant question: stackoverflow.com/questions/57726954/what-is-the-correct-way-handle-with-multidimensional-array-in-gekko-nonlinear-re Feel free to ask another question on StackOverflow with tag [gekko] if you need additional help.

[@bernsbuenaobra473]

Great tutorial - watch the master and learn. It would be nice if the notebook was shared too.

[@apm]

The source code is here: apmonitor.com/che263/index.php/Main/PythonDataRegression

[@hawraaabdulrazzaq8371]

Please how can i contact you

[@apm]

Unfortunately, I can't help with all of the individual requests for support I receive each week. I recommend posting your question to StackOverflow or else see additional resources here: github.com/APMonitor/data_science/blob/master/09.%20Interpolation.ipynb

[@Gustavo_0107]

Thank you! One question, how do I determine de uncertainty of the coefficients generated by polyfit?

[@apm]

Try using statsmodels OLS regression: apmonitor.com/me575/index.php/Main/LinearMultivariateRegression The polynomial regression can be posed as a linear regression. Here is additional information on uncertainty analysis: apmonitor.com/che263/index.php/Main/PythonRegressionStatistics

[@ericsonnen5248]

Thanks! I usually use R, but have a project started in Python.

[@apm]

As you are getting started, you may also be interested in the GEKKO Python package (apmonitor.com/wiki/index.php/Main/GekkoPythonOptimization) for regression. See #5 and #6 for examples of linear and nonlinear regression.

[@akshaysunil2852]

NameError: name 'polyfit' is not defined , sir how to solve this error?

[@apm]

You need to import numpy.polyfit with: "import numpy as np" and then use "np.polyfit()". If you haven't installed Numpy then there is help with installing packages here: apmonitor.com/pdc/index.php/Main/InstallPython

[@mountp1391]

Thank you! It helped a lot to apply some code to jupyter, but could you forward another code on how to get 5th-degree polynomial regression by python code?

[@apm]

Yes, just use polyfit with a 5th order. More details at apmonitor.github.io/data_science

[@wanderingwanderer1016]

why does xp = linspace(-2, 6, 100) have the effect of smoothing out the polynomial curve?

[@apm]

You are adding 100 linearly spaced points between -2 and 6. If you use xp = linspace(-2,6) then the default is only 50 points. If you are only plotting the regression points then it looks much less smooth.

[@bernsbuenaobra473]

It interpolates between points thereby adding more steps making a finer connection less jagged

[@ozwaltreacts4709]

@@apm The linspace is changing my x and my y value scale. is there a way to keep it the same?

[@amitshiuly3125]

For multivariate polynomial function of degree 8 I have obtain coefficient of polynomial as an array of size 126 (python). I can not understand the arrangement of polynomial function ie whose coefficients are this (vector of variables). Can any one help me regarding the multivariate polynomial regression?

[@apm]

Here is some help from the documentation: p=pndarray, shape (deg + 1,) or (deg + 1, K) Polynomial coefficients, highest power first. If y was 2-D, the coefficients for k-th data set are in p[:,k]. numpy.org/doc/stable/reference/generated/numpy.polyfit.html

[@amitshiuly3125]

@@apm I have go through this. However, I can not find my answer. My question is that how can I get equation of polynomial after polynomial regression. I am getting array of coefficients but I can not find the equation due to unknown polynomial orientation.

[@mirzasalkic5252]

Yo cannot do polynomial regression via importing packages to get Rsqr?

[@apm]

Here are a few options: stackoverflow.com/questions/893657/how-do-i-calculate-r-squared-using-python-and-numpy

[@anefuoche1053]

for some reasons i got different values from yours for all my print results. please why is that?. great video, i finally learnt the maths behind r squared

[@apm]

Try running the script at this URL: apmonitor.com/che263/index.php/Main/PythonDataRegression

[@peakxv13]

%matplotlib.pyplot inline not req in python3

[@apm]

Thanks!

[@JohnLaudun]

A little before the 8 minute mark you note that "polyfit" is a scipy function, but it's a Numpy function.

[@apm]

Thanks for the correction. I think Scipy inherits that function from Numpy so it may be available as numpy.polyfit and also as scipy.polyfit.

[@chouhanhimanshu]

Very helpful. Thanks for sharing !

[@tulsikumargupta7981]

>>> from sklearn import linear_model >>> reg = linear_model.LinearRegression() >>> reg.fit ([[0, 0], [1, 1], [2, 2]], [0, 1, 2]) what means this [0,1,2]

[@apm]

+tulsi kumar, those are sample weights as an optional argument: scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression.fit

[@darchcruise]

Is it possible to do a second regression video using multi-variarables i.e., x1(feature 1), x2 (feature 2), x3 (feature 3), x4 (feature 4), and y (output)? Thanks!

[@apm]

+darchz I'll add it to my list of next videos. Until then, check out the following links: stackoverflow.com/questions/11479064/multivariate-linear-regression-in-python and stackoverflow.com/questions/17690977/apply-non-linear-regression-for-multi-dimension-data-samples-in-python . I often prefer to use nonlinear programming solvers such as APM Python to have any model form, equations (up to 10,000,000), and parameters (up to ~10,000).

[@darchcruise]

+APMonitor.com Thanks : )

[@codicia.]

Amazing videos!!

[@junganggao5332]

"%" can not be recognized by python in windows. What should I do for this sign?

[@rrc]

Do you have numbers in percent units or do you need to include a comment? In Python you can divide by 100 to get the number into fractional form. You can include comments with # sign (not % like Matlab). I suppose I don't understand your question so feel free to ask again.

[@seandenny6430]

What if I want to solve a linear system where data is presented in the form of a matrix A(1000*100) and vector Y(1000*1) and linear system is A = B(beta).Y how do solve that with linear regression using python? I want to illustrate B(beta) using plot(). Thanks in advance

[@apm]

Here is a way to solve for y => A (1000x100) = B(1000x100) * y(100x100): A = B * Y => B^T * A = (B^T * B)*y => Y = (B^T * B)^-1 * B^T * A. This is common approach in linear regression.

[@seandenny6430]

But does this data have a linear relationship? Also, I don't know B(beta); how did you get (1000*100) I only have two sets of known data A and Y, the matrices, and getting them using randn() When I plot it using plot(); the output is crazy overlapping ball of dots.. I want to solve this using Python code. I would really appreciate your help here, thanks.

[@apm]

If the data is nonlinear and you'd like to use a nonlinear regression then there is some help here: apmonitor.com/che263/index.php/Main/PythonDataRegression If the system is a linear set of equations then B must be (1000x100) for consistency of the dimensions. The inner dimensions of any dot product of matrices must be equal (both to 100 in this case). The result is the outer dimensions of the multiplication (1000x100). Your original question has a discrepancy because of A(1000x100) = B * Y(1000x1). Did you intend the following: A(1000x100) = Y(1000x1) * B(1x100)?

[@seandenny6430]

You're right it is: A(1000x100) * B(?) = Y(1000x1) So I guess B should be (1000x100) then? What should be the result to solving this linear system and what would plotting B look like?

[@apm]

+anuj kumar in the case that you just mentioned, B should be (100x1). It is important which side of the equal sign it is on. I'm not sure what you are trying to show but in linear regression, you could show a bar chart of the magnitude of your regressed parameters.

[@marksegal9557]

Great Tutorial, is there a way to get the reduced chi-squared and degrees of freedom values for the polyfit function automatically? Is there a way to calculate the error for polyfit without graphing?

[@apm]

+Mark Segal Does this help? stackoverflow.com/questions/5477359/chi-square-numpy-polyfit-numpy

[@marksegal9557]

Yes thank you. I am new to python so this is very helpful. Once I have the chi_squared value and the degrees of freedom (d) I believe I can get the p-value by using, stats.chi2.sf(chi_squared,d), and reject the fit if the p-value is less than 0.05. Is that correct?

[@apm]

+Mark Segal That sounds right but I wouldn't know without digging into the documentation on the stats package. Please post again if you find additional information.

[@chobitsx9]

How can you use linregress with polynomials biggers than grade 1?

[@apm]

Is this what you need? stackoverflow.com/questions/33964913/equivalent-of-polyfit-for-a-2d-polynomial-in-python

[@rupalisurase13]

I had added scipy package but error arises like "ImportError: No module named scipy.interpolate"

[@apm]

It sounds like scipy wasn't installed correctly. Do you have multiple distributions of Python on your computer? You can follow these instructions to make sure SciPy is installed correctly: apmonitor.com/che263/index.php/Main/PythonIntroduction The first thing to do may be to uninstall all Python packages as a first step.

[@Lichlord]

I'm using Python 3.5 and Polyfit was actually part of NumPy. I was able to do this whole exercise without importing SciPy.Interpolate.

[@apm]

Bob, that is an error in my video to require scipy.interpolate. Importing scipy also imports all numpy routines as well so scipy.polyfit and numpy.polyfit are equivalent. To verify, I ran the following: import numpy import scipy print(numpy.polyfit==scipy.polyfit) It produces "True" meaning that the same module is imported.

[@Lichlord]

Huh, that's neat. I would not have guessed that was the case. Thank you for public the tutorials. I have generally found them very helpful.

[@rupalisurase13]

okk sir...thank you so much for your reply. Now it is more clear.

[@neutrinoBH]

Very useful video. Thanks.

[@alirezaabdolmanafi4311]

how can I write these codes on python with arc gis ? I need a solution how to do regression on arc gis and my supervisor said I can do this with the help of Model Builder !! do you have any advice for me about how can I do it ?

[@apm]

+Alireza Abdolmanafi maybe this link can help: desktop.arcgis.com/en/arcmap/10.3/analyze/modelbuilder/exporting-a-model-to-a-python-script.htm

[@alirezaabdolmanafi4311]

Thank you it really helped me. I've another question: with this example and codes can I do all sorts of regression on more complex equations ? like second or third degree equations ?

[@alirezaabdolmanafi4311]

And I would like to know that if we can write a code for nueral networks in python script and then import it in model builder on arc gis ? can we do that ?

[@apm]

From the link I sent earlier, it looks like you can export from arc gis but I didn't see any information about importing.

[@apm]

Yes, you can do higher order polynomial regressions in Python. Just use the option that tells what order of model you'd like to use. It will return a vector with the coefficients of that model.

[@shuxrat2600]

Thanks for the great video, I cannot figure out why plot(x, polyval(p2,x), 'b--') and plot(x, polyval(p3,x), 'm:') plotting several lines instead of one in my dataset. How do I calculate if I have nan values in my dataset? Thanks

[@apm]

Did you check that your x value is a vector and not a higher dimensional matrix?

[@shuxrat2600]

I have x, y numpy ndarray consisting of 26 values each.

[@Sukhmani_randhawa_artistry]

Hello sir First of all its a great video, thanks for that! Can you please let me know how to fit a regression line in 3D plots? because I'm dealing with huge data (in thousands), and 16 training sets compared to 1 output. So i needed to plot a 3D data. So how to fit a polynomial regression and compute errors?

[@apm]

Maybe try a bspline: apmonitor.com/wiki/index.php/Main/ObjectBspline

[@jaeen7665]

Pay $9,000 a semester to watch a real professor on KZbin. And people say taxation is theft.

[@apm]

Online learning is certainly an appealing alternative to a university education. The two main obstacles IMHO are certification of training and motivation of the students when it is self-paced without instructor or peer interaction. I like the concept of hybrid learning where some is online and some is active learning in a classroom.

[@anilpawar8908]

Thankyou sir for this helpful video. But sir how can we do it for multivariate data i.e x1, x2, x3, etc in place of x?

[@apm]

anil pawar, the APMonitor Optimization Suite can do multiple inputs or outputs simultaneously. See tutorials at apmonitor.com/do or the second example here: apmonitor.com/che263/index.php/Main/PythonDataRegression

[@darchcruise]

This is a great video! Any plans for teaching Calculus with SymPy?

[@apm]

+darchz Yes, we'll be starting the Python section of the Computational Tools course in a few weeks. Check out the material at apmonitor.com/che263

[@yamunalingam8380]

How do you calculate future values?

[@apm]

You can use the polyval function to evaluate the polynomial or linear correlation at different points.

[@laupii1]

thanks it worked, soomewhat i had to figure out how to call the funtion because it was saying that it was not defined but i added np. at the begginign and it worked.

[@LordTrayus]

Is there a way to conduct a multivariate linear regression using python?

[@apm]

+LordTrayus The best way to do multivariate linear regression is to use matrices. Suppose you have a model to regression (matrix B) in Y = X * B where Y and X are matrices of data. The solution to the multivariate linear regression is then B = (X^-1 * X) * X^-1 * Y. You would need to create matrices in Numpy (kzbin.info/www/bejne/o4C9YYh5mtqFjpY) or use another package that can compute this for you (see stackoverflow.com/questions/11479064/multivariate-linear-regression-in-python).

[@LordTrayus]

Actually, I lied...I'm a goofball. My target variable is binary...is it the same methodology as above for multivariate logistic regression? Or do I just change the binary variable to 0 and 1 and then just have it run as a continuous target?

[@apm]

+LordTrayus There are a couple options if you are doing classification work. You can use empirical methods such as k-nearest neighbors (en.wikipedia.org/wiki/K-nearest_neighbors_algorithm), artificial neural networks, fuzzy logic, etc. If you want to make one of your outputs 0 or 1 then you may also consider formulating it as a Mixed Integer Linear Program.

[@LordTrayus]

I'm aware of how these methods work when conducting it on SAS Enterprise Miner...I'm hoping you can point me to how I can do the same analysis (write the code) on Python.

[@apm]

You may want to look at Pyomo and a Mixed Integer Linear Programming solution. APMonitor can also solve MINLP problems: apmonitor.com/wiki/index.php/Main/PythonApp

[@rod2467]

Telling me 'polyfit' is not definied. What should I do?

[@apm]

+rod2467 make sure you import the polyfit function from scipy before using it as shown at 3:45. You can also download source code to test at apmonitor.com/che263/index.php/Main/PythonDataRegression. If you don't have the Scipy module, please see download and installation instructions here apmonitor.com/che263/index.php/Main/PythonIntroduction

[@cashphattichaddi]

The polyfit() method actually belongs to numpy NOT scipy. Just write :- from numpy import * .Now you can use the polyfit function

[@apm]

Thanks for catching that! This is a good example of why I should import numpy as 'import numpy as np' and then use 'np.polyfit()' to avoid module namespace conflicts and to make the code more readable.Thanks for catching this.

[@cashphattichaddi]

And Thank you for the tutorial!

[@rod2467]

thank you soo much! I'm gonna restart this tutorial again from the beginning. APMonitor.com cashphattichaddi

[@mionome501]

many thanks man for sharing this!! really good job!! ( btw I used spyder)

[@apm]

Thanks for the feedback on Spyder.

[@mionome501]

How do I select a renge of point on x and y on where to perform the fit ? (Do I have to create a shorter array x' and y' or there is a faster way ?) Thank you@@apm

[@PankajMishra-rt6hr]

I love u...no seriously I do....awesome video man...keep posting machine learning vids and can u please uploads video from kaggle ?

[@ririworks]

зашёл смотреть порараз и нашёл питон))) совпадение…

[@apm]

"I went to watch it and found a python))) a coincidence ... " for those who need a translation.

[@izumimizuno2390]

Thank you. It helps a lot!!

[@lucijeanej]

It would be great if the letters are bigger.

[@apm]

The video should also be available in 1080p. You may be able to see it better if you let it buffer.

[@zaraf]

are there really no adds in this video??? DAMN

[@apm]

If the video is watched from the course web-site (apmonitor.com/che263) there are no video adds.

[@patrickgoncalvesmclaren1598]

This was great.

[@AHMADKELIX]

permission to learn sir

[@apm]

Permission granted.

[@abhibrotomukherjee8239]

Very helpful. Thank you for sharing.

[@apm]

Thanks!

[@vijaykumarkari]

super......

[@federicobianchi7721]

I love you!!

[@jong7513]

You are f****** awesome

[@fernandosanchezvillanueva4762]

You're also

[@ahah8797]

crappy image quality. Screen capture, not video cam

[@apm]

You can set the video quality to a higher setting in KZbin. It will take longer to buffer if you have a slow connection. It is 1080p at the maximum setting.

[@marcoponts8942]

How can I force my linesr regression to go through a specific point? origin for example

[@apm]

Don't include an intercept in the equation or fix the intercept to zero with the optimizer.