Denoising Data with FFT [Python]

[@somethingironic2227]

the way this video is setup, overlaying him, the code and a whiteboard, is really slick

[@adityams1659]

Dude this guy is crazy, I still cant believe these videos are free.Thank You for making it free means a lot.❤️

[@josephtran1500]

I havent even watched the video yet and I can tell Im on for a ride. The content looks so good.

[@payman_azari]

I can't even believe these videos are made

[@jerickandrerodriguezmotta6809]

I've spent many hours trying to apply FFT to my data and I've finally done it with your amazing explanation. For sure, the best Fourier Transform video.

[@pipertripp]

Pure gold. Fourier analysis still feels a bit like voodoo to me as I'm just learning the basics, and your videos have been very helpful. The python examples are really handy. Thanks for taking the time to do these things in both MatLab and Python.

[@andrerangel7654]

Steve Brunton, I may never meet you in person but you helped me a lot with these videos. I wish you a good health and a prosper life.

[@felicemorgigi1764]

People like you make the world a better place. Free education helps everybody in the end. Thank you.

[@m.espejo]

This has been very useful for me. I am a Mechanical Engineer and I am working in dynamic studies of steel structures. This method is very practical to apply to the acquisition of accelerometer data in dynamic tests. Thank you so much Steve!!!

[@bArda26]

I was watching you when I was in college for controls, life has brought us back together lol

[@TNTsundar]

Brilliant. Just brilliant. The quality of this lecture is off the charts.

[@Via.Dolorosa]

Hi Steve, I watched many of your videos in Control the comments there are disable so I took this opportunity just to say thank you

[@ahmedgaafar5369]

me too

[@paulsenn1281]

this is mind-boggling-ly useful! (written after hours of scrolling in vain attempting to understand both the transform itself and the code that implements it .) Thank you!

[@abemartin2002]

Professor Brunton, I love all your videos.

[@monh964]

I told my self lets see what this guy is talking about... Realized I discovered a wizard. I will have to see all your videos. I always was afraid from fft. Thank you for simplifying it. Thanks again.

[@hfkssadfrew]

Very nice video professor Brunton. I think the idea behind is FOurier transform of white noise is a constant depending its amplitude. So for large signal-to-noise ratio , you will see a plateu + the real signal, then picking those peaks will work.

[@goodlack9093]

one of the best channels I could've ever stumbled upon...immediately subscribed😅

[@doneel.5338]

This is amazing! I am finally getting rid of the feeling that the topic is complicated thanks for these videos! I mean it is, but I also am capable of getting it with your explanation.

[@TungNguyen-gx1fy]

Take note: f is the noisy data 1. f_hat = FFT(f) - f_hat is a vector of complex Fourier coefficients (increasing frequency: low to high) with its magnitude. 2. PSD = |f_hat| - vector of real magnitude (power) for each frequency. 3. Filter: on PSD, keep the power > Threshold, and set 0 to the rest (because they are the noise frequency) and we have new f_hat. 4. f_denoised = iFFT(new f_hat)

[@babaksanaee1460]

This is so excellent. Absolutely the best, and most comprehensive video regarding real world FFT on youtube. Probably going to buy your book now. Thanks a lot!

[@Eigensteve]

Wow, thanks!

[@jinbobian1613]

Thanks very much! This really helps students who are struggling with denoising.

[@supervince110]

These video series are worth more than the courses you spend thousands of dollars in a university

[@HarrierBr]

Very good explanation. GG for the backwards writing, REALLY nice.

[@Eigensteve]

Many thanks!

[@afghanexperts7815]

One of the best explanation I have ever seen ❤️❤️

[@afsinyilmaz8665]

Effective! Your amplitude at per unit time teaching is above threshold.

[@L2.Lagrange]

Very helpful content. I have used MATLAB but not python. I want to use FFT and some basic signal processing techniques to learn how to use python, so these videos are very helpful

[@Michallote]

For those wondering how to filter non stationary series, you can use overlapping windows to perform the same process per window then blend the signals in the overlapping regions

[@shillowcollins6392]

Wow, I am super impressed, why didn't I find this channel early. Thanks so much for sharing this valuable information.

[@Akshay-cy9tu]

just in love with the way u teach concepts

[@yashodhanvivek8086]

Amazing Prof. Brunton. These videos are extremely helpful for DPS researchers

[@sennabullet]

A fantastic video with great informational content...and mindblowing production!!! How do you do all the overlays???

[@surajyergude]

Your lecture gives insight to the content what is in there books.

[@umedina98]

Your videos are crystal clear! I cant thank enough for sharing this high quality content. Loved the approach you took of writing parallel to the code!

[@larcomj]

Great application of the convolution theorem. Well explained.

[@saitaro]

super-duper, as always. Python examples are what we need!

[@JosephRajewskiWIII]

I had Steven Brunton as a professor at University of Washington in Applied Mathematics, and he was excellent. So glad I just stumbled upon his channel. It looks really good. Will be tuning in! I am hoping you will do a video on linear programming and optimization; I have a problem at work I want to apply this to. :)

[@Eigensteve]

Awesome, thanks Joe! I have some optimization videos in the works, so stay tuned!

[@giovanniiacobello2866]

Thank you very much Prof. Brunton for the interesting video! It is amazing this way of making videos. Which software and devices do you use for that? Is that a glass board? or something similar?

[@Eigensteve]

Yes, a glass board

[@DougWoodrow]

@@Eigensteve and yet the writing isn't back-to-front... clever! 🤔

[@tedgerhardt2619]

Thanks for such great videos! Been following you for years and I’ve learned so much and am truly inspired by your work. Just ordered the book and am excited to apply!

[@jacobanderson5693]

Would you consider doing a lecture on FFT in the context of NMR/Xray Crystallography/electron microscopy. Especially "Fourier Filtering"

[@Eigensteve]

Cool idea -- i'll add it to the list :)

[@彭九方]

Thanks a lot again Prof! Just wanted to mention that, when calculating the power spectrum (PSD), the data type of the production result of complex number with its conjugate is actually: complex with 0j (Python3)

[@kivancthewalrus]

You forgot to multiply your np.fft.fft(...) output by dt. np.fft module assumes the sampling spacing is 1. So we have to fix for this if we have a different sampling spacing. Also, you don't have to give n as the second argument of your np.fft.fft because n is also the length of your signal, so it's effectless.

[@ElCanalMeu]

Hello Steve, I just wanted to say thank you. It's been really helpful.

[@hardeepsingh68]

Beautifully explained ! Wonderful to learn from Steve !!

[@code2compass]

Every body's a gangsta until a man with glasses enters the room and explains fft

[@shahriarhabibi8382]

The magnitudes of the signals with 50 and 120 Hz depends on the random numbers that you generate at the beginning of the code. I had this problem that my magnitudes where different from those presented in this video.

[@NowanIlfideme]

I gotta say, watching these and re-remembering uni courses has been quite a blast. Thanks for your lectures! Also, what exactly do you use to make these videos? I assume you write on glass and then flip the video? It always turns out surprisingly clean, with the screen projection as well...

[@Eigensteve]

Thanks! Yep, glass and flip.

[@rafaelcardoso270]

You are awesome, dude. Thanks a lot. I wish I had a professor like you at the college.

[@ilAnfoosh]

Thanks for showing us how useful is learning Math!! Amazing!!

[@frankkoslowski6917]

Love your presentation. Interesting though that you got your code to work without errors. Casting complex values to floats ist verboten! 😏 Here's how it should be for the graphs to display without error: plt.plot(freq[L], np.absolute(PSD[L]), color='c', lw=2, label='Noisy') However, this does not solve the problem at it's root, because the Boolean return for a complex like `PSD` referenced to and integer as in: indices = PSD > 100 does only return a Boolean array filled with `False`, unless np.absolute(PSD) gets implemented. The best place to do this without undue repetition throughout the code is where the problem starts. Here we see the amendment that forces absolutes and brings all the trouble caused by attempting to handle complex values like floats to an end: PSD = np.absolute(fhat * np.conj(fhat) / n) # Power Spectrum Density (power of FFT) However, the error finds itself repeated in the last set of plots with the `Inverse FFT for filtered time signal` and is best corrected like this: ffilt = np.absolute(np.fft.ifft(fhat)) # Inverse FFT for filtered time signal 😉 All in all, a very clear and understandable `toy example`. 💚

[@Asparuh.Emilov]

This is one of the best videos on how to apply FFT in Python!!! Thank you so much! Is it possible to make a video with Time Series data denoising? And could be applied directly on the original Time Series Data Or first we should obtain the difference on 1st grade?

[@Ting3624]

thank you for the education kind sir! math + code example is a great idea

[@euclitian]

this!

[@StEvUgnIn]

7:25 Start of the explanation on FFT filtering

[@MandarPatkar_html_addict]

Thanks much Prof Brunton! This is gold.

[@kubumshel]

This lecture is very clean and precise. But I need to do this in matlab and I am really getting confused on how to implement the index vector and multiplying it from the noisy signal

[@dom996]

Thank you very much Prof. Brunton for the interesting video!

[@DiverselyArtistic]

Thank you so much sir ,i was really confused before this video

[@psheep8206]

Thanks Steve, great footage! Any chance to extend this series with a explainer on how to detrend a periodic signal in 2D data?

[@DueDue541]

import numpy as np import matplotlib.pyplot as plt plt.rcParams['figure.figsize'] = [16, 12] plt.rcParams.update({'font.size': 18}) # Create a simple signal by combining two frequencies dt = 0.001 t = np.arange(0,1,dt) f = np.sin(2*np.pi*50*t) + np.sin(2*np.pi*120*t) # sum of 2 frequencies f_clean = f f = f + 2.5*np.random.randn(len(t)) # add some noise plt.plot(t, f, color='c', LineWidth=1.5, label='Noisy') plt.plot(t, f_clean, color='k', LineWidth=2, label='Clean') plt.xlim(t[0], t[-1]) plt.legend() # Map the signal from Time Domain to Frequency Domain n = len(t) fhat = np.fft.fft(f,n) # Compute FFT PSD = fhat * np.conj(fhat) / n # Compute Power Spectral Density (Combine real & imaginary bins) freq = (1/(dt*n)) * np.arange(n) # Create x-axis of frequencies L = np.arange(1, np.floor(n/2), dtype='int') # Only plot first half of Freqencies fig, axs = plt.subplots(2,1) plt.sca(axs[0]) plt.plot(t,f,color='c',LineWidth=1.5,label='Noisy') plt.plot(t,f_clean,color='k',LineWidth=2,label='Clean') plt.xlim(t[0],t[-1]) plt.xlabel('time') plt.legend() plt.sca(axs[1]) plt.plot(freq[L], PSD[L], color='c', LineWidth=2, label='Power') plt.xlim(freq[L[0]],freq[L[-1]]) plt.xlabel('Hz') plt.legend() # Filter Signal based on PSD indices = PSD > 100 # Find all freqs with power greater than 100. [Boolean Array] PSDclean = PSD * indices # Filter PSD by multiplying by Boolean Array fhat = indices * fhat # Filter FFT coefficents that's Power < 100 # Inverse the cleaned FFT to get clean time domain signal ffilt = np.fft.ifft(fhat) # Plot fig,axs = plt.subplots(3,1) plt.sca(axs[0]) plt.plot(t,f,color='c',LineWidth=1.5,label='Noisy') plt.plot(t,f_clean,color='k',LineWidth=2,label='Clean') plt.xlim(t[0],t[-1]) plt.xlabel('time') plt.legend() plt.sca(axs[1]) plt.plot(t,f_clean,color='r',LineWidth=2,label='Clean') plt.plot(t,ffilt,color='k',LineWidth=2,label='Filtered') plt.xlim(t[0],t[-1]) plt.xlabel('time') plt.legend() plt.sca(axs[2]) plt.plot(freq[L], PSD[L], color='c', LineWidth=2, label='Power') plt.xlim(freq[L[0]],freq[L[-1]]) plt.xlabel('Hz') plt.legend() plt.show()

[@Eigensteve]

Thanks!!

[@EW-mb1ih]

To go further, it would be interesting to have a method to determine the threshold for PSD that we keep. In your case, it’s obvious that there are only two frequencies but what kind of methodology will you recommand if there no clear contrast between noise and real Frequency ( would a simple Normal t’est be sufficient?)?

[@Maria-mt2rz]

This video is adorably stylish!

[@azzouguenachour909]

Many many thanks professors this was very clear.

[@matiaharonyan5218]

Steve, you are amazing. Thank you a 1000 times.

[@Eigensteve]

You are so very welcome!

[@tatianaazarchenko7066]

Thank you a lot, Steve! Your lectures are absolutely amazing and extremely helpful!!

[@robertschofield9838]

Thanks for contributing! Exactly what I was looking for.

[@タナカコウイチ-w3p]

鏡文字書けるのすごいですね!( It's amazing to be able to write mirror writing!)

[@nitinupadhyay9193]

i liked and subscribed before even starting the video.. i knew it was going to be great!

[@arnoldsu]

Thanks for the explain, it is very details and easy to understand. One question, how can we demonstrate/apply the 95% significant test onto the plot? Thanks!

[@Neeba-q5x]

First time watching this guys video, pretty nice~ But one think that really attract my attention is that it looks like he is writing on an opposite way? I mean from what we see on the video, the scene is like he is writing on a glass board like thing And standing facing the camera, also keep the word readable from the side of camera. It's like we are looking at teacher from the other side of the blackboard but can also read the word at the same time. (the word should be mirrored!)

[@彭九方]

He is left handed and he mirrored the video

[@彭九方]

And he writes normally like us all

[@nhatson2265]

This work really helps, Sir! Thank you for your videos

[@Eigensteve]

Glad to hear that!

[@harisgulzar4605]

Wow great content. Python example just boosted my interest in signals processing by many folds. Thanks a lot Mr. Steve, I really wish to see your behind the scene setup by the way :p

[@Barasuisyou19]

Thank you very much for this amazing video! I was struggling to understand this, but you made it look easy.

[@cygtrafolia1007]

epic lighting. subscribed.

[@fastlab8506]

thanks for great explanation! You guys are doing it on a different level :)

[@Eigensteve]

Thanks so much!

[@midasredblade236]

Yes, THIS IS WHAT I WAS LOOKING FOR THANK YOU KZbin ALGORITHM

[@stijndeboer3721]

Thanks for your videos Steve.

[@arturfoden7139]

Thank you for the clean explanation

[@Asparuh.Emilov]

Thank you so much for those lectures! I love it! Is it actually possible to use Fourier series to create time series forecasting?

[@nischalsehrawat2130]

Hi Steve, I have a doubt. If we run a Kalman filter on this data, can the filter distort / change the frequency content of the signal? I tried to run a Kalman filter and sometimes I get funny results. I am not able to understand what might be going wrong. Can you please help? Thanks

[@Eigensteve]

That is a really good question. Yes, in general, filtering algorithms will change the spectrum of the signal, sometimes quite significantly. There is a lot of work in designing filters that have certain nice properties. For example, if we run a simple low-pass filter over the data, it will mess up the phase information in the signal (because there will be a small delay). So the 2nd order Butterworth filter is often used to prevent this phase corruption. Sometimes people also use the "filtfilt" function, which runs a forward and a backward pass of the filter to get rid of the phase delay. So long way of saying that yes, this is a real concern and it is not always easy to filter and preserve frequency content. But, remember, often when you filter, your goal is to modify the frequency content, for example to remove high-frequency noise. So there is no "silver bullet" for filtering.

[@Daniel88santos]

@@Eigensteve I'm using the filtfilt in sensor live data and that technic doesn't work for live data since I'm seeing a kind of "repaint" after I receive the last point on the data received previously. Let's say that I have 50 points ... when I receive the point 51 what was calculated on the previous 50 points will change when I receive this last point, so the filtfilt technic doesn't work well on the last point received. Another issue is when I compare the live output of my Butterworth filter to filtfilt output they are exactly the same on the last received data, meaning that the delay is there on the last point received, the delay is only taken away for that particular "event"(Let's point 50) when we receive more data( >=51 points). Can you advise me any technic that allow me to filter data in live stream having no or very few delay? My data is not periodic, I think that for example the FFT is not an option. Thanks you.

[@romanvereb7144]

@@Daniel88santos Applying FFT on live data sets, the size of which are not power-of-2? And then you're also filtering and inverse-transforming? Brave, yet interesting. My own power-of-2-cents would be... Don't allow the dataset to be sized dynamically. Keep a fixed size and let the live data "roll" through it. Make sure the size of this dataset is something like 64 instead of 50 or 51. Zero-pad if you must. If none of what I just advised made any sense to you, you may have to do a bit more research. Fourier Analysis and noise filtering is not a subject mastered with a few youtube videos.

[@Daniel88santos]

@@romanvereb7144 thanks by you reply, I've figure out how to do it with any size of points, and not using zero padding ... but I cannot comment on it, since is proprietary technology.

[@romanvereb7144]

​@@Daniel88santos Such is the way of the world... One learns without giving back. A seed planted, but no longer spread.

[@shillowcollins6392]

Pls I want to purchase your book, do I need to purchase both vol1 and vol2, or the vol2 is the second edition of volume 1?

[@Eigensteve]

No that is just the 2nd edition, so no need to buy both

[@artsyengineer2086]

Hi Steve, thanks for the nice and helpful video! One thing though, I was wondering if there is a "standard" way of choosing a filter value? Is your case of two "peaks" a good rule of thumb?

[@HackionSTx]

I'm not an expert so please don't take this as an absolute truth. I'd look at the graph first (or have so metrics, depending on the type of data) and see if we get only few very high peaks or something more regular. In the first case you'd probably have a lot of liberty if you wanted only the big peaks. In the second case I'd probably start being conservative and get some noise in the result if it means I'd get everything I wanted from the data. But depending on what I'd be working on I'd begin increasing the threshold until the data starts losing too much.

[@otaviocarvalho1140]

Thank you teacher for the awesome explanation!

[@goodlack9093]

thank you very much, you're a really good teacher!!

[@arkie3703]

Love your videos, I have a question tho. In the previous video Proffesor Brunton talked about how efficient the FFT was when our n was a power of 2. If I understand it correctly, here n = 1000? Why wouldnt we want n to be 1024?

[@rajinfootonchuriquen]

Thanks Steve for your knowledge.

[@vigneshrb1626]

Great lecture prof but I have one doubt. How to compute the x axis of the plot for any given data

[@juancarlosesquivel7855]

Thanks for your amazing videos and book. I'm watching them all, and I hope they'll help me with a Kaggle competition on detection of gravitational waves from time series obtained by LIGO and Virgo.

[@SP-ui6zn]

Thank you Dr. Brunton.

[@RupertBruce]

@6:20 Presentation Master! 🙂

[@robv3872]

what a great instructor! Thanks!

[@jean-francoisnoel9577]

Thank you very much for your amazing videos, but what about averaging to remove the Gaussian noise? (if the recordoing is long enough to perform several FFTs of course) It feels like a better method to remove noise since it can show information hidden below the noise floor that would otherwise be removed with the truncation method.

[@XwitterEye]

damn the video quality is amazing here. got a sub

[@inchae5279]

Thank you for making this video. I have one question with making a threshold, the threshold that you made, is subjective i thought. Is there any non-subjective threshold method? I mean, defined threshold to 100 isn't any valid reason.. (Sorry for my english)

[@babak5115]

Thank you Dear Steve for your best job, how can I see this similar example in Python?

[@jacobthomas939]

Agreed. A wealth of knowledge.

[@samsmusichub]

I ❤ FFTs. Thank you!

[@wahabfiles6260]

I have a silly question. Is it the correlation among the data that yields two spike in frequencies in power spectrum? because the white noise is uncorrelated.

[@Eigensteve]

The power spectral density and the autocorrelation are very closely related. It turns out that they are Fourier transform pairs. So it is definitely related.

[@wahabfiles6260]

@@Eigensteve makes sense. thanks

[@nightlyowll]

This video is amazing.. Just to the point !! I'm speechless on how good the video is.. ❤

[@dudidahan8721]

WOW, thank you for sharing this video!

[@Greg-ld4tr]

Thank you for the enlightening video. In order to calculate the PSD you have to use the prefactor 2/n**2, i.e. PSD = 2 * fhat * np.conj(fhat) /n**2, right?

[@monkeypoohonyou]

Amazing lecturer!

[@pdirac]

Brilliant. Just brilliant. Thanks

[@berkceyhan5031]

First I liked, then I watched. Amazing!