Making a Pitch Shifter

introductory guide to pitch shifting algorithms
#SoME3
code: github.com/JentGent/pitch-shift
===possibly helpful papers===
SOLA: doi.org/10.1109/ICASSP.1985.1168381
WSOLA: doi.org/10.1109/ICASSP.1993.319366
Phase vocoder: doi.org/10.1002/j.1538-7305.1966.tb01706.x
Phase vocoder: doi.org/10.1109/89.759041
===attributions===
auto-tuned music data from wikipedia
** images **
cher: flickr.com/photos/raph_ph/48932500988
t-pain: flickr.com/photos/willfolsom/6289614772
marimba: flaticon.com/free-icon/marimba_848242
** music ** (in order of appearance)
pixabay.com/music/110481
pixabay.com/music/2642
pixabay.com/music/146961
pixabay.com/music/111372
pixabay.com/music/3945
pixabay.com/music/156936
pixabay.com/music/161002
pixabay.com/music/143632
pixabay.com/music/143906
pixabay.com/music/4178
pixabay.com/music/110483

[@diplomaticfish]

This is really well made! I had no idea pitch shifting required this much complication.

[@wendolinmendoza517]

Same here. I really did not suspect so much math was needed for something this (apparently) simple.

[@Cineenvenordquist]

It doesn't. He should have done this with op amp circuits and it would be fine. Even if he skipped the window fudging thing to cut the pop, he could pick better transforms and meet success. I mean, why didn't he match the song effectiveness with what he demands from the filter; blam, a sing wrong fixer that has funkadelic qualia.

[@LuLeBe]

Funny that you say that, because this seems rather simple to me. No, I don't understand it at all, but the result is great for how "little" he did. I expected this quality to already need a huge amount of processing that only the gods at Antares (autotune company) have figured out.

[@DuckPerc]

@@LuLeBe The wizardry in autotune is all about interpolation between notes. The notes they sung, and the notes they wanted to sing. The repitching part probably isn't all that great if you just want a static effect. Like, imagine the math in this video, except the ratio between original pitch and target pitch is a function of the frequency content of the signal you started with.

[@MatthijsvanDuin]

@@Cineenvenordquist Ehm, no, what he's shown in the video is pretty much as simple as it gets for performing pitch shifting, which is just inherently difficult. I can't even guess what you could be referring to with the extremely vague "op amp circuits"... the function of an op amp in basically every circuit that uses one (except when used as a comparator) is to solve an equation, but the actual equation it solves is entirely defined by rest of the circuit, so describing something as an "op amp circuit" is utterly non-descriptive. Regardless, the fundamental problem of pitch shifting are not specific to how the audio is represented, and so are the solutions except they're much harder to implement using analog hardware than digital filtering, and getting phase to match for transient sounds seems like it would be particularly painful.

[@ianbelletti6241]

The original Alvin and the Chipmunks albums took advantage of speeding up the playback to change the pitch because they weren't being digitally processed. This means that the artists had to sing at an appropriately slower beat to account for the time shrinkage.

[@charleslambert3368]

And then you can play it on a 16RPM turntable to hear the singer sing really slowly and the band play down an octave or so

[@ianbelletti6241]

@@charleslambert3368 the way these things were recorded the singers sung lower notes at a slower beat. The instruments were played at normal rate. The singer track and the instrumental track were combined later.

[@raksipulikka]

Last semester did a project about phase vocoders at an audio signal processing course. Your explanation about is absolutely superb, we had to give a presentation at the end of the course and it was train wreck compared to what you managed to explain in 15 minutes. Well done!

[@mavaction]

Super smooth. Clear, concise and thorough... and then having a real demo audio to follow examples is a great presentation feature. As someone interested in spectrograms I'd like to see a video of which audio properties create the most problems for a shifter vs which properties are easiest to work with. I've got some guesses but I imagine you could demo it in a similarly illuminating video.

[@TheGamefreakr]

As a semi-professional producer and programmer, I am deeply fascinated by this.

[@a52productions]

This is an incredible explanation! The visuals are great, and the way the process is laid out ("lets try this -- oh that causes such and such problem") really helps understanding the rationale behind every aspect of the algorithm. I knew, as a physicist and electronic musician, that pitch shifting was hard. But I never knew how it worked, just that from a math side I couldn't come up with a good solution and from a musician side that choosing the right algorithm for a given sample was often difficult. This explains so much!! What a satisfying algorithm and a satisfying explanation.

[@HunterHerbst]

This video is truly amazing. I hope you make more coding content and simplify other random topics for literally anything. If it's just a one-off you still did amazing and I wish you well on future endeavors

[@amyvo2278]

Thanks

[@MysteryPancake]

6:28 The reason pitch modification is listed as "yes" for PSOLA is because it's the only algorithm that doesn't change the formant, which is important to produce natural sounding speech. Unlike SOLA and WSOLA, it doesn't introduce a chipmunk effect when speech is sped up. Because of this, it's heavily used in many autotune plugins today. It's possible to tell if a plugin uses PSOLA by shifting the vocal down: - If the formant sounds the same and there's gaps in the waveform, this indicates PSOLA was used, as it has no way to fill in those gaps. - If the formant sounds the same and there's no gaps, this indicates a frequency-based approach was used, as it can fill in those gaps. - if the formant sounds different, it's likely another approach was used.

[@samuelwyeth4178]

This is insane quality and content for such a small channel. Hoping you get the attention you deserve!

[@davidvomlehn4495]

Nice. My degree is in Physics. I assumed this was done with Fourier transforms, but I had always wondered how the finite sample sizes were handled. Basically, how they were glued together. This was well presented and I can see how much more there is to learn.

[@bentpen2805]

Do check out wavelets (including the continuous wavelet transform), as well as spectrograms. It’s all so cool.

[@davidvomlehn4495]

@@bentpen2805 Wavelets are on my list of things to check out, this is probably a fun place to start.

[@theb1rd]

I've wondered about this for decades. Great explanation, thank you!

[@user-yv6xw7ns3o]

This is something I’ve wondered about for a long time! Nicely done. It gives me hope when I see quality videos from small or new creators show up in my recommendations 😃

[@LaszloKorte]

One of the best signal processing videos I have seen yet

[@made.online2149]

Best video on an audio DSP topic I've ever seen tbh!

[@msclrhd]

The Overlapped Add (OLA) algorithms also form the basis of several text-to-speech vocoders like MBROLA, PSOLA, etc.

[@robertphilip1832]

I've understand so much more and so much better about pitchshifting by Your video. Thanks for it, really great job done!

[@MarkMendell]

This is fantastic. The animations and code are phenomenal for keeping up with the concepts. Great job!

[@Etobio]

This video is so well put together! I’d love to see this but with formant shifters. Love the content

[@mintsjams8862]

I've been watching a lot of SoME videos recently and this is one of my favs. Well done!

[@PhilipSmolen]

Thanks. I've always wondered how this was done. I didn't realize it was so complicated.

[@moralboundaries1]

Thanks for sharing this! Easy to follow and really interesting! Your pitch shifter sounded great!

[@byte.observer]

Excellent video! I've implemented these algorithms many times by now but having this video 5 years ago would have saved me a huge amount of time and headache :)

[@Cineenvenordquist]

Wait, why do you reimplement this bunches and why don't you just use an op amp circuit or get a singer who likes working to spec? Or a marimba with motorized dampers? Do you have to teach elliptical equations on python driven instruments?

[@TehBrian]

I lost it right around 11:00-ish. Nice to see where my current math knowledge tapers off. Hopefully I can revisit in a couple years and be able to understand the whole video. :)

[@Fasteroid]

I remember watching a video some time ago where this guy deconstructed a square wave with FFT, phase-shifted the frequencies to minimize the constructive interference between them, and then put them back together. The result was a signal that still sounded like a square wave, but could be pushed far louder. I wonder if certain "compressor" type filters also use this approach, and what they might do to transients having seen after seeing the results of naive pitch shifting with phase shifting.

[@gabrieldoudna6570]

isn't a pure square wave the loudest possible waveform to begin with? i would guess that you could only decrease its loudness with this technique (ie increase its crest factor)

[@aotmr1604]

@@gabrieldoudna6570 indeed, phase-shifting the components of a square wave by a quarter turn produces in theory a signal with infinite crest factor

[@yuvalne]

the demonstration are what really make this video. truly well done.

[@morgan0]

9:15 expanding on this, for N samples of time domain data, you get N samples of frequency domain data. however each frequency is a complex number, using two samples each, dc offset gets one. the nyquist frequency also gets only one, because phase shift between nyquist sampling frequency and input frequency at nyquist results in amplitude change. for every phase shift there is an amplitude that will make it fit the samples, and hence only one can be represented (and we never know which it is).

[@MatthijsvanDuin]

Though note that FFT outputs N-1…N/2+1 (note descending order) are the complex conjugates of outputs 1…N/2-1 and therefore are redundant, so if you discard those you've got a real numbers for DC and N/2-1 complex numbers for frequencies between DC and nyquist, and a final real number for nyquist, which all combined conveniently fits in the same amount of memory as the original N real samples :-) (All this is assuming N is even; if N is odd then there's (N-1)/2 frequency samples between DC and nyquist and no frequency sample _at_ nyquist.)

[@__MINT_]

​@@MatthijsvanDuinExactly, values are mirrored with nyquist bin being the center point, real part of mirrored values is the same, while for imaginary part the sign flips. So mirrored value of x = a + bi would be y = a - bi. For example, with 1024 frame size, 0th coefficient is DC, 512th is nyquist, and every other coefficient 1024 - n, where n is coefficient number from 1 to 511, is a mirrored version of coefficient n. 513 is mirrored 511, 514 is mirrored 510, etc. I assume that you know this, but maybe someone will learn something new.

[@danielchin1259]

Content presented around 8:00 is a bit misinformed. STFT preserves all info. Discrete time-domain repr doesn't include any info above Nyquist freq either. When pitch shifting, freqs higher than Nyquist is supposed to be lost, so scaling the frequency axis is not problematic. HOWEVER! The video up to 8:00 was very cool and taught me a lot of time-domain methods I didn't know about. Very inspring, and I appreciate that!

[@danielchin1259]

I just finished the video. 12:00+ was imprssive as well! You are utilizing assumptions about natural music to strike beyond theratical limits.

[@FF-94.mp3]

this taught me more than a whole half a semester of uni. good video. sent it to everyone and their grandma

[@Descenacre]

Subscribed instantly! This is an incredible video, thank you for visualizing something I've been incredibly curious about but had no clue how to find out! Beyond just being a math nerd I'm also a musician so a lot of this information might come in handy in the future...

[@michael-j-harrison]

Long before the existence of auto-tune there were pitch shifters that were used in live performance situations to eliminate audio feedback from the speakers to the microphones. These units had a very clever approach which I wonder why is this not used for auto-tune? The approach was based on something very similar to super-hetrodyne theory. The audio was multiplied by a carrier signal and the modulated signal was filterd to remove one of the side bands and then this filered signal was multiplied by another carrier signal that was slightly frequency shifted from the previous carrier frequency. This demodulation proces gave an output signal that was slightly frequency shifted from the original.

[@schmitzbeats6102]

What you describe are frequency shifters, not pitch shifters. When you apply too much of that, it messes up the harmonic relations of the sound. (A couple Hz of shift is not very audible, but removes the audio feedback.)

[@Cineenvenordquist]

That's a kinda DAW in the making, just add a tape loop or 7. Knocked away common mode and room resonance noise?

[@CFSworks]

My DSP experience is in digital telecommunications (OFDM! Costas loops! LMS equalizers! Oh my!), where this kind of operation isn't really ever necessary. I'd always wondered how my compatriots on the audio side of the DSP world managed to develop an algorithm to do this. Turns out it's an even more complex problem than I thought and I should perhaps rethink saying "I do basically the same thing, just at a higher sample rate." Excellent video editing and visualizations! With production this good, I'm surprised you only have ~1.57K subscribers - but count me among them now. :)

[@rb8049]

I’ve always wondered how this was done. Thank you. I could only really understand if I did it myself, but I get the gist.

[@CK-3K]

I like how almost every hurdle in audio processing comes back to phase when you trace it back far enough.

[@cornoc]

amazing video, took something i thought was magic and made it simple enough to the point where i feel i can implement it myself

[@4.0.4]

The piano tunes in the background were a nice touch!

[@calebdeming5515]

Jupiter notebooks unlocks alot of the mystery to this video.

[@aze4308]

WOAH this is incredible for your first video! i hope you blow up!

[@joelpetersson8742]

Amazing video! Thanks for putting so much effort into it!

[@mumiemonstret]

That was a concise walkthrough of the steps necessary to pitch shift. I never thought there were so many, and apparently you left some out. I can say that your end result still sounded way better than what Cool Edit Pro (predecessor to Adobe Audition) could accomplish. Of the left out steps, I would like to see formant preservation as the topic of a follow-up. It would be interesting to hear what happens when you move formants and pitch independently in a vocal sample.

[@WangleLine]

I've been hoping to see a video like this one day, thank you so much for making it

[@pepedecoatza]

What a video. I just saw its your first video in your channel, so, congratulations and keep it up!

[@triplezgames3882]

Damn, this channel is an underrated gem... I've been here at 661 subs, see you at many thousands soon

[@ri-gor]

Love this video. I took a college level course on digital signal processing, and we never even covered those hann windows. I definitely saw some leakage in my projects, but thought it was unavoidable (similar to the Gibbs phenomenon).

[@maokus]

"I don't like words", says youtuber who evidently read many, many words in order to make this video. We see through your trickery >:( In all seriousness though, this is an incredible video. Thanks for your contribution!!! What did you use to make the vid? Is it manim?

[@Aio-Project]

the amount of math videos i had to rewatch before returning to this

[@playbyan1453]

Very technical details that I love, clrear explanation and very inspiring. The vocoder result sounds so high quality.

[@vilkillian]

5:32. To be precise, auto-correlation is a convolution between signal and itself, but delayed, this is what auto- means, in this case it is just "correlation"

[@BKBluey]

This is EXACTLY what I was looking for.

[@JonHarris77]

Really enjoyed this! I would love a video on formant shifting please.

[@mikecase2033]

I love when I see something I take for granted (viewing youtube at x1.5) has a ton of math behind it. Awesome video

[@shinyless]

best explanation of pitch shifting I've seen so far, this is a 3blue1brown level of quality video, awesome work !!

[@ngideo]

This is really well done, I just feel like you could have given a quick recap of DFT, since you took a long time with the time domain explanation, but went super fast in the frequency domain explanation.

[@karlkastor]

Great video! I always thought pitch shifting was just translation in the frequency domain. I like how you explained the problems with the example.

[@JJRicks]

The silence after "the guy's name was just... Hann?....." is so perfect haha

[@SamAlton]

I also don't like reading words. Thanks for the pretty animations and explanation!

[@Cineenvenordquist]

Yeah, so why weren't the Jupyter notebooks in Sugar or a new Jaron Lanier language instead of python?

[@TheLaxOne]

Wow I've always wondered how pitch shifting algorithms work and you did a really great job illustrating how some of them do so with this vid!

[@Rajkumz]

Wonderful video. Excellent idea to calculate the volume ❤❤

[@fakesilver746]

really nice video. I appreciate that you took the time to define the jargon words.

[@FelixArchive]

Amazing video! I was actually looking for this type of video a year or so ago and couldn't find one. Thank you for making this!

[@ag-k]

only watched 4mins as of yet and i already learned more than i did in school lmao you deserve a sub

[@1495978707]

I’ve been wondering about this for a long time, but haven’t gotten around to reading a digital audio book to understand it

[@zorggn]

I wish i found this video sooner tbh, would have saved me some time, i believe. :3 After toiling for quite a bit of time trying to implement a realtime pitch shifter/time stretcher (hence i didn't want to touch fft or other things that would eat performance), i arrived at a solution where i only ever use two windows overlapping exactly 50%, but for each step to generate a new chunk, the windows are both resized to a size that's the chosen one plus/minus a small (uniformly or normally, i tried both) random deviation; Considering that a static window size was producing a very noticeable tone with my code with window sizes that were small enough to make the crossover points fast enough to go into audible pitch territory, the randomness introduced seemed to make that less noticeable quite nicely, basically akin to turning a pure tone into white noise. I still have issues with fluttering though, that i couldn't find time to implement a fix for yet.

[@hughmathimati]

Yo, I remember you from Khan Academy! Keep making awesome stuff dude.

[@kaishang6406]

Thank you very much, the ideas in this video are useful for more than just music.

[@riccardorosso2795]

Do more video like this one please!! Incredible work!

[@Ignisami]

Going to have to watch this a few times to really grok what’s going on, especially after the DFTs enter the picture. Been a while since I’ve done math to any serious level

[@eqwerewrqwerqre]

This is amazing!

[@apotheotic]

Excellent video, I hope to see more from you soon

[@ReginaCæliLætare]

"It's full of big word" brother, most of the words you used flew right over my head I understood almost nothing

[@DokterKaj]

So glad I clicked on this video, beautifully made!

[@gooball2005]

Incredibly interesting and well explained!

[@ThingsMadeSimple]

Excellent video! So clearly explained. I hope you will do many more like it!

[@fano72]

Awesome video and easy explanation of such complex stuff👍

[@wChris_]

Wait there was an almost identical presentation at the ADC22, just 4 Months ago! Coincidence? I think not!!

[@Cineenvenordquist]

Analog Semi does a conference? No, it's just audio not radar etc. so ...whoever they are, they have a YT Channel ADC. Or not, but some talks are there.

[@geraintluff]

That was me, here you go 😄 kzbin.info/www/bejne/nHu4nqCZfLCNg6s Yeah, this video hits many of the same points in a pretty similar order to mine - but presentation counts for a lot, and Jent's done a good job here. My goal is always for a train of thought to feel natural and obvious in retrospect, so if that's what happened here I'd be flattered.

[@tibr]

Fantastic visualisations! Tnx for the video!

[@sirati9770]

I am watching this video at about 4 Times speed. This actually becomes very important to me because these artifacts dictate how fast I can watch a video on a device (or in general)

[@bean_mhm]

Absolutely amazing

[@parlancex]

Informative with nice animations, thanks for making this.

[@mavisbavingson2299]

this video is seriously awesome and inspiring thank you for putting in the effort! 😊

[@seatyourself7082]

My Favorite SoMe

[@AK-vx4dy]

Your knowledge level is intimidating... Also talent to present it clearly. Thank you for your work. I once tried "pitch shifter" using doppler effect method, hardware couldn't even dream of Fourier . I wonder if you have knowledge how pitch preserver works like in youtube speed change

[@camerongrenier5405]

I studied physics for 3 years, and I’m currently studying computer science. That was wayyyyyyyy more complicated than I was expecting and a lot of it went over my head 💀

[@backpackvacuum9520]

Great video! I got a little lost with some of the Fourier stuff, but I haven't found a resource yet that doesn't lose me on that topic 😂 I'm surprised to see python doing this! I did a project doing DSP in python a few months ago and started hitting the limits of what could be done fast enough. Although I was doing real-time, not sure if you are. And I wasn't making extensive use of libraries written in C.

[@magran17]

Wow, great work

[@JorneRichard]

super great video and now im dizzy about phases... i will come back someday

[@juan23415]

great video!

[@PATRIK67KALLBACK]

Super great video, thanks!

[@Subbestionix]

How is this the first video you made?! 3b1b's software raised the bar man Keep it up!

[@malikamajjoud8324]

I love this video. You did an amazing job here :)

[@fluffsquirrel]

Are you the Jentacular Gent from Kahnacademy who made that incredibly Ice-cream cone smooth renderer? No worries if not, I just would imagine that your name being similar, it would make sense you would make a smashing video. Great work anyway!

[@timothydrewfilms5272]

WAIT! How the hell do you only have 174 subscribers!? I only noticed after watching the whole video and then going to check your channel for more xD Consider it 175 now

[@chadyonfire7878]

This is crazy love it

[@pmishraofficial]

This is really well explained!

[@geraintluff]

4:22 The fluttering is related to how much the output blocks cross over. Since you've fixed the analysis hop size (instead of deciding the output hop size and then figuring out what the input one should be) compressing in time means the output blocks overlap a lot more.

[@JentacularGent]

This seems right; I just tried since fixing the output hop size, and it does introduce fluttering when shifting down. But shifting down decreases hop size, which creates more overlap, so shouldn't it have had more fluttering?

[@geraintluff]

​@@JentacularGent More overlap reduces fluttering - the more blocks you're adding together, the more their fluctuations average out to be smooth.

[@noharahien]

Very helpful. Thank you!

[@dittilio]

Excellent video. I really hope the phases didn't smother your interest in making more videos. Instant subscription.

[@kruvik]

I implemented a pitch shifter and time stretcher in 2018 or so and was surprised how much non-trivial math was involved in doing that...

[@tomatosauce5775]

I’ve struggled to get anything with pitch shifting done without sounding bad which is why I prefer the “just doing it in the right key/in tune during recording” strategy. At least now I know why it’s so complicated 😭

[@wareya]

Could you improve the time-domain algorithm by splitting the signal up into several frequency bands with a brick-wall bandpass, timestretching each one, then recombining them? You could use smaller window sizes for higher-frequency bands, so transients would be less disturbed by bass frequency phase alignment.

[@Xzurii]

amazing content, 10/10 please continue