Making a Pitch Shifter

introductory guide to pitch shifting algorithms
#SoME3
code: github.com/JentGent/pitch-shift
===possibly helpful papers===
SOLA: doi.org/10.1109/ICASSP.1985.1168381
WSOLA: doi.org/10.1109/ICASSP.1993.319366
Phase vocoder: doi.org/10.1002/j.1538-7305.1966.tb01706.x
Phase vocoder: doi.org/10.1109/89.759041
===attributions===
auto-tuned music data from wikipedia
** images **
cher: flickr.com/photos/raph_ph/48932500988
t-pain: flickr.com/photos/willfolsom/6289614772
marimba: flaticon.com/free-icon/marimba_848242
** music ** (in order of appearance)
pixabay.com/music/110481
pixabay.com/music/2642
pixabay.com/music/146961
pixabay.com/music/111372
pixabay.com/music/3945
pixabay.com/music/156936
pixabay.com/music/161002
pixabay.com/music/143632
pixabay.com/music/143906
pixabay.com/music/4178
pixabay.com/music/110483

[@ianbelletti6241]

The original Alvin and the Chipmunks albums took advantage of speeding up the playback to change the pitch because they weren't being digitally processed. This means that the artists had to sing at an appropriately slower beat to account for the time shrinkage.

[@charleslambert3368]

And then you can play it on a 16RPM turntable to hear the singer sing really slowly and the band play down an octave or so

[@ianbelletti6241]

@@charleslambert3368 the way these things were recorded the singers sung lower notes at a slower beat. The instruments were played at normal rate. The singer track and the instrumental track were combined later.

[@diplomaticfish]

This is really well made! I had no idea pitch shifting required this much complication.

[@danielsantrikaphundo4517]

Same here. I really did not suspect so much math was needed for something this (apparently) simple.

[@Cineenvenordquist]

It doesn't. He should have done this with op amp circuits and it would be fine. Even if he skipped the window fudging thing to cut the pop, he could pick better transforms and meet success. I mean, why didn't he match the song effectiveness with what he demands from the filter; blam, a sing wrong fixer that has funkadelic qualia.

[@LuLeBe]

Funny that you say that, because this seems rather simple to me. No, I don't understand it at all, but the result is great for how "little" he did. I expected this quality to already need a huge amount of processing that only the gods at Antares (autotune company) have figured out.

[@ldcent8482]

@@LuLeBe The wizardry in autotune is all about interpolation between notes. The notes they sung, and the notes they wanted to sing. The repitching part probably isn't all that great if you just want a static effect. Like, imagine the math in this video, except the ratio between original pitch and target pitch is a function of the frequency content of the signal you started with.

[@MatthijsvanDuin]

@@Cineenvenordquist Ehm, no, what he's shown in the video is pretty much as simple as it gets for performing pitch shifting, which is just inherently difficult. I can't even guess what you could be referring to with the extremely vague "op amp circuits"... the function of an op amp in basically every circuit that uses one (except when used as a comparator) is to solve an equation, but the actual equation it solves is entirely defined by rest of the circuit, so describing something as an "op amp circuit" is utterly non-descriptive. Regardless, the fundamental problem of pitch shifting are not specific to how the audio is represented, and so are the solutions except they're much harder to implement using analog hardware than digital filtering, and getting phase to match for transient sounds seems like it would be particularly painful.

[@raksipulikka]

Last semester did a project about phase vocoders at an audio signal processing course. Your explanation about is absolutely superb, we had to give a presentation at the end of the course and it was train wreck compared to what you managed to explain in 15 minutes. Well done!

[@mavaction]

Super smooth. Clear, concise and thorough... and then having a real demo audio to follow examples is a great presentation feature. As someone interested in spectrograms I'd like to see a video of which audio properties create the most problems for a shifter vs which properties are easiest to work with. I've got some guesses but I imagine you could demo it in a similarly illuminating video.

[@TheGamefreakr]

As a semi-professional producer and programmer, I am deeply fascinated by this.

[@a52productions]

This is an incredible explanation! The visuals are great, and the way the process is laid out ("lets try this -- oh that causes such and such problem") really helps understanding the rationale behind every aspect of the algorithm. I knew, as a physicist and electronic musician, that pitch shifting was hard. But I never knew how it worked, just that from a math side I couldn't come up with a good solution and from a musician side that choosing the right algorithm for a given sample was often difficult. This explains so much!! What a satisfying algorithm and a satisfying explanation.

[@HunterHerbst]

This video is truly amazing. I hope you make more coding content and simplify other random topics for literally anything. If it's just a one-off you still did amazing and I wish you well on future endeavors

[@amyvo2278]

Thanks

[@MysteryPancake]

6:28 The reason pitch modification is listed as "yes" for PSOLA is because it's the only algorithm that doesn't change the formant, which is important to produce natural sounding speech. Unlike SOLA and WSOLA, it doesn't introduce a chipmunk effect when speech is sped up. Because of this, it's heavily used in many autotune plugins today. It's possible to tell if a plugin uses PSOLA by shifting the vocal down: - If the formant sounds the same and there's gaps in the waveform, this indicates PSOLA was used, as it has no way to fill in those gaps. - If the formant sounds the same and there's no gaps, this indicates a frequency-based approach was used, as it can fill in those gaps. - if the formant sounds different, it's likely another approach was used.

[@msclrhd]

The Overlapped Add (OLA) algorithms also form the basis of several text-to-speech vocoders like MBROLA, PSOLA, etc.

[@davidvomlehn4495]

Nice. My degree is in Physics. I assumed this was done with Fourier transforms, but I had always wondered how the finite sample sizes were handled. Basically, how they were glued together. This was well presented and I can see how much more there is to learn.

[@bentpen2805]

Do check out wavelets (including the continuous wavelet transform), as well as spectrograms. It’s all so cool.

[@davidvomlehn4495]

@@bentpen2805 Wavelets are on my list of things to check out, this is probably a fun place to start.

[@mikecase2033]

I love when I see something I take for granted (viewing youtube at x1.5) has a ton of math behind it. Awesome video

[@samuelwyeth4178]

This is insane quality and content for such a small channel. Hoping you get the attention you deserve!

[@Aio-Project]

the amount of math videos i had to rewatch before returning to this

[@user-yv6xw7ns3o]

This is something I’ve wondered about for a long time! Nicely done. It gives me hope when I see quality videos from small or new creators show up in my recommendations 😃

[@LaszloKorte]

One of the best signal processing videos I have seen yet

[@TehBrian]

I lost it right around 11:00-ish. Nice to see where my current math knowledge tapers off. Hopefully I can revisit in a couple years and be able to understand the whole video. :)

[@PhilipSmolen]

Thanks. I've always wondered how this was done. I didn't realize it was so complicated.

[@robertphilip1832]

I've understand so much more and so much better about pitchshifting by Your video. Thanks for it, really great job done!

[@pepedecoatza]

What a video. I just saw its your first video in your channel, so, congratulations and keep it up!

[@morgan0]

9:15 expanding on this, for N samples of time domain data, you get N samples of frequency domain data. however each frequency is a complex number, using two samples each, dc offset gets one. the nyquist frequency also gets only one, because phase shift between nyquist sampling frequency and input frequency at nyquist results in amplitude change. for every phase shift there is an amplitude that will make it fit the samples, and hence only one can be represented (and we never know which it is).

[@MatthijsvanDuin]

Though note that FFT outputs N-1…N/2+1 (note descending order) are the complex conjugates of outputs 1…N/2-1 and therefore are redundant, so if you discard those you've got a real numbers for DC and N/2-1 complex numbers for frequencies between DC and nyquist, and a final real number for nyquist, which all combined conveniently fits in the same amount of memory as the original N real samples :-) (All this is assuming N is even; if N is odd then there's (N-1)/2 frequency samples between DC and nyquist and no frequency sample _at_ nyquist.)

[@__MINT_]

​@@MatthijsvanDuinExactly, values are mirrored with nyquist bin being the center point, real part of mirrored values is the same, while for imaginary part the sign flips. So mirrored value of x = a + bi would be y = a - bi. For example, with 1024 frame size, 0th coefficient is DC, 512th is nyquist, and every other coefficient 1024 - n, where n is coefficient number from 1 to 511, is a mirrored version of coefficient n. 513 is mirrored 511, 514 is mirrored 510, etc. I assume that you know this, but maybe someone will learn something new.

[@Etobio]

This video is so well put together! I’d love to see this but with formant shifters. Love the content

[@MarkMendell]

This is fantastic. The animations and code are phenomenal for keeping up with the concepts. Great job!

[@mintsjams8862]

I've been watching a lot of SoME videos recently and this is one of my favs. Well done!

[@moralboundaries1]

Thanks for sharing this! Easy to follow and really interesting! Your pitch shifter sounded great!

[@CananaMan]

Subscribed instantly! This is an incredible video, thank you for visualizing something I've been incredibly curious about but had no clue how to find out! Beyond just being a math nerd I'm also a musician so a lot of this information might come in handy in the future...

[@danielchin1259]

Content presented around 8:00 is a bit misinformed. STFT preserves all info. Discrete time-domain repr doesn't include any info above Nyquist freq either. When pitch shifting, freqs higher than Nyquist is supposed to be lost, so scaling the frequency axis is not problematic. HOWEVER! The video up to 8:00 was very cool and taught me a lot of time-domain methods I didn't know about. Very inspring, and I appreciate that!

[@danielchin1259]

I just finished the video. 12:00+ was imprssive as well! You are utilizing assumptions about natural music to strike beyond theratical limits.

[@Fasteroid]

I remember watching a video some time ago where this guy deconstructed a square wave with FFT, phase-shifted the frequencies to minimize the constructive interference between them, and then put them back together. The result was a signal that still sounded like a square wave, but could be pushed far louder. I wonder if certain "compressor" type filters also use this approach, and what they might do to transients having seen after seeing the results of naive pitch shifting with phase shifting.

[@gabrieldoudna6570]

isn't a pure square wave the loudest possible waveform to begin with? i would guess that you could only decrease its loudness with this technique (ie increase its crest factor)

[@aotmr1604]

@@gabrieldoudna6570 indeed, phase-shifting the components of a square wave by a quarter turn produces in theory a signal with infinite crest factor

[@vilkillian]

5:32. To be precise, auto-correlation is a convolution between signal and itself, but delayed, this is what auto- means, in this case it is just "correlation"

[@yuvalne]

the demonstration are what really make this video. truly well done.

[@lnerqad--5268]

this taught me more than a whole half a semester of uni. good video. sent it to everyone and their grandma

[@aze4308]

WOAH this is incredible for your first video! i hope you blow up!

[@hughmathimati]

Yo, I remember you from Khan Academy! Keep making awesome stuff dude.

[@CFSworks]

My DSP experience is in digital telecommunications (OFDM! Costas loops! LMS equalizers! Oh my!), where this kind of operation isn't really ever necessary. I'd always wondered how my compatriots on the audio side of the DSP world managed to develop an algorithm to do this. Turns out it's an even more complex problem than I thought and I should perhaps rethink saying "I do basically the same thing, just at a higher sample rate." Excellent video editing and visualizations! With production this good, I'm surprised you only have ~1.57K subscribers - but count me among them now. :)

[@byte.observer]

Excellent video! I've implemented these algorithms many times by now but having this video 5 years ago would have saved me a huge amount of time and headache :)

[@Cineenvenordquist]

Wait, why do you reimplement this bunches and why don't you just use an op amp circuit or get a singer who likes working to spec? Or a marimba with motorized dampers? Do you have to teach elliptical equations on python driven instruments?

[@4.0.4]

The piano tunes in the background were a nice touch!

[@michaelharrison1093]

Long before the existence of auto-tune there were pitch shifters that were used in live performance situations to eliminate audio feedback from the speakers to the microphones. These units had a very clever approach which I wonder why is this not used for auto-tune? The approach was based on something very similar to super-hetrodyne theory. The audio was multiplied by a carrier signal and the modulated signal was filterd to remove one of the side bands and then this filered signal was multiplied by another carrier signal that was slightly frequency shifted from the previous carrier frequency. This demodulation proces gave an output signal that was slightly frequency shifted from the original.

[@schmitzbeats6102]

What you describe are frequency shifters, not pitch shifters. When you apply too much of that, it messes up the harmonic relations of the sound. (A couple Hz of shift is not very audible, but removes the audio feedback.)

[@Cineenvenordquist]

That's a kinda DAW in the making, just add a tape loop or 7. Knocked away common mode and room resonance noise?

[@calebdeming5515]

Jupiter notebooks unlocks alot of the mystery to this video.

[@playbyan1453]

Very technical details that I love, clrear explanation and very inspiring. The vocoder result sounds so high quality.

[@FelixArchive]

Amazing video! I was actually looking for this type of video a year or so ago and couldn't find one. Thank you for making this!

[@commiekiller3000]

I like how almost every hurdle in audio processing comes back to phase when you trace it back far enough.

[@joelpetersson8742]

Amazing video! Thanks for putting so much effort into it!

[@WangleLine]

I've been hoping to see a video like this one day, thank you so much for making it

[@cornoc]

amazing video, took something i thought was magic and made it simple enough to the point where i feel i can implement it myself

[@Rajkumz]

Wonderful video. Excellent idea to calculate the volume ❤❤

[@rb8049]

I’ve always wondered how this was done. Thank you. I could only really understand if I did it myself, but I get the gist.

[@TheLaxOne]

Wow I've always wondered how pitch shifting algorithms work and you did a really great job illustrating how some of them do so with this vid!

[@kaishang6406]

Thank you very much, the ideas in this video are useful for more than just music.

[@fakesilver746]

really nice video. I appreciate that you took the time to define the jargon words.

[@SamAlton]

I also don't like reading words. Thanks for the pretty animations and explanation!

[@Cineenvenordquist]

Yeah, so why weren't the Jupyter notebooks in Sugar or a new Jaron Lanier language instead of python?

[@therockafire]

Well, that clears that up! (For Albert Einstein!)

[@ri-gor]

Love this video. I took a college level course on digital signal processing, and we never even covered those hann windows. I definitely saw some leakage in my projects, but thought it was unavoidable (similar to the Gibbs phenomenon).

[@riccardorosso2795]

Do more video like this one please!! Incredible work!

[@made.online2149]

Best video on an audio DSP topic I've ever seen tbh!

[@apotheotic]

Excellent video, I hope to see more from you soon

[@fano72]

Awesome video and easy explanation of such complex stuff👍

[@Ignisami]

Going to have to watch this a few times to really grok what’s going on, especially after the DFTs enter the picture. Been a while since I’ve done math to any serious level

[@mavisbavingson2299]

this video is seriously awesome and inspiring thank you for putting in the effort! 😊

[@malikamajjoud8324]

I love this video. You did an amazing job here :)

[@ThingsMadeSimple]

Excellent video! So clearly explained. I hope you will do many more like it!

[@gooball2005]

Incredibly interesting and well explained!

[@JonHarris77]

Really enjoyed this! I would love a video on formant shifting please.

[@tibr]

Fantastic visualisations! Tnx for the video!

[@mumiemonstret]

That was a concise walkthrough of the steps necessary to pitch shift. I never thought there were so many, and apparently you left some out. I can say that your end result still sounded way better than what Cool Edit Pro (predecessor to Adobe Audition) could accomplish. Of the left out steps, I would like to see formant preservation as the topic of a follow-up. It would be interesting to hear what happens when you move formants and pitch independently in a vocal sample.

[@triplezgames3882]

Damn, this channel is an underrated gem... I've been here at 661 subs, see you at many thousands soon

[@JJRicks]

The silence after "the guy's name was just... Hann?....." is so perfect haha

[@ag-k]

only watched 4mins as of yet and i already learned more than i did in school lmao you deserve a sub

[@karlkastor]

Great video! I always thought pitch shifting was just translation in the frequency domain. I like how you explained the problems with the example.

[@wChris_]

Wait there was an almost identical presentation at the ADC22, just 4 Months ago! Coincidence? I think not!!

[@Cineenvenordquist]

Analog Semi does a conference? No, it's just audio not radar etc. so ...whoever they are, they have a YT Channel ADC. Or not, but some talks are there.

[@geraintluff]

That was me, here you go 😄 kzbin.info/www/bejne/nHu4nqCZfLCNg6s Yeah, this video hits many of the same points in a pretty similar order to mine - but presentation counts for a lot, and Jent's done a good job here. My goal is always for a train of thought to feel natural and obvious in retrospect, so if that's what happened here I'd be flattered.

[@PATRIK67KALLBACK]

Super great video, thanks!

[@parlancex]

Informative with nice animations, thanks for making this.

[@DokterKaj]

So glad I clicked on this video, beautifully made!

[@_engid]

This is really well done, I just feel like you could have given a quick recap of DFT, since you took a long time with the time domain explanation, but went super fast in the frequency domain explanation.

[@eqwerewrqwerqre]

This is amazing!

[@noharahien]

Very helpful. Thank you!

[@zorggn]

I wish i found this video sooner tbh, would have saved me some time, i believe. :3 After toiling for quite a bit of time trying to implement a realtime pitch shifter/time stretcher (hence i didn't want to touch fft or other things that would eat performance), i arrived at a solution where i only ever use two windows overlapping exactly 50%, but for each step to generate a new chunk, the windows are both resized to a size that's the chosen one plus/minus a small (uniformly or normally, i tried both) random deviation; Considering that a static window size was producing a very noticeable tone with my code with window sizes that were small enough to make the crossover points fast enough to go into audible pitch territory, the randomness introduced seemed to make that less noticeable quite nicely, basically akin to turning a pure tone into white noise. I still have issues with fluttering though, that i couldn't find time to implement a fix for yet.

[@bean_mhm]

Absolutely amazing

[@chadyonfire7878]

This is crazy love it

[@pmishraofficial]

This is really well explained!

[@seatyourself7082]

My Favorite SoMe

[@dittilio]

Excellent video. I really hope the phases didn't smother your interest in making more videos. Instant subscription.

[@desperatehouseflies]

Amazing vid, well done

[@baonguyen-ct6nj]

very well written analysis !!

[@magran17]

Wow, great work

[@sirati9770]

I am watching this video at about 4 Times speed. This actually becomes very important to me because these artifacts dictate how fast I can watch a video on a device (or in general)

[@juan23415]

great video!

[@1.4142]

I've noticed how much better chrome is than safari at 2x speed audio

[@Xzurii]

amazing content, 10/10 please continue

[@ollie-d]

Excellent video

[@mybestfriend13]

this is a very good video, thanks you

[@FiatMihiSecundumVerbum]

"It's full of big word" brother, most of the words you used flew right over my head I understood almost nothing

[@1495978707]

I’ve been wondering about this for a long time, but haven’t gotten around to reading a digital audio book to understand it

[@GeirThomassen]

Thank you! This is quality content, bravo! Subscribing and looking forward to more videos like this!

[@rs-met7170]

Very well done!

[@a.w.n.i.s]

i would not have the balls to do such a well made video at 1,7k followers

[@feelfree.1]

Well done 👍🏼😀

[@camerongrenier5405]

I studied physics for 3 years, and I’m currently studying computer science. That was wayyyyyyyy more complicated than I was expecting and a lot of it went over my head 💀

[@epicwhaleburger]

Great video!

[@kruvik]

I implemented a pitch shifter and time stretcher in 2018 or so and was surprised how much non-trivial math was involved in doing that...

[@RealNovgorod]

It's all about that phase, 'bout that phase.

[@lemonhscott7667]

Lovely video