Tune any musical instrument with ML5 and Crepe - Made with TensorFlow.js

Today on Made With TensorFlow.js we’re joined by Michelle Sun, an interaction designer, who solved a problem she had - never having a guitar tuner nearby when she needed one. Learn how Michelle created a system to tune any instrument (even your voice) live in the web browser using a pitch detection model known as Crepe without the need for any specialist hardware.
Learn more and try it out:
Guitar tuner → goo.gle/3dylLD3 ​
Snake game → goo.gle/3cMHMPk ​
Ukulele version → goo.gle/2R0emom ​
ML5 pitch detection → goo.gle/2OeIcEs ​
Crepe pitch tracker → goo.gle/2QX86xv​
Want to be on the show? Use #MadeWithTFJS to share your own creations on social media and we may feature you in our next show.
Catch more #MadeWithTFJS interviews → goo.gle/made-wi...
Subscribe to the TensorFlow channel → goo.gle/Tensor...

[@NicholasRenotte]

This is amazing @Michelle and a great, practical implementation of ml5!

[@zoeziyugao2079]

Like the color gradience! this so cute

[@ShahidulsPerspective]

🎯 Key Takeaways for quick navigation: 00:00 🎸 *Introduction and Background* - Michelle introduces herself, highlighting her UX background and current studies at ITP. - The problem she faced with finding a physical guitar tuner inspired her to create a machine learning solution using ML5.js. 01:22 🖥️ *Demo of the Tuner Interface* - Michelle demonstrates her guitar tuner interface using ML5.js and P5.js. - The interface maps guitar strings to visual elements, providing real-time feedback on pitch accuracy. - Visual design inspired by artist James Turrell, with a gradient indicating correct pitch. 03:51 🐍 *Bonus: Snake Game Integration* - Michelle showcases a snake game feature that activates when the strings are tuned correctly. - The game responds to the guitar strings, adding a playful element to the tuning experience. 04:18 🤖 *Model Used: Crepe under ML5* - Michelle explains her use of the pre-trained model Crepe from ML5 for pitch detection. - The Crepe model works across different instruments and even human voices, based on frequency analysis. 05:17 🌐 *Accessibility and Future Plans* - Discussion on the universality of the model across instruments and voices. - Michelle hints at possible future versions for accommodating different instruments. - Information on where viewers can try out the project on P5.js, shared in the video description. 06:11 💡 *Future Projects and Conclusion* - Michelle expresses her inspiration for light installations and plans to integrate machine learning. - Talks about returning to UX design and exploring more installation art before graduation. - The host acknowledges Michelle's contributions and expresses anticipation for her future projects. Made with HARPA AI

[@Jirayu.Kaewprateep]

📺💬 Colors and presence of the monitoring frequency. 🥺💬 That is nice work, I tell you the problem of song music recognition and she can answer how she can handle the problem. 🥺💬 For Guitar, it had multiple frequencies at once they can determine notes and rhythms but you need windows to the correct frequency, I see her playing means she understands and solved the problems. One note play can beat for 0.5 - 2 secs varies and continue playing the Guitar had some initial frequency that is mean it has their own rythms and she play well. 📺💬 For mine, it is about 6 frequency detection 📺💬 There always have somebody had to do it with 23 notes but they had a trick we can discuss it later. 🥺💬 That is because they are groups of notes the same as me I also cannot recognize those notes fast as they but they group and find some related frequency notes.

[@Rose-ng2zp]

Wait so she didn't find the Guitar tunar in play store?

[@JasonMayes]

Does that app allow you to play snake game too? By learning how to make it herself she can now do anything she wants. Knowledge is power. As she mentioned she wants to go on to go physical installations too using this tech. I'm pretty sure the free app in store doesn't support that.

[@LarryRiedel]

Perfect example of the "if all you have is a hammer" proverb

[@metroidandroid]

I don't understand why you'd use a deep learning model to solve such an easy dsp problem

[@JasonMayes]

As per the ML model's research paper: "The task of estimating the fundamental frequency of a monophonic sound recording, also known as pitch tracking, is fundamental to audio processing with multiple applications in speech processing and music information retrieval. To date, the best performing techniques, such as the pYIN algorithm, are based on a combination of DSP pipelines and heuristics. While such techniques perform very well on average, there remain many cases in which they fail to correctly estimate the pitch."