Motion Detection Algorithm Edits my Videos

Here's my naive attempt at building a motion detection algorithm that edits my videos.
AlgoExpert (30% off): www.algoexpert...
Thanks to AlgoExpert for sponsoring this video. Practice your technical interview skills and land your dream job (while supporting my channel!)
This is my first time doing image/video processing pixel by pixel. I'm sure motion/difference algorithms like this already exist but I figured it would be a great learning experience to implement one myself and integrate it with my video editing automation. So far this project does not use any machine learning or AI. There's still a lot we can do without it (and it might even be more efficient!). This is the second video in what may turn into a series. The core video automation functionality is built on top of FFmpeg and was built up in the last video. Now we can start designing more intelligent algorithms that edit videos for us :))
If you have no idea what's going on watch my last video: • Writing a Program that...
If you still have no idea what's going on, comment "I have no idea what's going on".
Thanks,
Devon
Social
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Website: devoncrawford.io
Twitter: / devoncrawford13
Snapchat: / devoncrawfordyt
Instagram: / devoncrawford_
Github: github.com/dev...
Discord: / discord
Gear
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Video Editing Software (Premiere Pro CC): prf.hn/l/BOomWo3
All Adobe Apps (I use this plan): prf.hn/l/Xv5qk5Q
Keyboard (Velocifire TKL01): amzn.to/2AQTjJQ
Monitors (Dell U2518D) : amzn.to/2AQQmco
Webcam (Logitech C920): amzn.to/2Cw588c
Desk Microphone (Blue Yeti): amzn.to/2FPFmzi
DSLR (Canon T7i): amzn.to/2T54mpZ
Wide Lens (Tokina 11-16mm F2.8): amzn.to/2MlpvcO
Music
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
e e v e e - lost found [2QQ3] Rare
Link: / lost-found-2001-rare
kloudbug - M&M's FOR BREAKFAST w/ Inteus
Link: / mms-for-breakfast-w-in...
crime tech. - GELATO
Link: / gelato
uncomfy - awake.
Link: / awake-2

[@gardnmi]

10 Episodes later...I've created this neural network to find the best frames.

[@tatzecom]

my thoughts man like, "so i rented this neural network thing and i dunno, i think i can do it better: We have this problem, and i just downloaded this library which enables us to do this and that"

[@borekworek69]

*next episode

[@2nafish117]

haha i just thought of recommending this paper to him medium.com/@thatipellianirudh/two-stream-convolutional-networks-for-action-recognition-in-videos-ffb924365658

[@natin9133]

he should use a neural network to then cut out ugly scenes after this one cut out the boring ones

[@HellsMascot]

If he keeps developing this idea I honestly think that's where it's headed

[@nabeeghahmed8530]

Programmer: Someone who solves a problem you didn't know you had in a way you do not understand.

[@PixemWeb]

That's one of my favorite programmer quotes. Cool phrase for a T Shirt.

[@srreal4821]

"When u go to the hospital to get tested and walk out diagnosed with a bunch of conditions"

[@nabeeghahmed8530]

@@PixemWeb interesting!!

[@okie9025]

Java Programmer* Seems like a better fit If Java was a tool (specifically a hammer), then every problem starts to look like a nail

[@Adrian_Galilea]

Oh that's why there is a sock on the mic

[@sebastianerivesmejia4474]

50% Google researching 45% Actual coding 5% Socializing A programmer’s life

[@blackwhattack]

Those are rookie numbers. Look up the 80/20 rule. 80% googling, 20% coding is the perfect combo. No time to socialize (wo)man.

[@imdone8243]

Socializing is not there.. haha..

[@RawPeds]

yeah, 10% socializing looks pretty optimistic

[@jongxina3595]

Should probably remove that 10% Socializing.

[@kaiherron1906]

As an actual programmer, I can confirm this but socializing is never an option

[@CarsOverdose]

Plot twist. This video was edited with the beta version.

[@degamersspelen6468]

how does this have no reactions, it has 87 likes so like what?

[@empoliyis1127]

@@degamersspelen6468 amazing 1 hour ago comment

[@reviveproject]

🤣🤣🤣🤣🤣🤣

[@Jejkobbb]

KZbin needs more channels like this. Absolutely love watching someone code a project. Super interesting.

[@stoic_sheep]

Will you support me if i made videos about my personal code projects?

[@fero5070]

Never clicked on a video faster, notification squad

[@reyariass]

Same!

[@TheoParis]

Fero same

[@jesusalejandrogarridocorde6687]

Here!!!!!!!!!!

[@dreammix9430]

Same

[@rodrickee]

Next video: writing an algorithm that writes my algorithms. Awesome video, Devon! Keep going. Edit: Shieeet dawg, thanks for the 260 likes in one day.

[@purnya2]

machine learning?

[@techpunk007]

👩‍💻 😂😂😂😂😂😂😂😂😂

[@ashiquemushtaq3333]

That's how mafia work

[@TheoParis]

lol

[@alfindelpiero7991]

that's.. that's actually possible if he has a something that can recieve brain neural signal to computer. :o

[@tompeters4234]

"It's just a lot of work, but it's fun." I loved that perspective!!

[@pesterenan]

Hi Devon, amazing video as always. Let me pitch my idea. Now that you're able to detect scene changes, you can also make a crude "face detection" so, if there's a face talking in the frames, the algorithm can ignore the cutting process, so the person is never cut out of the editing while talking. Keep up the good work, you inspire us with the amount of study you do to make these projects.

[@vincentbrandon7236]

I think he is passing pixel values as integers to the CPU in this video. If he goes the feature detection route, he'll want to use subsampling to pass less than the whole image to the GPU and run a CNN trained on his face and room to detect (a) when he's in the camera and (b) when something not in the normal room is in the camera. Super fast, can be done in greater than real time for shot video on any CUDA gpu.

[@shivampoke]

For face detection you need to preprocess each frame. it's not always of same color or something to match the face. You need to apply a filter and transformation to make that in a template and then use a template matching algo for the face. This is a crude method for face detection. These days CNN method is more reliable and common

[@perschrijver8884]

I think some kind of sound detection would also come in handy. I'm not sure how difficult it is to check for speaking and such but it could definitely be beneficial.

[@fellowhuman3541]

That's a really good idea. The majority of the footage he records is not him talking even when his face is in the frame, so if he does go this route, he would have to use face detection software in addition to what Per said about listening for sound to determine if he starts talking. But either way it would definitely be a nice feature to add.

[@FryuniGamer]

WTF Pesterenan você está em todo lugar.

[@Lina-ej1bt]

I know absolutely nothing about programming, yet I watched this whole thing with out skipping a second... what you do is really interesting

[@GabrielPohl]

True thing, watched everything while taking a shit

[@Cadenaz]

@@GabrielPohl HAHAHAHA SAVAGE!

[@hellosandile]

The only KZbin notification on my phone that matters

[@swahagaming5261]

SAME

[@Strandinator243]

"I just... 2 hours... an hour and a half of sitting here. And then we get a segfault" oof... too real

[@tomaskermes]

Auto editing of videos is so dope. Continue pls :D

[@vishalsharma5463]

Devon can you share a roadmap of how you learned all these skill and what resources do you use to learn ?? Please it will be a great help

[@mehedihasanhridoy1701]

VISHAL SHARMA yes. We want it

[@AcunPearce]

Yes, we want it

[@JoshSmeda]

Computer science and stack overflow. Good luck

[@vishalsharma5463]

@@JoshSmeda stack overflow is more like a guide or a reference , not an actual learning resource.

[@devolutheist5136]

Solve actual problems. When you do that you actually learn stuff.

[@lxlyzd]

Finally an upload! Nice job on the project. One idea I can suggest if I may is to blur the frames before calculating the difference using a 15x15 Gaussian filter. That would reduce the sensibility to noise and other elements entering the frame. You can also skip pixels (I tried to skip 2 or 3 pixels on a project and worked pretty fine) while looping or calculate the average in a 5x5 window that slides on the frame. Also, running the process on a GPU is much faster, someone in the comments suggested CUDA as an option.

[@frisosmit8920]

Those are great suggestions!

[@gregorymalchuk272]

What programming language is he using to do this project?

[@vexraill]

Anyone else think Devon should do some live streaming when he is building this stuff? I'd love to watch the process real time.

[@cinneylun5353]

vexraill omg yes!!!

[@Pyritenite]

Yeah!

[@Jan-ux7xp]

Plot twist: this video got edited by his software.

[@Reaver.]

That is not a plot twist at all.

[@Jan-ux7xp]

@@Reaver. u got the joke bro

[@pikuhana]

I think it was, if you notice towards the end his voice was cut off a little bit, he didnt finish what he was saying, so he probably hand picked the best clips and the rest was done by the software Example here: 9:31

[@Derienstephens]

As a newer Dev I have to say bravo, Devon. I don't know where/how you find the time to create amazing projects like this and shoot/edit such dope & useful content but I'm glad that you do. You're vids have inspired me to spend more time on my side projects and continue pushing myself! Keep it up, man and much respect from Detroit!

[@eL3ctric]

DONT Stop working on this! And people are joking about it, but definitely think about applying YOLO f.e. to the video for smart cutting

[@ROGUEyoutube]

I can't tell if you are for or against him continuing the series. By putting an exclamation mark after the "Don't" it seems like you are responding to his last question, which was "I'm not sure if I should continue working on this". Following that, you commented "Stop working on this". Not sure if that was intentional. If you are against the series then keep your comment as is. However, if you are for it then I would suggest replacing it with a bolded or capitalized "Don't" (e.g. *DON'T* stop working on this). To do this you can bold words by encasing them in single asterisks.

[@eL3ctric]

@@ROGUEyoutube oh yea you are right!

[@theyoutubeaccount8499]

@@eL3ctric damn admitted your mistake, respect. btw, do you know anything about what happened to devon?

[@eL3ctric]

@@theyoutubeaccount8499 Why wouldn't I? No, but according to reddit he is alive. I guess he found something even better than youtube.

[@ryan.connaughton]

@@theyoutubeaccount8499 He's working on 'something big' accoording to his Discord channel (and KZbin slows him down). Makes sense - but we want you back Devon!

[@ToDopeToFly]

Ok ok, I'm subscribing! I'm not waiting for your vids to pop up in my recommends anymore.

[@MalickLombion]

just did the same

[@admok]

I say keep working on this. It's super interesting how the edits actually look good :) Good job!

[@thev01d12]

Such cozy videos man i always come back to these.

[@ii7mdj_353]

*I’m very impressed by your program, Devon please continue these great videos.* *I’m a Grade 10 CS stundent*

[@kajij]

The first time that you not only explain what you are doing ( or want to do, but also how you are doing it ). Nice.

[@ba-qb9wo]

As a CS student, that reaction to the segmentation fault spoke to my soul ahaha 😂

[@Wereld03]

I dont need to study cs for that :)

[@SubterraneanChick]

Can't get enough of these videos of Devon creating and learning. It is a welcomed intellectual challenge in this baron scape that is YT Wix commercials.

[@curtisbridges]

Rather than scanning every frame, scan every 10 (or better, match your frame rate). When you detect a scene change, back-scan to the first frame of the change. This should improve your speed 10-60x.

[@techtonight882]

Oh my God, I have loved these two videos. Please make a series.

[@KyuubiRNG]

One of my most favorite projects you've done. Keep working on it, looks great.

[@philippelamy1796]

Very inspiring Devon!! I've been following you for a while and you really inspire me to better my skills in programming and hope to have the knowledge you have one day! Keep on doing some good stuff my man!

[@brakenthemole2377]

I see great potential for multi-threading

[@tatzecom]

My man Devon needs some serious upgrade, i mean, a 4C/4T or 2C/4T CPU aint nuff for that kinda stuff... He could cut some serious time with like an 8C/16T one and cut that 2.5h down to like 1 or somethin

[@frisosmit8920]

Gpu would be much quicker. It can basically do every pixel in its own thread

[@brakenthemole2377]

@@frisosmit8920 Oh hell yeah

[@tatzecom]

@@frisosmit8920 ye but I recon GPU implementation is a bitch... Also, what to use? CUDA? OpenCL? There are tons of options with their respective pros and cons and what not

[@brakenthemole2377]

@@tatzecom That wouldn't mean much unless he actually codes the thing to utilize the cores/threads. Better yet, the community could gather around and help our man Devon out with this

[@jandrew2514]

I would absolutely love a full series from this, i think it is very interesting and it has a lot of potential, cant wait for the day when you can just upload all your raw data and get a full edit without doing anything lol good luck

[@raintzrandmaa9829]

Devon, it's time, to acquire the power of neural networks

[@prateekmanta2097]

Bro what you made is awesome , seriously , the video edited by the algorithm is awesome

[@jcpluto4555]

Been my favorite KZbinr for a while. Keep up the good content bro

[@DevonCrawford]

Thanks man

[@neomage2021]

It's awesome that you are making these videos. Reminds me a lot of myself 10-12 years ago. It's great watching your journey of discovery as a programmer.

[@sidteche]

this program is good for creating a summary of vlogs (like an intro)

[@skizzz3227]

I love watching these videos because you have some really cool ideas I have never really thought of motion detection and how to make a program detect it but the RGB really opened my eyes to cool ways to do it! Keep it up!

[@TheoParis]

I was waiting for this video for a while and now it's here... EPIC VIDEO!!!!

[@powerverwirrt]

I don't understand most of this (at least in terms of coding) but damn, this is still sooo interesting. You're inspiring, dude. Keep thinking out of the box and please continue this as a series.

[@NitramiuZ]

You should make it so it don't cut in the middle of speech, Using audio detection.

[@reyisw]

Auto editing is so cool!! Please continue with this project!

[@nnaaaaaa]

I doubt you'll ever see this but... some suggestions on the implementation, lossy video compression is based on deltas, you may be able to make a super lean version that doesn't rasterize the bitstream and just detects how fast different parts of the scene is by looking at the magnitudes of the delta vectors, sum them up by region and then if enough regions are moving fast enough do a cut.

[@ideiasBrasil2023]

I've been programming for 5 year and and only few times I got so impressed as I am now for seeing your automatic editor. Liked a lot... You Got a huge potential dude. Keep up the hard work.

[@MeeZoid]

Amazing

[@brownyeet3981]

lol

[@bacho0218]

watchu doin here invis man

[@Thiago-jm7gr]

A brazilian guy called Filipi Deschamps has coded 4 robots that do web search, download and edit images, put them into a video with animations, render it and upload it to youtube with title, description, tags and so on... its amazing! The video is "Programei 4 robôs que criam vídeos para mim no KZbin"

[@YoriichiHeeheeRunFoyolife3334]

Dude for a future software engineer this is cool

[@YoriichiHeeheeRunFoyolife3334]

Thanks Guys

[@blehmeh8909]

Start now

[@arnomaddelein8369]

This is such a mindblowing project, I love the progress!

[@TheChangeYT]

*let the algorithm automatically sync up the cuts to the beat of the music*

[@wajeehdaouk1424]

the fact that u did in no less of a week a better editor/security cam is actually mind blowing considering how multiple security companies are failing to do so

[@Yazaar_]

2 months later - Brad 1 and Brad 2 lost their editing position for PewDiePie *AI takes over*

[@imdone8243]

Brad Alpha Omega, Enters the KZbin scene!

[@imdone8243]

Update: RETREAT RETREAT, The AI alpha Omega has taken over control of everything and making it completely organized. I CAN'T FIND MY EYES NOR MY DAUGHTER, GODA HELP US

[@qwerasdfhjkio]

Brad 1 actually left loll

[@jesperj86]

I love this! What you can add, when the frames don't change much for a long period. speed them up like you do while reading/coding (between 1:45 and 2:15 for example). show only 1 in 5, 10, 24/30 frames depending on the total length of the clip

[@Riiisuu]

Next episode. I create an algorithm to live my life

[@joelhayes9477]

This is so great, I was literally thinking about your editing project yesterday, and thought I wonder if you could tell the difference between frames to pick more interesting shots.

[@joelhayes9477]

So there are some libraries for face detection, and object detection. I wonder if you could run it on the footage, and get the timestamp when it's sure it recognizes an object or face, so then it would cut out a lot of shaky transitions (would also maybe cut out some shots that you want though like driving, but would mostly work).

[@ArkadioBRO]

why dont try implement cut any time the audio is silent and there isn't a scene change ??

[@JSheetzArtwork]

You could do so but you would have to select audio based on the volume or db of the audio. For example the issue would be that any background noises, coughs or other sounds would still be in the video. Thus you would have to edit by db volume instead of only esiting out silence. But you would have to be very close to mic so that the gain of your voice is much higher than anything else so it doesnt pick upbthe other sounds and only voice but then you run into problem of when you are at end of saying a word it will cut off part of the audio/video becauase the end of words tend to be slighly lower volume than beginning of words at times.

[@tortosvideos]

Solid idea! I'd probably just have it wait for new audio above a certain volume (as J.Sheets) suggested just in case there's a short pause between the dialogue.

[@JustinDuijn]

Can't wait for the first automated upload. I got something as well but it involves music and a template, which is way easier than what your doing here. Damn bro keep it up!!

[@whereismo99]

An algorithm that picks a frame based on the difference in pixels only won't get the required results. While editing a video there maybe a bunch of frames with no difference but actually can't be cut out of editing because the sound in them is different, therefore an editing algorithm can't just depend on visual parameters but on sound too. Building a machine learning model were the data set contains raw video footage and the the edited version would really make excellent results. It may be hard and challenging but totally doable.

[@DevonCrawford]

Yeah I haven't even touched sound yet. That's on my todo

[@bagel7860]

If you ever come back to KZbin, this would be an awesome project to apply machine learning!

[@millennialhardware]

You should build a program that automatically edits videos and become rich :)

[@raintzrandmaa9829]

That's what he is trying to do

[@incognito4746]

Yes or a program who edit vidéo off hours of surveillance camera and sell them your program

[@trym309]

@@incognito4746 that alredy exists.

[@incognito4746]

@@trym309 ohh okay name?

[@Dizzle2myNizzle]

@@incognito4746 spdload.com/blog/motion-detection-software/ there is plenty of it

[@ganon4]

Most of times is the first time you do something but what I like the most is the process and the way you do every step. It's like being in your mind and understanding what makes you choose this way. A big thank you for sharing I really enjoy everytime I see one of your videos.

[@bollingbollz9379]

You should write a new KZbin algorithm that detects *actual* inappropriate content.

[@kunalshah1002]

Watching this video and already freaking inspired. Kudos devon!

[@rotters2556]

It would require a lot of work, but integrating artificial intelligence (neural network deep learning) would bring your project to the next level. Check out the KZbin channel Carykh, he makes fascinating AI projects.

[@javierflores09]

A deep learning AI would need a huge dataset, not something you can really do with YT videos of a single channel. Better of going with simple machine learning first.

[@rotters2556]

@@javierflores09 True I didn't think about that

[@FarMoreAccurate]

I love this series and the in depth details you go into explaining the whole process to get your program completely working. I feel like a lot of more kickstarters should provide this kind of content to their endorsers...

[@dhavalkotak1751]

Likebutton.setEnable(true); 😇😇👌

[@Tullymastully]

Has Google hired you yet?

[@dhavalkotak1751]

@@Tullymastully i sent my resume 😂

[@dhavalkotak1751]

@Theikun he already knows I guess

[@ShivamJha00]

STFU

[@dhavalkotak1751]

@@ShivamJha00 lol 😂

[@jackzaic230]

When you’re over here struggling with your “Programming: Java I” class assignments and hoping you can become this cool someday

[@OndrejMusicIsMine]

I wasn't even sub but every time I got to the YoutTube I had that thought at the back of my head like "I wish that auto editing guy posted a followup on the last vid..". Nice vid, good edit, great program and gz on sponsor. Looking forward to what you come up with next.

[@KasonDS92]

Devon, dude this series is so neat. Please make more man!

[@loading685]

I didn't understand any of the bits in these two videos the first time I watched it a year ago when i was 13 Now I'm 15 and I'm working on a personal project that I came across media file formats, I saw this video again, and holy shit I finally understand what he is doing, this made me more into computer science, even if I just got into high school!

[@BearFromSteam]

In our company we are creating an app for videocameras now and this algorithm can be helpful for some basic motion detection :) Nice one Devon!

[@prithgg]

This is really awesome! Obviously need more of this series.

[@TheShinx317]

I love this kind of content so much. It's really inspires to get off my lazy ass and build something so cool like this!

[@umadbroyo2388]

Dude this series has me hooked!!

[@fatalsg92]

You really inspire me, man. I wish I could find a project that I would be so hooked on. I miss those days.

[@pawedawiec8562]

Devon, your consistency in execution is so impressive. Can’t wait to see how this algorithm evolves. Amazing work!

[@boris.karavasilev]

I can definitely see this as a highly demanded tool for daily vloggers! You should continue working on this! 😊

[@hemants1703]

He is the most curious guy I've ever seen my lifetime...! I just love his curiosity and the inspiration he emits me.! I hope he soon continues with KZbin.!

[@willemme758]

Love your stuff! You are one of the few channels that shows real interesting programming. You could advance your editing program by checking if you are speaking and only cut when you're done.

[@dariom8886]

These videos have the same effect on me as the ''study with me'' ones, always motivats me to start coding. Keep up the good work!

[@BrianGlaze]

I would love to see video editing automation as a series! I'm continually amazed at your work ethic.

[@Natei]

Im loving watching this progress!

[@SEWebDesign]

These projects are so cool. Awesome content man!

[@husseinalmosawi2001]

When I need inspiration and just a little bit entertainment I love coming back to this video!

[@Ash_Marshall]

I went to BETT trade show in London recently, and this kind of this is all they talked about, there were companies that had millions invested in them for this purpose and you're all up in here doing it be yourself in your bedroom. Amazing work man.

[@slimyslice]

bro you killing them videos as CS major you inspire me to do more on my side projects.

[@tahsinurrahman2170]

That urge to leave my degree program and start building random stuffs like Devon! It hits hard man. I've been watching his videos before I started my grad program. I'm re-watching all his videos again and none of them has made me think that I should stop watching them.

[@vincentcontrado899]

Definitely don’t give up you have serious potential with this program especially if you add voice recognition and could control sentence flow. You’re a genius man

[@igame1514]

This is a very fun series to see evolve haha

[@johnt1342]

An edited video on coding a utility on simplifying the process of editing videos about coding utilties. Nice

[@royilevy7322]

you should try making a program that uses the audio as a parameter. You'd most likely want the scenes with you talking to be in the video without cutting the scene while you are talking. you could do this by making the program compare your voice to the audio of the scene. if it detects your voice, it'll try to include the whole scene in which you are talking. You could do the opposite in the scenes you are not talking and add music. you give the program the song you want to add. it detects the scenes you are not talking in and adds the songs you gave it. this is like one of the coolest projects I've ever seen and I can't wait for the next part.

[@xxxvenmxxx2269]

I'm also a computer science student but I'm at beginner level and u always give me motivation to code more and more.One day I want to be a programmer like you🙂

[@Etredal]

Yes I want this to be a series! It's very interesting.

[@douguk2]

This is really cool. As and editor is also kinda scary, but still really cool.

[@Daco812]

This was a dope concept. It turned out pretty sick, too.

[@MuhammadAhmed-wh5kv]

Really, this is fun! Keep it up, man! Though the basic idea of the algorithm, just subtraction and average math, what it did was really amazing! The beauty is not to know a lot of crazy math, but the beauty is to know how to use it or apply it in real life! I enjoyed watching this project from the very beginning!

[@semiclean]

Yes Devon ! Keep working on this algorithm for scene detection ! Imagine the amount of time your will save yourself in the future if you spend a fraction of that time now !

[@HZLTV]

I have absolutely no idea what is happening in these videos but as someone who loves tech (just doesn’t understand it) this is honestly one of the most entertaining series I’ve seen haha

[@Maknopono]

Once again, sick amount of effort resulting in a sick video. Extremely well done.

[@pauldudley1273]

i love these videos because it really manages to capture the effort and time that goes into programming. Also interested to see more of this project.

[@aiwithrudiyt]

Best sales video for algo expert I have seen tbh 😂