Instance Segmentation using Mask-RCNN with PixelLib and Python

[@Deserthacker]

I'm gonna be one of those guys and say that you deserve WAY more subscribers than you currently have. Clear and concise with good info - thanks for putting this together :)

[@NicholasRenotte]

Ohhhh, thanks so much man! Going to have a custom version of it soon!

[@choffmeister]

Would love to see this done with a custom trained model!

[@NicholasRenotte]

You got it!

[@emmastone6496]

@@NicholasRenotte could you please do a video on faster rcnn??

[@RichardBronosky]

Same!

[@ATHARVA89]

@@NicholasRenotte Hi just wanted to know if you have made a video on custom training set, if you could share the link

[@brianm.4368]

Same!

[@Samuel-wl4fw]

Thanks for the tutorial. Am already using PixelLib a little but nice to get to know the library a little more thorough.

[@NicholasRenotte]

Awesome stuff @Samuel, definitely, will likely delve into it a whole lot more!

[@Samuel-wl4fw]

@@NicholasRenotte Btw will learn about reinforcement learning next semester, videos in this will also be appreciated :D

[@NicholasRenotte]

@@Samuel-wl4fw sweet! Awesome space to be playing in!

[@muditrustagi5775]

Beautiful implementation of masked RCNN

[@NicholasRenotte]

Thanks a ton @Mudit!

[@gustavojuantorena]

Another great video 👍👍 the model looks pretty fast!

[@NicholasRenotte]

Thanks so much @Gustavo, was actually suprised at how fast it was given how hardcore the model is!

[@sofienabidi6112]

Thank you for all your efforts, please make another video regarding labeling training and deploying custom dataset for instance segmentation using pixellib, thank you

[@katokyea12]

Awesome content :D currently learning about Object detection and segmentation, Learning about ResNets, MobileNet and the inner structure of the Mask-RCNN model you showed in the video. Thank you and look forward to more exciting application in the future.

[@NicholasRenotte]

So glad you enjoyed it @Jenz! Working on any interesting use cases?

[@nandankakadiya1494]

Another great video ..thanks please make full video on how to label image for segmentation and how to implement end to end.. thanks in advance 🙏

[@NicholasRenotte]

Sweet, was hoping someone would ask for it @Nandan!

[@nandankakadiya1494]

@@NicholasRenotte 🎉🎉

[@venkatesanr9455]

Thanks for your interesting and informative video.Kindly do custom semantic segmentation( based on our application )using this pixellib library that will be helpful.

[@NicholasRenotte]

Done and done, will start planning for it!

[@aboudezoa]

Great video ! Maybe next one on how to do custom dataset 👍🏻

[@NicholasRenotte]

You got it @thebigboss!

[@lillywinter7]

Awesome content and you're very good teacher. Can I suggest doing something like detecting missing things, for instance spotting the differences from two images. What I really want to see is a practical example like training a model for a completed item and recognize defects or missing things. Example would be like in a bottling company, it would spot missing or crooked bottle cap or labels. More power to you!

[@NicholasRenotte]

Ha, cool use case @Carlos. Would this be for defect detection?

[@lillywinter7]

@@NicholasRenotte Yes, you are absolutely correct. Always looking forward to your videos.

[@NicholasRenotte]

@@lillywinter7 might do something like that for a live stream! Been meaning to get back to it!

[@Lucasxdlol2]

@@NicholasRenotte I'm looking for this kind of video!!! You are amazing as always

[@davidbiggs8687]

A custom training video would be awesome!

[@solardad]

Excellent tutorial. I was able to go thru it with expected results and no errors. I am going to try and use segmentation to allow my outside robot to move about and stay on the grass areas and out of the flower beds. Don't know how to do that yet but this video gives me a good start. Maybe you could follow this video up with further analysing the segmentation data in order to follow one of the objects. I subscribed and rang the bell. Thanks

[@NicholasRenotte]

Sounds like an awesome use case @Solardad, I'm definitely going to try some real word implementations of ml on robotics soon!

[@solardad]

If I wanted to use a different model instead of coco is it as simple as downloading the model and changing the "load model" code line to point to the new model. I would like to be able to use a landscape model that would recognize grass, paths, flower beds etc to use with my outdoor robot

[@NicholasRenotte]

I believe it is @solardad, will be doing a deeper dive into it soon!

[@ebenezeragbozo]

this is epic! simple and neat! Thanks mate. just when I needed

[@sarahgarcia3063]

Hi ! Thanks for the tutoriel ! I am a begginer and I would like to know how could I save only the annotations mask without the background scene. Thanks you in advance ! :)

[@daljeetsinghranawat6359]

thank you so much for providing these tutorials..

[@Xirock-1220]

Great vid. Would love to see how you apply this for a custom trained model

[@NicholasRenotte]

On the cards, just working on dealing with memory limitations!

[@ledarius82]

Ive read everywhere that mask-rcnn fails for new versions of tensorflow and keras...so how come this pixellib is ok with tensorflow=2.4 if its based on the mask-rcnn

[@ledarius82]

Not only i read it, i actually experienced it. It throws multiple errors (mask-rcnn). Your code doesnt throw errors but at the end all the masks are randomly put anywhere on the image :S

[@sarahnawoya498]

Beautiful video, so how do i train the mask rcnn and pixel lib on custom dataset

[@AkaBhandari]

Hi Nicholas, Could you please do a complete tutorial on U-net semantic segmentation in pytorch?

[@lucasschimidt8338]

if u ever make a data science course, i would definiely buy

[@NicholasRenotte]

Woah, awesome. Stay tuned ;)

[@elle_ed]

how to do it with custom dataset, say we have starting point, width and height of object to be segmented, how'd you do it?

[@nazilaameli8639]

Hi Nicolas, I am almost new to image processing. I would like to use instance segmentation to segment three cervical vertebrae in medical images. I do not know where to start.

[@marekm7764]

PixelLib has not been update for over 2 years. Are there any other up to date libraries ready to use, like PixelLib?

[@suneelvarma4821]

Hey! I'm a video editor with computer science background. I'm here to understand whether it's possible to do subject segmentation in Video and save only subject footage, so that I can add green screen in the background and do my further edits. Ik the softwares do provide this but I'm just looking for cheaper alternative,. : ) Thanks

[@anziledge326]

Can you provide a code for automatic image labelling for training purpose. Like iOS 16 photos feature where we can just hold on any object and it will extract without any other backgrounds

[@mmshafique8491]

Hi sir, would be able to get inference on custom trained models of other networks like deeplab, SegNEt etc?? thanks

[@sreemath8526]

Well explained !!!!!!! Could you please show how to calculate AP and mAP, precision recall f1 score for this particular video instance segmentation...please update as soon as possible

[@phaminhhoang4478]

do you use mask r cnn to predict real world object size ?

[@DDDDD723]

Great lesson as always!

[@NicholasRenotte]

Thanks a bunch!!

[@timlin6571]

tried segmentFlame and segmentImage today. I think segmentImage can't work with real-time capture, it'll show something about numpy error bla bla. Then I find out, if I try to read a image file directly, I have to use segmentImage. If I read a file through cv2 (it has become a numpy array), I have to use segmentFrame. So cv2.VideoCapture(0) can only work under segmentFrame...Is it normal that the fps is quite slow when using real-time Mask-RCNN?

[@timlin6571]

and is it possible to convert the h5 weights of pixellib to pb file?

[@NicholasRenotte]

Yep, it's not the fastest as it's compute heavy model. pb file format for Tensorflow? Could just load the h5 into keras!

[@abu8422]

thank you share the video Nicholas Renotte , I have a question how to find the object`s center point and the farthest distance from the center point and can you make a video .

[@chukwujikeezemandujr.3912]

@nicholas renotte does Segment_image.load_model only work for mask rcnn coco?

[@limweilee6203]

Hi Nicholas, may I ask how to extract the segmented part and save it into a video?

[@vehinvelateker8078]

Thanks for beatiful Content ! Big fan your channel .

[@NicholasRenotte]

Thanks so much, glad you enjoyed it!

[@drjaikaal194]

Wanted a deep sort implemention please make a video on deep sort also

[@NicholasRenotte]

You got it!

[@pranavikulkarni215]

heyya, the web cam popup did not open when i used colab. what can be done?

[@ethiotech8970]

Bro I am working on a thesis to improve mask rcnn for instance segmentation. Can you give me some insight how to improve it?.

[@manisadeep2809]

I want to know why I am getting error as mediapipe.python.solutions has no attribute 'selfie_segmentation' . Can u pls help me

[@NicholasRenotte]

Might need to upgrade as it's only been added recently @Manisa!

[@eranfeit]

Hi, Did you discover how can I get the list of classes / objects that can be detected using this model ? Thank Eran

[@NicholasRenotte]

It's trained on COCO, check out the classes here: gist.github.com/AruniRC/7b3dadd004da04c80198557db5da4bda

[@prashanthkatakam6336]

it would be great to see a full project on custom data like you always do and deploy it on android phone using TensorFlow lite.

[@jacklee4096]

Great Video. I'm having trouble installing tensorflow and tensorflow-gpu. Please help me out with that. Thanks a lot

[@Penta_Penguin_237]

They are deprecated now exists only tensorflow.

[@KrishPatel-pf6ir]

how to do object detection if i already have image segmentation done and i have the model done for that ?

[@leopaulose9856]

bro i need ur help how to crop the face and its context(hair, neck ,ear)

[@pankajray5939]

Your video are osum....Please upload Yolov4 with deep sort...

[@NicholasRenotte]

Thanks so much @Pankaj, will take a look into it!

[@pankajray5939]

@@NicholasRenotte Thanks

[@manpreetkaur-pz7eg]

Very helpful video. How to create label for my data and train it? Please make video on segmentation with polygon labels

[@NicholasRenotte]

Ya, definitely, a vid on custom segmentation coming!

[@lanzvincentvencer3458]

Can you do this on custom synthetic data?

[@vishvajitviswanathan7970]

Is custom trained model code ready!?

[@kassemanis]

Hello, i have an intresting question, i would like to use the segmentation mask to do operations on the image partially (i mean only on the segmented area) for eg, apply some filters or feed it to another NN model. Does anyone have a clue ? "i know i should ask this in stackoverflow ahahah anyway"

[@rishabhgaur4552]

can this be used to segment lines from a printed document?

[@Dana130699]

I see Nicholas, I press like

[@NicholasRenotte]

Ayyyyyeee, thanks so much @Dana Andreea!

[@hellsing1324]

Is there a way to export the mask as an image file or shape file?

[@mikositumorang8680]

Thanks for sharing Nic. but i have some problem while Running !pip install tensorflow==2.4.1 tensorflow.gpu==2.4.1 pixellib opencv-python ERROR: astroid 2.2.5 requires typed-ast>=1.3.0; implementation_name == "cpython", which is not installed. ERROR: tf-models-official 2.5.0 has requirement tensorflow>=2.5.0, but you'll have tensorflow 2.4.1 which is incompatible. ERROR: lvis 0.5.3 has requirement matplotlib>=3.1.1, but you'll have matplotlib 3.1.0 which is incompatible. ERROR: apache-beam 2.30.0 has requirement avro-python3!=1.9.2,=1.8.1, but you'll have avro-python3 1.10.2 which is incompatible. ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'c:\\programdata\\anaconda3\\lib\\site-packages\ umpy\\compat\\py3k.py' Consider using the `--user` option or check the permissions. Do you have any idea with that issue ?. thanks

[@NicholasRenotte]

Heya @Miko, try adding --user to the end of your pip install line and running again!

[@mikositumorang8680]

@@NicholasRenotte Ok, Thanks Nic. I'll try that one

[@Penta_Penguin_237]

@@NicholasRenotte Man it would be great to have an update: ModuleNotFoundError Traceback (most recent call last) in () 7 get_ipython().system('pip install tensorflow==2.4.1 tensorflow-gpu==2.4.1 pixellib opencv-python --user') 8 ----> 9 import pixellib 10 from pixellib.instance import instance_segmentation 11 import cv2 ModuleNotFoundError: No module named 'pixellib' --------------------------------------------------------------------------- NOTE: If your import is failing due to a missing package, you can manually install dependencies using either !pip or !apt.

[@aruha2847]

Hello, I have the original and mask images like the data science bowl dataset. How to convert the dataset for pixellib without annotating so many images?

[@NicholasRenotte]

Working on a custom tutorial atm! Will shoot you a link when it's out!

[@aruha2847]

@@NicholasRenotte That's great news! Thanks!

[@ayarzuki]

Is this you are using your PC? the FPS is low

[@NicholasRenotte]

Yep, it's a pretty hardcore model!

[@sree7310]

My frame is not responding. How to correct that?

[@NicholasRenotte]

Heya @Sree, you might need to stop the notebook and restart it again if it's completely frozen up.

[@sree7310]

@@NicholasRenotte hey, thanks for the reply. I tried that so many times but it doesn't work and also it shows kernal appears to have died

[@NicholasRenotte]

@@sree7310 hmmm, not too sure man. Check to see if you're getting errors in the terminal?

[@Ajinomoto324]

Please provide the custom one too

[@NicholasRenotte]

You got it!

[@kanaktekwani2517]

I am not able to find my current working directory, where should i download the coco.h5 file??🙄

[@kanaktekwani2517]

@nicholas

[@NicholasRenotte]

In the same folder that your notebook is in :)

[@kanaktekwani2517]

@@NicholasRenotte i am doing in google colab

[@NicholasRenotte]

@@kanaktekwani2517 drop it in the main folder

[@jacsonferneycardenastorres9029]

Hi. Could you do it in google colab?, Thanks

[@NicholasRenotte]

Yup!

[@noemi5450]

Hi Nicholas, How can I run this on a CPU?

[@NicholasRenotte]

Install tf without GPU support, should be fine!

[@XX-vu5jo]

I want to modify the backbones or architectures :\ this is just using the already made ones

[@NicholasRenotte]

Ya, these are using pre-built architectures. You could make your own using Tensorflow or PyTorch. Can also fine tune these models however!

[@chukwujikeezemandujr.3912]

how can you extract the segmentations?

[@eranfeit]

very nice

[@vaishnavimore754]

Sir... Can you please explain this line... one more time... image = res[1]

[@NicholasRenotte]

The second index in the results variable contains the image :)

[@mohammedzain9876]

How to train a custom model Mask-RCNN for detecting car damages and deploy it

[@javlontursunov6527]

Bro please make this vidoe using Pycharm as well

[@peanuts675]

It's time for lane detection

[@NicholasRenotte]

YESSS!

[@dorrakadri1474]

I want to do the same but on pictures

[@PUBUDUCG]

Looking for this . . . . .

[@bellemarravelo6002]

i dont think my pc can handle that, pretty lag with your own pc high specs, how much more if I try my own pc with low specs

[@NicholasRenotte]

It'll work it'll just be a little slower. Could also do it on images @Bellemar Ravelo?

[@raypankaj4]

Yes me 2 wants to learn about Yolov4 with deep sort.

[@NicholasRenotte]

Woah, seems like deep sort is popular atm, will take a look!

[@starlife2781]

Make a video on Yolov4 with deep sort.

[@NicholasRenotte]

Definitely, will add it to the list!

[@architech5940]

I was excited for this module, but this module is absolutely butt. The model was 100% certain that I am a toothbrush and didn't segment any object in the frame correctly.

[@chmod-o]

please remake this tutorial on custom dataset

[@shivanshjayara6372]

Amazing video. I have one query, i got a task where i have given one single image and in that images there objects of different shape and they asked me to separate that..i am not getting it how to Separate it...can u refer me some video of ur where u have taught the thing like this... without using webcam and all because i have already that image. I would be happy if you can help me out.

[@sonas822]

Hi everyone, I am getting this error. File "C:\Users\student\PycharmProjects\errorsolve\fifth.py", line 9, in segmentation_model.segmentImage(".Cross/TrainingImg/semtry.jpg", ".Cross/TrainingImg/") File "C:\Users\student\errorsolve\lib\site-packages\pixellib\instance\__init__.py", line 191, in segmentImage new_img = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) cv2.error: OpenCV(4.6.0) D:\a\opencv-python\opencv-python\opencv\modules\imgproc\src\color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cv::cvtColor' How to resolve this?

[@redabenlekehal3273]

Still the best

[@deepvachhani3968]

great video bro but i am not able to use my laptop gpu to run this codded can you please help me with that if torch.cuda.is_available(): device = torch.device("cuda") print("GPU is available. Using GPU:", torch.cuda.get_device_name(0)) else: device = torch.device("cpu") print("GPU is not available. Using CPU.") cap = cv2.VideoCapture(0) while cap.isOpened(): ret, frame = cap.read() segment_model = segment.segmentFrame(frame, show_bboxes=True) video = segment_model[1] cv2.imshow('segmented', video) if cv2.waitKey(1) == ord('q'): break cap.release() cv2.destroyAllWindows() i tried this but my nvidea gpu is not running still the code is running on cpu and its gliching

[@chiragmalaviya2430]

Plz me tutorial on GANs and StyleGans

[@NicholasRenotte]

Definitely, check this out: kzbin.info/www/bejne/mHfInaeNq9-NprM

[@bastianstrahm6905]

Please make a reinforcment learning rockez league bot I would be so happy if I could see how you write your code and solve the problems and maybe after this im able to finish my own😂

[@NicholasRenotte]

Ya, will definitely dig into it. Will be interesting to see how it actually performs 😂

[@bastianstrahm6905]

@@NicholasRenotte Thx :)

[@ashleysami1640]

👌👌

[@NicholasRenotte]

Ayyyyeee!

[@vikashchand.]

🙌🙌

[@NicholasRenotte]

Ayyeeee!! What's been happening Vik?!

[@vikashchand.]

Let just say work from home is a killer man lol Would be cool to be able to apply instance segmentation on custom or single models though aye 🐱‍👤🔥🔥 lol ..what say?

[@NicholasRenotte]

@@vikashchand. oh you knoooowww it! Definitely coming in the pipeline!

[@rofiatulkhoiriyah9997]

please give me a tutorial using google colab

[@peanuts675]

Lord bless you 🙏

[@ishabhthakur5887]

how do you know hindi,indian words i am so surprised

[@NicholasRenotte]

So many of my friends and family speak Hindi! 😊

[@ashiqueraju1330]

how to segment human hair through this

[@walidelloumi2722]

Dear Nicholas, you are fantastic very very interesting!!! Please when i open your code with visual studio (i create virtual environnment after i use pip install jupyter and finally i use jupyter notebook) when i execute !pip install tensorflow==2.4.1 tensorflow-gpu==2.4.1 i have this error ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'C:\\Users\\Admin\\Desktop\\pip\\Opencv\\Walid_Elloumi\\PixelLibInstanceSegmentation-main\\PixelLibInstanceSegmentation-main\\venv\\Lib\\site-packages\\tensorflow\\include\\external\\com_github_grpc_grpc\\src\\core\\ext\\filters\\client_channel\\lb_policy\\grpclb\\client_load_reporting_filter.h'

[@joseluisnm1]

I also tried to use pixellib on virtual env but it didnt workrd (my error was on the import). the solution was not to use virtual env.

[@NicholasRenotte]

Heya @Walid, if running in an IDE, try installing the libraries separately at the command line.

[@nikleshkumar000]

Hi@UCHXa4OpASJEwrHrLeIzw7Yg, a fan of you and your contents, Need you to give it a shot on Multi Video stitching and Detection using Opencv and Mask-R-CNN.

[@ATHARVA89]

its pure disaster if you run it on CPU!!!😅😅

[@jacsonferneycardenastorres9029]

crack!

[@NicholasRenotte]

Ayyyyee thanks @Jacson!

[@kevinvigi9791]

Hi could you help with CUDNN please? I tried it but it shows open ssl error