COMPLETE DETECTRON2 TUTORIAL | Instance Segmentation, Object Detection, Keypoints Detection and more

This is a complete detectron2 tutorial for setting up detectron2, running it on images and videos. Detectron2 is FacebookAI's framework for object detection, instance segmentation, and keypoints detection written in PyTorch. Detectron2 makes it convenient to run pre-trained models or train the models from scratch. In this Detectron2 tutorial, pre-trained Instance Segmentation, Object detection, keypoints detection, panoptic segmentation, and LVIS segmentation models are covered.
In detectron2, MaskRCNN is used for semantic segmentation while Faster RCNN is used for object detection. You do not need to download any pre-trained model. The detectron2 takes care of it automatically.
** Code is available for our Patreon Supporters**
www.patreon.com/TheCodingBug
---------------------------------------------
► Time Stamps:
Introduction: (0:00)
Setup Detectron2: (0:25)
Object Detection: (2:45)
Instance Segmentation: (6:55)
Keypoints Detection: (8:49)
LVIS Segmentation: (9:29)
Panoptic Segmentation: (11:02)
Detectron2 on Videos: (12:41)
---------------------------------------------
► Commands:
git clone github.com/facebookresearch/detectron2.git
pip install -e .
conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch
pip install opencv-python
---------------------------------------------
► Links:
github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md
github.com/facebookresearch/detectron2
---------------------------------------------
Want to discuss more?
►Join my discord: discord.gg/kUUvTQTTau
#TheCodingBug
---------------------------------------------
► My Other Tutorials:
○ Instance Segmentation as Rendering: kzbin.info/www/bejne/nZ-7nXhvd7RjotE
○ Detectron2 Complete Tutorial: kzbin.info/www/bejne/hpOWoKN7e7Vsarc
○ Colorize Black and White Images and Videos using Python OpenCV: kzbin.info/www/bejne/e4u6eXSaZa57Z68
○ Face Detection Using OpenCV Python with CUDA GPU Acceleration: kzbin.info/www/bejne/fYnGqmp6npair9U
○ Build and Install OpenCV 4.5.1 With CUDA GPU Support on Windows 10: kzbin.info/www/bejne/j6TQmX6Xp52ihcU
○ Install TensorFlow Under 90 Seconds
kzbin.info/www/bejne/qqCtlmuQl6ube8U
○ Install PyTorch Under 90 Seconds
kzbin.info/www/bejne/qJKlnJuLpMqCftU
○ YOLOv4 On Android Using TFLite: kzbin.info/www/bejne/j6ukm3SJa7SmbsU
○ Custom YOLOv4 Object Detection with TensorFlow and TFLite : kzbin.info/www/bejne/rKu3dH2DZp2opa8
○ Darknet YOLOv4 Custom Object Detection: Part 2 (Training YOLOv4 Darknet): kzbin.info/www/bejne/Y3-oc2iGYrh_n5I
○ Darknet YOLOv4 Custom Object Detection: Part 1 (Preparing Custom Dataset): kzbin.info/www/bejne/qXynqqaEqdObgJY
○ YOLOv4 Object Detection with TensorFlow, TFLite and TensorRT: kzbin.info/www/bejne/qnTQdGqkrst9ppo
○ Darknet YOLOv4 Object Detection for Windows 10 on Images, Videos, and Webcams: kzbin.info/www/bejne/fHaVeHWbgNqkpsU
○ Real-Time Object Detection on Webcam and Videos Using OpenCV With YOLOv3 and YOLOv4 | Windows Linux: kzbin.info/www/bejne/fJvcd2NpqsaqqaM
○ Build and Install OpenCV 4.4.0 With CUDA (GPU) Support on Windows 10: kzbin.info/www/bejne/qpu7nIpmYpmag6c
○ Install TensorFlow GPU and PyTorch with CUDA on Windows 10 Anaconda | CUDA 10.1 cuDNN 7.6: kzbin.info/www/bejne/hp26aq13nJmWrpo
○ Real-time Multiple Object Tracking with YOLOv4 TensorFlow and Deep Sort | Linux, Windows: kzbin.info/www/bejne/fZLKenxrhaaYgck
---------------------------------------------
► Follow us on Twitter: BugCodingThe
► Support us on Patreon: www.patreon.com/TheCodingBug
---------------------------------------------
DISCLAIMER: Links included in this description might be affiliate links. If you purchase a product or service with the links that I provide I may receive a small commission. There is no additional charge to you!

[@TheCodingBug]

► My Other Tutorials: Instance Segmentation as Rendering kzbin.info/www/bejne/nZ-7nXhvd7RjotE Colorize Black and White Images and Videos using Python OpenCV kzbin.info/www/bejne/e4u6eXSaZa57Z68 Build and Install OpenCV 4.5.1 With CUDA GPU Support on Windows 10 kzbin.info/www/bejne/j6TQmX6Xp52ihcU Face Detection Using OpenCV Python with CUDA GPU Acceleration kzbin.info/www/bejne/fYnGqmp6npair9U YOLOv4 On Android Using TFLite kzbin.info/www/bejne/j6ukm3SJa7SmbsU Install TensorFlow Under 90 Seconds kzbin.info/www/bejne/qqCtlmuQl6ube8U Custom YOLOv4 Object Detection with TensorFlow and TFLite kzbin.info/www/bejne/rKu3dH2DZp2opa8 Darknet YOLOv4 Custom Object Detection: Part 2 (Training YOLOv4 Darknet) kzbin.info/www/bejne/Y3-oc2iGYrh_n5I Darknet YOLOv4 Custom Object Detection: Part 1 (Preparing Custom Dataset) kzbin.info/www/bejne/qXynqqaEqdObgJY YOLOv4 Object Detection with TensorFlow, TFLite and TensorRT kzbin.info/www/bejne/qnTQdGqkrst9ppo Darknet YOLOv4 Object Detection for Windows 10 on Images, Videos, and Webcams kzbin.info/www/bejne/fHaVeHWbgNqkpsU Real Time Object Detection on Webcam and Videos Using OpenCV With YOLOv3 and YOLOv4 | Windows Linux kzbin.info/www/bejne/fJvcd2NpqsaqqaM Build and Install OpenCV 4.4.0 with CUDA (GPU) Support on Windows 10 kzbin.info/www/bejne/qpu7nIpmYpmag6c Install TensorFlow GPU and PyTorch with CUDA on Windows 10 Anaconda | CUDA 10.1 cuDNN 7.6 kzbin.info/www/bejne/hp26aq13nJmWrpo Real-time Multiple Object Tracking with YOLOv4 TensorFlow and Deep Sort | Linux, Windows kzbin.info/www/bejne/fZLKenxrhaaYgck

[@coder5199]

You mentioned semantic segmentation at 0:09 . How to do semantic segmentation? It is not shown in this tutorial. Help!

[@TheCodingBug]

@@coder5199 Use panoptic segmentation as it's unified method for instance and semantic segmentation in detectron2.

[@coder5199]

@@TheCodingBug How can I turn off the number of instance segmentation for panoptic segmentation? My project do not want individual instances. It wants a single coloured masks for each category.

[@ricala9063]

hey man thanks, you saved me, i been trying to install this for 3 days and nothing looks to work but with your video i finally got it, thank you so much!!!

[@garyhuntress6871]

This was SO much better than the documentation!! I'm up and running again!

[@TheCodingBug]

I am glad it was helpful.

[@madhavgupta1212]

Crisp and informative, nice work. Thanks a lot for this video.

[@bithigh8301]

THIS. IS. THE. BEST. DETECTRON2. TUTORIAL. EVER! Thanks for sharing, very neat and good tutorial!

[@TheCodingBug]

I'm glad you found it useful.

[@coder5199]

Strongly Agree!

[@SanjeetKumar-nt3zo]

Awesome explanation ..Thanks for sharing

[@find127]

Dude great stuff!! Thank you

[@mohammadrajabi3152]

YESSSIR, i appreciate it man

[@amandarosa4653]

Very good video, thank you!!

[@TheCodingBug]

I am glad it was helpful.

[@sevemz3359]

Hi thanks for the detailed tutorial, how do I disable the name classes or label and the percentages from showing? Thanks

[@zunayedmahmud3515]

Thank you for the excellent explanations! I have implemented detectron2 myself but the inference speed seems very slow for videos. I'm getting around 2 fps on a Tesla V100 for videos with resolution 1920x480. I was hoping you might have an idea.

[@geonovelty]

with visual studio code at the "output" i get error plus that conda is not recognized

[@AaronFhd]

Quick Question how would I save a black and white image for just the detection of people? I want automate a task. Thanks

[@MuhammadHamza-rk1im]

kindly make a video for training custom semantic/instant segmentation dataset using detectron2

[@emmanuelkoupoh7979]

Hello, how to finetune on my own keypoints dataset ??? I don't find any tutorial that works for me. I need help please. I have a dataset with 4 keypoints per image

[@meiketas2293]

What is the format of the detector = Detector(model_type="OD")? Is this an array or a dictionary?

[@joewong7035]

Thanks for the tutorial! Just one problem, when I run it on video, the video’s frame rate become very slow, causing it to be very laggy. May I know how do I fix it?

[@HarithaWeerathunga]

you need better gpu

[@discovertop10any]

Hey, how to resume training in detectron2, i am using kaggle for training

[@ankitbatra270]

Is there a way to only segment one of the classes without changing the model? For example, in the instance segmentation model it detected sheep and person is there a way to do person only?

[@shubhamchoudhary5461]

Yup , you can apply if else condition there to segment only one class..... I'm writing sudo code here.. If detect-op == car : Visulize this Else: pass

[@ManuelHernandez-zq5em]

When I ran any of the tests python programs, I get this error: "ImportError: DLL load failed while importing win32file: The specified module could not be found." Any idea how to resolve this issue? 😕

[@Fernandomoraes13]

I don't even know what is Conda, but I need to use detectron2. What can I do? I couldn't install it the usual way... (file -> settings -> python interpreter -> serach for package)

[@deidy2005]

Man, I created a customized model with 3 classes (A, B and C). And I would like to take especific masks for a especific object. And I use this code: mask_A = outputs['instances'].pred_masks.cpu().numpy()[0]

[@jubayerhossainarnob9935]

how do I get an extracted list of segmented classes?

[@sachinpr7148]

I am not able to install detectron in windows as detectron2 is asking for torch version 1.8 or above with 1.8 still getting errors

[@gardief29]

very nice tutorial, I would like know how to do object detection of a specific class (for example person)?

[@TheCodingBug]

Model returns dictionary as detections. You can actually find the class labels from those dictionary. I guess class index for person is 1. So you can put a check that if class ==1, visualize the object, otherwise continue. Let me know if you are able to achieve that. Otherwise, I will make a small tutorial on how to do that.

[@vincentdupuis8134]

Amazing! Question for video panoptic segmentation, how to do for an object to keep its colour from frame to frame

[@isaacjacobharris]

Might be a bit late but I'd assume if you record the centre of mass of each object then the next frame when things have moved a little rerecord the centre of mass and assign the the new centre of masses to the closest old centre of massed

[@evanshlom1]

@@isaacjacobharris sounds like K-means is a useful tool. I've heard of people using clustering for computer vision but haven't looked into it

[@ashishnarvekar2051]

Hi. How to use the detectron2 repository? I want to try this demo

[@modassirafzal9560]

Can you make a tutorial for Semantic Segmentation because this is a really good tutorial

[@TheCodingBug]

I'll try to make one in November.

[@annmaryjose6786]

how can we use detectron2 to segment ultrasound images

[@snazalsingh2465]

can we give the same color to each object through the video??

[@manojbs5736]

How to add conda in visual studio code terminal

[@anissarker8543]

Can you make a video on densepose from detectron2?

[@adityasprojects7126]

Thanks for the awesome video. However, the inferencing part runs on CPU and not on cuda. It throws error if I change cpu to cuda - "output = viz.draw_instance_predictions(predictions["instances"].to("cuda"))" . How do i run videos/images on cuda

[@vy1w1yv]

did you resolve your issue with cuda?

[@nkchaitanya6980]

How to print classes in list after detection or segmentation

[@muddassarjilani4637]

how can find the area of each mask, any code

[@sufailmuhammed9670]

during installing the pytorch toolkit etc..CondaSSLError: OpenSSL appears to be unavailable on this machine. OpenSSL is required to download and install packages.error is happening. How can i fix that

[@Mr.Esmaily]

Dose detectron2 runs on rtx3090 ?

[@dheerajk1469]

I wanted to ask how can I perform Multitask learning using detectron2 , i mean i need to perform Obj Detection, Semantic segmentation and Depth estimation using a single network. Can you throw some ideas how I can I do this, how can I fuse the specific task heads to perform all the tasks together when I pass in an image.

[@yasminebaroud3965]

did you can make it in one model ?

[@VBFilms]

I keep getting the error: "ImportError: DLL load failed while importing win32file". I reinstalled the whole thing several times but I can't figure out why it's not working.

[@deepdiver4130]

just do conda install pywin32 It should solve your problem

[@abdelkarimlahmdani6793]

Thanks you, just were can i get the code colab please?

[@TheCodingBug]

I have another video for colab kzbin.info/www/bejne/mJvEqmqciZ5mars

[@discovertop10any]

Hey is it possible for training detectron2 cascade rcnn on custom dataset?

[@TheCodingBug]

Kindly check our detectron2 custom object detection videos. It's possible to train any detectron2 model using custom dataset.

[@discovertop10any]

@@TheCodingBug thank you for you lightning support :-)

[@joaopedropatricio6016]

Is this using gpu? or cuda? Can I run this on AMD graphics card? Can I get higher frame rates by doing parallel processing? Is it possible ?

[@TheCodingBug]

Yes it's using GPU. You cannot use GPU acceleration if you have AMD GPU. However, you can use it with CPU or Google colab in that case.

[@joaopedropatricio6016]

@@TheCodingBug why Google colab? How can I get higher frame rates ir video analysis?

[@muddassarjilani4637]

sir, while running this code on mac m1 pro, I am getting the following error height, width = original_image.shape[:2] AttributeError: 'NoneType' object has no attribute 'shape' kindly guide me

[@TheCodingBug]

You have a problem with reading the image.

[@tanujkhandelwal5306]

I'm getting the error "No module named Detectron2" after following your installation steps. Got this error while running a test file in the detectron2's test folder.

[@TheCodingBug]

Try importing it in command line while at main folder.

[@DManniex]

I had this error too, you need to change your enviroment to the detectron enviroment (in anaconda you select the enviroment before you launch your IDE)

[@Samurai-in5nr]

Is it possible to use detectron2 for my own custom dataset. I want to use it for finding dog keypoints? If yes can u please suggest how can i acheive that and if not then where can i start with?

[@TheCodingBug]

Yes it's possible. I have made video for custom detectron2 also. You would be able to find it on the channel.

[@Samurai-in5nr]

@@TheCodingBug yes sir I have seen that video, you have used it for image segmentation but how can I use it for keypoints. As I have went through the code I am not able to find where I have to make the changes.

[@johngrabner]

Predictor takes an BGR image not RGB according to the class DefaultPredictor comments. Unless cfg.INPUT.FORMAT is set to RGB.

[@TheCodingBug]

That's true. Therefore, we are converting the input image to BGR by the command "image[:,:,::-1]" and then the result back to RGB format.

[@johngrabner]

@@TheCodingBug Your code is correct. Did not realize that cv2.imread inputs as BGR. DefaultPredictor takes cfg.INPUT.FORMAT that defaults to BGR, consistent with cv2. But Visualizer wants RGB. Hence why you reverse the order when Visualizer is called.

[@paulyiu6389]

Thanks a lot for your video, but i face an error with the pip install -e. Building wheel for pycocotools (setup.py) ... error ERROR: Command errored out with exit status 1:

[@TheCodingBug]

You need to install pycocotools before building detectron2.

[@paulyiu6389]

@@TheCodingBug Thanks a lot for your comment, and the previous problem has been solved but a new problem occurs: ERROR: Command errored out with exit status 1: command: 'E:\anaconda3\envs\detectron_env\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\User\\detectron2\\setup.py'"'"'; __file__='"'"'C:\\Users\\User\\detectron2\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"' '"'"', '"'"' '"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps cwd: C:\Users\User\detectron2\

[@karthickraja1026]

@@paulyiu6389 Paul, I have faced same issue on install -e.How could resolve it?

[@dhlee2272]

Thank you for your good tutorial Could you help me with my problem? I have error that "COCO-Detection.faster_rcnn_R_101_FPN_3x.yaml not available in Model Zoo" How can I solve this problem?

[@pankajdeoli3889]

Put / instead of . after COCO-Detection

[@k2no548]

Have you solved the problem ?

[@k2no548]

@@pankajdeoli3889 I get the same issue even tho the expression is correct

[@aishwaryapanampilly5993]

@@pankajdeoli3889 I did this but still have the same error

[@aishwaryapanampilly5993]

Have you managed to solve the problem?

[@ilyassziz5]

hy, i get an error while executing the code. No module named 'detectron2.engine'. please help???

[@TheCodingBug]

You haven't installed detectron2 in the environment you're using.

[@ilyassziz5]

@@TheCodingBug i install it but i get the error!!

[@malyaj.mishra]

Hi. Thanks a ton for your tutorial. Really appreciate your hard work. Just a small query. I am getting below error while using the command: pip install -e . error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2022\\BuildTools\\VC\\Tools\\MSVC\\14.34.31933\\bin\\HostX86\\x64\\link.exe' failed with exit code 1120 Can you please tell me how to fix this? PS: Searched and tried many things, but couldn't find any solution.

[@bryanwang85]

Same, any solution?

[@adampetit7020]

@@bryanwang85 @malyaj.mishra same, did u find a solution ?

[@geonovelty]

I get an error with the pip install -e .

[@TheCodingBug]

What's the error?

[@gabrielprocopenco]

@@TheCodingBug ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode: C:\Users\Gabriel\PycharmProjects\detectron\detectron2

[@gabrielprocopenco]

this command must not be executed in the detectron2 folder

[@zenilgosher772]

How to use this with Google colab?

[@TheCodingBug]

I'll upload video on Monday.

[@welsonfy5246]

erreur install after pip install -e . : RuntimeError: Error compiling objects for extension [end of output] note: This error originates from a subprocess, and is likely not a problem with pip.

[@chelappankumar3187]

Yeah I also got the same error i didn't get solution

[@AbdulHaseeb-um8gn]

yup same error though need solution did u find any ??

[@saikoushik7026]

waiting for instance segmentation in google colab sir

[@TheCodingBug]

It' uploaded: kzbin.info/www/bejne/mJvEqmqciZ5mars

[@user-wz4xz1dl6h]

What am I doing wrong? File "c:\Users\User\.vscode\Detectron2\Main.py", line 5, in detector.onImage("Images/1.jpg") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "c:\Users\User\.vscode\Detectron2\Detector.py", line 24, in onImage prediction = self.predictor(image) ^^^^^^^^^^^^^^ AttributeError: 'Detector' object has no attribute 'predictor'

[@najmeb1786]

can you please not use any musical background?

[@TheCodingBug]

Sure.

[@generalroboskel]

Alot people don't get how distracting the musical background is. Thanks for bringing it up...why do people think it improves it, I'll never understand.

[@harshmankodiya9397]

why is the detectron2 documentation so bad and non detailed....

[@TheCodingBug]

Perhaps they abandoned the project.

[@marverickbin]

I followed exactly like discribed, but Im stuck because of this error, when running the main.py (in 6:33s): RuntimeError: COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml not available in Model Zoo! I checked for typos, there are none. (my code is exactly like the video: self.cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml")) self.cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml") Help? EDIT: I complete deleted the detectron folder and cloned and instaled again and it worked.

[@sohailali5741]

I am getting errors here, can you please let me know the solution? viz = Visualizer(image[:,:,::-1], metadata = MetadataCatalog.get(cfg.DATASETS.TRAIN[0]) NameError: name 'cfg' is not defined

[@TheCodingBug]

You need to import and initialize cfg

[@sohailali5741]

@@TheCodingBug Thanks, just solve the error by replacing MetadataCatalog.get(cfg.DATASETS.TRAIN[0]) with MetadataCatalog.get(self.cfg.DATASETS.TRAIN[0])

[@darrenho9564]

good tutorial. though if anyone got the "IndexError: too many indices for tensor of dimension 2" error for the panoptic segmentation code like i did, i managed to resolve it by replacing the original lines of code: else: predictions, segmentInfo = self.predictor(image)["panoptic_seg"] viz = Visualizer(image[:,:,::-1], MetadataCatalog.get(self.cfg.DATASETS.TRAIN[0])) output = viz.draw_instance_predictions(predictions["instances"].to("cpu")) with these lines of code instead: else: panoptic_result = self.predictor(image) panoptic_seg, segments_info = panoptic_result["panoptic_seg"] viz = Visualizer(image[:,:,::-1], MetadataCatalog.get(self.cfg.DATASETS.TRAIN[0])) output = viz.draw_panoptic_seg(panoptic_seg.to("cpu"), segments_info) hope it helps anyone who faces this issue