330 - Fine tuning Detectron2 for instance segmentation using custom data

[@SkalskiP]

Hi 👋 I'm the creator of MakeSense! Thanks a lot for using the tool in your tutorial!

[@DigitalSreeni]

Nice to meet you. Thanks for creating a very useful tool :)

[@utkarshdeshmukh3268]

@SkalskiP. You have made a fantastic tool. Great job! I found a minor bug in it. I believe the coco-json structure has evolved. Under "annotations" it is expected to have a category_id, which your tool does not populate.

[@sozno4222]

I honestly believe you have the best tutorial videos on youtube for vision ml. It's very thorough providing all the necessary information for somebody to work through it from scratch. Keep it up. Love these

[@ajay0909]

The hardest part is annotating the images, I have done a lot of annotations before. You have done a good job by explaining annotations in detail which is very rare. Can you please share the type of applications that are in demand so that we can try out new things with the knowledge you give us here. Thanks for all the extra effort you put in to explain each line of code.

[@temiwale88]

I love you Doc. I needed this. Thank you, DigitalSreeni!

[@QubitsQuantum]

KING YOU ARE A KING!!!!!!!!THANK UUUUUUUUUUUUUUUUUUUUUUU U ARE THE BEST!!!!!!! I THINK U SHOULD GET AN AWARD. THANK YOU!!!

[@michaelfenelon1423]

Hi Sreeni, many thanks for the this and amazing tuts. You are a life saver ! :)

[@satwikpandey9977]

Happy teacher's day bhaiya. You've made me fall in love with machine learning. Your videos are to the point and nicely explained. Thanks❤

[@patis.IA-AI]

Merci , super tuto , bien clair , bien explique en un mot "amazing"

[@moumitamoitra1829]

please make a tutorial on DETR instance segmentation. Your videos are really helpful.

[@zubairkhalid3209]

Appreciate your tutorials a lot! Keep the good work going.

[@astitva5002]

your resources have been incredibly helpful. thank you so much

[@msaoc22]

thank you again for this great tutorial. You make amazing job for us. Detectron works a lot better for me then Yolo8 or Faster RCNN. I will play a bit more with parameters in future, but even with small dataset I was able to get reasonable result =)

[@DigitalSreeni]

Similar experience as mine. I too like Detectron2 results better and I find it to be more reliable compared to other instance segmentation frameworks available.

[@buildingconstructioninnova6620]

Great job. We will reference you. Keep impacting

[@DigitalSreeni]

Awesome, thank you!

[@computing_T]

23:17 I love coffee as well hahaha . Thanks sir for the content

[@manuelsteele7755]

I downloaded the TBX11 dataset for a course to apply MM Detector Toolbox for Swin Backbone to localize/detect objects. The tutorial notebook to install and apply the code is two years out-of-date. Hence, the dependencies are broken. For example, it says to install mmcv-full but that doesn't work with python 3.12. I had to revert to 3.11. But then something else broke. I fixed that, then another dependency broke. It appears the MMDetector is hit or miss in terms of avoiding dependency issues. It's broken in my opinion in late 2024. I can't use it for my class right now. But your video for the detector2 is excellent. The TBX11 has (1) images and (2) annotation files in JSON COCO format. It appears I can just install the detectron2 on colab and load the TBX11 dataset and run it quickly. The code setup here and demo are far more realistic and efficient compared to MMDetection. I am very happy you posted this video. It will help me on a course for medical image processing and AI (or CNN) for segmentation and localization.

[@PoojaMaurya-i6n]

Thank you very much for this helpful tutorial. God bless!

[@billlee2641]

at 26:02, does initiate weight means exactly same as transfer learning ?

[@PoojaMaurya-i6n]

Thank you very much for this heloful tutorial. I have a question regarding the unites of area of each class- is it pixel?

[@drswapnil1234]

can you show how to use tensorrt in this to improve teh performance.

[@minhloinguyen8639]

Thank you for worderful lecture, but I get in trouble with dataset detached in description. I don't know how to convert it to COCO format to apply detectron2, I think data include image and label so I don't need to use makesense to label it manually. Give me some tips for using data provided.

[@zeynabbayat5596]

I appreciate your free tutorial. The problem is that I want to use the weights and the output of TensorFlow, but the result in the output folder is not like what I had in my YOLO training. I don't know how to find the weight folder and TensorFlow output images.

[@dimitheodoro]

Thanks for sharing knowledge! Again i cannot find the code link!!!!

[@DigitalSreeni]

You will find the code link now. There is a 4 to 5 hr gap between the video publishing time and me waking up and pushing the code to github. You caught me while I was still asleep :)

[@nguyenthehoang9148]

A point you made at the very end of this video is very important. U-NET is not capable of segmenting overlapping objects. Would you think of making a video comparing between different models for segmentation or other computer vision tasks? Thank you!

[@raihanvalentino2175]

Very nice content. But I think your link in description is not working well. Please fix it. Thank you!

[@DigitalSreeni]

If you are referring to the dataset link then try this link: www.dropbox.com/s/68yclbraqq1diza/platelet_data_1219.zip

[@rariwa]

Thanks for the tutorial, do you know a reference to train instance segmentation using label mask only (same image size with original image) ? I have one class but I would like to differentiate different object.

[@amesaleh6824]

how to draw confusion matrix in detectron2 ???

[@sudiptaproggaislam7321]

please make a video of detectron2 for panoptic segmentation...

[@yahaisha]

sir..can you show how to display result performance using tensorboard for this example .for total loss

[@jackyu6466]

if you start the training process ( trainer.train() ) inside the same block as default trainer, tensorboard should work

[@alexis.martin-comte]

Hey, i watched your tutorial and it was very clear but i would love to use custom data format (like tiffs with more than 3 bands) and i find it very difficult to do

[@junk90s92]

Could you provide the annotations and json files you used? I was not able to reproduce the results -- the accuracy on the test images was quite poor.

[@acmontesh102]

Thank you for such a great explanation. It would be very nice to complement the tutorial with the workflow to build on top of detectron2, i.e., to use the pooled feature map that the object detection and the masking model use to add a new module within the architecture.

[@seanolivieri4829]

for d in random.sample(train_dataset_dicts, 2): img = cv2.imread(d["file_name"]) visualizer = Visualizer(img[:, :, ::-1], metadata=train_metadata, scale=0.5) vis = visualizer.draw_dataset_dict(d) plt.imshow(vis.get_image()[:, :, ::-1]) plt.show() I dont see the right labels, they are labeled as 1-12 in my dataset instead of the category names. Probably its a problem with my json but does anyone know a solution?

[@muhammadzeeshan9548]

hi there love the content can you please do a video for panoptic segmentation using are own dataset and please make it from scratch meaning how we get the instance segmentation mask and the semantic segmentation mask then how we combine them and run it on own own dataset.

[@askitloud3168]

Thanks for this sir. I understand each and very thing very clearly. I have few questions. 1) - Why size of the model_final.pth is 334.2 MB and how we reduce size of model without loosing model accuracy and detection rate. 2) - For getting best accuracy and detection rate from detectron2 what should the minimum annotated labels per class and how many max classes we involve in single model. 3) - How, we check our model performance after training, is there any metrices and anything so that we can get rough understanding about model accuracy.

[@prayag_pawar]

Hey , as i work on aerial imagery, i thinking of using detectron2 resnet101 model and create my own pretrained model as they used coco datatset for creating pretrained in .pkl format than fine tune the model according my needs how can i do that thank you helping 👍

[@alexiscarlier2773]

Thanks again. Do you know if it is possible to train mask r cnn using only bounding boxes that still performs the segmentation after training?

[@soumen_das]

Excellent Tutorial!

[@chewchengkai2769]

Hi Sir, I want to ask about the evaluation part, how to find performance metrics other than mAP, such as confusion matrix, and IOU? Find so many code online no show example with using detectron2 to find these.

[@praveenavala2297]

same doubt bro, did u get it?

[@arkanaliraqi-r2g]

thanks a lot sir, can you show us how to run it in python under windows environment?

[@sushmanthreddymereddy3074]

@DigitalSreeni I have been thinking to fine-tune dataset on cell-lineage analysis for C .elegans??

[@DigitalSreeni]

Do you have the training data? Often times it comes down to data and knowledge to annotate.

[@saichandanakoganti1407]

Sir, I need to try instance segmentation with Mask RCNN using my custom dataset could you help me with this.

[@DigitalSreeni]

That is exactly what this video is all about.

[@saichandanakoganti1407]

Sir, i have a separate annotation file for every single image in the form of .csv can i convert all those files into single .json file?

[@ErikCarrieri]

Do you know how to adapt this process in order to work on images with multiple bands? For instance, for aerial images of homogeneus areas sometimes having just RGB images could not provide enough information and the accuracy of the segmentation could result too poor. Hence, to improve accuracy more bands (such as a Digital Terrain Model, or a slope band) can be used, since they provide more information for the segmentation of the object. Do you think something like this is feasible?

[@boppadapusaiteja8074]

You didn't discussed about finetuning why the tile that you have kept

[@nguyenthehoang9148]

I looked at some notebooks in the Sartorius - Cell Instance Segmentation competition on Kaggle but people don't report quantifying metrics. It's kind of confusing. Do people don't really care about the metrics like AP in segmentation tasks?

[@alocatnaf]

thanks for your great tutorials! are there any benefits using detectron2 with maskrcnn instead of, e.g, matterports maskrcnn?

[@DigitalSreeni]

Detectron2 makes it easy to access Mask R-CNN compared to matterport or other forms. Underneath, they are all using the same approach. So, no benefit when it comes to results but super beneficial when making the technology accessible.

[@alocatnaf]

@@DigitalSreeni ok i see. very helpful, thanks alot!

[@mahilkr]

Awesome explanation

[@shreyasinha7926]

Thanks for sharing! I really needed exactly this. I cannot find the dataset. The link is not working

[@DigitalSreeni]

Can you try this direct link for the dataset, I hope it works for you (unless region-blocked for some reason): www.dropbox.com/s/68yclbraqq1diza/platelet_data_1219.zip

[@pointrally1713]

Why do no results appear in the image after training with images with only 1 class? Of course I changed it to NUM.CLASSES = 1.

[@junk90s92]

Hey DigitalSreeni. Love the video! Where can I access the .json files and .png images? I see tiff files in the dropbox but not the others If unaccessible, do I need to create them myself?

[@sripadasidvilas1197]

same problem here did you get the solution

[@inquisitiverakib5844]

It will be very helpful if you look into MMsegmentation framework. It's quite similar to this. But there is very few videos about this...😊

[@jayakrishna7995]

for annotation tool lets use roboflow for auto segmentation process...🤚🤚🤚👍

[@DigitalSreeni]

You can try Roboflow on these images. Hopefully you will have better luck than I.

[@-_MUBESSIRULUMMAH]

​@@DigitalSreeniI tried using roboflow to annotate and apply your code for detection, but I experienced an error.

[@NandaKishoreBellamMuralidhar]

Is MakeSense safe to use? Any data security issues?

[@DigitalSreeni]

Not sure what you mean by safe. Please check their data policy. In general, if you are using any product or service for 'free', there is a good chance that your data will be used to improve their future offering. This can range from learning about how their product gets used to using customers' data to train their own DL models. Nothing is free in life!!!

[@vsbhadouria6768]

Can you make a video on how to perform virtual staining using deep learning?

[@DigitalSreeni]

Hopefully in future. Here is a link from my bookmarks. I saved it to test for future video but haven't found time yet. github.com/softmatterlab/live-apoptotic-virtual-staining

[@vsbhadouria6768]

@@DigitalSreeni Thanks a lot for your response. I will go through the github link and see if I can crack it.

[@Md.NahidulAlam]

very goood explanation,how can i contact u?

[@lukasb7013]

Detectron2 is a Two-Stage-Detector right? Is it therefore better at detecting and segmenting filigree defects? And does the library mentioned offer the possibility for hyperparameter tuning as i sparsely see any content about it with regards to the topic of instance segmentation

[@DigitalSreeni]

I do not know what Filigree is so I cannot share my opinion based on any experience. But if you are trying to detect specific objects in an image then this approach will work for you. Regarding hyperparameter tuning, you can do it via adjusting the parameters in the config file.

[@lukasb7013]

@@DigitalSreeni Thanks a lot. Filigree = very small details :)

[@DigitalSreeni]

Thanks for educating me :) Learning something new every day :)

[@lukasb7013]

@@DigitalSreeni :D Back to my question, 2 stage detectors should be better for such details right?

[@mohamadmahdinafar7638]

great video.

[@NandaKishoreBellamMuralidhar]

Is it safe to use MakeSense?

[@DigitalSreeni]

Not sure what you mean by safe. Please check their data policy. In general, if you are using any product or service for 'free', there is a good chance that your data will be used to improve their future offering. This can range from learning about how their product gets used to using customers' data to train their own DL models. Nothing is free in life!!!

[@kridsumangsri964]

Sir please guide how to deploy custom model on web app thx you ^^

[@DigitalSreeni]

kzbin.info/aero/PLZsOBAyNTZwYSk07iwEbWFaDxeRw1NXt6&si=d8qH4MK-ACg6pNya

[@kridsumangsri964]

Thank you❤

[@vtjustin333]

Dear Sreeni, I love all of your videos. I have a specific use case. Is there any chance to connect with you.? I have tried in Linkedin, but unfortunately couldn't connect with you yet. :(

[@yahaisha]

thanks sir .🎉🎉

[@DigitalSreeni]

Most welcome

[@Brickkzz]

What a pain to install it on Windows...

[@clabrop]

were you able to do it?

[@AnilAntony-y7p]

trainer.train() time error results like RuntimeError Traceback (most recent call last) in () ----> 1 trainer.train() #Start the training process 13 frames /content/detectron2/detectron2/layers/wrappers.py in nonzero_tuple(x) 151 return x.nonzero().unbind(1) 152 else: --> 153 return x.nonzero(as_tuple=True) 154 155 RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. Reason for the error??