330 - Fine tuning Detectron2 for instance segmentation using custom data

This video tutorial explains the process of fine tuning Detectron2 for instance segmentation using custom data. It walks you through the entire process, from annotating your data, to training a model, to segmenting images, to measuring object morphological parameters, to exporting individual masks (results) as images for further processing.
Code generated in the video can be downloaded from here: github.com/bnsreenu/python_fo...
All other code:
github.com/bnsreenu/python_fo...
Detectron2 repo: github.com/facebookresearch/d...
Annotations were done using Makesense: www.makesense.ai/
Dataset from: leapmanlab.github.io/dense-cell/
Direct link to the dataset: www.dropbox.com/s/68yclbraqq1...
Data courtesy of:
Guay, M.D., Emam, Z.A.S., Anderson, A.B. et al. ​
Dense cellular segmentation for EM using 2D-3D neural network ensembles. Sci Rep 11, 2561 (2021). ​
Data annotated for 4 classes:
1: Cell
2: Mitochondria​
3: Alpha granule​
4: Canalicular vessel​

[@SkalskiP]

Hi 👋 I'm the creator of MakeSense! Thanks a lot for using the tool in your tutorial!

[@DigitalSreeni]

Nice to meet you. Thanks for creating a very useful tool :)

[@utkarshdeshmukh3268]

@SkalskiP. You have made a fantastic tool. Great job! I found a minor bug in it. I believe the coco-json structure has evolved. Under "annotations" it is expected to have a category_id, which your tool does not populate.

[@QubitsQuantum]

KING YOU ARE A KING!!!!!!!!THANK UUUUUUUUUUUUUUUUUUUUUUU U ARE THE BEST!!!!!!! I THINK U SHOULD GET AN AWARD. THANK YOU!!!

[@temiwale88]

I love you Doc. I needed this. Thank you, DigitalSreeni!

[@zubairkhalid3209]

Appreciate your tutorials a lot! Keep the good work going.

[@astitva5002]

your resources have been incredibly helpful. thank you so much

[@sozno4222]

I honestly believe you have the best tutorial videos on youtube for vision ml. It's very thorough providing all the necessary information for somebody to work through it from scratch. Keep it up. Love these

[@satwikpandey9977]

Happy teacher's day bhaiya. You've made me fall in love with machine learning. Your videos are to the point and nicely explained. Thanks❤

[@ajay0909]

The hardest part is annotating the images, I have done a lot of annotations before. You have done a good job by explaining annotations in detail which is very rare. Can you please share the type of applications that are in demand so that we can try out new things with the knowledge you give us here. Thanks for all the extra effort you put in to explain each line of code.

[@user-qs5rb2tj1p]

Thank you very much for this helpful tutorial. God bless!

[@patis.IA-AI]

Merci , super tuto , bien clair , bien explique en un mot "amazing"

[@soumen_das]

Excellent Tutorial!

[@buildingconstructioninnova6620]

Great job. We will reference you. Keep impacting

[@DigitalSreeni]

Awesome, thank you!

[@mahilkr]

Awesome explanation

[@acmontesh102]

Thank you for such a great explanation. It would be very nice to complement the tutorial with the workflow to build on top of detectron2, i.e., to use the pooled feature map that the object detection and the masking model use to add a new module within the architecture.

[@moumitamoitra1829]

please make a tutorial on DETR instance segmentation. Your videos are really helpful.

[@msaoc22]

thank you again for this great tutorial. You make amazing job for us. Detectron works a lot better for me then Yolo8 or Faster RCNN. I will play a bit more with parameters in future, but even with small dataset I was able to get reasonable result =)

[@DigitalSreeni]

Similar experience as mine. I too like Detectron2 results better and I find it to be more reliable compared to other instance segmentation frameworks available.

[@computing_T]

23:17 I love coffee as well hahaha . Thanks sir for the content

[@nguyenthehoang9148]

A point you made at the very end of this video is very important. U-NET is not capable of segmenting overlapping objects. Would you think of making a video comparing between different models for segmentation or other computer vision tasks? Thank you!

[@alexiscarlier2773]

Thanks again. Do you know if it is possible to train mask r cnn using only bounding boxes that still performs the segmentation after training?

[@user-qs5rb2tj1p]

Thank you very much for this heloful tutorial. I have a question regarding the unites of area of each class- is it pixel?

[@askitloud3168]

Thanks for this sir. I understand each and very thing very clearly. I have few questions. 1) - Why size of the model_final.pth is 334.2 MB and how we reduce size of model without loosing model accuracy and detection rate. 2) - For getting best accuracy and detection rate from detectron2 what should the minimum annotated labels per class and how many max classes we involve in single model. 3) - How, we check our model performance after training, is there any metrices and anything so that we can get rough understanding about model accuracy.

[@alexis.martin-comte]

Hey, i watched your tutorial and it was very clear but i would love to use custom data format (like tiffs with more than 3 bands) and i find it very difficult to do

[@inquisitiverakib5844]

It will be very helpful if you look into MMsegmentation framework. It's quite similar to this. But there is very few videos about this...😊

[@rariwa]

Thanks for the tutorial, do you know a reference to train instance segmentation using label mask only (same image size with original image) ? I have one class but I would like to differentiate different object.

[@ErikCarrieri]

Do you know how to adapt this process in order to work on images with multiple bands? For instance, for aerial images of homogeneus areas sometimes having just RGB images could not provide enough information and the accuracy of the segmentation could result too poor. Hence, to improve accuracy more bands (such as a Digital Terrain Model, or a slope band) can be used, since they provide more information for the segmentation of the object. Do you think something like this is feasible?

[@junk90s92]

Hey DigitalSreeni. Love the video! Where can I access the .json files and .png images? I see tiff files in the dropbox but not the others If unaccessible, do I need to create them myself?

[@alocatnaf]

thanks for your great tutorials! are there any benefits using detectron2 with maskrcnn instead of, e.g, matterports maskrcnn?

[@DigitalSreeni]

Detectron2 makes it easy to access Mask R-CNN compared to matterport or other forms. Underneath, they are all using the same approach. So, no benefit when it comes to results but super beneficial when making the technology accessible.

[@alocatnaf]

@@DigitalSreeni ok i see. very helpful, thanks alot!

[@zeynabbayat5596]

I appreciate your free tutorial. The problem is that I want to use the weights and the output of TensorFlow, but the result in the output folder is not like what I had in my YOLO training. I don't know how to find the weight folder and TensorFlow output images.

[@yahaisha]

thanks sir .🎉🎉

[@DigitalSreeni]

Most welcome

[@minhloinguyen8639]

Thank you for worderful lecture, but I get in trouble with dataset detached in description. I don't know how to convert it to COCO format to apply detectron2, I think data include image and label so I don't need to use makesense to label it manually. Give me some tips for using data provided.

[@lukasb7013]

Detectron2 is a Two-Stage-Detector right? Is it therefore better at detecting and segmenting filigree defects? And does the library mentioned offer the possibility for hyperparameter tuning as i sparsely see any content about it with regards to the topic of instance segmentation

[@DigitalSreeni]

I do not know what Filigree is so I cannot share my opinion based on any experience. But if you are trying to detect specific objects in an image then this approach will work for you. Regarding hyperparameter tuning, you can do it via adjusting the parameters in the config file.

[@lukasb7013]

@@DigitalSreeni Thanks a lot. Filigree = very small details :)

[@DigitalSreeni]

Thanks for educating me :) Learning something new every day :)

[@lukasb7013]

@@DigitalSreeni :D Back to my question, 2 stage detectors should be better for such details right?

[@sushmanthreddymereddy3074]

@DigitalSreeni I have been thinking to fine-tune dataset on cell-lineage analysis for C .elegans??

[@DigitalSreeni]

Do you have the training data? Often times it comes down to data and knowledge to annotate.

[@prayag_pawar]

Hey , as i work on aerial imagery, i thinking of using detectron2 resnet101 model and create my own pretrained model as they used coco datatset for creating pretrained in .pkl format than fine tune the model according my needs how can i do that thank you helping 👍

[@shreyasinha7926]

Thanks for sharing! I really needed exactly this. I cannot find the dataset. The link is not working

[@DigitalSreeni]

Can you try this direct link for the dataset, I hope it works for you (unless region-blocked for some reason): www.dropbox.com/s/68yclbraqq1diza/platelet_data_1219.zip

[@yahaisha]

sir..can you show how to display result performance using tensorboard for this example .for total loss

[@jackyu6466]

if you start the training process ( trainer.train() ) inside the same block as default trainer, tensorboard should work

[@nguyenthehoang9148]

I looked at some notebooks in the Sartorius - Cell Instance Segmentation competition on Kaggle but people don't report quantifying metrics. It's kind of confusing. Do people don't really care about the metrics like AP in segmentation tasks?

[@raihanvalentino2175]

Very nice content. But I think your link in description is not working well. Please fix it. Thank you!

[@DigitalSreeni]

If you are referring to the dataset link then try this link: www.dropbox.com/s/68yclbraqq1diza/platelet_data_1219.zip

[@junk90s92]

Could you provide the annotations and json files you used? I was not able to reproduce the results -- the accuracy on the test images was quite poor.

[@billlee2641]

at 26:02, does initiate weight means exactly same as transfer learning ?

[@user-on1gn8wr6z]

thanks a lot sir, can you show us how to run it in python under windows environment?

[@chewchengkai2769]

Hi Sir, I want to ask about the evaluation part, how to find performance metrics other than mAP, such as confusion matrix, and IOU? Find so many code online no show example with using detectron2 to find these.

[@praveenavala2297]

same doubt bro, did u get it?

[@seanolivieri4829]

for d in random.sample(train_dataset_dicts, 2): img = cv2.imread(d["file_name"]) visualizer = Visualizer(img[:, :, ::-1], metadata=train_metadata, scale=0.5) vis = visualizer.draw_dataset_dict(d) plt.imshow(vis.get_image()[:, :, ::-1]) plt.show() I dont see the right labels, they are labeled as 1-12 in my dataset instead of the category names. Probably its a problem with my json but does anyone know a solution?

[@dimitheodoro]

Thanks for sharing knowledge! Again i cannot find the code link!!!!

[@DigitalSreeni]

You will find the code link now. There is a 4 to 5 hr gap between the video publishing time and me waking up and pushing the code to github. You caught me while I was still asleep :)

[@sudiptaproggaislam7321]

please make a video of detectron2 for panoptic segmentation...

[@pointrally1713]

Why do no results appear in the image after training with images with only 1 class? Of course I changed it to NUM.CLASSES = 1.

[@vsbhadouria6768]

Can you make a video on how to perform virtual staining using deep learning?

[@DigitalSreeni]

Hopefully in future. Here is a link from my bookmarks. I saved it to test for future video but haven't found time yet. github.com/softmatterlab/live-apoptotic-virtual-staining

[@vsbhadouria6768]

@@DigitalSreeni Thanks a lot for your response. I will go through the github link and see if I can crack it.

[@boppadapusaiteja8074]

You didn't discussed about finetuning why the tile that you have kept

[@user-fg9dj7dt8n]

very goood explanation,how can i contact u?

[@saichandanakoganti3410]

Sir, I need to try instance segmentation with Mask RCNN using my custom dataset could you help me with this.

[@DigitalSreeni]

That is exactly what this video is all about.

[@saichandanakoganti3410]

Sir, i have a separate annotation file for every single image in the form of .csv can i convert all those files into single .json file?

[@user-lv7wo4nl5n]

Is MakeSense safe to use? Any data security issues?

[@DigitalSreeni]

Not sure what you mean by safe. Please check their data policy. In general, if you are using any product or service for 'free', there is a good chance that your data will be used to improve their future offering. This can range from learning about how their product gets used to using customers' data to train their own DL models. Nothing is free in life!!!

[@jayakrishna7995]

for annotation tool lets use roboflow for auto segmentation process...🤚🤚🤚👍

[@DigitalSreeni]

You can try Roboflow on these images. Hopefully you will have better luck than I.

[@-_MUBESSIRULUMMAH]

​@@DigitalSreeniI tried using roboflow to annotate and apply your code for detection, but I experienced an error.

[@vtjustin333]

Dear Sreeni, I love all of your videos. I have a specific use case. Is there any chance to connect with you.? I have tried in Linkedin, but unfortunately couldn't connect with you yet. :(

[@user-lv7wo4nl5n]

Is it safe to use MakeSense?

[@DigitalSreeni]

Not sure what you mean by safe. Please check their data policy. In general, if you are using any product or service for 'free', there is a good chance that your data will be used to improve their future offering. This can range from learning about how their product gets used to using customers' data to train their own DL models. Nothing is free in life!!!

[@kridsumangsri964]

Sir please guide how to deploy custom model on web app thx you ^^

[@DigitalSreeni]

kzbin.info/aero/PLZsOBAyNTZwYSk07iwEbWFaDxeRw1NXt6&si=d8qH4MK-ACg6pNya

[@kridsumangsri964]

Thank you❤

[@Brickkzz]

What a pain to install it on Windows...

[@clabrop]

were you able to do it?

[@user-dd6gz1wk3r]

trainer.train() time error results like RuntimeError Traceback (most recent call last) in () ----> 1 trainer.train() #Start the training process 13 frames /content/detectron2/detectron2/layers/wrappers.py in nonzero_tuple(x) 151 return x.nonzero().unbind(1) 152 else: --> 153 return x.nonzero(as_tuple=True) 154 155 RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. Reason for the error??