286 - Object detection using Mask RCNN: end-to-end from annotation to prediction

Code generated in the video can be downloaded from here:
github.com/bns...
All other code:
github.com/bns...
This video helps you with end-to-end Mask RCNN project, all the way from annotations to training to prediction. Handling of VGG and Coco style JSON annotations is demonstrated in the video. Code is also made available for both approaches.
For this video, I've used the annotation tool from www.makesense.ai/
You can try other annotation tools like:
www.makesense.ai/
labelstud.io/
github.com/Doo...
labelme.csail.m...
github.com/ope...
www.robots.ox....
coco weights can be downloaded from: github.com/mat...

[@Guide4Ever]

Your videos are always long enough, informative and beyond what anybody does. Very polite and understanding delivery. Kudos!

[@DigitalSreeni]

I appreciate that!

[@joaosousa9620]

I'm sure the is more elegant ways of referencing paths than the crude form I employed here, but it fixed my problems. Running the code in full, no problem was detected. It just wasn't generating the weights. Going line by line or groups of lines by group of lines, following the video, I corrected some things that were wrong, printing things to see were the paths were pointing to. My changes are crude, so you will have to manually change the paths, as your path will not be like mine. All this on the coco style In my file 'F:\joaopedro\downloads\python_for_microscopists-master\286-Object detection using mask RCNN - end to end\286-marbles_maskrcnn_coco_style_labels.py', I had to make the following changes to make it work: line before change: dataset_train.load_data('marble_dataset/train/labels/marbles_two_class_coco_json_format.json', 'marble_dataset/train') line after change: dataset_train.load_data('F:/joaopedro/downloads\python_for_microscopists-master/286-Object detection using mask RCNN - end to end/marble_dataset/train/labels/marbles_two_class_coco_json_format.json', 'F:/joaopedro/downloads\python_for_microscopists-master/286-Object detection using mask RCNN - end to end/marble_dataset/train') similarly for the 'COCO_WEIGHTS_PATH = '... line, the line after the change became: COCO_WEIGHTS_PATH = 'F:/joaopedro/downloads/python_for_microscopists-master/286-Object detection using mask RCNN - end to end/coco_weights/mask_rcnn_coco.h5' Also added lines to save the weights (I'm not sure how Sreenivas was able to save it without those lines...), below the last line of the training section of the code, 'model.train(dataset_train, dataset_train, learning_rate=config.LEARNING_RATE, epochs=3, layers='heads')': # Save the trained model weights ##print(os.path.join(DEFAULT_LOGS_DIR, "mask_rcnn_trained_weights.h5")) model_path = os.path.join(DEFAULT_LOGS_DIR, "mask_rcnn_trained_weights.h5") model.keras_model.save_weights(model_path) print(f"Trained weights saved to {model_path}") -==-=- Now on the inference (which I put in a separate file), I've made these changes: line before change: model = MaskRCNN(mode='inference', model_dir='logs', config=cfg) line after the change: model = MaskRCNN(mode='inference', model_dir='F:/joaopedro/downloads/python_for_microscopists-master/286-Object detection using mask RCNN - end to end/logs', config=cfg) line before change: model.load_weights('logs/mask_rcnn_trained_weights.h5', by_name=True) line after change: model.load_weights('F:/joaopedro/downloads/python_for_microscopists-master/286-Object detection using mask RCNN - end to end/logs/mask_rcnn_trained_weights.h5', by_name=True) -==-=-=- Now on the section 'Show detected objects in color and all others in B&W' I changed 1 line (below '#line changed 1')and added another (below '#line added 2') to specify an output folder, otherwise it was outputting in my C: in my user. The changed code is below: ############################################## #Show detected objects in color and all others in B&W def color_splash(img, mask): """Apply color splash effect. image: RGB image [height, width, 3] mask: instance segmentation mask [height, width, instance count] Returns result image. """ # Make a grayscale copy of the image. The grayscale copy still # has 3 RGB channels, though. gray = skimage.color.gray2rgb(skimage.color.rgb2gray(img)) * 255 # Copy color pixels from the original color image where mask is set if mask.shape[-1] > 0: # We're treating all instances as one, so collapse the mask into one layer mask = (np.sum(mask, -1, keepdims=True) >= 1) splash = np.where(mask, img, gray).astype(np.uint8) else: splash = gray.astype(np.uint8) return splash import skimage def detect_and_color_splash(model, image_path=None, video_path=None): assert image_path or video_path # line added 2 output_dir = 'F:/joaopedro/downloads/python_for_microscopists-master/286-Object detection using mask RCNN - end to end/viz/' # Image or video? if image_path: # Run model detection and generate the color splash effect #print("Running on {}".format(img)) # Read image img = skimage.io.imread(image_path) # Detect objects r = model.detect([img], verbose=1)[0] # Color splash splash = color_splash(img, r['masks']) #line changed 1 file_name = output_dir + "splash_{:%Y%m%dT%H%M%S}.png".format(datetime.datetime.now()) skimage.io.imsave(file_name, splash) elif video_path: import cv2 # Video capture vcapture = cv2.VideoCapture(video_path) width = int(vcapture.get(cv2.CAP_PROP_FRAME_WIDTH)) height = int(vcapture.get(cv2.CAP_PROP_FRAME_HEIGHT)) fps = vcapture.get(cv2.CAP_PROP_FPS) # Define codec and create video writer file_name = "splash_{:%Y%m%dT%H%M%S}.avi".format(datetime.datetime.now()) vwriter = cv2.VideoWriter(file_name, cv2.VideoWriter_fourcc(*'MJPG'), fps, (width, height)) count = 0 success = True while success: print("frame: ", count) # Read next image success, img = vcapture.read() if success: # OpenCV returns images as BGR, convert to RGB img = img[..., ::-1] # Detect objects r = model.detect([img], verbose=0)[0] # Color splash splash = color_splash(img, r['masks']) # RGB -> BGR to save image to video splash = splash[..., ::-1] # Add image to video writer vwriter.write(splash) count += 1 vwriter.release() print("Saved to ", file_name) detect_and_color_splash(model, image_path="F:/joaopedro/downloads/python_for_microscopists-master/286-Object detection using mask RCNN - end to end/marble_dataset/val/test4.jpg") ###################################################### =-=-=-=-

[@lukeoluoch1617]

thank you so much for this. I've been quite confused on how the weights were getting saved

[@Kutulu101]

Your videos on Mask RCNN is great thank you!

[@caiyu538]

Keep on learning from your great lectures.

[@Dheeraj24]

highly informative presentation and explanation!!

[@joaosousa9620]

It is not saving the weights after training... no folder 'logs' is being created... Trying to find a solution, no apparent error message...

[@joaosousa9620]

The coco style code.

[@joaosousa9620]

I'm going to try running the VGG style to see.

[@joaosousa9620]

I solved it. There is a comment explaining quickly what I did.

[@mengyingzhang5210]

I have been watching your entire series on Mask RCNN this afternoon and learned so much from your video, Sreeni! Super grateful for your wonderful work here! I'll be following your code tomorrow step by step :)

[@ry_zakari]

Hey!

[@andresrodriguez6392]

how can I update or use my own weights (.h5) once I have trained my own models? from where and how can I implement new own ones? Thanks!

[@vincentnaayem]

Hello, I need to use a model to do semantic segmentation of human organs from CT scans (3D). I have an important question, do you think I should test both Mask RCNN and Unets? I am well documented on U nets since it seems to be used a lot for these medical tasks. I used Mask RCNN with Detectron2 in the past but have no clue how to code one from scratch. Do you think Mask RCNN is not well suited when you have only few data?

[@sambitpritam1929]

I am getting the train mAP as 0.000. Could you please let me know what could be the reason behind this?

[@nihadhadjsahraoui4782]

hey, just a question i have an issue with training maskrcnn on my custom dataset after reaching 100 epochs the training restart from the beginning any idea how to resolve this i followed the exact same tutorial. would really appreciate it im stuck any solution please

[@billlee2641]

Dear Sir, I am using this code for my own image object detection(the image is .png format). However, I encountered a issue which confused me a lot. it is at detection stage, where "detected = model.detect([marbles_img])", and I got errors saying "mold_image return images.astype(np.float32) - config.MEAN_PIXEL ValueError: operands could not be broadcast together with shapes (1024,1024,4) (3,)" it seems that my image has 4 channels so I checked that png indeed has 4 channels. Therefore, I changed all my png files to jpg files and re ran the whole training process. However, when the code executes at the same location, it still reports the same errors. Do you have any idea what caused this error? I appreciate your kind help. Thank you!

[@datapro007]

Hi Sreeni, Is the mask r-cnn model in the Tensorflow model zoo the same as Matterport?

[@noureddineelharyky1619]

nice tutorial, thank you.what's about the confusion matrix

[@alele5410]

your presentation is nice and pretty good. would please show how to classify image(Remote sensing image such as sentinel 1,2 or land sat) by CNN in jupyter notebook

[@abhishekwadighare8613]

Hi Sreeni sir, thank you for sharing mask rcnn workflow, I was using it, I am struggling to deploy it using redis as this model has custom objects and class, could you please help

[@harinathreddya8439]

Hi Sreeni i really like your videos and thanks a lot. But I have a scenario here if you want to detect whether a person is turning right or left or back what is the best approach you suggest?

[@limchengwei8853]

Hi, thanks for the video. Can we get coco format output for prediction?

[@rajbhanushali9152]

I used this to train my custom dataset on Vehicle Damage Segmentation but it's results are very poor. I trained it for 100 epochs but still it cannot work correctly on training images. Please help me to solve this. Thank you in advance

[@mayaltaee2963]

Please make a video for object detection using Faster R-CNN from annotation to end.

[@kavyasubramanian8314]

Sir how we will calculate accuracy here..?

[@stefanAH97]

amazing session, thnx a lot can't find the code on Github?

[@biplabdas8635]

Hello bro I always get this error load_image_gt() got an unexpected keyword argument 'use_mini_mask' When I run the map score from inference part Can you help me how I can solve this

[@pacodecoco4349]

Does anyone knows why I cannot find the logs of the trained model? I created a logs folder but it is empty, what am I doing wrong? Please help

[@joaosousa9620]

Similar problem here, on the coco style code. Did you find a solution? In my case not even the 'logs' folder is being created automatically, as I think is supposed to. Even puting an exact and direct path for the variable did not work.

[@aminmemar416]

How to train with binary mask?

[@guillermovc]

Hi Sreeni, im following the tutorial and doing some modifications to adjust it to my needs, but im not able to use my system's GPU, do you have any recomendation? i have cuda 11.7 installed but i dont know if i need to do another extra step. Thanks for your content!

[@rajbhanushali9152]

For that you have to upgrade the TF version but it has some dependencies on skimage so you cannot do it. Have you trained on CPU?

[@joaosousa9620]

On Medium there is a good guide on how to install 'Install CUDA and CUDNN on Windows & Linux'. CUDNN had to be installed manually, as there is no installer for older versions.

[@joaosousa9620]

You need to look up the compatibilities between the programs. My GPU is RTX 2060 Super. cdnn used by me: cuDNN v7.6.5 (November 5th, 2019), for CUDA 10.1 (had to remove the developer link... but in the 'cuDNN Archive' you can find it) cuda used: 10.1 python used: 3.7.11 tensorflow-2.2.0 Here you can find cuda and cdnn compatibility, search on google, as I cannot put links here: tensorflow tested_build_configurations Another resource that gives a little bit of context, search on google: stack exchange Dlerror: cudnn64_7.dll not found

[@ชานนท์จิรวัฒนานุกูล]

ValueError: Error when checking input: expected input_image_meta to have shape (16,) but got array with shape (15,) Are anyguy have this ploblem ? And how can i fix it

[@asqarsankibaev1789]

It's late, but I'll post it, maybe it will help someone, perhaps the number of specified classes is different NUM_CLASSES = 1 + count_your_classes

[@yangyangwen1158]

Hi, Thanks for your video! I get a maP=0.0 while I was implementing this video. Do you know what may be the different? I would appreciate if you can share your mask_rcnn_marble_cfg_0003.h5 file to Github so that I can try to run and see if maP=0.0 to figure out my problem. Thanks!

[@biplabdas8635]

Hello bro when I run the map I always get load_image_gt() got an unexpected keyword argument 'use_mini_mask' Can you tell me whether you also have got this type of error.And can you suggest me how to solve this

[@jamilal-idrus1905]

Does RCNN work like Yolo?

[@DigitalSreeni]

The actual way it works is different but if object segmentation is what you need then both Mask R-CNN and Yolo will do the job.

[@karlkeller3324]

Very useful presentation. Nice to see an end-to-end example!

[@DigitalSreeni]

Glad it was helpful!

[@tmacnba-eg8er]

sir please make a video about nnunet with costum dataset on colab.

[@datapro007]

Fantastic video! I'm going to run through the code in detail next. Many thanks.

[@nik-nj5bo]

Can you make a video to do the whole process in cloud I mean google collab

[@Guide4Ever]

What could be wrong if the training stops at the message "Epoch 1/30"? There is no loss information or anything loading like in your case. This occurs after model.train has been run. The process where it gets stuck is: get_next_batch() -> get() -> get() -> wait() -> wait() -> wait(), so I believe it doesn't get the required responses just yet or it's looping.

[@elmRoz]

Thank you for the presentation, makes the whole thing so much clearer.

[@DigitalSreeni]

Glad it was helpful!

[@Mira-wo9bp]

Your video is totally informative, totally love it. However, I am having a small problem with the model= MaskRCNN line, it keeps telling me that type object "Config" has no attribute "image_shape". Can you please tell me what has gone wrong and what is the best way to fix it? Thanks a lot and I'm looking forward to your reply!!

[@justinamichael316]

Hello sir... Wonderful lecture... I tried the same... Couldnt see where the trained weights are stored... So they are not loaded during the 'inference' and shows Train mAP value as 0.000. Please address the issue...

[@akritirani4600]

I am constantly getting the error "You are using the default load_mask(), maybe you need to define your own one." But I already did custom it . Someone please help

[@saqibqamar9270]

Thanks for video. I have trained MaskRNN model with my custom dataset. Could you tell me about how to measure large image where few objects span to another patch. In that case, objects metrics are not accurate.

[@prolong835]

Where can I find a description of the COCO file? And where do I find the requirements.txt information? Please also provide information on various auxiliary programs and versions that need to be installed.

[@alessandrosetzi1571]

Hello, I've an error when I try to import the model. The error says that there is no module named 'keras.engine'. How can i fix it?

[@moumitamoitra1829]

I have learned a lot from your videos. Could you please upload any video on Cell Nuclei segmentation using Mask R-CNN?

[@h2rpubg961]

Sir can you tell why are you loading the coco weights we have,nt trained the model yet and also coc o weights are not included in mask repo folder

[@dheerendrapratapsingh3535]

Sir please give some content for point cloud data processing by deep learning methods

[@saqibqamar9270]

Hi, Excellent explanation step by step to understand MaskRCNN. Pls, let us know how we can calculate the accuracy of generated mask instead of the mAP of object detection?

[@saikatdas5474]

How much data is required for such kind of transfer learning? Is 40 - 50 images enough?

[@DigitalSreeni]

Usually 50 objects per class is a good starting point. The more the merrier!

[@iggyk3773]

Great course! What is the best way for small object detection?

[@DigitalSreeni]

This works for small and large objects. Although, if your small objects are roundish, give StarDist a try.

[@jasterno8335]

i am using google colab and got an error AttributeError: module 'tensorflow.python.framework.ops' has no attribute '_TensorLike'

[@jasterno8335]

please help me

[@jasterno8335]

I'm still waiting for your reply

[@dronnet]

I think that need use oldest version tensorflow.