How to apply MaskRCNN for Medical Images: Explanations and Code

In this video we learn about MaskRCNN and its usage in Medical Imaging.
Github code available at: github.com/soh...
Data available at: github.com/Aak...
Some existing works on the topic are:
1. www.kaggle.com...
2. www.kdnuggets....
Most existing solutions are in Tensorflow 1 which poses difficulty for fine tuning.
3. Paper for comparison Unet vs MRCNN is at arxiv.org/abs/...

[@rashmininayanathara5304]

Thank you, mam. I had a big problem with finding compatible versions of Keras and TensorFlow. but after watching this video I solved that problem within a few minutes...Thank you again ....

[@AIwithSohini]

Thanks a lot for sharing your feedback. It supports so many of us facing major bottlenecks everyday. Stay tuned.

[@datasciencewitharbaaz5221]

Hello are you working on the MRCNN what are the versions you used can you please share.

[@wakandavernon1412]

is there any video on implementing maskrcnn on custom dataset?

[@umarsadique6207]

how I can apply mask R-CNN to the SVS files of size 0.5GB to 3.5GB. I have data from the Microscope. I am interested to collect two different types of nuclei. How I can collect this data using Mask-RCNN. 2nd question: is instance segmentation is enough for this task or I have also to do first semantic segmentation and then instance segmentation. 3rd question is: If I apply instance segmentation. I worried it may be given me more than two classes but I am interested only in two classes. thank you for your precious time.

[@AIwithSohini]

Hello Umar, so your problem requires semantic segmentation and NOT instance segmentation (since you are inetersted in the two classes specifically). Use something like Unet for your use case. I have several videos on this channel on Unet. Good luck and stay tuned!

[@saqibqamar9270]

Hi Sohini, I have trained MaskRNN model with my custom dataset. Could you tell me about how to measure large image where few objects span to another patch. In that case, objects metrics are not accurate.

[@AIwithSohini]

Hello and thanks for your question. Yes in such cases you have to extract shorter patches from the image. Once all patches have the regions detected you will need to put the patches back to form the dimensions of the original image and then proceed with evaluation. For regions where several regions from multiple patches coincide, pick only one (maybe the largest region) as the final object. In these situations semantic segmentation is more preferable rather than object counting per patch. Hope this helps. Thanks and stay tuned

[@saqibqamar9270]

Hi Sohini, Excellent explanation to implement MaskRCNN with custom dataset. Pls, let us know how we can calculate the accuracy of generated mask instead of the mAP of object detection? What would be the best metric to evaluate the instance segmentation, not detection?

[@AIwithSohini]

Hello and many thanks for the question. So instance segmentation has two components, one the area of segmentations and the number of regains segmented. For area metrics like average intersection over union or average dice scores are good. If you have multiple frames across which you are tracking instances then you use the MOTS metric to account for instance level tracking

[@RS-vw1ny]

Hi, thank you for the video, everything was clearly explained. Just have a brief question: I have some images from centrioles (they are some components of cells important for cell division) and my objective is to identify every centriole. However, for training the model, it would be useful to give an input of more than one image (instead of one image per input, which is the common thing to do, just like you did in your example). This happens because for identifying centrioles we use two images, the first one identifies with a green marker one protein of the centrioles (CEP135), and the other picture identifies another protein with a red marker (Centrin). I can merge both images into one, but the identification becomes much more difficult to do, especially when centrioles overlap (which happens regularly in the images). It's easier for a human to identify how many centrioles there are when he has the two images separate. My question is: do you know if it's possible to feed more than one image for training sample? Do I have to change the MaskRCNN architecture? Is there another model prepared for this type of input? Thank you and best regards.

[@AIwithSohini]

Hello there and thanks for your query. Your use case is super interesting must say. So any semantic or instance segmentation model operates under the paradigm that input is one image and output can be multiple regions. So you may use multi class U-net model to extract the green and red regions as separate masks. However if you need to have multiple inputs for a single output image you need to do sensor fusion, early or late fusion. There are separate models to do that. Hope this helps. Thanks and stay tuned

[@vishrutjain1427]

hi, thank you so much for this video. I wanted to ask if there is a way to use Mask RCNN pretrained model with .nii files (MRI scans of heart). The dataset itself is already annotated but both the images and the annotations are in .nii format.

[@AIwithSohini]

yes that is very much possible! you will need to rely on the python library for nib. Here is a tutorial idiotdeveloper.com/read-3d-nifti-images-in-python3/. Once you read the nii files and convert it to jpg/png that will enable you to reuse MaskRCNN. Hope this helps. Thanks and stay tuned!

[@KrithikaAnbuDeviMPHD]

hai mam,you clearly explained everything,thank you

[@AIwithSohini]

Thabks so much for joining our journey. Please stay tuned

[@saqibqamar9270]

Hi Sohini, Thank you for your video. I am interested to do instance segmentation of wood data that associates with 3 classes including background. In the dataset, few object regions in the classes are overlapped. My questions are 1- Can I use MaskRCNN in my problem? 2- How can I measure the overlapped region of classes? Or the quantitative analysis will give the actual output as Compre to qualitative analysis (visual)..pls reply

[@AIwithSohini]

Hello, I hope thiis response stream helps you www.reddit.com/r/MachineLearning/comments/cxw2eq/d_overlapping_instance_segmentation_mask_rcnn/ Good luck and stay tuned!

[@saqibqamar9270]

@@AIwithSohini Thank you so much Sohini, it's very helpful. Can you suggest me Annotation tool for tiff files who can give JSON format annotated data into single file?

[@AIwithSohini]

@@saqibqamar9270 Yes LabelMe can handle all images. kzbin.info/www/bejne/ZpzXfqypmMmceZo

[@lijinp3430]

I'm getting the following error"could not broadcast input array from shape (896,896,3) into shape (896,896)".In my dataset, images are RGB and masks also RGB.how to change the code in the load mask section

[@AIwithSohini]

Hello you will need to convert your images to grayscale using cv2 library. Then you will be able to feed the images to this codebase

[@lijinp3430]

@@AIwithSohini ok thankyou

[@lijinp3430]

Mam. In your dataset the code iterate through ids. Images and masks are stored respective folders having folder id same as of images. But in some datasets there is no such a formate. Is there any other way to convert a dataset in such a way

[@tiomixt]

Hey thanks for this video. Unfortunately, when I try to run the %tensorflow_version cell in Colab it’s written « tensorflow 1 is unsupported in Colab ». Do you know how to fix this ?

[@AIwithSohini]

Hello and thanks for your comment. Yes the versioning can be a real challenge for such code bases. This example was run on a system and not on colab. You can look at colab.research.google.com/github/tensorflow/tpu/blob/master/models/official/mask_rcnn/mask_rcnn_demo.ipynb and modify this for your set of images. Hope this helps.

[@datasciencewitharbaaz5221]

Hello mam, I am getting mAP as nan error: mrcnn/utils.py:734: RuntimeWarning: invalid value encountered in true_divide recalls = np.cumsum(pred_match > -1).astype(np.float32) / len(gt_match) mAP: nan Please help on this

[@AIwithSohini]

Seems like this ground truth image had no regions. So GT-match is empty which causes a divide by zero error returning a nan value. Hope this helps.

[@vivekanandchimman2933]

Hi Sohaniji. I am getting error ' importErrro: can not import name 'saving' from keras.engine' while loading weights. please help to overcome this.

[@AIwithSohini]

Please send a snapshot of the error to my email. I can take a look. Thanks

[@vivekanandchimman2933]

I got it solved. I got another error somedays back while opening that colab notebook. I am sending snapshot, please help to resolve.

[@vivekanandchimman2933]

Great! Hi Sohaniji How can I do it on video dataset for? (Eg:vehicle classification) I mean how I can detect different types of object (eg: vehicles) in a single video.

[@AIwithSohini]

Hello so the original implementation of Mask rcnn is to detect multiple vehicles . Here is the code base github.com/matterport/Mask_RCNN

[@vivekanandchimman2933]

@@AIwithSohini Thank you for such a quick response. I tried this only but it's giving error. So I submitted a query form in website. Please chek and help.

[@irfanullah7574]

Hi Mem Thank you .......Mem please One video on Faster RCNN in Brain tumor Data set

[@AIwithSohini]

Hello and thanks for the question. Typically for brain tumor data segmentation methods are more used like unet or maskrcnn, since the intention is to isolate the whole tumor region. Fasterrcnn does bounding box detection. So it places rectangular boxes around a tumor region. That is not clinical helpful other than for diagnostics. That’s why for medical images segmentation methods are more useful than bounding box detections. Hope this helps

[@muhammadasim1072]

Hello madam here is asim working on segmentation my data set is related to cows but i have have some problems like keras layer issues during training can you help me """ module 'keras.engine' has no attribute 'Layer' """""""""""

[@AIwithSohini]

Hello, this is a versioning error. Keras is rolled into tensorflow under tf2. Check your keras and tensorflow versions. To correct your error here you have to find the location of Layer. I believe it is tf.layers.Layer. Stackoverflow is very useful in these cases.

[@AmadeaPermanaS]

thank you for your explanation it's very helpfull, but I want to ask about the evaluation part. Did mAP is the only evaluation that can evaluate when implementing mask rcnn?

[@AIwithSohini]

Hello and thanks for your question. To evaluate the semantic maps mAP, precision recall, IOU and Jacard, Dice scores are mostly used. However, if you are applying instance segmentation on a volume scan, so if you are tracking an instance across frames (like tracking a pedestrian in consecutive video frames, or a tumor across a volume scan), then MOTS (multi object tracking) metrics should be used. Here is a paper where we use multiple metrics arxiv.org/abs/2005.07796. Hope this helps. Thanks again and stay tuned!

[@emanwaqar9347]

Hi Sohini, I am trying to segment lung x-rays using maskRCNN. I am new to this and having several issues. Please let me know how you created the masks for training? I am trying to find a way to do that but can't find it. It will be of great help. Thanks.

[@AIwithSohini]

Hello there. Please see this video on Labelme to generate segmentation masks for such tasks kzbin.info/www/bejne/ZpzXfqypmMmceZo

[@hadjersaidi7881]

Hi Sohini Thank you very much for this meaninggful explanation . I have few questions please 1. How did you create the Masks in the treaning file ? 2. Is there a way to use this code for .jason annotations?

[@AIwithSohini]

Thanks for the question. I typically use Labelme to label my images that has an add on to convert json to images. Please check out my video on Labelme. Good luck:

[@hadjersaidi2972]

@@AIwithSohini Thanks for your quick answer, good luck 🤝

[@emanwaqar9347]

Hi Hadjer, did you find out a way to create masks?

[@mohammedfadel1405]

Hi Can I apply mask rcnn with transfer learning for images classification?

[@AIwithSohini]

Hello there, maskrcnn is useful for segmentation. Classification is a much simpler task and does not need something as complex as maskrcnn. Recent image classifiers are the EfficientnetB7 versions. Hope that helps.

[@rajum9478]

im getting IndexError: list index out of range i

[@AIwithSohini]

This is a typical input output error. Please check input output dimensions

[@Ccomputervision]

Thank for u video. I have a question how to create data mask in video 14:02

[@AIwithSohini]

Hello there. Annotation or data masks can be generated using Labelme. Here is the video kzbin.info/www/bejne/ZpzXfqypmMmceZo

[@venkatesanr9455]

Hi Sohini mam, Thank you for your valuable inputs/videos. I have one simple query that the example contains the explanation of instance segmentation and the data(nuclei images before training) are prepared as shown by you (13:50) one nuclei input image with separate masks of each nuclei. 1. My question is if I am trying U-net for nuclei semantic segmentation (dataset used by you) I believe my first nuclei input image and its mask (only one mask (all combined annotated masks of first nuclei)) 2. If I am trying MaskRCNN for semantic segmentation and I believe that the data preparation looks like the same discussed in question 1 (i.e., first nuclei input image and its only one mask image(containing all combined annotated masks) before training. Thanks and looking forward to your next videos/inputs/plans. Happy learning

[@AIwithSohini]

Hello Venkatesan. The answer to Q1 is yes, you are right, to use Unet for semantic segmentation you will combine all masks to one groundtruth image. For Q2, you should actually NOT apply MaskRCNN for semantic segmentation. The major issue is one of the utility functions computes bounding box around each cell nuclei mask. Now if you generate a composite image by combining all the nuclei, the bounding box around the combination will NOT reflect the actual regions of interest, in which case the loss functions will NOT converge as needed. So for MASKRCNN, the data should always be separated per region. You may however interpret the output of MaskRCNN as a single region (so all cell nuclei correpond to the same label), but the input format should not change. Hope this helps!

[@venkatesanr9455]

@@AIwithSohini Thanks for your valuable/helping inputs. Like to see your next videos/plans.

[@ushendrasengineeringtutori9450]

Not able to find the video on semantic segmentation. Can I have the link.

[@AIwithSohini]

kzbin.info/www/bejne/g2Wop5mvpLubmJY Hope this helps. Stay tuned!

[@sultantota1776]

you are awsome

[@AIwithSohini]

Thanks so much for the encouragement

[@kelixoderamirez]

permission to learn miss. where I can take this datasets? thank yoy

[@AIwithSohini]

In the description I have added the data set link github.com/AakashSudhakar/2018-data-science-bowl/tree/master/compressed_files. It appears in the code as well. Thanks for tuning in.

[@kelixoderamirez]

@@AIwithSohini oke. thank you, miss.

[@datasciencewitharbaaz5221]

What python version are you using for this project mam ?

[@AIwithSohini]

Hello, so for this project I had python 3.6. Any version between 2.7 to 3.6 would work for tensorflow 1x

[@mardeenosman8979]

Hi Sohini Thank you thank you thank you for amazing videos. I have few questions please 1. Can I use mask RCNN for DICOM images? 2. How would the data look like after preparing? You know its already at least 600 images and there would be masks for every object in all images?? I would love to see a sample please. Thanks alot for your time

[@AIwithSohini]

Hello, the dicom images by default are not supported by these models. So one way is to convert them to jpg and then applying the same methods. I have videos to help convert dicom images. Check out www.tensorflow.org/io/tutorials/dicom

[@vivekanandchimman2933]

Please video on this👇

[@shubhamchoudhary5461]

mam, why not we're using tf2 ??

[@AIwithSohini]

hey shubham, so the original codebase that we are reusing is in TF1, since we are reusing functions, this is the faster solution.

[@Jennifer-yz6uq]

🤣🤣🤣🤣🤣