YOLO-NAS + SAM : Image Segmentation Using YOLO-NAS and Segment Anything Model

[@DinitoThompson]

Would be nice if you could link the source code in the description

[@helloansuman]

How to train this on 8 GPUs on DGX A100?

[@CodeWithAarohi]

docs.deci.ai/super-gradients/documentation/source/device.html#4-ddp-distributed-data-parallel

[@helloansuman]

@@CodeWithAarohi thank you. I tried this but I will try again

[@balveersingh3051]

wonderful

[@CodeWithAarohi]

Thank you!

[@nandiniloku7747]

excellent explanation mam.. can you please show me how to save the segmented image (not the mask i want only segmented image to be saved in the other folder

[@junioraguilar7880]

could you provide the notebook of this video?

[@sebaarr]

Thanks, good video! it's possible to run the YOLO-NAS + SAM in a Jetson Nano?

[@CodeWithAarohi]

Will try this on Jetson Nano and reply you back 🙂

[@ZeynepC3]

Thanks for your videos! Do you think YOLO-NAS + SAM is suitable for realtime applications? Have you ever checked FPS? I'm curious about your thoughts.

[@CodeWithAarohi]

I think Yes, YOLO-NAS is a better choice as compare to yolov8 because of the quantization blocks it has.

[@전주석-m5f]

Hello. Thank you for the good video. I'm leaving a message because I have one question. Is it possible to segment objects that have a hole in the middle, like a donut? And if you can, what labeling tool do you use for things like donuts? Most labeling is in polygon format, but I don't know what to do in cases like donuts, so I'm asking.

[@CodeWithAarohi]

Yes, you can segment objects with holes like donuts by using image labeling tools that support multiple polygon annotations for a single object, enabling you to outline both the outer shape and the inner hole separately. Tools like Labelbox, VGG Image Annotator (VIA), and Roboflow offer this functionality.

[@cocon8287]

what are the benefits of using yolo nas instead of just conditioning sam with a text prompt "person" ? I am newbie in computer vision but not deep learning

[@priyanshupandey3148]

I have different folders for different classes of images with it's mask images in the same folder. I want to perform segmentation with YOLO+SAM. Please suggest other advance possible approaches and codes.

[@jasonl2860]

what if there is some other things inside that person bounding box, will the SAM model segmented those things as well? thanks

[@CodeWithAarohi]

We need to train our object detection model for those other things. And when our Object detection model will generate the bounding box for those other things. All the boxes will be the input to SAM and SAM will generate Masks for all of them.

[@TY_B_39_SejalParate]

how can i use this code to perform instance segmentation in real time using a laptop's webcam?

[@CodeWithAarohi]

You need to change the source of input to webcam of our laptop.

[@jeffreyeiyike122]

I dont know why the detection_pred= model.predict(image_path, conf=conf_threshold)._images_prediction_lst is not print the results. I want to extract the use the image feature vectors, bounding box for another neural network. I have tried the code but not get the rsults

[@NnannaEze]

Did you find the solution to this? I also experienced this

[@jeffreyeiyike2358]

@@NnannaEze not yet.

[@ArbaazShaikh-y3t]

what if I have multiple person in the image, YOLO-NAS will detect all the persons and will those all person will get SAM segmentation?

[@CodeWithAarohi]

Correct

[@ArbaazShaikh-y3t]

@@CodeWithAarohi I have a bounding box data of "face" of person in an image, from that bounding box can I segment the entire person body. If yes can you please explain how ? one thing that thought was to get the centroid of the face bounding box and use that centroid point as the input for SAM model rather than using bounding box. From the centroid point can i get the entire person body segmented ? in the demo of SAM, I can see we can but with the code how ? can you please explain mam. It would be really helpful. Thank you in advance.

[@Andre_Villela]

I don't understand how do you select just the person class to be the "zero class" on coco dataset.

[@CodeWithAarohi]

Person class have 0 id in coco dataset. I just ran the Object detection model which gave me the id of Object detected in the image which is Person in this case.

[@jyothir07]

Can't we use SAM alone to get the bbox instead of using 2 DL models? by getting the min rectangle for the segmented part alone.

[@CodeWithAarohi]

As per my understanding, The Segment Anything Model (SAM) predicts object masks given prompts that indicate the desired object. Prompts can be provided via the predict method to efficiently predict masks from those prompts. The model can take as input both point and box prompts. Also, SAM model predicts object masks only and does not generate labels. Suppose you have a image which have person and bus class. SAM will put masks on both the objects (it will even put the mask on subparts of the objects like separate mask for tyre, windows etc.). We can get a mask for specific object using point or box prompts. Suppose we chose box prompt to display mask. When we pass our image to SAM, it will give us boxes for all the objects. But we only want a box on Person. To put the mask on the box that corresponds to a person, you need to first identify the mask that represents the person. You can do this by checking the label associated with each mask and selecting the mask with the label "person". SAM is not trained on corresponding labels. Here you need a detection algorithm which will give you the box co-ordinates of the object you are interested in.

[@jyothir07]

I agree, but using the point prompt we can segment the object as shown in the official notebook. we could use an interface for that and then simply take the minimum area rectangle of the contour of interest using opencv. correct me if wrong.

[@tusharvora3921]

@@jyothir07 you can definetly do that as long as you are not interested in class labels ]

[@meg33333]

Maam can you also make a model which convert hindi text or sentence to image. If possible pls give your tips.

[@CodeWithAarohi]

I will try

[@meg33333]

@@CodeWithAarohi Hindi text to sign language images something like that bec now a day AI become more advance so this project may be benefit to others.

[@deepakts8941]

hi mam, which Yolo model is better for small object detection ( like grains ) now I am using Yolo v7 can I switch to V8 or NAS

[@abdurrahimbalta1015]

you should try the sahi algorithm

[@deepakts8941]

@@abdurrahimbalta1015Sahi we have used we want our model to run quickly with less gpu usuage

[@abdurrahimbalta1015]

@@deepakts8941 okey, maybe try the efficent-net. i think best cpu object detector model

[@robertocenci3834]

Hi Aarohi, would it be possible to download your sample code used in the video? Thank you

[@CodeWithAarohi]

Hi, This same code Ultralytics implemented in their latest ultralytics package. Check this video: kzbin.info/www/bejne/gZC6iqCpjs11e9k

[@grayelearning772]

Good work. Please share the code

[@CodeWithAarohi]

Please email me at [e-mail]

[@philipplagrange314]

Is it possible that the background does not get a mask? It is currently displayed in purple

[@CodeWithAarohi]

Yes it is possible.

[@philipplagrange314]

Can you show me how to do this? The background should still be visible, but without a mask. That would be really helpful!

[@assiachahidi5162]

Thanks for your videos. Can you consider sharing your code please.

[@grayelearning772]

Please share the code

[@CodeWithAarohi]

Please email me at [e-mail]

[@盛毅]

where is the source code please??

[@pearlmarysamuel4809]

❤

[@world66699]

# Extract desired outputs bboxes_xyxy = detection_pred[0].prediction.bboxes_xyxy.tolist() confidence = detection_pred[0].prediction.confidenceftolist() labels = detection_pred[0].prediction.labels.tolist() print("Bounding Boxes (xyxy):", bboxes_xyxy) print("Confidence:", confidence) print("Labels:", labels) bboxes_xyxy, confidence, labels TypeError: 'generator' object is not subscriptable

[@shubhamrathod9969]

Since the model outputs a generator. Convert the Generator to a list and then Extract the desired output. model_pred = list(model_nas.predict(image, conf = conf_threshold)._images_prediction_lst)