SAM 2 | Segment Anything Model 2

Meta AI has released Segment Anything 2 (SAM 2), an advanced image and video segmentation foundation model. SAM 2 allows users to input points in an image to generate segmentation masks for those points, and it can also generate and track segmentation masks across frames in a video.
Segment Anything 2 (SAM 2) by Meta AI is open-source. It is a follow-up to the original Segment Anything Model (SAM) and is designed to enable zero-shot segmentation of objects in images.
Building on the original SAM model released by Meta last year, SAM 2 has been utilized for various vision applications, including image segmentation and as an image labeling assistant. Meta reports that SAM 2 is six times more accurate than its predecessor in image segmentation tasks.
In this guide, we will explore what Segment Anything 2 is, how it functions, and how you can leverage the model for image segmentation tasks.
Steps to run SAM 2:
conda create -n samm python=3.12
conda activate samm
git clone github.com/fac...
cd segment-anything-2
python setup.py build_ext --inplace
%cd checkpoints
double click on it. Model checkpoints will be downloaded in your directory.
Then install this- we need this to us the SAM 2 predictor and run the example notebooks,
pip install --no-build-isolation -e ".[demo]"
conda install jupyter notebook
jupyter notebook

[@mylifeasaarav]

I appreciate you creating these videos on modern technology.

[@CodeWithAarohi]

Thank you so much!

[@World-um5vo]

Madam if we want to fine tune the model for videos based on mask decoder how to do it.

[@nicolasarayacaro94]

Thanks Aarohi, greetings from chile

[@CodeWithAarohi]

Welcome 🙂

[@shekharkumar1902]

As usual great video. Image segmentation. Two questions.. 1st how to find the appropriate point cordinate of the object of interest? 2nd Can you add demonstrations of image and video segmentation other than the existing git repo? Just to see actual model response?

[@CodeWithAarohi]

You can use object detection model to find the coordinates and then pass those coordinates to SAM in order to put mask on.

[@devavratpro7061]

@@CodeWithAarohi But What if I want reverse operation? How to perform object detection using SAM? Any suggestion would be help in better understanding.

[@kalkagyani1726]

I just wanna know how can we use this model to our project, as an AI enthusiast, one of biggest question is that, is their any need to create our own model now, or we have to change few lines in others jupyter notebook code claim it is as ours? BTW great explanation

[@CodeWithAarohi]

SAM 2 is open-source and designed to enable zero-shot segmentation of objects in images. The decision to create your own AI model versus adapting existing ones depends on several factors: 1- If you have a unique problem or a highly specialized dataset that existing models are not well-suited for, creating your own model may be necessary. 2- If you need a model that is highly optimized for your specific use case, including performance, accuracy, or computational efficiency. 3- If owning the intellectual property of the model is important for your business or research. For many common problems, such as image classification, object detection, and text generation, existing models are mostly sufficient and can be fine-tuned or adapted to your specific needs. Training a new model from scratch can be resource-intensive in terms of both time and computational power, so fine-tuning existing models can be a more efficient approach.

[@anujshah7350]

How can we label the identified masked, let's say list of labels? what could be thought process sort of? coming from context like template matching

[@CnnrVizuals]

so how can you actually export the masked out subject as a separate video without the extra stuff? Where are the elements of the graph and other assets

[@CodeWithAarohi]

Noted!

[@JayYoung-r5s]

This is very helpful to me. However, regarding the details of the memory module, I still need to study them further. I'm just a beginner 😭

[@CodeWithAarohi]

Glad it was helpful!

[@RatherBeCancelledThanHandled]

Thanks . Great explanation 👍

[@CodeWithAarohi]

Glad it was helpful!

[@lorenzoleongutierrez7927]

Great explanation, thanks a lot 😊

[@CodeWithAarohi]

Welcome 😊

[@GianmarcoGoycocheaCasas]

Thanks Madam, great explanation 👍

[@CodeWithAarohi]

You are welcome 😊

[@nitishrathore14]

Hi can you create a video using tensorRT on SAM2 to reduce inference time?

[@CodeWithAarohi]

Noted!

[@shantilalzanwar8687]

Gr8 video as usual. Pls share code. Also why it is 6x faster ?

[@CodeWithAarohi]

SAM 2 utilizes a more efficient model architecture compared to SAM, which reduces the computational complexity and speeds up the segmentation process.

[@shantilalzanwar8687]

@CodeWithAarohi need to run it and chk. Tt

[@deemon101]

ok, but if we compare SAM 2 to Yolo, then what are the differences, also I guess different use-cases? I mean, with yolo, I train the model to detect a certain item on a picture or a video, and after that I can feed it random new video/picture and my items will get detected. But with SAM 2, is there a similar use case?

[@work.aniketkumar]

SAM is segmentation mode designed to segment anything in an image where as yolo is object detection model designed to identify object in video. Correct me If I am wrong.

[@deemon101]

@@work.aniketkumar you're correct. My current usecase is to use SAM2 for dataset creation. So I created a tool that uses SAM2 to create segment label files for images. Basically I created a roboflow copy, but it all lives in my workstation.

[@devavratpro7061]

@@work.aniketkumar remember, Mask-RCNN has both segmentation and detection. So can we think of a pipeline using SAM that is similar to Mask RCNN?

[@SaumyaSrivastava-y1d]

From where to get python codes for this?

[@darkmatter9583]

can you do it with videos like the blog meta demo please?

[@CodeWithAarohi]

Sure

[@shantilalzanwar8687]

Aarohi, pip install -e . is not working, you said in description details provided but no details found.

[@CodeWithAarohi]

I have added now. Please check description section.

[@shantilalzanwar8687]

@@CodeWithAarohi thanks

[@shantilalzanwar8687]

Your videos are great help for new people like us. WOndering why you choose anaconda over Google colab for SAM 2 ? ALso in SAM1 they provided colab notebooks from Meta, did they provided those this time ?

[@avinashrai7588]

Mam Can we run it on Jetson agx orin

[@CodeWithAarohi]

Yes

[@mouizjarral6486]

Highly Appreciated Aarohi. When you have time kindly make a videos on Semi and Self Supervised Learning. A Humble request. :)

[@CodeWithAarohi]

Noted. Sure I will make videos on Semi and Self Supervised Learning.

[@mouizjarral6486]

@@CodeWithAarohi Thank you so much. Stay Blessed ✨

[@shantilalzanwar8687]

did you faced this error ? --------------------------------------------------------------------------- OSError Traceback (most recent call last) Cell In[1], line 5 3 os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1" 4 import numpy as np ----> 5 import torch 6 import matplotlib.pyplot as plt 7 from PIL import Image File ~\AppData\Local\anaconda3\envs\sam2\Lib\site-packages\torch\__init__.py:148 146 err = ctypes.WinError(ctypes.get_last_error()) 147 err.strerror += f' Error loading "{dll}" or one of its dependencies.' --> 148 raise err 150 kernel32.SetErrorMode(prev_error_mode) 153 def _preload_cuda_deps(lib_folder, lib_name): OSError: [WinError 126] The specified module could not be found. Error loading "C:\Users\ss7a0328\AppData\Local\anaconda3\envs\sam2\Lib\site-packages\torch\lib\fbgemm.dll" or one of its dependencies.