PaliGemma by Google: Train Model on Custom Detection Dataset

[@rairorr]

The evaluation technique of visualizing the learning rate is something I’ve never seen before. Beautiful.

[@Roboflow]

I’m super glad you liked it. I’ve invested a bit of time to make it look good.

[@abdshomad]

Always the first to teach others all conputer vision tools ... Thank you ... 🎉

[@Roboflow]

You are always first to comment! Thank you… 🙏🏻

[@lucasbeyer2985]

Great dive, I'm happy to see you figured this all out without us documenting it well yet! For the issue about bad performance on card detection, I think most the reasons you listed contribute, except I don't think larger model is needed. 1) It suffers from the same issues as pix2seq (and all max-likelihood detection models) of being "conservative" and needing either lots of tricks/augmentations (see pix2seq) to fix that, or RL-tuning (see "Tuning computer vision models with task rewards" paper). 2) Order should be fine if trained for long enough 3) probably attention-only with SGD (what's done in the colab) fine-tuning is too weak for "harder" tasks like this. (not sure about this one)

[@Roboflow]

Thanks a lot! I have almost none intuition with this category of models, so any ideas and suggestions are super helpful. I intend to spend more time battling this issue. Going to start here: - arxiv.org/abs/2302.08242 - arxiv.org/abs/2109.10852

[@Roboflow]

Also if you need any help documenting it I’m happy to help ;)

[@lucasbeyer2985]

@@Roboflow yup, those two are good reads to start. Probably pix2seq first, as it introduce the general idea but also lots of hacks to make it work OK, and then RL-tuning second, because it then shows how to skip all of the tricks in a nice way (with RL), but in the writing, assumes you kinda know pix2seq already.

[@RipNonoRasta]

amazing work!

[@Roboflow]

Thank you!

[@gokayfem]

excellent tutorial!

[@trezero]

I'd love to see your take on Phi 3 Vision with Grounding Dino.

[@Roboflow]

I have not tried Phi 3 yet

[@Pingu_astrocat21]

Thank you for this :)

[@lovol2]

Fantastic. Thank you.

[@codeandrobotid2466]

Good job, bro!

[@AashishMtho]

VIdeo on inference please!

[@uttamdwivedi7709]

Great tutorial @roboflow !!! Is it possible to train same model for VQA as well as object detection? Can you provide any example of how the JSONL file should look like in such cases?

[@Roboflow]

We will soon add support for VQA datasets in roboflow. I plan to roll out tutorials covering this topic soon.

[@유영재-c9c]

I think the LLaVA families we are familiar with are not specific for detection or segmentation tasks, is that correct?

[@MABatin-hd1mt]

As always another great video. However, do you think few shot detection is possible with PaliGemma where I can fine-tune the model with as little as a single image rather than a large dataset like rf100?

[@kaifshaheem7546]

Once the model is saved in .npz format in locally, how it can load the model to make inferences with different case of input?

[@techtb2923]

did you find the answer??

[@marcc0183]

thank you for this. One question, I've been researching for a while and I haven't found a way. I want to train a model that has information about my trips, meal plans with friends, etc. so I can ask it. I don't know if it is best to use vector embeddings with images or use vision models to convert the images to text, and this text together with the metadata of the photos (with Google Photos API) create a RAG on the model that has information about this in text. What do you recommend vector embeddings with images or RAG with text? all the best

[@hegalzhang1457]

Greate work !Do you have example code for finetuning OCR task?

[@Roboflow]

Not yet. But I plan to play with OCR and VQA tasks finetuning.

[@mahirafser3736]

I am getting 0 mAP and blank confusion matrix on a medical image dataset (tooth endodonics). Can you tell me the reasons @Roboflow

[@thesuriya_3]

can you do as well as in VQA😊 using pytorch code

[@haricharan4474]

When I'm running the last step in the notebook I'm getting an error saying "An error occured when getting the model upload URL: 404 Client Error: Not Found for url" how to overcome this ?

[@BingxinYang-j7t]

if I want to fine-tune the PaliGemma model for the segmentation task， what dataset should I prepare?

[@Roboflow]

We don’t have a written tutorial but I made Google Colab some time ago. Would that be enough?

[@BingxinYang-j7t]

@@Roboflow sorry, I cannot find the Google Colab about fine-tuning the PaliGemma model for the segmentation task. Would you please give me the web address? Thank you very much.

[@JingyunYang-i1r]

@@BingxinYang-j7t Have you found it yet? Thanks!

[@BingxinYang-j7t]

@@JingyunYang-i1r No, I have not found it.😮‍💨

[@safiraghulam1862]

Hi, Can I fine-tune the model on a medical dataset? Currently, the model is not performing well on this data, and the results indicate that it is not suitable for medical data out-of-the-box. If I fine-tune the model on my dataset, which consists of approximately 200 to 300 images, will it work better? Additionally, is it possible to quantize this model to reduce its size from 3B to something smaller without significantly compromising its performance? Thank you.

[@Roboflow]

It is possible to fine-tune on medical images I done that on several use-cases like tumors detection.

[@safiraghulam1862]

@@Roboflow okay

[@toobasheikh106]

Hi, I fine tuned the model on a small medical object detection dataset with just 223 training examples and after training for more than 20 epochs it starts giving random predictions (classes that are not there in the dataset and even some spam classes). Could you please highlight what could be the reason?

[@Roboflow]

Can you send me the link to your dataset?

[@toobasheikh106]

@@Roboflow sorry it's a private dataset for organoid cell detection and has 4 classes

[@toobasheikh106]

It gives me random predictions like this: metaphase; talaga; ExecuteAsync; tooth; JvmSlicer#

[@suphotnarapong355]

Thank you

[@AkramKhanyt]

How did you deploy the model and also give code for how to download dependencies supervision of paligemma ........

[@Roboflow]

For now I deployed it simply by cloning this HF space: huggingface.co/spaces/big-vision/paligemma But we are wrapping up the work on github.com/roboflow/inference integration.

[@IceStormSerenade]

@@Roboflow Hii can you explain how the deployment works please? I haven't used gradio before so I am having difficulty with it...

[@president2]

What about liquid AI? I see a video from years ago. Any updates on this process?

[@ttkrpink]

Great tutorial. I followed the tutorial online, everything worked great. When I tried on a local machine, I got the following error on the line: for image, _, caption in make_predictions(validation_data_iterator(), num_examples=4, batch_size=4): ValueError: Sharding GSPMDSharding({devices=[3]

[@Roboflow]

Do you have multiple GPUs locally?

[@ttkrpink]

@@Roboflow yes, I do have 3 GPUs

[@Roboflow]

​@@ttkrpink please try using `batch_size` value that is devisable by 3. 3, 6, 9, 12, 15 something like this. It looks like it tries to split your data equally across all the GPUs but you set batch_size=4.

[@shubhamkumarjha1868]

What is the best model for training multiple bounding boxes

[@Roboflow]

If we talk about VLMs, that’s probably Florence-2

[@shubhamkumarjha1868]

@@Roboflow Thanks for the reply. 😀

[@darklord96423]

Good job, bro, but don't u think that not including the answers bout photo, Yolo is better than paligemma. I mean in detection

[@Roboflow]

YOLO is definitely a better detection model, so if you don’t care about other capabilities, you’ll probably be better off staying with YOLO.

[@유영재-c9c]

jax is difficult for me. huggingface is right for me 😅