Florence-2: Fine-tune Microsoft’s Multimodal Model

Learn how to fine-tune Microsoft's Florence-2, a powerful open-source Vision Language Model, for custom object detection tasks. This in-depth tutorial guides you through setting up your environment in Google Colab, preparing datasets, and optimizing the model using LoRA.
Chapters:
- 00:00 Introduction: Unlock the Power of Florence-2
- 01:09 Getting Started: Prepare for VLM Fine-Tuning
- 03:55 Florence-2 in Action: Explore Pre-trained Capabilities
- 07:00 Dataset Deep Dive: PyTorch Data Loading for Florence-2
- 13:02 LoRA: Optimize Your VLM Training
- 14:21 Fine-Tuning: Unleash Florence-2's Custom Object Detection
- 17:30 Model Evaluation: Measure Your VLM's Success
- 21:37 Florence-2 vs Other Computer Vision Models
- 24:09 Conclusion and Next Steps
Resources:
- Roboflow: roboflow.com
- 🔴 Community Session July 3th, 2024 at 08:00 AM PST / 11:00 AM EST / 05:00 PM CET: roboflow.stream
- ⭐ Notebooks GitHub: github.com/roboflow/notebooks
- 📓 Florence notebook: colab.research.google.com/git...
- 🗞 Florence-2 arXiv paper: arxiv.org/abs/2311.06242
- 🗞 Florence-2 overview blog post: blog.roboflow.com/florence-2
- 🗞 Florence-2 fine-tuning blog post: blog.roboflow.com/fine-tune-f...
- 🔗 Florence-2 HF Space: huggingface.co/spaces/gokaygo...
- 🗞 Mean Average Precision (mAP) blog post: blog.roboflow.com/mean-averag...
- 🗞 Confusion Matrix blog post: blog.roboflow.com/what-is-a-c...
Stay updated with the projects I'm working on at github.com/roboflow and github.com/SkalskiP! ⭐

[@abdshomad]

I've been waiting for this tutorial for days. Thank you again for being the first to comprehensively review this new model. Super exited! 🎉🥳

[@Roboflow]

As usual you are the first one to comment on the video! Thanks a lot for all the support! 🔥

[@jk_c66]

thank you roboflow for providing such nice and lovely tutorials for free and with a nice instructions

[@Roboflow]

Pleasure!

[@artem-yw8km]

Thank you for this turtorial, was working on these kind of setup for a couple of days. You definetely could save lot of time

[@Roboflow]

Sad I didn’t save your time this time.

[@SatyamKumar-cb2mt]

Thanks a ton for this awesome video! Every single term is explained so clearly-it's super helpful. I can't wait to dive in the code and start putting this knowledge to use!

[@Roboflow]

Thanks a lot! I really put an effort and try not to fall into a bias (not assume that people know those things).

[@SridharanS-vz7re]

how to train this model on custom dataset for OCR

[@kylewang6704]

Thank you for the awesome tutorial! I wonder what about the detection accuracy comparing to YOLO based model?

[@Roboflow]

We cover that topic in the video ;) something tells me you didn’t watch till the end.

[@sandrojunioraraujo3706]

Wonderful tutorial! Could you make a tutorial about how to fine tune florence 2 for the segmentation task?

[@suphotnarapong355]

Thank you

[@Jordufi]

Nice video, as usual

[@Roboflow]

Thanks a lot!

[@user-mc7tg4pf3i]

Hell Sir Thanks for your all videos and efforts. I am following your channel, but I request you please upload one detail video on how to finetuning Yolov5 model for custome images classification.

[@Roboflow]

Does YOLOv5 support classification?

[@user-mc7tg4pf3i]

@@Roboflow Yes

[@VLM234]

Very informative video. Thanks for making auch a valuable video free of cost. Just one request when your you make tutorials if possible try to do inferencing, training or fine tuning on agricultural or satellite related data.

[@Roboflow]

Next time I will try to find some cool datasets from this domains

[@yj1548]

Good video.and I'm curious about what can be done to improve mispelled class names on object detection tasks，do you have any ideas?

[@Roboflow]

I think you asked me this question on Twitter, but let me answers here as well. 1. Longer training could fix it. 2. Fuzzy class matching. In the video we filter out anything that is not exact match. Hamming distance for example.

[@dabaizhang-x5b]

Master, could you please tell me if Florence-2 can perform SER (Semantic Entity Recognition) and RE (Relation Extraction) tasks? If so, what should my dataset look like? 🤔

[@jimshtepa5423]

can you please upload recording of teh community session for those of us who are in different zone or might otherwise miss the call?

[@Roboflow]

Sure! All our community sessions are available to re-watch on YT channel

[@3DFinalCut1]

Thank you for this very informative video. Something like this helps enormously. Dziękuję! I have a question and maybe someone here can give me a tip. I am looking for a tool that searches for similar images in a folder and shows me the results so that I can clean up the data set later. I have already found tools that do this, but they mainly work with image hashing methods or use fuzzy matching algorithms. But I wonder if there aren't already tools that use AI to solve this task. Does anyone use such a tool?

[@Roboflow]

You can make it happen using CLIP model. We covered that topic here: kzbin.info/www/bejne/j6mtnHhse9t6fJY

[@8-P]

For the community session I have a couple of (beginner) questions: - the google collabs on roboflow seem to be linux based, is there an easy way to make them work on windows? - in general, how do I download a model (YOLO) to use in a python app (on windows) - are there models that would run for realtime video detection on a regular laptop with an integrated iGPU? - I am planning to use a YOLO model for a sports live stream, but only have a simple 3 Year old mid range laptop on me - would it be better to send the stream over to my desktop PC with an Rtx3060Ti-8GB and let the model run there (and send back the detection back and sync on the laptop) - if a laptop is underpowered? - for simple applications, like the realtime sports detection of yours, would it be better to run it on my own hardware or investigate in cloud servers for inference? Thank you very much for your tutorials, the help a lot!

[@geniusxbyofejiroagbaduta8665]

Thanks Sir. Please do fine-tuning for Oct, captioning and segmentation task

[@Roboflow]

Did you tried to run OCR with pre-trained model?

[@UhuruSsemakula]

Can you please make one for Object detection using web camera?

[@Roboflow]

You mean using Florence-2 and Webcam? Or webcam in general?

[@UhuruSsemakula]

Yes Florence-2 and Webcam

[@richardobiri2642]

what if I want to detect fake and authentic certificates ? please any help

[@geniusxbyofejiroagbaduta8665]

Please sir also tech us how to annotate with it

[@Roboflow]

You mean how to automatically annotate images?

[@bladethirst1]

Is this applicable to grade handwritten pdf math assignments?

[@Roboflow]

Florence-2 can be really good at OCR processing of handwritten text. Not sure about math equations. We would need to confirm that.

[@bladethirst1]

@@Roboflow {'': 'In this image we can see a book with some text on it.'} This is the test output of a handwritten math problem deduction, is there someway to get more detailed caption or the OCR output?