How to Train DETR Object Detection Transformer on Custom Dataset

[@abdshomad]

When i see notification appeared from Roboflow, I always press like button before watching the show 👍😁

[@abdshomad]

Always quality content.

[@Roboflow]

@@abdshomad thanks a lot! I’m so happy that someone is clicking on those notifications 🔔

[@sihameghaprabashwara6025]

​@@RoboflowIs there any way to detect in a video ?

[@Megan-gl7pi]

Great tutorial. I'm going to train Detr on a custom medical dataset with x-ray images to detect bolus, stent, catheter, etc. Hopefully this will turn out to be fruitful. There hasn't been much research using Detr on medical datasets yet.

[@soumenhalder2811]

Best ever Deep Learning video I have found. please make such video

[@ALOKAGIMHARI]

Thank you for your tutorial.❤ Let me know how to use this trained model for test an image thats not in test dataset.

[@raymond-andrade]

Great Video. I resonate with your intro, on how it's hard not to get imposter syndrome in this fast changing AI/ML landscape, when you don't know how to implement the latest "thing".

[@Roboflow]

Thanks a lot. At the same time there is nothing better than just to face your latest “thing” and learn!

[@venusshah4476]

Hey, This is a great lesson, do you have any idea that this way we can also train Grounding Dino on our custom dataset?

[@MiaLovEm]

Thank you for such a great tutorial! I have learnt a lot. And I also watched other videos you made. They really helped me understanding the models and training process better. About the false detection you mentioned, I came acorss similar problems on my dataset. I'm thinking maybe add null images to the dataset (about 10%) could help? I am just about to try!

[@arfashaha4074]

Thank you so much for the great video! Your quality content has helped me a lot in my thesis ☺. Just wanna ask, can we apply our trained model to predict a video/webcam directly?

[@Roboflow]

Not entirely “directly” but you can use tools in Supervision to process video

[@arfashaha4074]

@@Roboflow Ah I see, is the concept the same as your video about detect and count object in polygon zone?

[@Roboflow]

@@arfashaha4074 yes exactly! Generate frames from video. Process frame. Annotate. Save to result file.

[@arfashaha4074]

@@Roboflow Perfect, thanks for the help! You've been wonderful

[@Roboflow]

@@arfashaha4074 thanks a lot for kind words!

[@katogyn]

annotations are not found during the evaluation process any way out for this model

[@Roboflow]

Would you be a bit more specific?

[@helloansuman]

Nice video. Can you please tell how to run the training on multi-gpu? Which lines to be added or modified?

[@MuhammadAdnan-tq3fx]

Thank you so much for this video. i followed your video and received this results Accumulating evaluation results... DONE (t=0.16s). IoU metric: bbox Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.546 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.766 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.629 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.328 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.558 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.578 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.605 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.605 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.343 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.617 please guide me how i can increase my value upto 95%. i use same parameter that you have used in your video.i will be very thankful to you

[@Roboflow]

Did you used the same dataset we did?

[@MuhammadAdnan-tq3fx]

I used the "SeaShip" dataset, downloaded from the Roboflow website, which consisted of 7,900 training images and approximately 1,661 validation images. For both the train_dataloader and val_dataloader , I set the batch size to 4. training and model parameters was: model = Detr(lr=1e-4, lr_backbone=1e-5, weight_decay=1e-4) trainer = Trainer(devices=1, accelerator="gpu", max_epochs=50, gradient_clip_val=0.1, accumulate_grad_batches=8, log_every_n_steps=5) on these i received the results that i have uploaded in my first comment. I again starts a second training iteration with significant adjustments to hyperparameters. In this instance, I enlarged the training dataset to 9,843 images while simultaneously reducing the validation dataset to a mere 300 images., also increase the epochs up to 100 and for train_dataloader batch_size=4 and for valid_dataloader batch_size=2, also reduce the learning rate from lr=1e-4 to lr=1e-5. during first time total training time was 23 hours but this time my training is still running from last 18 hours it will take one more day i think.after that i will share the result again. and one more thing i have NVIDIA rtx 6000 (24GB)

[@MuhammadAdnan-tq3fx]

@@Roboflow Second time accuracy comes 81% how can i increase the accuracy.

[@anaisabelrua189]

Loved it!! have you tried using DETR with Detectron2? BTW, it wasn´t long for me. Very precise learned a lot!

[@Roboflow]

Thanks a lot for that positive feedback. I actually didn’t trained DETR with Detectron2. I came a bit late to the party and went straight with Transformers.

[@canciones381]

It is possible mix this training custom data set with transformers with the PolygoneZone tool? Great Video!

[@10straws59]

Thank you so much for this tutorial! I am getting this error: KeyError: 'annotations'. I think this is because "predictions" is an empty dict, so is "results". However "orig_target_sizes" is not empty. I have tried with 20 epochs, is it because the model is not learning well enough to be able to actually make predictions? Furthermore, I tried to annotate images in the train and val datasets, and I am not seeing any detected boxes around the objects (when comparing to the ground truth) Would rly appreciate some help regarding this issue!

[@Roboflow]

Could you create new issue here: github.com/roboflow/notebooks/issues and describe all those issues.

[@SalmanFarsi-qt6fr]

Bruh i am also getting this type of error continuously please if you have come up with it guide me too. Will really appreciate ❤ Thanks

[@afriquemodel2375]

NameError: name 'image_processor' is not defined

[@umarkhalid5463]

Can this be applied to realtime surveillance cameras to detect shoplifting?

[@Thedrive17]

Thanks for the great video! Just wanna ask can we detect image from image that we upload in colab?

[@Roboflow]

Thanks a lot. I’m not sure I understand. Could you rephrase the question?

[@accongnetwork1208]

@@Roboflow Hi, may I ask you a question? How can we show the IoU after the detection?

[@Roboflow]

@@accongnetwork1208 and I responded a month ago :) IoU of what? Detections and ground truth?

[@Thedrive17]

⁠​⁠@@Roboflow Yes, the IoU of detections and ground truth, I dont know how to show the coordinat of detections, and Is there any code to show the IoU of Ground Truth and Detections automatically? Thank youu so much

[@chrisboutsikas5973]

Is there reason in particular why you install a spesific verison of supervision (i.e. supervision==0.3.0) ?

[@body1024]

amazing tutorials keep it coming

[@Roboflow]

Thanks a lot! Stay tuned. I already have idea for next one ☝️

[@SalmanFarsi-qt6fr]

I am getting the 'Key Error'='annotations' And Type of Error: Detections.___Getitem__ not supported for index of type .... Please Do guide me with the following error how can i come up with it. Will really appreciate it.

[@sangamithramurugesan8178]

Hello, the acuial purpose of DETR is to ELIMINATE the use of NMS. But your code uses NMS. why?. obtained from the original paper:"Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression procedure or anchor generation that explicitly encode our prior knowledge about the task"- PLEASE EXPLAIN

[@Roboflow]

The answer is pretty simple. I run the model and faced a lot of problems with double detections. Results were pretty much unusable without NMS and that’s why I added it.

[@ComedyArchives]

great video, thanks!

[@Roboflow]

Pleasure!

[@aaminataskeen3448]

Hi, very informative video! I’ve been working with DETR for mitosis object detection and your code has been of great help. But when im training the model, the loss is decreasing very very slowly. Like even after 50 epochs my loss has gone down from 3.7 to 3.1 . So I was thinking maybe it’s coz of the learning rate. Is there any way I could change the learning rate coz when I tried doing that during DETR class instantiation, it threw an error while error. So as of now I can’t change the learning rate and my loss is almost stabilized at 3 and just fluctuating between the above mentioned values. Appreciate your reply :))

[@QuickKeysAcademy]

Great video👍, I have used my own dataset and got annotations using roboflow. I got this error can you please resolve this --> 243 raise TypeError( 244 f"Detections.__getitem__ not supported for index of type {type(index)}." 245 ) TypeError: Detections.__getitem__ not supported for index of type .

[@Roboflow]

What is the version of supervision that you have installed?

[@QuickKeysAcademy]

@@Roboflow torch: 1.13 ; cuda: cu116 ; supervision: 0.3.0 ; transformers: 4.26.1

[@Roboflow]

@@QuickKeysAcademy wow, this is unexpected. Could you please create a bug report here: github.com/roboflow/supervision/issues and attach link to your notebook. I'll try to debug it live.

[@QuickKeysAcademy]

@@Roboflow ok thank you

[@SalmanFarsi-qt6fr]

Bruh did You resolved this issue. If yeah then guide me too i am also getting this for too long. Will really appreciate Thanks❤😊

[@leandrodiascarneiro5663]

Excellent video. Thanks!!

[@chandank5266]

Hey Roboflow bro! I found this vidio really helpfull and I have a question striking me: where to use the checkpoint, I want to load the model using checkpoint that got saved while logging inside lightning_logs and not by the config file saved in custom-model directory; is there a way?

[@nimrakhan5785]

which object detection model is best for accuracy, Yolov8, dert, transofrmer or Faster RCNN?

[@Roboflow]

You only care about top accuracy? Speed does not matter? Or top accuracy in real time detection?

[@iMJKhan]

@@Roboflow Thanks for the great content. How about each use case? 1. Accuracy, 2. Speed, 3. Accuracy in RT-detection, and 4. Speed in RT-detection

[@michaelscheinfeild9768]

HI THANKS FOR VIDEO. Where i can read article of model

[@AmarnathReddySuarapuReddy]

Great lesson

[@nhauwu]

I tried to use it with my own model, but it said "NameError: name 'image_processor' is not defined" in the first code cell of "Create COCO data loaders

[@Roboflow]

I’d need more details. Could you create Agnew thread here: github.com/roboflow/notebooks/discussions/categories/q-a and give me a bit more details?

[@sihameghaprabashwara6025]

Is there any way to detect in a video ?

[@mahachaabene6750]

great content! I have tried using it on my own dataset but the labels were not correct with the coco format, although i tried yolov5 and it did good . any explanation for that ?

[@Roboflow]

Boxes were detected but with incorrect labels?

[@mahachaabene6750]

@@Roboflow yes that"s exactly what happened

[@mahachaabene6750]

my dataset contains numbers stamped on metal in a curved way, the model was detecting the boxes but with the wrong labels in this case, but it worked fine when i tried yolov5

[@Roboflow]

@@mahachaabene6750 any pattern? Some numbers are always generated as other number? Maybe there is some mapping problem?

[@mahachaabene6750]

@@Roboflow it seems like each number is detected as its successive number, like 1 is detected as 2, 3 detected as 4.....

[@bharanis1004]

hey any idea on how to use pretrained detr

[@rezaazhar1292]

hi i wanna ask. can i deploy this model to mobile android or drone like dji tello?

[@Roboflow]

You can convert the model to ONNX and in this format it is deployable on Android Phones. Maybe there are other ways, but I’m pretty sure this one should work.

[@МаксимМалахов-у4ш]

How do I use my saved pre-trained model in a new laptop with my own weights?

[@DigiDriftZone]

What is the impact of small batch sizes on inference accuracy? - I only have 8gb of vram on my 3060ti and for the larger models I’m only getting batch sizes of 1 or 2.

[@محمدالفقى-ي4ب]

AttributeError: type object 'Detections' has no attribute 'from_coco_annotation' I didn't find any raise issue on github or any stackOverflow so python``` sv.Detections.from_coco_annotation ``` and i get this error

[@Roboflow]

What version of supervision you use? Tutorial was dedicated for 0.3.0. from_coco_annotation got drop from supervision API.

[@biswajitroy-zp6lk]

TypeError: Detections.__getitem__ not supported for index of type . Hi this is a error i am getting when i am about to detect bounding box on the test dataset .i have one class

[@Roboflow]

Hi 👋! Is that happening with the current version of the notebook without any changes?

[@tomaszbazelczuk4987]

Piotr thanks a lot👍

[@Roboflow]

Our viewership in Poland 🇵🇱 grew a lot over the last few months.

[@felipegerard1613]

Hi, first thanks for this great work, I am having a problem if I try to create this notebook on my development server, when I use "inputs=image_processor(images=image,return_tensor='pt').to(DEVICE)" (device(type='cuda', index=0)) I get in transformers/feature_extraction_utils.py:229 "TypeError: is_floating_point(): argument 'input' (position 1) must be Tensor, not list" any help will be very appreciated

[@俊凱-c8x]

Hi, thanks for your great video!! I am trying to train detr on lab server. I tried to set up the environment, but when I typed pip install -q ./transformers and pip install -q pytorch-lightning in terminal. I got the following errors ERROR: Invalid requirement: './transformers' Hint: It looks like a path. File './transformers' does not exist. Can you help me with this? Thanks a lot !!

[@Roboflow]

I’m a bit confused. Why ./ before transformers? I believe it should not be there :

[@俊凱-c8x]

@@Roboflow oh, that's my fault ! Thank you!

[@Roboflow]

@@俊凱-c8x no problem! 😉

[@afriquemodel2375]

how to configure to work in local environnement

[@tongtongchen-qo6qj]

AttributeError: 'ToTensor' object has no attribute 'pad' that How to solve this problem ?

[@МаксимМалахов-у4ш]

How do I count the various objects in the photo?

[@Roboflow]

You can do it quite easily, for example like that: count = len(sv.Detections.from_transformers(transformers_results=results))

[@МаксимМалахов-у4ш]

@@Roboflow And how do I determine the number of objects for each class in the photo?

[@Roboflow]

@@МаксимМалахов-у4ш we do not have that functionality yet, but I'll add it to our roadmap. In the meantime you can do something like that: { int(class_id): len(detections[detections.class_id == class_id]) for class_id in np.unique(detections.class_id) }

[@МаксимМалахов-у4ш]

@@Roboflow Thank you very much!

[@pranks4167]

Hello Peter, I waana you to become my teacher , how can we do it ?

[@kilikia8939]

Why you dont show, how look annotation?

[@afriquemodel2375]

it is possible to do the same thing in local environnement??

[@Roboflow]

Yes. But you really should have GPU train it.

[@afriquemodel2375]

@@Roboflow i have Gtx nvdia 1050 4Go

[@seroshmannan]

how can we tweak with network block

[@varshanathkm6947]

Is there any way to detect in a vedio

[@Roboflow]

Yes! you can use process_video function from supervision package: roboflow.github.io/supervision/video/#process_video to automatically process every frame from video.

[@olanrewajuatanda533]

@@Roboflow the link leads to nowhere. Please, can you check again? I'd appreciate if you can do that. Also, how can one see the mAP, F1 and Recall from the training and validation?

[@muhdnajhan886]

what code for random run any image

[@YahyaSamet-vi3pu]

how to test this model in real time

[@DrShroom.]

Cool!

[@TheArkLade]

Does anyone know why " __init__() got an unexpected keyword argument 'gpus'" error shows up when training with pytorch_lightning? It used to work...hummmm. Its this part: "trainer = Trainer(gpus=1, max_epochs=MAX_EPOCHS, gradient_clip_val=0.1, accumulate_grad_batches=8, log_every_n_steps=5)"

[@Roboflow]

Did the error accrue when you run our notebook? Or you have done some custom changes?

[@dreamin4444]

@@Roboflow Tak, na waszym notebooku. Mam ten sam problem :/

[@Roboflow]

@@dreamin4444 I'm working on fix right now

[@Roboflow]

I just fixed the notebook. There was a change in pytorch-lightning. And that broke our notebook. It should work fine now.

[@dreamin4444]

@@Roboflow Thank u so much!

[@JUANG0MEZ]

hello good afternoon, is it possible to detect objects and that the bounding box has a rotation?

[@Roboflow]

What do you mean by retention?

[@raymond-andrade]

Generally if you want more accuracy than a horizontal bounding box, I feel most people opt to use Segmentation instead of Bounding Boxes, since Bounding Box is meant to just have XY center position with Width Height (to minimize model output needed to identify the object)

[@Roboflow]

@@raymond-andrade I’m still not sure if that was the original question, but if so than I agree.

[@miladfarzalizadeh3058]

how inference time for each frame of video?

[@Roboflow]

You should be able to get around 20-25 fps with ResNet50 backbone.

[@systembreaker4651]

Why you didn't make with pycharm its hard to do like in the video 😢

[@Roboflow]

You men in colab?

[@systembreaker4651]

Yes on it

[@systembreaker4651]

@@Roboflow i need to make it on my laptop

[@afriquemodel2375]

--> 366 raise MisconfigurationException("No supported gpu backend found!") MisconfigurationException: No supported gpu backend found!

[@rupalibalabantray5178]

Hello sir I am a Research Scholar I want a dataset of thermal images of a Distribution transformer can you help me out.

[@lorenzoleongutierrez7927]

great !

[@Roboflow]

Thanks a lot! 🙏

[@lorenzoleongutierrez7927]

@@Roboflow, really thanks to all of you for the teaching and truly democratization of this knowledge changing the world. As an user following almost 3 years ago the evolution of roboflow, it seems to me incredible how well is directed the whole ecosystem service and clear information for so many people (Including me, as an agronomist interested in locating weeds and crops!). a Big Bravo from Chile and have a great weekend!!

[@Roboflow]

@@lorenzoleongutierrez7927 This is so kind! Thanks a lot 🙏I'll pass your message to the rest of the team.

[@FirstNameLastName-fv4eu]

hahahaha !! 12324343 KM from reality