Mask RCNN - COCO - instance segmentation

[@Augmented_AI]

Would really love a video tutorial :)

[@tahirhanyildizoglu3408]

at 0:48 it finds reflection of car in mirror at right side , Really good !

[@smart_world7928]

at 1:23 it also finds a bus right there ;)

[@ojjoooooo]

Would be good if it understood it's a reflection.

[@KarolMajek]

Imagine advertisements, billboards with cars or people...

[@liberator328]

Wouldn't that be bad? Ideally it should detect that it's a mirror of a car, not another different car.

[@KarolMajek]

You need a lot of examples and a class for that. For autonomous machines it's a lot easier to use lidar.

[@thisnameisnotavailable]

15:19 "Clock 98% ... " - it's ventilation, man! VENTILATION!

[@KarolMajek]

no ventilation in COCO set cocodataset.org that's the reason

[@mathiashaudgaard9360]

However this ventilation has a lot of the same features as a clock. If you look closer you can see how they relate :-)

[@OscarPrice007]

What about the bus (van) and the three sinks (litterly just road)?

[@dx31900]

That's why they never labeled anything 100%. Machine vision can "never" be accurate as human eye.

[@denanfamily5890]

T-900 point of view

[@piyush9555]

how precisely accurate computers have become, we're in a real trouble now

[@YanuarTriAdityaNugraha]

Really good implementation, but perhaps for safety critical use-cases, I'll limit on using the net only for vehicle and pedestrian detection only, with probably extension to traffic light detection (spot on). There's plenty of mislabel for traffic signs, almost all signs were detected as stop signs.

[@walterrural]

May I use this footage in a project? I'm looking for this kind of footage, but nothing I find licensed in Creative Commons compares to this.

[@KarolMajek]

Ok, but please share the result with me. Just wanted to know. Thanks for asking!

[@parkerlopez3197]

The most impressive part of this is the detection of the car via a reflection on the glass

[@primodernious]

now we need just to make it detect more details of objects and translate these speed signs into numbers for speed regulation and get enough detail to tell what ever the car is a farrari or a ford as well as being able to detect red green and blue color on traffic lights and make it do a stop on red light or a stop if a person is on the way. had also been cool if the program could detect if the person what a male or female, long hair or short hair, sun glasses or no sun glasses and other details. i hope such detail will be included in future versions.

[@cemrekaplan1480]

you've been watched -person of interest

[@battalll]

aga ben seni daha önce de bir ai videosunun yorumlarında görmüştüm sanki

[@vardaan3282]

It was You are being watched! I think!

[@hawklee8582]

Great job, but I wonder if it is possible to hide the square outline and the shade of each detected object. Instead, I want the TensorFlow tells us the name of an object when we touch it on the screen.

[@KarolMajek]

Nice idea for an app. It is possible

[@hawklee8582]

Could you show me how to do it?

[@richardburrows9202]

Very impressive work Karol. I'm working on a home surveillance tool (personal project) that utilizes IP cameras and an object detection neural network. Have you done much work in low-light environments? Any tips for working with images that have IR illumination (most security cameras use IR for night mode so the images are in greyscale).

[@KarolMajek]

It's hard if you have bad illumination. I was working with thermal camera, but not with RGB+illumination. Did you already tried to run something on such videos? If you want I can run some algorithms and publish results here, on my channel

[@mohamedouziane]

Question: for segmentation semantic models : Bisenet & segnet, it is necessery to prepare the masque for training? if yes how why to prepare this masque ? thank you for anwser me.

[@Tetsujinfr]

How is it possible that with such a high resolution the detection is so unstable. E.g. the utility vehicule at 10:00 right in front of the camera get undetected several secondes...

[@KarolMajek]

It's not a matter of resolution. This net was trained on COCO, and here you have images from different distribution. I think Mask RCNN trained on Open Images v5 would work much better check also YOLACT which is much faster

[@nicom.1500]

Great video, Karol. Thanks for sharing. I think yours is the only implementation where you see only one color for each label (people, car, etc) which makes it much more understandable vs the original. How are you doing that? Thanks!

[@furkancetin3195]

Bro did you find an answer? I am trying to the same thing

[@KarolMajek]

It's not a secret, I plan to release code and create tutorial. Sorry that you are waiting so long

[@furkancetin3195]

It will be great I am waiting for that tutorial thank you :)

[@magdalenamackowiak1065]

Dzień dobry, Czy możliwa jest realizacja tego typu projektu w środowisku Google Colab?

[@KarolMajek]

Uruchomić się da, ale Colab jest bardzo niewdzięczny. W wersji pro mocniejsze maszyny, ale lepiej wychodzi gdy się uruchomi na laptopie - sesja się nagle nie skończy i nie trzeba dostosowywać kodu. Zapraszam do mailingu mojego, tam wygodniej się pogada, albo przez jakiś formularz kontaktowy. A chodzi konkretnie o uruchomienie inferencji by dostać segmentację instancji? Może wystarczą ramki jednak, bo szybciej

[@Musicoder-es3nt]

it's really impressive.

[@MasterJ202]

I’m interested in using this with Unity. I already have Tensorflow and I have been playing around with an object detection example project from the internet and it works pretty well but I like the idea of Masking the objects instead of just bounding boxes. Do you think it’s possible?

[@zhiqiangxu7548]

you can try it in colab

[@KarolMajek]

Colab != Unity.

[@KarolMajek]

Try newer networks. YOLACT, DETR

[@dantwister5106]

Transparent colors looks great. The question is, is it possible to integrate this into google glass, so that you have a real life color hack? I think it is because it is easy process screenshots in real time at 640x360 resolution, no?

[@KarolMajek]

Thanks, mask rcnn is too slow. Check YOLACT or DETR

[@KaeMCe1]

Dzień dobry, należy przyznać, że algorytm segmentacji w połączeniu z YOLO(?) TensorFlow (?) działa naprawdę dobrze, jestem pod wrażeniem. Mam pytanie odnośnie użytego sprzętu i jak wyglądało użycie CPU i GPU oraz czy algorytm wykorzystywał obraz w oryginalnej rozdzielczości 4k, czy też był on specjalnie zmniejszany do obliczeń? Jaki FPS osiągnięto? Podejrzewam, że o ile samo wykrywanie obiektów może odbywać się w czasie rzeczywistym o tyle połączenie tego z segmentacją już nie koniecznie, jak to wyglądało w praktyce?

[@KarolMajek]

Dzień dobry, Wejście 1024x1024. Mask RCNN arxiv.org/abs/1703.06870 w Tensorflow. Na GPU: GTX1080 ~1FPS (muszę to jeszcze sprawdzić bo w hoście mam jeszcze K40 i FPS też nie jestem pewien) A więc działa to znacznie wolniej niż YOLO czy SSD, ale wyjście jest bardziej atrakcyjne

[@KaeMCe1]

Dziękuję za odpowiedź. Myślę, że w wolnej chwili spróbuję uruchomić YOLO albo tensorflow z bazą COCO. Nie myślał Pan o przetestowaniu tego typu algorytmu na filmie z platformy latającej (dronie)? Osobiście jestem ciekaw czy baza COCO pozwoli na detekcję obiektów widzianych z góry i z dużej odległości.

[@KarolMajek]

YOLO może mieć problem - źle działa dla małych obiektów. Bardzo interesujący problem, niestety nie mam takiego materiału video. Co do samego uruchamiania na dronie - proszę sprawdzić ten film - na telefonie: kzbin.info/www/bejne/mYKtkmx3i6p0f6s Co do danych z góry - kwestia przygotowania zbioru danych i wytrenowania sieci neuronowej.

[@nguyenmanhcuong9657]

Hi, I want to know which is the annotation tool to create the dataset (like MS Coco). I used labelImg but it can create only the Pascal VOC data set. How to create the mask for image in case of MS Coco ? Thank for your help !

[@KarolMajek]

Do you want to create contours of objects? There is a tool available, I don't remember name/link, sorry

[@justcurious2906]

Is this implementation better than YOLO? Also, how do you count uniquely the objects detected, without counting the same object twice?

[@KarolMajek]

This is single frame detection, without any tracking, so you wouldn't know. But in each frame you are able to count objects, as output you get list of instances + contours

[@MrSwan-tm5wj]

Would it be possible to track rats going around a track, and get their order ? Is yolo a better choice to do this ?

[@KarolMajek]

YOLO/SSD are faster and should do this, but you will need also tracking I guess

[@MrSwan-tm5wj]

Thanks.

[@bigbook5307]

Is there a cpp implementation with caffe?? I have not found on github.

[@KarolMajek]

+Xuling Chang I don't know any. I googled but haven't found any results

[@shariarshakhawat]

I want to learn this stuffs. Just installed Cvat on my MacBook air. Can you please suggest me from where should I start? As I have pretty vague knowledge about this things.

[@KarolMajek]

I don't think cvat is a good start, but it depends what is your goal, what do you want to achieve

[@jinwu4715]

Karol, Super cool video! I wonder where you got those input videos. Can I use them in my own demo? Best,

[@KarolMajek]

My own, recorded with a phone. Yes, go ahead! Can you share results publicly? Would be great to see!

[@jinwu4715]

Will do if I have a proper video produced. Thanks!

[@DanFrederiksen]

Funny that it sees the fans in the tunnels as clocks.

[@faruknane]

how much space is required for the network model to be trained? in my case for 400x400 image (batchsize=14) I need 11gb memory (U net basic model).

[@KarolMajek]

For mask RCNN I don't know. If you reduce batch size you can use smaller GPU, but bigger batch is better. Faster RCNN with NASNet requires 8gb for 1 image in batch...

[@faruknane]

​@@KarolMajek 8gb is really huge wow. Thank you for the information.

[@KarolMajek]

Go to tensorflow object detection model zoo and check prediction speed. Slower nets need more memory to train

[@fabricmax7024]

Hey dude, I also made a video demo and i had a webcam test, but the speed is very slow, only about 1~2 fps. Do you know how to improve the test speed ? I want to make a real time application.

[@KarolMajek]

Same for me, I also have 1-2 fps. This video is not realtime. I increased speed to match 30fps after detection

[@fabricmax7024]

Yeah, same for me. I tested on a GTX TITAN X, what about 1080 ti? If you have progress about speed, could you please update your github? Appreciate it a lot!

[@KarolMajek]

GTX 980m, measured only the time of inference. There are computational heavy steps required to blend masks with input video

[@my1347]

In order to get better result , do you train with 4K image pairs? i

[@KarolMajek]

This is original result from a model trained on COCO dataset. 4k video is for test purpose only

[@my1347]

@@KarolMajek got it, Thank you!

[@张聪-r3t]

Hi, what are the hardware requirements for it to run in real-time? Thanks.

[@Chocolovers]

Its not ment for realtime too much weight

[@KarolMajek]

True. And it's quite old. Check YOLACT and DETR

[@kaibinxu6090]

Around 10:00 there are moments the mask rCNN doesn't recognize the truck right in front. What may cause it?

[@jerelvelarde2829]

It's not trained to recognize it.

[@kawo666]

@@jerelvelarde2829 But if it's not trained to recognize it, why does it recognize it in other frames?

[@aniketji1435]

It happens when the accuracy of detection falls below a given threshold which is set as criteria to display. And in continuous videos only we find such glitches where the frame might recognise it as 2 objects with less confidence in each.

[@shairozsohail1059]

Could be because the tires are occluded and because of the lack of reflective surface/depth

[@masifakbar]

can i see the code to this project. your repository only have gifs and base code

[@KarolMajek]

Are you looking for code for inference on video? I was modyfying this demo notebook github.com/karolmajek/Mask_RCNN/blob/master/demo.ipynb

[@masifakbar]

@@KarolMajek but how did u read video, frame by frame or did u use any other method? And any idea on how to use imagenet weights and dataset instead of Coco??

[@HomelessRafi]

I am trying to segment and detect 2 different images, however the results i get are identical so that every image has the masks and detection of only the first image. Does anyone else have this problem?

[@KarolMajek]

You mean to put 2 images as input (batch size=2), not to run prediction 2 times (run 2x with batch size=1)? Can you share your code?

[@HomelessRafi]

The first part. Currently I have the prediction running 2 times (batch size=1). But I want to be able to run 2 images simultaneously with the mask updating for each image.I hope I explained that clearly, let me know.

[@HomelessRafi]

Quick follow up question, how did you get the mask to stay a constant color for each frame?

[@yuejohnson7661]

Hi, It is very good job , and I can run matterport/Mask_RCNN but it is so slow(GTX 1060 , about 450ms per frame which frame size is 640*480 ), How fast you are?

[@KarolMajek]

Thanks! I confirm this is slow. It took day or two to process this video :-) I need to add fps display next time

[@yuejohnson7661]

I'm sorry, I didn't see the previous comments. Mybe most of us are interested in using Mask Rcnn for real-time.

[@matiqulislam4026]

I have successfully run the demo of Mask-RCNN but problem is that it does not display the output image

[@KarolMajek]

matiqul islam tu can use matplot or cv imshow. I can send you demo code via email if you want

[@matiqulislam4026]

Thanks brother please send me the code. My email address is : [e-mail]

[@matiqulislam4026]

Dear Karol Majek I have another Question: Do you tell me Where I get or How I can generate "mask_rcnn_shapes.h5" Shapes trained weights. Also send me the demo code of my email: [e-mail]

[@jaydipsinhparmar6010]

hello sir i have download the source link but i am not able to run the proj.please give me the doc if you have

[@KarolMajek]

You will need python, tensorflow and jupyter notebook installed. Then try to run this: github.com/karolmajek/Mask_RCNN/blob/master/demo.ipynb

[@jaydipsinhparmar6010]

first thanks for your reply.i have allready install the python,tensorflow and jupyter notebook.i have soma cuda error can you give the specific version of tensorflow and cuda toolkit version

[@aiai-nn5jd]

Thanks for share this video, how can i get this code to learn?

[@zieddatascientist546]

I'm always asking my studient what we can do with that ? It's beautiful, nice, etc. but why

[@AashuPrasad]

Of many possibilities, it can be used as a vision system for self driving cars!

[@stangerdanger7252]

Terminators will use this to identity Humans and Machines.

[@ltsgroupcompany]

Hello, this video is so nice! Could I use this video for my youtube channel? If it is okay, then I will put the link of the video on description box.

[@KarolMajek]

Yes, ok. Can you put here the link to your video so I can share it in a post?

[@ltsgroupcompany]

@@KarolMajek Thank you so much :)) ok, I will.

[@ltsgroupcompany]

@@KarolMajek kzbin.info/www/bejne/e16YY6qOoqxgZ7M 3:17 - 3:25 I used the clip from this video for my youtube channel! Hope you enjoy this video!!

[@sublimoud3196]

hi, i am trying to set my own dataset with create_pet_tf_record.py i have mask, xml and png, but when i launch train and eval masks are not displayed... have you an idea ? no one answer me

[@shikharsaxena9989]

Is this sematic segmentation because all the cars have same colour

[@KarolMajek]

No, it's instance segmentation. I set consistent colors for each class

[@lukas_543]

is the underlying data available (video + instance segmentations for each frame)?

[@KarolMajek]

Only the input video is shared

[@t-lm]

Very nice job, Could you please add code example for IP camera live RTSP streaming on github? For static counting cars usage.

[@KarolMajek]

You will not get live detections since mask RCNN is pretty slow. You can use opencv to receive such stream. Then just pass the image through net and you will see your results

[@t-lm]

@@KarolMajek I use ROCM with Sapphire NITRO+ RX 580 8 GB and no issue with live detection from webcam, i just need to figure out how to do it with IP cam.

[@KarolMajek]

Ok. Just use opencv camera capture with url as input

[@t-lm]

@@KarolMajek i did that but after a few sec it crashes with this: [rtsp @ 0x7f762b264700] RTP: PT=60: bad cseq 4e86 expected=30c3 i think is a FFmpeg bug.

[@KarolMajek]

Can be. I don't have solution for that problem

[@2mvX]

Myślisz że jest szansa implementacji tego na raspberry? Jakiego hardware to wymaga?

[@KarolMajek]

Na RPi już same boxy działają wolno. Zobacz Tencent NCNN. Zobacz NanoDet (jutro film będzie tu)

[@KarolMajek]

Przy okazji. Znasz DeepDrive.pl/30 Trochę filmów o sieciach + parę miesięcy maili o Deep Learningu

[@yesu399]

Is it real-time? Or do you analyze each frame and write masks onto video?

[@KarolMajek]

Not realtime

[@yesu399]

Emmmm.. So you process image by image and combine into a video?

[@KarolMajek]

Correct. Mask RCNN is slow. I plan to run 4 other maskr rcnns with different detectors and will put fps there

[@yesu399]

Yeah. It's indeed slow, I apply it on my own dataset and the best I got is 5 FPS. Do you have any recommendation for accelerating the prediction? Like any nice paper？ Thank you!

[@KarolMajek]

Not yet, I think solution is waiting somewhere, but I haven't found it. I would give a try to any pytorch implementation. Should be faster because of channel-first image representation

[@marr73]

this is more like semantic segmentation? how can i reproduce the result?

[@kanimozhisoundararajan9850]

Hai !! am new to this mask RCNN. but i was really impressed by ur work.am having some doubt in running the github work. Can i mail u ?

[@gilmeregildo1093]

why its hard to breath while working on this??

[@KarolMajek]

Is it? Maybe it's the air pollution

[@neparkiraj]

Can you make it recognize different types of trees from a long way?

[@KarolMajek]

You will need manually annotated dataset. But from the large distance the results will not be perfect

[@Kevin-ze5gk]

what Hardware did you use for the recognition and on how much FPS you run it?

[@KarolMajek]

desktop 1080. it was slow. I don't remember now. In next mask RCNN vid I will try to add FPS

[@abdullahzaqebah7919]

hi when i runs the code, in the step of Create Model and Load Trained Weights i faced this error: AttributeError: 'TensorShape' object has no attribute 'rank' any help?

[@KarolMajek]

I think you're using newer version of TensorFlow

[@abdullahzaqebah7919]

@@KarolMajek am using tensorflow 1.9.0

[@KarolMajek]

@@abdullahzaqebah7919 Try Matterport original repo github.com/matterport/Mask_RCNN and their updated demo github.com/matterport/Mask_RCNN/blob/master/samples/demo.ipynb TF version seems ok (>1.3.0)

[@aaronag7876]

interesting the system picked up cars in the reflection of windows

[@hlliu8638]

Good！ How to deal with the video? I can only use the source code to detect a single picture.

[@KarolMajek]

maybe I forgot to push. Basically you need to predict for every from and them make video from it - I am using ffmpeg, can recommend

[@HumbertoManeto]

Is it possible to measure movement speed?

[@julayelei]

Hi, there. Good job for this! I'm really surprised! By this day, I'm working with Mask RCNN, but I could not figure out how to implement a live video, would give me a tip or something? I would really appreciate it! Thanks anyway!

[@KarolMajek]

Julio Rivera basically I am doing prediction for every frame and writing result to file. Then I am using ffmpeg to create a video. Of course you can use camera instead of mp4 file thanks to opencv.

[@julayelei]

Thank you!

[@KarolMajek]

Julio Rivera if you will need some help, feel free to ask

[@julayelei]

Karol Majek Cool! Do you have an email? Mine is [e-mail] You are the man, haha

[@KarolMajek]

Julio Rivera [e-mail]

[@fouziaehsan2859]

HI Can anybody help me how can i use mask rcnn to train for mainly 3 object classes on my own video dataset? What we need to modify in train_shapes.ipynb for loading images from own dataset? thanks !

[@tomaszgiba]

Impressive! This looks like V.A.T.S.

[@emretatbak]

Hello what is your gpu and cpu?

[@KarolMajek]

980m

[@muhammadmonjurulkarim1614]

how did you set same color for all cars?

[@KarolMajek]

I am going to share this code finally, but have too many other things. Thank you for patience

[@thejanwijesinghe2393]

What is the average fps it could run on a GPU like titan x?

[@KarolMajek]

Hard to say Ignore Mask RCNN, chech YOLACT - it's much faster!

[@mohamedouziane]

Great job

[@PrasannaRoutray97]

Inference speed? Can we use it in real-time?

[@KarolMajek]

It's super slow. For realtime check: YOLACT

[@bryangarcia4153]

This is not live, right?

[@KarolMajek]

Yes, it's super slow. Check my newest video, there is something much more online

[@TankNSSpank]

Holy shit. Good job.

[@강새이잡이]

Can you tell me about hardware requirement for this?

[@KarolMajek]

NVIDIA GPU. With 980M it works pretty slow. Now you can find faster methods. This was below 1fps as I remember

[@twiekfkwniejwjrj7867]

what is the mask?

[@abbkeihatsu]

Can it calculate trajectory and velocity?

[@bormisha]

I guess this is still image processing applied to every frame of the video, so the results are not being correlated between frames. Compared to this, trajectory and velocity estimation once the objects have been detected and classified seems an easier task to me.

[@yuxiangwu4573]

Nice result

[@KarolMajek]

Thanks!

[@Booruvcheek]

Source video not available

[@KarolMajek]

It's THE DAY - I finally update the link in desc: Input 4K video: [NEW LINK!!!] archive.org/details/0002201705192

[@andersonriciamorim4895]

Which FPS did you achieve?

[@KarolMajek]

Super slow, seconds per frame. For faster inference check detectron2 and yolact

[@jeffersondesouza7783]

Boa noite, Caso possa adicionar as legendas em inglês eu consigo configurar a tradução automática para português e acompanhar com mais facilidade. Eu tenho aprimorado o meu inglês, mas ainda tenho dificuldades em alguns termos e fico com a sensação de estar perdendo algo importante. Você tem uma extensa gama de materiais excelentes, então se puder fazer a adição de legendas neles, seria de enorme ajuda. Mais uma vez parabéns pelo excelente trabalho. Grato,

[@ІванПилипко-у9с]

impressive

[@s2maschmeyer]

It can't seem to decide if motorcycles are cars or people? On a highway, I guess it doesn't matter in an accident....very little protection. Motorcycles are sometimes more unpredictable than cars and therefore, higher danger....It's a Bird... It's a Plane... It's a Boat.... It's Superman!

[@sekolahrakyat4434]

Hi carol can u send me the code ?

[@sekolahrakyat4434]

Please send me @ [e-mail]

[@KarolMajek]

look in description - github.com/karolmajek/Mask_RCNN

[@saifu2write]

source: github.com/karolmajek/Mask_RCNN please tell me how to work and install to proper steps. because some error i am facing. please help me.

[@deacix]

Crazy shit!

[@arcadetv8537]

where is it? turkey?

[@KarolMajek]

Warsaw, Poland

[@chanyy6838]

8:29 “The flag is also a kite”

[@fetullahatas3927]

what is the fps

[@KarolMajek]

Really low. Seconds per frame is a better metric Check Detectron2 or YOLACT

[@fetullahatas3927]

@@KarolMajek Thanks for reply, I did train YOLACT, accuracy is nothing close to Maskrcnn

[@fetullahatas3927]

@@KarolMajek I actually just need a real-time detector, in my scenario there is a lot of overlapping objects, do you have any recommendations for this task ?

[@KarolMajek]

@@fetullahatas3927 then check mask rcnn in detectron2

[@sampark8422]

in real life wall hacks?

[@PIDOtomasyon]

I can send a gift, but Paypal blocked in here(Turkey), I can send over Patreon.

[@KarolMajek]

Thank you, what about Revolut? I had created patreon, let me check. Thanks!

[@KarolMajek]

www.patreon.com/karolmajek

[@Ken-kt9br]

Reported for walls, enjoy your vac

[@gemobnyi4474]

inspired by bigpackets

[@uccao304]

code

[@neptune4909]

esp irl

[@HikariSakai]

Nice aimbot

[@AnshumanKumar007]

This vide was sponsored by the rich people with awesome GPU gang.

[@KarolMajek]

3yo laptop with GTX980, nothing special

[@cezexcezex9888]

skynet

[@chuckiepan6380]

牛逼

[@cjhuang3835]

nice try