First hour with a Kaggle Challenge

Neural Networks from Scratch: nnfs.io
Channel membership: / @sentdex
Discord: / discord
Support the content: pythonprogramm...
Twitter: / sentdex
Instagram: / sentdex
Facebook: / pythonprogramming.net
Twitch: / sentdex

[@SexySnorlax]

"keep social distance" sir know your audience we already are

[@abcdxx1059]

nope your were at distance when everyone was outside the house

[@GaneshKumar-zz9py]

@@abcdxx1059 mjhjnn bjm

[@GaneshKumar-zz9py]

@@abcdxx1059 khdw

[@suzikang9283]

0:00 intro to dataset 3:18 browsing through files 5:58 loading files into python program 8:43 (cleaning & structuring) getting keys from text and storing into variables 15:07 thinking about “extracting meaning from text” --> NLP. i.e. what are we looking for? --> keywords in papers that are consistent 19:03 looking for “incubation” in text 32:18 using regular expressions 42:08 plotting 44:20 adding rest of files to script 45:21 looking at other kernels on kaggle

[@alexdavis9324]

Thank you for not editing out the mistake at the 30 minute mark. That makes me feel a lot better about my own silly mistakes.

[@sentdex]

Heh, happy to keep it realistic :D

[@wongright]

@@sentdex Improved approachability was the big benefit when watching you debug in real time. Thank you.

[@hammerofheaven1313]

Not to be confused with kegel challenges.

[@whoisabishag3433]

... that's the word ... She whispered to me ...

[@Alex-pd5xc]

hah!

[@caseymeehan5901]

So glad you left in that error (printing full text ~30 min mark). It makes me feel so much better :) Thanks for doing this, it is so rad! I was looking at the kaggle competition but I am too much of a noob to know where to start.

[@TheAcolossus]

Everyone: What are you working on? Me: A Covid-19 machine machine learning model Everyone: How does it help us with Covid-19? Me: It doesn't

[@sentdex]

If you're helping to parse through the insanely dense amount of information and research to answer the questions that are being asked, you *are* helping.

[@owoled282]

Hey, is machine learning squared deeper than deep learning ?

[@cesarp6761]

if it keeps you home for hours doing this.. it does help! :p

[@fuba44]

I liked this "come as you are" format, could have easily been longer..

[@pw7225]

In fact, I think this is way better for learning. Since you see the actual process. Like George Hotz' coding sessions.

[@non_complete]

@@pw7225 I love george's sessions. You might like Jon Gjengset too he has a similar style, mostly does rust development.

[@parkerdinkins5541]

@@pw7225 geohot is an absolute mad lad! this format is definitely much better than structured sessions. it really captures the trial and error process of programming

[@danielcolomer7815]

@@parkerdinkins5541 same here for quantum computing kzbin.info/door/-2knDbf4kzT3uzWo7iTJyw (disclosure I own the channel xD) and that's exactly the reason i chose the style as well! I find showing the real process is the best way to help ppl learn

[@FlorianLinscheid]

Just to answer that very basic question. Make the decimal part optional by grouping it and then make the whole thing a group and you're good to go. re.findall(r'( \d{1,2}(\.\d{1,2})? day[s]?)', sentence)

[@sentdex]

Heh, thanks!

[@floxire7042]

Could you please explain why his technique with the parentheses didn't work ?

[@sentdex]

@@floxire7042 if you just have 1 set of parentheses, you'll find only examples that match the full string u searched for. But only return the part of the match inside the parentheses

[@FlorianLinscheid]

The main catch here was that he made that one group by using parentheses. Regex will only output what's inside the parentheses then. So to get the whole number again, you need to make the whole expression another group then. Putting the first or the second half in parenthesis doesn't matter in this case. I just found it more logical to have always the first two digits, followed by an optional decimal. Other way round works just as well. Hope that was clear.

[@floxire7042]

@@sentdex Oh ok thanks I didn't know that

[@nmertsch8725]

About the cite-fields in the JSON data: When you write a scholarly article and use findings of other articles (e.g. to compare your results with them or to build your study on earlier findings), you cite the original articles. "Because others already found out that rotten fish smells (see 1-4), ..." would contain a cite range from 1 to 4, because the cited articles 1 to 4 have shown that rotten fish smells and you build up on that without investigating it yourself. In the bottom of the article there is a list of references, where each number is associated with another scholarly article.

[@hewypy9015]

loving these live coding videos where you explore real world datasets. thank you!

[@balthazaromeyer4334]

Sentdex you are one of the best teacher I ever encountered. Keep Strong, Keep Teaching Us! People like you should be Glorified! Leaving your error in he videos is humble and remembers the mortals where we come from.

[@gabrielk3733]

I've been waching your channel for a few months now...the way you are talking, the way you are thinking, your python knowledge and experience is absolutely amazing for me, you're a GENIUS!

[@sebbecht]

Just got started browsing kaggle for future challenges two days ago, great to see a series on this :) excited to see how far you take this!

[@TheSaintsVEVO]

😂 “that went so fast” - yeah, you do remember you have a supercomputer right?

[@ankushbisht-0055]

Love how you keep the debuging part and small mistakes. Keep doing such great live coding , was looking for such content from a long time.

[@thesitcomaddict]

Your thought process is so clear! Thanks for showing this was really enlightening to watch :)!

[@cyruscuenca]

I'm learning to analyze image data right now, and even though you're analyzing text, I found this really helpful. Thanks!

[@merth17]

he's just that inspiring

[@connor4440]

SO happy you left the error in lol, shows that every programmer, no matter the skill level can have stupid little errors like that

[@HellTriX]

I think the most impressive part of this challenge, is a 50 minute challenge of not mentioning that which shall be demonetized :)

[@nicky_buttigieg]

For the incubation day regex you can also do: re.findall(r"(\d{1,2}\.?\d{1,2}) day", sentence) This will find any integer/decimal number followed by 'day' but only output the number, avoiding the need to split.

[@BrentBrewington]

you got my like & subscribe, my dude. wow, this was super useful to watch - i'm a Sr Data Analyst looking to go Data Scientist, so looking to learn from more people like you. also kind of interesting to watch this 1 yr later

[@RepiGameplays]

To get the names of folder I usually just use F2 instead of right clicking and rename. I rename stuff a lot and it surely has been helping. Great video!

[@attentiondeficitdisorder]

These are so awesome to watch. It really helps to see your logic and thought process. As someone new to trying to process datasets like this, it's great to have confirmation that I'm not doing it some weird, crazy way.

[@PositronQ]

Formula: Pf = the probability of infection on the virus C = the consequences of the situation Dn+1 = C*Pf Dn = another_day Dn+1 = next_day or actual day So Dn+1/Dn= “the porcentage of the increase of days” example: 22/11=2 so 2 is the porcentage of that days in increase but if you want to predict the next day so multiply the actual_day(dn+1 * “the porcentage of increase” in This case 22*2= 44) and this a formula if you want to predict all days of your country or in the world.

[@erosennin950]

That's what im talking about a kaggle challenge MAN! big thanks :)) I would like to learn from a pro, how to approach problems and solve them the fastest way possible.

[@Mahmoud_Gabr]

I’m sure I’m not the first to ask, but please do more videos like this!! The lack of editing is also very helpful. Thank you 👍

[@kar-s6716]

That print(t) made my day .. 😂😂

[@DanipBlog]

I'm glad you decided to leave the 'print(t)' blooper in the video 😂😂

[@ramzykaram296]

I can keep watching you programming the whole quarantine time, seriously your videos are so interesting so please do more videos

[@clearthinking5441]

Great video Harrison! I really enjoy seeing how you think, it gives the viewer a more accurate picture as to what coding is really like. Keep these videos coming please!

[@shivamshukla438]

this is really nice i think we can apply more re's and logic to get more information as you suggested like cleaning too

[@Evilleoleo]

dude you have so many good videos, was going through your data analysis playlist yesterday, so today this was perfect thanks dude!

[@sentdex]

Glad you like them!

[@DP-dc2vv]

Super informative, thanks for posting. Two thoughts: (1) Get VS code or Spyder/Anaconda; I use Spyder for general purpose Python stuff--the iPython integration is the best I've found. VS Code is potentially better still (depending on preferences), as it provides access to a terminal as well (though the iPython implementation is run through Jupyter and pretty janky). (2) Re the regex stuff, no shame in googling. There isn't a SINGLE programmer of any sort that doesn't need to google at times. I've been a professional for over a decade and regularly need to google syntax on basic methods, etc. Regex is a much more involved beast, and unless you use it daily I'd be amazed if you remembered syntax needed for specific applications.

[@qaispalekar]

Thanks for maiking this video. It would be great if you make more videos like this. Will get a rough idea of how to tackle such big data.

[@not-lain]

24:59 beautiful cup noises

[@marcelo403]

maybe you mentioned it at some point in the video, but if not: it is important to note that since you got the last number before the word "day" or "days", likely your estimation is upward biased, because you might catch the mean incubation period but also the upper bound of those papers that report lower and upper bound, such as "from 4 to 12 days", or "4 - 12 days", and so on...

[@vaibhavkhobragade9773]

You are so swift. It seems you are invincible in coding. I love your walkthrough for the kaggle challenge.

[@hectoralarcon4888]

I envy the fluency of your python programming. :( I always get stuck during preprocess for a while.

[@Luckylesss]

I LOVE these types of videos. Please keep them coming! Maybe even show us your googling to find answers to problems like needing a regex refresher.

[@sentdex]

Given more time, I would have included that. I've including some of my internet searching in the past, seems like people really enjoyed this format of video, so maybe more to come :)

[@KylePapili]

Very interesting seeing your thought process working through a new dataset like in this vid. Loved it!

[@Tony-mt4pi]

When I saw that he did not notice the "print(t)" line, I wanted to shout "into" the screen to let him know that.

[@TiboLatte]

This was really useful please continue ! You're doing awesome work thanks

[@ralphlagos4210]

Love this channel! So glad I found it, thanks for uploading :).

[@jackbillimack7159]

You are the man sentdex! It's hard to express how much you have inspired me while introducing great concepts. Keep up the great work. Does anyone know if the scientific community is making strides to standardize raw data and move from PDF-type papers that need more cleanup to interactive IPython-type papers that could store all findings? A move like this seems like it would open the flood gates of open-source hypothesis testing and review. Hosting and publicizing poor analysis could be a problem, but I would appreciate any information and opinions folks have.

[@Dhukino]

in university theses you can have abstracts in multiple languages, hence the list structure. in papers? probably not

[@pinakeekaushik7803]

print( " Really loving this bro, can you please continue it like for other kaggle challenges too" )

[@sentdex]

I could try some others like this, sure

[@onlyme0349]

You're hardcoding a format to search information that has wildly different formats. Just parse out any number not separated by a space. Since you can expect it in the format of "8 to 12 days" you'd have to write something for that too eventually.

[@cayanaraycaudhuri]

I got 9.93 days with the data I downloaded. They have removed non commercial stuff. Have you looked into the nltk package? This video was awesome, and I learnt something I never thought was possible.

[@waron999]

Make a data extractor. Extract plot and tabular data from pdf files and add meta data about the methodologies. Many of the coronavirus research papers a made open access This would create a database that could be highly useful.

[@crazyoldhippieguy]

25-03-2020.Hi l just found you today.l,m the guy who gave the talk in 1990, called Joy and Creativety in the qauntum realm With the blessing of IBM and Honeywell in Malta, the laser cooled ion team from Boulder, was there, it was to pick up were fieyman left off.Thanck you.

[@whoisabishag3433]

"Planting The Flag Of The First Comment"!" Kaggle And Corona ... New Tactics

[@shkronjax]

very nice. Im glad this knowledge is open source.

[@souradeepsinha]

Do a live chat while doing this challenge.. No one has much to do anyway and we can brainstorm together! :P

[@alexr7530]

36:56 I guess in the regular expression you wanted to make a non-capturing group: '(?: ...)'

[@fuat7775]

Hey thanks for the video. Abstract is a list because each paragraph is a text item. Look for the schema file included in the directory, that might help you. Cheers

[@josephsmy1994]

Hey sentdex, one of the wacky things about re in python is that when you use parentheses, you need to specify that you're not trying to capture that as a group when matching a whole pattern. (?: .... ) denotes a non-capture group you're looking for this... "(?:\d{1,2}\.)?\d{1,2} day" at 36:00

[@ahmetdiril824]

This also worked for me: " \d{1,2}\.?\d{1,2} day" I think the ? considers all of the expression before it. I also dropped the r at the very front.

[@Gamegankk]

the forward slash works and it always works

[@mdougf]

Thank you so much! I’ve been so intimidated by even approaching a Kaggle problem!!!!

[@Shubham-ny2ce]

you are really doing great to teach the scholars and non scholars . . Why don't you start some open source projects or start building a community if you are not planning a new startup . You are a great learner and far better teacher.

[@techystuffs371]

It was the coffee mug for me :)

[@nighteagle9961]

38:13 I think this works: single_day = re.findall(r" (?:\d{1,2}[.])?\d{1,2} [D,d]ay", sentence) Thanks for the videos and keep this good work. Learning a lot from you.

[@junaidmahmud2894]

Can you please do some more competitions like this? This is amazing?

[@mihaisabadac9631]

Great tutorial and good theme also :) For me worked re.findall(r" \d{1,2}\.*\d{1,2} day", sentence). I don't know if someone else wrote some other solution, too many comments :D Thanks sentdex

[@cosmosnomad]

Probably want to avoid having more than one incubation time per paper to avoid skewing the plot/mean. Make sure they're associated with Covid-19. I saw one to do with an avian flu strain.

[@sentdex]

This was addressed in the video with that exact example :p

[@nassehk]

Hello. Great to see your workflow. I think median is a better measure of finding the average rather than mean in your case because you are looking at a population of incubation times.

[@adeeb12321]

thank you

[@ramil17998]

Really enjoyed the video. Thanks for making all the mistaies and raising my confidence bar :P

[@sentdex]

Heh, happy to help

[@puneetsingh5219]

Yo, this video was long due. Thank you.

[@ambarishkapil8004]

Nice and Insightful tutorial.

[@classicrockman90]

Definitely look into glob from the standard library. Much easier than nested for loops to pick up files recursively in a folder structure with a pattern like *.json

[@adityask277]

Hey sentdex. Not sure if you will see this. I have been having problems with the recent updates in certain packages. For example. BeautifulSoup.findAll() returns empty list, but beautiful Soup.find() isn't. I'm using version 4.6.

[@kuldeepsingh2983]

i am in love with shark-coffee

[@mubinabdulkader1525]

Just quit from my 30Hr online course after watching this...

[@JustSomeAussie1]

The forward slash in os.listdir(f"{}/{}") definitely works on Windows, i just tested it. (tested with Python 3.6.4)

[@SomebodyOutTh3re]

22:31 hahaha. Great video thank you!

[@leonshamsschaal]

Thank you so much! I have always wanted to do Kaggle competitions but never really known how to approach them.

[@sentdex]

Happy to help!

[@treelight1707]

Hey sentdex. I think the cite_span key at 5:25 are the location of the article in the publication print. 'start' /'end' is like start page-end page, 'S1' stands for supplementary material, usually put at the end of the print, like an appendix. 'abstract' is like the summary of the entire publication, but there seems to be text from other parts of the article, that might be more specialized stuff. Don't know if this info would help.

[@sentdex]

Thanks for the info!

[@thinboxdictator6720]

@@sentdex $ cat CORD-19-research-challenge/json_schema.txt | less

[@borispapic9510]

Wow just took up this challenge a few days ago but hit a wall and didnt know how to proceed. This is a godsend!

[@ahmetdiril824]

I put the work done here in a notebook. I improved the regex and added some cleaning: www.kaggle.com/ahmetdiril/01-incubation-from-sentdex

[@noctreik]

Also, with your style of programming, I recommend you to run things in ipython shell and copy/paste fragments of working code in sublime text.

[@mahdi7d1rostami]

your text editor looks futuristic. i will use gedit more frequently if i now the name of theme.

[@TheMaidenOnes]

its not gedit, its sublimetext

[@-nepherim]

He's using the Sublime text editor.

[@mahdi7d1rostami]

at the beginning when he opened one of the json files he used gedit. beside that black gtk theme for the whole system was great. i wanted to know its name.

[@adityavarma131]

one can use os.path.join() to overcome any issues with forward or backslashes in different operating systems.

[@abdelrhmandameen2215]

Programmers are masters in selling themselves short

[@tarsala1995]

Your machine took over the camera view. Who knows where is this going

[@mayukh_]

I am starting to like your mugs

[@LKokos]

28:40 you still had print(t) on line 42 thats why it printed all edit: nvm

[@MistaT44]

This is an excellent series! kudos

[@EranM]

Harrison! Well put video! I very much enjoyed it! You are hilarious!

[@leosdeoilha]

Always great videos! Why don’t you use spacy for nlp? It takes a lot of the re out of the way!

[@Mr3zoozee]

what a Coincidence i was looking for videos like this thx sentex

[@chrisherring8733]

You had a capturing group there. You need to make it non capturing group like ((?:\d{1, 2}\.)\d{1, 2} day)

[@mbappekawani9716]

nice data cleanup buddy

[@sameerzahid3544]

I really like how your operating system looks and the text editor 😍👌

[@ВолодимирКузько-б5ж]

why did you define the incubation = df[df['full_text'].......some code..] - was it some kind of wrapping?

[@clumsydnkey29]

Such a helpful video! Thank you!

[@noctreik]

I recommend you setting up i3wm for desktop environment. You will be much more efficient. You will not want to come back after spending couple of weeks using it.

[@selcukmisir2399]

You are the best sentdex!!!

[@alexr7530]

Thanks for the video. Hope you'll continue the rubric

[@teresitaeyzaguirre4741]

new Fave channel

[@Pythonenthusiast]

I don't know if others mentioned it before, but you got some cool mugs! I guess you can make a video on that as well!