How do I work with dates and times in pandas?

[@dataschool]

Starting in pandas version 0.18.1, you can create a new datetime column directly from a DataFrame, based solely on the column names! It's a useful trick, which I explain in this video: kzbin.info/www/bejne/Y3_Fimp7bs1-rs0

[@cossanfran]

Hello! I have a table with the date column. I want to group the data by month / year how do I do this?

[@dikshyantthapa3367]

I love you Bruh.. 😂.. No homo..thanks a lot!!!!

[@cobbdouglas690]

You've saved my job on multiple occasions sir, thank you.

[@dataschool]

That's awesome to hear! 🙌

[@kasiditauable]

I am an aspiring data scientist. I just found a series of your videos. Thank you for doing this for all of us. Keep doing great work!

[@dataschool]

Thanks for your kind words, and good luck to you!

[@witotiw]

2022 and you're still saving us. Thanks for the excelent content

[@dataschool]

You're welcome!

[@goktugerce]

I was actually going to ask you about this. I started learning Pandas thanks to your video series and I feel like I learnt a lot. Thanks for all the awesome videos.

[@dataschool]

Wow, that's really great to hear! You're very welcome!

[@goktugerce]

Actually, I still need some help. I want to create a column for "year/month" from datetime column. For example, if datetime is 2016-07-06 15:56:19, I want to map "2016-07" into a column. Of course I can get first seven characters by converting it to a string, but what is the correct way to do this? I have following the following lambda function: get_month = lambda x: '{}-{:02}'.format(x.to_datetime().year, x.to_datetime().month) my_df["year_month"] = my_df["timestamp"].map(get_month) For now, I do it like this but I am sure a better and more efficient way exists for the job. I'd be glad if you can help!

[@dataschool]

If your desired end result is a string (such as '2016-07'), then I think using string methods is the way to go! Perhaps something like this: ufo.Time.dt.year.astype(str).str.cat(ufo.Time.dt.month.astype(str), sep='-') However, there is probably an even simpler approach that I'm not thinking of...

[@goktugerce]

Awesome. It really is simpler than what I was doing. Thanks!

[@archidar1]

Hi, I know its a year late, but in case you (or anyone else is interested) "pd.Series.dt.strftime" is an easy way to output dates as strings in whatever format you like. pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.dt.strftime.html#pandas.Series.dt.strftime

[@Cam-p2z]

These are the first videos I look for when I have pandas questions.

[@dataschool]

Thanks!

[@Martin-lv1xw]

You are naturally born to impart knowledge. Thanks for your videos br!

[@dataschool]

Wow, I really appreciate your kind words! 🙏

[@akashmishra2661]

I understood each and everything in this video and it helped me alot for my project. I just want to thank the instructor.

[@dataschool]

Great to hear!

[@oleholeynikov8659]

5 years video, but the best on KZbin!!!

[@dataschool]

Thanks very much for your kind words!

[@boubacaramaiga4408]

I love the way you teach, easy to follow and to understand. Many Thanks.

[@dataschool]

You're welcome!

[@christopherclark2205]

Phenomenal description of working with dates and times in Pandas, very helpful.

[@jsmit7730]

I'm impressed. Simple explanations with examples. subbed and hit that bell. Thanks for the vid!

[@dataschool]

Awesome, thank you!

[@Diachron]

I've only recently stumbled onto your videos. Very clear and concise delivery. Good job!

[@dataschool]

Thanks so much! Glad you are enjoying them.

[@senz3757]

Thanks for your great video! It seems pandas.Series.dt.weekday_name is removed in pandas version 0.23.0. and one can use pandas.Series.dt.day_name() instead.

[@wilmarperez4962]

Thanks for the tip. Was facing the same issue!

[@Vishnu_Datta_1698]

Thanks a lot

[@jasper8291]

Thanks :)

[@aartiladdha01]

All your videos are worth watching. I have learned a lot about pandas just from your videos. Thanks a ton :)

[@dataschool]

Thanks Aarti!

[@RossThompson89]

I really enjoyed this video and your teaching style! To the point, but well explained at a nice pace. Thank you for this :)

[@dataschool]

Thank you so much!

[@goktugerce]

For anyone who wants to convert timezones, here's how you do it. First, you need to specify which timezone the date originally belongs to. In my case, it was UTC (Coordinated Universal Time). In my DataFrame (let's call it df), I have a column named "timestamp", with type datetime. I localized the column first: timestamp_utc = df["timestamp"].dt.tz_localize("UTC") Then I overwrite the timestamp column: df["timestamp"] = timestamp_utc.dt.tz_convert("Europe/Istanbul") I hope this method is correct and it helps someone!

[@dataschool]

Awesome! Thanks so much for taking the time to share your code.

[@SetarehFasihnia]

Hi, I just tried your code but it returned all NaT values? my original column was in datetime format and UTC + x timezone. I tried to convert it to US/Eastern time. Update: Apologies, turns out the error lies in trying to convert my 'timestamp' equivalent column into date time from object and inserting errors=coerce produced the NaT values.

[@payalbhatia5244]

@Data School , Mark, you are just amazing. :) You make it appear everything quite simple. My humble request is to kindle make a series on machine learning algorithms too.

[@dataschool]

Thanks for your kind words and your suggestion!

[@cmdeckermusic]

Thanks for this! You explain things very clearly and concisely.

[@dataschool]

Thanks!

[@marco4570]

That bonus is what I needed. Thank you so much!

[@dataschool]

You are so welcome!

[@AsyraffLatiffi]

I've been watching your videos for a while now but never got the chance to comment on it i just want to say keep up the great work! You are just awesome!

[@dataschool]

Thanks very much for your kind words! Much appreciated :)

[@aakashvarma9960]

Excellent explanation .I suggested these video series to most of my friends.

[@dataschool]

Thanks so much! I really appreciate you spreading the word.

[@deepakpanigrahi9601]

Data School is the best of all resource available on Pandas. thanks a ton!!

[@dataschool]

Thanks!

[@termica7349]

Thnks for this explanation buddy! So clear and concise!

[@dataschool]

Thanks for your kind words!

[@gurkanyesilyurt4461]

You're king of this area man!!!!

[@dataschool]

Thank you!

[@pranavmendiratta3166]

loved the tutorial, cleared my doubts! I like how you explain so patiently.

[@ΠαναγιώταΜωραΐτη-τ2μ]

So many great videos . Absolutely guidness. thanks from GREECE !

[@dataschool]

Thank you!

[@jannahawkins865]

This is a great video. Thank you so much! Also, I wish the pandas API reference still looked like it does in this older video. It's harder to read now.

[@dataschool]

I agree...

[@slavaslavia4085]

New to Pandas, new to your channel, and soon New year =) Thanks for the videos!

[@dataschool]

You're very welcome!

[@SatishKumar-yz4tn]

Thank you so much in taking time to explain so nicely.

[@dataschool]

You're very welcome!

[@rodrigoviverosphoto]

This is EXACTLY was a looking for. I love you.

[@dataschool]

Awesome!

[@nowyouknow2249]

Thanks so much. You are one of the best teachers I have ever known. Thanks so much once more you are a darling.

[@dataschool]

Wow! Thank you so much for your kind words! :)

[@samcathcart5388]

Hi DataSchool tk u for gr8 vid on working with dates and times. I am trying to work out how to group data for days, months and years in the same plot, e.g. bar graphs for months and different colours for the years

[@shilpikulshrestha9487]

Hello sir, you are awesome teacher. Great videos Thank you very much

[@dataschool]

Thanks so much for your kind words!

[@gustavoviana1178]

I'm so thankful for your tutorials

[@dataschool]

You're welcome!

[@debashissahoo5031]

Your video is really good, we'll be really helpful, if you make some more videos on Dates and Times. Thank you.

[@dataschool]

I cover it a bit more in this series: kzbin.info/aero/PL5-da3qGB5IBITZj_dYSFqnd_15JgqwA6

[@debashissahoo5031]

thank you, Now I am planning to cover this series too.

[@VijayKumar-zx5bm]

Excellent videos.Please consider giving tutorials on time series forecasting ( with various statistical models ) with Pandas.

[@dataschool]

Thanks for the suggestion! I'll consider it for the future.

[@leandrorberto]

Excellent class! As always! Cheers!

[@dataschool]

Thank you!

[@sinabaghaei3504]

Special thanks for your valuable work. Is the attribute "weekday_name" in panda version '1.1.3' now deprecated? and instead now we have 'day_name()' method?

[@benjerome9738]

I have a column containing times in the format hour, minute, second (e.g. 00:24:43) and are currently an 'object' type. How can I calculate the average time for this column? I have tried converting to a pandas datetime but this throws up several different errors when I try to calculate the mean? Also, I have three separate groups (full match, T1 and T2) how can I use groupby to figure out the mean times for each group? Thanks

[@raghwendranarayanshandilya4525]

Hi, I liked your way of presentation. It is very precise and nice. I would like to see any tutorial where you explain about taking statistics (mean, median, etc.) of data by filtering any column (say date-time column).

[@deepakebenezer2089]

Hi - qq - I have an excel sheet that has a column that includes dates, some of the dates have errors like '4/4/4/2020' or '/1/12/2020' - - is there a way python generate a dateframe column that lists all of these errors with their corresponding row information?

[@ganwilliam43]

Very good video tutorials! Very much thanks, Kevin.

[@dataschool]

You're very welcome!

[@annakornikova1119]

Very nice, just what I was looking for . Thanks!

[@dataschool]

You're welcome!

[@farzanaislamadiba7987]

Hi, I have applied the format as you had shown. But I am getting error called "AttributeError: 'DataFrame' object has no attribute 'Time'". Time is the column for my Date and time. How to solve it?

[@iubob98]

wow.. you're tutorials are just so awesome !!!!!

[@dataschool]

Thank you!

[@majidm4215]

Hello , I have a dataset with datatime index col. and it is weekly data , do I need to set freq='W' to apply forecasting models such as Holt-Winters, I tried : df.index.freq ='W', and got this error: OverflowError: int too big to convert"" please help me to fix this. Thank you

[@АлексейВереник-г1р]

Great job, bro! Thnx dude!

[@dataschool]

You're welcome!

[@riderblack6401]

Big thanks, Kevin! Great job!

[@dataschool]

You're welcome!

[@vaishnavipadala9457]

Hi, Very useful basics covered in your videos, thank you very much!

[@dataschool]

You're very welcome!

[@j4y0785]

Hi Data School, Just a quick question, I am still new at this and my apologies if this question as been answered already, how would I go about changing multiple columns to datetime? Thank you.

[@AnalyticsMaster]

Hi Kevin, Just like we have 'weekday_name' to know whether it is 'Sunday', 'Monday' etc... what should we use to find the 'month name' ? I serched for help in pandas doc, but not able to find. Kindly suggest.

[@dataschool]

Great question! I'm not sure if there is a built-in way in pandas to do this. I would probably write my own code to do this using the map method, explained here: kzbin.info/www/bejne/hpDUYaehjtapic0

[@francescopallottino4201]

you are my new hero!

[@dataschool]

Ha! Thanks for your kind comment :)

[@carloseduardocorreacoimbra4040]

Kevin, could we define a ts_min and ts_max, and select the events during this interval?

[@myselfandpesit]

Thanks for the tutorial. I have a question. How can i find unique items under a given column as some could have been repeated?

[@retrofutur1st]

thank you for this, I have a lot to learn

[@dataschool]

You're welcome!

[@XinhLe]

4:25 .weekday_name is not working, used day_name() instead (2022 update). Thank u!

[@dataschool]

Thanks for sharing!

[@legiegrieve99]

Exactly what I wanted!! Thanks.

[@dataschool]

You're welcome!

[@robotinthebrain]

You made my day, thank you

[@dataschool]

You're very welcome!

[@subasishbiswal3481]

Hi , I have a column in date/mm/yy i want to remove the yy can i do this ?

[@kirahman2]

thank you so much for this video, it saved me so much time, thank you. Wow. so good.

[@devanshukala626]

I want to show the week of months as per the datetime columns. How can i do that? Please advise.

[@jessicas2978]

I hope you could add some time series data mining analysis for us in the future. I really want to know how to mine time series data in Panda. Thank you very much!

[@dataschool]

Thanks for the suggestion, I'll consider it for the future!

[@nadie7480]

How can I export this "datetime64" (YYYY-MM-DD HH:MM:SS) to a CSV file with the format "DD/MM/YYYY HH:MM" ?

[@HealthyFoodBae_]

Hello, I have 2 large datasets and want to compare time differences by seconds for instance. I want to Group-by a certain column first, and then see the time differences or duration for a certain action. Can I do this in Python

[@dataschool]

I'm sure you can, but it's hard for me to say how off-hand. Sorry!

[@anilkushwaha7164]

Please make a video for Cleaning of "FINANCIAL DATA " Your Tutorial is very Lucid and informative

[@dataschool]

Thanks for your suggestion!

[@anilsangani8862]

AB['Date']=pd.to_datetime(AB.Date) i am getting error Unknown string format: TOTAL my format is dd/mm/yyyy in column Date of a data frame named AB showing dtype as object .

[@u0000-u2x]

DataFrame.resample() is also a very, very useful feature in Pandas for working with time series.

[@dataschool]

Agreed!

[@TorIvanBoine]

awesome videos. been watching quite a few now. So, I'm playing with my gpx running data. And I'm trying to convert the duration of my runs so I can plot them. But I just fail. How would you convert ints like 33:28 and 01:44:42 so it would be understood as 33 minutes and 1 hour 44 minutes and so on?

[@dataschool]

Glad you like the videos! As for your question, it seems like extracting the datetime attributes (hours, minutes, seconds) and then doing the math with those attributes would solve your problem. Hope that helps!

[@prof_albert]

Bravo man, really really useful. 👌💞🤩💪

[@dataschool]

Thank you!

[@metwill]

Thanks, needed just this!

[@dataschool]

You're welcome!

[@metingercek]

I wrote this code after this video date = input("Input your birthday:") dt = pd.to_datetime(date) today = pd.to_datetime('today') day = today - dt print("After you born", day," hours passed")

[@dataschool]

Cool!

[@ranaalameedee9762]

Thank you so much, wonderful explanation

[@dataschool]

You're welcome!

[@Skandawin78]

Python is really not intuitive when compared to R.. datetime comparison makes me split my hair . Still try to figure out how to compare a date in a dataframe ( which in datetime format) to today's date and do some action if they both match.. Any help is appreciated

[@dembobademboba6924]

Very helpful. Thank you so much bro

[@dataschool]

You're welcome!

[@vimultivi]

hi in the last line graph what to do to introduce line with arrow after 1960 to point some annotations and also customise x axis labels ? I want to know more about axis ticks ,labels and formating of axis as per our coveinence and also annotations in graphs .Your teaching is very simple and easy to understand and interesting !

[@dataschool]

You'll need to use matplotlib for customizations. This post should help you get started: pbpython.com/effective-matplotlib.html

[@akahn8311]

thanks for the tutorial. is there any way to change the year column meaning change the year series. In year series - starts with 1930 and then go on till 1933, how to change this to 2013 to 2016 in the csv file ? thanks lot for the time and help ! cheers

[@sebastianplazaponte6766]

How do i subtract dates? I need to get a range between my last date time and 6 months before. Thanks! Your videos are great.

[@dataschool]

I'm not sure off-hand, sorry!

[@balavkrish]

Thank you for the input

[@dataschool]

You're welcome!

[@timpinckney]

Great explanation Kevin. Do you have any videos exclusively on matplotlib?

[@dataschool]

I don't... sorry! I'll consider them for the future!

[@timpinckney]

Thanks Kevin

[@sarikadatta3706]

this is an awesome video but the to_datetime is not working for me, it keeps giving me error like "hour must be in 0..23: 10/11/2006 24:00" I've tried everything

[@deutschejar]

Nice video man, in 9:20 min, if you wanna group by month and year, how could you do that?

[@Yagnik_Sojitra]

I am plotting a chart in that I need to show time only, how to trim time from pandas datetime ?

[@_AbUser]

I did't get what is the difference between ufo.time* and ufo.time.dt* operations. Would You explain plz...

[@rahulkambadur147]

Hello, In My date column has the date of month missing. how do I add the date to the existing column Ex: My column is 04-1982 (which is not in date format) and I want to make it 30-04-1982.. and want to repeat for all the other sections.. please help. and how to add a date if there is no date available

[@dataschool]

I'm sure there's a string method that can help: kzbin.info/www/bejne/mKDJknZmfsieftE&index=12&list=PL5-da3qGB5ICCsgW1MxlZ0Hq8LL5U3u9y

[@NirajSingh-nv2jb]

Great Sir. And please dont says to like videos our hands automatically get clicked on like after watching this.

[@dataschool]

Ha! :)

[@somnathmalik7200]

Thank you, it helped a lot

[@varshadevgankar8242]

hi, i want a function in python that identify which column have date in them??

[@manishsharma2211]

You are gem bro. Thank you

[@dataschool]

Thank you!

[@teklehaimanotaman3150]

Hi Kevin, lets say you have a date column containing only hour, minute and second. While changing into date format using pd.to_datetime, it added automatically years and days. How could keep only hours, minutes and seconds? Thank you.

[@dataschool]

Great question! I can't remember off-hand, but I think you need to use a pandas timedelta object instead of a datetime object.

[@udaykiran-rh8es]

could you please make videos on Matplotlib ,seaborn and power bi also for data analysis . I like your way of teaching.

[@dataschool]

Thanks for your suggestion!

[@udaykiran-rh8es]

@@dataschool EDA also

[@nousername2520]

@Data School Do you have any video on how to fill missing dates with zero (desired number) in a large csv file?

[@dataschool]

kzbin.info/www/bejne/nHSwo4KVi9-Ygpo

[@robertowbatista]

Thanks for the video. Quick question: How can I change the ​data type of several columns at the same time?

[@dataschool]

Great question! See part 3 of this video: kzbin.info/www/bejne/Y3_Fimp7bs1-rs0

[@monica-ww8gi]

THANK YOU! Very helpful video!! :)

[@dataschool]

You're very welcome! :)

[@hiteshkumavat6335]

The videos is with very nice explanation .I am getting the time data only in hour:min:sec format and when i convert it from object to time then it gives also the current date with time stamp. I want to fill the missing seconds values so is there any other function available ?

[@dataschool]

I'm not sure off-hand, I'm sorry!

[@ariramkilowan8051]

As always (literally), Great video! Quick question, Is it trivial to combine the times from two independent time series' then resample both time series' (the data at those times) to the new combined times? I can't quite figure out how to do this pythonically. pandas.resample doesn't seem to be too friendly when using irregular timestamps.

[@dataschool]

Thanks for your kind comment! Regarding your question, it's hard for me to envision exactly how to do this... I'm sorry! Feel free to let me know if you figure out a good strategy.

[@sdoken]

It is interesting that while ufo.dtypes tells you the type as “datetime64[ns]”, when you look at each element individually type(ufo.loc[0, ‘Time’]) returns a Timestamp object. Seems a bit inconsistent? No big deal, I’ll just have to get used to it. With timedelta however, both the column and the individual elements are of type timedelta.

[@dataschool]

Yes, it takes some getting used to.... datetime is the type of the Series, timestamp is the type of the element, and timedelta is the type of a time range!

[@enian82]

For some reason This through s an error ufo.Time.dt.weekday_name but ufo.Time.dt.weekday this works any idea.... Thank you

[@yop1001]

Love this course as usual. Thank you. I have a question. How could I do to "aggregate" rows by month or days ? (each row is a sell of a product over years)

[@yop1001]

Note to myself : use resample and learn to read comments (df3 = data.resample('M', on='date').sum())

[@pradeepmada7515]

when i am trying to convert my data column into datetype its showing an error unknown string format