How do I work with dates and times in pandas?

[@dataschool]

Starting in pandas version 0.18.1, you can create a new datetime column directly from a DataFrame, based solely on the column names! It's a useful trick, which I explain in this video: kzbin.info/www/bejne/Y3_Fimp7bs1-rs0

[@cossanfran]

Hello! I have a table with the date column. I want to group the data by month / year how do I do this?

[@dikshyantthapa3367]

I love you Bruh.. 😂.. No homo..thanks a lot!!!!

[@cobbdouglas690]

You've saved my job on multiple occasions sir, thank you.

[@dataschool]

That's awesome to hear! 🙌

[@kasiditauable]

I am an aspiring data scientist. I just found a series of your videos. Thank you for doing this for all of us. Keep doing great work!

[@dataschool]

Thanks for your kind words, and good luck to you!

[@Martin-lv1xw]

You are naturally born to impart knowledge. Thanks for your videos br!

[@dataschool]

Wow, I really appreciate your kind words! 🙏

[@goktugerce]

I was actually going to ask you about this. I started learning Pandas thanks to your video series and I feel like I learnt a lot. Thanks for all the awesome videos.

[@dataschool]

Wow, that's really great to hear! You're very welcome!

[@goktugerce]

Actually, I still need some help. I want to create a column for "year/month" from datetime column. For example, if datetime is 2016-07-06 15:56:19, I want to map "2016-07" into a column. Of course I can get first seven characters by converting it to a string, but what is the correct way to do this? I have following the following lambda function: get_month = lambda x: '{}-{:02}'.format(x.to_datetime().year, x.to_datetime().month) my_df["year_month"] = my_df["timestamp"].map(get_month) For now, I do it like this but I am sure a better and more efficient way exists for the job. I'd be glad if you can help!

[@dataschool]

If your desired end result is a string (such as '2016-07'), then I think using string methods is the way to go! Perhaps something like this: ufo.Time.dt.year.astype(str).str.cat(ufo.Time.dt.month.astype(str), sep='-') However, there is probably an even simpler approach that I'm not thinking of...

[@goktugerce]

Awesome. It really is simpler than what I was doing. Thanks!

[@archidar1]

Hi, I know its a year late, but in case you (or anyone else is interested) "pd.Series.dt.strftime" is an easy way to output dates as strings in whatever format you like. pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.dt.strftime.html#pandas.Series.dt.strftime

[@witotiw]

2022 and you're still saving us. Thanks for the excelent content

[@dataschool]

You're welcome!

[@akashmishra2661]

I understood each and everything in this video and it helped me alot for my project. I just want to thank the instructor.

[@dataschool]

Great to hear!

[@oleholeynikov8659]

5 years video, but the best on KZbin!!!

[@dataschool]

Thanks very much for your kind words!

[@christopherclark2205]

Phenomenal description of working with dates and times in Pandas, very helpful.

[@Diachron]

I've only recently stumbled onto your videos. Very clear and concise delivery. Good job!

[@dataschool]

Thanks so much! Glad you are enjoying them.

[@Cam-p2z]

These are the first videos I look for when I have pandas questions.

[@dataschool]

Thanks!

[@jsmit7730]

I'm impressed. Simple explanations with examples. subbed and hit that bell. Thanks for the vid!

[@dataschool]

Awesome, thank you!

[@cmdeckermusic]

Thanks for this! You explain things very clearly and concisely.

[@dataschool]

Thanks!

[@aartiladdha01]

All your videos are worth watching. I have learned a lot about pandas just from your videos. Thanks a ton :)

[@dataschool]

Thanks Aarti!

[@boubacaramaiga4408]

I love the way you teach, easy to follow and to understand. Many Thanks.

[@dataschool]

You're welcome!

[@senz3757]

Thanks for your great video! It seems pandas.Series.dt.weekday_name is removed in pandas version 0.23.0. and one can use pandas.Series.dt.day_name() instead.

[@wilmarperez4962]

Thanks for the tip. Was facing the same issue!

[@Vishnu_Datta_1698]

Thanks a lot

[@jasper8291]

Thanks :)

[@RossThompson89]

I really enjoyed this video and your teaching style! To the point, but well explained at a nice pace. Thank you for this :)

[@dataschool]

Thank you so much!

[@AsyraffLatiffi]

I've been watching your videos for a while now but never got the chance to comment on it i just want to say keep up the great work! You are just awesome!

[@dataschool]

Thanks very much for your kind words! Much appreciated :)

[@benjerome9738]

I have a column containing times in the format hour, minute, second (e.g. 00:24:43) and are currently an 'object' type. How can I calculate the average time for this column? I have tried converting to a pandas datetime but this throws up several different errors when I try to calculate the mean? Also, I have three separate groups (full match, T1 and T2) how can I use groupby to figure out the mean times for each group? Thanks

[@XinhLe]

4:25 .weekday_name is not working, used day_name() instead (2022 update). Thank u!

[@dataschool]

Thanks for sharing!

[@payalbhatia5244]

@Data School , Mark, you are just amazing. :) You make it appear everything quite simple. My humble request is to kindle make a series on machine learning algorithms too.

[@dataschool]

Thanks for your kind words and your suggestion!

[@deepakpanigrahi9601]

Data School is the best of all resource available on Pandas. thanks a ton!!

[@dataschool]

Thanks!

[@aakashvarma9960]

Excellent explanation .I suggested these video series to most of my friends.

[@dataschool]

Thanks so much! I really appreciate you spreading the word.

[@marco4570]

That bonus is what I needed. Thank you so much!

[@dataschool]

You are so welcome!

[@slavaslavia4085]

New to Pandas, new to your channel, and soon New year =) Thanks for the videos!

[@dataschool]

You're very welcome!

[@python_by_abhishek]

Hi Kevin I have a doubt.. I have 1 column in which time is 42368.149155 When I convert it into year, days, month, hrs, min, sec I am getting 1970-01-01, 17:22:13.453068 I read that 1970 is the default year. How can I convert it into some other year, say 2016 or any other year. Kindly help.

[@dataschool]

You might need to adjust the "unit", see examples here: pandas.pydata.org/docs/reference/api/pandas.to_datetime.html

[@python_by_abhishek]

@@dataschool Thank You Kevin. Got it now.

[@ΠαναγιώταΜωραΐτη-τ2μ]

So many great videos . Absolutely guidness. thanks from GREECE !

[@dataschool]

Thank you!

[@termica7349]

Thnks for this explanation buddy! So clear and concise!

[@dataschool]

Thanks for your kind words!

[@goktugerce]

For anyone who wants to convert timezones, here's how you do it. First, you need to specify which timezone the date originally belongs to. In my case, it was UTC (Coordinated Universal Time). In my DataFrame (let's call it df), I have a column named "timestamp", with type datetime. I localized the column first: timestamp_utc = df["timestamp"].dt.tz_localize("UTC") Then I overwrite the timestamp column: df["timestamp"] = timestamp_utc.dt.tz_convert("Europe/Istanbul") I hope this method is correct and it helps someone!

[@dataschool]

Awesome! Thanks so much for taking the time to share your code.

[@SetarehFasihnia]

Hi, I just tried your code but it returned all NaT values? my original column was in datetime format and UTC + x timezone. I tried to convert it to US/Eastern time. Update: Apologies, turns out the error lies in trying to convert my 'timestamp' equivalent column into date time from object and inserting errors=coerce produced the NaT values.

[@SatishKumar-yz4tn]

Thank you so much in taking time to explain so nicely.

[@dataschool]

You're very welcome!

[@pranavmendiratta3166]

loved the tutorial, cleared my doubts! I like how you explain so patiently.

[@pythonradio6497]

I have a series of " 23:21:06 31/07/2019" format how to convert this to a time series

[@rodrigoviverosphoto]

This is EXACTLY was a looking for. I love you.

[@dataschool]

Awesome!

[@nowyouknow2249]

Thanks so much. You are one of the best teachers I have ever known. Thanks so much once more you are a darling.

[@dataschool]

Wow! Thank you so much for your kind words! :)

[@sarikadatta3706]

this is an awesome video but the to_datetime is not working for me, it keeps giving me error like "hour must be in 0..23: 10/11/2006 24:00" I've tried everything

[@sdoken]

When he says at 2:46 that to_datetime() is a top-level function, what does it mean? Is to_datetime() a pandas function? Or is it a python function? Thanks!

[@dataschool]

It's a pandas function. It's called a top-level function because it's at the top level of the pandas namespace, meaning you call it using 'pd.function_name(df.series_name)' rather than 'df.series_name.function_name()'. The latter is called a Series method. Hope that helps!

[@TorIvanBoine]

awesome videos. been watching quite a few now. So, I'm playing with my gpx running data. And I'm trying to convert the duration of my runs so I can plot them. But I just fail. How would you convert ints like 33:28 and 01:44:42 so it would be understood as 33 minutes and 1 hour 44 minutes and so on?

[@dataschool]

Glad you like the videos! As for your question, it seems like extracting the datetime attributes (hours, minutes, seconds) and then doing the math with those attributes would solve your problem. Hope that helps!

[@sinabaghaei3504]

Special thanks for your valuable work. Is the attribute "weekday_name" in panda version '1.1.3' now deprecated? and instead now we have 'day_name()' method?

[@frankdeng624]

HI Kevin, I met a problem when I read the csv file with date time. I use the following code to read the csv file, but got 2 warnings said PST and PDT can't be understood... Can you please help me solve this problem? thank you! data = pd.read_csv("datetime.csv",parse_dates = ['date/time']) Mar 3, 2019 12:16:44 AM PST UnknownTimezoneWarning: tzname PST identified but not understood. Pass `tzinfos` argument in order to correctly return a timezone-aware datetime. In a future version, this will raise an exception. category=UnknownTimezoneWarning)

[@dataschool]

It's hard to say without investigating, I'm sorry!

[@gurkanyesilyurt4461]

You're king of this area man!!!!

[@dataschool]

Thank you!

[@jannahawkins865]

This is a great video. Thank you so much! Also, I wish the pandas API reference still looked like it does in this older video. It's harder to read now.

[@dataschool]

I agree...

[@carloseduardocorreacoimbra4040]

Kevin, could we define a ts_min and ts_max, and select the events during this interval?

[@farzanaislamadiba7987]

Hi, I have applied the format as you had shown. But I am getting error called "AttributeError: 'DataFrame' object has no attribute 'Time'". Time is the column for my Date and time. How to solve it?

[@leandrorberto]

Excellent class! As always! Cheers!

[@dataschool]

Thank you!

[@Gargi_3106]

I'm unable to use ufo.Time.dt.weekday_name as an attribute. What might be wrong?

[@bloomingtruth3399]

Try ufo.Time.dt.day_name()

[@deutschejar]

Nice video man, in 9:20 min, if you wanna group by month and year, how could you do that?

[@gustavoviana1178]

I'm so thankful for your tutorials

[@dataschool]

You're welcome!

[@deepakebenezer2089]

Hi - qq - I have an excel sheet that has a column that includes dates, some of the dates have errors like '4/4/4/2020' or '/1/12/2020' - - is there a way python generate a dateframe column that lists all of these errors with their corresponding row information?

[@samcathcart5388]

Hi DataSchool tk u for gr8 vid on working with dates and times. I am trying to work out how to group data for days, months and years in the same plot, e.g. bar graphs for months and different colours for the years

[@shilpikulshrestha9487]

Hello sir, you are awesome teacher. Great videos Thank you very much

[@dataschool]

Thanks so much for your kind words!

[@majidm4215]

Hello , I have a dataset with datatime index col. and it is weekly data , do I need to set freq='W' to apply forecasting models such as Holt-Winters, I tried : df.index.freq ='W', and got this error: OverflowError: int too big to convert"" please help me to fix this. Thank you

[@j4y0785]

Hi Data School, Just a quick question, I am still new at this and my apologies if this question as been answered already, how would I go about changing multiple columns to datetime? Thank you.

[@manumadhavan9264]

Hi, I have Hour column in format- 0,1,2,...22,23.How can i convert to 00:00:00,01:00:00,02:00:00...,22:00:00,23:00:00. Thanks in Advance Manu Madhavan

[@dataschool]

I'm not sure the easiest way to do that off-hand. Sorry! And good luck!

[@OmarMiranda]

for h in hour_column: datetime.time(h, 0, 0)

[@devanshukala626]

I want to show the week of months as per the datetime columns. How can i do that? Please advise.

[@myselfandpesit]

Thanks for the tutorial. I have a question. How can i find unique items under a given column as some could have been repeated?

[@jameszanzarelli9255]

I tried to use pd.to_datetime(df.variable) with my date variable (read as an object), but I'm getting this error: OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 1410-02-25 00:00:00 What am I doing wrong?

[@dataschool]

I'm not sure, sorry! I haven't seen that error before.

[@jameszanzarelli9255]

found it! python couldn't handle a date with year 1410

[@dataschool]

Ha! Good to know!

[@iubob98]

wow.. you're tutorials are just so awesome !!!!!

[@dataschool]

Thank you!

[@gilbertgutabaga8255]

I did like this but getting error OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 1-01-21 00:00:00 , from tutorial example on web scraping of Tramp's lie

[@dataschool]

I'm sorry, I'm not sure what the cause of this error would be.

[@Jazzcrime1]

@@dataschool Great Tutorial! i also encountered this problem because of a date outlier (year 2981). Pandas would read the column as object and was unable to convert to datetime see: stackoverflow.com/questions/32888124/pandas-out-of-bounds-nanosecond-timestamp-after-offset-rollforward-plus-adding-a i added this parameter df['date'] = pd.to_datetime(df['date'], errors = 'coerce') seems to work, the column is converted and the outlier has NaN value.

[@annakornikova1119]

Very nice, just what I was looking for . Thanks!

[@dataschool]

You're welcome!

[@jjunior1283]

Thanks for the tutorials. I want to compute the difference between two dates and return the result in integer. Much like the last example you showed. can that number of days be returned as integer? thanks

[@dataschool]

Actually, this already returns the result as an integer: (ufo.Time.max() - ufo.Time.min()).days

[@ganwilliam43]

Very good video tutorials! Very much thanks, Kevin.

[@dataschool]

You're very welcome!

[@HealthyFoodBae_]

Hello, I have 2 large datasets and want to compare time differences by seconds for instance. I want to Group-by a certain column first, and then see the time differences or duration for a certain action. Can I do this in Python

[@dataschool]

I'm sure you can, but it's hard for me to say how off-hand. Sorry!

[@AnalyticsMaster]

Hi Kevin, Just like we have 'weekday_name' to know whether it is 'Sunday', 'Monday' etc... what should we use to find the 'month name' ? I serched for help in pandas doc, but not able to find. Kindly suggest.

[@dataschool]

Great question! I'm not sure if there is a built-in way in pandas to do this. I would probably write my own code to do this using the map method, explained here: kzbin.info/www/bejne/hpDUYaehjtapic0

[@didierleprince6106]

A Huge Merci from Les Sables d'Olonne (France) (:

[@dataschool]

You're very welcome!

[@VijayKumar-zx5bm]

Excellent videos.Please consider giving tutorials on time series forecasting ( with various statistical models ) with Pandas.

[@dataschool]

Thanks for the suggestion! I'll consider it for the future.

[@debashissahoo5031]

Your video is really good, we'll be really helpful, if you make some more videos on Dates and Times. Thank you.

[@dataschool]

I cover it a bit more in this series: kzbin.info/aero/PL5-da3qGB5IBITZj_dYSFqnd_15JgqwA6

[@debashissahoo5031]

thank you, Now I am planning to cover this series too.

[@hiteshkumavat6335]

The videos is with very nice explanation .I am getting the time data only in hour:min:sec format and when i convert it from object to time then it gives also the current date with time stamp. I want to fill the missing seconds values so is there any other function available ?

[@dataschool]

I'm not sure off-hand, I'm sorry!

[@subasishbiswal3481]

Hi , I have a column in date/mm/yy i want to remove the yy can i do this ?

[@ssam7195]

Thanks for your videos. I have data df['DateTime'] as '02/12/2019 11:00:00' , ...02/13/2019 23:00:00'., 02/13/2019 24:00:00' in column 48hrs of hourly data. While plot only ' 02/XX/2019 24:00:00' ( only 24:00:00 ) hrs point missing on graph. :( How to fix this? Please support me.

[@kunalkumar2717]

how can we handle ranges ( suppose salary ranges ) in a dataset? it would be great if any of you can tell.

[@vaishnavipadala9457]

Hi, Very useful basics covered in your videos, thank you very much!

[@dataschool]

You're very welcome!

[@vimultivi]

hi in the last line graph what to do to introduce line with arrow after 1960 to point some annotations and also customise x axis labels ? I want to know more about axis ticks ,labels and formating of axis as per our coveinence and also annotations in graphs .Your teaching is very simple and easy to understand and interesting !

[@dataschool]

You'll need to use matplotlib for customizations. This post should help you get started: pbpython.com/effective-matplotlib.html

[@dineshkundu4623]

If I want specific dates in train data and remaining in test data how can we do tht

[@varshadevgankar8242]

hi, i want a function in python that identify which column have date in them??

[@sdoken]

It is interesting that while ufo.dtypes tells you the type as “datetime64[ns]”, when you look at each element individually type(ufo.loc[0, ‘Time’]) returns a Timestamp object. Seems a bit inconsistent? No big deal, I’ll just have to get used to it. With timedelta however, both the column and the individual elements are of type timedelta.

[@dataschool]

Yes, it takes some getting used to.... datetime is the type of the Series, timestamp is the type of the element, and timedelta is the type of a time range!

[@akahn8311]

thanks for the tutorial. is there any way to change the year column meaning change the year series. In year series - starts with 1930 and then go on till 1933, how to change this to 2013 to 2016 in the csv file ? thanks lot for the time and help ! cheers

[@АлексейВереник-г1р]

Great job, bro! Thnx dude!

[@dataschool]

You're welcome!

[@pradeepmada7515]

when i am trying to convert my data column into datetype its showing an error unknown string format

[@ariramkilowan8051]

As always (literally), Great video! Quick question, Is it trivial to combine the times from two independent time series' then resample both time series' (the data at those times) to the new combined times? I can't quite figure out how to do this pythonically. pandas.resample doesn't seem to be too friendly when using irregular timestamps.

[@dataschool]

Thanks for your kind comment! Regarding your question, it's hard for me to envision exactly how to do this... I'm sorry! Feel free to let me know if you figure out a good strategy.

[@Yagnik_Sojitra]

I am plotting a chart in that I need to show time only, how to trim time from pandas datetime ?

[@yucelozyazgan]

Basically how can i do a simple math operation like increasing 2 months to the month of any date when i do that as at the following, i'm taking an error like : 'datetime.date' objects is not writable not wirtable *** dt = pd.to_datetime('2016/9/28') dt.month = dt.month + 2

[@dataschool]

Great question! Here's how I would do it: dt = pd.to_datetime('2016/9/28') td = pd.to_timedelta('60 days') dt + td I don't think you can specify time units as months because of the ambiguity. It makes sense when you say 9/28 plus 2 months (11/28), but what about 12/31 plus 2 months (2/31?)

[@yucelozyazgan]

Holly teacher! :D

[@_AbUser]

I did't get what is the difference between ufo.time* and ufo.time.dt* operations. Would You explain plz...

[@teklehaimanotaman3150]

Hi Kevin, lets say you have a date column containing only hour, minute and second. While changing into date format using pd.to_datetime, it added automatically years and days. How could keep only hours, minutes and seconds? Thank you.

[@dataschool]

Great question! I can't remember off-hand, but I think you need to use a pandas timedelta object instead of a datetime object.

[@anilsangani8862]

AB['Date']=pd.to_datetime(AB.Date) i am getting error Unknown string format: TOTAL my format is dd/mm/yyyy in column Date of a data frame named AB showing dtype as object .

[@galymzhankenesbekov2924]

thank you for your video! I have a question. for if one of your columns has the following values,like 1.x,y,d,z,w 2.x,t,y,d,z 3,x,y,t,d,w and so on. numbers mean different rows So, i need split the dataFrame into x,y,d columns(by values). P.S. so how to tell the program that if particular value within a selected column satisfies my desire I need to take that value and put them into seperate column

[@dataschool]

Sorry, I don't quite understand your question... good luck!

[@09january]

Is there any simple method to calculate sum of consecutive days say employees who worked continuously for 2 weeks including weekend? Can this be done using timedelta?

[@dataschool]

It's hard for me to say off-hand, sorry!

[@townheadbluesboy]

Hi Thanks for all your videos - what's the easiest way to get seconds since the epoch from a datetime object - Python 2 ?

[@dataschool]

Sorry, I don't know off-hand. Good luck!

[@riderblack6401]

Big thanks, Kevin! Great job!

[@dataschool]

You're welcome!

[@rashmikoparde826]

I have a csv file which consist of placeid and date .I want to calculate in a particular month how many times is the particular place visited..how this could be done in python code..Please help @ Data School

[@dataschool]

Perhaps you could store the 'month' attribute from the 'date' column as a new column, then filter the DataFrame by 'placeid' column, and then check the shape of the DataFrame. This video might be helpful to you: kzbin.info/www/bejne/aHKpeIOag9NnfK8

[@timpinckney]

Great explanation Kevin. Do you have any videos exclusively on matplotlib?

[@dataschool]

I don't... sorry! I'll consider them for the future!

[@timpinckney]

Thanks Kevin

[@nousername2520]

@Data School Do you have any video on how to fill missing dates with zero (desired number) in a large csv file?

[@dataschool]

kzbin.info/www/bejne/nHSwo4KVi9-Ygpo

[@Nehisniche]

Hey thanks for the video really cleared so many doubts, I had one question tho, I extracted the microsecond from my date-time format and now if I want to plot my microsecond as x axis and all 3 other columns(that has float values) as y axis how do I do that?

[@dataschool]

I'm not sure, sorry!

[@nadie7480]

How can I export this "datetime64" (YYYY-MM-DD HH:MM:SS) to a CSV file with the format "DD/MM/YYYY HH:MM" ?

[@matildanordahl3500]

What if I want to know the mean of entries for each month? Is there more ufo-sightings in summer as an example?

[@dataschool]

Is this what you are looking for? ufo['Month'] = ufo.Time.dt.month ufo.Month.value_counts().sort_index()

[@Skandawin78]

Python is really not intuitive when compared to R.. datetime comparison makes me split my hair . Still try to figure out how to compare a date in a dataframe ( which in datetime format) to today's date and do some action if they both match.. Any help is appreciated

[@raghwendranarayanshandilya4525]

Hi, I liked your way of presentation. It is very precise and nice. I would like to see any tutorial where you explain about taking statistics (mean, median, etc.) of data by filtering any column (say date-time column).

[@debaniedalaielizondoflores4404]

Is there any way to separate the time (h,m,s) that's on the same column of an imported excel data frame and pass it all to seconds?

[@dataschool]

I'm sorry, I don't understand your question. Could you clarify? Thanks!

[@hajaya1]

Thanks for the awesome videos! Any thoughts on how I can use time series functionality with a data set that contains multiple rows with the same date? I mean I use day-level dates for one-year data, and every day may contain multiple observations (my data set has no unique column). Thanks

[@dataschool]

I think what you're looking for is resampling. Thanks for your kind comment, and good luck!

[@diakitem.youssouf7916]

Thanks You. But a have a query If I want to get the data between two dates How I can do ?

[@dataschool]

I don't know the code off-hand, I'm sorry!

[@diakitem.youssouf7916]

@@dataschool OK. Thanks . But according some of my research to have the data between a period we should set data index as time stamp type.

[@robertowbatista]

Thanks for the video. Quick question: How can I change the ​data type of several columns at the same time?

[@dataschool]

Great question! See part 3 of this video: kzbin.info/www/bejne/Y3_Fimp7bs1-rs0