How do I work with dates and times in pandas?

[@dataschool]

Starting in pandas version 0.18.1, you can create a new datetime column directly from a DataFrame, based solely on the column names! It's a useful trick, which I explain in this video: kzbin.info/www/bejne/Y3_Fimp7bs1-rs0

[@cossanfran]

Hello! I have a table with the date column. I want to group the data by month / year how do I do this?

[@dikshyantthapa3367]

I love you Bruh.. 😂.. No homo..thanks a lot!!!!

[@cobbdouglas690]

You've saved my job on multiple occasions sir, thank you.

[@dataschool]

That's awesome to hear! 🙌

[@senz3757]

Thanks for your great video! It seems pandas.Series.dt.weekday_name is removed in pandas version 0.23.0. and one can use pandas.Series.dt.day_name() instead.

[@wilmarperez4962]

Thanks for the tip. Was facing the same issue!

[@Vishnu_Datta_1698]

Thanks a lot

[@jasper8291]

Thanks :)

[@XinhLe]

4:25 .weekday_name is not working, used day_name() instead (2022 update). Thank u!

[@dataschool]

Thanks for sharing!

[@jsmit7730]

I'm impressed. Simple explanations with examples. subbed and hit that bell. Thanks for the vid!

[@dataschool]

Awesome, thank you!

[@shilpikulshrestha9487]

Hello sir, you are awesome teacher. Great videos Thank you very much

[@dataschool]

Thanks so much for your kind words!

[@АлексейВереник-г1р]

Great job, bro! Thnx dude!

[@dataschool]

You're welcome!

[@termica7349]

Thnks for this explanation buddy! So clear and concise!

[@dataschool]

Thanks for your kind words!

[@baltalutjens5569]

thank you, you fucking save me for my homework dude apreciation

[@dataschool]

😄

[@akashmishra2661]

I understood each and everything in this video and it helped me alot for my project. I just want to thank the instructor.

[@dataschool]

Great to hear!

[@goktugerce]

I was actually going to ask you about this. I started learning Pandas thanks to your video series and I feel like I learnt a lot. Thanks for all the awesome videos.

[@dataschool]

Wow, that's really great to hear! You're very welcome!

[@goktugerce]

Actually, I still need some help. I want to create a column for "year/month" from datetime column. For example, if datetime is 2016-07-06 15:56:19, I want to map "2016-07" into a column. Of course I can get first seven characters by converting it to a string, but what is the correct way to do this? I have following the following lambda function: get_month = lambda x: '{}-{:02}'.format(x.to_datetime().year, x.to_datetime().month) my_df["year_month"] = my_df["timestamp"].map(get_month) For now, I do it like this but I am sure a better and more efficient way exists for the job. I'd be glad if you can help!

[@dataschool]

If your desired end result is a string (such as '2016-07'), then I think using string methods is the way to go! Perhaps something like this: ufo.Time.dt.year.astype(str).str.cat(ufo.Time.dt.month.astype(str), sep='-') However, there is probably an even simpler approach that I'm not thinking of...

[@goktugerce]

Awesome. It really is simpler than what I was doing. Thanks!

[@archidar1]

Hi, I know its a year late, but in case you (or anyone else is interested) "pd.Series.dt.strftime" is an easy way to output dates as strings in whatever format you like. pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.dt.strftime.html#pandas.Series.dt.strftime

[@bhaskarsharan4280]

I've created a dataframe in python using pandas. The index used is a series of timestamp of type int64. However, for time series analysis, the index need to be type dates. Can somebody help me to do the conversion ? first few rows of the dataset is 'Elapsed time','ECG I' 'hh:mm:ss.mmm','mV' '0:00.000',-0.08 '0:00.002',-0.08 '0:00.004',-0.07 '0:00.006',-0.07 '0:00.008',-0.09 '0:00.010',-0.09 '0:00.012',-0.10 '0:00.014',-0.10 '0:00.016',-0.10 thanks in advance :)

[@RossThompson89]

I really enjoyed this video and your teaching style! To the point, but well explained at a nice pace. Thank you for this :)

[@dataschool]

Thank you so much!

[@prof_albert]

Bravo man, really really useful. 👌💞🤩💪

[@dataschool]

Thank you!

[@sinabaghaei3504]

Special thanks for your valuable work. Is the attribute "weekday_name" in panda version '1.1.3' now deprecated? and instead now we have 'day_name()' method?

[@goktugerce]

For anyone who wants to convert timezones, here's how you do it. First, you need to specify which timezone the date originally belongs to. In my case, it was UTC (Coordinated Universal Time). In my DataFrame (let's call it df), I have a column named "timestamp", with type datetime. I localized the column first: timestamp_utc = df["timestamp"].dt.tz_localize("UTC") Then I overwrite the timestamp column: df["timestamp"] = timestamp_utc.dt.tz_convert("Europe/Istanbul") I hope this method is correct and it helps someone!

[@dataschool]

Awesome! Thanks so much for taking the time to share your code.

[@SetarehFasihnia]

Hi, I just tried your code but it returned all NaT values? my original column was in datetime format and UTC + x timezone. I tried to convert it to US/Eastern time. Update: Apologies, turns out the error lies in trying to convert my 'timestamp' equivalent column into date time from object and inserting errors=coerce produced the NaT values.

[@nadie7480]

How can I export this "datetime64" (YYYY-MM-DD HH:MM:SS) to a CSV file with the format "DD/MM/YYYY HH:MM" ?

[@majidm4215]

Hello , I have a dataset with datatime index col. and it is weekly data , do I need to set freq='W' to apply forecasting models such as Holt-Winters, I tried : df.index.freq ='W', and got this error: OverflowError: int too big to convert"" please help me to fix this. Thank you

[@kasiditauable]

I am an aspiring data scientist. I just found a series of your videos. Thank you for doing this for all of us. Keep doing great work!

[@dataschool]

Thanks for your kind words, and good luck to you!

[@farzanaislamadiba7987]

Hi, I have applied the format as you had shown. But I am getting error called "AttributeError: 'DataFrame' object has no attribute 'Time'". Time is the column for my Date and time. How to solve it?

[@deepakebenezer2089]

Hi - qq - I have an excel sheet that has a column that includes dates, some of the dates have errors like '4/4/4/2020' or '/1/12/2020' - - is there a way python generate a dateframe column that lists all of these errors with their corresponding row information?

[@Skandawin78]

Python is really not intuitive when compared to R.. datetime comparison makes me split my hair . Still try to figure out how to compare a date in a dataframe ( which in datetime format) to today's date and do some action if they both match.. Any help is appreciated

[@sarikadatta3706]

this is an awesome video but the to_datetime is not working for me, it keeps giving me error like "hour must be in 0..23: 10/11/2006 24:00" I've tried everything

[@_AbUser]

I did't get what is the difference between ufo.time* and ufo.time.dt* operations. Would You explain plz...

[@kushjuvekar1080]

Anyone watching this today? I am getting 'DataFrame' object has no attribute 'to_datetime'

[@debashissahoo5031]

Your video is really good, we'll be really helpful, if you make some more videos on Dates and Times. Thank you.

[@dataschool]

I cover it a bit more in this series: kzbin.info/aero/PL5-da3qGB5IBITZj_dYSFqnd_15JgqwA6

[@debashissahoo5031]

thank you, Now I am planning to cover this series too.

[@saiful305]

How I should sort "How many trips are occurred per month?" and "Which hour of the day had the most frequent trips?"

[@samcathcart5388]

Hi DataSchool tk u for gr8 vid on working with dates and times. I am trying to work out how to group data for days, months and years in the same plot, e.g. bar graphs for months and different colours for the years

[@Martin-lv1xw]

You are naturally born to impart knowledge. Thanks for your videos br!

[@dataschool]

Wow, I really appreciate your kind words! 🙏

[@shivanshsahu3781]

i think in new versions "weekday_name" is replaced by "day_name".

[@aartiladdha01]

All your videos are worth watching. I have learned a lot about pandas just from your videos. Thanks a ton :)

[@dataschool]

Thanks Aarti!

[@boubacaramaiga4408]

I love the way you teach, easy to follow and to understand. Many Thanks.

[@dataschool]

You're welcome!

[@pythonradio6497]

I have a series of " 23:21:06 31/07/2019" format how to convert this to a time series

[@cmdeckermusic]

Thanks for this! You explain things very clearly and concisely.

[@dataschool]

Thanks!

[@deutschejar]

Nice video man, in 9:20 min, if you wanna group by month and year, how could you do that?

[@subasishbiswal3481]

Hi , I have a column in date/mm/yy i want to remove the yy can i do this ?

[@frankdeng624]

HI Kevin, I met a problem when I read the csv file with date time. I use the following code to read the csv file, but got 2 warnings said PST and PDT can't be understood... Can you please help me solve this problem? thank you! data = pd.read_csv("datetime.csv",parse_dates = ['date/time']) Mar 3, 2019 12:16:44 AM PST UnknownTimezoneWarning: tzname PST identified but not understood. Pass `tzinfos` argument in order to correctly return a timezone-aware datetime. In a future version, this will raise an exception. category=UnknownTimezoneWarning)

[@dataschool]

It's hard to say without investigating, I'm sorry!

[@kunalkumar2717]

how can we handle ranges ( suppose salary ranges ) in a dataset? it would be great if any of you can tell.

[@lyejiajun]

Hey ! Great video. In the example where you select all data > than a certain date , i want to do something similar , except this time i want to do it in terms of date range . EG Start date = today (19-Jun-19) Split into 4 columns (4 diff category) Very old (20 years old or older) old (newer than 20years old , older than 10 years old) new (10 years old to 3 years old ) ... etc I have tried Something like print(data['WHOIS_REGDATE'].loc[ts10 ts20]) (where ts10 is 10 yrs ago and ts20 is 20 years ago) and i have error. I am certain i am doing it wrongly. Do you have any advice on how i should approach this problem? Thank you for taking your time to read this

[@dataschool]

Sorry, it's hard for me to help with this... good luck!

[@balavkrish]

Thank you for the input

[@dataschool]

You're welcome!

[@payalbhatia5244]

@Data School , Mark, you are just amazing. :) You make it appear everything quite simple. My humble request is to kindle make a series on machine learning algorithms too.

[@dataschool]

Thanks for your kind words and your suggestion!

[@devanshukala626]

I want to show the week of months as per the datetime columns. How can i do that? Please advise.

[@pradeepmada7515]

when i am trying to convert my data column into datetype its showing an error unknown string format

[@metingercek]

I wrote this code after this video date = input("Input your birthday:") dt = pd.to_datetime(date) today = pd.to_datetime('today') day = today - dt print("After you born", day," hours passed")

[@dataschool]

Cool!

[@nanaosanasam9937]

Hello!!! I am facing an issue regarding the DateTime problem in the data set. the problem is that when i execute the to_datetime function, there is an error pop out. the error is given below. The question is how to handle such kind of dataset. "ValueError: ('Unknown string format:', '25 - Apr-16')"" I need your help.

[@dataschool]

You may have to define the format for to_datetime so that it understands the format. Hope that helps!

[@yucelozyazgan]

Basically how can i do a simple math operation like increasing 2 months to the month of any date when i do that as at the following, i'm taking an error like : 'datetime.date' objects is not writable not wirtable *** dt = pd.to_datetime('2016/9/28') dt.month = dt.month + 2

[@dataschool]

Great question! Here's how I would do it: dt = pd.to_datetime('2016/9/28') td = pd.to_timedelta('60 days') dt + td I don't think you can specify time units as months because of the ambiguity. It makes sense when you say 9/28 plus 2 months (11/28), but what about 12/31 plus 2 months (2/31?)

[@yucelozyazgan]

Holly teacher! :D

[@geetanjaliusb2823]

how can i calculate median for column of type datetime64 ns

[@TheMeltone1]

How would you convert from a column of times in UTC?

[@slavaslavia4085]

New to Pandas, new to your channel, and soon New year =) Thanks for the videos!

[@dataschool]

You're very welcome!

[@ΠαναγιώταΜωραΐτη-τ2μ]

So many great videos . Absolutely guidness. thanks from GREECE !

[@dataschool]

Thank you!

[@DookyButter]

For anyone using pandas version (pd.__version__) 1.0.3 and higher, you will have to type ufo.Time.dt.day_name() to get the weekday name.

[@dataschool]

Good to know, thank you!

[@metingercek]

great video. legend:)

[@dataschool]

Thank you!

[@HealthyFoodBae_]

Hello, I have 2 large datasets and want to compare time differences by seconds for instance. I want to Group-by a certain column first, and then see the time differences or duration for a certain action. Can I do this in Python

[@dataschool]

I'm sure you can, but it's hard for me to say how off-hand. Sorry!

[@ssam7195]

Thanks for your videos. I have data df['DateTime'] as '02/12/2019 11:00:00' , ...02/13/2019 23:00:00'., 02/13/2019 24:00:00' in column 48hrs of hourly data. While plot only ' 02/XX/2019 24:00:00' ( only 24:00:00 ) hrs point missing on graph. :( How to fix this? Please support me.

[@mardozaaaa]

What if column 'Time' has a 2 words name? How do you call it on the expression ufo.Time.dt.week? I tried with ufo.ufo['Two Words'].dt.week but it doesn't work.

[@dataschool]

ufo['Two Words'].dt.week

[@Cam-p2z]

These are the first videos I look for when I have pandas questions.

[@dataschool]

Thanks!

[@hardikvig2723]

Hey Kevin.. Could you please make a 2-3 hr video on ML project covering all the tools from sklearn based on real life dataset let’s say cancer prediction model or something like that trying to combine all the elements important for making a model. That would be a great help.

[@dataschool]

Thanks for your suggestion! I suggest checking out my course: courses.dataschool.io/building-an-effective-machine-learning-workflow-with-scikit-learn

[@varshadevgankar8242]

hi, i want a function in python that identify which column have date in them??

[@anilsangani8862]

AB['Date']=pd.to_datetime(AB.Date) i am getting error Unknown string format: TOTAL my format is dd/mm/yyyy in column Date of a data frame named AB showing dtype as object .

[@teklehaimanotaman3150]

Hi Kevin, lets say you have a date column containing only hour, minute and second. While changing into date format using pd.to_datetime, it added automatically years and days. How could keep only hours, minutes and seconds? Thank you.

[@dataschool]

Great question! I can't remember off-hand, but I think you need to use a pandas timedelta object instead of a datetime object.

[@benjerome9738]

I have a column containing times in the format hour, minute, second (e.g. 00:24:43) and are currently an 'object' type. How can I calculate the average time for this column? I have tried converting to a pandas datetime but this throws up several different errors when I try to calculate the mean? Also, I have three separate groups (full match, T1 and T2) how can I use groupby to figure out the mean times for each group? Thanks

[@mashroomguo2124]

ConnectionResetError: [WinError 10054] 远程主机强迫关闭了一个现有的连接。

[@dataschool]

Sorry, I won't be able to help you troubleshoot! Good luck!

[@j4y0785]

Hi Data School, Just a quick question, I am still new at this and my apologies if this question as been answered already, how would I go about changing multiple columns to datetime? Thank you.

[@dineshkundu4623]

If I want specific dates in train data and remaining in test data how can we do tht

[@akashderasree6716]

How can I add a number of days column like 10 20 120 to a date column in 2020-06-27 format???

[@dataschool]

I believe you will need to use a timedelta

[@rodrigoviverosphoto]

This is EXACTLY was a looking for. I love you.

[@dataschool]

Awesome!

[@dineshkundu4623]

Bro can we use this date to split data into train and test date if yes how we can do that

[@nousername2520]

@Data School Do you have any video on how to fill missing dates with zero (desired number) in a large csv file?

[@dataschool]

kzbin.info/www/bejne/nHSwo4KVi9-Ygpo

[@marco4570]

That bonus is what I needed. Thank you so much!

[@dataschool]

You are so welcome!

[@rashmikoparde826]

I have a csv file which consist of placeid and date .I want to calculate in a particular month how many times is the particular place visited..how this could be done in python code..Please help @ Data School

[@dataschool]

Perhaps you could store the 'month' attribute from the 'date' column as a new column, then filter the DataFrame by 'placeid' column, and then check the shape of the DataFrame. This video might be helpful to you: kzbin.info/www/bejne/aHKpeIOag9NnfK8

[@python_by_abhishek]

Hi Kevin I have a doubt.. I have 1 column in which time is 42368.149155 When I convert it into year, days, month, hrs, min, sec I am getting 1970-01-01, 17:22:13.453068 I read that 1970 is the default year. How can I convert it into some other year, say 2016 or any other year. Kindly help.

[@dataschool]

You might need to adjust the "unit", see examples here: pandas.pydata.org/docs/reference/api/pandas.to_datetime.html

[@python_by_abhishek]

@@dataschool Thank You Kevin. Got it now.

[@myselfandpesit]

Thanks for the tutorial. I have a question. How can i find unique items under a given column as some could have been repeated?

[@hiteshkumavat6335]

The videos is with very nice explanation .I am getting the time data only in hour:min:sec format and when i convert it from object to time then it gives also the current date with time stamp. I want to fill the missing seconds values so is there any other function available ?

[@dataschool]

I'm not sure off-hand, I'm sorry!

[@akahn8311]

thanks for the tutorial. is there any way to change the year column meaning change the year series. In year series - starts with 1930 and then go on till 1933, how to change this to 2013 to 2016 in the csv file ? thanks lot for the time and help ! cheers

[@carloseduardocorreacoimbra4040]

Kevin, could we define a ts_min and ts_max, and select the events during this interval?

[@saisrinivasbikkavalli5013]

Hi actually I had a column name time consumed which contains time in hours minutes and seconds but some of them contains only min and seconds.can u please tell me how can I get the time consumed rows within 30 sec

[@dataschool]

Sorry, I won't be able to help... good luck!

[@kirahman2]

thank you so much for this video, it saved me so much time, thank you. Wow. so good.

[@nanaamapinto2352]

This is the head of my data frame Station Date Precipitation 0 TA00016 1/1/2021 0 1 TA00016 1/2/2021 0 2 TA00016 1/3/2021 0 3 TA00016 1/4/2021 0 I run the code you presented for changing the date column to a pandas date time format and this is the feedback i receive as shown below File "C:\Users\pinto009\.spyder-py3\temp.py", line 26, in rain["Date"]=pd.to_datetime(rain.Date) File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\generic.py", line 5465, in __getattr__ return object.__getattribute__(self, name) AttributeError: 'DataFrame' object has no attribute 'Date' What could i be doing wrongly. Will be glad to hear from you soon. Thanks

[@dataschool]

Maybe the 'Date' column has a space before or after? Try running rain.columns to check.

[@nanaamapinto2352]

@@dataschool Thanks very much it worked now. There was a space after the name indicated for the column. Thanks again

[@dataschool]

Great to hear!

[@hokapokas]

i have a question panda time series where all my columns are dates and i want to find the hrs spent by employee. Pls help me with your Github id so can i can post details there. please help

[@dataschool]

I'm sorry, I won't be able to help. Good luck!

[@TorIvanBoine]

awesome videos. been watching quite a few now. So, I'm playing with my gpx running data. And I'm trying to convert the duration of my runs so I can plot them. But I just fail. How would you convert ints like 33:28 and 01:44:42 so it would be understood as 33 minutes and 1 hour 44 minutes and so on?

[@dataschool]

Glad you like the videos! As for your question, it seems like extracting the datetime attributes (hours, minutes, seconds) and then doing the math with those attributes would solve your problem. Hope that helps!

[@somnathmalik7200]

Thank you, it helped a lot

[@gadellikrishna1826]

How to subtract to times and output required only time with out days ?

[@dataschool]

Not sure off-hand, sorry!

[@raghwendranarayanshandilya4525]

Hi, I liked your way of presentation. It is very precise and nice. I would like to see any tutorial where you explain about taking statistics (mean, median, etc.) of data by filtering any column (say date-time column).

[@jackf3499]

Exception has occurred: AttributeError 'DataFrame' object has no attribute 'Time' I keep getting this error when I try to use: df['date'] = pd.to_datetime(df.Time)

[@dataschool]

Perhaps you read the file incorrectly or have a typo somewhere?

[@AnalyticsMaster]

Hi Kevin, Just like we have 'weekday_name' to know whether it is 'Sunday', 'Monday' etc... what should we use to find the 'month name' ? I serched for help in pandas doc, but not able to find. Kindly suggest.

[@dataschool]

Great question! I'm not sure if there is a built-in way in pandas to do this. I would probably write my own code to do this using the map method, explained here: kzbin.info/www/bejne/hpDUYaehjtapic0

[@dikshaanand5531]

thanks for the video it is very helpful. It is very conceptual. though I have a doubt like---- what to do if I want to create another column consisting of days from TIME column to today's date i.e. today's date for example 2019-01-24 subtracted from the values consisting in TIME column?

[@dataschool]

I'm sorry, I don't know off-hand... good luck!

[@SatishKumar-yz4tn]

Thank you so much in taking time to explain so nicely.

[@dataschool]

You're very welcome!

[@enian82]

For some reason This through s an error ufo.Time.dt.weekday_name but ufo.Time.dt.weekday this works any idea.... Thank you

[@galymzhankenesbekov2924]

thank you for your video! I have a question. for if one of your columns has the following values,like 1.x,y,d,z,w 2.x,t,y,d,z 3,x,y,t,d,w and so on. numbers mean different rows So, i need split the dataFrame into x,y,d columns(by values). P.S. so how to tell the program that if particular value within a selected column satisfies my desire I need to take that value and put them into seperate column

[@dataschool]

Sorry, I don't quite understand your question... good luck!

[@anilkushwaha7164]

Please make a video for Cleaning of "FINANCIAL DATA " Your Tutorial is very Lucid and informative

[@dataschool]

Thanks for your suggestion!

[@Camerooony11]

The code (ufo.Time.max() - ufo.Time.min()).days returned 25781L. Might be a dumb question but what does the L at the end mean?

[@dataschool]

It means "long integer". It's a technical detail that doesn't matter to end users like you and me :)

[@gcm4312]

DataFrame.resample() is also a very, very useful feature in Pandas for working with time series.

[@dataschool]

Agreed!

[@pranavmendiratta3166]

loved the tutorial, cleared my doubts! I like how you explain so patiently.

[@christopherclark2205]

Phenomenal description of working with dates and times in Pandas, very helpful.

[@sdoken]

It is interesting that while ufo.dtypes tells you the type as “datetime64[ns]”, when you look at each element individually type(ufo.loc[0, ‘Time’]) returns a Timestamp object. Seems a bit inconsistent? No big deal, I’ll just have to get used to it. With timedelta however, both the column and the individual elements are of type timedelta.

[@dataschool]

Yes, it takes some getting used to.... datetime is the type of the Series, timestamp is the type of the element, and timedelta is the type of a time range!

[@timpinckney]

Great explanation Kevin. Do you have any videos exclusively on matplotlib?

[@dataschool]

I don't... sorry! I'll consider them for the future!

[@timpinckney]

Thanks Kevin

[@pickarpit]

Thanks a lot Sir ! It was very Helpful. Another thing i am stuck with is how to prepare a dataset for a Machine Learning Model which contains GPS Co-ordinates. Any Guidance or Suggestions?

[@dataschool]

That's really a question of feature engineering, meaning that you need to decide how to best represent the data so that a model can learn from it. The best approach depends on the problem you are solving, but my default approach would be to keep the latitude and longitude as separate features, and represent them in decimal format (not degrees/minutes/seconds). Hope that helps!

[@udaykiran-rh8es]

could you please make videos on Matplotlib ,seaborn and power bi also for data analysis . I like your way of teaching.

[@dataschool]

Thanks for your suggestion!

[@udaykiran-rh8es]

@@dataschool EDA also

[@rahulkambadur147]

Hello, In My date column has the date of month missing. how do I add the date to the existing column Ex: My column is 04-1982 (which is not in date format) and I want to make it 30-04-1982.. and want to repeat for all the other sections.. please help. and how to add a date if there is no date available

[@dataschool]

I'm sure there's a string method that can help: kzbin.info/www/bejne/mKDJknZmfsieftE&index=12&list=PL5-da3qGB5ICCsgW1MxlZ0Hq8LL5U3u9y

[@mahendraprajapati3359]

how to retrive the data between( time interval as minutes) from dataframe

[@dataschool]

I'm sorry, I won't be able to help you... good luck!